Erich Gamma did a keynote presentation on Eclipse 3.1 news. One of the things I remembered, was his description on how the team made sure that Eclipse 3.1 got faster in respect to 3.0. Basically, they set up a series of unit tests, which measured the runtime of some code and compared it to some reference timing (pseudocode):
public void testFoo()
So this fitted in the framework, one is used to.
More important, they did learn a lesson: It is not enough to measure things, but to have someone (e.g. the community) to kick ass if there is too much red in the results.