I have come across some poorly written “performance” tests in the last couple of month, persuing my little hobby improving md5 algos for .NET. Here are some basic things to always have in the back of your head while doing performance optimizations and tests in .NET (and other languages):

NEVER assume you know in advance what will be faster or slower

This is the first and probably only RULE in optimization. Don’t even assume that unsafe code must be faster. Actually that is quite wrong since the built-in optimizer in Microsoft’s .NET is very good at optimizing regular stuff. That means: ALWAYS test your assumptions and “optimizations”.

Always have a spinup cycle that does exactly the same

If you are testing an algorithm or a specific method in your class, always call it at least once BEFORE actually starting your test. This will get rid of the compilation that is done while executing a codepath for the first time. .NET is an interpreted language. Don’t underestimate the time it take even today’s cpus to do that initial step. I am talking here about hundrets of seconds. Be aware that SQL 2005 does some first time compilation of T-SQL statements as well that will not be visible during productive usage.

Be aware of caching

While the “spinup cycle” is very important one has to be absolutely sure that no caching will interfere the process of the “second call”. Especially be careful of the earlier stated caching and compiling of T-SQL statements! This might result in way faster test times than your actual production code will ever run.

Utilize statistics

Never assume you know everything about the test by just performing it once. There are a lot of factors that play into the “actual performance right now” on a multithreaded GUI system. Try creating the environment that will exist during every day usage of the production code. Sum over multiple runs to get more accurate readings of the actual performance during normal operation.

Be aware of Debugging

Don’t run your performance test by hitting F5 or CTRL+F5 in Visual Studio. This will get you readings that are very different from you actual production performance. First of all you probably used “Debug” as compiler settings. Plus Visual Studio will watch variables and breakpoints for you by attaching to the process of you test. This will slow down execution time considerably.

Be aware of Console output

Writing to console or updating textfields is very slow and therefore no diagnostic output should be produced during the actual testruns. Always save that for time after/in between testruns. If you want to write to the console between testruns concider doing a Thread.Sleep(50); before starting the next run.

Frameworks and compilers differ

Never assume that the optimization you made will be the best on all platforms and/or compiler. In the .NET world there is the Microsoft .NET Framework and the Mono Framework. Their compilers are very different when it comes to optimization philosophy, focus and maturity. Plus especially the mono interpreter allows for a huge range of additional runtime optimizations. Different code optimizations will gain or loose, depending on compiler and interpreter!

That’s the basics. I could get into “testing the empty test-loop”, etc. but that’s for the hardcore testers. These points above though are very basic, but must be obeyed, if the test timing should be anything close to the actual production timing.

Happy testing. :)

Read on: