I've been evaluating some popular cloud systems lately and unsurprisingly enough, I'm finding it really hard to make a final call on a set of results. Have I tuned the systems I'm comparing appropriately for the workloads I'm subjecting them to? Are the workloads reasonable? Is the system itself the bottleneck or is it a set of external factors? There were so many questions piling up that it's made me rather skeptical about system evaluations I find in papers today.

So I've picked up this book, and found this minimal checklist to go through to avoid common mistakes when conducting a performance evaluation.

1. Is the system correctly defined and the goals clearly stated? 
2. Are the goals stated in an unbiased manner? 
3. Have all the steps of the analysis followed systematically? 
4. Is the problem clearly understood before analyzing it? 
5. Are the performance metrics relevant for this problem? 
6. Is the workload correct for this problem? 
7. Is the evaluation technique appropriate? 
8. Is the list of parameters that affect performance complete? 
9. Have all parameters that affect performance been chosen as factors to be varied? 
10. Is the experimental design efficient in terms of time and results? 
11. Is the level of detail proper? 
12. Is the measured data presented with analysis and interpretation? 
13. Is the analysis statistically correct? 
14. Has the sensitivity analysis been done? 
15. Would errors in the input cause an insignificant change in the results? 
16. Have the outliers in the input or output been treated properly? 
17. Have the future changes in the system and workload been modeled? 
18. Has the variance of input been taken into account? 
19. Has the variance of the results been analyzed? 
20. Is the analysis easy to explain? 
21. Is the presentation style suitable for its audience? 
22. Have the results been presented graphically as much as possible? 
23. Are the assumptions and limitations of the analysis clearly documented?