How to review a data analysis

July 18, 2016

Writing good code is hard, writing a good analysis is harder. Peer-review is an essential tool to fight repetitive errors, omissions and more generally divulge knowledge. I found the use of a checklist to be invaluable to help me remember the most important things I should watch out for during a review. It’s far too easy to focus on few details and ignore others which might be catched (or not) in a successive round.

I don’t religiously apply every bullet point of the following checklist to every analysis, nor is this list complete; more items would have to be added depending on the language, framework, libraries, models, etc. used.

  • Is the question the analysis should answer clearly stated?

  • Is the best/fastest dataset that can answer the question being used?

  • Do the variables used measure the right thing (e.g. submission date vs activity date)?

  • Is a representative sample being used?

  • Are all data inputs checked (for the correct type, length, format, and range) and encoded?

  • Do outliers need to be filtered or treated differently?

  • Is seasonality being accounted for?

  • Is sufficient data being used to answer the question?

  • Are comparisons performed with hypotheses tests?

  • Are estimates bounded with confidence intervals?

  • Should the results be normalized?

  • If any statistical method is being used, are the assumptions of the model met?

  • Is correlation confused with causation?

  • Does each plot communicate an important piece of information or address a question of interest?

  • Are legends and axes labelled and do the they start from 0?

  • Is the analysis easily reproducible?

  • Does the code work, i.e. does it perform its intended function?

  • Is there a more efficient way to solve the problem, assuming performance matters?

  • Does the code read like prose?

  • Does the code conform to the agreed coding conventions?

  • Is there any redundant or duplicate code?

  • Is the code as modular as possible?

  • Can any global variables be replaced?

  • Is there any commented out code and can it be removed?

  • Is logging missing?

  • Can any of the code be replaced with library functions?

  • Can any debugging code be removed?

  • Where third-party utilities are used, are returning errors being caught?

  • Is any public API commented?

  • Is any unusual behavior or edge-case handling described?

  • Is there any incomplete code? If so, should it be removed or flagged with a suitable marker like ‘TODO’?

  • Is the code easily testable?

  • Do tests exist and do they actually test that the code is performing the intended functionality?

Written by Roberto Vitillo

Want to learn how to build scalable and fault-tolerant cloud applications?

My book explains the core principles of distributed systems that will help you design, build, and maintain cloud applications that scale and don't fall over.

Sign up for the book's newsletter to get the first two chapters delivered straight to your inbox.

    I respect your privacy. Unsubscribe at any time.