‘Encouraging’ study examines static typing and quantifying detectable bugs in JavaScript

James is editor in chief of TechForge Media, with a passion for how technologies influence business and several Mobile World Congress events under his belt. James has interviewed a variety of leading figures in his career, from former Mafia boss Michael Franzese, to Steve Wozniak, and Jean Michel Jarre. James can be found tweeting at @James_T_Bourne.

Are you a proponent of static typing or dynamic typing in JavaScript, or indeed another language? A new study from researchers at University College London (UCL), alongside Microsoft, put a couple of products to the test – with pretty solid results.

The researchers tested Flow, Facebook’s static type system, and TypeScript – from Microsoft, of course – and found that, at a conservative estimate, each product can spot approximately 15% of bugs which would otherwise have ended up in committed code.

The importance of static typing over dynamic typing rests on the ability for code to be checked at compile time rather than runtime – at which point it is often too late. As Anders Hejlsberg, technical fellow at Microsoft, noted when introducing TypeScript back in 2012, writing application-scale JavaScript is ‘hard… [it] was never really designed to be a programming language for big applications’, and thus lacking static typing.

As the report introduction puts it: “Proponents of static typing argue that it detects bugs before execution, increases runtime efficiency, improves program understanding, and enables compiler optimisation. Dynamic typing, its advocates claim, is well-suited for prototyping, since it allows a developer to quickly write code that works on a handful of examples without the cost of adding type annotations.”

Yet as the research shows, it may be worth taking the effort to add type annotations after all.

Having examined the total number of closed issues for JavaScript projects on GitHub – 3,910,969 as of August 2015 – the researchers plumped instead for 400 studied bugs it tested on Flow and TypeScript, having added type annotations (below).

Of the 400, Flow successfully detected 59 and TypeScript 58 at first analysis, although after labelling all the bugs as either detectable or undetectable under Flow and TypeScript, the number rose to 60 for each. Factoring in the confidence level, the eventual percentage of between 11.5% and 18.5%, with the mean at 15%. According to the data, the vast majority of the bugs picked up were done so by both programs, with only three on each side only being preventable by Flow and TypeScript alone.

The study aimed to focus on the effects of annotations rather than the annotator – much in the same way surgery trials seek to draw conclusions over the surgeries rather than the surgeon – with the researchers opting for methods to negate ‘learning effects’ during the tests.

Ultimately, the question is one of whether a 15% reduction in bugs is worth the extra work in annotation. As Adrian Colyer put it in a blog post assessing the study, Flow’s more lightweight approach is potentially better suited to dynamic typing advocates, with TypeScript a better option for strong-typing advocates coming from other languages. The researchers added the results from the study were “encouraging.”

You can read the full report here (pdf, 12 pages).

View Comments
Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *