By Sue Denim (pseudonym) – LockDownSkeptics.com:
Imperial finally released a derivative of Ferguson’s code. I figured I’d do a review of it and send you some of the things I noticed. I don’t know your background so apologies if some of this is pitched at the wrong level.
My background. I have been writing software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. See more in linked article.
Summary Statement. Speaking for myself, based on what I’ve seen in this situation, my recommendation is that ALL ACADEMIC EPIDEMIOLOGY BE DEFUNDED. This sort of work is best done by the insurance sector. Insurers employ modellers and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.
Key Details:
- It isn’t the code Ferguson ran to produce his famous Report 9. A request for the original code was made 8 days ago but ignored, and it will probably take some kind of legal compulsion to make them release it. Clearly, Imperial are too embarrassed by the state of it ever to release it of their own free will, which is unacceptable given that it was paid for by the taxpayer and belongs to them.
- What the model is doing is best described as “SimCity without the graphics”. It attempts to simulate households, schools, offices, people and their movements, etc.
- Unpredictable. The documentation says the model is “stochastic. Multiple runs with different seeds should be undertaken to see average behaviour. ” ‘Stochastic’ is just a scientific-sounding word for “random”. That’s not a problem if the randomness is intentional, i.e. the randomness is derived from a starting “seed” which is iterated to produce the random numbers. BUT investigation reveals that this code produces critically different results, even for identical starting seeds and parameters.
- Buggy. Their code is so deeply riddled with bugs and they struggled so much to fix them that they got into the habit of simply averaging the results of multiple runs to cover it up… and eventually this behaviour became normalised within the team.
- No tests. In the discussion of the fix for the first bug, Imperial state the code used to be deterministic in that place but they broke it without noticing when changing the code. Regressions like that are common when working on a complex piece of software, which is why industrial software-engineering teams write automated regression tests. The Imperial code doesn’t seem to have working regression tests.
- Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.
