This week’s reading happened almost all in one day, when I took time off work to recharge. I read articles and blog posts on COVID-19 models, focusing on the question of what it means to model something. There are also a couple of garden variety engineering & data science posts.
Leading with the Unknowns in COVID-19 Models
Author: Dr. Jessica Hullman, in Scientific American
I found this from Twitter and it caught my eye; I’ve read one of the papers out of Dr. Hullman’s lab before about visualizing uncertainty, so I was excited to see this pop up.
Hullman’s thesis is that there are lots of sources of uncertainty implicit in all of the COVID-19 models being circulated:
What worries me as an expert in reasoning under uncertainty is a more difficult type of uncertainty: the uncertainty that arises from the many unknowns underlying COVID-19 data and models. We can’t easily quantify this uncertainty, and it is easy to overlook, since it is not conveyed by model estimates alone.
The most obvious example is that for a long time, the number of cases was mostly a function of testing capacity. This is why Washington led the nation in cases for a couple of weeks. A larger number of cases, then, does not necessarily equate to a larger risk.
Other sources of uncertainty, Hullman argues, come from assumptions made about disease transmission or from the unpredictability of human behavior. Every model accounts for these differently, and these assumptions rarely make it into public-facing results. It’s important to recognize that these models are based on limited information, and that more reliable data will emerge over time.
Clear presentation of uncertainty can make model estimates seem less reassuring, but can prevent people from blaming the forecaster or the scientific enterprise itself, when, as we should expect, the model is wrong. Trading public trust in science in the future is not worth feeling more assured in the short term, no matter how much we seek to eliminate uncertainty.
My thoughts: I’ve seen similar ideas from various folks in data science & statistical modeling Twitter circles. It’s clear that a lot of the nuance behind the COVID-19 forecasts is being lost, and it leads to interesting questions about how to best communicate information.
And predictably so! It’s much easier to remember “Chicago was supposed to peak on Saturday” than remember “… given that social distancing continues until the end of May,” and all the other assumptions implicit in the model (which model?) that produced it.
This raises lots of interesting questions. How do we best communicate the assumptions behind these models? Which assumptions are necessary, and which are acceptable to leave out (bearing in mind the space constraints of things like press releases, headlines, and tweets)? How do the answers here change when communicating to different audiences?
Models are not predictions meant to be right or wrong, either. There are feedback loops implicit in how they are used: if a governor uses a model to inform statewide policy, and the policy changes people’s behavior, then of course the model will be “wrong.” I have begun thinking of models as “explorations of dynamics,” rather than “predictions” (see: xkcd 2289).
Just a tweet by Richard McElreath
People who make model-based forecasts don’t (usually) think it is possible to get the model right, in a situation as realistic and complex as an epidemic. So we study many models and try to understand the structural risks of the situation that are common across models. Forecasts are not predictions, but ways to influence policy and behavior. The public and politicians engage with predictions in a very different way, and I think this is a dangerous gap between the way people like me think of complex systems and the way the public thinks of the purpose of forecasts (to predict the future).
I largely agree, but also have some reservations about the idea that forecasts aren’t supposed to be “right.”
Public health experts and researchers are able to unpack all the nuance behind the use of the word “right,” and indeed I regularly speak at work about how “all our models are wrong.” I’m confident in doing so when my audience is other data scientists, but this isn’t the sort of thing that we can say to clients (yet).
I suspect that the public may hear “the model isn’t supposed to be right” as “the model is useless.” McElreath gets at this in his last sentence when he speaks of forecasts as predicting the future. Perhaps we might reframe this by saying that modeling the numbers isn’t as important as modeling the mechanisms of the spread or exploring what different mechanisms (and interventions) might mean.
Another Twitter thread by Carl Bergstrom on visualization
Bergstrom writes:
- When plotting epidemic curves or death totals, should we divide by population size? Here on twitter this question has generated a lot more heat than light.
The answer is a bit subtle and so while I’ve tweeted about this before I want to address it in more detail.
The answer is complex (and makes me annoyed at Twitter’s platform for being awful at conveying anything with nuance). One thing that stands out, though, is that if you divide by population size, then you should start with a constant fraction of the population, not a constant number of cases. The thread is interesting otherwise, too.
Hacking for coronavirus needs more humility
Author: Nik Marda (at The Stanford Daily)
Technologists are engaging with the pandemic in new ways, often good:
- Bill Gates is funding factories to develop candidate vaccines and supporting at-home testing
- Zuckerberg is promoting helpful information at the top of the News Feed
- Google and Apple are partnering on contact tracing
On the other hand, Elon Musk is on the record saying “The coronavirus panic is dumb” (that was an awful tweet at the time, and aged even more poorly) and that Tesla would make ventilators “if there is a shortage” (there was a shortage, literally at that moment, and had been for weeks). Others in Silicon Valley who have acted and spoken irresponsibly, too.
This, Marda argues, is a symptom of Silicon Valley saviorism:
The [Stanford] administration proposed a “Hacking Hate” event, where students had five hours to research and propose innovative ways to address racial hate and violence. In response, Jasmine Sun ’21 articulated how racism is a complicated, multi-pronged problem that can’t be solved by “hacking.”She proceeded to describe Silicon Valley’s saviorism problem, where “inexperienced innovators parachute into a community they don’t understand to build products that no one needs, claiming a resume line and virtue signaling along the way.”
The behavior of many technologists during this pandemic has been downright irresponsible. It takes a profound amount of arrogance to believe that “hacking together solutions” will be anything but a profound waste of time (as a Texas group tried to do).
To all the software engineers, data scientists, and non-epidemiologists interested in modeling COVID-19 or hacking together solutions: step back. It’s not just hard to verify if these “solutions” help; it’s outright dangerous.
Other articles
The metric we need to manage COVID-19 is a blog post (with accompanying notebook) by Kevin Systrom (founder of Instagram) implementing a paper by Bettencourt & Ribiero (2008). The goal is to estimate R0 at a state level using a multilevel Bayesian approach. This was a fun read!
Yet another Twitter thread by Carl Bergstrom critiques a Medium article by Aaron Ginn, which Marda above cited as one of the most glaring examples of Silicon Valley saviorism. “You don’t need a special degree to understand what the data says and doesn’t say,” Ginn writes arrogantly, later claiming that his experience in making products go viral helps him to understand actual diseases. The thread is a great breakdown of the problems with articles like this.
Social media companies must flatten the curve of misinformation by Dr. Joan Donovan (research director of the Shorenstein Center on Media, Politics and Public Policy at Harvard) is a Nature World View article discussing how the need for social media companies to combat misinformation is more critical than ever. Her central claim is:
Researchers and policymakers can — and must — make sure social-media companies identify, implement and evaluate ways to curtail the spread of dangerous misinformation, and that independent researchers can assess this work.
I’m disappointed in this article, because while Donovan states that “preventing misinformation requires curating knowledge and prioritizing science,” there’s no suggestion of how to do this in practice. I trust her expertise, and believe in the need to combat misinformation, but without anything concrete the article offers little substance.
Markov Models - Essays on Data Science is an essay by Eric Ma that motivates and develops the theory behind various classes of Markov models, then implements a Bayesian inference procedure for them in PyMC3. This is a thorough treatment that really helped me to understand some of the concepts behind these models, and I’d recommend it as a great resource for anyone trying to learn more.
Code that is “too simple to break” is yet another Twitter thread by David Tanzer talking about, when he teaches TDD, he will sometimes not test code that is too simple to break. He defines this as code where (1) the chance of the code breaking is low, and (2) if it were to break, the failure would be noticed immediately. This is likely “write-only” code (written once and never touched again), where you can instead test the callers of this code, and code that is used so frequently (like loading or writing to a file) that would be tested as part of any test elsewhere in the application anyway.