Lessons learned from the COVID-19 pandemic: a modeling perspective

By: Feline Velthuis

During the COVID-19 pandemic, all eyes were on the mathematical models that would tell us how we could control the outbreak, how occupied hospitals would be and if we could celebrate Christmas together. Worldwide, lots of different models were used to make predictions on the development of the outbreak and the impact of measures taken by the government. In the Netherlands, policy decisions on how to control the pandemic were largely based on the model predictions of the Institute for Public Health and Environment (RIVM). These were presented by the chair of the Outbreak Management Team (OMT), during the regular technical briefings with the government¹. Now that measures in the Netherlands are lifted, we can reflect on the models used by the Dutch government to take the lessons learned from this pandemic to a potential next one.

For such reflection, the five principles of modeling developed by the Neglected Tropical Disease (NTD) consortium can be used. These have been set up as a guide to communicate the quality and relevance of models to the stakeholders, such as the government and policy makers and are based on a literature review on existing modeling guidelines². The five principles are not only applicable to tropical disease models, but to all mathematical models aiming to support policy decision-making, including the COVID-19 models.

Principle 1: Stakeholder engagement

The most important aspect when providing relevant model outcomes for decision making is stakeholder engagement. Stakeholders such as policy-makers, behavioral scientists, economists and psychologists should be invited to participate in the modeling process. After all, decisions should be made keeping the impact of the decision on the economy and mental well-being of people in mind.

One of the aspects of how fast a disease spreads is people’s behavior; whether or not they stay at home or keep distance from each other. Wise et al.³ showed that individuals who perceived a high personal risk, engaged in protective behaviors, such as hand-washing and social distancing. This behavior is critical to reduce transmission of the disease. However, it might be the case that behavior changes over time and people, for example, follow the rules less strictly. From the models of the RIVM it is unclear if such behaviors were implemented in their model, for instance to predict the number of occupied ICU beds⁴. For future pandemics it could be valuable to invite stakeholders, such as behavioral scientists, to take part in the modeling process to ensure that such aspects are modeled. Figure 1 shows one of the topics from behavioral and social science that can provide valuable insights for modeling COVID-19.

Principle 2: Complete description of the data

The second principle states that it should be clear which datasets are used to allow others to assess the data quality and the relevance of using that dataset. Furthermore, complete descriptions of datasets are important for building confidence in the assumptions made in the model. From the slides of the technical briefings presented by the chair of the OMT, it is clear what the sources of data are to inform the models¹. Examples of data sources the RIVM uses for their model are hospital data, data from COVID-19 test locations, vaccination rates and their own research⁶. Examples of the type of data that the RIVM uses to inform their model are readily available on their website. However, due to privacy issues, part of the datasets is replaced by anonymised data⁷. Therefore, the available data do not contain the exact same information as the RIVM uses for their models. Still, it is clear what data the RIVM used.

Principle 3: Complete model description

In line with the second principle, the model should be completely reported or published in a peer-reviewed journal in order for other researchers to rerun the analysis. The RIVM uses different models for predicting different aspects of the pandemic; one model is for example used for predictions on the intensive care occupancy. Although the RIVM reports the type of model they use as well as anonymised data, exact reproduction of this model is not possible with the information available⁷. According to the RIVM, their model is not fully described because that would require sharing personal information, which is not allowed⁷. Therefore, other researchers cannot track down what “settings” were used for the predictions of different scenarios⁴. This hampers discussion and comparison with other models which is something that could be encouraged in the future. In the UK, privacy issues were addressed by signed agreements to be able to share all data⁸, which could have been considered in the Netherlands as well.

Principle 4: Communicating uncertainty

For stakeholders making policy decisions, it is important to know how much uncertainty the model contains in order to make a balanced decision. If there is a lot of uncertainty, different decisions might be made, in comparison to when there is only little uncertainty. For example, if you have no idea how much damage a virus will cause, you might take more drastic measures than when you have become familiar with the virus. In the technical briefings, uncertainty was communicated well, as can be seen in Figure 2. Here, the RIVM predicted the impact of different combinations of measures on the number of occupied ICU beds and showed the 95% confidence intervals. In their description of the model, the RIVM also explains how the uncertainty of parameter values was calculated and used to weight the parameter combinations in the model.

Model comparison is another way that provides more insight into uncertainty. If different modelers are asked to come up with a COVID-19 model, they will use different methods to develop a model, each based on a different perspective. These models will all have their own uncertainties which gives a better understanding of the different scenarios modeled; if all modelers draw the same conclusion from their model, the government has a better argument to favor a specific scenario. This is more difficult when largely one model is used to advise the government, which was the case in the Netherlands. Contrastingly, in the UK there was a diverse range of external (outside government) modeling groups who were extensively involved in the surveillance and reporting of the spread of the disease. A subgroup of the Scientific Advisory Group of Emergencies (SAGE), similar to the OMT in the Netherlands, combined the information from the various models to support decision making by the government⁸. For a next pandemic it might be favorable for the government to ask different modeling groups to develop a model, which all are presented by the OMT.

Principle 5: Testable model outcomes

To investigate whether the impact of the measures was as predicted, data should be compared with the predicted scenarios. In this way, it can be evaluated whether the scenarios were too optimistic or too pessimistic and whether the model should be adapted. Towards the end of December 2021, the transmission model of the RIVM showed a peak in the number of occupied ICU beds due to the new variant Omicron. One of the assumptions was that Omicron would cause as much disease as Delta, the previous variant. Based on these predictions, the government decided that a lockdown was necessary once more. However, looking back, the number of occupied ICU beds stayed low and didn’t reach the expected number of occupied ICU beds during lockdown, by far (black line Figure 3). Due to this discrepancy, it would have been valuable to, with the new information on the Omicron virus, model the scenarios again and evaluate what adaptations to the model were necessary to predict the scenario that became reality. This is important to prevent certain measures such as a lockdown being taken while less drastic measures would suffice to flatten the curve.

Lessons learned

According to this reflection of the COVID-19 models used in the Netherlands based on the five principles of the NTD consortium, one can conclude that several lessons were learned. First, it is important to include stakeholders such as behavioral scientists in the modeling process. This can be encouraged when the government and RIVM ask experts from other disciplines to participate in the modeling process. Furthermore, effort should be made to make all model code better available to the public. This facilitates the reproduction of the RIVM models by others. Also, the government should ensure that it is informed by models from different research groups, which will provide more insight into the uncertainty in model outcomes. Finally, evaluation of the scenarios is valuable for future predictions and should take place when the actual development is different from the prediction. Taking these lessons into account during a next pandemic will lead to more relevant and high-quality models to support policy decision-making and to better predictions on how we can control the spread of the disease.

About the author

Further reading

1. Tweede Kamer (n.d.). Coronavirus. Accessed on 30-05-2023. https://www.tweedekamer.nl/kamerleden-en-commissies/commissies/volksgezondheid-welzijn-en-sport/thema-coronavirus
2. Behrend MR, Basáñez MG, Hamley JID, et al. (2020) Modelling for policy: The five principles of the Neglected Tropical Diseases Modelling Consortium. PLOS Neglected Tropical Diseases 14(4): e0008033. https://doi.org/10.1371/journal.pntd.0008033
3. Wise T, Zbozinek TD, Michelini G, et al. (2020). Changes in risk perception and self-reported protective behaviour during the first week of the COVID-19 pandemic in the United States. Royal Society open science, 7(9), 200742. https://doi.org/10.1098/rsos.200742
4. Houtekamer C, Berkhout K. (2022, 19 January). Bereken meer dan besmettingen – en nog twee ideeën voor een bredere coronastrategie. NRC. https://www.nrc.nl/nieuws/2022/01/19/oproep-modelleurs-nederland-heeft-recht-op-een-second-opinion-a4081121
5. Bavel JJV, Baicker K, Boggio PS, et al. (2020). Using social and behavioural science to support COVID-19 pandemic response. Nature Human Behaviour, 4(5), 460-471. https://doi.org/10.1038/s41562-020-0884-z
6. Backer JA, Mollema L, Vos ER, et al. (2021). Impact of physical distancing measures against COVID-19 on contacts and mixing patterns: repeated cross-sectional surveys, the Netherlands, 2016–17, April 2020 and June 2020. Eurosurveillance, 26(8), 2000994. https://doi.org/10.2807/1560-7917.ES.2021.26.8.2000994
7. RIVM. (2021). Beschrijving transmissiemodel berekening zorgbelasting. https://www.rivm.nl/documenten/beschrijving-transmissiemodel-berekening-zorgbelasting
8. UK government. (2022). Independent report, Chapter 5: modelling. Assessed on 06-04-2023. https://www.gov.uk/government/publications/technical-report-on-the-covid-19-pandemic-in-the-uk/chapter-5-modelling#fn:23
9. RIVM (2021, 13 January). Presentatie van dhr. Jaap van Dissel (directeur van het Centrum voor Infectieziektebestrijding van het RIVM) (2). https://www.tweedekamer.nl/downloads/document?id=2021D00933

Cover photo credits: John Hopkins Bloomberg School of Public Health. (2022, 8 August). Johns Hopkins Offers New Free Virtual Course on Infectious Disease Transmission Models for Decision Makers. https://publichealth.jhu.edu/2022/johns-hopkins-offers-new-free-virtual-course-on-infectious-disease-transmission-models-for-decision-makers