American enterprise institute. Event is hosted by the American Enterprise interested institute. I am a senior fellow at the American Enterprise interested to enterprise institute. Thank you for coming. Thanks to those who are tuning in virtually. What i thought i would do is before getting into our panel, we have a terrific lineup and two excellent panels i am make excited about. I want to start by saying a few words about why we are having this event here now i way of background. I think it would be uncontroversial to say the covid19 crisis was a crisis. We think about it, it is something that we will be grappling with for some time to come and certainly among the most important events of recent times. It was a Public Health crisis, arguably the worst in 100 years, which resulted in unspeakable amount of death and illness. It also has resulted in a set of policy and political decisions which were unprecedented, the effects of which we are grappling with and trying to understand and the effects of covid can be measured now only in terms of the massive amount of carnage but also the secondary effects of the crisis. Thinking in terms of learning loss, Mental Health issues, other sorts of medical problems that were exacerbated by the crisis. Also the affectional crisis had on our social fabric and political life. If you spend time reading the history of epidemics, which is a the marling demoralizing exercise, you will find this is a very frequent pattern and one of the worst effects of the pandemic is the cost social and political conflict and an erosion of social trust and we will be grappling with that for a long time. All of that would be reason enough to have serious reflection on what happened during the covid crisis, what we did well and what we did badly. There is another arguably more important reason which is that covert is not the last Public Health crisis we will face. It is likely not the less pandemic we will face. In the years to come. While recognizing with humility that there are deep patterns that can occur, we are not helpless. We do have tools that we can use and draw on to think about how to grapple with crises like this. One thing we can agree on, we did not handle covid well, the response particular in the u. S. , is not an exemplar of good politics are policy and the question we have to think about today is how we can do better next time, what kinds of lessons we can learn for policy, for thinking about how we use evidence for policymaking, which is the subject of the first panel but also how we think about institutional reform and building resilience into our federal and other institutions which is what we will talk more about in the second panel. As the crisis proceeds, the window in which to learn those lessons, risks closing. This is why we need to have the conversation now and one might have expected a national Blueribbon Commission to look at the covid crisis and to think about what we did well and maybe something akin to what happened after the 9 11 attacks. Demoralizing we demoralizingly, we have seen little action in washington compared to the scale of the crisis and its importance with thinking about the future. As it happens, a group of experts got together in 2021 with the purpose of trying to we ate a commission trying to create a Commission Like the 9 11 commission and had an adequate and had adequate success inadequate success. They formed the Covid Crisis Group which was an independent group of experts which tried to examine what we did well and badly during the crisis and tried to put forward recommendations and these recommendations came together to report a book called lessons from the covid war. Part of what we want to do today is highlight the work of the report and engage with its findings and recommendations so we are fortunate to have a member of the Covid Crisis Group on each of our panels. We hope we will this will be the beginning of a set of conversations, only the start of more to come and we are here to convene and contribute to that so without further ado, i would like to introduce our first panel so weekend here less from me and more from the experts. Joining such as, we have a professor of epidemiology. During the pandemic, he was named the founding codirector of the new center within the center of Disease Control and prevention prevention. We have an independent research scholar, one of the divisions within the National Institute of health where emily leads the epidemiology and Data Management unit. We have the dubious distinction jennifer fuller who has the dubious distinction of being a fellow scholar. He is also a professor and as well as the cofounder and Deputy Editor of the philosophy of medicine. I will like it to pass it over to mark. Thanks for the invitation to be here. I think it is an opportunity opportunity to opportunity Time Opportunity opportune time. I should say although i do hold a parttime i am speaking not as a representative of the cdc. I was one of the people that was on the Covid Crisis Group, which produces this group that many this book many you have seen many of you have seen. Lessons on the covid war. Tony asked me to say words on the process of that. The original idea was to be the groundwork layers for a fullscale Investigative Commission that might be set up by the government or in some other way. That began in 2021 through the efforts of someone who led the 9 11 commission as the director and who work with four foundations, the rock effector the rockefellers are foundation to stand together to try to get diversity of funding and diversity of viewpoints on the planning what was the Covid CommissionPlanning Group and he began assembling that group through a snowball process of surveying people who were involved in the pandemic of various ways, asking some of us after the interviews to come and join the group that was doing the interview so we assembled in that way. As tony said, the goal had been to put together documents and oral history and some fresh memories for the potential future commission and, as we also also all, that was not adopted in the various pieces of legislation that could have adopted it. So the book was written by philip, which is good because he is a great writer and some of us are not most of us are not as good writers as he. I will go into the details i will not go into the details and one of the things that is good about it is that it is extremely readable and his ability to write well even about bureaucratic and institutional history and we have read books about those topics that are not well written and it is easier to write about personalities and interval stories but the goal was and individual stories with the goal was to write about systemic issues but the goal was to write about systemic issues and why it was decided and why it is yet to be decided in certain ways that lead to less good outcomes then we would have liked. That is part of the value of the book. It does have individual characters. It does have stories. It really focuses on the structures and on the unfortunate fact that had we had the best leadership possible at that time in all parts of government, we would have had a uphill battle. There were clear failures of leadership and they were discussed in the book but the system was not designed it talks about the Grover Cleveland era are of our Public Health system and the analogy between the way we deliver Public Health and the articles of federation. That gives you a taste of the level in which it was written and nonetheless, it is easily readable. I will leave it there and we will come back to some of the topics that are in the book during the course of the discussion. I will now sort of shift to a few perspectives of my own that i think are consistent, probably not all in the book. I think, seriously consistent with what the book tried to lay out. The topic of our panel is the role of science in pandemic decisionmaking. The perspectives i want to bring our thinking about the time evolution of that question. The pandemic was an exercise in decisionmaking under uncertainty and the uncertainty was greatest in the beginning and narrowed as it went on and as we learned more. It persisted. I think it is helpful to think about how you should ask in relation to the science when the science is minimal, when it is somewhat more established and when it is more mature, which we can think of as the beginning, middle and the later phases of the pandemic so in the beginning, there is a need to be cautionary and use shreds of evidence, even shreds up indirect evidence to make policy that will delay the bad outcome. I think there was a process of education that happened at the beginning of the pandemic where people darted to understand what it means to have an exponential growing threat that can be small today and large tomorrow and small is easier to control the large and where delay is at the is the name of the game. If you got covid in february, in april of 2020, you faced overloaded hospitals, very few medical countermeasures, very few evidence of how countermeasures might work. If you got it elite dear later, if you got it a year later, people knew how to treat the disease. There were other procedures in the hospital. You are better off. If it was not new york at the height of its first wave, you are less likely to be in an overcrowded hospital so delay is valuable, flattening the curve, as hackneyed as the term became, really does mean fewer people get affected under the course, as well as those who get affected infected later. You are trying your best as a society to delay the spread. We did not do that as well as we could have. That is one of the biggest issues. It is important to mention the issue of the complementarity of following clients in that environment. I was one of the few people among my friends who was in favor initially of border restrictions. It was not a popular view among many Public Health experts. In retrospect, i made an assumption that was wrong which was if we can delay entry by a month or delay the spread of the virus by a month, we would have time to prepare. In fact, that delay is only useful if you have time to prepare and because we were not doing much preparation in february and march, that delay was not as valuable as it could have been so i think i was right on the principle but wrong in the event. As you get to the end and the precautionary approach means you do a lot of intervention that should be marked with a very big red asterix as temporary and based on our current ignorance and School Closures is a good example. It was a right thing to close schools initially because we do not know the role of children would be in spreading it. It made a big and made a lot of sense given our understanding of how many respiratory viruses spread. They should have had an asterisk saying this is what we know and we will revisit those decisions and that is a general issue i will mention. Evidence accumulated but it is not decisive but the need so they need to eliminate evidence from different sources, from social science, from economics, from biology, from randomized trials, particular interventions becomes more necessary. As we try to figure out how to live with what turned out to be a longterm threat. The goal has to be to make sense of different kinds of evidence, none of which is completely conclusive by itself and to treat it as a multiple input decisionmaking problem. I think emily will talk about this in more detail but in the later stages, what struck me was the importance of observational evidence as opposed to the demised trial evidence as opposed to minimized trial evidence. Most of what we know about vaccines are based on observational studies because we did not do trials with this variant. There are a lot of things we cannot do trials on. We have to understand the value of observational evidence and take steps to make it better. I want to finish by saying a couple things i think we can learn from other countries responses and how they integrate science. The u. S. Has among the most informal processes at major industrialized countries were integrating scientific input into government Decision Making on Public Health and putting aside the fda, for four decisions of the sort of many kinds during the pandemic, there was no centralized group of effort of experts who filtered signs from policymaking and contracts to the u. K. In contrast to the u. K. It is a formal structure to filter evidence and provide decisionmakers with a sense of what the Scientific Community believes. Another lesson, and i think that happened informally, physically in many states, there were advisors assembled but no structure. I was on the one in massachusetts, several in massachusetts and because it was informally assembled, it was a week Time Commitment not a many hour a week Time Commitment. We couldnt respond to formal requests for evidence, for example. Another thing that u. K. Did well. I focus on the u. K. Because it was one of the best in terms of evidence was to put in place structures together evidence. They put surveillance studies that made them understand what was going on at a regional level at all times in a very comparable way across jurisdictions that we never had. There is the preparedness aspects of putting in place data structures and data authorities for the federal agencies to collect data from the states and structures for reporting from states and from health care to be more effective, and there is a discussion to be had about that but i will leave it as a teaser. Thank you so much for having me here. I am an infectious i am also here in a personal capacity to give my and my opinions are my own. I am here to talk about data. A lot of what mark said, having to get that evidence, having a lot of misunderstanding and uncertainty as we moved through the pandemic, how we answer those questions, how we become certain is gathering data and doing research and having Surveillance Systems. Unfortunately, data is difficult to collect and manage even at the best of times but it is made more difficult during an epidemic or pandemic. I want to talk about some of the things that went wrong, some of things you could have done better during we could have done better during the pandemic in terms of Data Collection and survey and research. There is a difference in the reasons on why we collect data and how we collected and based where we collected based on where we understand we want to know how many cases of infection are in certain locations and it is different from how well vaccine works. We need to set right surveillance and research and other data needs because it is not just all a one size its all Data Collection. Not all data will collect every question so that is what i want to talk about. My expertise is in design and conduction of critical studies, not observational studies and how and why we collect data and how we use it and how we manage it so that is what i am hoping to bring to this discussion. I had the opportunity to work at a state health department. I work in Academic Year and i work with cdc programs. I had a unique opportunity to understand Data Collection from a wide variety of different areas. Couple of problems that i want to highlight that we ran into during the covid pandemic, although are not surprises to anyone who has done research or the surveillance, it is difficult to collect large amounts of data, particularly sensitive data. In a pandemic, you are collecting information on peoples health, on who is sick, and demographic information. We learn a lot about people through the illnesses they have and that information can get reported and people can have various feelings about that. Some of us dont mind having that information shared and other people may have reasons why they dont want that data to be shared so we need to make sure we are doing collecting data in a sensitive manner. We are doing it in a way that protects peoples privacy. That is difficult to do, especially when you collect data through different platforms. We need to collect data in a realtime fashion, so having data from three weeks ago doesnt help us if we are trying to respond to our pandemic so we need to have data on how many people are getting tested. Are people able to get treatment . Are they taking those treatments . We need them in real time to understand how the situation is evolving and how to understand how we can respond to it and how the responses are working. Otherwise, why are we investings in investing in these interventions . There is the Economic Situation to making sure we are using the speed and Health Reform how we are doing in the Public Health response. We need a debt a standardization so just having a bunch of data instead of is better than having no data but it takes a lot of time to manage data. In this the data science world, we joe, we spent a lot of time cleaning data people handouts and only doing 30 of handling it. We get Surveillance Data from Health Departments and Health Care Companies and medicare services. None of these data systems are standardized. Thats not how our system works in the u. S. It is problematic when it comes time to the pandemic. We need that data to be interoperable to join together so we can say, we may not have one federal system of data but we have a different date at that can coverage and manage what we need. We need to be able to collect the data in realtime and be able to have a standardized and interoperable and we need to have people who cant analyze it and you cant analyze it and utilize it and we need to get that data from the people analyzing it up to our decisionmakers who can act those decisions based on the data and provide a response and unfortunately, in all of these different settings, the United States has fallen short in a pandemic situation and mark mentioned, we dont have those oversight organizations who can take data and the things we find from it and make it actionable. That is one of the things the report talks about and i hope we get to touch more on that. Who is responsible for taking the data and the insights and making them actionable . There are evan mentioned we have surveillance and research. We have have different places we can get data from area a lots of people think data appears. I wish that was the case. It is not. We can get data from Surveillance Systems. We have a distinct Surveillance Data in the u. S. There are pilot states around the country who report data on certain diseases to the cdc which collects information. Each state has a list of portable illnesses that if someone has it, they give it to the doctor there. If they have salmonella, that has to be reported. There is data transferred electronically to the cdc. One thing we can talk about is the potential to strengthen those systems to pay more attention to them, invest in them, and expand them. That is one way to help fix the problem of the lack of data we have, although there are many others. We also collect data by doing observational studies. Studies where you are not randomizing people to particular interventions. But we cannot always do them. They are expensive. You have to set them up with mindfulness and standardization and protocols which takes time, and money. You can be in a place where there is just no way to get enough people enrolled because there are not enough cases or cases already blew through in everyone is immune. There are a number of ways in which we cannot use a randomized study so we turn to Something Else like observational studies. At this time, there is not much work on standardizing observational studies and prethinking about protocols and case report forms. Things we can spend time on and prepare for, to have them ready when a pandemic situation emerges, to take them into the field, and collect data. Which can than be used which can then be used. It may places where you want a controlled trial but you can. Or during the pandemic, we had vaccine studies, but that left out huge parts of the population. We are not able to you to use pregnant women or people with illnesses. So standardizing this and making sure people have the money and people to actually do this work is important and something we should be discussing in terms of its own preparedness initiative. Finally, we have real dwell data realworld data. We use this all the time for research and it can be an opportunity for data. In the u. S. , we do not have a centralized Health Care System so it relies on working with Different Companies and different Hospital Systems around the country. They are not necessarily required to send data to the cdc in general or in the event of an emergency. They are not interoperable. I had the opportunity and misfortune of irking with a couple of them. They can be timeconsuming to work with. This slows us down and makes it hard to respond. So how can we work with these corporations and Health Care Systems who create a lot collect a life our realworld data . Two if the data we need to answer questions. There is a lot of opportunity to work with the data to improve it. Having better data will enable us to prepare and respond to outbreak situations in a much more efficient and precise manner. That is what i am here to talk about. I am looking for a discussion about that forward to a discussion about that. Thank you. I am tony from the American Enterprise institute. I am representing my own views and not those of my institutions. We heard about data and i want to turn to thinking about evidence, a related concept roughly when used data to support conclusions happens. I want to talk about evidence based policy which is a term often thrown around in government, and sometimes in Public Health. On some readings, evidencebased policy may mean using some kind of data or rationale to support this decision. But the expression evidencebased comes from the 1990s with the first evidence of evidencebased medicine which brought a number of different principles and solutions when thinking about evidence. Some are quite useful especially when trying to evaluate trials of novel therapeutics but some may be awkward or inappropriate. This medicine can be boiled down to a few. It says the highest quality evidence comes from randomized controlled trials, the kind emily mentions, in which we are randomizing individuals. In order to compare the relative effectiveness of different interventions or policy. We also might want to synthesize evidence in a certain way by doing a systematic review of the literature where possible, pooling results from multiple, similar trials in a metaanalysis to come up with a singular estimate of the quote affect quote unquote effect of an intervention. This can have a similar effect on people but in other cases may not. In which the effectiveness might not be dependent on social context. The other thing is the phrase too much medicine which is a Movement Popular in the u. K. And in the u. S. And elsewhere. This is the idea that medicine, especially in recent decades, has been prone to overtreatment, overdiagnosis, too much medicine. Scientists like clinical epidemiology can serve as an important check on unbridled medical enthusiasm by making sure we are only using intervention supported by the best evidence which shows they are effective. There is a more political kind of bent the movement that might tempora a kind of intervention might temper a kind of intervention. What do we need for evidencebased policy in Public Health . We need to think differently. What mark mentioned we need in Public Health is the decision to rely on diversity of evidence and answer diverse questions we want to have answers to and understanding better transition of a pathogen and rates and the role of schools in transitions and the role of therapeutics and vaccines and different policies in order to mitigate transmission and so on. And different kind of evidence. We would not use Natural History studies in which we are not intervening to understand the effectiveness of an intervention. This means we need a kind of approach to evidencebased into mission, including evidence beyond randomized trials. We also need diverse loans of thinking and reasoning. Probably for a reason mark mentioned which is that in an evolving Public Health emergency, there are different bases in which we have relative amounts of data and evidence available. We might advise a precautionary answer early on based on data that makes possible assumptions and outcomes or possibly supports the use of certain policies that we might collect more data on and reevaluate as we go along. Relying on the highest quality evidence in the early stage might, on the contrary, lead us to be too cautious and not act to avert a catastrophe. We also need diverse experts. Public Health Science is an interdisciplinary field. We need diverse experts to avoid blind spots and print that particular scientific disciplines might suffer from. As well as to come up with an understanding of complex health problems. The pandemic is a prime example of a complex health problem. But of course this leads to scientific communities are used to handling certain kinds of evidence that could end up butting heads or having different understandings of what kinds of studies we need and what kinds are appropriate. Partly to deal with this challenge of the need for interdisciplinary, the fact different disciplines have different ways of thinking through evidence and decisionmaking, i actually want to second marks call for a Scientific Advisory group for emergencies in the u. S. A Group Similar to what they have in the United Kingdom and what advises the World Health Organization as a Standing Group during the interpandemic period that a wrong that involves epidemiology and other Public Health scientists and Public Health scientists, including economics. It can help answer important questions in between Public Health emergencies and to think through what kind of evidence we may need when an emergency hits. The independence of government, both for the perception of having freedom from political interference, but also for the purpose of being, relatively independent of political influence. This is a group of people that should be in between. The second thing i need is government institutions to lead the way on preparing for the kinds of Research Studies we may want, having dedicated finding streams with. Coming up with protocol in order to standardize Research Across multiple areas and it comes time to collect relevant data. Identify important priorities for gaps in research, and then be able to kick in and partner with private and public groups based on resist existing relationships in order to get studies done quickly when we need them. It should not be just left to academia, even though academia did a great job during the pandemic into answering many questions. The second thing i want to say is that was under the heading of evidencebased policy of Public Health. But we also need to talk, not just about evidence when we are talking about evidencebased policy, but also value and reasoning. We have heard about evidencebased policy and follow the science. Often times, they are invoked just to put a stamp or justification on a policy. Sometimes one may be to be doing no more than saying, there is some evidence and data that might be relevant to this partake of the problem. But data only becomes evidence when we use it to support a conclusion in the context of an argument we are making. It often depends on what evidence that we need. I suggested this for a precautionary reasoning or planning for reasonable worstcase scenario. We might just need modeling results or might just need evidence to make certain that outcomes possible. This may be the best way to do in early conditions about certainty. On the other hand, if we are trying to justify a intervention, we need to know how this may prove may prevent harm to third parties and may stop the spread of infection from one person to another. We are doing harm benefit analysis. We need lots of evidence about the diverse effects interventions have. This might not be available early on. This is not just evidence but also requires thinking about values. What is it we are trying to achieve . Are we trying to prevent harm to vulnerable groups that may have been made vulnerable because of a legacy of injustice that put them at heightened risks during a pandemic . It also means that because values and decisionmaking are necessary in evidencebased policy that when groups disagree over the relevant values and decisionmaking, it can sometimes look like they are disagreeing over the evidence but are also relevant to making that decision. Sometimes it can see easier to think exit i disagree with the value of priorities you are relying on bible criticize the strength of evidence you are using to support it. Because that makes my arguments against her position seymour at objective and politically neutral so we have to make sure we are not letting this agreement masquerade as stimson agreements over evidence and policies as they are wont to do in the heightened clinical environment to a pandemic. If there is a more concrete issue, it may be we need to focus on communicating the values and the kinds of reasoning not just invoke data or evidence and we need to public scrutiny, not just make data or evidence open for viewing. The last point and want to make is, it may seem trite but in Public Health, we need Public Health approaches to problems and not just medical program not just medical approaches. This means we need to think about epidemics according to what they really are. If they are problems of a population and have a life of their own. We need to adopt certain tools to understand them at the population level in different ways for thinking about how to intervene or track the spread of a pandemic. We might deploy interventions for different purposes and to might employ face masks to protect individuals from becoming infected and harmed. This may lead us to look for certain kinds of evidence that tells us whether or not are good for certain kinds of pathogens. We might also look at the ability to stop the spread of other pathogens. To pull another agenda, testing for infections can have various purposes. It can be for surveillance, diagnosing individual patients with having disease, and spreading individuals who may be contagious to others. We need to think about what we are trying to achieve and these must include Public Health outcomes that are directed at the community and not exclusively outcomes directed at sick individual patients. This may be one kind of philosophical way of supporting the common refrain that we need to support Public Health and not just clinical medicine. This includes regulation policy guidance and scientific advice as well. I think i will leave it there and turn it back over to tony. Thank you all. There are a lot of topics i want to come back to. I thought to begin, before getting into the weeds on standards of evidence, i would like to point to mark as well. One thing we could start with is thinking about what we did wrong initially. This is a team with mark and emily, a lot of what you were saying. Some people agree that as the pandemic unfolded, a lot of the interventions we had were controversial and not always effective rated a lot in our its not always effective. A lot of our polarization came from that and masks and vaccines, etc. One of the ironies is if we are better of front, we would have been able to be more precise and have hammered tight interventions. The fact we did not have a clear picture of what was going on. The first question i want to ask mark and emily is, why was that . What did we lack that did not give us the picture we would have needed in order to have early, more bespoke kinds of policy interventions to avoid the hammered tight interventions we wound up with that were controversial yucca that were so controversial . Data. We did not have the data. We started back in march 2020 but we probably had people that came in earlier than that. Once you turn on the tap, the water is running. So not knowing that people had come in infected, where they were, and who they may have been infecting all led to compounding cases spreading where we had no idea. If you do not know, how do you intervene . Contact tracing works better if you have one person and 20 to 30 contacts then if you have 20 people and 2000 contacts. Everything create increases exponentially everything increases exponentially as you go. The earlier that you know what the situation is, the earlier you can catch it. One thing was the lack of data. The other was to believe this would be a problem in america for us definitely contributed to our early stumbling. This was one of the first big times that a pandemic impacted americans directly. Sure, we had pieces of ebola we had to worry about and yes, we have the flu. But this was one where more people wanted to not have the data or evidence and say, we do not have anything to say it is here. They allow that ignorance, deliberate not snowing, del iberate not knowing, to get out of hand. Have we started collecting data earlier, we would have known how to respond. I agree with all of that. It was a partly willful decision to not respond in the worst two months, january or february. I was one person but not the only person in february to say this was going to be a Global Pandemic and what affect majority of people in the world. I was not the only one because it was clear to epidemiologists if there are cases in dozens of countries that it is the weakest link problem that an epidemic spreading anywhere becomes a risk of an epidemic spreading to new places. Even if china, as it did, can control the source, once it is out into the rest of the world, it is the weakest link. And there is age and the chance it gets back into the bottle is nil. We did not spend the first two months securing up and preparing as much as we could have. This includes preparing better data. I think there were some specific institutional and weedy things about the inability to use, for example, studies sets out for flu in seattle to detect covid. It was eventually broken through. But that last that lost time. We could have had more flexibility to see what was going on how we planned our human subjects approval in advance to cover a little more emergency needs. But i agree with emily that it was a lack of data but also a lack of positioning the ability to respond to what was clear to many people as a threat, even if we did not have data to prove it was get a threat. One of the other things, just talked a lot about in the report, is i feel a lot of agencies did not know whose responsibility it was to respond to these things and collect data. Everyone assumes the cdc. That is not the case. The cdc can make recommendations but it is up to the state to do a lot to the bond. If it is in certain states, the department of Homeland Security has the responsibility to report cases. There was confusion about who should be doing the responding and who should be collecting data and what is the purview of different agencies. This is something that i dont know if it ever got cleared up. And the development of the white house Covid Commission did not help, saying there was a lot of movement around there was a lot of confusion about early responsibility which again let us down. One thing we need to get the data, as the book talks about a lot, is have the ability to rapidly scale up and distribute the kind of tests we need to collect the information we want. So we can get an eye on where the enemy is and how quickly it is spreading. Then eventually allow us to more safely return to work, school, and other environments. Which it took americans a long time to do because we did not have a National Testing strategy of the kind Certain Groups like the sapper center were working on developing curated and eventually collaborating with the covid collaborative to think carefully. There was no national or federal strategy for thinking about what we would need to do and how we need to do it in order to surveilled covid effectively and also return people safely to their various environments. It required to have the Production Capacity in place in order to manufacture the tests once we had the sequence of the virus and are able to produce them. Another thing the book mentions is the lack of an industrial policy. It ends of a preexisting nonsuspicious or not overly suspicious relationship between government and private industry. They give the example of south korea. Because of their experience with mers, the previous coronavirus, they have the ability to build test and had set up the infrastructure. Nothing physical Production Capacity to my knowledge, that the agreements and relationships and knowledge of who could do it so when this virus came, south korea was in the position to scale up the manufacture of tests. If that does not give you policy, that is a separate problem. But at least the supply of tests. If we want to point to the early friend errors of the stashed the early front errors of the cdc to the erly front errors of the cdc. I want to get to issues and data of evidence you raised. John, in your opening remarks, and something you have written about, one of the difficulty of complex problems where you have expertise that is wired as you can have that is required is you can have conflicts among domains of experts. One of the cultural clashes that you have written about is that between the domains in medicine and Public Health and control trials generated by that, and the domains in reliance on modeling especially with population evidence. I wonder if you can say more about those two approaches and how this shake the debate about covid. Part of what happens in a lot of these policy discussions is public controversy. If you lifted the hood, you would have a complex situation with in the community as well. These two things were not always clearly identified and seemed to muddy the waters in some cases. One thing is i want to start off by saying there is an Important Role for clinical trials, especially randomized and during the pandemic. We need them to understand the effectiveness of novels like vaccines and we need them in order to figure out which of the many possible repurposed drugs we may consider actually effective. There are great examples from u. K. With the Coverage File and the Critical Care plot from trial in the u. S. , where they delivered good, high quality and fairly ontime results to care for patients. I am not someone who thinks we do not need critical trials for medicine but there can be more tension when we are thinking about the relative roles and different kinds of evidence in examples like face masks. Maybe i will stick with that example because right from the beginning of the pandemic to just a few months ago, there has been payments among different sectors of the Scientific Community who are used to dealing with different kinds of interventions or problems. Early in the pandemic, there was the question of whether or not we should be face masks in order to protect ourselves or others. There were a few individuals in the evidencebased Medicine Community who pointed out that from previous reviews of the evidence, including our randomized and nonrandomized studies, face masks for other respiratory pathogens and studies of the effectiveness of those face masks were often of lower, uncertain quality. It did not give us a clear direction of whether or not face masks may be useful. So arguing from a kind of lack of evidence from caution against appointing face masks. They may be analogous to a drug that has some kind of theoretical possibility but ultimately turns out not to be effective. Others however, who are possibly a little more acclimatized to the reality of Public Health decisionmaking suggested that there is another kind of approach we can take too with the question of face masks. That is that if we mask ourselves, what are the worst case outcomes we are trying to prevent . What are the possible harms. Given there is some uncertainty about whether or not face masks might be useful here, is there a different way we can reason. Not asking ourselves how good is the evidence and is it strong or high quality, but rather on plausible benefits and harms, is it a good and to wearing face masks, specifically for in the Community Among the asymptomatic. There is a different question about whether to wear face masks when you have a system and guidance was a good thing to do. Fastforward to three years later where we have collected quite a bit more data on the ability of face masks to slow Community Transition. Some is observational. We are looking into different regions that implement policy mandates versus not. As well as two or three randomized trials comparing different individuals or community that use face masks to slow community transmission. Despite that fact, we have some more studies in the context of the pandemic and a wealth of different kinds of evidence from before the pandemic about the ability of face masks in a laboratory setting. We have a wealth of more evidence during the pandemic about the role of presymptomatic and asymptomatic spreads as far as cove to as sarscov2. We have a wealth of evidence demonstrating that airborne transmission for small respiratory particles is the dominant mode of spreading sarscov2. There still remains some disagreement among certain kinds of scientists. Our corporate systematic review which is the Gold Standard for evaluating evidence and evidencebased medicine a few months ago, did an update on literature of face masks, focusing only on randomized controlled trials in which face masks were used during the pandemic in the context of other viruses, trying to revisit whether face masks were effective for mitigating community transmission. It concluded that it was that it probably had little or no effect based on evidence of variable quality but often according to the review of lower or uncertain quality in which i turned to face masks was highly variable in which the role of face masks was also highly variable from providers in Health Care Settings to Community Members living in villages. They essentially used an orthodox based approach in order to come up with a single estimate the ability of face masks, compared to no face masks with security transmission. Some studies as well compare different kinds of faith studies and respirators kinds of face masks and respirators. This will be concluded on an orthodox based medicine approach that they are not effective. But if you look too many members of the health community, they are unconvinced, because they are looking at a broader range of diversity of evidence. They are not asking the question, what is the effect size of face masks . You are instead asking, how can we understand the results of multiple studies in context and understand the role of mechanism by which this intervention may have a function or role in mitigating persontoperson transmission . It is not just that they are looking at broader range of evidence but also that they are looking at the evidence in a different way, a way i would think of as trying to explain the results of different studies holistic leave rather than pulling to get results to come up with a more precise estimate of a single effect which does not exist in this case. Because the range of studies that are being looked at are so heterogeneous in terms of policies that are being studied, the context, settings, outcome, and so on. There is no singular effect size of masking respiratory pathogens to be found in the first place. This is an example of what you may call scientific culture clash where a Certain Group of people who are used to evaluating certain kinds of interventions and medicine using different methodologies apply these in different contexts in which, to my mind, they have important limitations. This seems like a good example of where needing space in which to have these kinds of debates and informing policy would have been very helpful. It strikes me listening to the description you gave about the state of science and disagreement and reasoning that goes into the policy recommendations is in stark contrast with what we tended to get in the media and public sphere. Which arguably contributed to a lot of polarization and trust issues. I want to make sure we have time for questions from the audience. I have more i would like to ask but the but if there are questions, feel free. We have a microphone coming. You mentioned seattle and im curious of your judgment about whether now, given the technical capacity we have, and the folks that were doing the testing and knew there was Community Transition going on. This information filter up . Would we make that same mistake again . Or do you think current policies would allow the information to start informing policy earlier on yucca do you want to take this one . I will start. But i do not know the detailed timeline of where information got from that. The point i was trying to make was that for a long time, it was inconsistent with their human subjects approvals that they had to do. So their study had been viewed as the seattle blue study and they were not allowed to test temples for other things. So there was a delay in being able to do this, due to the limitations of how consent forms were written. I think once the case was reported, it made national news, as i recall. It was not a problem of getting information out then. I think there was the Public Health agencies always have the problem that they like to stick to data and do not like to speculate. I think Public Health agencies need to do control speculation with an asterisk saying i am speculating now, this is not data but informed expert opinion. If this is the case of seattle, which we found in that way, and the cases in other countries, it is stupid to think there are not cases in other places and the u. S. The fact we have not seen the needs to be noted but the stories should not be, we have a case in seattle. The story should be that we have a case in seattle and one role of Infectious Disease models and epidemiological reasoning is to say, if there is a case in seattle, we have reason to believe there are cases elsewhere and we need to be preparing for that. Certain Public Health departments are in conversation with a good epidemiologist and were thinking that way very early on but not all. I think one of the things you are probably trying to get a day, is if to get at, is is a university an institution develops a great test, why cant we use that . There is something to be set on both sides. Should we as a country be more flexible in allowing evidence from different sources like academia . I think so. But is there a reason why every random joe on the street elon musk cannot get into the Testing Industry and come up with any test . There is a lot of garbage that can get into the system and we can run the risk of, if someone has a fault positive and we say have to stay home for 14 days, now this person loses 40 days worth of income when it was a garbage test and should not have, then that is when we mess up. We need innovation between allowing the historically standardized groups who always made the tests and balance incorporating them and allowing them to come to the table with also making sure we are still following rigorous, Clinical Health standards. Because there are implications for having this data being identified as a case for putting money into this region because we think there is transmission. I think there will probably be more conversation Going Forward about how we can incorporate different academic institutions or private companies who may have new genome technology. I do think we have to mix where there are still standards and regulations there because it could go badly. There is a lot of damage they could do that people may not think of initially. Like, more data, more information. The microphone. Thinking about new technologies coming on line and garbage out, i would like to online and gabage in garbage out, i would like to talk about wastewater samples that were coming out as the pandemic was unfolding and predicting forecast information. The much more recent use of Large Language Models in the context of sarscov2 genomes as opposed to future language to predict the evolution of the spike protein. In both cases, these are new approaches that have been historically outside the Public Health and medical arenas. As a result of the pandemic, they were and are being deployed. They certainly do not set the mode of randomized control throttle controlled trials, but they do represent where technology and leadingedge curiosity based signs can be deployed during the Public Health emergency to yield positive outcomes for the public. I am wondering if any of you can respond to this. Absolutely. Wastewater surveillance has been done since polio. We have used it as a mechanism for Public Health surveillance in the past. As long as it is done in an Ethical Fashion and done in such a way that it is able to answer questions that you have, why not . If we have money and we have people to do it, why not have that data . I think maybe people disagree with me but i think it is a great way. It also removes the need for humans in that situation. If you can observe the wastewater, you dont have to have people go to the doctor. You can go into areas where there may not be as much access to health care. It can be a good way to innovate getting information about people that are historically excluded from our Surveillance System because of other issues we could definitely talk about. In terms of ai and language learning, i think this is a really great area for the utilization of this. I am an epidemiologist and i do machine learning. I am usually very much a proponent of taking this carefully and making sure we really think through. When it comes to Predictive Modeling of things like mutations or target sites for vaccines, we could really utilize ai. We have a life quality data, good quality sleep uses, structural models as a virus in different proteins we can put in. We have good quality data we could put into an ai system and model things that will not immediately impact patients and their wellbeing. Then we can say, maybe they are different options of target so maybe we can tests through a more traditional observational study and then randomized controlled if we need to. I definitely think those are two instances of newer technologies or repurchasing of older technologies that we could use in a pandemic situation and in an fireman where it could contribute and add to our body of knowledge as opposed to adding more conflicting knowledge or lowquality evidence. I would not say, lets go put ai on everything and it will save the world. Just as with mathematical models, you have to be mindful with how you are using them and building them. But i think there is room for these technologies, given we have people thinking through the riskbenefit tradeoff and we make sure we are using them in an Ethical Fashion. I think we have time for one more question. The front. Thank you. When we think about the fact our Health Care System is both public and private, we do not have a universal system, the opportunity to make it better next time how much of it is structural and how much his policy . I know they are related. When you look at the two sectors, public and private, and the structural versus the policy side, how does this play out in terms of making it better next time . I think the federal structure of our Public Health system is the biggest constraint because it will probably not go away. Meeting the centralized and at the state or local level meaning decentralized and at the state or local level. Policies can impact decisionmaking by empowering the more decentralized areas of government in making them better. One thing we are focused on at the cdc center is working with local authorities to pile it and spread the best, analytic approaches. Really as a hedge against federal inability or national inability to respond to. It is good to have capacity at multiple levels. There is a workforce issue as well. Then, at the same time, we need to redundantly, in a way, find better as all panelists have said to get Health Care Data and Health Care System data from the private Health Care System integrated into Public Health. This does not happen as much as it shed. It happens to some degree but there is an opportunity to make it timely and to find better ways to collaborate rather than a sort of data buying a block of data and transferring it which means it will be old before you can use it. And you dont necessarily know all the details of what the data means. But to build collaborative systems of Public Health to do analyses is one thing we are trying to work through. Also at the cdc center. I think the structure is, in a way, a constraint in the policy could work within that constraint. I think we have a lot we can improve through those kinds of purchases. Unfortunately, we are out of time and i want to make sure we have the chance to think our panelists and prepare for our next panel. Please stick around enjoying be in thinking them. [applause]