[ is distinct conversations ]. and live here this morning on c spann span 3, a discussion about transparency and oversight at the food and drug administration. we'll be hearing from dr. scott gottlieb starting off the conference. introductions under way. >> efforts to pass and implement the food safety modernization act, focus on medical devices, especially the potential of using the unique device identifier to improve monitoring devices throughout the product lifecycle. we have successfully worked on products 10 to courage innovation particularly around antibiotics and 10 to sure appropriate use of them. we focused on drug safety and transparency in the supply chain and especially on the safety of compounded drugs. among other things, pew submitted transparency to the task force back in 2010 if the. they touch on some of the topics we'll be discussing today. reading the background papers for today's meet regular mind med of a meeting we convened in a room directly above of us here a few years ago about regular plo predictability and what do we mean what we talk about that? one of the things that emerged from that is something you may touch on today, sometimes the fda asks for new data on applications late in the game and companies are left wondering why are they asking about this at this late stage? and the fda said sometimes we learn things from other pr ducts that we're looking at but we can't tell them what we know or why we know. one of the idea that came out of that was making the fda's complete response letters available. with that as with so many things today, you'll be thinking about the tradeoff between public good and private rights and how do you balance those and find the right sweet spot. all of this is just to say that we share with other stakeholders in the room an interest in transparency and effective governance at the fda and commissioner gottlieb prior to his current role shared with many of you the perspective of an outsider watching the agency closely and trying to figure out how do we achieve an optimal regulatory system. josh, of course in his time at the fda was responsible for fda track which brought together hundreds of performance indicators at the agency in an attempt to move towards transparency and an external dashboard that the people could use to monitor the agency. so you're in good hands today, as you grapple with these key questions about transpatiencesive various different kinds proffer says and clinical data. so with that let me turn it over to josh. >> good morning and thank you, alan, and the pew foundation for hosting today's event. my name is josh sar of seen it i'm the associate dean for public health practice and training any johns hopkins bloomberg school of public health and the diabetes commissioner of the u.s. food and drug administration. i also want to acknowledge and thank the laura and john arnold foundation which response erpd the blueprint for transparency that we will be talking about today as well as the special issue of journal of law, methods and ethics that's being released today and for responsering today's event. i want to thank anna davis, jim miller and caleb alexander from the johns hopkins bloomberg school of health, alex kessel heim, and joe ross and margaret mccarthy from yale medical school and law school who all worked with me and with our students and chauddry and brian smith in drafting the blueprint that is now available on line. today's discussion will take this document as a starting point and focus on critical aspects of transparency. and i also want to thank nick enquest and barbara venom and susan myrrh row and the whole team at the school public health that's made today's convenient possible. now, i first became interest in transparency in fda when i was part of the transition in 2009 at fda where we met with about 60 different groups that were engaged with the agency. i noticed that many of them seemed to be mad at the fda and for very different reasons. in fact, they could be mad about the same decision the fda made from completely different vantage points. and as we went through just meeting with group after group after group, i realized that every single one of them, if they didn't understand an agency decision, would think about it as a worst case scenario. had is what it meant. they would bring all kinds of assumptions even if they were wildly different from group to group. it's unquestionably true that fda can handle a very challenging regulatory matter well, in fact, do a better job than any other agency in the world and if the answer is not explained can still fail in the court of public opinion. later when i was serving at fda i attended a conference where the agency announced that fda had worked with nih and industry to make available to researchers data from the placebo arms of the studies on alzheimer's disease and that that data was then going to help drive research forward faster. and i appreciated on the one hand how hard janet wood kok and the team had worked to create that transparency of data. and other on the other hand, the information that the fda uses and the information that it brings has value far beyond a regulatory decision. even if one drug fails, the information generated could set the stage for the next advance. now, you heard that i was involved in transparency in 2009 and 2010, and that initial work was kind of lonely. everyone at fda is focused on the issues of the day, and there are many different issues that hit fda all the time, you just can pick up any newspaper and see them. so it was a little lonely. but support came from some unusual sources, including scott gottlieb. even though we worked on -- in administrations with different political colors, scott was incredibly generous with his time when i started in the obama administration, and we talked about all of the essential and critical and timeless issues at fa fda, including hr, the clearance process and transparency. it is no surprise to me that dr. gat leb's tenure so far has been productive. he's pursued important efforts against dangerous and ineffective products, worked to end the ghams undermine the market for generic drugs and supported the work of thousands of scientists on the agency to keep on top of rapidly changing science. he spent time at the johns hopkins public school of health where he's impressed students, faculty and others with his deep understanding of many different issues often in response to rapid-fire questions from students. so it was a very, very important. people stop me in the halls and say who was that again? who came? when i visit the fda for a talk on the opioid epidemic, i found scott eating lunch with the employees in the cafeteria, not locked in a sound-proof booth. most importantly, he has kept fda's focus on its public health mission, and that's why i'm so pleased to introduce him for today's opening remarks. dr. gottlieb. [ applause ] >> thanks a lot. thanks for the opportunity to be here today. dr. sar of seen it and i have had a dialogue for many years and i appreciate the relationship we've had talking about policy issues. when he was at fda i wrote an article calling on the agency to start releasing the complete response letters, and then i wrote a subsequent article saying what's taking you so long, josh? so i've been waiting for him to return the favor. and i just got a copy of it, so i appreciate it very much. thanks for the opportunity to be here today. thanks for johns hopkins for hosting this and pew for providing the space. transparency can be a powerful tool for innovation, as was noted. new platforms for analyzing and visualizing large, complex data sets present important new opportunities for the research and development community for regulators and for clinicians it allows foam harness date to advance the safe and effective and efficient development of new therapies, to address unmet medical needs, to tailor clinical care to individual patient preferences, and to communicate information supporting regulatory decision making in ways that engender greater public confidence in those decisions. the fda's goal is to advance transparency without reducing incentives to innovate. this can be accomplished by meeting our statutory responsibility for protecting confidential, trade secrets and personally identifiable data while having more information that supports and i forms regulatory decisions and patient care once products are approved for use. with these safeguards in place, transparency's a tide that can lift all boats. it cannen able stakeholders to better address common challenges from a development process to identify are identify areas requiring post market data and product safety and efficacy. the morn transparency strategy began with josh's leadership at the transparency task force and subsequent recommendations in 2010, and we continued to expand fda's commitment to transparency today. but stakeholders continue to request greater transparency in the drug approval process and greater access to useable information on those approvals. and i believe they're right. we look -- we need to look for ways to be more forthcoming with the information we have within the boundaries of our stat ought to and o-- statute and our obligations. much of our information contains pro duktss that are integral to people's families, the foods they eat, the pets they love, and the medicines they use. people deserve to have as much information about these products as we can possibly reliably provide. and there are more tools available than ever before to help consumers fill the they're information and to help third parties harness it into more useable knowledge. we have a public health obligation to keep people fully informed of the safety and benefits of the products they use to improve their lives. and we're taking some new steps to fulfill these commitments. today i'm announcing two important initiatives i hope will provide greater transparency and efficiency in the agency's review of the new drug april occasions, biological license applications. together these initiatives will give pishts aatients and resear new insight behind the approval process. first, beginning this month, our center for drugs is launching a pilot to evaluate whether disclosing certain summaries of clinical information called clinical study reports or csrs following the approval of new drug app occasions clications c the process. to start, the pilot's going to include up to nine drug applications who sponsors volunteers to participate. pilot's going to test the process to be run by fda's drug center for selecting, redacting and posting csr information on the public website drugs at fda. that gives fda -- that fda uses to provide approval information on new drugs. applications to be included in the pilot will be selected based on criteria including their novelty, scientific interest, and whether or not the drug is a new molecular entity. a csr is a portion of the drug file related to a clinical trial that contains a detailed summary of bottom-line information on the methods and results of the trial. and csv a scientific document addressing the efficacy and safety. we expect that make something csrs publicly available after approval will provide stakeholders with more information on an application's clinical evidence and fda's decision-make process. it can help those interested in that detailed data, for example academic researchers who want to study a specific drug and need access to summaries bottom-line information in order to inform that research. this will be the first time that fda's privately disclosing clinical summary reports from sponsors to the public. the csr sections disclosed as part of this pilot will include study report bodies. the study protocol and amendments, and statistical anal sees plans for each product's pivotal study. clinical summary reports will be posted on a new fda web page describing the program in addition to appearing drugs on fda with the sponsor's drug approval information alongside it at the time that it's approved. one goal of the pilot is to enhance access to clinical information by medical researchers. the drug development community, and the public. while fda does provide access to drug application action packages that include all discipline reviews after a drug has been improved, the utility is sometimes limited by their complexity. while these summaries, these new summaries provide information, they can at times make it difficult for external audience to decipher the evidence. we believe that by posting key portions of the csrs publicly in response we'll both comply with our existing obligations to make certain summary materials available upon approval and make that material significantly more practical and user friendly for key collaborators. for example, medical researchers who want to further study a specific drug beyond the clinical evidence that's supported its approval and is disclosed typically right now. at the same time we're taking another new step to make it easier for patients cland anythings to track new therapies as they advancing through product development and different regulatory check points. right now the majority of publicly and privately trials around the world register in the national institute's after health's database which i'm sure many of you are familiar with. this website provides easy access to information on studies in a wide range of diseases and conditions. ultimately some of these trials involve new drugs and form the base is of an application seeking fda approval. these trial -- in it's active state on clinical trials gov in noo a product label can be very challenging at times. in response, we plan to increase transparency -- posted on clinical trials.gov, that all they registered through clinical trials.gov to use to link to those specific drugs including in the product labeling and our advisory committee materials. this new number will be called an nct number. members of the patient and academic and scientific communities will then use this number to follow and track clinical research from a drug's development through key regulatory milestones. this number will make it easier to correlate the clinical trial listings on clinical trials.gov to fda communications about specific drugs, including product labeling and even our advisory committee materials. including this number on fda's materials could greatly benefit all those interested in following the progress krf specific clinical research. we're also continuing to explore whether it would be possible, as we discuss at the outset under existing statutory authority or through change in the governing law to release additional information from the complete response letters relate to the clinical and safety efficacy that could have significant public health value. releasing all the crls would be administratively challenging given the likelihood we would have to redact all proprietary information from these letters and not all letters have information that would directly inform clinical practice. many letters relate to manufacturing smort cummings with new drug applications that are eventually resolved. but some of the letters do contain information that could be directly relevant to patients. we're eval waiting whether there's a subset of the complete response letters where there is especially important public health reasons to redact and release these letters. this could include letters that have safety-related findings or recommendations that could help inform patients and providers about the profile of already marketed drugs. releasing this information could enhance patient safety by reducing the number of potentially futile trials and spare patient's suppose exposure to potential risks without the prospect of a likely benefit. it can also help better inform clinical practice. we're committed to enhancing transparency throughout the work we do at fda in the is especially true when it comes to the efforts that have the potential to foster further research and discovery across the scientific community in clinical care and we'll continue to seek additional opportunities to foster greater access to key scientific and information and clarity about regulatory decisions wherever appropriate. the powerful synergy between high-quality data sets and therapeutic innovation has been evident for more than 50 years, long before the advent of the desktop computer, let alone machine learning. arguably two of the most important in public health initiatives undertakement post war era were the launch of the framingham hard study in 1948 and the certain general's report on smoking in 1964. the surgeon general's report was deeply informed by the epidemiologic evidence from the framingham study along with a similar report from albany, new york. a new generations of participants, spouses, and children helped revolutionize our understanding of risk factors for cardiovascular disease including smoking, obesity, and high kohles ter roll that led to the development of the framingham risk score ars well. that risk score reduced the risk of heart disease to the why use stan continue drugs to smoke sensation. the combination better science and better informed public led to a dramatic decline in mortality from all forms of cardio vas vascular disease including coronary heart disease, stroke, and coronary artery disease. there are reports that the death rate from cardiovascular disease peaked just before the surgeon general res report and started falling from the late 1960s. from 1968 to 2010, the age adjusted death rate from cardiovascular disease has declined by nearly 70%. today, the framingham study has expanded to include biospecimen banks and collection of data and any researcher can request access to the underlying data sets. technology, no matter how powerful, is always just a tool. it's our commitment to the transparent and science-based use of those tools that gives them their potential to save and improve lives and build many more practical framingham studies for the 21st sent tries based on a growing number of data tools and data sets. to reach their fool potential, open data, real world data, wearable and implant able diagnostics will all have to be validated through transparency processes that gain the trust of stakeholders in their reliability and the fda will be at the the epicenter of this process. not by inventing new values but by reinventing ourselves to century long commitment for product safety and esfekttiveness and remaining the benchmark for global excellence. thank you. thanks for your time today, look forward to your questions. >> so i think people should have index cards and if you can put questions down there and just raise your -- raise your hand with them, we'll collect them and i'll get them and we'll be able to ask questions. i'm going to come over and get started. dr. gottlieb, thank you very much. >> thanks. >> that was a terrific talk and did not disappoint with some news about transparency at the agency. i wanted to start with a general question and then follow up with some of the specifics about the announcements that you made. some people think of fda as a black box, that the agency sits out there in white oak and makes decisions and that's really the important thing, how the decisions get made, not so important. how do you think what's the metaphor that you would like people to think of fda as you're leading the agency into a very different time. >> i think would you agree with this think there it's important that people understand exactly how the decisions get made and the processes that inform how we go about analyzing data and making decisions. and also the values that inform the decisions. because ultimately a lot of the decisions that we do make are policy calls. i mean, you can -- you can scientifically define the risks and the benefits of a product-related decision, but ultimately a decision on how you balance those risks against those benefits involves a lot of policy decision making. and i think we need to be very transparent and engage the public in how we ultimately do that balancing. and i think we do do that. i think we do a good job of that. and it's gotten a lot better over time. i think the adviser committee process provides a lot of transparency. we hold a lot of public work smops and engage the public, we post online through guidance and other vehicles detailed documentation on how we do that balancing and make those decisions. and there's more of an impetus, i think, for them to engage with the public as much as possible to speak to communities and other kinds of workshop to articulate our philosophy. think ultimately one of the challenges that we face is that the complexity of what the agency does sometimes makes it hard to access. and think that's something we struggle with historically. and that's something that i think we try to -- we're trying to address through, for example, engaging patients for actively in the mechanics of the review process, trying to elevate patient involvement in some of the regulatory decision making on the advisory committees. but i think that that's been a historical challenge dpoint thi. i don't think it's how do we sort of make the philosophy and that policy more transparent? >> yeah. well, let me ask you about the csr announcement that you just made, the complete study reports. these are pretty complex documents. >> right. >> they're big, they have a lot of data in them. what are the kinds of benefits that you see from this pilot program? >> i think first of all it's going to make the review process more efficient. and we're doing it as a prooi pilot program, but we think over time a lot of the manufacturers are going to see the value in make these public. because a lot of review memos right now are kurting and pasting from the review documents. and the information, the bottom line clinical information informs the review isn't very accessible in those documents. think when -- when we're able to link to this csrs, the bottom-line information in the csrs is going to make the review document much more efficient, more accessible and cop provide a lot of efficiency to the review process itself. this is part broader reforms that are under way with the office of new drugs that we're trying to come up with a more consolidated review memo. right now if you look at the review process, a lot of the documents this made public at the time of a product approval are different documents related to the different consultative aspects of the review process. a different components of the agency act as consultants to the review process and write their own mem moss. so you have multiple memos. we want to work in a direction of coming up with a more consolidated memo so you have a more team-based approach where everyone's working together on trying to develop a consolidated document that explains the agency's philosophy so you don't have five different doms documents put have one document where you can see nefrg one place. >> that would be released? >> that would be released at time of approval it would be a different way of presenting the information. but having the csr available and being able to link to it makes it much more feasible to come up with a consolidated document. it's also just to finish that, so that's one virtue of doing it. that's sort of an efficiency. but then the other is that the csr is it's one level above patient-specific data. it's not paper-specific data but it's more bottom-line data. i think that having it available is going to allow third parties do more analytics around our decision making. and do more analysis. and so, you know, you'll have research better informed by actual data. right now some drug companies release summaries of the csr but they don't release the whole csr. so this would be the whole csr. >> so two questions. one, should people be worried that patient-level data will be disclosed or they'll be approaches to making sure that there's not a confidentiality -- >> this isn't a patient level data and we're very mindful to make sure -- that's where we're going to have to go through it to make sure that there's no way to sort of decouple the presentation. i mean, we saw that experiment i think it was with aol. >> yeah. >> where they looked at searches and were able to go to people's homes and find out who was doing searches by just decoupling the data. that's a concern. you want to make sure that the date is presented in a way that someone who san intelligent researcher and has access to another clinical stud i can can't correlate it in a way that he can get down to patient data. that's one of the things we're very focused on. we feel pretty confident that that's not going to be the case. the csrs are released, i don't know all the specifics around it, reporters will probably do a better job than me at getting at the details of this. but in europe the csrs are available but they're not readily available. they're available on a proprietary database think that's maintained by the ema where certain researchers can get access to it. so there is some precedent for allowing certain qualified researchers access to some of the csrs. we want to do it on a more wholesale fashion and we're working through the legal issues and we'll see where it zblgoes. >> initially you're starting out as voluntary? >> we're reaching out to products we think that this could be particularly relevant and allow for, you know, more efficient process and then we'll see where it goes. i do believe that there's -- you know, we talk about whether wednesday up making this voluntary or mandatory going forward, we'll work that out through the pilot. >> you're guessing what question is on the card here? >> yes. but i am making a pitch for the fact that i think when you look at the -- what we're hable to do by making the in making the csr publicly available in terms of how it changes the nature of how the review memo gets derived and that then changes how the review process works, i think people are going to see it creates a lot of efficiencies. because the krcreation of those reviewer memos and how people formulate their opinions does drive a lot of the structure of how the review process works. to the extent you can develop a more consolidated review opinion that more easily services areas of agreement, disagreement, it's all in one document it lends itself to a more team-based approach which is the direction we want to move in. so in order to move in that direction of a team-based review approach where people are part of a consolidated team, the memo itself needs to be a team-based effort. >> so the -- >> it's going to create a more efficient review process with greater input from people who have -- who have expect tertise different displinciplines. >> i see. that woen courage. >> that would enkaur rage debate, yes. >> moving on to the clinical trial number, which is another recommendation that is in the blueprint for transparency, so much appreciated. one question there. it has been the case that some researchers sometimes don't register their trial appropriately or don't do the linkages themselves. obviously there's one aspect of it which is just getting the information to people so they can see where there's a trial that has been registered with nih so they can look it up. if companies don't do that, have you thought, you know, if they don't provide the information it turns out -- i don't mean companies, researchers actually don't -- don't tell you which trials they are correctly or maybe they tell you some but not others, do you see that this could be an area that lends itself to some expectations that fda has and then enforces on companies? >> we'd have to -- we're going to see how it goes. >> okay. >> in terms of creating requirements we'd have to do that through some kind of notice of common process. think we're going to see how this works and then there's questions of, you know, some of the issues that you're alluding to are things that would be managed by library the medicine. >> right. >> some would be issues that we would directly address. >> what about devices, would these first two policies have anything do with devices or now you're starting with drugs? >> right now we're starting with drugs. i'm not sure what the analogous piece would be on the medical device side, it's different. but the broader goals on the medical device side are similar in that we're trying to move towards a team-based approach in the review process. and do that we want to have more consolidated types of communications. so it's easier to sort of -- to read the narrative of what the agency's thinking was. that say chal life-threatening length right now when you look at a review package in terms of looking at the different opinions to read a lot of different memos. and we think sort of a more consolidated narrative will lend to more transparency around the agency's considerations. >> it sounds like what you're saying is it's clearer for people to understand, still allows room for disagreement. >> absolutely. >> but allows for the agency to connect to some of the original data in a way that's a little different than now. >> right. >> instead of just saying here are five different memos plus three data sets, you're saying here's the memo and at the right part of the memo it will be referring to data. >> exact. >> i it's part of a bicker approach to reflecting what fda's thinking. >> right now a lot of the -- if you look at the memos they're long but a lot of it is cutting and pasting of information out of the package that the sponsors provided. to sometimes it's not very accessible. if you can provide a memo that focuses more on the agency's think and linkages to the underlying data because have you this electronic document available and it contains all the data, so you're looking at their analytics and their bottom-line conclusions then you can link right to the data. that's a more accessible document for lay people because it's more focus dollars open the agency's philosophies. and for researchers because noe now they can link back to the data sets that were used to derief some of these conclusions. >> last couple questions on complete response letters. you knew that would come up. 2010 a wise person wrote the public de storves see the agency's explanations for its -- i think that was you. and so the complete response letters are the letters where fda explains to a company why it couldn't approve the product. that may include challenges with safety or effectiveness of the product and there's been some research by fda showing that companies will release some information about why their drugs weren't approved, for example, they often don't release other information including some safety concerns. i want -- you made a really interesting comment that you're most focus on the public health side of this issue. you can talk a little bit more about where you see public health coming into play with complete response letters? like someone might say for example if a drug's never going to be approved at all, you know, why is it necessary to let people know twhwhy it failed? where's the potential for public health benefit? >> well, it's hard to draw a conclusion that a drug's never going to be approved at all. if you look at the statistics, most drugs that make it to the nda stage eventual, are approved think there it's upwards of 90%, you know the statistics as well as i do. but a lot of times the -- not a lot of times, but sometimes the basis for the decision to issue a complete response will be, you know, safety-related issue where the agency's skd askiasking for information about a particular drug on a concern it has where that concern might be regular to other drugs. >> in a class. >> in a similar category, in a class. and that's highly relevant. or, you know, information that can inform different classes of drugs where the agency has sort of an overall philosophy about a therapeutic space or drug interactions. so those are the kinds of situations where i think that making that information could help inform clinical practice and public health. what we don't want to do, this is very resource intensive, as you know. peter's been around this issue, you've been around this issue, and the challenges have been the resource intensive nature of redacting and making these letters public. i want to see if there's a way to make an easy cut not easy, but something that can be implemented as a matter of policy between letters that have that kind of public health value and those that, you know, have less value might be more interesting to the investment community but not necessarily have probative value from a public health standpoint. i'm not sure that that's doable. i'm inclined to think that it might be. we have someone working right now, we've sort of pulled like 100 crls. >> okay. >> and we have something going through them to see whether or not there are some identifiable things. if you look at a complete response letter, there's above the line issues and below the line issues, it's kind of like the kitchen sink when you do a complete response letter they'll have ten different issues but there's only two driving the agency's decision is the so is there a way in terms of how it's organized or certain kinds of information where can he can extract it and say letters that have this kind of a safety issue identified in this part of the letter or contain this kind of information, we're going to make those public to start. and that's what we're looking at right now. i think that there might be a wait to get there, to find the subset that really is important from public health standpoint. it might be places, for example, i'm just speculating so don't hold me to this, place where's we're asking for an additional clinical trial related to a cyst issue for a drug where there's other marketed drugs where it could have some relevancy. that might be an easily identifiable subset. those are the kinds of things we want to look at. what i don't want to do is become -- we have processes in place to make referrals to the sec when there's misrepresentations and stuff like that. i don't want to use this kind of initiative where i'm get to gog to be eating a lot of resources to redact these documents just for purposes of market efficiency. this has to have some bottom-line value to patients. >> so two last questions, this is really interesting. one more on the complete response letters. when we've thought about the public health value complete response letters, for those that are for drugs already on the market for unapproved uses have risen pretty high up because sometimes there's a little bit of data out there and maybe, you know, some intense interest on a company's part for the drug to be used for another indication. but when the -- are down and fda's doing its review, they don't see evidence of the effectiveness. and because that drug is on the marquette and because there may even be articles in the medical literature saying why not try this drug for this reason, the fda's view of that could be quite influential, you know, coming from the perspective of the fda's really the neutral expert agency that's able to decide whether it's a ball or a strike. do you see potential public health value there? >> yeah, i mean, that's certainly a category, you know, issues -- crls related to already marketed drugs. i would -- i would make two points. i mean, nefrms making a cut around that, that's a pretty small subset. so, you know, we might be trying to look a little broader than that. and that information generally is made public. now that said, there could be a lot of value doing that. so i'm not by any means saying i don't think that's a good idea. i think we're trying to look a little bit more broadly at that at a little bit bigger subset and that could certainly be scoped into what we do but a subset where the information might not diffuse as readily but for our ability to release the crls. i do want to try to look a little bit more broadly than that. that could be part of a policy. >> great. in the -- in the journal there's a piece by one of your predecessors where he really lights up on the issue of transparency. is the agency correcting this information that's out there? you just alluded to that possibility. and, you know, i think he personally was just incredibly frustrated when people would say things that he knew not to be true and he felt sort of policies or procedures or whatever at the agency made it really hard for him to just go out there and go that's just not the case. do you feel that kind of frustration? do you think this sis an area where the agency may be willing to jump in depending on the -- >> i feel the sfrus administration every day there's usually about 15 or 20 tweets that i craft that my media staff stop me from sending out. but that's a challenge. it's a challenge for the agency because we have to be very careful about the disclosure of information. you know, and to appreciate that, in order to do our job, we -- we get access to a lot of commercial confidential information and trade secret information. and people at the agency take that responsibility very seriously. they take it seriously because there's a criminal sanction that attaches to the, you know, inadvertent disclosure of trade secret information. but they take it seriously because they know how important it is to do their public health job to have people feel confident in sharing information with the agency. and so there's a concern that if we demonstrate that we are either, you know, not careful in safeguarding the information or will to sort of follow our policies sort of change our posture easily with respect to how we handle information, it could make sponsors more reluctant to disclose information and to be fight us ton when we need it. and that could be a real challenge. i've -- i come up against that all the thyme where for example with the shortage situation, we want to disclose more information about what we're seeing in the marketplace and at the same time it's manufactured date and we need to be careful about disclosing it and we won't get it as readily. we'll have to fight to get it. so it's a constant challenge. i know you've grappled with it as well, but the problems with sort of correcting stuff in the market is often times to correct stuff vuf to put out stuff and sometimes the stuff you're putting out is also confidential information. you know, we do make referrals, as you know, when it's in the context of, you know, securities markets. >> yeah. >> but that's not the same as trying to correct things in realtime on a public health basis. >> right. well, thank you so much. i'm just going to make a prediction, which is what you've announced today is going to be very well received, that transparency has a constituency that cuts across politics. and i think you -- people can see the, you know, or think about what the potential barriers are out there, but think you're going to -- and i hope that it develops its own momentum and the steps you're announcing today lead to more and your ability to feel more confident about doing more things that help people understand the agency that help others develop more effective medical products based on information that the agency has. congratulations on the steps you're taking today. thanks so much for coming today and for your leadership. >> thanks a lot. thanks for your input. [ applause ] >> fantastic. so i am now going to transition us to the next part of the agenda. thanks to all of you for giving questions, i got through a whole bunch of them here and we're going to use the index cards for the next session too. so feel free to write down questions. i'm going to introduce the moderator for the next session and he is going to introduce the man task panels that we have. our moderator is dr. joseph ross associate professor of medicine and public health at the yale school of medicine. he is a member of the center for outcomes research and evaluation at yale new haven health system and codirector of the national clinical scholars program at yale. he is an incredibly prolific writer and researcher on transparency at fda and many associated topics including the importance of data for improving clinical practice. dr. ross. [ applause ] >> thank you, everybody. it's such a test meant to josh to have such an incredible group of people here in the room, and i think we're done, spritright? scott came up and talked about csrs. this portion of the meeting we're talking about open data. the way i like to think about it is more around opportunities, challenges, and implications. specifically, we're going to be talking about two of the recommendations made in the blueprint. we're going to be talking about the sort of category two recommendations around -- that fda should proactively disclose communications to companies when products are not approved, that fda should make pickup public it's clinical and statistical reviews of products not approved, and fda should make its poold data sets, including -- that are masked and deidentified as probl appropriate and the fda's nal sees of data sets. we're also going to be discussing the category five recommendations, that's more around sort of the bigger picture of open data. fda should disclose clinical study reports and given cot's announcement i think for us the onus is on how do that and do it in a twha protects patient privacy while ensuring use ability and feasibility. that the fda should also release the final reports that fulfill post marketing requirement and post marketing commitments, and that when there are clinical trial data the fda should take steps to proactively release this ipd. we're going to have three different people speaking today, and i'll introduce each of the three and then we'll just have them each take turns presenting at the podium and then we'll stit down and take questions. the first speaker is kate dickerson, professor of epidemiology at the bloomberg health. she's also the director of the u.s. cochran center. the second is sean cody, a statistician in the epidemiology branch with nhlbi who runs the biolink data repository and he'll talk a little bit about that. and last is matt herder, canadian harkness associate fellow in healthcare policy and practice and the director of the health law institute at -- you know matt -- how do you say -- the university in halifax, nova scotia. he's been working with our group at yale, c.r.i.t. thank you to everyone who's here and i'll transition to kay. >> thanks very much, joe, and hello to all of you. and thanks to dr. gottlieb for the announcement about the csr. so, my job is to lay, in ten minutes, the groundwork for what you'll hear the rest of the day, and i think a lot of what i said was geared to why we need csrs public. now, you can see just how much information they'll give us. right. okay. and i'm going to speed through these. the idea was that if the slides are made public, then you all will get to see a story, and the full story is here, but you may hear me say, and i'm going to skip this slide, and it's because it summarizes a piece of the story. okay. so, declaration of interest, the usual things. it's federal grants for the most part. so, most of you are aware of this. you may or may not have seen a slide like this, but the evidence generation piece of what we're talking about, the clinical trials is what the fda sees, the clinical trials and some observational studies. then, these are made into systematic reviews where you gather the evidence together and synthesize it qualitatively and quantitatively. then we make guidelines based on the systematic reviews and probably the hardest part of all is getting it into practice, which is implementing the best practices. but what we're talking about is evidence generation and getting the full reports of clinical trials. the perspective that my group is coming from is, how do we get good systematic reviews that are based on actual data? but it's also, as we heard from dr. gottlieb, what the public can see, looking at trials but either you do it in your head or you do it formally and i'm talking about doing it formally in systematic reviews. many of you or most of you have seen this in front of the national academy of sciences and it's einstein and he said the right to search for truth also implies a duty. one must not conceal any part of what one has recognized to be true. and you're also probably familiar with the lancet series in 2014 that said we shouldn't waste research data, and i think in the u.s., it was a little offensive at first to hear people talk about waste, but then, as we thought about it, no, we shouldn't waste anything, and i'm from the generation that turned ketchup bottles upside down, so i don't waste anything and certainly not research. so if the research has been done, i want to know about it, and i want to know everything so that i don't have to go hunt for it in a difficult spot or whatever. okay. so, many of you may not be familiar with a report that was put out by the national academy of sciences labeled "institute of medicine" until recently. you're familiar with the 2015 report, no doubt, talking about the need for open access to data. but maybe you're not familiar with the 1985 report, which has almost identical recommendations, because you have to be old like me to know that it exists. but we've been making these recommendations a long time agent open access to data, and it's just wonderful that we're finally getting there. so, thank you very much to the group that's calling this into fda for what they're doing. okay. so i'm going to think about three things that we need to make clinical trials better reported. one is -- oops, sorry. we have to ask a clear question. and that's harder than it sounds. and what we say is p.i.c.o., population, intervention, or exposure if you have observational studies, comparator and outcome. does the system mathatic review know them? when we compared publications, what's published, and that's how most of us get our information, to what's in a protocol, they differ. so we don't really know whether what was planned was what was done or we're just going after what's easy to report or the editors of the journal are saying you can't report all this, and when people have compared protocols to publications, we see the publications don't describe clearly at all what the population is, what the intervention is, what the comparator or the outcome. and there's a lot of work that's h hidden in this one slide and where we're going from here and a lot of work that's ongoing. but this is very important. we don't know the basics, the p.i.c.o., without looking at, for example, the csrs. but what we get from publications, which is where most of us get our information, it just isn't enough information. so, i'm going to give you some information from studies we've done within our group, and many of the group are here today. called m.u.d.s and it's about multiple sources of data, which is what most of us in science deal with. that is, we aren't dealing with just a single source of, here's what the data show, either individual participant data or csr, but we're dealing with a publication, most of us and multiple publications or sources of data, including the csrs, and then what do we do if those data conflict or something is missing? so i'm going to tell you a little bit about our studies just because there are so many studies out there now that have shown not what m.u.d.s. has shown but little pieces and this has brought everything together. we looked at gabapentin for noo neuropathic pain. it's what's called an off label use and quetiapine for bipolar disorder, which is allowed under the fda, but under a different mechanism. okay. so looking at just two cases. we're not looking at everything that's out there, and in m.u.d.s., we found 21 trials. i'm just going to show you the results that are relevant to what i'm saying for gabapentin. we found 21 trials where the question was -- remember the question was the first thing we had to do -- was it effective in relieving neuropathic pain and we found 21 trials and these were reported in 80 sources so the person who's trying to tie all this information together, whether it's the public or a scientist doing a systematic review has to find 80 sources and that's not easy. but anyway, of these 80, 68 were public and 12 were not public. that meant, either you couldn't find them or you had to go -- in our case, we found nonpublic sources not for everything but for a few of the trials because there had been litigation and we found them online. we were able to compare what's public and what's nonpublic with all the information for these trials. so, looking at p.i.c.o., what we found is what's green or blue there is it wasn't clear. it's not that it was -- it disagreed between the nonpublic and the public sources. that's in red. but what's in green is, it just wasn't there some of the time and so we had to look in multiple sources, including nonpublic sources, to be able to find what the p.i.c.o. is. and the bottom line, based on what we heard from dr. gottlieb today is, wow, the csrs are going to tell us a lot. the blue there, if it becomes public, can now become something that we all go to first to try to find out what the p.i.c.o. is. that information is there. it's just you have to look in these nonpublic sources and if they're public, he's going to do a pilot first, but if all the csrs are public, then we should be able to find out what p.i.c.o. is. so, this is a study some of you may be familiar with, also on the gabapentin for neuropathic pain and other outcomes and it's putting all of it in one slide. i'll see how that works. the red spots are what was reported -- sorry, what's reported in the csr, and then the blue spots are what's reported in the publication, the outcomes. and so, what's in the csr is not statistically significant. it's above that 0.05 line. and yet, different outcomes, then, were mentioned in the publication than were mentioned in the csr, and this is confusing. the ones that were mentioned in the publication were statistically significant. the ones that were in the protocol, designated in the protocols, the ones i wanted to look at, are in the csr. so in the csr, if it's made public, we're now going to be able to see all the outcomes that were examined and not just the statistically significant ones or predominantly the statistically significant ones. so that's outcome. o. p.i.c.o. okay. we suggest some standards and i'm not going to go through these because they would take too much time but it will tell you the whole story, that is, that p.i.c.o. is not clearly described or it's not what was in the protocol and you may not be able to put the data from what's published into a meta analysis, but it's there. it's in the csr. so, if we make the csr public, this should be good, and the data are already aggregated there with an individual participant data, the data aren't already aggregated. you would have to do it yourself and we do have some ways of getting access to that, including y.o.d.a. so this is risk of bias. and this is a complicated graph, and if you're interested in it, you can go to the original publication. evan, who's in the audience, is the first author. each of the trials, of the 20 trials that i mentioned, we're going to assess, are these good trials, what's the risk of bias. and so for each trial, sometimes there is only one publication. that's where there's a dot. and where down below i use this pointer but there are multiple screens. so, sometimes there's only one publication or report, and sometimes there are lots of them. like that one out on the far right, saying 2013. and what you see is on the inner circle, you see, if you only looked at one, what's the worst case situation? and on the outer is, can you find out what you need to find out about the risk of bias. and the answer is, yes, you can. you just need what was nonpublic to be made public. so, the information is there. here it is for all the risk of bias items. the information is there. it's just hidden right now. it's in nonpublic information, like csrs. the one exception to that is missing data, and so we need to deal with how missing data are handled to be able to understand, because people -- and this is statistical, for some of you -- there are different ways of handling missing data that are sometimes not highly regarded. so, finally, we need to meta bias. we immedianeed to minimize metan the systematic review and this shows that using our data from m.u.d.s. and lee has a paper on this that has been submitted, at least, and it's also in mayo-wilson's paper that you can get anything from, yes, there was a statistically significant result using this drug, to, no, there was not a statistically significant result for the same outcome. that's shown all sources up there, the purple, and because the outer confidence limits for the outer tails don't cross over, you can get different results using simulation, depending on which data source you use. and this is kind of scary. you can see the fda source here is the most positive about the outcomes, what we found. we only had two studies where we had fda data. so -- but the scary part here, that we all should notice, is that depending on the source you use, you can get different results for the same outcome. okay. and so, finally, making the individual participant data, the csrs, and the pool data sets and the final research reports public will improve patient care. that's that bottom line. that's the graph i showed you to begin with. it will tell us more about the methods, the risk of bias, and the results. hopefully i've convinced you of that. thank you. >> good morning. i, too, would like to express my thanks to everybody for -- especially the pew and to joe for -- take off the tag? okay. so, what i came to talk about briefly was how our institute has made data from institute-supported clinical studies widely available to the general scientific community. i'm in the division of cardiovascular sciences at the national heart, lung, and blood institute and basically to give you a 50,000-foot view of how we started, back in 1999, the institute wanted to have a more formal approach to data sharing. and so some of the staff got together. we put together a protocol. it was approved by the nhlbi irb and roughly in the year 2000, our first kind of website opened up. and that irb protocol continues to this day. so, the purpose, the reason we wanted to make data more widely available, was in order to take full advantage of institute-supported clinical studies. and the best way, we felt, to do that was to make the data as widely available as possible, under appropriate terms and conditions, to the largest possible audience. so in order to kind of fulfill that purpose, we had to consider, essentially, multiple stakeholders. and of course study participants being first and foremost. so, study participants, this is not true so much in the clinical trials, but the observational studies often have tiered consents, and so the -- when we share that data, we need to make sure that we're taking the informed consent restrictions into account. we also want to be sure that we're taking any confidentiality and privacy concerns into account. but also respecting participants' altruism in participating in clinical research. also, study investigators. study investigators have a considerable investment in time and also their intellectual input in the design, implementation, and conduct of a study. the scientific community making data reasonably accessible in a reasonable amount of time. and of course the public, since it is the public dollars that have been invested in this clinical research. okay. so, all these are somewhat intertwined. i'm going to try to hit upon each of them in turn but just keeping in mind that they are, essentially, all linked together in one way or another. so, for prepares, for the study investigators, when they're preparing data for submission to the data repository, first and foremost is you want as much documentation as you can get as carefully prepared as you can get. so the clinical forms, the manual procedures protocols, a publications list, other relevant documentation such as algorithms for calculated variables, just full descriptions of the data. the expectation is that all the data that was collected during the conduct of the study will be made available. that includes things like procedural tests, you know, things like echo, cts, questionnaire data, the outcome data. of course, the data should be prepared in a manner consistent with the informed consent. so, due to tiered consents, there could be situations where there's essentially multiple -- i don't want to say versions of the data, but the data set then subseted to participants consistent with that informed consent. for example, commercial entities would get a data set that's been appropriately subseted to those who gave consent for commercial use. and again, there may be research use restrictions, such as only certain types of studies can use this data or any type of research. so we need to make sure we take those into consideration. so, of course, obvious identifiers are removed, and from there, hipaa is the primary guideline for removing obvious identifiers. we do record all dates so the date of randomization may be day zero and then all other dates are relevant to that data randomization. geography is generally removed, not always, but if you have a study with hundreds of clinical sites, you may not be particularly concerned. you give them a random site identifier. if you have just a few sites, then you might want to be a little bit more careful about geography. sensitive data is removed when not integral to the study. we have, for example, a large population-based study, collected data. they were young adults when they were enrolled and so there's some data on things such as drug use, incarceration history. that, since it's a general population study, that would be withheld, and not part of the widely accessible data. other data may be recoded to minimize the possibility of any sort of reidentification, and there, we're talking about putting into reasonable groups things such as race, ethnicity, marital status, education, income, things that we would refer to as kind of casual knowledge type of traits that we do group together. also, free text fields are always a concern. oftentimes, you see names or dates or, you know, precise hospital names, things like that. so, we pay particular attention to free text fields. okay, so, for the timing for release. so, for our trials, the guidelines are that three years after the final visit of the participant to their clinical trial site or two years after publication of the primary outcome paper. the idea is that data then becomes widely available to the general research community. and similarly, for observational studies, three years after the completion of an examination cycle or a follow-up cycle, or once -- or two years after a baseline, that baseline data, the follow-up data, the outcome data, once it's been finalized and ready for release to the study investigators, then the idea is it becomes generally available to the general scientific community. so, the intent, really, is that study investigators have this two-year protected period of time. now, during that two-year period, that doesn't mean data is not being shared. almost all of our investigators are open to collaborations and things like that. it's just that data sharing is occurring through internal study-based procedures rather than being widely available to anyone who requests it. the data access process, we do have a website. the first step is to register on the website. fairly straightforward. there's this data request sheet that asks for fairly simple information about the data set you're requesting and some of the things you want to look at in terms of that data set. we do require either an irb approval or certification of exemption from irb review. we do conduct what's known as just an administrative review of the data access request. basically, all we're looking at is to make sure that what you want to do with the data is consistent with the terms and conditions of the data use agreement. and so within that data use agreement, there is a prohibited -- it's prohibited to share the data with another third party. you are limited to three years, but after three years, you just either renew or you tell us that you've destroyed the data. it does have an acknowledgment for publications. and if you do significantly change your project or you want to add project, we just ask that you let us know about it. again, just for kind of an administrative level review type situation. and then on the data use agreement, institutional signoff is required along with the investigator signoff. okay. so one thing i thought i would do -- we have all this and i thought i would give you a rundown of perhaps some of the outcomes from our data-sharing activities. so since we are the national heart, lung, and blood institute, you can see the type of studies here that's part of our portfolio of studies. right now, we have 42 observati observational studies available and 122 trials so almost 800,000 participants represented at present. cardiovascular, heart failure, asthma, sickle cell disease. what you'd expect from a heart, lung, and blood institute. this is essentially cumulative access request for data and the take-home message that i wanted to do with this slide, that red line is community request for observational data and the blue line is for trial-based data and kind of the message here is that while observational studies have kind of dominated the data access request process, you can notice that the differences narrow quite a bit. there's been a considerable uptick in demand for trial data for secondary use. here, what we're looking at is kind of the variety of uses that our data ends up getting used for. so, of course, new hypothesis-generating questions is the most popular. but along with it, people ask for our data for meta analyses with our data and other data they've acquired. what i kind of call clinical trial methods, things looking at trial design or different aspects of analysis of clinical trials. a comparison group. for instance, one of our population studies was a comparison group to a study of nfl players. clinical prediction. it can -- that's been a more recent -- that's -- that top kind of blue bar up there, very recently, clinical prediction models, risk prediction, our data's been used a lot for that of recent. and then other reasons, pilot data for grant submission, simulation studies, things like that. publications, you know, probably the primary metric for us in terms of the value that we get from releasing data, making it widely available. we have, presently, 872 publications, and you can kind of see the steady rise in the publications. of course, as more data becomes available, as the requests do. that's all. oh, i should summarize. so, demand from -- for secondary use continues to steadily increase. we grow each and every year. clinical trial and observational data from repositories, they fulfill a range of needs. it's not just -- it's not just to replicate. it's not just for new questions. there's a range of needs that are -- that repository data can fulfill. review for deidentification usability and post-request support on nontrivial tests. once you make that data widely available, there is an expectation that you're going to support it. and so these activities do occupy considerable amount of time. just to kind of give a brief overview of the type of level of effort, and this is mainly on our contractor side. but, so, the web platform, the maintenance, about -- almost 2.5 ftes, analytic support -- i didn't talk about our biospecimen side, but the bio-repository requires a significant level of effort as well. along with communication efforts and then the total management. and that was all i had. >> so, thank you to the organizers for having me. it's a pleasure to be here. so, after hearing a fair bit about the why and the how, my task is to tackle the question of what it will mean from a regulatory point of view for the fda to increase its institution openness. and by openness, i mean greater openness in the fda's decision making as well as the data behind its decisions. recommendations 8 through 10 and 16 through 18 of the blueprint articulate concrete ways in which to improve transparency. specifically, the blueprint calls upon the fda to publish not only its decision to approve drugs and its underlying action packages but also applications that have been rejected, withdrawn, or abandoned, which i'll refer to collectively as negative decisions. and the blueprint also calls upon the fda to publish a great deal of data underlying those decisions, including pool data sets, csrs and in some circumstances, individual patient level data. separate from the benefits of patient safety and innovation efficiency that we've heard a little bit about already, publishing a more fulsome set of data and decisions will also afford three important benefits to the institution of the fda. so, the first institutional reason for greater openness concerns institutional memory and agency discretion. the second surrounds regulatory ha harmonization. and the third is governance. i'll start with institutional memory. it has become received wisdom within the fda that certain information, including negative decisions and nonsummary data, cannot be disclosed because it constitutes commercial confidential information. this obscures a much more nuanced history within the fda itself about what information it has chosen to disclose at different points in time, and it also obscures flexibilities that legal scholars outside the fda have repeatedly suggested could be invoked in the absence of legislative reform to improve transparency at the fda. at times, the fda has agitated for and successfully exercised its discretion in the direction of transparency. the remarks of former commissioner donald emphasizing a democratic rationale for transparent in the late 1970s are often cited. the efforts of various fda officials, some of whom, like robert temple, who are still at the fda today, to push for public disclosure of what was referred to in the late 1970s as the two-inch, two-foot, and 20-foot data reports should be known more widely. but even before those initiatives in 1972, the fda has authored a set of public regulations which allowed public disclosure of the safety and effectiveness data behind drug applications that were rejected, abandoned or withdrawn unless, quote, extraordinary circumstances were shown. in other words, as early as 1972, the agency proposed and briefly implemented a practice of disclosing safety and effectiveness data, underlying the various sorts of negative decisions that the authors of the blueprint are calling for today. as well in 1974, the fda announced its intention to publish summaries of safety and effectiveness data behind drugs it approved, even though a statutory authority to do so was not codified until the passage of the hatch-waxman act in 1984. on its face, section 104 of that legislation also imported the agency's interpretation of the foia law but widening it to include the safety and effectiveness of data rejected, withdrawn, or abandoned but also approved ones as well. then congressman henry waxman sponsored the addition of the disclosure provision to the bill. after it was passed in the house, however, then commissioner of the fda frank young wrote to senator hatch, advising that the fda would consider that the extraordinary circumstances test would have been met provided the sponsor demonstrated the data had ongoing commercial value. the fundamental point here is less about where the fda ended up and more about the discretion that the agency continuously exercised about what is and what is not cci. court decisions through the 1980s and beyond constrained that discretion, but so too has the agency gained a statutory mandate from congress to disclose safety alerts, warning lerts letters, as well as, quote, other materials deemed relevant by the fda to the question of drug safety. implementing the blueprints recommendations would serve as a critically important reminder of the discretion that the fda continues to enjoy about what information it can and cannot share and to correct the problem which is to some extent a problem of its own making. the second institutional reason to improve the transparency of fda's decisions and data concerns regulatory harmonization. for years, regulators have espoused this goal. apart from the pilot that we heard about this morning, insofar as the fda continues to withhold negative decisions and nonsummary level data, including csrs, and individual patient-level data from public purview, the u.s. regulator is apt to be out of step with other jurisdictions. the european medicines agency is already proactively publishing csrs and other regulators are following suit. health canada has introduced regulations into canada's parliament which would require publication of safety and effectiveness data behind all approved, rejected or withdrawn drugs unless the data was not actually relied upon during the review process or the method is deemed an asset that is exclusively use bd by the spons in question. if they succeed, the fda's policy of nondisclosure will put patients, physicians, researchers, and companies at a relative disadvantage in terms of what they can learn about the fda's decision-making compared to foreign regulators. further, if the fda chooses to go its own way and maintain the confidentiality of negative decisions and nonsummary data, it may soon find itself forced to disclose information in response to regulatory decisions outside the united states. that is because the legal category of commercial confidential information does not extend to information already in the public domain. as the ema in health canada render regulatory decisions and publish the underlying data, the fda may face an increase in requests under foia legislation for information, which by virtue of its availability from foreign regulators, can be cci no more. in this sense, the fda would not only be out of step with its regulatory partners but risk losing a level of control over its own processes, in particular about the timing of disclosure. which takes me to my third and final reason for improving the transparency of fda decision making. good governance. much has been said about the potential to better inform actors outside the fda. where the transparency of its decisions to improve. if the fda really is unique in the analyzable data sets it secures from industry, then the benefits to pharmaceutical knowledge production and patient safety can hardly be understated. however, the benefits of opening up the fda's decision making to public scrutiny are not one-directional nor purely instrumental. rather, there are benefits to be gained from within, in terms of the decision-making performance of the agency itself and in terms of the larger public function the agency serves. as daniel carpenter puts it, access to the fda's decisions and the reasons that underpin them, not just the data, are important for patients and researchers but also citizens. studies comparing the outcomes of the fda and the ema reviews of the same drugs based ostensibly on the same data vaguely allude to the fact that economic, political, and social cultural factors may account for any observed discrepancies between the two regulatory outcomes. in a cadre of social scientists who have closely observed the fda and other regulators from within have done well to explain what those factors look like in a series of cases. these studies provide only a glimpse of the full complexity, con tin je contingency and contested nature of regulatory decision making. adopting the blueprint's recommendations and perhaps going even further, not simply disclosing the data but rather providing the full sets of reasons for a decision, for example, how it's been informed by past reviews in the same class of drugs, would represent a remarkable expression of the fda's commitment to good governance. i imagine that that level of openness might sit uneasily with some regulatory officials, but the fda only ever exists in controversy, and at a time when public agencies are under significant threat, revealing the full complexity of the fda's decision making, rather than hiding it, may help preserve its legitimacy as an institution. thanks very much. >> we have a bit of time for questions. if anyone has question. i'll help get us started. just thank everybody -- the three panelists for getting us started on a great discussion. i think that while people are passing in questions, i think it would be useful for us to start with the issues around sort of usability and feasibility. when we think about open data. there's an interesting -- there's a number of moving parts. you know, when sean -- when you spoke, you talked about the work that's done to deidentify the data and prepare it. that's mostly around the issue of distributing ipd, not the clinical study reports, but if we focus on the clinical study reports, you can imagine that the -- the efforts that are needed to privatize or make sure that there's no patient information in a csr, actually more difficult than in a flat data set. where it's easier to sort of aggregate up and to strip identifiers in whole as opposed to page by page. so, what i'm -- and at the same time, peter and tom jefferson and comments within -- on the blueprint report that were published in concert with the blueprint in the journal talked about the need for easier access and in terms of whether you need to have an application to apply for the csr as opposed to ipd and dr. gottlieb himself talked about just postal thing them on where anyone can download them. the question i'd like to hear the panel reflect on a little bit is how to balance these usability and feasibility issues in concert with the work that's going on to deidentify csrs for proactive distribution by the ema. how do we make sure that the csrs that are made available can be used and what kind of a pathway for access do you envision is the best? i think ideally, what dr. gottlieb proposed, which is bold, would be best for science. but maybe you guys could sort of talk about it one by one. >> you and i have talked about this. i think csrs are hard to use. but because they're aggregated data, i think, in a way, they're easier to use than the individual participant data. one of the reasons they're difficult is because they have everything -- everything in it. and in multiple places. and so, let's say two people do the data extraction in an 8,000-page document. they might actually extract different information. so, i think, as dr. gottlieb was saying, having an electronic document with pins or pointers might help us quite a bit, and maybe the whole electronic nature of things can make this more doable. but it's definitely something we have to pay attention to. >> you know, i'd have to say i would concur. as an nih institute, you know, we fund primarily academic type of trials, so it's a little difficult for me to relate. you know, some of the things that we do involve not only checking for obvious things with the data but will also have to, because of the usability side of it, is we basically have to check all the data and compare it to all of the documentation. that's what's kind of the heavy burden part of our review process. but -- so some of those, in a way, relate, but i do think that there are -- i think there are technology types of approaches that could probably help tremendously. >> i guess the only point i would add, and i'm a legal scholar, so i'm less able to think about usability from a research of the data perspective, are the institutional considerations about when exactly to disclose and what kinds of processes need to be in places or not when the regulator is trying to make that decision. i find -- just, you know, i know i'm raising a little bit of ancient history, but i think it's useful, as a reminder, because some of these sort of very detailed questions on the how side have been -- have been thought about in the past, and it's interesting that even though it seems to be lost in the present policy discussion, that the debates in the '70s around releasing way more data, one of the strongest recommendations that came out of this independent panel of drug regulations is that we should release this information not when decisions are made but actually during the regulatory review process. so -- and that doesn't even seem to be a question right now. everyone's talking about after decisions are final in some way. so, the question of timing, i think, actually could be worthy of some further thinking, although everyone's taking it for granted that we'll wait until an actual decision is made, as opposed to actually engaging independent researchers more actively in the review process. i think that's something that we could think about more. and i'd be curious what clinical researchers think about that. the other point is around these exceptions and redactions, and there are obvious resources constraints that we need to think through, but the wording of some of the exceptions that are being put into place about information that will be withheld is quite controversial is i'm not sure we've come up with good language as of yet. i alluded to the canadian policy developments and there, the thinking is two exceptions following a disclosure of the regulatory decision and that is information that is not actually relied upon but is somehow being exchan exchanged by the sponsor with the regulator, and secondly, methods, assays, other kinds of sort of information that is more procedural in nature which is in, quote, the exclusive possession or use of the sponsor in question. and i worry, with the latter exception, that that potential -- that has a potential to be quite wide and the regulator may not have the capacity to screen for whether it's exclusively used or not. so i think more work needs to be done when we think about the idolized model, about the wording of exceptions to disclosure following a decision. >> just going to read this question that came in for matt, which may enable you to give us a little bit more background on the canadian policy, because someone is asking, can you provide us with additional information on the effects, if any, thus far, from health canada's regulatory decision around open data. >> sure. so it's a decision that is very much still in process. so, health canada passed major amendments to the food and drugs act in 2014, and there were a variety of provisions around transparency in that legislation. one of them included -- and this is where, i think, as a matter of law, the canadian approach may be on stronger footing than the european one -- and that is because there's an explicit power in health canada's governing statute to revise and change what the scope of proprietary information is. so that was added to the law. at the time, it introduced a broad definition of, we call it, confidential business information, which basically codified health canada's long-standing practice of treating unpublished safety data as confidential business information, but it included a regulation-making power to change that definition or pull specific kinds of information outside that category. and so it introduced, in mid-december, regulations that actually would do that, use that power to say, once a decision is made, information which we have treated as cci or in canadian terms, cbi, we are going to call it clinical information, and release it. and so, with those two exceptions that i've mentioned. so, those are regulations are open for public consultation, including to folks outside of canada, and after that, the health canada has announced it will actually produce a guidance further interpreting it, and we think they're going to look like what the ema has done. but the ema has largely done what it's done as a matter of policy as opposed to governing law, in my view, so i think the canadian approach is actually on stronger legal footing. >> i'm going to ask a question. what i was -- i was struck by, in sean's slides, the vast uptick in use of the shared data other time and the sort of low -- the low usage initially that sort of certainly has grown exponentially and that's an experience we've had working on our project. and contrasting that with -- the blueprint is really about forward thinking going forward, what can we do to make the regulatory world, public health world, better, more informed going forward, yet when kay, your presentation, the vast majority of the work you're describing is about gabapentin, a drug that's been on the market for 25 years, but remains one of the most commonly used today. how do we think about retroactively applying some of these recommendations? how do we think about, you know, these drugs that are in wide use where the likelihood of cci is probably substantially diminished around both the ipd and the csrs. >> i'm so glad you brought that up, because there's so many drugs that are in current use and are highly used, like gabapentin, and that's off label use, so we haven't even gone there. and how are we going to get the data. so we could go -- and that 2015 data sharing recommendation from the iom recommends starting, i think, 2007. anyway, recently. and -- but there are many, many drugs we use that were approved before that. i think some of this stuff can be released, and it's there. and maybe we should just release what we can. >> do you think that there's a -- there's a group -- there's medical researchers who are going to take advantage of it? that's always the question in the slow uptick. >> systematic reviewers will. absolutely. but again, we have to deal with this, wading our way through the csrs, which can be a big job, and i don't know that electronically, they're set up right now. one of the things i was going to bring up when we were having the discussion, and maybe it's just my naivete, but do we know how csrs are put together? and my understanding is that it's multiple people at industry, at a company, are putting it together, and maybe if we modify how they're put together, they can be easier to wade through and less duplication of effort. >> it's a great question. i'd probably have to ask a question first. so, in terms of what gets released as -- with the csrs, would that be prepared by industry? or would that -- is that something that fda is going to do? >> i think it's a good question. i would expect it will be prepared by the sponsors as part of the submission process, but fda would probably have a close hand in ensuring that it's appropriately deidentified. >> so, you know, one of the issues that we have with -- >> correct. >> so, one of the issues that we have with, you know, these academic trials and studies done by a lot of different, you know, academic units, is that there can be a lack of consistency in the way they're prepared. and i think that's -- that's one of our kind of hurdles is to try to get more consistency, but you got to maintain that flexibility, too, in the way that data is prepared. so, if some of those could be done more centrally, then probably, you know, i'm just kind of in a way talking off the top of my head, just from our background, then some of the legacy types of csrs, you know, that might be the way to get those released. but it would probably have to be targeted, and i'm not sure how you would try to set your priorities on targeting those. >> so, i would just add a wrinkle there is that, you know, consistency in the name of usability seems like a good thing, but it's -- if we're standardizing, the question of standardizing to what becomes quite critical. because you know, if we think about multiple jurisdictions with different sort of definitions of what's proprietary, what's not, how far are you need to go in terms of deidentifying the data in light of privacy concerns, there's a concern, also, that if we make it easier, such that industry formats this one time for multiple submissions for public disclosure eventually that the sort of greatest protection of privacy, for example, in europe, which i understand have to be stronger as a matter of law, is going to compromise the ultimate usability of those documents and i don't think it's hard to imagine the highest standard in terms of protecting privacy and the largest scope of what's proprietary and what's not, doing a lot of work in the process of coming up with one common submission. so, admittedly, there are efficiency concerns about having to do this multiple times, but there's also a tradeoff with, which standard is going to undermine the usability of those dn documents as well. >> we've been doing our data repository for 20 years and we can't come up with one. because we're up against nih policy and, you know, our own policy and people's own perceptions and things like that. it's hard. >> well, what strikes me with that comment is, on the one hand, in your comments, you talked about how, as canada and the eu get ahead in terms of what they're releasing with respect to cci, it can, you know, undermine the argument for retaining the information within fda, but on the other hand, if they're not released certain patient identifiable information that makes the documents usable, the fda may actually get out ahead of them in terms of their -- >> yep. >> which sets a -- they're going in slightly different directions but both are moving toward greater transparency. >> it's quite dynamic as to which regulator is going to lift or pull others up or down. >> but don't we have reporting standards for papers and then each journal says, yes, and we also want this or whatever. like icmg asks for registration by a certain date, which is, you know, a fine-tuning, and then they either endorse or implement the reporting standards, and it hasn't actually worked that well. but i think that if we have reporting standards, we can certainly have standards for the csrs. >> well, so, i'll just note that having participated in some of the discussions on the european side, there is a real need for standard-setting, particularly from statistical associations around appropriate, you know, deidentification and what needs to be taken out as opposed to the hipaa standards that fda generally uses. the european union is, you know, is pushing much farther, which has the sort of risk of making the data unuseful. sean, i wonder if you could speak to something. so you know, one of the biggest concerns, when you start talking about making data more available, around both among medical product industry and clinical researchers is that people won't want to participate in trials because they're concerned their data is going to now be made available. this came in as a question. what was your experience or the experience of nhlbi when it put this policy forward, started making the data available? how did it affect clinical trial enrollment? how did it affect clinical investigators applying to use nhlbi funds knowing they're going to have to make their data available. >> another great question. so, in the beginning, there was resistance. i think from both the observational studies and the trialists. probably the observational studies people were a little bit more open, because by design, you know, an observational study is meant to generate many questions, you know? but everybody wanted to make sure they had that opportunity to ask the questions they really wanted to. and i think observational studies folks kind of caught on a little quicker. probably the clinical trialists now are about where the observational study people were, you know, as they started to get -- to get -- to be more open. from the trialist perspective, a lot of it has to do with understanding the nuance of the data. and the trialists who put so much effort into that study feel that they're in the best position to analyze it. and you know, we can understand that. that's why we do have, you know, a two-year protected period of time. given the whole push with the institute, well, the medical journal editors report, the iom report and even with clinicaltrials.gov, i think the trialists are at the point where they understand the need for open sharing, but again, the hows are still, i think, a problem. you know, being a funder, if you offer the money, they will take it. you know? so, the general approach is, you know, there are -- there are the bigger studies where we can get this in the notice of award. and you know, they will do it, especially if that's what you do to get the money. it does seem like we need to really have a better way of rewarding sharing, and i don't think we have that yet, and i don't think -- i haven't really heard anything that would really jump that, i think, to kind of the next level, to reward the sharing that investigators do. >> so i just got the wrap-up note from josh, but i wanted to thank our panelists. i wanted to thank josh for bringing us together. i want to thank alan for having pew host us. thank you. and we'll keep going. >> thanks. >> well, thank you very much. that was really terrific, and really right on point to have an announcement about the study reports being made available and then to have such an incredibly informed discussion about the utility of those very reports and some of the challenges, strengths and weaknesses of those reports to relevance medical science right afterwards was terrific. so, thanks so much, and thanks to dr. ross for moderating. i also wanted to specifically thank mike for his work with the arnold foundation, getting this whole project together, and to thank c-span3 for being here and broadcasting. made me think, as i was listening to these presentations, you know, if somebody is just clicking around, doesn't know all the acronyms, lands on this discussion, what is it about? how does it affect them? and in fact, transparency at fda has just a huge impact on many different people, and i'll give some examples. i think patients may want to know what happens to promising products. they've heard about it. there's something in development. did it -- is it -- did it get approved? does it work? if it didn't get approved, why not? those are pretty basic questions that if you or someone you love is suffering from a disease, you want an answer to. doctors care a lot about transparency. i don't know if this is going to be playing in any waiting rooms, but the physicians can learn a lot from the data, and we talked about that. you heard this on the previous panel, about whether drugs actually work for certain topics, for certain populations of patients, and the data we're talking about can really extend what's in the published literature and really help inform clinical practice. researchers can learn about development pathways that don't make sense or, as dr. gottlieb said, some findings that are -- happen with one drug could have implications for other drugs. it could actually save time and money and lead to cures faster. and then, we're now going to talk about another constituency that's pretty important to the whole process of medical products, and that's the companies that make them and the people who invest in those companies. and so it is my pleasure, for this part of the conversation, to invite jonathan leff to join me up here, and i will come over and introduce him. and we'll get started. >> thank you so much for coming. >> thank you. i need to turn my mike on. >> great. jonathan is a partner with deerfield management and chairman of the deerfield institute. he joined this company in 2013. the company focuses on venture capital and structured investments in biotechnology and pharmaceuticals. prior to joining deerfield, for more than 16 years, jonathan was with warburg pinkus. he's been active in policy discussions related to health care and medical innovation, served on the executive committee of the board of the national venture capital association, and led that group's life sciences efforts. he's also served on the emerging company section board of the biotechnology industry organization. he served on many related nonprofit boards, including the reagan foundation for the u.s. food and drug administration. he is a graduate of harvard university and the stanford school of business, and we really appreciate your coming. >> thank you. thank you, josh, for that very kind introduction, and it's good to be here. >> good. well, i really appreciate it. i think i want to jump in. we're going to talk and we'll have a chance for some questions again through the index cards. i'm going to start with a provocative question. does the investment community view the u.s. food and drug administration as a necessary evil or as a critical ally or something else? >> well, that's a great question. i obviously can't speak on behalf of the entire investment community, nor certainly the biopharma companies and their executives, but i would say that overwhelmingly, fda is viewed as a critical ally, and it is clearly recognized that the ability to do what we all do, to invest the huge quantities, the hundreds of millions of dollars necessary to develop drugs, and to bring those drugs to market in a drugs to market in a way that is sustainable is a function of having the fda, the gold standard of the fda as the regulator. and i think that distinguishes that from most other industries where they're a necessary evil or bureaucracy that slows things down. certainly from the industry you hear complaints about specific decisions or specific interactions with fda and that is to be expected and inevitable, there are so many perspectives and complexities here, but it is almost universally recognized without a robust and effective fda, we couldn't make the bio, pharma work. >> what is it that investors want to hear and see. what kinds of information can come out of the agency, what kinds of explanations are people interested in. >> stepping back, i want to thank you you and co-authors for the focus you are bringing to this important issue. it is one i feel passionately about. i have been asked at various times in the last 20 plus years that i have been in the investment business and venture capital business and biotech what are the public policy initiatives that could be implemented that would make that system of innovation work better. one of the things i have often said is greater transparency and access, public access to the kinds of information that exist at fda. i often get a stoney silence from bio pharma companies when i say that, they take a somewhat more cautious and skeptical perspective and we can talk about why, but from my point of view this is a great opportunity not only to do the kinds of things highlighted already today, to enhance the decision making medical professionals can make in terms of using marketed drugs but to make the whole enterprise of developing drugs better and more innovative, more efficient. now why is that. the reason i say that is very simply the whole purpose of the capital markets, investment markets is to direct investment capital to the most promising opportunities. and we all have seen the statistics about the hundreds of millions of dollars it cost to develop a drug and the high failure rates in this sector. every time that huge dollars are invested in a drug that doesn't ultimately succeed and get to patients, those are dollars that cannot be invested somewhere else, those are dollars that were in effect wasted. now, you're always going to have that happen because there's so much. >> can't be right all the time. >> can't be right all the time. to the extent we can be right more often than we currently are and that dollars can be allocated more thought fully, based on better information will get more information, better drugs and better return on investment that turns into more capital, more dollars going into things and dollars going to projects that ultimately serve the public health better. >> fair to say that transparency has direct and indirect effect. directly because you can pick up a document the fda releases, response letter, and learn something that could influence your investment decision based on likelihood a product will succeed or fail, and indirectly, if the fda releases large quantities of information, people like professor dickerson and others are using that and producing academic materials that provide insight into the decisions you're making. >> absolutely right. there's the immediate company sp information one wants as investor to make the best decision, one wants to know as much as possible what can be known about the drug and experience that's been had in patients in clinical trials to date and incredibly importantly one wants to know what fda and the company has said to each other about it, one wants to know if they've reached agreement on a path to approval, and whether the company is studyi studyi studying end points that the fda deems informative to support approval or whether fda has serious questions and uncertainty about that. one wants to know that. so that's the immediate company specific or drug specific information that one wants. but then also as you say to the extent that the whole enterprise becomes more educated about what's going on and one can inf infer learnings from one set of trials to other drugs in the same class or same condition that are being developed, that's extraordinarily important. not just clinical trial information but regulatory insights. when one company has or develops together with fda a better idea about how to develop a drug for a particular condition, what patient population to study, what end points to study to provide the most robust, sensitive treatment effect that's measurable, detectible. one wants to know that not only for that drug being developed but for the whole community that's active in that area. these are incredibly important issues. i will just say i endorse essentially everything in the blueprint for greater transparency for the fda while recognizing there are counter veiling arguments, complex tease with commercially confidential information and burden put on fda, and when you get to patient level data, confidentiality, and they're legitimate concerns that must be addressed and managed on the other side. but endorse essentially everything that's there. i would say i would go much further. easy for me to say that, but i would certainly endorse release of complete response letters or at least as was talked about with the commissioner this morning, release of clinical and regulatory information, not so much manufacturing information that may be confidential. understanding why fda, what fda found to be deficient about a company's file for approval and what they're asking for in terms of additional information to try to address that is incredibly important. but that even for me is the end of the road. most of the time we're investing in companies or drug development programs that are not yet unfortunate enough to have received a complete response letter, but we're still very, very interested in what fda has to say about it and what the company has presented to fda in the context of the back and forth. >> right. someone might say why not ask the company, if you're financing in, why wouldn't the company just hand over all of the documents? i mean, help me understand the need for fda to have a policy rather than for you to store it with the company. >> that's a great question and often what fda officials will say, don't put this burden on us. companies can release anything they want, it can release the full correspondence if they choose to. companies essentially never do. i can count on one hand the number of times i have seen primary fda documents released in entirety by companies. now why is that. this is a great perversity in the capital markets as i see it. i spend most time investing in private companies, venture capital type companies. my colleagues, others at my firm spend a great deal of time investing in public companies. when we invest in private companies, we enter into confidentiality agreements and as investors always demand to see the primary fda documents. that's a condition to invest and we will essentially never invest if a company won't hand that over to us under a confidential agreement. sometimes companies resist a bit but ultimately if they want our capital, they're going to provide that. and those documents, the back and forth between the company and the fda throughout the drug development process are very often the most informative things, among the most informative things we get to see in making investment decisions about companies. and often we learn things we wouldn't have learned anywhere else. unfortunately, often we find out, or sometimes find out the way management has conveyed their interactions with fda are not quite reflecting the full texture of what's in the documents, so they're incredibly informative. when we invest in public companies, we essentially never get to see that, we can't demand that. you can demand it and companies can say no and they don't release this information. so the perversity of if it is the most important information as an investor in a private company, a similar situated public company, billions of dollars of investment decisions are being made all the time in the absence of anyone outside those companies, and the fda seeing that information. that's an enormous perversity. another great perversity that always strikes me is there are requirements for public companies to release material information about their business and the sec is the traffic cop for all of that. companies are required to release any material agreements that they enter into in the course of business, if they're public companies. and agreements are filed as attachments to sec filings. i can and any member of the public can go and read the lease that a public company, a biotech company entered into with a landlord. but i can't go read the spa that the company entered into with the fda. >> stop. key criteria for success of studies? >> if you ask me how many times i care what's in the lease that they entered into versus what's in the fda correspondence, the difference, there's no comparison. there's a great perversity in what is out there in public information in biotech. >> it sounds like it is your view more data, better decisions that you could make, more likely that medical products we develop that are successful, that will get support that patients that want drugs to be developed will be able to have drugs developed with greater transparency. i want to go back to the stoney faces. despite this advantage and people investing in companies, really wanting to see this information, companies are skeptical, and you know, i wonder whether this is a case where in their own case they're worried about something, but you know, for the whole enterprise it would advance if there were greater transparency. i am wondering, how do you understand their stoney face as you put it. >> well, there's a secret unspoken reason for it from my point of view, and i'll tell you what it is here. companies are afraid their stock price will go down if all of that information is in the public domain. now, and i don't mean to minimize legitimate concerns that are there as well, and the competitively sensitive information is very much at the top of the list, and that's a very legitimate concern, but i believe that, you know, thoughtful people working through that can manage that in the same way you can redact competitively sensitive information from sec filings, but it doesn't change the fact that it is deemed important by the sec as it should be for capital markets to have access to the robust flow of information while redacting what is competitively sensitive. it is a bit of a prisoner's dilemma because companies are afraid if they're the only one to release their documents and this goes back to the question as to why don't companies do it, if one company does it and nobody else does, in general, what companies say about their fda interactions is putting the most positive face on what's in the documents. if you release the documents, you see all of the sordid details and questions that come up in the process and uncertainty and so on. now, i firmly believe if that were out there for every company in a level playing field, if it were automatically released, most company stock prices would actually be higher because the greater transparency will lead to greater confidence and investment decisions. what happens in reality is every time we as investors hear a company describe their interactions with fda, we have to put a discount factor on because most companies are forthright. most companies describe it accurately, although even then there may be details you wish you had that aren't described at all, but most describe it fairly. some don't. because you don't know in any given case, you have to ascribe some kind of discount factor to things i may not know about the company's interactions with fda that i may find later that will burn me. >> how does the discount factor have an impact? means less money for them? what's the impact. >> it means instead of being 100% confident that i feel like i know that when the company tells me, for example, we agree with fda that the end point that we're using in phase three program is supportive of approval, so if we hit that end point, fda will agree we have demonstrated efficacy in a manner to support approval. they ask that question all the time. when you hear that, you say okay, i am 90% sure. therefore, whatever the valuation is on the business, if i think the company is worth $300 million, maybe it will be worth $350 million, if i were really sure what they were telling me was true, and then -- >> that has all kinds of implications. >> yes, for the company's ability to raise capital, issue stock, raise capital to advance clinical trial development programs, the companies care immensely as they need to what is the valuation of their company, what is their stock price because that information their ability to raise capital. >> so if we have a situation where individual companies which are very worried about specific products don't want to be the first one to put all their information out there and potentially have it picked apart where all of the competitors aren't doing that, yet at the same time if all the information was out there, the whole enterprise would go forward, maybe stock prices would be higher, and more importantly, investors from the company perspective would be interested in in investing more. how do we get over that hump. in the context of what dr. gottlieb announced today, he is taking some steps that he hasn't taken before, he is taking modest steps. he is starting with the csrs, assigned someone to look at 100 complete response letters. he seems to have a few ideas how to get the companies to move forward. he seems to be saying that part of the deal will be a different more coherent review they'll get if they sign up for this. but what are the kienlnds of ths people can do, not just the investment community but others, if they want to support what the commissioner is doing and see him go further? >> i think it is a matter of public policy. i don't think that this prisoner's dilemma can be resolved by ethem releasing information, they aren't going to unless everybody else does and everybody else won't unless they have to for some reason. seems to me, and i am not the expert on the legalities, but ultimately it falls to some kind of legislation or policy decision by regulators to say we're going to make available this information or we're going to require that companies make available certain categories of information. simple way of thinking about it, simple way to do this would be for fda to release everything in the way the sec releases essentially everything. i mean, i can go and see not only a company's sec filings but see drafts of the sec filings that were submitted to the sec that the sec commented on and back and forth that led to the final filing. would certainly like to be able to have a similar concept at fda where all of the documents that already exist, companies make submissions to fda in the context of formal meetings, fda responds to those with formal responses, all of those meetings result in formal minutes that are ultimately released or provided to the company. this is probably not practical in a single step, but i would like to see if we could envision where we would love to be and get to somehow. if you could release all of that with mechanisms to redact and enable companies to redact information that really is competitively sensitive, that would solve the problem. you know, we're not imposing a whole lot of burden on fda, they're already producing the documents, we're saying we're going to post them on a server somewhere so the public can access. this redaction process would require some meaningful resources, i don't mean to trivialize that, but it is a whole lot easier than what is often talked about and that i think fda struggles with for good reason, which is the idea that fda should police company's public statements which in the absence of doing anything like i'm talking about here, it probably is a good thing for fda to be able to counter truly misleading information that's put out into the marketplace. so i would endorse that. but that's a huge burden on fda to figure out, track that information, figure out when they should take that step versus when they shouldn't, potential for it to become political is enormous. it would be a lot easier if fda didn't have to police it because the real information was in the public domain, so if companies decide to extend that information, everybody can see it. >> well, i hear loud and clear your confidence in fda in terms of the products and interest in transparency for the sake of investments. i am curious where you are on the spectrum. let's assume, i can understand you're not maybe getting standing ovations with this in a roomful of biotech ceos who are worried about their particular products. what about in the investor communities, are you all the way on the edge of the spectrum or do you think there are other people out in the investment community that share your perspective on transparency? >> perhaps we'll get to test that because people will see what i've said, see what reaction we get to that. honestly, it is not something that's talked about enough. i served on board of venture capital association, led their life science effort, and this was never high on the agenda, and maybe it will be now that people like yourself and others here are taking a step to give this issue a greater public profile and greater prominence. i think when investors really think this through, it becomes hard for me to see how investors would find greater transparency of fda related information to be anything other than good for the innovation enterprise. it is a more complicated question as we talked about for biotech and pharma company executives who are worried a lot about losing control of the flow of information and also legitimately worried about enabling competitors. >> has to be done well. and maybe a little more across the board. do you think that the investment community has any leverage that could be brought, maybe as one investor, you can get a public company to change its behavior, but if a bunch of investors said you know, we need to know the complete response letter, so it should be released, or you know, some statement. is that a potential source of leverage, even if it is to be leveraged directly on companies, it could be leveraged through the policy process for people to realize there's a broader constituency, not a democrat or republican issue at all, that it would advance the science? >> the investors and community when it gets organized through trade groups can be effective like national venture capital association can bring to bear some positive attention to these kinds of issues on the political sphere, because it is not viewed as when formal or biotech trade organizations say something, it is always a question through the lens of is this trying to increase the profits of the companies. when investors try and speak about greater efficiency in the entire investment process, innovation, enterprise, i think very often policy makers respond favorably to that. now, overcoming if this is something that biotech and pharma industries fundamentally resist, that's a very difficult thing to overcome, and i'm not the political expert, but that can be a very challenging environment. i think investors can play a real role helping to voice the issue. >> i think there's a con v convergence, you have interest at the top of the fda, you have what other regulatory agencies are doing around the world, interest in the investor community, strong public health rationale, and hopefully this can come together and everyone can play a role in really getting the process to a new stage of transparency that benefits from multiple constituency, including companies. i think that's the goal. i think checking whether there are other questions. let me say how happy i am that you're here. wish i had known you at the fda, could have seemed up at things. it is important for people to hear different perspectives and it is very important to understand, and if there's one real message from today, it is that transparency has benefits for all of our ability to get better medical products and it is a team effort and there's a lot of opportunity for the fda to move forward. fantastic. thanks so much. so i think we're going to keep going. we're going to move on to the last panel which is about correcting this information. i'm going to ask dr. kesselheim to introduce the panel. he is an associate professor of medicine at harvard medical school and faculty member in the divisi division, he created and leads the program on regulation, they are -- therapeutics and law. he has written hundreds of papers. i know he is on vacation when i only have three papers to review in a month. there's a group of people in the academic world that believe that aaron is not just one person, but multiple people that are aaron kesselheim. he is an author off books on medications and devices, from policy and research and legal perspective. editor of the journal of law medicine he d medicine ethics, and he is also very, very committed to having the fda work more effectively for people, including all of us that count on it. aaron, thank you so much. >> thank you very much, josh, for that night introduction on behalf of the collective aaron kesselheim. i am aaron kesselheim to introduce the final panel of the day that relates to another important set of public health functions, that greater transparency about information and regulatory decisions about the fda can play. i want to thank alyou for hosti this, for organizing the publishing of the special supplement, and of course to commissioner gottlieb for his remarks at the beginning of the day. so the focus of this final panel today is the fact that in certain high profile cases, manufacturers have released incorrect or incomplete information into the market. that was directly counter to information that contemporaneous information the fda had. and in the 2010 fda transparency taskforce that he led, they noted selective publication of clinical trial results has in the past created a misleading picture of safety and efficacy of a product with negative implications for public health, and as the practice now still stands, manufacturers have wide latitude to publicly characterize daylight characterize data submitted to the fda without the fda correcting the record. something we'll talk about in the next hour is in these cases, diagnosis the fda have public health or ethical responsibility to correct the record and if so how. in the blueprint, we outlined a couple of principles that might help organize this kind of approach. first, should the agency adapt a basic set of standards when to correct misinformation. should the agency give advance notice to manufacturers regarding concerns that they have, if so, how would that happen, and should the fda disclose scientific information that's the basis for its concern about the information. hopefully we'll be able to get into some of the topics with the distinguished group of panelists that we have to talk about this to. i'm going to introduce the panelists who are going to proceed in giving their remarks and then giving some of their experience from public health advocacy and academic points of view. and meanwhile, if you all have the cards still, please do start writing down questions on those cards so that we can address them in an efficient way after they're finished with their comments. the panelists today, first is peter lur i.e., president of the center for science and public interesting. previously, commissioner for public health strategy analysis and analysis at the fda, and in your packets there are much longer bios for these very distinguished people. second, dr. eric turner who is an associate professor of industry at oregon health and science university. his research aimed at increasing medical transparency to make the evidence base more complete, truthful, reliable. and then third, dr. michael carome, director of public citizens health research group, one of the leading public health advocacy groups in the country, and editor of public citizens worst pills, best pills news, a monthly newsletter that provides reviews of safety and prescription over the counter drugs. unfortunately, the fourth panelist, shannon brownly couldn't be here because of extended jury duty she had. she's serving the public in a different regard today, but is unfortunately not able to be with us. i would like to start with dr. lurie, and go through the three panelists. save your questions, we will collect them at the end and hopefully have a good discussion at that point. thank you very much. >> good morning, everybody. thanks for the opportunity to speak. aaron said that one of the -- what this panel is about is the difference between information that the fda might hold contemporaneous with information in the public realm. my background had been an activist group, public citizen. imagine me really rubbing my hands together with glee when employed at the fda and got a chance to see some of this contemporaneous information. and the information that i am going to present today is really about finding a way to make some of that contemporaneous information public by aggregating data in such a way that you wouldn't be able to identify any particular company or product, but would nonetheless be able to convey certain information about this con te contemporaneous information as it were. i am with the center for scientific of public interest. this research was done while at the fda, speak not as an fda employee since i am not one. some of my colleagues from fda helped work with me on the papers are present today. so really, the presentation consists of two parts, presentation about two separate papers. one of them you've heard a little about through the course of the day, a publication on complete response letters which for the benefit of people watching, viewers from afar, are complete response letters are letters that disclose reasons why the fda might have decided not to approve a drug. the second paper relates to publication of post marketing requirements or use the term pmrs which are requirements that the fda can now impose upon a manufacturer whose drug has been approved, in order that they conduct follow-up studies to better inform the safety or effectiveness or other profile of the drug. those are the two studies. they have quite a bit in common as it turns out. and i'll turn to them in turn. here's the first one. it relates to the complete response letters or crls. what we did in our study is look at all of the crls for new drug applications put out by the center for drug evaluation and research, only that part of fda, for the period of august, 2008 to 2013. we did something i presume similar to what dr. gottlieb is having someone at fda do, we took 61 of these crls and chopped them up. and chopped them into seven different domains like safety, effectiveness, and then each and every sentence or group of sentences within the crl was assigned to one of the domains. then go on to describe the frequency of those. we actually get further into what we call sub domains, but i left that out of this talk. that's available in the bmj paper for which the reference is at the bottom. then we looked at the percentage of crls that had a magic threshold. what's in the crl, nobody really knew. the second part was given that we in fda could see the crl, what was it that the public was able to see. what could they see in a press release, what percentage of statements in the press release, if there was a press release, how much information was disclosed. we also looked at securities and exchange commission sec filings to see if there was any supplementary information there. so again, the first question was why does the fda turn down a drug. and that's a very fundamental question to which there was no public answer. in the absence of information, there were at least some people who believed the drugs were sometimes not approved for lack of a crossed t or lack of a dotted i. and our experience was that was not the case. we thought that they were for substantive reasons and wanted to set out finding out whether that was true. there were 61 crls as i mentioned, and this is the number of the crls that made mention of a reason for disapproval in these seven domains to which i referred. you can see first of all, they were an array of different reasons they were turned down. you can tell by the numbers here that many had much more than one reason, right, since there were 61 and many of them have in the 40s. and you can see efficacy, safety, and clinical manufacturing and controls were the primary reasons. but most importantly about two-thirds of the crls had an efficacy reason for turning down a product and it was similar numbers to safety. at the bottom, 87% had either safety or efficacy reason included in the crl and 48% had both. we drew the conclusion from this that in fact the fda does not turn down products for trivial reasons and hopefully that has helped to counter the misinformation on that point in a general way. of course, that doesn't answer the question of what might be happening for any particular crl because that is not yet being released. okay. so in here we're looking again at crls as the unit of analysis. and the question is what did the companies do when they received their crl? the answer was that in about 18% of cases, 11 of 61, there's no press release and no way for anybody to know the product had been rejected. in an additional 13 the company put out a press release that said that a complete response letter had been released but provided no details whatsoever of what that reason was. then the rest of this are the fraction of the statements that were matched. we have 61 crls, there may be ten pages long a piece. we broke them into statements. 61 crls had an afternoverage of pages with ten separate statements. the question is what fraction of statements were there. and you can see that in those that had press release that gave reasons, you can see typically only one to 25% of statements matched, only occasionally were higher fractions of statements actually disclosed. now we are looking, changing the frame instead of looking at the crls, now the unit of analysis is the statements themselves of which i mentioned, there were 687 now we are looking at the same seven domains i mentioned earlier. in the red are statements that never appeared in a press release, and in yellow are the statements that did. you can see what the matching rates are. the highest matching rates for safety and effectiveness. overall matching rate is 14%. higher in safety and effect i haveness and much lower in others. companies are leaving out 86% of the information. they're more likely to disclose safety and effectiveness information, even their critical information is often not included. for example, there were seven crls that mention statistically significant or at least increase in mortality rates between treated and comparison groups. only one of seven trials had a matching press release that disclosed that fact. if there was a requirement for an additional clinical trial which is much the key turning point for the company as to whether or not they will be able to rapidly get a product approv approved or not, that was only disclosed 59% of the time. we looked at the securities and exchanges commission filings to see if there was additional information there, we did find a small amount of additional information which brought the overall amount of public information just a little 14 to 15%. most of the information is not made available. again, this is at the statement level in either source. so the second study which i will hurry through, about post marketing requirements. pmrs. we looked at all of those fulfilled. this doesn't get to the question of whether or not the studies are done, this is among those that are done. we asked whether or not at least three years, pmr had been published in scientific literature, clinicaltrials.gov or either source. these are results. take home number is probably this one here. the pmrs were published in either scientific literature in 64% of cases. there were higher rates in literature and clinicaltrials.gov. that obscures the fact that there's really interaction based on what kind of trial it was. these are the clinical trials as opposed to other trials that were fulfilled. in the first place, more interventional clinical trials, and you can see that greater fraction of interventional clinical trials are published than the case for the other trials. so the first bullet point is saying that difference i mentioned was statistically significant. many of you ask the question whether or not there was difference in publication rate with those with favorable or unfavorable outcome and we did not find statistical difference. so just in conclusion, again, two different kinds of studies that turn out to have a lot in common. the first is that the publication of both letters and the pmr results are recommended in the transparency report back as far as 2010, and our data showed in both cases reporting has been less than optimal. broadly they share the same potential solutions, despite being different kinds of documents. on one hand, companies could in theory publish voluntarily, but obviously are not doing so in a consistent way. or on the other hand, the fda could publish the relevant document, although in each case there would be requirement for some regulatory changes. the advantages of publication would be we heard much of this this morning, more informed discussion by everyone of the scientific reasons for the agency's actions which could in turn facilitate drug discovery. there could be in general more transparency on the agency's regulatory processes and decision making, and misperceptions about reasons drugs weren't approved or what results of these pmr studies, any misconceptions could be clarified. advantages are as we heard from the last discussion, there's industry concern that competitors might be advantaged. there's potential for release of confidential commercial trade secret information, i think most of us in this room would agree the fda is at least as good as it ought to be in protecting that information. and finally, the ever present issue of agency work load. there i think there is a legitimate issue, having once been a person with work load at fda. i think that what we are especially interested in and dr. gottlieb got at this is information that relates to safety and effectiveness much less than about manufacturing. i think that information that's most related to clinical outcomes might be a way to cabin the information such that almost everything important to the public would be released but that the work load on the agency could be kept to a reasonable level. thank you. >> can you hear me? maybe i can turn off the microphone? no? leave it on. i do not have power point. i am so used to speaking with power point, i am a little intimidated here. it has been a long time. i'll use old fashioned note cards like back in high school. i'm going to speak, my name is eric turner. sorry, i didn't introduce myself. i am a psychiatrist by training and former fda reviewer. may i speak from perspective as a clinician and also as former fda reviewer, as consumer of the information as well as participant in production of the information. so first of all, and one theme i want to touch on is not just qualitative types of thing disclosed but the manner in which it is disclosed and get at the idea of transparency versus translus ensee. you can think about more transparency, and that's a continuum, not either or thing. one thing i want to talk about is drug approval packages where i have done most of my research work with. i was told back when i was an fda reviewer when the fruits of my labor were complete and the drug were approved, then my review would be up on the internet for the world to see and that was kind of sobering. i knew that i was charged with an important job and that if i messed up, it would be there for the world to see. so one of my first reviews i worked on was, and i think i can say it now, as i learned earlier today, something that's public is no longer commercial information, this is now public, i was given an nda new drug application to work on on the drug ra box teen, which was approved in europe as an antidepressant the year before i got there in '97 i started working on it in '98. that was a long time ago. i was immediately struck by the quantity of negative studies, i had never seen this before. i had done a fellowship at national institute of health and thought i was at mecca there, thought i should have access to everything one would need to know about drugs, but i found out i had really been in the dark and my eyes were opened up on my fda experience there in seeing again all of the negative studies, i mentioned it to my boss, i said is this a fluke? he said no, this happens all the time. and that kind of -- it was in a way i could see the humor in it, but at the same time it was disturbing that in my other hat as in private practice, i should be a clinician, and would have been if not for my fda position, would have been totally unaware this was going on, blissfully prescribing these drugs believing they were always placebo, and i was learning this was not at all the case. later after leaving the fda did some formal studies of the phenomenon, first study and one perhaps best known looking at antidepressants published in new england journal of medicine in 2008, took 12 aentidepressants and through 2004 wiand looked a the studies and tracked them in published literature and asked two questions. first of all, was the study in question published and number two, if it was published, was it in a way consistent with the fda. turned out there was quite a discrepancy between the two views. what as a clinician you would see, you would get the impression, loobi impression, looking at journal articles, greater than 90% of the articles conveyed positive conclusion, ones that were published. on the other hand, if you look at fda reviews, you saw it was really about 50/50. and by positive, i mean reached statist statistical significance with the drug being placebo. striking difference between 100% and 50%. and then from meta-an lit i can review, there was similar discrepancy, not as striking, but shift in the effect size by a third. the effect size went up according to fda, calculate .3 standard deviations using journal data, calculate .4. that was one-third increase in effect size. and then later did other studies like this using drugs for anxiety and drugs for schizophrenia and got similar view of things. then other people have done these studies making use of fda data, and i guess what i would like to segue to now is that it has been a bit disappointing that not more has been done with this approach, that the fda data is there, it is on the fda website, but it is still strikingly underutilized, and i think underappreciated, and so it has been there since '97 that's when the reviews started to go online. yet very few people seem to know about it. when i say few people, not only obviously lay people but even researchers. i think most people in medical research are unaware of this. there was a study done in denmark about cochran reviews. and the cochran collaboration as you know is highly revered as the people that do the most highly renowned systemic reviews and meta analysis. one of the questions they asked, how many cochran reviewers are using regulatory data. he found 97% have not used regulatory data. this is bad. so why is that? i think part of this is lack of appreciation, perhaps some people simply didn't know about it, our training is that we are taught that published literature is the be all, end all, the most authoritative information on medical interventions. so we might be skeptical that there might be something even better elsewhere. and also i think the fact that there are obstacles, if they do know about it, they find -- for lack of a better word, user unfriendliness of reviews. to talk about drug approval packages, for instance, first of all you have to -- you need a number of mouse clicks to find your way to drugs at fda and then to the drug you're looking for, then sorting out which one has the efficacy and safety reviews to separate that out from the generics which have no efficacy and safety reviews, that's one issue. secondly, you will typically for a drug, if looking for the first indication it was approved for the drug in question, you're in luck. you probably will find that. if on the other hand, you're looking for indication number two or three, you're not in luck. you usually will not find that. this has been attributed largely to work load issue. we don't have the personnel to put the reviews up online for the second and third indications, but we might do that if we get a number of foia requests for that. navigating to reviews, finding what you're looking for. once you get there, you might find a medical review, couple of hundred pages, might be broken into four different pdf files. and then i guess because of size of the document. then when you get to it, you'll often find for newer drugs, you're in luck, they're searchable, but for older drugs, five years or older, and as we heard previously from dr. dickerson, most of what we're using out there are older drugs. they're not going away, i mentioned prozac included, it is alive and well and used quite a bit. you get to the review and find out you've got one big, long image document. and the reason for that is that it has been -- the review is printed out, at one point it was redacted in the old days, they would redact with white out even, and then they would scan it in, it would be scanned in as an image. the only way to look through it is with your eyeballs, the old fashioned way. you can print it out, sort pages. if you're high tech, you can use a program and use optical character recognition, but that's limited too. you can't go back to the original user friendliness of the original pdf created by the reviewer because inside tables, you've got numbers and ocr cannot, it usually freaks out when it hits tables and won't read them. so it is very limited. and then table of contents, it would be nice if there were table of contents you could navigate around. there's table of contents in the views, but it is not hyper linked. like to mention one other type of fda document besides drug approval packages, and that's advisory committee documents as well. they can also be quite valuable for correcting misinformation. example would be a drug, there was an advisory committee in the early 2000s that led to a jama paper by steve nissen and colleagues and it, not saying there's cause and effect here, but the fda decided not to approve. rumor has it that they were leaning towards approval, but it was not 100%. correcting misinformation had impact using the advisory committee documents. another type of document i have not used at all, pediatric documents, pediatric approvals and found a nice example of a drug that was actually not approved, which is unusual, like paxil, for pediatric depression, subject of a lot of controversy, the fda review is online, but i never would have known it until recently. i won't say how i was pointed to it, but i was given the exact url and able to find the review. later, a few days later, i went where was that, and i went to google and tried to find it using key words and i was unable to find it. so it is there somewhere, buried in the website, but i don't think -- good luck finding it. let me move to my sort of wish list, and the flip side of many comments i made already, it would be nice if we could have access to move further along on this transparency from translucency, getting more indications on the website. secondly, the ability to navigate through the documents i think we can take older documents, these documents were created in microsoft word and they were converted to pdf, and at that point they were searchable, including the text and tables and numbers. and those could then be, sort of backtracking, might sound like a lot of busy work, but again, older drugs are the ones in greater use than newer drugs, they could be redacted electronically the way the newer drugs are. it would be nice to have more organizational approach, rather than look in different places, one for advisory committee documents and i didn't complain about advisory committee documents, you basically have to know what the committee was and when it was, and then find your way to it. rather than have that organization or indexing method, why not put it with the drug, indexed by drug. you could also have index by indication. if looking for depression, you could find drugs, maybe find devices that are approved for depression. and then once you find the drug or intervention you're looking for, you can find the drug approval package, of course, and you can also find perhaps the new complete response letters, if that indeed is going to come to pass, find pediatric approval documents, and advisory committee documents. just some -- just about out of time. on the topic of advisory committee, i want to make a plea, just a suggestion, that we are talking, this is a symposium on transparency at the fda. why not an advisory committee for transparency at the fda. rather than once every ten years, we could have an on-going discussion between the various stakeholders. >> good morning. i'm from public citizen and i'd like to acknowledge that one of my colleagues, one of our other health researchers helped co-author the comments i'm going to make and as the final speaker of the day, you're going to find i'm going to be reiterating many of the points you've already heard today. so much of the information upon which the fda relies when making pivotal regulatory decisions with regard to regulated products is kept secret and one prominent example of the fda's lack of transparency concerns new drug applications or ndas, including supplemental ndas that have been rejected or withdrawn by the company. the long standing policy is that the fda does not release its on analysises of data submitted for ndas or supplemental ndas. nor does the agency notify the public that such rejections or withdrawals have occurred. by contrast, the fda releases to the public its detailed analyses and findings related to data supporting the approval of a drug's first nda and a request by three individuals of supplemental ndas for new uses of our already marketed drugs. so i'd like to bracket my general comments with two real-life case examples. in the first case, it vols, valdecoxib, which is an nsaid known as a selective cyclooxygenaze that was marketed under the brand name bextra before it was returned in 2005. in january, 2001, jd searle, then a subsidiary of pharmacia, submitted the initial nda to the fda for approval to market this drug for four indications, relief of the signs and symptoms of osteoarthritis and of rheumatoid arthritis, treatment of primary dysmenorrhea, or menstrual cycle pain, and prevention and treatment of acute pain, including opioid sparing and prevention of operative pain. in november, 2001, the fda approved the drug for the first three indications but not the fourth. in december 2001 public citizen requested from fda a copy of the approval package for bextra. and a copy was posted on the web site but a few days later before a public citizen staff and at the request of sterile fda removed from its web site the package. at a later date, the fda reposted the approval package but redacted the information regarding the acute pain indication and the safety and efficacy related to that claiming the information involved trade secrets. in may, 2002, a medical journal article and a related press release were published touting bextra for treating acute pain, in this case associated with dental surgery. the article was co-sponsored by pfizer and pharmacia and three of the five authors were employees of pharmacia. public citizen subsequently submitted a foia request for the unredacted approval package and in response to a lawsuit filed by our organization in 2004, the fda released much of the information previously redacted from the approval package which showed that the fda had denied approval of bextra for treating acute pain because of safety concerns. in particular, safety data from a trial that tested valdecoxib as an add judgmejunct revealed exercise adverse events including death in subjects receiving valdecoxib compared to placebo. so in terms of general comments, the 2017 blueprint builds on the work of the fda's 2010 transparency task force which recommended that the agency, among other things, release complete response letters to shed light on why drug marketing applications were refused. the blueprint report noted several potential benefits from releasing such information, including that the clinical community can benefit from the insight, expertise, and analyses of fda reviewers and researchers can learn from the failures of previous medical products and subsequent research programs. keeping the public in the dark about unapproved drug marketing applications prevents patients, researchers and health care providers from gaining insight into why a drug's application was not improved. this lack of transparency is particularly troubling in cases where the fda has found a currently marketed drug to be ineffective or unsafe for a newly proposed indication. disclosure of the fda's finding in such cases would promote public health by encouraging health care providers to avoid prescribing drugs for unapproved or off-label uses that the agency deemed to be potentially dangerous or ineffective. this is especially important given the endemic practice where pharmaceutical industry is engaged in illegal marketing drugs for off label uses. disclosure of complete response letters is all the more important given the current permissive framework allowing in some ways the promotion of marketed drugs for unapproved uses, existing fda guidance already permits drug and medical device manufacturers to market their products to physicians for unapproved uses through dissemination of scientific or medical journal articles and reference publications and congress is considering legislation that would further expand the scope of such off label promotion. such erosion of restrictions on awful label marketing make it vital that health care professionals be informed of off label uses that were deemed by the fda to be too dangerous or ineffective for patients. failing to provide information on unapproved ndas also gives companies free rein to craft their own self-serving narratives as to why their applications were turned down, as the doctor discussed in detail. finally a new policy related to rejected applications for already approved drugs would be cop sis tent with the belmont report's basic ethical principle of beneficence. this establishes an ethical obligation to maximize potential benefits to human subjects participating in research. in the event an application is rejected by the fda because the fda determined the drug's harms outweigh its benefits for particular use, both the drug company and the agency have an ethical obligation to make this determination public in order to avoid future clinical trials of the drug or in some cases a similar drug in the same class that would unnecessarily expose human subjects to harm. now a policy whereby the agency, the fda, releases complete response letters in the underlying analyses leading to the agency's decision not to approve an application is certainly feasible. in 2004 the european union required that the ema make publicly accessible information about all refusals of human drug marketing applications and the reason for them. the eu law stipulated that after an ema decision rejecting a sponsor's drug application or where the sponsor has withdrawn the application before the ema completed its assessment the agency must publish, among other things, a report containing the ema's analyses and conclusions relating to the clinical trial data in the application. one can now research the ema's web site for all reports with specific searches available for drugs that have been refused marketing authorization or have been suspended or withdrawn from the market. health canada followed suit in 2015 when it announced it would make available all regulatory decision summaries which contained the rationale for health candidates' decisions on drug marketing applications. this decision notably includes for public release final negative decisions and cancellations for all marketing applications of new drugs and new indications for existing drugs and this data can be searched for on the health canada web site. so let me conclude with a second case example. in september, 2008, johnson & johnson submitted an application to the ema for an additional indication for its anti-psychotic drug, the treatment of acute manic episodes associated with bipolar one disorder. they withdrew their application 85 days into the ema's review of the application based on feedback from the early evaluation indicating the data provided were not sufficient to support approval the ema posted the company's letter requested the withdrawal and publishing a q&a fact sheet on the application describing the study's use to support the application and linking to the company's letter explaining why it was withdrawn. we could not find a record indicating whether the company has submitted a similar snda to the f.d.a. the fda did refer in pass an unrelated medical review document on another nda published six months after the announcement to pre-nda meets with the company to discuss the bipolar disorder application. the company did not disclose to the readers that its approval for invega was insufficient to support approval. off-label use of second generation antibiotics such as invega for psychiatric conditions for which they are not approved is widespread. physicians may have been prescribing invega for bipolar disorder with no knowledge of the ema's regulatory history. it should be noted that johnson & johnson, invega's maker, was forced to reach a $2.2 billion settlement with the federal government in 2013 over, in part, allegations of off label promotion of its anti-psychotic drug invega andr risperdal. in conclusion, the fda must join the ema and health canada in allowing the public to know when a drug is deemed unsafe or ineffective. even not withstanding the public health benefits that such disclosure would reap, the public has a right to know when, how, and why the nation's largest public health agency reaches major decisions on the product it regulates. thank you. >> all right, maybe i'll ask our panelists to sit down and you can start q&a. are there -- there are cards being passed out so please provide your questions on the cards and we'll get to them. so thank you all very much for your comments. so maybe first we'll start with peter. so you mentioned in your comments you touched on two different kinds of correcting this information. there is correcting -- that the fda can engage in. the first is correcting information about a specific product and the second is more about correcting information about what the fda does in terms of understanding that it provides these complete response letters and there might be different perceptions or understandings of the review that the fda takes and the information that it provides back to manufacturers. so those seem like -- it seems like you're -- the papers you talked about are -- were about correcting the second kind of misinformation and maybe there's a public health functionality to that kind of correcting this misinformation as well, whereas i think a lot of things people presume when they hear about correcting misinformation, that i think about a specific product. so do you think you can talk a little bit more about the different public health roles that each of those functions of correcting misinformation? >> let me make sure -- can you say the two kinds again? >> correcting misinformation about a specific product where there is a particular piece of information out there and the fda has other information, then there's correcting misinformation out there about what the fda's role is in general. >> i see. yes. well, that's right. i think there are misperceptions about what the agency does and i agree with the doctor in his piece in the journal from my own personal experience that certainly one of the most frustrating experiences as an fda person is to see information that you know to be inaccurate. when i was working on expanded access to approved drugs there was information put out by -- alleging a certain set of event which is we knew to be much, much more complicated than that. and the place that the fda especially gets handcuffed, which has not so much been talked about at this forum, is that the policy has been not to even acknowledge the existence of the investigational new drug application or of the new drug application itsemeaning -- for e who don't know the terminology, the initial application to provide the drug to humans for the first time or the attempt to seek approval for marketing. that's what handcuffs the agency. because they get in this place where not only can they not actually correct the misinformation, they can't even acknowledge that the drug about which there is misinformation is a drug that's before the agency. i mean, that's how far back with holding of information goes. and that is what really handcuff it is agency and puts the agency in this frustrating position of seeing its actions and the overall lay of the land in the scientific literature, they see it being distorted and there's very little that can be done so i think both are important, obviously. i think just one piece of clarification which i think is of interest with regard to our complete response letter paper is this. we went out when we began the paper i think believing that we might find places where the companies had actively misstated information. what we found were not errors of commission but errors of omission. very high rates of omission. 85% rates overall. but overall the information that is put out a very heavily vetted by the lawyers. the s.e.c. documents are rife with cut and pastes within the document and between the press release and the s.e.c. documents themselves so they're quite careful in what they put out. but errors of emissions themselves can absolutely create a misleading profile of the drug, its regulatory status and what the agency has been doing and that's what i think needs to be remedied? >> that's a good transition to something you said, eric, about errors of omission in regard to the reports about the antidepressants medications that you studied. it seems an error of omission and error of commission can have similarly different problematic and public health outcomes. right. they can both affect whether you're looking at the overall proportion of positive versus negative trials or if you're looking at it from a metaatlantic point of view. the omission, that can be called study publication bias where the entire study hadn't been published then there's outcome reporting bias. it goes by many names, spin, et cetera, et cetera but they can have an impact as well. >> one of the points you brought up in your comments is this idea of user unfriendliness of the information. so i was wondering if we could brainstorm on optimal ways the fda could take on this role of improving information out there. do we want -- would it be through tweets or would it be through medical journal articles or just putting the information out there on the web site. can we think on better ways to improve the process or make the process more user friendly. >> the information has been online since 1997 but it's striking how few people know about it so anyone trying to sell a product would never just arrange to have the marketplaced on the shelves and never advertised. there needs to be a push. twitter could be one way of doing it, there could be deliberate attempt to get the information out there to spread it, this concept of being hog tied or however you put it, it reminds me of another story where the fda tends not to speak at all. it tends to be the silent -- the black box as was alluded to earlier. there was a case of a drug that had been back in the '90s that was -- europeans -- people were complaining this drug was -- that the fda was holding up the show on this new drug but the truth was the company never submitted it to the fda so there was nothing for the fda to act on but they couldn't say, hey, don't yell at us, the sponsor hasn't applied, hasn't send in an ind or nda. they couldn't say snoig tanythi the public is left with the impression that the fda is sitting on it and not doing anything. so the lack of communication being the common denominator there. >> i can make two points to that. i don't think i have a techie answer to this but i have an optimistic response all the same which is i firmly believe that when these documents spoken collectively are increasingly made public they will improve. i believe that. i think people will find ways knowing that their documents will be public is a very powerful pressure to do a better job of it. and i think inevitably there will be little hiccups along the way as the documents are made public for the first time but pretty soon people will see the mistakes and they'll redesign the documents in ways that make sense. within complete response letters, for example it would help if all of these statements that we analyzed will put -- very clear subheads, perhaps the same seven that we use so people would know if something needed to be redacted for some reason, you would know from where it came. and you wouldn't think the fda is taking out critical advocacy information because it would be under the manufacturing heading. and this is the general history of transparency writ large in this country. when more transparency comes to bear the quality of the information over time itself improves. the other point i want to enlarge on something mike said which is critical. we are at a moment right now in fda regulatory history in which the companies with respect to their free speech rights are being unleashed and there's a series of court case, which you're no doubt familiar with than i am that are slowly but surely unleashing the companies to speak more freely. the fda can't sit silent in that circumstance because it's a one-sided unleashing and the problem that all of us in this room are so concerned about will only get worse if the companies speak more and the fda speaks no more than it currently does. >> the drugs at fda, we use it a lot in our work and once you're familiar with it it's pretty usable. it takes a few clicks to get to the particular review but once you've done it a couple times you can pretty within a minute get -- if it's posted get to those documents you can make it word searchable in a pdf program so there are -- i think it's very useful, it's got a wealth of information that i think an -- and fda should have a web page to give instructions on how to use it, how the access it and to the extent they can make documents searchable when they're posted. >> so i wanted to come back to this idea of the resources that the fda has and the ability that the fda has to try to identify every piece of information out there and respond in a reasonable period of time. i do see the issue with trying to put that sort of responsibility on the fda. on the other hand, maybe there are certain hot spots or particular points in time such as the complete response letter or the submission of an snda or the case you pointed out, the case where, you know, a decision is made by the ema about a product on the market in the u.s., that seemed like particular important points in time in the life of a drug were getting the optimal information out there is useful. so as beginning to try to develop the standards, maybe particularly putting out information around these events in the life cycle of a drug is one way of going forward. so mike, you mentioned in invega sto story, it would seem like one thing that would be useful is for the fda to report what was going on with this drug in europe and the ema putting out this report about the supplemental use of invega, it seems like that kind of -- that should be relatively straightforward to do so maybe you want to -- i don't know if you can comment on the extent to which the fda can weigh the -- you know, can try to balance amount of time and effort to police everything out there, just trying to be much more strategic about it. >> so the examples i gave are imper fe imperfect. so for example the fda gets an application for a drug for that same indication and in searching for evidence to the enhanced example we're not sure if that's the case. so in cases where fda hasn't received such an application, i don't know what role they have in terms of monitoring the drug for other indications that are -- where approval is being sought by -- in other countries, by the regulations of other countries, i suspect they would push back against that idea. where the fda did get a parallel submission for that indication and reached a decision and that decision was to reject it, obviously in that case we think they should be fully transparent about that whether they should be transparent about -- make known information again by other regulators and countries before they make the decision, they'd probably push back hard against that as well. >> so i think -- if we were to have in place all of the documents outlining the blueprint, that goes a pretty long way to addressing your problem. if the existence of the i.d. was public, that would be -- that would answer a good fraction of the question. when we hear about some company that's claiming the fact that they haven't even filed an nda, to know with certainty that it had or had not would be to change the debate over the project. but with regard to misstatements of information, there's no way the fda will be policing the entire yuan veshs of information, that's impractical but the instance where that comes up are fairly identifiable within the agency. there are a small number of cases, a modest number of cases in which you know it's happening and i can say that senior officials at that point have a debate over whether or not to set about correcting it and as the article in bmj shows, there have been a limited number of cases in which the fda has actually gone out most recently in the case by dr. caleb, and pointed out to places where the information was misleading. i think they should be doing more of it. but i'm here to say the number of important case which is will mostly be about new molecular entities, occasionally about supplemental implications is limited in number and the agency know which is ones they are. >> so we have a question from our online listeners. in canada, psychiatrists don't realize that risperidone is an active metabolite of the parent drug so they are chemically linked so what are the implications -- the question is what are the implications for transparency if prescribers don't make a connection between one brand name drug and another and is there a responsibility for the fda to bridge that gap for prescribers? >> i'll comment on that being a psychiatrist. i believe if i recall the labelling it does not make it -- the labelling does not make it clear that haliperidone is not the same. i don't know why that is allowed because it seems to be an error of omission if full. i know there's a nomenclature where they describe what drugs can be named. i know less about the generic names but that's -- that's a question i don't know the answer to. i'd like to because i think it is potentially -- imagine if you took a survey of psychiatrists and you somehow were to be able to disguise the question to ascertain whether they actually know that that's the case, my guess is that a minority would be aware of that. sorry i don't have the comment besides that. >> okay, another question for the audience. do you worry more disclosure might encourage reviewers to write less down? bringing up the point that maybe foia has had unintended consequences of having telephone conversations rather than written correspondence among people within government? >> well, i'm sure there would be a certain measure of that but in the end you're a medical reviewer, you're making the recommendation for or against approval of a product you're probably going to want to put your best case forward. you don't want to put forward a skimpily argued document, either, on such a critical question as approval or disapproval. i'm sure there will be some behind-the-scenes discussions. prance i've even availed myself of the telephone once or twice for that very purpose. but i think in the end people's pride in their own work and need to make a convincing case now in the court of public opinion will raise the boat rather than lower i it. >> another question from the audience? we've talked about documents as if we want something like paper documents out there. but don't we really want information in an organized and user friendly way? i'm going to talk about the difference between the fda trying to put information out there to correct misinformation just about releasing more information as opposed to trying to organize and try to, you know, construct a -- sort of a science-based response to whatever their concern is. >> one of the proposals is exactly on that point. the agency does do a certain amount of its own analysis. there's an analyses that offers placebo groups on trials but that information could be aggregated and provide very useful information about natural history of diseases. they have a database of ekg information which i believe is housed at duke at this point. that information is information generated by the agency. and there is a request for information from two three more well, more years ago than that looking for input on whether or not that information should be made public so let me tell people a secret as long as we're here and i'm not divulging anything. while the agency is requesting information about whether or not this information should be made public if anybody here were to foil a freedom of information act request, the agency would provide it. it's available under the freedom of information act. so there's something that's more discloseable than you know. hopefully the agency will discrease the information but file your freedom of information act request for existing information because, i agree, you have a lot of smart people that have aggregate it had data in useful ways. there's no reason the public shouldn't be able to have it as long as personal information is removed, as long as product and company information is removed and i believe it can. that will be use fful. so if the agency has done its own analysis that uses elements of submitted indications, submitted drugs approved or otherwise, if they took the placebo groups, combined them and reached some wisdom about the natural history of the disease, that analysis if the identifiers is removed, is available under foia. >> so final question for the audience. does anybody here have thoughts about other products and transparency related to other products and medical devices, other fda regulated products and how that -- whether or not this -- there are similar issues in that kind of a context? >> i think the devices could be brought under the same -- there could be an organization by -- by the target indication, you could have not only drugs but devices as well and perhaps there may be interventions in the dietary supplement category. that would be one thing that has to be done. the other method about types of interventions being reviewed is quite different. that could be an issue. >> now we're mostly in food and i think that's a whole area not perhaps -- it's of interest to the collected group here but -- other than consuming it. but there is, for example, there's probably a lot that can be done in that area. for example there are outbreaks of food related illness in which supermarkets have contained the con dam nated product are not named by fda. that's one transparency. there's another process whereby companies can self-certify that they believe an attitude they're putting into a product is generally recognized as safe. they are not required to provide information on what information they did so. all of that would be helpful so as often is the case at fda, the drug area is what leads us into new conceptual areas like transparency but i think your question is an excellent one and we should be thinking as well about whether there are tobacco applications and the like that ought to be made public veterinary medicine applications. those haven't been the subject of debate but ought to be. >> thank you very much. it's been a useful conversation and hopefully this will help genera generate ideas adds we try to think about the different standards and recommendations for the fda. it seems like the fda is very engaged so thank you for your comments and contributions. >> thank you. [ applause ] >> thank you so much. that was terrific and that brings to a close today's session i just want to say on behalf of the johns hopkins bloomberg school of public health and dean mckenzie i want to thank all of the panelists, all the people who came, the fda, c-span about everyone who's watch watched or will yet watch when this is played many times. i'll tell you the web site address, it's jhsp jhsph.edu/fda/transparency. so people can watch any parts they want to play again. you can hear the different parts where people said "i'm going to tell you a secret." that happened a few times. you can relive those exciting moments but the other all message that came through is that transparency is an issue that's been around fda for a while and has tremendous potential for improving information available to patients, doctors, researchers companies and it may be an issue whose time as has come and it will lead to more and we can see the value of transparency so thank you all for coming and participating. [ applause ] [ indistinct conversation ] [ indistinct conversation ] [ indistinct conversation ] [ indistinct conversation ] >> congress returns today facing a government shutdown deadline of midnight friday. the house gavels in at noon for speeches, 2:00 p.m. eastern for legislative work to extend government funding and later in the week take up an abortion bill. the senate back at 4:30 working on a house-passed fisa reauthorization. later in the week, the senate will have to work on government funding as well. you can watch the house live on c-span and the senate live on c-span 2. the trump administration recently announced it will allow states to impose medicaid work requirements. here's more. >> mr. wexell, good morning to you. >> good morning, pedro. >> can you talk about what these rules are? >> so the intent is to target the newly eligible medicaid people. so it won't be the elderly or the disabled, it's going to be targeted at "the able-bodied people" who are eligible for medicaid right now. >> and so then specifically what qualifies as able-bodied? >> it's -- well, that's up to every state to determine so that's part of the issue here is that the federal government said states can determine who is able bodied who is not able-bodied. but generally speaking if you're not elderly, if you're not disabled, if you're not pregnant and if you're not under the age of 16, you qualify as able-bodie able-bodied. so if the state decides to implement this, you'll have to do work, community engagement activity for however long the state decides that you want to do it in order to be eligible. so that's generally who they're targeting. >> is this something new under the trump administration or could other administrations have applied this same type of standard? >> no, this is brand new this is something very unique to the trump administration. the idea of working for benefits and requiring that you need to be self-sufficient to get the benefits provided by the government is not a unique idea. but the traiump administration the only one that granted states the ability to require medicaid beneficiaries to work. previously under medicaid if you were eligible for medicaid you are eligible for medicaid. there's some income requirements and basically that's it. so no other state has done this before, no administration has done this before, this is something that is brand new and just in the trump administration. so if the burden goes to the states, how have states responded? >> so far there are ten states right now that have waivers pending. nine states that have waivers pending before the federal government. kentucky is the first state that just got theirs approved last week so they're going to be the test case here. they got this waiver approved that can let them -- require these able-bodied medicaid beneficiaries to have 80 hours of community engagement. so they're the first ones, they're the guinea pigs but they seem confident that other states will follow their lead. it's mainly going to be republican states, too. >> i was going to ask you as far as the states that have responded how many would you classify as red states or blue states? >> i would say mostly all of them are red states so far. there are nine states that have required and they all but one have a republican governor. >> for the critics of this administration, what have they said and are there legal challenges to this? >> nobody has sued yet the that's pending. there are certain advocacy groups promising lawsuits. they have said they are ready to sue as soon as they get their paperwork sergeant to there will be legal challenges advocates say this is not in the nature of the medicaid program. the idea of medicaid is to make sure that everybody gets coverage and that's the requirement of medicaid and that this policy goes against it and makes it harder for people to get medicaid coverage. >> nathaniel weixel, is the administration or at least republicans looking at this as a test case to see if they can apply it to other social net programs? >> well, there already are work requirements on other sort of social net programs, there was in the old welfare system and there's still requirements for the food stamp program so thigh ear not necessarily looking to others in that respect. medicaid is one of the last ones you have to -- that's an open-ended social benefit so i'm not sure they're looking to others because there are work requirements. >> nathaniel weixel who covers health care for the hill talking about the new requirements under medicaid when it comes to work, thank you for your time, sir. >> thank you. >> on wednesday, members of congress will share their ideas on ways to reform the use of federal earmarks. you can watch the first part of this two day house rules committee live at 10:30 a.m. eastern on c-span 3 and on thursday journalist michael wolff will talk about his book "fire and fury," about the inner workings of the trump administration. lye coverage at 7:00 p.m. eastern on c-span 2. house minority leader nancy pelosi and congressman mike thompson held a town hall meeting with constituents on the tax reform law and immigration policy including an update on negotiations involving the federal budget and the dhaka program. the university of san francisco hosted this event

Related Keywords

Acton , Ontario , Canada , United States , Kentucky , Guinea , Mecca , Makkah , Saudi Arabia , Togo , Oregon , Canadian , Andr Risperdal , Sean Cody , Joe Ross , Anna Davis , Warburg Pinkus , Jonathan Leff , Janet Wood Kok , Eric Turner , Steve Nissen , Johns Hopkins , Caleb Alexander , Josh Sar ,