Play. I want to thank allen and pew for hosting this meeting, for organizing the publishing of the special supplement and of course to mr. Gottlieb for his remarks at the beginning of the day. So the focus of this final panel today is the fact that in certain high profile cases manufacturers have released incorrect or incomplete information into the market that was directly counter to the information, the contemporaneous information the fda had. They noted that selective publication of Clinical Trial results has in the past created a misleading picture of the safety and efficacy of a product with negative implications for Public Health. And as a the practice now still stands, manufactures have wide latitude to publicly characterize data that were submitted to the fda without much risk that the fda will correct the record. I think were going to try to talk a little bit about in the next hour is, you know, in these cases does the fda have a Public Health or an ethical responsibility to correct the record . And if so, how. So in the blueprint we outlined a couple principles that might help organize this kind of approach. So first, should the agency adapt a basic set of standards for when to correct misinformation . Should the agency give advance notice to manufacturers regarding any concerns they have, and if so how would that happen . And should the fda disclose the scientific information thats the basis for its concern about misinformation. Hopefully well be able to get into some of those topics with the distinguished group of panelists we have today. Im going to introduce the pa panelists who are going to give their remarks and their academic points of view. Meanwhile, if you have those cards please do start writing down your questions on those cards so we can address them in an efficient way. Our panelist today first up is peter lurry. Previously he was the associate commissioner for Public Health strategy and analysis at the fda. And in our packets there are much longer bios for these very distinguished people. Second is dr. Erin turner. His research is aimed at increasing medical transparency to make the Evidence Base more complete, truthful and reliable. And then third we have dr. Michael curome. Health citizen is one of the leading Public Health advocacy groups in the country and editor of a monthly newsletter that provides reviews of the safety and prescription of over the counter drugs. Unfortunately our fourth panelists was not able to be here because of extended jury duty she had. So she is serving the public in a different duty today but is not able to be with us. I would like to start with dr. Lurry and please save all our your questions and well collect them at the end and hopefully have a good discussion at that point. Thank you very much. Okay, good morning, everybody. And thanks for the opportunity to speak. Aaron said that with this panel is about is the difference between information that the fda might hold contemporaneous with information in the public realm. My background had been in an activist group and Public Citizen, and you can imagine me really rubbing my hands together with glee when i was employed at the fda and finally got a chance to see some of this contemporaneous information. And the information that im going to present today is really about finding a way to make some of that contemporaneous information public by aggregating data in such a way that you wouldnt be able to identify a particular company or product but would nonetheless be able to convey certain information about this contemporaneous information. So thats the basic outline. As mentioned im now with the center for science in Public Interest to this research i will present was done while at the fda. Im speaking not as an fda employee since im not one, but some of my colleagues from fda who helped work with me on these papers are present today. So really the presentation consists of two parts. A presentation about two separate papers. One of them youve actually kind of heard a little bit about through the course of the day, a publication on complete response letters. Which for the benefit of those watching, complete response letters are letters that disclose the reasons why the fda might have decided not to approve a drug. The second paper relates to publication of postmarketing requirements or ill use the term pmrs, which are requirements the fda can now impose upon a manufacturer whose drug has been improved in order they conduct follow up studies. So those are the two studies, and they have quite a bit in common as it turns out, and ill turn to them in turn. So heres the first one. It relates to complete response letters or crls. And what we did in our study was to look at all the crls for new drug applications that had been put out for center for Drug Research for the fda for the period of 2008 through 2018. What we did is i assume is similar to something that dr. Gottlieb is having someone at fda do is we chopped them up into seven different domains like safety, effectiveness, pharma, pharma and so forth. And each was assigned to these seven domains. We actually get further into what we call subdomains, but ive left that out of this talk. And then we look at the percentage of crls that had a matching press release. Question one was whats in the crl because nobody really knew. And the second part was given that we at the fda could see the crl, what was it that the public could see . What could they see in a press release . What percentage of the statements appear in the press release . Or do they release a press release and if so, how much information was actually disclosed . And we actually look at the fcc filings to see if there was any supplementary information over there. Again, the first question was why does the fda turn down a drug . And that a fundamental question to when there was no public answer. And in the absence of information there were at least some people who believed the drugs were sometimes not approved for lack of a crossed t or lack of a dotted i. And our experience was that was not it case. It was usually for substantive reasons and we went about finding if that was true. This is the number of the crls that made mention of a reason for disapproval in these seven domains. And you can see, first of all, that there were a array of Different Reasons that the products were turned down. You can tell by the numbers here that many had much more than one reason, right, since there were 61 and many of them are in the 40s. And you can also see the efficacy, safety, and this is clinical manufacturing and controls were the primary reasons. But most importantly about two thirds of the crls had an efficacy reason for turning down the product. 80 had either a safety or efficacy reason included in the crl, and 48 had both. So we drew the conclusion from this that in fact the fda does not turn down products for trivial reasons, and hopefully that has helped to counter the misinformation in that point in a general way, of course, that doesnt answer the question of what might be happening for any particular crl because that has yet been released. Were looking at crls as a unit of analysis, and the question is what did companies do when they received their crl . The answer was that in about 18 of cases, 11 of the 61 crls, there is no press release whatsoever and really no way for anybody to know that the product had been rejected. In an additional 13, the company put out a press release that said that a complete response letter had been released but provided no details whatsoever what that reason was. And then the rest of this or the fraction of the statements that were matched. So we have 61 crls. Theyre maybe about ten pages long apiece. 61 crls had an average of 10 to 11 statements. The question is what fraction of the statements were actually there . And you can see that in those that had a press release that actually gave reasons, you can see that typically only one to 25 of the statements were matched and only occasionally were higher fractions of the statements actually disclosed. Okay, so now were changing the frame to instead of looking at the crls, now the unit of analysis is the statement themselves of which i mentioned there were 687. And now were looking at those same several domains i mentioned earlier. And in the red are statements that never appeared in a press release. And in the yellow are the statements that did. And so you can see what the matching rates are. The highest matching rates are for safety and effectiveness. The overall matching rate is 14 . Higher in safety and effectiveness, like i say, and much lower in these other things. So the companies are leaving out, well, 86 of the information. Theyre more likely to disclose safety and effectiveness information. But even their critical information is often not included. For example, there were seven crls that mentioned a statistically significant or at least an increase in mortality rates between treated and comparison groups. And only one of those seven trials had a matching press release that disclosed that fact. There was if there was a requirement for an additional Clinical Trial, which is very much, you know, the key sort of turning point for the company as to whether or not they will be able to rapidly get a product approved or not, even that was only disclosed about 59 of the time. Okay, finally we looked at the Securities Exchange commission filings to see if there was any Additional Information over there. And we did find a small amount of Additional Information, which brought the overall amount of Public Information just a little bit to 14 to 15 . But most of the information is not made available. Again, this is at the statement level, in either source. So the second study, which ill hurry through, is about these post marketing requirements, the pmrs. We look at all of those that were fulfilled. So this does not get at the action of whether or not the studies are actually done. This is among those that are done. And we asked whether or not by at least three years later, 2016, the pmr had been published either in a scientific literature, in clinicaltrials. Gov or in either source. The pmrs were published either in the Science Literature or on the website, and you can see there are higher rates in Science Literature than in clinicaltrials. Gov. These are the interventional Clinical Trials as opposed to other trials that were fulfilled. Theyre in the first place more interventional Clinical Trials than other kinds of trials. And you can also see the greater fraction of interventional Clinical Trials are published than is the case for the other trials. The first bullet point here saying that bullet point i just mentioned is a statistical difference. In the interest of time ill just pause over what the predictor predictors are there. Not especially strong or strike ing what happens. Just a conclusion, again two different kind of studies that turn out to have a lot in common. Both the letters were recommended in the transparency report as back as far as 010 and that data shows in both cases the reporting has been less than optimal. Broadly, they share the same potential solutions despite being very different kinds of documents. On the one hand the companies could, at least in theory publish voluntarily. But they obviously are not doing so in a consistent way. Or other hand, the fda could publish the relevant document. Although in each case there would be a requirement for some regulatory changes. The advantages of publication would be weve heard much of this this morning, more informed discussion of everyone of the scientific reasons for the agencys action, which could in turn potentially facilitate drug discovery. There could be in general more transparency on the agencys Decision Making and misperception about the reasons why drugs were not approved and what the results of these pmr studies, any misperceptions could be clarified. And the disadvantages as weve heard from the last discussion theres industry concern that competitors might be advantaged. Theres the potential of release for confidential trade secret information, although i think most of us in this room would agree that the fda is at least as good as it ought to be in protecting that information. Ask finally theres the ever present issue of agency workload. And there, i think, there is a legitimate issue, having once been a person with a workload at fda. You know, i think that what were especially interested in, and i think dr. Gottlieb got at this, is that information that relates to safety and effectiveness much less than information about manufacturing. So i think that information that is most related to drug discovery, most related to clinical outcomes, might be a way to cabin the information such that almost all the information to the public would be released but everything related to the agency could be kept to a reasonable level. Thank you. Can you hear me . Actually, maybe i can turn off the microphone since i no, leave it on . Okay, so i do not have powerpoint. Im so used to speaking with powerpoint im a little intimidated here. Its been a long time. So im using oldfashioned note cards here like back in high school. So my name is eric turner. Sorry, i didnt introduce myself. Im a psychiatrist by training and a former fda reviewer. Im going to speak from a perspective as a clinician and also as a former fda reviewer and consumer of the information as well as serving in production of the information. One theme i want to touch on is not just the qualitative, the types of things that are disclosed but also the manner in which its disclosed and get at the idea of transparency versus tran translucency. Transparency, indeed, is a continuum. Its not an either or thing. One thing id talk about was the drug approval practices. I was told when was an fda reviewer when the works of my labor was complete and the drug were approved then my review would be up on the internet for the world to see. And that was kind of sobering. I knew i was charged with an important job, and if i messed up it would be there for the world to see. So one of my first reviews that i worked on, i think i can say it now because as i learned earlier today some things that become public is no longer confidential information. This is public. I was given an mda application to work on a new drug roboxtan. I started working on it in 98. Gosh, that was a long time ago. And i was immediately struck by the quantity of negative studies. I had never seen this before. I had done a fellowship at National Institutes of health and thought i was at mecca there, and thought i should have access to everything one would need it know about drugs. But i found out i really had been in the dark. But my eyes were opened up there in my fda experience and seeing all these negative studies. And i mentioned to my boss, is this a fluke, and he said no, this happens all the time. In a way i could see the humor in it, but at the same time it was disturbing that in my other hat, that in private practice i should be a clinician and would have been if not for my fda position would have been totally unaware this was going on and blissfully prescribing these drugs believing tat the drugs always be a placebo and here i was learning this was not always the case. So after leaving the fda i did some formal studies of this phenomenon. And my first such study and one thats perhaps best known is looking at antidepressants. It was published n new england journal of medicine in 2008 and took a cohort of 12 antidepressants and looked at all the studies that had been registered with the fda and then tracked those into published literature and asked two questions. First of all, was the study in question published and if it was published, was it published in a way that was consistent with the fda . And it turned out there was quite a discrepancy between these two views. As a clinician you would get the impression looking at Journal Articles that nearly 100 , it was greater than 90 of the articles conveyed a positive conclusion, the ones that were published. On the other hand, if you were looking at fda reviews, you saw it was about 50 50. So quite a striking difference between nearly 100 and 50 . If youre a person doing systematic reviews, it was a shift in the effect size by a third. You calculate a 0. 3 standard deviations using standard data you calculate 0. 4. Then we later did other studies like this using drugs for anxiety and drugs for schizophrenia and got a similar view of things. And then other people have done these studies, making use of fda data. And i guess what id like to segue to now is its been a bit disappointing that not more has been done with this approach, that the fda data is there. Its on the fda website, but its still strikingly underutilized and i think underappreciated. And 97 is when the reviews started going up online. Yet very few people seem to know about it. And obviously not only lay people but even researchers. I think most people in medical research are unaware of this. There was a study done in denmark serving authors of cochran reviews. And its very highly revered as people who do the most highly revered systematic reviews. He found that 97 have not used regulatory data. This is bad. So why is that . I think part of this is part of a lack of appreciation. Perhaps some people simply didnt know about it. Our training is that we are taught that the published literature is the be all and endall of the most authoritative information on medical interventions. So we might be skeptical about there might be something even better elsewhere. And also i think the fact there are obstacles. If they do know about it, then they find, you know, some well, for lack of a better word, user unfriendliness of the reviews. So to talk about drug approval practice, for instance, first of all you have to you need a number of mouse clicks to even find your way to drugs at fda and then to the drug that youre looking for and then sorting out which one has the efficacy and safety reviews to separate that out for the generics which have no efficacy and safety reviews. Thats one issue. Secondly, you will typically for a drug, if youre looking for the very first indication that was approved for the drug in question, then youre in luck. You probably will find that. If oernn the other hand youre looking for an indication number two or number three, youre not in luck. You really will not find that. This is usually tributed to a workload issue. So navigating to the reviews, finding what youre looking for, and then once you get there you might find youll find, for instance, a medical review which is a couple 100 pages often might be broken up into at least four different pdf files. And then when you i guess because of the size of the document. And then when you get to it, you will often find dau now for nr drugs youre in luck. But for older drugs, five years and older as weve heard previously most of what were using out there are these older drugs. Theyre not going away. I mentioned prozac included. Its still very much alive and well and being used quite a bit. So youll get to the review and youll find out youve got one big, long image document. And the reason for that is that its been the review is printed out. At one point it was redacted. In the old days they would redact it with white out even, and they would scan it in as an image. And so the only way to look through it was with your eyeballs. If youre high tech you can run a program and use optical character recognition. But thats limited, too. You cant go back to the userfriendliness of the original pdf that was created by the reviewer. Because inside youve got tables and it freaks out with tables and it wont read them. So its very limited. So the table of contents, it would be nice there were a table of contents. There is a table of contents in the reviews but its not hyperlinked because the whole thing is an image document as i mentioned. Id like to mention one other type of document and thats Advisory Committee documents as well. They can also be quite valuable for correcting misinformation. There was an Advisory Committee in the early 2000s that led to a paper and colleagues, and not saying theres cause and effect here, but the fda decided not to approve. Rumor has it that they were leaning towards approval, but it was not. So correcting the information there had an impact using Advisory Committee documents. Another type of document that i have not used at all yet, the pediatric documents, the pediatric approvals. And i found a nice example of a drug that was actually that was not approved, which is unusual. Peroxotine for pediatric, which has been controversial, the fda review has actually been online. And i wouldnt have known it until recently. And i wont say how i was pointed to it, but i was given the exact url and was able to find the review. Later i tried to google it using certain key words and i was unabeal u unable to find it. So its there somewhere, buried in the website. There are a flip side to many of the comments ive made already. It would be nice if we could have access to move further along on this transparency from translucency to transparency continuum, getting more of the indications, the subsequent indications up on the website. Secondly, the ability to navigate through the documents. I think we could take these older documents. These documents were created in microsoft word, and they were converted to pdf. And at that point including the texts and the tables and the numbers, and those could then be this is sort of backtracking. It might sound like a lot of busywork, but again these older drugs are the ones in greater use than your drugs. They could then be redacted electronically than the way your drugs are. It would be nice to as a more organizeal approach rather than looking in these different places. One place for an Advisory Committee and i didnt complain about Advisory Committee doctrine. You have to know basically where it was and when it was. Rather than having that indexing method, why not have an index by drug and also have it index by indication. So if youre looking for depression, you could find drugs and also find dwieevices that a approved for depression. And then you can find the drug approval package, of course. And then you can find also perhaps the new complete response letters, if that indeed is going to come to pass. Fun pediatric approval documents and the Advisory Committee documents. So just some just about out of time, so on the topic of Advisory Committee, i just want to make a plea, just a suggestion that we are talking this is symposium on transparency at the fda. Why not an Advisory Committee for transparency at the fda . Rather the once every ten years we could have an ongoing discussion between the various stakeholders. Thank you. Good morning. Id like to acknowledge that one my colleagues, one of our other Health Researchers helped coauthor the comments im going to make. And as the final speaker of the day youre going to find im going to be reiterating many of the points youve already heard today. So much of the information upon which the fda relies when making pivotal regulatory decisions is kept secret. And one prominent example of the fdas lack of transparency concerns new drug applications or mdas including supplementing mdas that have been rejected by the agency or withdrawn from buy the company. The fdas longstanding policy is that it does not release its analyses of data submitted for such applications or disclose response letters or reason for those actions. Nor the agency notify the public such reactions or withdrawals have occurred. By contrast the fda releases to the public its detailed analyses and findings related to data pending approval of support by request of at least three individual for new uses of around marketed drugs. So id like to bracket my general comments with two reallife case examples. The first involves a nonsteroidal antiinflammatory drug that was marketed in the u. S. Jointly under a brand name before it was removed from the market for safety concerns in 2005. In january 2001 one submitted the initial mda to the fda for approval to market this drug for four indications. Relief of the sign and symptoms of ought yo arthritis and of Rheumatoid Arthritis and prevention and treatment of acute pain including opioid sparing and prevention of operative pain. In november of 2001 the fda approved the drug for only the first three indications but not the fourth acute pain. Public citizen requested from fda a copy of the approval packets. And in early 2002 the fda posted on its website a complete copy of the requested approve package. But a few days later before Public Citizen staff saw the information and at the request of cyril the fda removed the information on its pack claiming that the information involved trade secrets and confidential information. In may 2002 a press release for touting dextra. The article was cosponsored and three of its five authors were employees of pharmacia. The fda released much of the information previously redacted from the approval package which showed that the fda had denied approval for treating acute pain because of safety concerns. In particular safety data from a trial in patients undergoing coronary bypass surgery the 2017 blueprint builds on the work of the fdas 2012 Transparency Task force which recommended that the agency among other things release complete response letters to shed light on why drug applications were refused. The blueprint noted several potential benefits of releasing such information including that the Clinical Community can benefit from the insight, expertise and analyses of fda reviewers and researchers can learn from the failures of Subsequent Research programs. Keeping the public in the dark about unapproved drug marketing applications prevents patients, researchers and Health Care Providers from gaining insight into why a drugs application was not approved. This lack of transparency is particularly troubling in cases where the fda has found a currently marketed drug to be ineffective or unsafe for a newly proposed indication. Disclosure of the fdas findings in such cases would promote Public Health by encouraging Health Care Providers from prescribing drugs that the fda has proven to be dangerous or ineffective. Disclosure of complete response letters is all the more important given the current permissive framework allowing in some ways the market of drugs. Existing fda always permanents drug and manufacturers to limit through scientific journals and reference publications. Such erosion of restrictions on off label marketing make it vital that the public be informed of those deemed by the fda to be in effective or dangerous for patients. Failing to provide information also gives Companies Free reign to craft their own selfserving narrative as to why their products were turned down. Finally a new policy of transparency whereby the fda discloses the existence of in data related to new drugs and new indications for already approved drugs would also be consistent with the bell minute reports principle which benefits human research. In the event that a drug marketing application is rejected by the fda because of the fda determines the drugs harms outweighs its benefits for a particular use both the drug company and agency have an obligation to make this determination public in order to avoid future Clinical Trials of the drug or a drug in the same class that would unnecessarily expose human subjects to harm. Now, a policy whereby the agency, the fda releases public response letters in the underlying analyses leading to the agencys decision not to approve an application is certainly feasible. In 2004 the European Union required bma that the bma make publicly accessible the reasons for them. The eu law stipulated that after a decision where the sponsor has withdra withdrawn, the agency must publish among other things a public assessment report relating to the Clinical Trial data in the application. One can now resource the ema website. Now, Health Canada followed suit when it announce today the public it would make available this decision notably includes final releases decisions for all marketing applics for new drugs and existing drugs. And this data can be searched for on the Health Canada website. So let me conclude with a second case example. In september 2008 johnson and johnson submitted an application to the ema for its antipsychotic drug. The company withdrew its application 85 days into emas review of the application. Quote, based on feedback indicating that the data provided were not sufficient to support approval for this indication and the companys reviewed that it was not in position to adquentequately add this issue at the time