Hi, welcome. So im mac, a professor here at georgetown, in the law school and Computer Science department. I welcome you to our after lunch panel. Thank you for sticking around. On Digital Technologies and coding. We have on our panel who i will introduce very briefly, four very distinguished experts at various parts of the digital elections landscape, and the technology and the risks of some of the Underlying Technology that is inherent there. And well be talking about this subject for, from a very wide range of different perspectives, but all with a very technological bias. So im going to very briefly introduce these people and i want to apologize now for being extremely incomplete in my introductions because they would take up the entire panel. Andrew appel is a profess her of Computer Science at princeton, where he served since 1986, and in fact, he was one of the members of my Doctoral Committee at princeton. His research focuses on software verification, security programming languages, and technology policy, and in the case of the latter, particularly with a focus on Election Technology, and Voting Systems. The next, on our panel, is kim zetter who is an Award Winning journalist who was covered Cyber Security and National Security since 1999. She has been on the staff of wired magazine for over a decade. And had written about tech and security for publication for instance in the new york times, politico, the Washington Post, and many others, she is also the author of would it serve absolutely the best book on the stuck net virus and everyone should run out and buy it and read it. Philip stark is a professor in the department of statistics at uc berkeley. His work has influenced many aspects of Public Policy from the u. S. Census to credit risk modeling, and so forth. Its a huge range of incredibly influential and important work that he has done. Most relevantly to us, his message for auditing for Election Outcomes, his method for auditing Election Outcomes are largely recognized as the standard for liable election tasks to make use of technology, something you will be hearing about quite a bit during our panel. And then finally, Barbara Simmons is chair of the board of verified voting. Shes a Computer Scientist whose work has been at the intersection of technology and policy for as long as i can remember. She is a former president of the association, and she was one of the first voices to look at the risks of the use of technology in votesing systems. And again, ive not done justice to any of these people, but im sure they will introduce themselves as well. So im going to just briefly introduce our panel an kind of frame some of what were talking about here. Technology is all over our elections. We have Voting Systems which are probably the most prominent and the most highly visible part of elections. In particular, the technology in Voting Systems can sometimes be essential for the integrity of the ballot itself, for the integrity of the vote count and so forth. Voting systems are probably the main subject of the help america vote act. And there are obviously quite important, theyre obviously quite prominent, theyre targeted by corrupt candidates and their supporter, who have the aim of altering the official outcome of elections. So the risks are quite serious in Voting Systems, the integrity issues are quite important. But thats not the only place that technology exists in election systems. We also have the election management infrastructure. This is less visible. Its less standardized. It tends to be built and maintained by individual counties and its dependent upon the logistics of elections includingth registration of voters and the voting books to check voters in to determine whether they are allowed to obtain ballots in the first place. These systems as well are exposed to risks and to the attention of hostile actor, particularly hostile state actors who may not be seeking to choose the outcome of an election so much as disrupt the election or cast doubt on the length macy of its outcome. And these systems are much larger and more exposed than the Voting Systems that we tend to focus on. So the issues in technology and Election Integrity are vast, and we will be touching on them quite a bit. I think this all starts with a photo. If you recognize this photo, that dates to you some extent, as having been around at the turn of century. Who recognizes this photo . This is of course the recount of the 2000 president ial election in florida, and this photo and variants of it show, you know, somebody applying scrutiny to a piece of cardboard that was used to express the voters choice during the election. And this photo is interesting, because we all know it because it was on the news and in the newspaper pretty much every day for over a month, as the example of how terrible things were in the florida election. This is a national embarrassment. And we have to do something to make sure that this never happens again. And i think its interesting that this photo has almost inverted in its meaning, in the two decades since, as also a look at the 2000 election, that a human being could look at a physical artifact and make a judgment about what the voter may have intended, and we may agree or disagree about but we can at least get closer to the truth by examining this. So where does this come from . And it came in fact from a Technology Failure in the voting equipment that was used that did not itself actually involve computers or even electricity in the voting booth. This was the voting machine that produced those ballots, it is a voteomattic, and the technology involves taking a perforated punch card, inserting it in the top of the, and making your choice among a column of hole positions, using a stylus, by punching through the ballot. And that would have been produced, a punch card, that could be put into an electronic calculator, and the vote tallied according to what hole positions had been punched out. The interesting thing about this is that the only electricity even involved in the voting booth itself is for the lights in the room. This is a completely mechanical device. And yet it had a failure rate, which a Computer Science would recognize probably as a garbage collection failure, in that the, as more voters than usual showed up for the voting, for voting, eventually, the little pieces of cardboard that were punched out from the more popular candidates would back up behind the position where the ballot was to be punched out. And as the day would wear on, in a hotlycontested and very popular election, eventually would become physically harder to vote for your candidate of choice, and the difficulty would ule actually be proportional to how popular the candidate was, because of the pieces of cardboard behind the ballot. And the consequence of that was that, you know, a properlypunched ballot, if you look at hole number 68 there right in the center, will be cleanly punched out and easily read by the optical electronic tally device that these cards were fed through. But what might happen, is that if we look in the center, instead of the cleanly punched ballot hole that we see in the upper right corner, we might see only a dimpling of the little cardboard square, a term that became known to every american, the dimpled chad, or we might create a flap, where the hole might have been, it might close up, and neither of these two, a hanging chad, in this terminology of these machines, and what would happen is that these cards, a human being would be able to look at them and say ahha, there is a dimple in that position, but the reader, which worked by casting a beam of light through each position, would interpret either of these conditions as no vote. And what we saw in a very close race is that the number of people in certain counties who didnt vote was within the margin of victory, for the winner. Greater. Sorry, the greater, the margin of victory was within the margin of nonvotes so we had to resort to this lengthy tedious recount that you can read about now in the history books. So the country, unlike today, was heavily divided on who should be president. It was a very sharplydivided, not at all bipartisan world, where there was bipartisan agreement on one thing, which we should replace these florida punch cards Voting Machines. And congress very willingly and bipartisanly passed the help america vote after the 2000 elections and mandated that states shift to acceptable voting technology, that would generally mean electronic touch screens with Adaptive Technology that people could use if they couldnt interact with paper, or with an ordinary type of interface. So there would be various assistive technologies available. And it also provided substantial funding to the states to purchase new equipment. Unfortunately, for the most part, the equipment mandated by the help america vote act didnt really exist at the time that the, the task in full production form, and the understanding of how to do, build equipment, that would comply with help america vote in a reliable way, was not well understood by the technical community, and to the extent it was sbunderstood, it really wast a design consideration in the certification of the equipment. So they allowed for a number of different types of voting equipment. The most prominent is the direct recording, electronic voting machine, which is essentially a computer that stores the tally of the votes cast on it internally. In the computer memory. It also permitted optical scanned ballots and absentee ballots that are mailed in on paper, as well as assistive technologies like ballot marking devices which im sure will be discussed by the panel as we two on. But i want to talk a little bit about our Voting Machines, because these probably received the majority of attention from people concerned about technological risks. These are essentially computerized Voting Machines, these are computers in a particular form that make them look like a voting machine, but these are really computers just like your laptop or your desk top or your phone, in that theyre controlled by software. And the tally of votes is maintained inside the memory of the machine, under the control of the Software Running on the equipment. And so many questions have been raised over the years, about whether vre Voting Machines can be made reliable enough to use in election, and the overwhelming consensus of experts is that in fact they cannot, yet many states are still to this day using them. So with that, id like to turn this over to our first panelist, andrew appel. I think if we just keep going right here. Yes. No, its not. Okay, so ive been studying Voting Machines and computerized Voting Systems since about the hanging chad bee cackle. In 2017, i was asked to serve on a National Academy of Sciences Medicine an consensus study panel which met for five, twoday meetings over 18 months, we heard from panels of witness, both scientific and election administrators. On the panel, there were two University President s cochairing, five Computer Scientists, mathematician, two social scientist, one law professor and three actual Election Officials. And we were asked to write down what is the Scientific Consensus. Dont invent any new science. And we wrote a report, here it is. Which has many recommendations backed up by even more pages of scientific explanation. So here are the key recommendations which i dont nearly have time to describe all of. Now, let me get to the main two or three points. Elections should be conducted with human paper ballots, marked by hand or by machine. That was the Scientific Consensus in 2018. That no longer the Scientific Consensus and ill come back to that. Counted by hand or machine by optical scanner and audits should be made by human inspection of the ballots. Voting machines that do not provide capacity for independent auditing, machines that do not produce a voter verifiable paper audit should be removed from service as soon as possible. So these are the dre machines that you just heard about. They dont have a paper trail. You interact with them on the touch screen. And at the end of the day, the Computer Program in there says how many votes each candidate got. And so it is very easy to commit largescale election fraud, by a Computer Program that states, make sure it doesnt cheat except on election day, like those volkswagen diesels, they wont test in theyre in the test environment outside of election day, and get them installed on all of the Voting Machines and there are various ways to do that. So heres me installing some software on a new jersey voting machine, well, its in new jersey, i own this one, and it would be a felony to do it on a real in use in the state of emergency voting machine but modern computer, it is much easier to install Software Just through network promle gation, and in fact, any Computer System nowadays has so many lay efrs software, the top layer is the application that counts your votes and below that, the operating system, and the hyper binder and the bios, there are millions of lines of code, thousands of software bug, and amid the software bugs, some portion are exploitable vulnerabilities to allow hackers to install a different application on top. Vote stealing software instead of vote counting software. It can propagate through a note work and it can also propagate on removable media which stucks net did. And ill tell you more about that. All right. So anybody with a bachelors degree in Computer Science could write that Computer Program that shifts some of the votes around but doesnt do it when it is not election day. So the solution recommended in the National Academy report, the Scientific Consensus is to vote on optical scan forms. This is a much Better Technology by the way for punch card, for a couple of reasons. One is that you have it next to the candidate on the change sheet of paper and the pen is intended to be read by humans as well as read by humans and the Technology Works and highly accurate and there are measurements of how accurate it is. And then we count those in some sort of machine, even a precinct count optical scanner that the voter feeds the form in in the precinct or deposited in a ballot box or mailed in for a central count. And this year, here is how were going to vote. This information is from the database of the verified voting foundation. In white or cream colored, optical scan form, counted by op scan computers recounted by hand. And in light, light brown, are places where for accessibility, theyre going to use paperless dres or maybe dres with paper. It is not a great idea. And then in dark, dark brown, and in red, are places where theyre going to use dres without a paper trail, or ballot marking devices with an inadequate paper trail. So part of the good news is that most states use about the right technology for voting. They use the most auditible, least insecure technology. And there are a few laggard states such as my own state of new jersey, that are still using paperless dres. Now, there is a bigger problem, that most states dont actually audit their paper ballots, or even recount them with any reasonable probability, and well come back to that. So, heres how we vote. The voter marks an optical scan ballot, feeds the scanner, and the scanner is just a computer and all my remarks goo to a hacker that could be part of that machine as well and if the hack her fraudulent software, that software could shift the election by miscounting the ballots. What saves you is that the paper ballot drops into a sealed ballot box and you can maintain a reliable chain of custody of that ballot box to the polls place where it can be stored for auditing and recounting later, then you can trust the results of the election independently of any possibly hacked computer. So if you have to recount the ballots by hand though, whats the point of having a computer . And the answer is, you can do a random sample audit of the paper about the ballots to, it be assured of a statistical guaranteed probability that the outcome of the election is consistent as reported by the computers, its consistent with what is actually on the paper ballot. Now, many states do some sort of random audit. But only recently have there been a real science of what kind of random audit will give you that guarantee. And professor stark will be talking about that later. So ill just cite the National Academys consensus study report that says that states should mandate, risk audits prior to the certification of election results. All right. So some states do some sort of audit. The states in pink here do no audits or completely unsatisfactory audits. The states in yellow do mostly unsatisfactory audits. The states in blue do moderately unsatisfactory audits and all of those states you see in green are doing satisfactory audits. But several other states are following along with pilot projects to, on the process of adopting high quality audits. Now, ballot marking devices. Ballot marks devices remanded by the Health America vote act as an assistive technology, that is for voters who cannot mark a paper ballot by hand, they can use some sort of touch screen that will mark a ballot for them. Now, you might ask, how is the touch screen going to help the blind voter . But the answer is that there is an audio part that they can plug into that device with the mashing of the ballot. Some states in the last year or two have started adopting bmds for all ballot marking devices for all voters voting by ballot that is fed into the optical scanner. In the National Academy report, there was some concern about ballot marking devices, about voters actually respect what is printed out on that piece of paper, after the voter makes the selection from a touch screen, and the card comes out, that records their choices supposedly, if the computer and touch screen hasnt been hacked. Will they examine it, before they put it in the ballot box . And there has been no research on that. So the Scientific Consensus as of september 2018 didnt really have any answers there. But since then, there has been research. There have been two studies, and here i can show one of them, that show that in real polling places, voters dont really look at the cards. And when they do look at the cards, they look at them for just a couple of seconds, not nearly long enough to check that every contact on the ballot has printed on it the choice that they indicated on the touch screen. So that if the computer had been hacked, and is misrepresenting on to the card their vote, what will happen in the recount . The recount will recount whats printed on paper. And a newer study from the university of michigan shows that 93 of the voters dont notice. This study was done with real voters, in a fake election, right . Something set up as an experiment in a public library. When the Voting Machines deliberately changed a voters selection and printed in one context a different name on the paper ballot, 93 of the voters didnt notice. So this means that the paper trail that comes out of a ballot marking device is not a reliable indication if the computer has been hacked of what the voters have indicated. You might think, well, at least 70 of the voters were noted, and they can serve as a check that detects the rest of them but in an analysis that we did and published last year, shows that this doesnt really work. Suppose the hacked ballot marking device changes 10 of the votes, in the race for sheriff, and if i could steal 10 of the votes i could convert a landslide loss into a marginal win. And suppose 10 of the voters actually examined the ballot, enough to notice, and 50 of them actually know what theyre supposed to do, and just alert the poll worker, and then what happens . One out of 200 voters will raise their hands, and whats the poll worker supposed to do . Void the ballot . Let the voters recast the ballot and lets assume the machine doesnt cheat that time. So that means that the hacked ballot marking device exceeds in stealing 9. 5 of the votes instead of 10 . Now you might think well the voter caught the machine cheating redhanded, we can do something about that. And the answer is no, you cant. There is nothing you can do. Then the voter says, this paper doesnt have on it what i indicated on the touch screen, theres no evidence of that. The voter might be wrong. The voter might be mistaken. And furthermore, we cant invalidate an election because one out of every 200 voters raised their hand and said the machine made a mistake. If we did, then you can see what the attack on elections would be there. Just get your friends to raise their hand. They should not be used for most voters only voters who cannot mark a paper ballot by hand and do not wish to vote by mail in some other way should use ballot mashing devices. And states and jurisdictions should not adopt ballot marking devices to be used by all voters because of the danger of hacking. All right. The next category has quite a bit to say about internet voting. There is no known or immediately foreseeable technology that can secure internet voting. But i wont talk about that at length because one of the other panelists will. And the report makes recommendations in many other areas regarding Voter Registration, Voter Registration database security, electronic poll book, ballot design, bad ballot designs on touch screens or paper, and can cause voters to overlook a certain context and whether it is even there, so states should require the use of well understood and published principles for ballots. The congress should fund the Election Assistance Commission to do its job, and fund mist to work with the Election Assistance Commission as it does. States should participate in eric, the electronic Registration Information center. And so on and so forth. But i will wrap up here as im out of time. Okay, thank you. [ applause ] okay, so im going to talk a kbit abo bit about telling a story how voting machine vendors control the message about their machine, and feed misinformation, some people might say lies, to Election Officials and that information then gets passed to the public, to the media and the public. So weve heard a lot from voting machine vendors and Election Officials that Voting Machines cant be hacked because theyre not connected to the internet. All of those vulnerables that andrew talked about are not a problem because no one can access the machines. And it turns out that that messages theyve been giving us for years in particularly after the 2016 elections just isnt true. So this is the message that was given by a voting machine vendor and has been repeated by the National Association of secretaries of state, in this case, denise meryl, is the president , was saying that our Voting Machines are not really cyber at all. And we assume that she means by that that theyre not connected to the internet. Dhs had said that the machines are noninternet connected. And cbas, the Election Assistance Commission that is responsible for testing and certifying machines also wrote an oped prior to the 2016 election the month before in the Washington Post saying that everyone calm down, forget all of the hype, you cant actually hack these machines. No eac certified voting machine is connected to the internet. Thats not true. There is a lot of voting machine, in a lot of districts that after the election is over, they want Rapid Transmission of results. Usually, are the results of an election are sort of a memory card inside the voting machine, and at the end of the election, the poll worker shuts down the machine and takes off the memory card and then drives it into the election office. And there is a lot of pressure to get back, because in iowa, people are impatient to get the rapid reporting so Voting Machines, sold officials on the modems. So at the end of the election, the machine goes into shutdown mode. And then this option pops up about loading in the results and so the system will ultimately then dial in and send the votes over a cellular modem to a server on the internet that collects the results. So everyone will tell you and they told me, every time ive spoke within them, cellular modems are not Internet Connectivity and they will say that doesnt mean it is connected to the internet, it is using a cellular modem, thats not true. The election systems and sort of wear which is the top Software Maker in the country, this is one of their statements over and over again. But this is one of their own diagrams that they actually gave to rhode island in 2015, and you can see that circular part in the center, there showing that modem transmission, using the wireless mode dem, its right there, it is on their own diagram that it is going over the internet. So what happens is that the transmission of the votes, the voting machine will dial in, using the cellular modem and it contacts the nearest cell tower and then the data goes through that cell tower into the carriers back end network and then the data has to get to that county network and it goes over to the internet to a system of server thats on the internet to serve those votes. So weve already, we already know basically showing the misinformation of the transmission of those votes. So they will then say it doesnt matter because all of that process is cured. So the transmission of the votes are secured so that no one can intercept them and read the vote ors alter them. The modem is configured in such a way that no one can actually dial in to the modem. It can only dial out. And it can only dial out when the machine shuts down at the end of the election. So apparently there are all of these safeguards. And the back end system that receives the transmitted votes is supposed to communicate only with one of those authenticated machines with the modem. The problem is none of this has been tested or certified. So Voting Machines themselves go through sort of a federal testing lab process, and through certification, and the modem transmissions dont. So we dont know what is inside those modems. We dont know how they work. They we dont know how theyre configured and if they dont have a track record on implementing security. So we dont know that the way that these are being transmitted is actually the case. What can happen of a modem of a machine . I dont know if youre familiar with something called a stingray. Its a device that Law Enforcement uses and the military uses and what it does, is it masquerades as a legitimate cell tower. It transmits a much more powerful signal than the nearby cell tower so your cell phone will connect to the stingray instead of the cell tower. And then it might pass you on to the cell tower as well. It is mostly used for tracking phones, but also sting rays are used to intercept the content of communications. So if youve got a cellular modem in a voting machine, a rogue person can put a rogue cell tower near some kind of voting precinct, and instead of the modem connecting to that cell tower, it connects to the rogue cell tower. And you can intercept data, if it is not properly encrypted, you could independent the data and change the results or basically swap out the whole package of results if it is not authentically signed and replace it with your own package of results and those go on to the server. Election officials will tell you, it doesnt matter, these are unofficial results on Election Night that get transmitted, the real results are the memory card that gets walked in. But if you have transmitted results at the end of the election that dont match the votes on the modem, you can imagine the mayhem that is going to result to that. There is a mistrust in that election. Thats not the worst problem with the modem transmission. So if youve got that cell tower, you got a rogue cell tower, that is actually an entry point back into that voting machine. If that voting machine is connecting to a rogue tower, if there is a vulnerability in that modem, a hacker can actually transmit mallware back to that voting machine to the rogue cell tower. Once youre in that voting machine, that way, or maybe you got into the voting machine prior when it was programmed and you now control the configuration of the modem. So even if that modem will only work at the end of the election and will only call out and not receive calls in, if you control the configuration of that modem, you change all of that. You can have that machine contact your system at any time you want, for however you want, and you can do reconnaissance on that machine and study it. And establish your attack. So thats a stingray. Again, these are high end systems that are used by Law Enforcement and military. Its not theoretical though to have a rogue cell tower. This is a story that is published a while ago, about the rogue cell towers that were placed somewhere in the vicinity of the white house in washington, d. C. , and possibly by nation state ak, to we dont know, but it is not just people who are sophisticated, this is a sample of a homemade do it yourself stingray. It is self encoded entry catcher that was made by a hacker and presented at the conversationfe 2007, with 1500 and that antenna can pick up data from a phone, one two miles away. Thats a pretty raw sample. This is now 2007, much later, you will have much more sophisticated, more powerful systems that will cost only a couple hundred dollars in hardware it put them together. And rogue hackers or someone who wants to disrupt can do. That and lets talk about the voting machine that was never conducted conducted connected to the internet by anyone. This is a statement made by 2018. Matt puts together a voting machine hacking village at the hack con in las vegas in the summer and in 2018, when they were coming up to a couple of weeks before the conference, esms got very nervous because their machines were included in the machines the hackers would be looking and they put out a statement which i was able to grab to their customers and the statement, this is my annotations on really what they were saying and what they werent saying and one of the things they say here, dont worry, first of all, again, the hackers, if they find vulnerabilities in the system, it doesnt matter because they have unfettered access to the machine, and thats not a real world scenario. Again, they said, the errors are pointing, these systems are connected to the internet. What they actually say is no vote tabulation system is connected to the internet. Which is interesting. Because the voting machine itself isnt actually the final tabulator, right . It turns out that that back end system that does tabulate the votes is connected to the internet. So they have lied. So we will say that Voting Machines are connected to the internet, except when they are. So when i, i wrote a story in 2018, february 2018, for the new york times, about the modem transmission, and the reap cushions, the risks that that cracied, and there were a group of researchers that decided that they would try and see if they can find those back end servers that received the votes that are transmitted by modem. So if you thought something was transmitting the votes over the cellular network, theres something that has to be connected to the internet to receive them. So there is a server. And it turns out that they could actually, based on Configuration Information that is publicly available on the internet, that the voting machine vendors provide to election offices and the election offices post on the internet, they described the type of fire wall that they used, made by cisco, they described the type of sap software that tallies votes on the server, they described the whole configuration, including the type of cellular modem im bedded in the machines. Based on that information, they decided to see if they could look for that very specific footprint of these machines that are receiving the votes and they did a scan and they were able to find the systems on the internet. So they found nine responsible counties that had systems connected it the internet seven florida counties and four michigan counties. They found systems in ten different states but these were the primary ones and of course these are all important critical swing states. So heres the thing. Election officials will tell you, well, the modem transmissions dont matter, because we only turn on the Voting Machines for a very brief period, less than a minute at the end of the election to transmit and that is not sufficient time to hack. And the experts on the panel will tell you it is more than efficient time to get the system. And the back ends systems that receive the votes are quite often connected year round. You can see when theyre being scanned, you can sometimes see them, they come up a couple of week, some of them that are temporary will sometimes come up a couple of weeks before the election because they want to test the transmission and leave it on for those weeks before the election, and after the election, they might forget to take it down, and it might be a couple of other weeks and there are some who dont take them down as well and one of them was on year round. And this is a problem. These systems, what are they . I started with the server. But its sounds kind of benign. What is happening is the system, the information transmitted and on the receiving end it is fire wall connected to the internet and behind the fire wall there is a server that the data for both are transmitted on. And now if that ftc server is supposed to serve as a d number z, dmz, a safe zone. And it turns out thats not the case at all. This is a diagram that is illustrated and handed out to Election Officials. And you can see the votes are coming over the internet that cloud, and they are coming in the fire wall and you can see all of the wires connected and you can see at the bottom there the ems, that is the election Management System, that is the system that tabulates the final results. So even though they say that that transmission of votes over the internet is just unofficial, connected to that system that is receiving those unofficial votes is also the system that is tabulating the official results. Whats more, that election Management System is also used to program all of the Voting Machines prior to an election. So all of these systems are connected to the internet. They are Critical Systems. Oops. Only a couple more slides. So when i brought this to emss attention, they didnt say that nothing is connected to the internet, what they said was none of those Critical Systems are pingable from the internet because there is a fire wall in front. So essentially what theyre saying is that even though now, first they said that none of these are connected to the internet and now when theyre faced with someone saying they actually are connected to the internet, well, they maybe configured in some manner that youre showing, but there is a fire wall in front of them, and therefore, you cant see, and you cant see what is behind the fire wall but if you could find the fire wall, then you find the systems that are behind the fire wall. So can i have a couple more minutes . About one more minute. Im wrapping up. One minute. So the systems, the election Management System, they arent anything sophisticated. Just a laptop with the election Management System software on it. That fire wall that sits in front. The only thing that is protecting anyone from getting into those Critical Systems behind the fire wall are the rules of the fire wall that say only these certain systems can connect, and only these certain systems can get it. That is simply software. It is configuration rules. And if you misconfigure that software, then anything can get into that fire wall. And of course, many, many hacks happen because fire walls are reconfigured. So this was, here is a another problem though. If that fire wall has hasnt Software Vulnerability itself, you can actually bypass any of those rules and any of that protection and get into the, past the fire wall to the critical system. And the very cisco fire wall that ems has used had a critical vulnerability announced in january 2008 by cisco, a password was released, but the patch was released and it turns out the patch was insufficient and many systems remained vulnerable for a long time before the patch was available. And because a patch is available doesnt mean the system is patched. And shortly after the patch was available, Craig Williams announces that they are seeing exploits of the vulnerability in the wild. Hurry up, everyone, patch your system. A warning. People are actively trying to attack systems that have this vulnerability. So we see now, one more, in wisconsin, those systems that were connected to the internet, would see the cisco announces in january 2008, i read the story in february 2018, and i asked, are you aware of the system vulnerability and are you patching it . And my impression is that they were not aware of it, and at that point, they started notifying customers. So february, 2018, they begin notifying customers. March 2018, wisconsin election commission, cant immediately establish that patch, a long process for adding patches to servers and they yash yated process in march and they dont actually do the system until july. So six months now in which a vul inability is public and actively being exploited and the system is not getting patched. These systems are fragile. Thank you. So philip . [ applause ] thanks a lot. Thank you very much for having me. Its an honor to share the podium with this distinguished group. So a lot of this has been covered already by andrew and kim, but the standard argues that our elections cant be hacked in the u. S. Are some combination of physical security, you cant get access to devices, theyre not connected to the internet, theyre tested before election day, and the system is too decentralized. Its run by a bunch of individual Election Officials and individual counties, and jurisdictions. So its a hard target. Most of this has already been debunked by previous speakers, but physical security is securi. Equipment has sleepovers in school gymnasiums and churches. There are lots of examples on the internet of photos of election equipment warehouses where the door is propped open. It isnt true that the machines arent connected to the internet and even if they werent connected to the internet, teld still be hackable through other means. One of the things that hasnt been mentioned yet is supply chain hacks. There are components that come from foreign countries. Chinese pop songs were found in the memory of a voting machine he bought on the internet. Those songs made it through the Quality Control of the election equipment vendor and then through however many elections it was used in by the local election official and were still there. Moreover, theres an issue in reporting which im not going to talk about that much. There are a number of states that outsource their reporting of elections to third parties, some of which are corporations based in other countries like spain. Youve got to trust that the aggregation of the votes and the recording of the votes is accurate as well. Tested before election day, something can behave differently when its an operation than when its being tested, if it knows whether its being tested. One clear tale tail, the decentralization turns out not to be true. There are some mom and pop shops that are responsible for configuring machines. They have very little security of their own. And another feature of decentralized systems is that there will be the weakest link. To tip the results of largest contests, the outcome of the 2016 president ial election could have changed by altering Something Like i think fewer than 20,000 votes if you did it in the right places. You dont have to hack the whole country. You just need to get the right precincts and a few swing states and you can change whats going on in a tight election. Lets see if i can find the right button here. You can use the spacebar too. Great. As andrew mentioned, the thing that we really need to be working with is paper. Why is paper so special . Its hard to think of paper as a technology, but its a Wonderful Technology for this purpose. It has important security properties. First of all, its tangible and accountable. If youre careful, you can keep track of how many ballots you sent to a polling place, how many came back voted, spoiled, unvoted. In order to do that you want to do things like use ballot stock that can be distinguished from other paper you can buy at best buy or Something Like that. Its temper evident. You cant absolutely remove marks or alter marks, but its hard to erase without leaving some kind of trace. If you have good protocols, you can trust it. Roger johnston who did physical security for Argonne National lab, at some point got interested in Election Security and went to his local election official to say show me how you use your seals and was shown this box of ballots. You turn it upside down, you can open the box from the bottom without disturbing the tape. Having serious protocols around seals, making sure theyre not something thats easily defeated with solevant. If were talking about the human readable portion of it, matt talked about that as well, one of the features of hanging chads is that something could actually look and see that whereas you cant look at the electronic state of memory of a Voting System. This is becoming troubled again because there are places where the vote of record on the paper ballot is no longer the human readable portion of the paper ballot. Georgia is flirting with right now making the qr code of the ballot the essential part. In order to make large attacks on paperbased voting, that requires a lot of accomplices. You need to make a lot of paper disappear, fall on a truck Something Like that. Electronic hacking of Voting Systems can be done by a small number of people. You dont need physical access. So paper is a great thing, but paper is not a panacea. It matters how you market, take care of it, how you tabulate it and how you audit it at the end of the day. Andrew talked at some length about why ballotmarking devices are not a good way to mark the paper if you want to know that the paper has a trustworthy record of what voters expressed to the equipment. Ill talk about that a little bit more. Lets suppose that we have generated a trustworthy paper trail and have kept track of it. How do we figure out whether the winner won . We cant trust the computers, we cant trust the people who were involved, we can maybe trust some aspects of the system. What can we do . So right now were in a situation where we have to trust Election Officials for just about everything. And im not saying that theyre untrustworthy, but it would be nice to not have to trust people. In the u. S. , we have procedurebased elections. The way an election official finishes an election, i followed the rules, i used certified equipment, this is the result. Trust it. And i would liken this to a brain surgeon doing brain surgery and saying i used a sterile scalpel, the patient is fine, right . You actually ought to look. Thats what auditing is about. You want to look and have evidence whether things are conne correct and not just rely on procedures. Handcounting ballots in a group can make mistakes and perhaps does. The question is, is it true that despite all of the things that might have gone wrong, the reported winners really won, or did these problems actually rise to a level of altering the political outcome of the contest . Its an unattainable goal to insist on counting every last vote with perfect accuracy. Its not going to happen especially if we allow handmarked paper ballots. People are going to mark ballots in funny ways. Humans looking at the ballot can tell what was intended and there are examples from recounts in minnesota where what was the fraction of votes that were ambiguous. 99. 99 . Were unambiguous. Some people will say, we cant have handmarked paper ballots because machines cant read them. That doesnt matter. What matters is a human can read it at the end of the day. The goal is count well enough to get the right political outcome, winner or winners. All right. So here are the three cs of Election Integrity evidencebased elections. The voters need to create a voterverified audit trail. Then you have to take care of it. There needs to be adequate curation. Thats a nontrivial problem. Its a physical security and an accounting problem, not a cybersecurity problem. Its the kind of thing that if if we ought to be able to count on local Election Officials to do anything, we ought to count on them to keep track of the paper. That seems like job number one. And we need to have some kind of of an audit of the reported results against that paper trail done in a rigorous way that has the possibility of correcting the outcome if the outcome is wrong and has a large chance of correcting the outcome if the outcome is wrong. And that takes us to risklimiting audits. How can you catch wrong outcomes . If you have a paper trail, you can count the votes by hand and that will tell you who really won. Thats expensive. So what are we going to do instead . If youre willing to permit a small risk of not correcting the outcome, you dont need to look at that many ballots if the outcome is right. Its any procedure that has a known chance of correcting the reported outcome if the reported outcome is wrong and it will never make a right outcome wrong. It will only make wrong outcomes right. Wrong means that if you were to count the votes on a paper trail accurately, you would get a different answer. Trut worthy means that the paper trail reflects how the voters voted. The only way they work is by resorting to a full hand count if they dont get Strong Enough evidence that its right. So the basic rule, you keep collecting evidence until you have convincing evidence that the reported outcomes are right. If you never get convincing evidence that the reported outcomes are right, you look at everything. You stop only if it becomes clear that its pointless continue. Its not looking for a smoking gun, its looking for evidence that the outcomes are right. Its been endorsed by a lot of people. Theres a number of ways of doing it. Im going to talk about one of the lowest tech ways of doing it which is a battle polling audit. Its like an exit poll. You ask the ballots what votes they have on them. They have to tell you the truth. I just want to give you some examples of how much work this is. We looked at a number of every president ial race is really 51 contests, 50 states and d. C. If we looked at what it would take to limit the risk to 10 for the president ial contest from 92 to 2012, you would look at less than 308 ballots statewide for half of those. This is not a heavy burden. Some states would have to look at a lot. If youre talking about a tiny margin in the swing state thats going to be different. Even in the 2016 president ial election, we could have audited looking at less than a half of a a half of a percent. Thank you. Thank you. Barbara . So i was going to call this talk i can bank on line, why cant i vote on line . But then iowa happened. And so ive been kind of obsessed for the past few days with trying to figure out what happened in iowa. Its not really voting, per se, because theyre returning the results, not casting ballots. I have information about iowa. In fact i just had a 45minute conversation with the ceo of shadow. Some of you may know thats the company that did the app. Jeff gary from georgetown was with me. Is he in the room . Hes over there. If you have questions about it afterwards and im not around, go talk to jeff. So this is the outline. I did put iowa last because we have ten minutes. And i figured if we dont get to it now, we can talk about it later if youre interested. This way. So what we should not do, we should not do internet voting including cell phone and block chain. Im not talking about the voting. There have been multiple warnings as my panelists have said. Robert mueller most notably when he testified was most animated when he talked about russian interference in our elections. James mattis, the secretary of defense, former, i think, warned about russian interference. Christopher wray who is still the fbi director did likewise. And the intelligence communities have been warning consistly about the threats including the Senate Intelligence committee in a bipartisan statement that said the department of Homeland Security assessed that it included all 50 states on voter information, election System Software and Election Service companies. So if you werent already worried about the 2020 election, you should be now. Now, when it comes to previous elections, like the 2016 election, we dont actually have evidence that votes were changed, but then again, we didnt do a careful search to try to find out. Thats one of the problems is that to say that theres no evidence that something bad happened doesnt mean that we know for sure that something bad didnt happen. Its different. So internet voting is the return of a voted battle over the internet via web attachment or email. I think its probably still the case that there are some people who dont understand that email is internet. Ive certainly had Election Officials say we do email voting so we dont do internet voting. But thats not true. It goes out over the internet. Email voting can be modified en route, you can have lost ballots, you run the danger of having no secret ballot if you dont have good protections. You can ballotbox stuffing with counterfeit ballots. Email voting is not secure nor is webbased voting. You can vote on your personal computer, a smart phone, smart tablet and so forth. There is Ongoing Research thats trying to use crypto to come up with secure ways, but theres nothing that is commercially available that is robust and its a very hard problem. Many of us are well, my guess is if we ever get to secure internet voting, its not going to be for a while. What do these institutions all have in common . Theyve all been hacked. It could have made multiple slides listing all of the institutions starting i guess, huawei is one of the most recent ones where there was a major hack. To state the obvious, how can underfunded, understaffed, un r underresourced local officials protect their services in an internetbased election from wellfinanced adversaries, rogue hackers and maybe even your teenage son . Now, vulnerabilities are well known among Election Security community. One is authentication. On the internet, nobody knows youre a dog. How do you know youre the voter either . Thats a major issue that really hasnt been solved certainly in the united states. Malware could change the voters selection before it goes over the internet and i think theres parts of the general public who arent experts with computers, this notion that what you see on the screen is what gets stored in computer memory, neither is true. Computers are components of things. What you see on your screen may not have anything to do with what goes out on the internet. You cant count on what you see. You can have denial of service attacks that can prevent real ballots from reaching Election Officials. I was actually in new york when Hurricane Sandy hit and i was trying to change my reservation on united airlines, it took me two days to get through. Thats one of the things that happened in iowa. Everybody was trying to call in. So that was a denial of service not necessarily an attack. But a denial of service. Penetration attacks on the vote servers. Where you send the vote can be can result in changing the votes. You cannot audit an internetbased election because you dont know if what you have at the other end is what was intended to be sent out. You have to worry about vote buying and selling and voter coercion and this is true for any kind of remote voting. Not just internet voting. This issue is also raised when you have widespread vote by mail with people who dont need to vote by mail. Regulations, there are, as you see, none. There is no independent standards, theres no independent testing, theres no government oversight, theres no legal accountability, and theres no ability to do a recount. So the National Institute of standards and technologies were asked to develop standards and they threw up their hands and said we cant do this. These are quotes from nist, technology is not able to mitigate many of the threats to cast ballots via the web. And they said malware pose a serious threat that could compromise the voters ballot. Nonetheless, we still have internet voting in the united states. Roughly 30 states another internet voting, mainly for military and overseas voters. Many of us many of the people on the stage have been involved with fighting internet voting. Its an ongoing task because a lot of people want to do it u. And weve been successful so far in primarily keeping it limited to military and overseas voters, although theres a lot of pressure to change that. When it comes to military voters, the argument was, well, people the military cant get their ballots back on time. In 2009, i believe, the move act was passed. And what the move act does is it requires states to provide only blank ballots that can be downloaded by military and overseas voters so they can download them, print them out, fill them out manually, stick them in an envelope, and mail it back. And for military voters, theres expedited return of ballots. The beauty of the move act, it cuts down the travel time. You download it over the issue. That has Security Issues, but its relatively small. I will argue that internet voting is a solution in search of a problem. There was a major study done in British Columbia in 2014. They spend 400,000 of canadian money to look into internet voting. They were able to do this because ontario allows internet voting at the municipal level. They had multiple elections done that they could look at. And based on that, their conclusion was that internet voting does little to nothing to increase Voter Participation in general and much to their surprise, it does not increase participation by young people. Now, this talk about myths. We were talking about myths earlier. This is an incredible myth that people dont let go of. There is no there is really little to no evidence that internet voting is going to increase Voter Participation. Similar results have been found in estonia and switzerland. So this is what were confronted with today quite a bit, block chain voting. Block chains are a data structure. Thats where you store your information at the end. After the voting has already happened. A block chain could be a single or multiple owner. When there are multiple owners you have to have a majority agreement about transactions, you could have collusion among owners, outside hackers, theres no Central Authority to Police Activities and of course with voting, block chain is likely to be owned by a single owner, either a vendor or local election official which eliminates the extra security you get from multiple owners and most of the point, all internet vulnerabilities are still present with block chains. Quoting the National Academy of science again, that report from 2018 says in the particular case of internet voting, block chain methods do not address the Security Issues associated with internet voting. So i just want to talk about votes which is the largest and most aggressive vendor of block chain voting. Theres no federal state certification for block chain voting. Votes claims they dont need to be certified because theyre not a Voting System because they dont tabulate the votes, but they send the votes over the internet. They do not disclose their source code, theres no testing, and, by the way, a mock election is the minimal thing to do to open it up to outsiders to try to hack it because if you try to hack into a real election you can go to jail with a big fine, also. They claim to have done security audits but nothing has been made public. So the solution as weve heard is paper ballots, strong chain of custody and audits. That brings us to iowa, do i have one minute to talk about iowa . Yeah. Very fast. Im not going to say much about it because theres a lot to say. You can ask me questions. The reason they have they moved they tried to bring in technologies because of all the criticism the caucuses have gotten because theyre so undemocratic. Its difficult for poor and Single Parents to participate in these caucuses. Iowa and nevada were initially going to allow people to vote on their cell phones. And fortunately that was stopped by the dnc. But because there are major security threats with that as i have already said. They decided to go ahead with this app which apparently, as i just learned this morning, this afternoon, the app was part of the whole project. The project was going to be the app plus the phone voting. They chopped off the phone voting, then they would redo the rfp for the app. They didnt start working on the app until october. They had no time to do it. It was, in my opinion, fairly irresponsible. So anyway the app was developed by shadow. I was talking to the ceo and i can talk to you about that. My time is up. One quick comment, they have an nda, he couldnt tell me all he said was that it was an enterprise level security firm. He couldnt tell me who it was. He couldnt tell me why they have an nda. Ill leave you with that. [ applause ] before we open it up, i just want to ask one question of the whole panel and please feel free to riff on whatever perspective on this you choose. Ive been working in this area of Election Technology for a couple of decades now and consistently during that time i, like i think all technologiologists who worked in this area, are accused of bei being im saying we shouldnt use computer technology, we shouldnt trust technology. Im just being a worry wart, sort of the perspective that is thrown back. We hear things like, well, we put a man on the moon, surely, we can build a reliable Voting System. We hear everythi has vulnerabilities, sure, your Voting Systems have some vulnerabilities, but so does everything else. We also hear we rely on computers for everything and we rely on it for the banking system. We rely on it to protect nuclear weapons. Why is this application so different . Stop being so fussy is the overwhelming that ive heard a lot. Now, i had a brief moment of optimism in 2018 when the National Academys report came out. We had what should be credible weighty consensus from the best experts, this is what the you know, we all agree on, this is the uncontroversial baseline and, you know, its at use paper ballots, do risklimiting audits, build systems to be softwareindependent. Independent of software for its outcome. And i thought, great, we now have a strong statement of what to do and yet when we see any of these maps of what is used in the country, were doing very poorly at actually deploying this and we hear again, we put a man on the moon, everything has vulnerabilities, we rely on computers for everything. What is you know, i would like to ask each of you, what is your response to that other than despair . Optimism. Since the high watermark of paperless dre, touch screen Voting Machines in about 2003, one state well, many states were using paper all along. Almost all of the states that unwisely rushed to adopt Voting Machines soon after the year 2000 have abandoned them and moved to optical scan paper ballots. They have done it for the right reason. Theres been a widespread understanding among officials that the Voting Machines are hackable and that the paper ballot is the gold standard. Theres been a lot less progress in consistent ways to audit and recount paper ballots. But even there, theres begun to be more understanding and more states of the need for audits. So i see the glass with more than 40 states now using optical scan paper ballots to record the vast majority of votes, i see the glass as at least fourfifths full. Okay. Fourfifths. So theres a lot of focus on the need for federal legislation. Everyone is very frustrated that mcconnell isnt pushing legislation forward that would require paper trails or audits. And federal law would be great, but we dont actually need a federal law for the reasons that andrew has just described. States have been very effective when they actually move and when they actually do take action of dealing with the issue in their own jurisdiction. Obviously it would be great to have it all at once, but in the absence of that, theres still optimism because the states do eventually come around. Its often after something has happened. California was the first state to demand require legally a paper trail and that happened after one of the voting machine vendors lied about the software that was installed in the systems, they said it was one version of software and it turned out to be an uncertified version. Florida passed a law requires paper trails after a major mishap in 2008 that was using paperless machines and more than 19,000 ballots didnt have a vote cast in a particular race and that caused them to demand paper trails. Unfortunately, it does tend to take some kind of mishap that moves progress along. I say thank god for the russians because we would not be sitting here today having a conversation, we would not be having election hearings, we would not be having any attention on Election Security without the russians in 2016. I think its optimistic. I dont know where to start. Lets start with florida. Florida has paper, but if i understand their laws correctly, its also illegal to audit more than one contest and the audit is not allowed to change the outcome. Having paper doesnt matter if you dont look at the paper. You have to look at the paper and the paper has to be kept secure. The right order of operations for me is, get paper, hand mark for everybody who can, learn to keep track of the paper. And thats something that no states are doing a very good job of by law or regulation. Individual Election Officials do a good job, but were not regulating that very well. Once you have trustworthy paper, you can do an audit and the audit means something. Thats a math problem. We know how to do that. Theres an enormous amount of logistical complexity in conducting an audit. The methods you can use depend on the Voting System. The simplest one requires paper ballots. But how the jurisdiction organizations its paper matters. Whether the state wants to audit from the top down as a whole or let each county draw its own samples. It does involve a lot of people figuring things out and getting practice with it which is why the pilot audits are important. In terms of why we are where we are, i would say follow the money. The people who are really benefits from a lot of this stuff are the Voting System vendors who are selling equipment at enormously inflated prices and then by having states go to all ballotmarking devices can kind of triple the revenue that they get from the states. What they say to the states doesnt seem to get policed. We caught them lying to the senate about whether the equipment is connected to the internet or not. Theres no consequences for that. Pennsylvania, philadelphia recently bought express vote xl ballotmarking devices after having fined es s 2 1 2 Million Dollars for violations of the procurement process and yet theyre still trusted as a vendor. Why would you trust them to count the votes if theyre not playing fair. Theyre telling lies to Election Officials about the security of the systems and theyre trying they sell their systems as making life easier for Election Officials. In some senses, they might. In other senses, they dont. I think that probably the best leverage here is the fear of public humiliation. The problem is that often cuts the other way. If you have a paperless Voting System, its very difficult to be publicly humiliated because theres no way to tell what actually happened. How or if you dont audit. Similarly. If you dont look at the paper, you dont know. Yeah. So im maybe not quite as optimistic as my colleagues up here. I am very worried about this whole push for block chain voting. Its being pushed by a particular vendor, but the vendor is being funded by a very wealthy entrepreneur who i believe im not positive, but i think he actually believes this is going to increase Voter Participation. Although weve been trying to tell him that it wont. So thats votes. But theres a lot of money behind votes and its not for making money. Its for pushing this idea. Theres another concern which is that the language that were using, some of it has been taken over by people who arent doing what were saying. For example, seattle is going to have a form of internet voting for an obscure election and what so people can vote on their cell phones. And they print out the paper and they say we have voterverified paper trail which is nonsense because the voter hasnt verified it and you have no idea if it was sent from the cell phone. And theyre talking about having a voterverified paper trail. This is a case where weve been too successful because people are adopting our language without adopting the rules that it entails. Im concerned about policymakers who want to do the right thing but who dont consult the experts and i think weve seen that in iowa which actually cuts both ways as far as im concerned because im also hoping that iowa will help kill block chain voting as people say, you know, you didnt test this properly and why should we trust the block chain voting. But sometimes people dont make those connections. I dont know how people can advocate for internet voting. Theres still this push for internet voting. I want to change my answer. [ laughter ] im not optimistic. Im trying to respond to his outside of despair, whats your response. We have as ann says, we have had some progress and theres a lot of attention now and yet despite that attention, there is this push toward internet and mobile voting. Election officials seem to say, we get it now, right after 2016, they were very much in denial and now they say, we get it and were willing to have dhs come in and help us and yet they make bad decisions over and over again and theres phillip was saying theres nothing to force them to make good decisions. Another worry i have right now is a lot of federal funding has been released for states to do whatever they want with related to elections. There arent have many constraints if any on the money being spent intelligently on things that improve security, reliability, et cetera some states are spending it in an unwise way. Were going to be stuck with the systems that people are buying into right now for a while. And its not enough money to begin with. Well congresss answer is to throw money at the problem. But the problem was that the money they threw at the issue in 2002 that enabled states to buy insecure machines. Throwing more money at it isnt going to solve the problem. Thanks for that. Ill adopt andrews framing and saying that im onefifth pessimistic. I would like to open it up to the audience for questions. Please use the microphone. As one member of the audience, i would like to donate two minutes to barbara to hear more juicy details on shadow. You put that tantalizing thing out there. Have you learned anything thats not yet public . What did you learn about the amount of money they spent . You said a little bit about the time they spent it wont leave the room, dont worry. I didnt ask about the amount of money. I think thats public. They got money from nevada and iowa. And candidates. And the candidates, sorry. Its a small app. Its a small app. The fact that they didnt start until october, i find mindblowing. They did do some security testing, they did not do red team testing. They found one major problem which they corrected and two or three highpriority problems which they corrected. They said the app worked find. In terms of usability testing, that was inadequate. And this app was going to be used by elderly people who arent accustomed to smartphones, they didnt get the app until late and i think part of that was the Iowa Democratic party, this notion of security through obscurity. Any security person will tell you that is wrong. You want to make it public and let people try to hack it and find the vulnerabilities. I do think this was the case where you had policymakers making decisions, not listening to people who know whats going on. My coauthor i coauthored a book and my coauthor kodoug jos is a professor at the university of iowa and he was yelling and screaming about this and they werent listening to him. They werent listening. I have four pages. I cant go through it all. Good afternoon, im david lavine. Few of you spoke about the National Academy sciences report as the report. I take a look at the report. That recommendation about audits which i think the case is compelling about being the gold standard, includes the lines that folks should adopt it within a decade. And that line is right there in recommendation 5. 12. I think the question that i have for the folks here, and professor stark began to touch on it with the lomgistics, righ, this is the goldstar report. If this is something that professor blaze has talked about, urgently adopt, why was that the recommendation . And i would welcome feedback from anyone on the panel. Let me first explain the rational for that recommendation. In many, many states now, we have paper ballots. If theres a serious question about a particular election, they are available for recount. But a risklimiting audit, orange county, california, as an example. They did an audit in the summer of 2018. Theyre first one in 2011. That involved handling 1. 8 million sheets of paper. A risklimiting audit is a logistical process as well as a statistical process. You need to design it to be in accordance with their election laws, you need to train the staff who are going to do it, you start by training them why you even need to do this at all, and then you move into the logistics of keeping track of all that paper and the paper handling. To think that we could snap our fingers now and have risklimiting audits for the november 2020 election in every state would lead to a meltdown of audits. I cant be done overnight. We have to get there as quickly as we can with, you know, training and design of these administrative and logistical procedures. Im going to push back on that slightly. Ive been involved in excess of 20 pilot audits now in local jurisdictions. If we want to audit contests that are on every ballot, then the big bottle neck is not so much the paper handling, its knowing where the ballots are and how theyre stored. My feeling is that, that should be a noexcuse demand of local Election Officials. If they cant tell you how many ballots theyve got and where they are, theyre not doing their jobs. Its the Public Service announcements when i was a kid, its 11 00 at night, do you know where your kids are . Its the morning after the election, do you know where your ballots are . The answer ought to be yes. If you want to audit every contest on the ballot, thats a bigger logistical problem. If you want to audit things in the most efficient, possible way, that may require replacing equipment. To be able to do something that still limits the risk, it would be in reach in states that have paper for 2020. If i could join in also, its not only the technology that isnt necessarily available everywhere because we still have dres, but also laws. Phillip mentioned a little bit, in michigan, i dont know if this is still the law or not, but in 2016, because i was involved in the recount effort, there was a law that said if the number of ballots that youve counted does not match the people who signed, you may not recount. Which is the opposite of what it should be. You know why . The only reason i can think of is you want to be able to rig the election. I think protection against exactly when you do most want to recount you cant. Its not just a matter of getting the technology, we have to do significant changing of laws at the state level and thats a nontrivial thing and thats where georgetown law can help us. We need your help. Audits are funded at the county level and the state level. In some cases, the question is, you know, do we want the potholes repaired or do we want risklimiting audits . There are problems at every level here in rolling this out and some of them are just really simple. Again, if were talking about a top of the ticket contest, 308 ballots statewide from more than half the time, thats not thats an expense. That doesnt require a lot of funding. It requires doing it. You have to do it. I was going to say because i witnessed the colorado audit, i went out there when they were doing it. And it took a decade to get there. You and many people were working behind the scenes with colorado to get to the point where it could do a risklimiting audit and it was an extremely complicated process of keeping track of ballots and they had a system down to it was very scientific in which they did it. And it was remarkable in the way that they did it, they knew exactly what row, what box, what number in that box was the ballot they were looking for. But that doesnt happen overnight. It took a decade over there. And colorado is a relatively small state in terms of numbers. Roughly three years after that ten years was arguments between the secretary of state and the counties over what they were willing to do and what was going to be covered by the open records act. A couple of years, maybe three of years of it was a procurement process because they decided they wanted to change all of their equipment in a particular way. It made the audit easier there are a lot of things. Yes, it took ten years. Not all because the auditing part is hard. Im lucy thompson, georgetown alum. A group of us at the American Bar Association drafted a resolution that would be presented to the house of delegates on february 17th which contains recommendations for Congress Consistent with the national economys report and it says that congress for federal elections should provide funding for nist and they should be mandatory in all voting equipment and software and related items should be certified. So its a fairly comprehensive resolution. If you all want copies, im happy to provide it. Thank you. Based on all of your important work. Thank you. I think we have time for one more efficiently asked questions. There are various formal guarantees of security correctness, risklimiting assessments or block chain. What is it that attracts people to certain such formal guarantees . Its clearly not which ones are most effective. What attracts people to what . Various sorts of mathematical guarantees. How can we succeed, is that your question . Sure, broadly. There are various places that formal methods can play a role, trying to prove that software is correct and does what its supposed to do and verifiable Voting Systems or this and that. There are some aspects of the voting security problem that are amenable to quantitative analysis that are always going to be qualitative. How do you know that the paper trail has been secure . Thats not something you put a number on. Youre going to look at security logs and video and chain of custody and all of these other things and make an assessment that says, yeah, im pretty sure its okay. Im not sure im answering your question. But i dont think that theres a formal answer to all of it. It seems that the risk assessment, which is a statistical guarantee, is not attractive to, say, votes type startup people who like to be involved in this business whereas block chain is. Yeah. Why is that . We have to somehow convince people that accuracy matters. And another thing which i would like to see people take on is the timing it shouldnt be the ultimate thing you have to get the reports in right away. I think thats one of the things that impacted iowa. They felt the need to get these results in quickly for the the press wants it. Lets take the time and do it right. People dont want to hear that, but i think thats an important message. Okay. Thank you. [ applause ] we have about ten minutes for a break. Anybody who wants to talk about iowa [ laughter ] monday, president s day, American History tv is live at 9 00 a. M. Eastern from mt. Vernon, the home, library and museum of our first president , George Washington with doug bradburn. Its also the start of museum week with exhibits exploring the american