Order. For 40 years, cspan has been providing america unfiltered coverage of congress, the white house, the Supreme Court and Public Policy events from washington dc and around the country so you can make up your own mind. Created by cable in 1979, c span is brought to you by your local cable or satellite provider. Cspan, your unfiltered view of government. Next on cspan three, a discussion on federal standards and oversight of Artificial Intelligence. The National Institute of standards and technology and the Information Technology and Innovation Foundation center for data innovation is the host of this discussion. Good morning everybody. I am a senior policy analyst with the center for data innovation. We are a nonpartisan, nonprofit Public Policy think take focusing on the intersection of data, technology and Public Policy. We are based in washington dc. We are affiliated with the Information Technology and Innovation Foundation which has been ranked for several years as a top science and technology think tank. We are happy to be here today to be partnering with this event. Particularly this panel. I am joined by lynn parker, the assistant director of Artificial Intelligence at the White House Office of science and technology policy. Next we have jason, the general manager for the Corporate Standards Group at microsoft. And anthony robbins, the Vice President for north American Public sector. We have a great group of public and private sector experts that can talk about the activities going on in the Sand Development phase and the people helping define u. S. Leadership in ai. Before we get into the discussion, we will have audience q a and i want to do some stage setting here. The term ai is often used to describe two different but related topics. There are technical standards. Things like reliability performance, accuracy. And then there is the oversight. These are very different but related things. Standards are a prerequisite for oversight. The oversight side of the conversation receives a disproportionately larger share of the attention from policymakers. Concerns about algorithmic bias, the potential for these blackbox systems to be running amok without knowing what is going on is what is dominating conversations and policymakers understandably want to address the concerns. Unfortunately, this privatization of oversight is seemingly coming at the expense of the focus of standard development. The activities required to develop standards requires a really robust Scientific Understanding that can serve as a technological underpinning for this oversight. So when we say things like we want a mandate for algorithm transparency was is what some people are calling for, right now algorithmic transparency does not have a definition. We dont know what that means. We dont know how to compare the transparency of one system to another. And rushing to make those rules without actually doing those scientific legwork behind it is going to be shortsighted and any rules will necessarily be arbitrary. So i guess the challenge for us is how do we get nontechnical policymakers to care about this really important technical work. And it is a challenge. Im sure folks know that. But we hope to get out of this conversation today and Going Forward, to educate policymakers about the importance of this kind of scientific legwork in shaping the future of oversight of ai and all the concerns that there are in the public about the potential for the misuse of these systems. We have to make sure that translates into momentum for this kind of scientific investment. So to start off, i am going to tee up with a pretty easy question for the panelists to get the ball rolling. What is your primary focus when it comes to ai Standards Development going on the line . Are you working on developing standards yourself . How are you engaging the community . What did they mean to your business or to your role in the federal government . Good morning and thank you for being here. This is an important activity. My role in this process right now at the white house, i am the assistant director for ai which means i oversee the white houses activities an a i particularly the promote side of ai. One of the important areas i am working on now is all the deliverables that are called for in the executive order and you have heard this morning the main deliverable we are discussing today which is the creation of a plan for how the federal government should prioritize its engagement and teco health Standards Development for ai. This obviously is one of the key actions in the executive order. It is recognized by the administration as an area that we as a nation need to get engaged and there are a lot of good reasons for doing that. We could go into that as we go through the panel. But right now, my role is looking at the great work being done and leading the way here. Certainly the rfi is an important way for stakeholders to provide feedback. We encourage you and your colleagues to provide that feedback. This workshop is an important opportunity to hear from everyone about what the federal government should be emphasizing. As the plan is developed and is issued for public comment, we encourage you to respond to that and provide feedback. I am cheering on all the great work being done here and everything you are doing to contribute to that. Good morning. It is a pleasure to be here as well. Thank you for nist getting this important process underway. At microsoft, i am running a team of Global Practitioners who are involved in standardization at the International Level through primarily, in relation to Artificial Intelligence at jtc one and we will be talking more about that throughout the day. I will point out that as an organization, microsoft is looking at Artificial Intelligence and the role that it will play in a broader spectrum and recognize that standards are one element of the ways that the Technical Community will deal with interoperability. People also looking heavily at Opensource Software and the methods happening there were people are working on exchanges. And in the policy environment, i absolutely agree and support the need for thinking about accountability. And when you go down this path of accountability, and principles or any of the other countries around the world looking at this, we are also recognizing that the relationship between the regulatory approach and the accountability comes down to the criteria by which you measure. That criteria is going to be fundamentally predicated on the good standardization work being done. So i look forward to this panel and thank you again for joining us. Good morning. My name is anthony robbins. For the last 30 years, i have spent my career at the intersection of government and commercial industry. Most of which has been with Silicon Valley companies. In my role at nvidia, i am the person in the field that is trying to predict, guide and coach the federal governments progress as it relates to ai and make sure i do the right work to interpret the need from nvidia and commercial industries. So we were excited in february to see the executive order that was signed on ai. If you look at the executive order, you might convince yourself that it was the first time that an executive order had been signed on one of these Big Technology waves. These big waves being the work that we did with client servers early on to mobility, to cloud to ai. Ai is one thing different than the previous that came before and it is bigger than all of them combined. I will grab on to something. We do spend a lot of time as people and as leaders in this Community Talking about some of the challenges with ai and how secure and robust it is. And where the biased exists and things like that. I think it is really important for us as leaders and especially in this room and in this community, to spend a lot of time talking about the progress that is being made and the things that are being done to help mankind and improve the planet and make a significant contribution. There are children literally and i will say that for High School School kids making a profound impact on the world that we live in and the work that they are doing around ai. As much as we want to go act out of concern and we want to address standards and international and know that the role that they have with the strategies i think it is important for us as leaders to celebrate the amazing progress that is underway because if you look back into the history books i guess the federal government has been touching ai since the 50s and we had a major breakthrough since 2012. I am excited to be a part of this panel and make the contribution. Thank you. The rush to develop the standard is new but the recognition of the importance we have seen this cycle happened before with previous generations of technology. So what kind of precedent exists for how we will be approaching the standards. What other models can we look from where there are fundamentally new challenges that we have to address. I think if you look at historically the way that standards have developed, lets look at the computer era Information Age where we have a lot of technology being developed and we have a lot for the technology. Historically the worldclass ideas were coming from American Companies. Certainly we very much support the voluntary consensus open transparent industry led Standards Development process. They were coming from American Companies for the most part. The folks that were at the table conversing about technical standards were primarily American Companies so the process has worked out well but i think we cannot assume that because it has worked out well for American Companies in the past it will continue to work out while Going Forward. Now the landscape has changed. The economic competitiveness has changed. We have strategic competitors who are also recognizing the importance of technical standards. I think that the process that has worked well in the past, we want to foster that Going Forward but we have to recognize that we are in a new climate now. That Global Competitiveness requires us to be more intentionally proactive in promoting that process so that we can make sure that all of our good ideas are coming out of the United States and have an equal footing. We are not afraid of competition internationally but i think the standards process cant just presume in some sense the federal government broadly speaking is not recognizing the importance of standards because we presume that the process in the past continues to work well Going Forward. I think that the fact that the process has worked well for the computer innovations means we do not need to let our guard down and presume that we do not need to be more proactive promoting the open process of developing technical standards. I concur that if we use history as a guide you look back and say how is it that u. S. Industry in the sector has been so strong. Theres no question that one of the underpinnings has been a dynamic standardization of the environment. That has been created by keeping open all possibilities of the spectrum as it approaches everything from consortia to national processes to the International Processes. U. S. Industry has been adept at making use of that full spectrum depending on what it is that you are trying to get done and what time. I want to take a step back to my opening comments and recognize i agree that things have changed but they changed in a fundamentally different way than people are thinking about. Im going to mention again open source software. You will see a new factor that will play in how technical interoperability will be addressed. Engineers will move to a much more rapid pace of technical interoperability work. That does not then take away from the role that standardization is going to play but it means that things are fundamentally different not in a way that things will be less but it will be function of how engineers and contributors come to the table and played with innovation and ideas that bring about a foundation of understanding that the issues are transparent and you can understand what the technologies are and policies can be built. That is a misuse of standardization if you put up barriers. It will be a function of using standards in a way that they are strong and leading discussion but you do want to preserve that system as much as possible following the principles that have led to the great success. To me i think theres a great deal of merit looking at where the standardization has been but recognizing that there has been a fundamental change but its about who is racing to get the first standard. Those of us in industry will tell you that they can be marketmakers but the products come down to the aggregation of dozens if not hundreds of standards and it is really about the layering of heil value work above standardization that is about market success and so i would put it in a slightly different context. So try not to repeat anything that has been said, what have we learned about standards as they have come across. Most recently would be the work has occurred across where 160 Different Countries have rallied around different aspects of standardization. That is a pretty good model as we think about standard and scale. Where the model may break down a little bit is it is still not at the scale that we are talking about here in the complexity we are talking about with respect to ai. We are talking about nations that want to create immense value for themselves. We are talking about concerns with the application of ai so we have to deal with that. The other thing is as i mentioned in my opening comments that when we think about the society only think about the trust and security an important aspect of the adoption will be the improvement of societys belief and trust in the technology and its ability to be there for. Standards play a really Important Role. I think that nist is really important. There are a couple of things. It is important to the progress that we have made the other thing i would recommend is that it is not just about the standard itself. Its about the use cases for the federal government. On the civilian agency side we think about waste and fraud and abuse there are Different Things we may consider out of the data and citizen privacy then maybe on the department of defense. Or how we think about Cyber Security. So as one of the reports that came out the benchmarks and standards and prototypes i think it is important that we get started and build some prototypes and Lessons Learned that relate to how the federal government might adopt and deploy ai because it may inform some learning and the position on standards. I want to pick up on a team that all of you addressed on this idea about u. S. Leadership and standards of development. Its one of those issues that we talk about folks in industries than in the International StatusBody Community that these organizations exist they are sending huge amounts of delegates that are coordinated. It is a very clear specific goals in mind where the u. S. Approach is different we have industry representatives and we are not trying to champion any particular company but create a competitive and fair open standard. The concern we keep hearing about is that china is much more effective about potentially ship in the scales if they choose to to favor them in an anticompetitive way as they have done in the past so what is the solution here for the United States and how can industry and government Work Together to kind of create a fair and even playing field. I am happy to start but im sure everybody has good opinions on this. It seems to be we are honorable today. I think that there is zero evidence that the chinese have an unfair advantage in the International Standard system. The reason that i say that is because the principles by which most of the Standards Organization that we are talking about follow very explicitly prevent or have rules in place to diminish dominance. I speak about this as a large corporate player. I recognize the dynamics that are playing these bodies. As industry we would far rather have them involved in the International System where they are engaged in one country one vote dynamics or in an environment where they need to bring ideas to the table like the germans and americans and japanese and argue them out in the community rather than put them behind a wall in the country and use things like one belt one road or trade agreements to impose their system on others as part of financial engagements. Those outcomes will be far worse for us than encouraging all countries to be at the table in an international dynamic where you have rules that are in place to protect the minority voice. I recognize that there was a big discussion in the country around 5g right now. I want to broaden it out and say that i have people involved in a wide array of discussions and engineers. There is very little evidence that the contributions carry more weight than anybody elses or get adopted. They have bounties in the system for making contributions. We have seen wikipedia articles cut and paste and submitted in the standard bodies in order to get the payment back home. That is not the path to success in the standards body. The chinese are learning and coming to the table. If we take that example which is such an immediate today they are hiring real, decent, strong professionals from all around the globe so their ideas can be heard and listened to and engaged with. There are certainly scenarios where people have been concerned about certain technical elements. I am not here to be an apologist for that. What i am saying is we should think about and protect that which has been incredibly important to the Economic Growth as a country which has been a standardization system that is fundamentally predicated on these principles and have led to an incredibly strong outcome over a long period of time. To the extent that it is easy to point out one player and say im concerned about them, i do not believe that the standardization system is the place for that. Investments in r d and expanded intentionality on National Security concerns much like we are seeing over the past week those are government mechanisms that have no place for me to make comments on but i would say that this system itself will boomerang as an industry if we start using it to beat other countries over the head with it. I just want to clarify. You are absolutely right i was not particularly clear my question. We are not seen evidence that that is happening yet. The concern is that it could happen and we might not be well equipped to respond. Im not trying to cast any dispersion on anyone. I completely agree that the objective of having the federal government more engaged in federal standards is not to change the process. As i said a moment ago it is absolutely did foster the consensus driven approach. In the Cyber Security area i have read multiple reports i am not an expert so i am sure that others cant can comment more on that. The challenges over the International Standard setting process as it relates specifically to china is the internal standards for anybody that wants as an industry to operate in that domain so they have to operate under standards that are set by the local nation and not based on International Technical standards. The challenge to industries outside of that nation is that now you have to make two technical standards. If nations do not want to go along with that in a very large market. That is one of the concerns. In some sense this may be a political question of how do you encourage all nations of the world to abide by and participate in this International Standard setting approach which has worked so well. If you believe in the prospects of ai for good then every country has something to contribute. That would be the u. S. Just like china. In the case of china it is a very big and powerful country and they have made very broad commitments and sometimes people worry about that. The development where we are right now this is a global team sport which will require Companies Like microsoft and nvidia. It will require Higher Education institutions around the world. It will require those that are part of the Industrial Base so there is a lot of work as i mentioned earlier theres actually few ai standards at the moment and many of the thinking is preliminary. We need to be careful not to cast all this fear and uncertainty and doubt on the prospects of a i forget him we have to let this innovation cycle go and take hold. It will get the point of the matter is that we have a bigger Human Resources challenge today than we ever did in cyber. It is getting bigger and faster so we require global participation i will paraphrase in part some conversation that i had with the senior government leader and she said she was actually less worried about any particular country and more worried about the United States. I think this nation and this country has done extraordinary things for this planet and of course for our country. I think if we are focused critically on our ability to contribute we will be just fine on the global stage. So picking up on something you just mentioned talking about the standards themselves and how much work there is to do here. . How mature are the standards today. Is there a lot more work to be done or are there particular areas where we have seen some early success that we should be looking to as we flush out the entire system. We are really early. Of course there has been several meetings and we have mentioned sc 42 and there are four subcommittees that are part of that and there is work that is going on here in the u. S. Are we have done some work around benchmarking which is an area that people wanted some kind of standardization around. There is some open source tools and processes relative to share things. Then there is the set of meanings that have occurred on standards around the globe. The mist and civilian agencies are part of that but it is really early. Both the committees and the leadership and the roles and the like. I will say it is really important that i think we could be more active in the work that has occurred thus far that we are today. Whether it is nist on behalf of the federal government and ecosystem partners my own company included i think we could be more assertive on that. I also concur that we are on the front end of a lot of work we will also take a moment and look back as we talked about something critical like sharing data sets and all of those discussions there are dozens of critical existing standards around data and the format interchanges that will be critical to the work of ai. There are standards around computing that will also be critical. You need to recognize that is the inventory what is important for a i dont just throw out the baby with the bathwater. Theres a number of things that are essential to that next step that we will talk about. You then have new work and we are at the front end of that. I would say the definitional work right now is the place where nist is already active and the government is engaged with industry and we encourage that to continue rather than doing something repetitive and going after that same process they will participate and stay engaged where the experts are already working. Then we talked about open source. It is really interesting to see what is happening there. Onyx is a great example to the Exchange Format but cronus is also hosting an nef. Competition is good. You will see a different format going after Different Exchange ideas that are also going to show up posted by consortium or foundation that might crop up for they will be in competition not only with each other but with a more formalized that my come through. The market has a funny way of rearing its head and saying i like this and i dont like that. It is a critical part of the process as well. We have a huge existing swath of work that is really important. There is always this question about the timing. When is the right time to push for a standard . Ai is still very new but people are concerned about issues of governance and oversight you cant govern and oversee if you do not have good ways of measuring. Thats a good area for digging into technical standards. If you standardize to really you are missing out on good ideas because it is still new. It can also really accelerate projects in the field. I have done Robotics Research for my whole career. The open source standard is when the operating system created not only a standard in the community for how you begin building up robot systems if you want to build a sensor interpretation system it allowed the community to come together and interchange all of those parts that the community had built so they can Work Together to accelerate the field. I think very critically defined areas of standards work can accelerate. That is the key karstens questions. Had a we prioritize that that will be most impactful to the development. We have not talked about that but it is worth noting and thinking about. When we think about these big waves of technology whether it is the clients or the mobility cloud or ai there is one thing that is distinctly different or unique about what is occurring. There are many but one is the democratization of the access to the technology. When you think about Cloud Computing there arent that many worldclass Cloud Computing companies. What is there . 10 that dominate the cloud. In the case of developers theres millions. Theres going to be a lot more. The access to the technology is something that i think the world would benefit from. It is something to Pay Attention to because the development is very different than some of the Big Technology ways that have come before i want to get back to something that i think is really important. There is different kinds of standards. Theres Different Things that get standardized in different ways. There are foundational standards that you can go after those things can become the basis for Public Policy. They are just simply what does this word mean and how will you use it and can everybody agree with the same use of it. You get to technical interoperability or technical standards. Robotics is a great example. That could be de facto by nature of adoption. Enough people are using it and it becomes the means by which common practices are set. Then we get into this new frame for the i. T. Industry in particular. There has been this push toward Management Practices and governance practices that get standardized for the purpose of audit. I can see gordon sitting out here and the work that they have done to help people start to understand the cascade of conformity and conformity assessment for an industry that is moving to become more regulated. We have a need to be able to declare what it is that the criteria is important. If we will talk about trustworthy what do you mean . We can talk about algorithm transparency if you want but if we crack it open you will not understand it. Even the people that wrote it will not understand. You have to talk about transparency rather than in a different way people move to explain ability. They talk about what is the source data and how did you establish appropriate methodologies. Afterwards can you describe what the outcome was and how do you track that and measure that. Those are not going to be written in code. They will write out the behaviors and that is a standard. It does not get standardized but if you take a Large Organization and start putting in place behavioral requirements those become standards that have a market affect on supply chains or large. I think that it really behooves us to keep in mind that it will be a much broader picture than whether or not there is a protocol for big data access or some sort of magic protocol. I want to pick up on that thread and something that was mentioned about the concern around the use of standards for things like transparency or safety. You most often hear it described as ethical ai. Do we know what technical standards or ethics look like . Is it more useful to think about it in discrete component parts. Is this a meaningless phrase . I understand if nobody wants to jump on that one. My team is tired of hearing me on this but i will share with you since they are not here to stop me. If you talk to an engineer about utilitarian ethic or different ethical models they will look at you really blankly because it is not something that they have studied. We have a need to have people who are truly trained and people who have a basis for the discussion and i have a strong opinion that there are things that the Engineering Community absolutely candy. You can describe in the context the systems and you can say take Something Like bias. Statistically you can map out in the statistics models that there is bias in all of the data. It might have to do with the race of the individual but it also might simply be we were collecting data before hospitals and he collected more from that one. There are things that engineers can talk about to address in the description of how you think about bias and systems. The fairness of that bias needs to be talked about by not the Engineering Community but by people who are deeply steeped in those issues and then the reaction from the community is one of responsibility for what gets built and how they build it. We have to find the joining of the conversation to say that its not going to be some sort of joining of the competition. I guarantee that the french see it differently. You do not want to have a standard that supports the ethic because he will get ignored by everybody. Good engineering practices that support the application of strong ethics to matter where you are applying them is a different discussion. I might add that all of the engineering teams that are good are having the conversation. All customer meetings that have rich Technical Depth to it this is a part of the conversation. I think part of it is acknowledged that this is an important thing. If you follow the government theres all kinds of work going on as it relates to ethics. I know that the director of the Defense Innovation board and he has been conducting listening sessions. They are listening to the community or to companies or innovators and the like. Just like most aspects we are still early. I will say the u. S. Is playing a role here but the conversation occurs every day across many aspects of the business at the intersection of customers in these cases. I will add to that. These challenges also go into areas like safety. If you think about the safety of a healthcare device that is ai based you have to perhaps involve this Multidisciplinary Team that includes Healthcare Providers that can include what it really means to be safe. Its not just about the engineers but it is really a multidisciplinary activity across the board for all of the different applications and that is the real challenge i think is often getting these communities to understand each other. I cant tell you how many meetings we had people come in that were trained in one discipline and they were talking people are trained in another discipline and they seem to have no understanding of what is happening with each other. So this gets into the Educational Area where we have to begin training ourselves to have more of an interdisciplinary focus so that we are not just a Computer Scientist so that we have both combined in multiple areas. Does this mean that my son who is taking liberal arts will have a job . As long as he takes some Computer Science classes . We talked about the industry is developing standards and i want to build on that a little bit. Standards are like toothbrushes. Everybody wants to use their own. I understand why a company would want to use their own standards. We know that the mutual adoption standards are better for competition and consumers and innovation. Can you describe how industry is playing nice here. What are the incentives to cooperate . Then i would like to hear a little bit about what the federal governments role is or what you hope the role should be to kind of foster this cooperation. I want to hear from you. Standardization is by definition collaborative and competitive at the same time. Everyone who walks into the room is doing so on purpose. Nobody is ever tripped and fallen down and made a standard. You show up with contributions or as a participant and overtime People Choose to implement. We would always advocate for a rich competitive environment both of standards bodies and participants. That competition drives the best possible outcome which are those things that moved to the top. Those have been involved in things that are successful and that is part of the process. Sometimes those dynamics that play out have nonlinear pressures on them so you can look at vhs and betamax as the example of the alaska technical standard one out because of other factors. We want that richness. That is exactly what has propelled the Technology Industry to be so strong but i will take a step further and recognize that the technologies we are talking about now are not about the Technology Industry at all. This is about ubiquitous technologies that affect every sector and every sector will turn on it and do their own things. Equipment manufacturers will have processes that they will deal with and healthcare folks and automotive folks will do it and the way that we break out of the recently espoused opinions that there will be a winner take all model we reject that notion. We reject it categorically because what it really comes down to is can the competitive nature of Industries Come to bear but also bring to bear all of their resources and pull resources around data and do shared learning. That will be a new model for Many Organizations but standards at the vertical level and the horizontal level will assist them at and that is the way that you break out of this concept. On the International Perspective the standards will exist. They will take some time to develop and most countries a consequence will participate gainfully. As i said this is one of those technologies we hope they will be affected by a i forget. If you believe in the prospects of a i forget i think that the world either does or wants to believe in that case. You are open to the notion the federal government plays an Important Role. We have talked about Global Competitiveness the contribution that nist has made. I am certainly hopeful and confident that we get there if there is this exertion over the federal government this International Standard thing breaks. One of the things it provides us the opportunity to convene people. They make request of the federal government. That is the point of this executive order is to find out from you is the federal government is, what are areas the federal government can be more engaged. Clearly we are very fortunate to have missed here who has been an honest broker. The Government Works or further the industry goals of standardization. The Important Role of federal government is to listen and act on the information that we get from this community on how we can help more. We have just under 15 minutes left. It looks like there are two microphones on the side. Can you say who you are asking a question and do not do the terrible thing of making a statement rather than a question. We have technical standards and standards of responsibility and ethics. And explainable ai. You could definitely standardize things like algorithms and the level of explain ability. I am kind of curious as the standardization of Different Levels of explain ability. Im thinking about things like the standards of accessibility for standards on accessibility i think one Important Role is the rmd that informs. It is a perfect example because the technical approaches to achieving the kind of explain ability that we want do not exist. Its not that nist is doing that r d but it is an Important Role so i think it gets to the question of timing and whether or not we know enough to set up a standard. We can begin look at the studies that need to be done i do not think we should talk about standardizing innovation for one of the parts that it is trying to take his tools. There is fundamental research that will come around explain ability dont necessarily exist. It is about Building Tools that help you or creates an opportunity for other technical standards that come from that understanding. I would be really careful about using the term standardizing the algorithm. Theres level of explain ability. I appreciate the response. We actually have until 10 45. We have plenty of time for questions and then the moderator will post a couple of closing questions. So the doctor mentioned what is happening right now with the privacy framework. They have the standard. Can each of you sort of address the appropriateness of the developing getting into that framework. So i am not familiar with those conversations. The perfect terminology for standards perfectly privacy clearly there is an important connection mayor. There are many issues with data at lunch that are not necessarily only ai issues. I cannot speak to what the activities are in terms of connecting them. The privacy discussion is one of prescript of light prescriptive regulations. If you are working on a i. What we have to do theres a really interesting duration of data sets. The responsible behaviors or use of data. Those become the measuring stick by which will be held accountable. We have to factor that in and now that you have the laws they need to be held to the same standards. They underpin the responsibility they give you some peace the puzzle. If you refer back to the oecd principles the second one that they talk about is kind of the role of law. You be careful not to bring those into standards to really. The National Standards institute. I wanted to get this panels perspective on engagement and tools and activities. The federal agency is engaged on the Mission Space which i am sure that you all know. They support the decision regarding regulation or the procurement agency. This is unique in the space that drives the wide range of development activities. You will see this technical staff ranging from open source to voluntary consensus in the whole range. So just a few thoughts from the fan on the members about the different types of level consideration. So from an industry perspective and being on the outside looking in i think you touched on it there are different aspects of standards when we think about what the Intelligence Community might deal and what the department of defense might do or what civilian agencies do. That is why i recommended earlier that i think we have to look at some of these really big use cases and we have to create some environments where we can test these use cases out. Because what is occurring in Cyber Security relates to the federal government. This in 2016 was the last time that is a big challenge to go after and the way that the standards relate to that might be different than the way that we worry about platform sustainment for the department of defense where the issue may be more data rights. There may be interesting value to be gleaned by looking at use cases associated with different aspects of the federal government. One of the challenges we have is an education challenge and an understanding of technical standards on the Technical Missions on all the agencies. We looked back at the testimony i believe it was walt and looked at some of the comments he had made of why it is to the benefit of the u. S. Government to have it with the vibrant standards of development process. You think of all the things that they purchase you can imagine that there are a lot of standards but it is an agency that does a lot of acquisition. It provides incentives and opportunities it encourages longterm growth for u. S. Enterprises for the alliance of the u. S. Government for Cost Effective goods. Those are the agencies they should want to experience. There is not a broad understanding of how the standards can feed into that. These are advantages that are appropriate. At a minimal level. They will help raise awareness across the agency the engagement could contribute significantly to furthering the standards of the development which benefits their own agency. It is always a challenge that people that are busy with the here and now but the longterm also pays off. That is an important part and the executive order is to raise awareness across all of the agencies. I want to emphasize the importance of the executive order. This document is a central the standardization is the cornerstone of the system. You mentioned before convening discussions like this and the central role the government can play and participate. The agencies have real needs an understanding of how they want to deliver Government Services how they are working with those and how they are establishing policies and regulations and how are they supporting usa interest. You dont pick winners in the marketplace. Secondly dont use standards as a trade area. Did that answer your question . I want to touch on one of the points you made and you have spoken about the fundamental question is it something new completely or dependent on the capabilities. It is the capability that they already have with the Risk Management testing development. They look at bolstering the capabilities that you have. That pushes us in the direction. They have to explain ability and Security Privacy the whole set of those. I think the equation changes dramatically i think it goes back to this conversation back to the points you made. I think it is really started by us saying what is it . Is it something completely new . Is it really a some of a number of parts that already exist. That is obviously a part of the discussion in the workshops. Is it new . No. The federal government has been on record doing this work for 60 or 70 years. I think 1957 is the earliest data we have seen. What is new and different is the amount of developers and the access to the technology. It used to be 40 years ago or 30 years ago that you would need supercomputers that took up Old Buildings and phds to do the kind of work that was required for these Neural Networks and the like. Today people are doing these on commodity. It is kind of like the democratization of a i has brought the world into potential developers and has created the application and all kinds of use cases for good and otherwise. To build on that i think the fact that we have accelerated to the point that we do have a lot of use cases that brings the question of what is under the curtain and what is in the black box. Some of the techniques, particularly the deep learning techniques that are popular today and contributed to these cases been successful now, that technology is the black box. That is because we are using it very frequently. That raises the questions about is it being used appropriately. A lot of the techniques to verify the system based on the underlying mathematical models are not applicable for a variety of reasons. That is kind of the new piece of the application that we dont fully understand. I agree. There are a lot of things that exist. They could fundamentally change the game. Steve harrison from the department of defense. I had a question to be interested in the panels you and the role and type of standards to promote aggregated unofficial intelligence and intellect. Of somebody who has been doing heterogeneous robot interaction you have different kinds of systems and you want to build a largescale system that can interoperate and talk to each other clearly standards are pretty important because otherwise the systems do not know how to talk to each other. I think you can do that through one off kind of implementations but it does not scale well. You can add new kinds of systems into the largescale distributed team. In order to have interoperable capabilities where you are mixing and matching amount components the standards are very important. This may not come out of the standard body. It may come out of the github project. We could use a normalized approach but it may not be coming out of the governance. We have about 10 minutes left so please keep your questions and answers sort short. In the spirit of algorithmic fairness i saw him first. My name is calling im with the National Center security for excellence. I have a question about adversarial machine learning. It seems like an area that is ripe for these kinds of efforts we are talking about. It is really a concern of these critical systems. I wonder what efforts are these Companies Like microsoft and nvidia taken to mitigate these sorts of risks. The applications that could be developed that could encourage safety minded ai development. I will answer briefly because i am not an expert but what i will say is we have active Research Going on in the space. The starting point frankly has been modeling and understanding even with the picture looks like before you get to actions you can take. The other step is to start looking at your Development Practices and start putting in place mike in the past you would look at the Security Development and make sure that we are training the workforce appropriately of the things that we are doing but i will stop there because you have put your finger on an incredibly important point and one that is needing a great deal more research and consideration. I will say the same thing. Check will have a National Initiative around cyber. If you look at what is happening around cyber come even before he was confirmed they talked about the importance and role of a i. This is a brave new field. It is really interesting because when we mentioned that 160 countries have standards relative to cyber. The world is about to change. Something promoting the change is 5g anymore. We go from hundreds of millions to billions. This is a giant challenge. We have been working on actually developing some of our own tools. That will allow us to get to these Big Data Analytics things that relates to the cyber. We are still very early and thats a the best contribution from the nvidia perspective is how we intersect the federal government. There is ideas to do that. A lot of work still to do. One of the areas are frequently asked about. What areas that we can work on the International Level. This issue of how to build safe and secure a is an issue that has international importance. We want them to have safe and secure. This is an area that you have engagement. One of the things are highlighted as being important. The project. I agree with the statement very much. How people working in the open source sent the parties to contribute to the process. Then conversely, how the federal government can incur the government private and public sector. The federal government. They did actually find some of the development of the open source work because it is just built by the community. The federal government does try to provide some resources for these types of open source projects that are clearly having a broad impact. The people that use the words, the community. If you look at the most significant projects that are heavily funded, there is a lot of deep research organizations. It is going to come down to each project. Different foundations. Something like that they can put in place their own governments practices relative to project. That is a choice they have to make. People come into the table. Does not have that sense of responsibility. I cannot underestimate the importance that it is going to play relative to the nature of Integration Development and interoperability within the context of future emerging technologies. That does not mean just like we talk. People are not going to be able to say, i do not have to follow it. There not want to have appetite for that. The california law. A do not follow it because it was open. They had to take on responsibilities and everybody who is acting in the community. Is like we have to make questions let. Im here in maryland. My first question, this question is actually for jason. Munich elaborate the side the Standard Organization to minimize resources. What is there progress on creating ai standards. What is missing. My other question. We have used the word standards. It is an interoperable compliance. My question, given all these very kinds of standards, would you prioritize in the plan . How would you prioritize our work and effort. Our investment in those different types of areas. Which is that we should just try to approach the mall. What levels are we in terms of ai. I will translate quickly. The International Process is inherently collaborative. Is also important to note that it is a process that leads to must. Know what he says here is my standard and everybody has to listen to me. It is something where the debate goes back and forth. The earth of the people who push hard for those ideas. The process is designed to encourage that collaboration. Everybody has a agenda. Participation is essential. You people in the room who are experts. I will let you take this question. For me it is all about. Looking at whatever the activity is. What could it lead to. To help us economically. To help with lots of different kinds of applications. The type of the standardization process or approach. I think it is more of what will impact so we can move the software forward. I would just say, just acknowledged what is going on. The train has left the station. This is the biggest technological transformation. The most important work we have to do in the business is related to this. You look at what is going on in the Intelligence Community. The executive order. The federal government has indicated financial commitment. I will challenge us on how we can play a role as leaders in helping this thing go faster. That is a really important aspect. I know that is a overused turn. We need to go faster as a fellow government and American Public. We have the tools to do that. Effective leadership will be important to his barn that journey. Before we turn over, i want to thank them for hosting us. This will be available online. Please feel free to visit our website. We are doing a lot of work. Again, thank you again to our families. The California DemocraticCongress Talks about her memoir. Surviving johnson. Some encourage and fighting back. When people say it was an accident, it was not a massive suicide. They were forced to drink this toxic brew