Piloting Accessible Voting Technology in the Field A NIST/EAC Sponsored Accessible Voting Technology Initiative Webinar January 10, 2014, 1:00 – 2:30 PM ET Operator: Ladies and gentlemen thank you for standing by. Welcome to the Piloting Accessible Voting Technology in the Field Webinar. During the presentation all participants will be in a listen only mode. Afterwards we will conduct a question and answer session. At that time, if you have a question, you'd press the 1 followed by the 4 on your telephone. If you would like to ask a question during the presentation, please use the chat feature located in the lower left corner of your screen. If you need to reach an operator at any time, you may press the Star followed by the 0. As a reminder this conference is being recorded today, Friday, January 10, 2014. I would now like to turn the conference over to Shaneé Dawkins. Please go ahead, madam. Shaneé Dawkins: Hello everyone and thank you for joining us for our first webinar from the AVTI grant from EAC. And I want to welcome you all. And for our agenda, first, I will start out. A little click - there we go. And then I'll turn it over to Pat Leahy from the EAC. And we have three wonderful presenters today. We have Juan Gilbert out of Clemson. We have Sarah Swierenga out of Michigan State and Dana Chisnell of Center for Civic Design. We will follow up with Q and A. And I will give some closing remarks. So first we have Pat the wonderful advisor from the EAC over the AVTI grant. And he has been great in working with us and getting the word out and really commanding the research. And he his dog - I'm sure, with him (Galahad). So Pat, if you could go ahead. Pat Leahy: Thanks a lot Shaneé. Hi this is Pat Leahy. I work over at Election Assistance Commission as Senior Advisor. And this has been a great project to be involved with. We see AVTI a lot so Assessable Voting Technology Initiative. And what that is, back in 2010 congress appropriated 8 million dollars to Election Assistance Commission to come up with a grant program that would improve voting assess for all Americans. And we put together a package in a grant proposal and a way to get that out to the public and received a lot of great impute and great proposals. Ultimately Clemson entity led by Clemson University and Dr. Juan Gilbert won one of the intermediately grants and ITIF Information Technology and Innovation Foundation. And that side of the grant was led by Daniel Castro. Both organizations have just done a fantastic job over the past two and a half years of leading their projects. So now where we are is at the stage of getting this information out there really talking about the pilots that are possible and the pilots that have been done - also talking about the results of the grant. And what are the results of all of this work. My background is - on the hill - I helped (unintelligible) HAVA - - Help America Vote Act back in 2002. And just gave me a great experience to see this bill and to see the landscape of elections across the country and what we need in the area of accessibility. So we've come a long way. I'd like to say we've come a long way for accessibility. But we have a ways to go. And this grant helps in that area. So for accessible voting technology initiative Dr. Gilbert's team at Clemson and Daniel's team over at ITIF have put together over 45 initiatives to improve voting access. Some of them are complete with their reports and research and development. And then some are still kind of in process. So I think what you'll hear about today is the great pilot work that Dr. Gilbert's done across the country. You'll hear about from folks like Dana and Sara about the possibility of pilots and how they can get them out there and how they would like to - kind of moving forward. And the last part I'd like to mention is that as we go through today and talk, one of the big parts of this grant have been we want it to be collaborative. We want to hear from you - - folks in the field, state election officials, folks in the disability community just anything dealing with universal design and access and how we can improve the process. So please feel free to, you know, ask questions today. Reach out to me. Reach out to the folks at NIST, the grantees - just anyone that you would like to contact. We're all very accessible and would love to hear from you. So it has been a great experience to work with the folks over NIST with Sharon and Shaneé and Daniel and Juan with the grantees. So looking forward to today's call so we can begin to share the results and talk about the piloting. Shaneé Dawkins: Great first up, we have Dr. Juan Gilbert. I'll have - all of you know - full disclosure. He is my former advisor when I was at Auburn University getting my PhD. And I also worked on the Prime III System. Juan Gilbert is the Presidential Endowed Professor and Chair of the Human Center Computing Division in the school at competing at Clemson University where he leads the HCC lab which is the Human Center Computing. He is also the primary investigator on a U.S. EAC Assessable Voting Technology Research Grant titled the Research Alliance for Accessible Voting. Juan. Dr. Juan Gilbert: All right, thank you. All right so I'm going to talk about the piloting that we've done and our project which is Prime III and how we're moving forward with a lot of those pilot projects. So as then noted, we received one of the accessible voting technology grants from the EAC. And as part of that we were task with doing research about making voting more accessible - and again, for everyone. And we were also tasked with researching, innovations and technologies in this space. With that said, we wanted to also do pilots. So if you do research on these technologies, innovations and then pilot them and report how well they work and things like that. So a list of our partners that form the correlation - - the Research Alliance for Accessible Voting are raised. And also a list of the institutions, records, Carnegie Mellon and Silicon Valley - these were the partners that helped start the grant. So before I can talk about the pilot, I need to tell you about what we actually use in the pilot. So I begin with Prime III which stands for Premier Third Generation Voting System. And Prime III is our technology that we created that started in 2003 at Auburn University. It's been over a decade of research and development and testing that we've done. So Prime III is a software based research platform that we created. And it has a multimodal user interface. And the goal was to create a universally designed system. Back in 2003 when we started working on this and telling people about our ideas, conventional wisdom was that you had to have an assessable voting machine for people with disabilities. And then everybody else voted on a separate machine or a different method. And we said we wanted to build one machine that everyone voted on independent of their ability or disability. So that's where the multimodal user interface comes in. On the visual side we have a touch screen where large fonts. We could put images - many pictures of candidates on there or not. We've done some of that. And we can talk about that later. Have names - you touch names. It's a touch screen. And you have one contest per screen. And the voter confirms. So that's a visual aspect of it. And again, we're software based. So we run on different devices. Prime III is developed primarily from an HTML5 perspective. So it can run - it runs in browsers - - phones or tablets or laptops or whatever. We also have a switch as you can see here it's a two button switch. On the right hand side it says - goes next. And on the left hand side is select. So if I have a voter that's using a switch that goes from prompt to prompt to prompt and select, prompt to prompt select things like that. We have a headset with a microphone. The headset allows the text to speech to be spoken back to the voter. And then the microphone is used for the voter to respond. So we'll say, for example, to vote for Pat Leahy, say vote. And the voter can respond by saying vote or the voter can respond by simply blowing into the microphone. So it's a 1.5 second window that the system is waiting for you to respond and then you can make a selection that way. So you can use your voice, the switch or touching the screen. So we created this technology. What we discovered is that people that couldn't see - - people that can't hear, people who can't read, people without arms, - - could all privately and independently vote using this method - this multimodal interface. So they're voting on the same machine as anyone else. So we have one machine that everybody votes on that same machine. So naturally, people ask about security. So let me tell you about that briefly before I get into the pilot. We have what we call a voter verified ballot. Especially when you start with Prime III - you walk up to the machine and the assistant or poll worker - so it's a single sheet of paper in a printer. You vote. And then it prints out your ballot. And when it does that it prints a ballot that looks like this. So that ballots has just the contest and the individual or individuals that you selected. And you can see a picture there where we can put pictures or not. The pictures have been useful. And I'll talk more about that a little later - the implications of pictures. But essentially you can see here in this ballot. It eliminates ambiguities with respect with who the - what the voter's intent is because it clearly shows those selections. Now in the lower right hand corner you'll see a number 1889. That's not printed on the ballot. That actually is placed on the ballot after it goes in the ballot box. That's one thing that we've done. You don't have to do that. But we do it. Where you collect the ballot - they go in the ballot box. Then you take them out. And we actually put a unique number on each ballot. And then those ballots are - then those ballots are scanned. And our OCR Optical Character Recognition scan it. So essentially, what happens is a picture is taken of each ballot. And we have software that looks at those pictures. And then can read them and tally. So it was a challenge the election and we had these numbers on the ballot. So we can do what we call a ballot level audit. Meaning we can go pull ballot number 1889 and actually match it - match the image to the physical ballot as well. So essentially that's how - in a nut shell - there's more features of Prime III. But essentially that's an overview of Prime III. I'm more than happy to give you more detailed information about what we're doing and how it works. But let me talk about some of the pilots. So since 2008, the National Society of Black Engineers - which is the world's largest student run organization - has used Prime III for their national election. And so that organization - as a group of college students - very few students with disabilities were participants in help with election. And they had large contest. In that particular pilot there's a student election. But given the size of the organization, they had multiple contests. On the order of ten or more contest for - when I say - I mean election. So there's 10 to 12 ballots and 4 or 500 people voting. There's a lot of moving parts. So that helped us in managing our ballot styles and things like that. I will say - I didn't put this one on here. It's not really a pilot. But it was a study. We actually started with Prime III our first - the first people who tested Prime III went to the Alabama Institute for the Deaf and Blind. And we went there and let their population use Prime III. They were the first people to actually use Prime III. National Counsel Independent Living - we did their election - two years. I can't remember which years we did it. But it was in D.C. So we did pilots there. We did a presidential mock election at Auburn University. That was in I think 2008. And then we did the State of Oregon in May of 2012 use Prime III in a presidential primary. So that was the first election that we had done that was not a student or mock election. And that election - as many of you are probably thinking - well Oregon is a mail in state. Yes that's true. But what they did - they took some tablets. In this case it was Aces tablets with using Prime III to rehab centers and nursing homes. And they let people vote using Prime III which resulted in a ballot. And that ballot was then used to fill out a mail in ballot. And that mail in ballot was the ballot of record. So that's how Oregon used it. To date Oregon has our software. And they're using it for something. But I can't remember. But they're still using the Prime III software in some context. Then in 2012, self-advocate is becoming empowered - this was a very important election for us. SABE is an organization of people with cognitive disabilities. And we had about 270 people vote. And in this election, all the candies had pictures as you saw on the previous example. So we didn't - we knew that the voters were at different reading levels. But we didn't know who and what level that individual was at. In this election we didn't' have any problems with anyone reading the ballot. We did notice that people when they verify their ballot, they would say yes that's him or her. They we're referring to the pictures. So they were using the pictures to actually vote. And so we - that was our hypothesis. But we didn't have any evidence of it. So we came back to Clemson a week or two later. We went to Clemson Elementary school. And we let the elementary school do a presidential mock election. And we had voters that we knew could not read. And we had the pictures on the ballots. And they voted. So what was fascinating about the pictures is that they were enabling people with different reading levels to participate and not exposing their reading levels. Okay, So on April 1st - this is probably our biggest pilot. On April 1st, we'll be in the State of Wisconsin doing an election there in Sheboygan County. We will have three - we'll be in three voting places - set up for Prime III. People will vote on Prime III - will tally with our optical character recognition. We'll do the whole election in those three locations. And so this particular pilot is going to probably be our biggest and most visible pilot. And, you know, again, we're inviting people to come and witness that pilot. So let me talk about challenges and what we've done. Some of the biggest challenges were with election officials. In many cases, they were unwilling participate. And it takes a lot of work to convince some of them. And that's understandable because it's additional work to do a pilot. And it's risky. So we're going to do this pilot. What if it doesn't work? And we have issues. How do we resolve them? It's an extra burden on election officials. So it took a lot of convincing for years to get election officials to warm up to us. Now I will admit the way that we were able to do this is we had really good friends - starting with Doug Lewis in the election center. So the election center helped us with that communication with election officials and gave us opportunities to meet with election officials to actually get in and do these pilots. So that helped us out a lot. And they're a partner on the RAAV grant. I would say that vendors - this was another challenge for us. We were often thought of as competition. Vendors would say are you our competition. They didn't know who we were and what we did. They immediately thought we were trying to, you know, steal their cliental or something. From our perspective, we are a research entity. We are not vendors. Prime III is a research and development system use to and the open market. We'll give you our code. We'll show you what we do. We're not going to be a manufacturer of voting systems. However, we are here to show you the best practices on how these things work. So show you how to do universal design. And if you look at the machines that are the next generation coming out, you'll see universal design and universal acceptable machines being produced. And so we are having some effect on what's the next generation. But we are not vendors. So vendors were a little taken back by us at first. But I think we've gotten to the point where they warmed up to us. And that hurdle we've kind of gotten over it. Then participate - so if you're going to do a universally designed technology, you need to test it across a large diverse group of participants. And that is all range of abilities, disabilities, ages, source of demographics. So we've been doing this for - like I said - over ten years. And the only demographic that we - to date have not been able to use Prime III is deaf, blind. Now we recently attended a workshop - - the National Federation for the Blind - - where we been introduced to some refreshable brail interfaces. And we're looking to integrate that with Prime III in the future. So but that the challenge is, how do you get participation from a broad audience to gain confidence that you cover various levels of ability or disability? So I'll stop there. And that's what we've done. And I'm more than happy to take questions, I guess, at the end or now - whatever, Shaneé says. Shaneé Dawkins: We do have one quick question from Kathryn Summers. I believe it's in regards to your Prime III system. She said, "For disabled voters, how does the paper get to the scanner unless they request assistance?" Dr. Juan Gilbert: In March or April, we're going to release a 100% hands free design on how to do that. Right now, there has to be assistance to get the paper to the ballot box. But we have an implementation a design that we're going to release that will only require the poll working to put the piece of paper in the printer. And then after that, no one touches the paper again until they take it out of the ballot box. Shaneé Dawkins: Another quick question from Alicia Kundtz. "In terms of using images for voting, have you looked at the studies about using photographs and the image of the voter having an effect of an election?" Dr. Juan Gilbert: Yes, well we haven't found any data that goes either way on that. We've been asked that a lot. What I would do is I could solicit - we've spent a ton of time trying to figure out how to design a study to get at that question. And the problem is that people don't want to expose the biases. And it's hard to really determine if I put up a particular face, will that create a bias for or against an individual based on how it looks. And studding that in an election context is very difficult - we - because the bias is very difficult to measure. That - we don't know exactly how to capture that. So if someone has input on that, I'd be more than happy to receive your comments - if you have an idea. How would you conduct a study to identify if pictures create bias verses not - non pictures. Shaneé Dawkins: Okay and we have one last question before we move on to the next presentation. If anyone else has any other questions, please save them for the end. From Sanford Morganstein or (Morganstein), "In your pilot elections, do these elections count?" Dr. Juan Gilbert: In Oregon it counted. And in Wisconsin it will count. Those are the binding elections that are non-student orient. But the student and the other elections - yes they counted as well. Every one of those counted for that organization that did it. The Alabama Institute for the Deaf and Blind that was a study. So that one did not count per say. But all the others were binding elections. Shaneé Dawkins: Okay thanks a lot. And as I said before, he will be available at the end of the webinar for any further questions. So up next we have Sarah Swierenga. She is the Director of Usability, Accessibility Research in Consulting Lab at Michigan State University. She leads several user center designed research grants and assessable voting system, outdoor recreation knowledge management and healthcare IT environment. Her lab also does user experience consulting projects with government, academic, military and corporate clients. Sarah is a member of the User Experience Professional's Association Voting and Usability Project. And she is an invited expert on the W3C WCAG 2.0, Evaluation and Methodology Task Force. Sarah. Sarah Swierenga: Hello everyone this is Sara Swierenga. Thank you for taking the time out of your busy, early start of the new year to hear a bit about some of the research projects going on in accessible voting systems. We are very excited about this project and happy for the opportunity to get a little more attention given to specific technologies as opposed to Juan's nice overall system. So first I'd like to - maybe if I can move - do I have control? Shaneé Dawkins: Yes you can click next at the next arrow at the bottom of the screen to move on or you can double click the slide. Sarah Swierenga: I am - there we go. All right, we're on our way. First I'd like to just acknowledge a fantastic research team. My collogue James Jackson who's on the line with me today to help handle questions at the end. But Graham Pierce, Jennifer Ismirle, Robert Decloniemaclennan are all user experience researchers in my lab. And Stephen Blosser is a rehab engineer who has done a lot of work with people with disabilities doing customization projects for specific disability combinations - wonderful resource from the Michigan State University of Resource Center for Persons with Disabilities. And Aditya Mathew is a recent Ph.D. graduate who has actually has stayed on to work in our CPD on accessible technologies for people with disabilities. We also - one of the other aspects of this project that I won't deal with too much in today's talk. But the initial prototype for the Smart Voting Joystick was done in conjunction with the engineering students at Michigan State University. There's something called a capstone project where at the end of their senior year - during their senior year - they have to take on a real world design project, implement it and have it up and running in one semester. And the team that was excited to work on this project included Yangyi Chen, Tyler Dennis, Graham Pence, Behdad Rashidian, and Joy Yang. And in addition, this grant was also able to fund a number of introductory engineering students to develop universal mounting devises. I'll speak about that a little bit later. But the exciting part about this is, is that we now exposed a number of engineering practitioners who will go out into the workforce - wherever that might be - to accessible design. And that is an experience that now will, you know, ingratiate all the organizations that they end up with. So that's very exciting for us here as a side benefit of this project. This particular grant project grew out from a number of other research projects that we've been working on in accessible voting context. One of the - we've worked on - actually we've worked on protocols that vendors might use to test the voting system for people with disabilities - particularly blind - low vision and people with dexterity issues. We recently and are currently working on what would an interface design look like for people particularly with dexterity and mobility issues - if they were able to handle voting on a small screen like a tablet. And - excuse me - and then additionally, the smart voting juristic project. And one of the things that came out of some the prior research is that - and I'm sure this is not a surprise - is that there's several areas where there's room for improvement in electronic accessible voting systems. And particularly we are focused, in this particular project - on people who have dexterity and mobility challenges. And in typical systems today, your regular voting systems - quote, unquote - there's usually a touch screen that requires that the person be able to lift their arm. It requires some shoulder strength. It requires the ability to continue pressing on the contest and the candidates and maybe the write in - so for periods of time. Or there's like use - some systems have a button panel. And so you can go up, down, left and right and various combinations. But this should for people who have very serious dexterity or mobility issues is the repetitive nature of this task and just the - how the technology is currently implemented. So our group did some thinking about how we can we move beyond the current choices - most of which focus on two button. There's a picture of that on the screen right now - - a two button switch. They either say yes or no. Or they say, you know, "next" or highlighted select. But the basic idea is that you use one button to move to the right or top to button. And you move the other to select and/or sift and puff which works on the same type of design strategy. And the issue with these is that it requires a lot of button pressing or sifting or puffing. And to vote something like in this standard ballot which has representative types of context for various - that you would see on typical ballots in this country. And the idea is that it just takes a lot of button pressing. And so how can we reduce the burden on the voter who is already in high pressure situation there might be somebody else in line. And how can we reduce the burden of the actual (unintelligible). Let's see here. So our goal for the project was to create what we're calling a Smart Voting Joy Stick that would be able to plug into an electronic voting systems. And then get some feedback on that design from users who have dexterity and mobility limitations and working with the Michigan State University Engineering design teams that I mentioned a little earlier. So what is showing on the screen right now is a relatively basic looking box with a joy stick mounting device with a box and a little golf ball. It turns out that that's the size that's comfortable for some people who have dexterity issues. And there's a small select button on to the left. So this was the first generation of the joy stick that the engineering team working with us in UARC Usability/Accessability, Researching Consulting and with the engineers from the research center for persons with disabilities RCPD. So this is how far they got in the semester. It was considered a very successful project from that standpoint. But we knew it needed more work. So over the summer - so this was designed the first semester - so January - a year ago at this time - starting. And it was it done in May. Then over the summer of 2013, we worked to refine the joystick. We worked - the engineers worked with several configurations - - a smaller stem, and a smaller knob - - to make it a little more like some of the joysticks out that are available on wheelchairs. And then we had some larger buttons put in place that are labeled, et cetera. And while we're not going to talk a lot about the technology, I'd like to talk more about the user experience with it. The joystick itself has - provides tactile haptic feedbacks. So when you press on the joystick in any direction, it gives a physical reaction. So you can feel that the joystick has moved. And the buttons, of course, give a tactile feedback as well. And we have a - we developed a sample - a prototype ballot to go with it. So that we could interact with this joystick on a regular PC screen via the joystick is hooked by a USB port. And so there's a press release on it. I was hoping to show the video. But you'll have to just - when the slides become available, it would be easier if you'd just go on out there and take a look. One of the participants in our study and I were interviewed for this article. And it just shows a little bit of the person using the joystick. So we were very excited to get that coverage. Because it gave some attention to this project that we wouldn't otherwise have gotten. So just briefly, before I go into the usability study that we ran the engineering 100 students - there were six or seven teams. And they came up with a variety of mounting devises. Because when you think about it in most voting systems, the actual accessible button panel or other technologies like mounted to the machine or right next to it. And we need to think about some more flexible design so that they could set the (unintelligible) app. They could use the foot pedal. It could be mounted on a table or an arm. And then so the students just thought of some different - different ways that you could actually hook this type of device in the voting system setting. So I, you know, we're happy to take more questions about that later. But I'd like to now use the remaining part of this time just to talk about the actual study that we did. So this is - what's showing on the screen right now is the basic configuration where we have the joystick, the three buttons for enter, help and review and then just a screen that shows the contest - the contest sorry where the joystick if you move it up and down goes through the contest - or the items within the contest and then left to right. You can either use the arrow or double move the joystick to the right to move between contests in either direction. And let's see what else might there be interest. So (unintelligible) 205 and the ballot that was designed is got a lot of flexibility in it. But it still needs more work. But the participants were recruited from Michigan State University. We had a few students involved. And we also reached out to our networks in the community. And had some adults from the Michigan area participate as well. And we ran the study in our lab that has, you know, accessible access accessible table height. And we could configure the environment to best suit the needs of the individual users. So, six people with mobility and/or dexterity issues participated in our study. For those of you who don't know a lot about usability methodology, basically the idea is that you have representative users. People who have the disability were the primary users of interest for this study. And then we want them to do a representative task - a typical task. And in this case we want them to vote a portion of the NIST the standard ballot - - the test ballot. And then the facilitator (unintelligible) intake. What kinds of - what areas did the participants have trouble with. So we're looking at how long. Were they successful in doing the task? If not, what kinds of errors. And, you know, how did it go? And then we gave them a post study questionnaire that asked them some questions about what it was like to use it. And then we also talked to them after they finished the voting task about what the experience was like. So the usability measures that used were effectiveness. The percentage of votes completed accurately - efficiency - the average time to complete the voting and satisfaction - again, the comments and ratings from the qualitative part of the session. What can we find? Well in our particular set of participants, they seemed to - they split very clearly into two groups which we called more moderate disability. These are all dexterity mobility issues. And these were the participants who were able to use the joystick without a lot of configuring. And they were able to - they had more of the dexterity side of the challenge. The sever disability group included a couple of participants who had more physical disabilities then like you were dealing with spasticity - not being able to - not having much verbal communication. And so the two groups - the data was different enough even in this very small set that we handled them separately. And so the basic results were that in the moderate disability group all four users were able to complete the voting task. Three of the four were able to vote it as instructed. In other words, we said vote for this person in this contest. Skip that contest. Vote for these three people in this contest. So we gave them instructions. And they needed to vote according to those instructions. So that we could determine whether they were actually voting the way we wanted - we intended them to vote. The average time to complete it was about nine and a half minutes. And the average time to change a vote was 30 seconds. This compared to some of the basic pilot testing that we had done earlier is actually very good numbers. And compared to some of the systems that might be out there in terms of the number of button presses, this is pretty dramatic. For the sever disability group, one of the two could complete the voting task. But one person - the one person who did complete the ballot, did make a few errors. So they technically would not have voted as they intended if it were a real election. And the second person just was not able to work with the ballot. The disabilities were pretty severe. And, you know, people have an off day. That's life. We're all human. So we went, more or less, to interviewing and just trying to get some feedback about what might be helpful for them so that we could then include in the recommendations. So the time for the one person to complete the ballot was almost 30 minutes. And the average time to change a vote was about five and a half minutes - so significantly longer but still doable. And all participants gave pretty positive ratings about this joystick especially thinking about what the alternatives were in the voting setting. So given more time - more money - and/or working with other teams - we're open to all of those. We really think more attention - there needs to be more refinement to this innovative technology. One thing I didn't mention earlier about this joystick - because you're thinking, well, what's so fancy about a joystick. You move it left it goes left. But typically it will go left one character or one word. This particular one was designed so that you could actually feel when you moved from president and vice president. So it'd read the presidents name, the vice presidents name and their party. And then when you move the joystick it would move to the next item. And then item could be a two or three line, you know, with the names and the parties. So it was the being able to have a sense of moving between the items on the ballot as opposed to just words or characters. And the joystick does need more refinement - the ability to adjust the - how much force is needed. And it was too sensitive for a couple of the more challenged users. It was that maybe - it was okay for the rest. But this needs to be adaptable and flexible. And for a couple of the participants it would really make more sense. We went ahead and tested the dual access. So you could move, you know, up, down, left and right with the joystick. But for some of the more severe situations - that you - they clearly indicated that they would rather just have the left, right - so left, right select. The joystick could be configured in a different way. We tried several configurations. But there's always room for some thinking about other choices. You know, maybe thicker and shorter and more spherical more like other joysticks that are available. And really thinking about the arm support - you have to - and that's where the mounting devices come into play. So you have to be able to support weakened - weakened limbs. So those are the for future research directions what might be at more interest to the participants here is we would be excited about trying this out in something even with an accessible voting machine. We were in the lab for this particular project. But I think one of the next steps is that it would be helpful if there would be a way to connect this to a voting system. And give it a try to another usability evaluation after we've had - or the team or a wider team would have the funding in that to take a look at some of the more detailed characteristics of the joystick. Shaneé Dawkins: Sarah, your time is up. Sarah Swierenga: So, I just wanted to say that the smart voting joystick defiantly has potential to be part of a next generation voting system. It did allow people to vote independently and privately even at the more extreme limits. And we have received some positive reaction both from the media coverage that we got. And we are certainly interested in any one who would like to think about refining and moving on for commercial development. Thank you for your time. And I am finished with my presentation. Thanks again. Shaneé Dawkins: Great thanks Sarah. We have two quick questions for you Sarah before we move on to Dana. The first ones from I would say (Stephanie) Babb. "How many contests were on the ballot used for the study?" Sarah Swierenga: We worked with [the NIST standard ballot]. So we used the entire ballot except for the right-in. There's a - one of the candidates or one of the contest on that ballot includes a write-in. We didn't use that one. So I can count up quickly. And at the question time, we can answer that more completely. But it was several contests. So like 20. James Jackson: Something like 20 - 10 or 20 something like that. Sarah Swierenga: Yes, maybe 10 to 15 would be my guess. But we have the report right here. We can look that up momentarily. Shaneé Dawkins: Okay and then last questions is from Diana Mitchell-Fulford. She says, "thank you. In terms of mounting a device, most of the choices I have for my voters is having the voter put the device on their lap. I find this discouraging. What do you recommend for arm support?" Sarah Swierenga: Thank you for the question. And that was what these engineering teams were thinking about. And some of the exciting designs were taking advantage of things that people have in their workstations at work where you have like a mounting arm that comes out and can - the joystick can - or whatever device it is - could be mounted to that. The other was a free standing device. This was pretty simple stuff. If any of you play music, you know, like a music stand. It was something like those - that type of concept where it could be positioned close to the participant - or sorry the voter. And there were also some wheelchair mounting type devices since most wheelchairs also have adaptive technology already mounted to them. But I agree that trying to set a foreign voting device in someone's lap is a bit disconcerting for everyone involved. There were a few designs though that kind of took advantage of things like you do if you're like reading in bed. There's like a tablet with a pillow underneath it. And you could set the devise on that. The Velcro was very important so that you could position the buttons and the joystick where you want to make it the best for you. Thank you. Shaneé Dawkins: All right. Thank you Sarah. And Sarah as well as Juan will be available at the end of the webinar to address any further questions any of you may have or if you have any comments for them. Up next, we have Dana Chisnell. And I have [wonderful] bio for Dana Chisnell. When Dana Chisnell signed up to work on what ended up being the anywhere ballot project with Drew Davies' Oxide Design and Kathryn Summers of University of Baltimore, she didn't' want to do the project. It seemed unlikely that anything real would come out of the project. The outcome would just molder in a dark corner of the Web somewhere and neglected. But it turned out that anywhere a ballot has been one of the best projects she has ever been involved in within her 13 years of working on design in voting and elections. Dana is the Co-Director of the Center for Civic Design, Dana. Dana Chisnell: Thanks a lot Shaneé. This project was a design project really - as much as a research project on a sub grant from ITIF housed at the University of Baltimore. And as Shaneé mentioned, I was honored to work with Kathryn Summers and some of her great students at the University of Baltimore and Drew Davis from Oxide Design Company. I'm going to give your some background and talk about the realities. I will try to get this into some kind of pilot in a realistic situation. But first, I want to give you a little bit of background. We started with a question from a workshop that ITIF put on. And that question was what if anyone could vote on any devise anywhere, anytime. That's how the name Anywhere Ballot was born. So what this is a digital front end - is built in HTML5 and CSS3 which means for people who are not nerds - that you can use this ballot through any standards compliant browser - on any devise. And the wonderful idea behind that was that people could conceivably in some future - that we were visualizing - use it on their own devise. In fact, I carry around my iPad and I make people try it out where ever I am. So besides being a digital front end - really too - what we envision eventually being something like a ballot marking devise - okay imagine this ballot as being implemented on Prime III for example as the ballot user interface or as the ballot for whatever system that Sarah' s team put together with assess through the joystick. This would also be available through a creative commons license. And in fact we created a pattern library. This is a compilation of the design elements that are part of the anywhere ballot that make up the visual design, the graphic design and the interaction design. So you can go borrow parts of that. The pattern library also includes information about why we did it the way we did it. And what's good about doing it that particular way. And I'll give you a link for that at the end of my talk. In our ideal world, right now, what we'd like to see is that the anywhere ballot could be imbedded into any kind of academic expedient really to voting - why reinvent a ballot for your experiments when you could just use this one in your system. It has some limitations. But I think it's pretty flexible for that kind of thing. Now we started this design based on applied research that was sponsored by the EAC and this on ballot design and language and instruction on ballots. Because some of us on the team had actually done some of that research. We were pretty confident going into the lab that we had something that was going to work pretty well for people. But we learned some really useful things in the usability testing. And I'm going to talk about that a little bit too. Right now, what we're all about is sort of continuous improvement. Our grant took us pretty far in the design of this ballot front end. But it's not - well, it's not perfect. The design is beautiful and is very accessible. Pat even tried it out for us. Pat Leahy tried it out. We'd like to iron out some of the flows that we already know about. Our charter on this project was to test with people who had low literacy and mild cognitive disorders. We'd like to test with people who have other disabilities and combinations of disabilities. And we'd like to test with a lot more older people. We had a few in our study and working with them was really fun. But they have some - some of them have special issues that we would like to explore some more. There's a lot that we don't know about how the summary works for people, for example. The way this ballot works is you - on a touch screen device of whatever size - you touch the screen to make your selections. And then they all collect together. And it shows you a summary of your choices. Sort of like - you're at a restaurant. And you get a menu in the beginning. And you pick some things from that menu. And your bill at the end of the meal shows up with the things that you picked only - not with the whole menu. So it shows the summary screen. And while part of the way the study that we did was constructed asked people to interact with the summary a lot. And we think that we have a pretty good design. That there's - there may be more that we can do to that design that will compel voters to actually check their votes before they cast their ballot without somebody like me moderating to say, okay you need to check now. We think it's probably acceptable within most states' elections laws. Also, there's another issue. We designed our task ballot based on the NIST medium complexity ballot. In total there were 25 pages to this ballot. We put one contest per page. And we get to decide what the wording was. We use a typical hierarchy - in terms of what the contests were. So top of the ballot was president, vice president. Bottom of the ballot was judges to state referenda, et cetera. But we would like to - we'd like to pilot this. So we can see how it will react with state election laws and the constraints that you encounter out there in the field. Would the design manage? There is a lot of interest in the Anywhere Ballot. Whenever we show it off at a NIST event or at a conference, people get really excited about it. Don DeFord from the State of Oregon, for example, just thinks that this is the most beautiful ballot he's ever seen. And we do too. But we would like to see in real life if this is a thing. Our project only went up to designing this template. It didn't include a pilot. And so we've been working the last few months since we sort of wrapped up design to try to make a pilot happen. There's quite a lot of interest - as I say. We've had a few states say, oh my god, we really, really want this including states that are of a by mail. Because as Dr. Gilbert pointed out, they go out to rehab facilities and nursing homes and other community centers to help people vote there. And this is a very portable way to do that. It's also an awesome way to pilot things out. A couple of states have contacted us about the possibility of using it as a basis of their UOCAVA voting system. The open source digital voting foundation has been working on a voting system for some years. And they have said out loud in public that their ideal ballot is based on the anywhere ballot. The VSAP project in Los Angelo's County which is in design and development to create a new voting system that can handle LA County's many constraints and their huge scale - has shown some interest as well. One of the neat things is that Eddie Perez who's at Hart InterCivic has taken everything he possibly can from the design of Anywhere Ballot to implement in the user interface on their touch screen systems for their new voting systems that should be coming out soon. I hope. I know that they're testing. So there's hope that this will see the light of day - contrary to my original concerns about this project. So the big thing is that there are some challenges with getting a thing like this out into the world. And we've encountered almost all of them in just trying to make this thing happen. Dr. Gilbert mentioned, that there's a long lead time in talking election officials into doing this because this is not a small thing. It's hard enough within an election as it is without somebody coming along and saying, will you try this out in a real election too. We saw how this worked a little bit - some of us in Minnesota in the November elections a few jurisdictions tried out some electronic poll books in St. Paul. And the way they did it was they had the poll books operating on the electronic versions. But they also had paper backups. So they needed more poll workers. They needed more training for the poll workers. They also had to include, you know, some explanations to voters about what they were doing. So there's some considerable overhead just on Election Day. But backing up from that, you have to make sure that you're in the right place in the election cycle to sort of introduce this and make sure everybody's ready to go. There's not a small amount of IT implementation. There takes a lot of coordination there. And if you don't control your own voting system, you have to work with your vendors to do these things. It's not a small thing there. It takes some planning and coordination. And then the question is, who would do the implementation. Can this be done eternally in an IT department or is this something that the vendor actually has to do? Procurement - even though this is creative commons license open source, there are resources that have to be made available to do a pilot of something like this. And procurement is - I don't have to tell you all who our election officials are government officials. This is not a straight forward or easy thing to deal with these days when you're talking about - especially kind of a formative design project. One of things that we've encountered really is we've run up against a lack of political will where we've received questions about well if this is such a great idea how come nobody's done it yet. But somebody needs to go first. And right now nobody wants to take the risk of going first. The other part of the risk is that election laws often get in the way of doing these things - coordinating with who can do the work - and next, where this fits into the process. And voting legislation - election legislation is not built for the digital age in most places. So how do you handle wording that has to be on the ballot for navigating that may be different from what's in the legislation. So those are just some of the objections that we've encountered. But I want to talk about the difference between usability testing verses piloting. And fortunately, I've had the good luck of having Juan go first and talk about piloting a lot and Sarah go next and talk about lab testing. And so now I can meld them together for you. So what we did was what we call formative testing where we did rapid interactions. So we did two rounds one with a paper prototype where we just had what we wanted to design on sheets of paper that we had a graduate student of Kathryn's [observe]. She was acting as the computer. So when a participant did something, we just show it another piece of paper that would show the results on the digital. And then in the second round we created a digital prototype that operated on an IPad. We did 33 individual sessions this way so one voter with me moderating. There was a lot of interaction between me moderating and the participant to ask follow up questions to understand what kinds of problems they were encountering. What was frustrating them? What kinds of questions they had - what was going on there? This is not a thing that you can do in a pilot. And it's often not the kind of thing that you do in a lab test either - a lab usability test. Usually usability testing is fairly controlled. There's a direct observation. Because as you can imagine and you can probably see from the first picture in the series. You know, there are two people there. It's kind of a cooperative thing. And usually testing is really about informing the design. Piloting is a very different beast. While you can get some of that same kind of information, you know, what's frustrated people or where did they get stuck, you're very much more hands off. But it does give you realistic context. You know, it puts this thing out into the field in a way that you don't have any control or you have very little control. And it uses real election data going in and going out. In addition, we had 33 individual sessions. Sarah's study sample was fairly small too. And though Juan's projects have covered a broader range of people in their experiments, now you're letting it out to people who you can't predict are going to come into the room. So the research is very important. And I am excited that we got to do this. But I'm really hoping that we get to do some piloting as well. So, you know, just the tradeoffs here, the research can give us a baseline. And it's a way to experiment with ideas. A well designed research project can show what's possible and lower the risk. On a pilot - piloting gives you some access to some noncommercial ideas early. If you're an election official, this may be appealing to you. Likewise, you can let your requirements be known as the purchaser - - as the client - - kind of outside our fee situation. So this is a way for you to kind of test out ideas about what you think you need. But also piloting can show you what you need to change operationally and in training. And possibly give some preview to what needs to be changed in legislation - if at all. So one of the important aspects of pilot is making some determination about how to do that and scale really is the - probably the most important thing right next to having a plan for backing out of it. St Paul had a great plan for doing it. They had a paper backup. So they could continue to get people registered and sign the roster to be able to vote whether the e-poll vote worked or not. And they kept it to a few locations where they knew the other variables very well. You know, what - how well the poll workers were trained. And what the leadership of the pole workers was like. And what the physical environment was like there. Who are the kind of voters were in the precincts, et cetera. And so the question you have to ask when you start to do pilots is, you know, what if the pilot actually fails while it's happening. What does failure mean? How you're going to measure that? And what does success mean? How will you know whether what you're seeing actually is a successful outcome? But this is also another opportunity to assess usability. And there a whole bunch of different ways to measure that as well with, you know, post voting surveys or questions or little postcards. One of the things that the team that created - that did the research behind the EAC's best practices for ballot design - they handed out post cards with just a couple of questions to voters when they were on their way out of the polling place. And they were prepaid. And voters returned those with, you know, little checkbox answers and a couple of comments. It was really helpful just to get that kind of feedback. So in summary, while the Anywhere Ballot is a beautiful, accessible thing we've encountered questions about fusibility and willingness to go ahead, we are intending to continue to improve this thing somehow. Local association elections, university alumni elections things like that may come into play. And we're interested in seeing what we can find out about what the implementation obstacles are in those smaller elections. But I want to emphasize here that usability testing and piloting together are really important in making a successful system whether it's a voting system or not. So this is the team behind field guide (unintelligible) insuring voter intent. And so I love how the line broke in the URL. But if you are interested in the work that we do, you'll want to go look at civicdesigning.org/fieldguides. You can download free PDF's of those. But you can also - also try out the Anywhere Ballot for yourself. Go to anywhereballot.com in any browser. And the first pages of the ballot will come up. You'll get to go through it for yourself. If you're interested in the pattern library that we've created, go to anywhereballot.com/library. And I want to thank you very much for being here today. Shaneé Dawkins: Great thanks, Dana. Okay so now we'll open it up. We have about 15 minutes for questions. I would let this start with one from Pat Leahy. Oh wait, I'm sorry. Dana has individual questions. Oh that's from the EAC. Okay, so I'll start with Pat Leahy's question. He says it's a question for Juan, "What would you suggest as the first few steps for people wanting to pilot new technologies?" Dr. Juan Gilbert: The first steps? Shaneé Dawkins: The first few steps. Dr. Juan Gilbert: Relationship building - targeting, you know - assuming you already have your product ready or your technology ready for piloting - relationship building. You got to build relationships with election officials, et cetera. That's the first step. Shaneé Dawkins: Okay... Pat Leahy: And that's exactly - that's exactly where I was headed Dr. Gilbert that you've done such a good job of building the relationships - first with the associations and then building that out to elections. Shaneé Dawkins: Okay I do also want to make a correction really quickly. On the slides - seeing the first slide - her name and she's from the Center for Civic Designs. It sort of looks like the Center for Civic Design is a part of National Institute for Standards and technology, I just wanted to clarify - thanks to Sharon's comment that, that's how it appeared. That it is not... Dana Chisnell: Sorry about that. Shaneé Dawkins: Sharon is the only person that noticed that. So I did want to clarify that. And I'll go on. We have about six more questions from chat. And after I go through those questions, I will open it up to live questions where you can - the operator will let you in to answer your questions. So I'll continue on. There is a question from Sanford Morganstien for Dr. Gilbert. Is your system EAC certified? Juan? Dr. Juan Gilbert: Can you hear me? Shaneé Dawkins: Now we can. Dr. Juan Gilbert: Okay I was on mute sorry. No we're not a vendor. So we will not be EAC certified. And that's just not our intent. Shaneé Dawkins: Okay, next one we have from Stephanie Babb. "Will hands free" - oh this is for Juan. Sharon Laskowski: Juan. Shaneé Dawkins: "Will hands free feature be available for the Wisconsin primary on April 1st?" Dr. Juan Gilbert: No it will not. We're going to do probably a press release - the demo of the technology. So we will - no, it won't be ready for primetime in 2014 at all. Shaneé Dawkins: Okay, next is from James Johnson. This is for Juan. You're popular Juan. Is there anyone offering commercial implementations of the Clemson Prime III system? If not, what is the time frame?" Dr. Juan Gilbert: That is an interesting question. So I don't want to answer this. So I won't use any names. But there are vendors that are creating solutions that when you look at them, you're going say, oh, that looks familiar. And that's fair. We didn't patent this. It's in the open domain. They can model, mimic, replicate. And steal all they want from what we're doing. That's fine. There are vendors and individuals who have our code who are integrating it to service solutions. And I'm not privy to what they're doing beyond that. So it is out there. And as you start looking at the next generation machines, if you see Prime III then you look at what's being done, I think you'll figure it out. Shaneé Dawkins: Okay, back to... Sharon Laskowski: Sarah. Shaneé Dawkins: Sarah, for Stephanie Babb's original question about how many contest were on the ballot for your study. Did you... ((Crosstalk)) Sarah Swierenga I - we have - there were - we used 21 of the contest. And the task also involved going back and changing your decision or your vote on one of the contest. So it was built into the script. Shaneé Dawkins: Okay for Sarah, again, David Baquis has a question. He says, "There is occupational therapy research on use of assistive technology switches. Can that help inform voting technology research? Example - - factors affecting the use of a single switch with assistive technology devices." And he has a link to a PDF that I will send out at the end of the webinar. Sarah Swierenga: Thank you for the question. We know that there's a lot of switch technology and some research out there as well. Stephen Blosser, the Rehab Engineer, has utilized many different versions and actually created a number of those switches that are out there. So he was thinking about that - about those issues and the in-depth details of the joystick and the buttons and the mounting because he's done this for 30 some years and created individualized, customized accommodations for facility staff and students and even people in the community over the course of his career. So yes, those types - that type of research is very helpful. That particular article you referenced talks about, you know, some of the key items of interest or like the key differentiating for what might be (unintelligible) for people with mobility and dexterity issues. But we were really focusing more on - okay we know that having one button or two buttons leads to having lots, and lots of button presses to get to the ballot. How can we reduce that issue? And that's why we focused on the joystick. Thank you. Shaneé Dawkins: Okay on the last chat question - before we open it up for live questions, is from Sanford Morganstein. This question is for EAC so Pat I believe that's you. “And the question is for EAC and goes to your “political will” observations. Does the EAC have a way to get new innovations piloted in a real election when such innovations may not be certified and therefore may run into problems with HAVA Law - for example, accuracy requirement?” Pat Leahy: Can you hear me okay, Shaneé? Shaneé Dawkins: Yes. Pat Leahy: Okay, it's a great question. We've been tossing that around with our AVTI group as well. And I quests I would just say, that - and Dr. Gilbert mentioned this as well - that on the certification part that's where some of this becomes a little more challenging. But currently what we're doing is, you know, sponsoring, you know, these 45 initiatives to get new technology and new R&D out there. And I guess we'll just need to see what the future holds as far as on the legislative front and at the state and local as well for elections. How we can improve and get more out there for promoting R&D and new pilots into elections. And I think you've kind of seen some of that kind displayed today with working with associations first. And then moving that into, you know, maybe a primary or something like that and making it - maybe not the full election at first but a certain precinct or certain area of the county or a county. So we're working on it. And our focus is to kind of keep pushing forward and to improve access. Shaneé Dawkins: Okay, great. I was going to open up for live. But we have one more chat question for Sarah from Sherri Charleston. "Can you explain how this can be" - this I assume meaning your joystick - "can be integrated with existing technology or is the expectation that the joystick will be attached to a "COTS - I'm sorry "commercial off the shelf hardware and operational live using the software that you have developed?" Sarah Swierenga: Basic idea for that, again, this is an open source type project as well through this grant which we're very grateful to receive. So the design itself is meant to be flexible. And we use a USB port. Because obviously with the time and the funding and the resources it made sense to just integrate with a ballot on a PC. But the basic design could be interfaced with whatever the voting system can accommodate. So this is really more of a proof of concept at this point. But it certainly has the characteristics to be commercialized. Shaneé Dawkins: Okay, all right. So before - we're running close to 2:30. So I just want to say in the chat box online I put each of the presenters contact information and their main Web site. We will still open it up for live calls in a moment. So for Juan Gilbert for those of you who are not on the Web, he's at juan@clemson.edu and it's humancentercomputing.org is his Web site. Sara Swierenga is sswieren@msu.edu and her Web site usability.msu.edu. Dana Chisnell is dana@centerforcivicdesign.org. And her Web site is centerfordesign.org. And I'll just close with a few more things before we open for live questions in case anyone has to go. Then we are having another webinar on Thursday, January 23. It's Accessible Voting: People and Technology. And it's from 10:00 am to 11:30 pm. Sharon Laskowski: AM. Shaneé Dawkins: Oh I'm sorry - 10:00 am to 11:30 am. Yes that would be a long webinar. And we have the speakers for that webinar. We have Amanda Beals out of Paraquad, Brad Fain from Georgia Tech Research Institute, and Jon Sanford from Georgia Institute of Technology. And their registration link is available. I will send it out after the webinar. And it is on the Web screen now. If you would like to join the AVT email list - if you haven't already done so, you can send your contact information to avtvote@nist.gov. Also we have AVT Web portal online at nist.gov/itl/vote/accessiblevoting. An archive of today's webinar will be available within the next week. And we hope that you all give us feedback on this webinar. And if it turns out well for this webinar in the next one there's a potential to have two more webinars this year with some of the other research that has been done under the AVTI grant. So after this webinar here, you'll be able to provide some open feedback. You'll receive an email about that. And then we'll send the next week you'll receive a survey to complete on the webinar. So, Nelson, our operator, if you can open it up for live questions please. Operator: Thank you. Ladies and gentlemen, if you would like to register for a question, please press the 1 followed 4 on your telephone keypad. Your line will then be accessed to obtain your information. If your question has been answered and you would like to withdraw your registration, you may press the 1 followed by the 3. If using a speaker phone, please lift your handset before entering your request. Once again to register for a question, via phone is the 1 followed by the 4 on your telephone keypad. You may also ask a question using the chat feature located in the lower left corner of your screen. Shaneé Dawkins: Thank you, Nelson. So if you have questions for any of the presenters, Juan, Sarah, or Dana, Pat Leahy is available from the EAC. And from NIST you have me, Shaneé Dawkins and Sharon Laskowski is here with me as well. Operator: And as a reminder, to register a question via the phone is the 1 followed by the 4 on your telephone keypad. I am showing no questions at this time. Shaneé Dawkins: Oh, okay, well if no one has any further questions, as I said before all the contact information is online. I will send out follow up emails with all the speakers contact information and links to some of their references from today including Sarah's video as well as the link that David Baquis has sent out. And Dana Chisnell's pattern library that she also mentioned. You will also receive a link for the registration for our next webinar which is Accessible Voting People and Technology. All right if there are no more questions, Nelson, I think we'll close here. Operator: Okay, Ladies and gentlemen that does conclude today's webinar. We thank you for your participation. And ask that you please disconnect your line. END