Test-first development for voting system certification?

At the OSCON session on Hacking Open Government, Secretary of State Debra Bowen talked about the mismatch between the process of certifying voting systems, the changing nature of voting requirements, and the goal of open source voting software.

Currently, voting systems need to be certified in order to be used in elections. The certification process entails submitting code to a testing agency that keeps the code, tests, and results proprietary. The Secretary of State’s office has access to the data. Citizens don’t. The testing process is long and cumbersome. This imposes a significant barrier to new entrants, including open source voting systems. When new requirements are added, the system needs to be re-certified. This imposes a long delay on the adoption of modifications.

This testing process is based on a model that is older than current best practices for software design. The testing process is based on a “waterfall” fall method where software is developed, and testing is done, all in one piece after the fact.

Current best practices are different in a number of ways.
* Software is developed incrementally, and testing is done continuously, as the software is built.
* Tests are written before the software is developed. Tests serve as the detailed specification for the way the software is intended to function
* Tests are written incrementally. New tests are added to govern new behavior.
* There are automated test suites that verify that the system continues to pass tests, with old and new behavior

This suggests a different process for voting system certification.
* Tests are made publicly available. Detailed tests serve as specifications for the behavior of the voting system.
* There is an automated test suite that continually tests the behavior of voting software.
* New functionality can be added to systems and tests incrementally. Tests will verify that the system continues to function correctly, for old behavior and new.
* Results of tests are publicly available.

Using an incremental, test-driven process for voting system development and certification would improve the reliability of the process, by enabling more scrutiny. It would shorten the time needed to introduce new voting system improvements. And it would lower the barrier to new entrants, including open source systems.

This testing would cover only functional behavior of the system – are votes counted correctly, does the administrative process work. There is still a need for security and penetration testing, which goes beyond the function of the code, includes all aspects of the system, including physical security, authentication practices, data integrity, and more. And there is still a need for usability testing – which as far as I know is not yet part of voting system certification. Usability problems result in a larger portion of day-to-day voting system failure than technical failures, although technical failures can have disastrous results.

Still, opening up the functional testing process, and running it incrementally, seems as though it might offer significant benefits.

For practitioners of modern software development and testing – what do you think about this suggestion? Are there any big gaping holes that would make this nonsensical or unfeasable? Feedback most welcome.

Architecture for civic participation

Last week’s brainstorming session on the use of social media for voter education got me thinking about the architecture that is needed for civic participation. The underlying concept is that the government provides basic infrastructure services and data. Citizens can participate in oversight and decision-making, and build tools for additional engagement, through access to services and data.

To facilitate participation, openness is needed in several layers.

  • open code and open data. These are two related families of practices that engage the community in the development and review of technology; and that make public information available to the public. Open data includes basic availability, as well as support for standards and licences that enable re-use and participation.
  • open APIs. Application programming interfaces enable developers to build on basic government infrastructure services, creating a broader ecosystem of applications that deliver value to the public without additional government funding, and that provide services that the government can’t.
  • Effective practices for social participation. Several attendees noted the problems with simple comment systems that devolve into anti-social anarchy, driving away constructive citizen participation. There are many techniques, tools, and social practices to overcome these problems. Solutions are context-dependent – there is no one-size-fits all solution.

It is exciting to participate in discussions such as the Social Media for Voter Education, the Hacking Open Government session at OSCON, and Transparency Camp West, coming up this weekend in Mountain View, that are helping to spread these ideas and encourage their implementation.

Earthshatteringly bad service from Earthlink

It took over 5 hours to recover from an expired domain name. The experience with Earthlink was one of the worst customer experiences I’ve had in my life, and I can’t remember the other ones.
The domain had been my primary web and email address for 10 years. It took me a few days to realize that the problem was the domain, not just an email delivery problem. Once I realized the problem was with the domain, I looked up the domain on whois and saw that it was registered at register.com.
Then came a comedy of redirection. I called Register, who sent me to their partner division, who sent to Earthlink, who sent me back to Register, who sent me back to Earthlink again. Each of these episodes involved long waits on hold. The last Register clerk who sent me to Earthlink a second time was extremely patient and volunteered to stay on the line while I talked to the Earthlink folk who denied I was registered through them.
This time the Earthlink clerk recognized the registration. But she was unwilling to give me any information about the account unless I provided her with the credit card number I used to register the account a decade ago. I don’t know about you but I save financial information for 7 years, which is the recommendation for tax purposes. Finally she referred me to another number. She wouldn’t explain why, or what the other number was. I stayed on hold at that number for 30 minutes, then gave up and tried Earthlink chat.
Finally, the chat clerk explained what the problem was. The number was a collections agency. Apparently I had owed money to Earthlink (even though my records showed that I had been paying them regularly). Another side trip to the billing department at Earthlink who confirmed that I was paid up. A re-visit to Earthlink chat. Now, the clerk identified that there were *two* accounts, one of which was delinquent (I hadn’t gotten any notices that I remembered). But he couldn’t give me any more help unless I remembered the decade-old credit card. He was about to invoke a supervisor confirmation procedure that would take another day to verify my identity.
But then I had an epiphany. Amazon.com to the rescue!!! I realized that I’d been buying stuff from Amazon for a decade. I looked up the history of my Amazon account. Lo and behold, there were the 10 year old credit card numbers. At this point, Earthlink’s computer system went down and I needed to call back 30 minutes later. Finally, they were able to look up my account, and sent me to the collections people with instructions to get a confirmation number once I had paid my bill.
By this time, the collections office was really closed (they hadn’t picked up the phone earlier when they were supposedly open). I tried them in the morning, the first minute they opened, and someone answered the phone. They confirmed that my account was paid up. They couldn’t give me a confirmation number, since I didn’t owe any money. To get out of the catch 22, I offered to pay them just to get a confirmation number. The supervisor discovered a fee of $50 (real or fictional, I’m not sure), and I used this to ransom the confirmation number.
Then I called back Earthlink, gave them the number, and was ready to renew the domain. Ok, in order to renew your domain, you need to transfer the hosting to us, said the clerk. Oh, no I don’t. It took a few steps of back and forth to cause them to renew my domain name without getting their hosting services which I very definitively did not want. I gave them a new credit card, and paid $30, which is $20 more than the going rate.
A couple of hours later, alevin.com was back in service. But that wasn’t even the end. I got an email from them, saying that my credit card had been rejected. They hadn’t used the new credit card number I gave them, but the 10 year old number that had been expired for years. I gave them the current credit card number, again.
I think I registered the domain with Mindspring, back in the day, when they were good. Earthlink bought mindspring and went through years of turmoil and decline. Now they have acheived a level of customer service that is dramatically awful.
In 30 days, I can transfer the domain to some other registrar and be rid of Earthlink forever.

She’s Geeky

This week, I went to She’s Geeky, an unconference for women in technology. There were sessions on topics technical, organizational, entrepreneurial, and personal.
One interesting session was on managing groups of men. The conversation dealt with some of the style differences between women and men, the list below comes from that session and some others that dealt with the topic:
* women communicate by telling stories that put the issue into context; men are more likely communicate with bullet points and arguments
* women often try to lead conversations by asking questions and getting others to contribute; this can be read as weakness
* decisiveness and strong opinions from women can be read as bitchiness. People varied in their reaction to this, ranging from “claim your inner bitch” to “learn to respect people with alternative skills and styles”
* see above: women may care too much about what other people think about them.
* women sometimes have trouble saying no; there was a whole session on the topic that I didn’t go to.
* on the whole, more men believe they’re above average, and more women believe they are below average (think about this for a moment…) women need to learn to filter men’s boasts when they aren’t matched by reality, realize their own competence, and get safe support to build confidence.
There were also some rather unfunny stories of traditional sexism: the only female engineer in a group being asked to decorate a new office; a woman who found she was making less than similarly qualified men; a woman executive being asked to regularly provide fashion advice to her CEO (and she seemed to feel obligated to do it). (I suggested that she refer him to the neiman marcus personal shopping service.

The Box

Last post’s exploration of “the way things work” was “Infrastructure”, Brian Hayes’ photo survey of industrial infrastructure. This week’s episode of “Richard-Scarry-for-grownups” is The Box, by former Economist editor Marc Levinson, which delves into the history of container shipping.
The Box is compelling history of things and people. It dives into the details of industry structure, finance and technology and assembles an intricate picture of transformative change. And recounts the adventures of the competing entrepreneurs racing to get the system working, beat competition, and outwit regulation.
Container shipping appears inevitable from the perspective of technological determinism. Boxes, trains, trucks, motorized ships, cranes, none of the technology was dramatically new. Container systems had been tried in the railroad shipping since the 1920s. The old system, where each item needed to be loaded, unloaded, and reloaded with manual labor, was costly and slow. But, a clear view is not the same as a short distance, to quote Paul Saffo.
The incumbent industry had strong incentives to preserve the status quo. Shipping, trucking, and trains were regulated industries with centrally set prices and terms of service, established cartels, and a focus on the mechanics rather than on the service of transport. It took an innovative entrepreneur and some well-timed government handouts to break the logjam. Malcom McLean, a trucking magnate, envisioned the system in his minds eye, drove the engineering for the interlocking containers and the fast-loading cranes, put together aggressive debt financing, and benefited from the US government’s giveaway of WWII surplus transport ships. Far-sighted port agencies in New Jersey, Long Beach, and Singapore invested heavily in container ports, securing early leads. When change came, it was rapid. Levinson writes, “Three years after containerships first sailed to Europe, only two American companies were still operating breakbulk ships across the North Atlantic.”
But even the folks who saw change coming had very imperfect foresight. Many cities invested in ports, but only a few succeeded, and others invested heavily without return. After making a fortune in his first container ventures, McLean himself bet badly, on a fast, fuel-guzzling container ship that hit the market during the 1970s oil crisis, then on a huge slow ship that was introduced just in time for the 80s oil price crash. From a distance, the transition to container shipping seems orderly and logical, like water flowing downhill. Close up, it’s rapids.
And its attention to evidence shows a more complicated picture of the relationship between labor, capital, and government than would be predicted by ideology. Much economic writing in the popular press has a clear ideological slant. The free market generates the most efficient economic outcomes, while regulation, government subsidy, and labor protection reduce economic growth. Alternatively, regulations protect against excessive corporate power, subsidies protect infant industries and local economies, and unions empower workers.
Levinson’s history of the rise of container shipping uncovers a more mixed and subtle story. The early innovators in container shipping got a jumpstart from a government fire sale of surplus WWII ships. WIthout the gift of lowcost ships, the capital costs of ships would have been higher than the entrepreneurs could carry. Early on, some port cities and agencies invested heavily in the creation of container ports. The government investment paid off spectacularly well for some, and badly for others.
At the same time, the shipping, trucking,and rail industries were highly regulated. Players were attuned to manipulating the regulatory agency rather than competing. Much later on, the successful container industry helped drive deregulation. Levinson doesn’t touch the reasons that the railroads got regulated in the first place; they had been an overly powerful oligopoly that abused their market power. So, when does it make sense for government to subsidize or regulate industry? Sometimes, in the cases of early industries, very high capital investments, and to combat market power. And sometimes regulations and subsidies outlive their usefulness.
The biggest expense in shipping was not the transport itself, but the repeated loading and unloading of every item. Longshoreman’s unions arose to protect workers against an abusive contingent labor system, where workers scrambled every day for the chance to unload the days ships. The union policies provided steady work, but also created work rules that mandated more workers than were needed to do the job. The longshoreman protested containerization vehemently. In some regions, protracted labor conflicts kept the port from adapting to the new technology; by the time the union lost, the container ports had been set up elsewhere. But in the US west coast, the union negotiated a settlement where longshoremen whose jobs were made obsolete received retirement payouts. The benefit of containerization was shared with the workers.
The Box tells a story that is more complicated than an ideolog would prefer. Unions and government actions are sometimes helpful and sometimes harmful, and helpful structures can outlive their usefulness and need replacing.

Freedom from Cingular

So, it sounds like Cingular and other phone companies have been blocking calls to Freeconference.com>
I am very eager to try the Nokia E61i with wifi, and to see what the OpenMoko project comes up with. How long til someone sells voip phones for $49 in cities with good public net? Tony Bowden, a Socialtext colleague who lives in Estonia which has great wifi, was tryign the skype phone approach. Wonder how that was going.

Speaking of followup, muni wireless

installations have been around for long enough to come back and test which ones are working. Novarum, a consultancy specializing in wireless broadband, has gotten behind the hype and the skepticism, and tested muni wireless networks by coverage and speed. The best rated system were Saint Cloud Florida and Mountain View (which worked when I was there). The first thing to note is that according to the study, some of them actually do seem to work. Second, reasonable performance depends on more transmitters; early estimates recommended 20 transmitters per square mile, but it appears as though 40 are needed for adequate performance.

Novarum also ranked the cellular broadband networks, and included them in an overall ranking with the wifi nets. The Saint Cloud net came in first overall, and the Google Mountain View net came in number ten on the combined list. The cellular nets rank better because they have better coverage. Wifi nets, when you can get them, are faster than cellular.
One puzzlement is that Palo Alto appears on the wireless list, ranked number 8, at 2.45 on a scale of 5. On the University drag, there are plenty of locations offering free wifi, but what is the muni offering? Is it the lame Anchorfree service that has poor connectivity and a horridly annoying registration system? If that’s the case, then it’s below the cutoff where a rational person would consider the system to “work.” Santa Clara is above it, ranked at 2.65. A field trip may be in order for some anecdotal testing. I wonder where in the Santa Clara sprawl the network is to be found?
What was the population in the survey? Hard to say. To build that top 10 list, how many citiies did they visit. Ten? Fifty? They don’t say on their website. This makes it impossible to draw conclusions about the overall state of muni wireless investments.
Novarum plans to come back in six months to test again.