Test-first development for voting system certification?

At the OSCON session on Hacking Open Government, Secretary of State Debra Bowen talked about the mismatch between the process of certifying voting systems, the changing nature of voting requirements, and the goal of open source voting software.

Currently, voting systems need to be certified in order to be used in elections. The certification process entails submitting code to a testing agency that keeps the code, tests, and results proprietary. The Secretary of State’s office has access to the data. Citizens don’t. The testing process is long and cumbersome. This imposes a significant barrier to new entrants, including open source voting systems. When new requirements are added, the system needs to be re-certified. This imposes a long delay on the adoption of modifications.

This testing process is based on a model that is older than current best practices for software design. The testing process is based on a “waterfall” fall method where software is developed, and testing is done, all in one piece after the fact.

Current best practices are different in a number of ways.
* Software is developed incrementally, and testing is done continuously, as the software is built.
* Tests are written before the software is developed. Tests serve as the detailed specification for the way the software is intended to function
* Tests are written incrementally. New tests are added to govern new behavior.
* There are automated test suites that verify that the system continues to pass tests, with old and new behavior

This suggests a different process for voting system certification.
* Tests are made publicly available. Detailed tests serve as specifications for the behavior of the voting system.
* There is an automated test suite that continually tests the behavior of voting software.
* New functionality can be added to systems and tests incrementally. Tests will verify that the system continues to function correctly, for old behavior and new.
* Results of tests are publicly available.

Using an incremental, test-driven process for voting system development and certification would improve the reliability of the process, by enabling more scrutiny. It would shorten the time needed to introduce new voting system improvements. And it would lower the barrier to new entrants, including open source systems.

This testing would cover only functional behavior of the system – are votes counted correctly, does the administrative process work. There is still a need for security and penetration testing, which goes beyond the function of the code, includes all aspects of the system, including physical security, authentication practices, data integrity, and more. And there is still a need for usability testing – which as far as I know is not yet part of voting system certification. Usability problems result in a larger portion of day-to-day voting system failure than technical failures, although technical failures can have disastrous results.

Still, opening up the functional testing process, and running it incrementally, seems as though it might offer significant benefits.

For practitioners of modern software development and testing – what do you think about this suggestion? Are there any big gaping holes that would make this nonsensical or unfeasable? Feedback most welcome.