On Monday and Tuesday (6-7th Nov, 2017) I was privileged to attend a two day workshop at Facebook, London on Testing and Verification organised by Mark Harman and Peter O’Hearn, along with about 90 other people. Unusually for these kinds of events it was a very balanced mix of industry and academia.
The theme was testing and verification and some kind of rapprochement between the two groups. As it turned out this was hardly contentious as everyone seemed to agree that these two areas are mutually supportive.
As ever it seems a bit invidious to pick out the key lessons, but for me, I was most struck by:
- Google has ~2 billion LOC. That is hard to appreciate!
- Practitioners strongly dislike false positives (i.e. predicted errors that do not exist), it was suggested more than 10% is problematic.
- Testing multiple interleaved threads is complex and it’s possible to occasionally observe very unexpected behaviours.
- Testing costs are a key concern to developers but not something that researchers, yet, have a good handle on.
- The quality and sophistication of some automated testing tools e.g. DiffBlue and Sapienz (see the paper by Ke Mao, Mark Harman and Yue Jia) and their ability to scale is seriously impressive!
So thank you speakers and organisers!