Room
Room F3 - Track 3: Talks
Audience
Tester, Test Manager, Manager, Developers.
Key-Learning
- Our code history contains information about where bugs were fixed in the past. This can point us to systematic problems in our processes.
- Untested changes are most error prone. In large, complex systems, surprisingly many code changes are never tested and cause most of the field bugs.
- Test gap analysis can reveal untested changes prior to release, for all types test: automated or manual, low-level or end2end.
- Test suites often contain a lot of redundancy. Optimized smoke test suites, without that redundancy, often find like 80% of the bugs in 5% of the time
- Defect prediction trains AI on past bugs & code features to predict future bugs. It has won research awards, but doesn't work at all and is useless.
Can Data in Our Repos Help us Find More Bugs Faster? Where Can AI Help, and How Well?
We have to test more and more functionality in less and less time, as successful software grows from release to release, but release cycles are getting shorter and shorter. Historically grown test suites are often not up to this challenge, since they test too much and too little at the same time. Too much, since they contain redundant tests that cause execution and maintenance costs but provide little value over similar tests. Too little, since important functionality remains untested. We must make these test suites more effective (i.e. find more bugs) and more efficient (i.e. faster/cheaper) to succeed in the long run.
Our research community has worked on approaches to increase test effectiveness and efficiency for decades. In recent years, AI based approaches have appeared that also promise to help us find more bugs faster.
In this talk, I present several approaches (e.g. historic bug analysis, test-gap-analysis, predictive test selection, test impact analysis and defect prediction) to find more bugs in less time. We have implemented each of these approaches, have done empirical research on how well they work and employed them with customers and in own development. For each approach, I show how well it works so that you can decide, whether it helps you to find more bugs in less time.