November 10 – 12, 2020

Online Edition!

Designing Your Tests Up to Fail

Jenny Bramble

Automated tests are information, not bug detection systems

Testing should verify EXPECTED behavior. A defect purposefully released to prod is now expected behavior

Automated tests should pass on EXPECTED behavior even if that behavior is objectively a defect (AKA designing to fail)

Using TODOs, documentation, and other methods to keep your team informed of the expected behavior of your automation suite

EUROPE'S GREATEST AGILE SOFTWARE TESTING FESTIVAL!

Designing Your Tests Up to Fail

We have to write tests on features that have defects--so let's make them useful.

It’s a fact of life that we often have to write automated UI tests for features that have defects, or that interact with 3rd party APIs that aren’t returning the right responses, or for items that we know aren’t working right.  When the team has decided that the behavior isn’t going to be fixed, what’s an automation engineer to do?  Let the tests fail?  Not write them? Champion harder for the defects?

Jenny Bramble suggests writing your tests to pass.

By creating tests that pass on the current expected behavior (the defect), we are in a perfect position to tell when the defect is resolved, or the api is returning the correct information, or any of the other error cases we may be encountering.  This prevents failure fatigue (from seeing a test ‘always fail), while still providing meaningful, actionable information out of our test suite.

She will discuss several cases that she’s experienced that this method has worked for as well as how to keep the rest of the team informed through TODOs, Jira stories, and documentation.  And—of course!—what to do when your test finally fails.

You’ll walk away from this talk with a clear idea of when to design your tests to fail, how to use TODOs and other indicators to let the rest of the test team know what’s going on, as well as a frank discussion of automation as information—not as a bug detection system!


  • Automated tests are information, not bug detection systems

  • Automated tests should pass on EXPECTED behavior even if that behavior is objectively a defect (AKA designing to fail)

  • TODOs, documentation, and other methods to keep your team informed of the expected behavior of your automation suite


More Related Sessions


30-min New Voice Talk

12:05 p.m. – 12:45 p.m.

30-min New Voice Talk

10:15 a.m. – 10:55 a.m.

95-min Workshop

No Recording
1:45 p.m. – 3:20 p.m. Equipment required

30-minute Talk

4:30 p.m. – 5:10 p.m.

If you like AgileTD you might also be interested in :

Your privacy matters

We use cookies to understand how you use our site and to give you the best experience on our website. If you continue to use this site we will assume that you are happy with it and accept our use of cookies, Privacy Policy and Terms of Use.