How to handle flaky tests?

Randomly flickering tests which might indicate a false alarm for an incident can be very annoying but they happen. In unstable environments like the world wide web, they are part of the testing and monitoring game. Third party javascript libraries may not be that robust, images might be stored on different servers, the website is loading a lot of advertisements. There are a lot of reasons a website can be "broken" or indicate a failure while running your checks which then disappears during the following cycle of checks.

Leankoala provides three solutions for flakey tests:

• Alert n in a row
  The alerting mechanism of Leankoala can be configured to triggering an alert solely if the error occurs a specific number of times (=n) in a row. The alert will not be sent upon previous detections of errors, but you will be able to see the failures on your status dashboard.

• Non-Strict mode
  The default behaviour of Leankoala is the display of every failure on the dashboard. However, when the non-strict mode is activated a failure must occur at least twice in a row to be a real failure. Please also read "What’s the non-strict mode?" for more information on this configuration.
• Tool Configuration
  If the non-strict mode is not enough or too much effort to configure it is possible to set that option directly within the tool. This option can be found under „Configuration > Tools“. Just activate the overwrite switch and enter the number of "errors in a row". All checks from this tool in that project will now follow that same rule.