When changing the configuration in Leankoala for a check the very next run of this check will already use the new configuration. If the change was done for a single component that check will be re-run immediately. If the new configuration affects all components of a system the checks will be run at the standard times.
It is possible that the old check configuration was already submitted to the Leankoala test queue. It is not possible to stop this check from being executed. In general, this is no problem but it has one side-effect. Sometimes the new configuration gets a re-run and is executed before the old config. The check succeeds. Afterwards, the old configuration gets executed. The check fails. This should only happen once.