This helps encode the idea of eventual consistency and uses eventual consistency to optimize the system by not requiring total consistency which prevents people from iterating quickly.
Again, “people don’t write incorrect code, they write code they think is correct”. So it’s important to consider good UX as being “innocent” until proven guilty rather than guilty until proven innocent.
link to “software development process and feedback loop is a distributed system”
CI failure (for the deferred jobs) automatically creates an issue, linking to the PR, the PR author, and relevant code. A timer is then set; if the build isn’t passing soon enough, all all deployments are halted until the failing tests pass. Deferred compliance can be used for any long running CI run that does not need to block deployments while creating a call for action for CI run failures
This worked because:
- The long running jobs were required for on-prem, not for deploying to cloud.
- Consequently, they were effectively only required once every two weeks.
- Eventual consistency was enforced in a timely enough manner that fixes didn’t become outstanding.
- Crucially, the fix window is much shorter than the release window, so releases are never observably broken.
- This workflow was not a common path, so it wasn’t continually happening.
- That is, it’s also important to realize that most tests are ran every time but will never fail.