- CI should be stateless
- CI should be transient and ephemeral
- CI should be idempotent
- CI should be quick to start up cold
- CI should ideally be minimal
- CI should be sound. Completeness is impossible
Testing pipelines should be quick. They should finish by the time someone commits, opens a browser, goes to the repository, and looks at the results of their pipeline. 20-30 seconds, ideally.
- Not everything needs to be tested every time
- Not every test needs to run on every code change
- Compiling can be done incrementally if caches are configured well
There are 3 tiers of tests. Each corresponds to the context required to test a falsifiable assertion of a Good Explanation.
- Warning: This article shows why it’s important to write even the logic tests within the context of Good Explanations.
- As a general rule, the more stateless an application is, and the less IO it does, the more its total complexity can be tested with pure logic tests.
Logic: Stateless, idempotent. No IO or threading. (
- Ex: Unit tests, Generative testing
Interface: Stateless, Idempotent. (
- Composition of logic preserves semantics
- Tests composition invariants that can’t be represented in the type system
Environment/Context. “The full thing” (no limit) 10%
- Tests things as fully and interactively as possible from the perspective of the end user (ie their full environment with their full context)
- Schedule tests and CI runs. Run Env/Context tests nightly.
- Use branches as control flow. Example: Use CI to auto merge to the “staged” branch. Test the staged branch. Auto merge to master if it passes the long nightly test suite.
- Compiling should ideally be reproducible
- CI test results should be ergonomically reproducible in a development environment
- Properties of CI/CD/CC
- Don’t break the build/
- Many large projets see to follow the optimization of calculating changed files, then calculating the minimal sets of targets to build and then building only those targes. Particularly for monorepos.