Continuous Integration

  • CI should be stateless
  • CI should be transient and ephemeral
  • CI should be idempotent
  • CI should be quick to start up cold
  • CI should ideally be minimal
  • CI should be sound. Completeness is impossible
  • Testing pipelines should be quick. They should finish by the time someone commits, opens a browser, goes to the repository, and looks at the results of their pipeline. 20-30 seconds, ideally.
    • Not everything needs to be tested every time
    • Not every test needs to run on every code change
    • Compiling can be done incrementally if caches are configured well
    • There are 3 tiers of tests. Each corresponds to the context required to test a falsifiable assertion of a Good Explanation.
      • Warning: This article shows why it’s important to write even the logic tests within the context of Good Explanations.
      • As a general rule, the more stateless an application is, and the less IO it does, the more its total complexity can be tested with pure logic tests.
      • Logic: Stateless, idempotent. No IO or threading. (<1ms) 70%
        • Ex: Unit tests, Generative testing
      • Interface: Stateless, Idempotent. (<1s) 20%
        • Composition of logic preserves semantics
        • Tests composition invariants that can’t be represented in the type system
      • Environment/Context. “The full thing” (no limit) 10%
        • Tests things as fully and interactively as possible from the perspective of the end user (ie their full environment with their full context)
    • Schedule tests and CI runs. Run Env/Context tests nightly.
    • Use branches as control flow. Example: Use CI to auto merge to the “staged” branch. Test the staged branch. Auto merge to master if it passes the long nightly test suite.
  • Compiling should ideally be reproducible
  • CI test results should be ergonomically reproducible in a development environment
  • Properties of CI/CD/CC
  • Don’t break the build/
  • Many large projets see to follow the optimization of calculating changed files, then calculating the minimal sets of targets to build and then building only those targes. Particularly for monorepos.
Links to this page
#CI