Continuous Integration

  • CI should be stateless
  • CI should be transient and ephemeral
  • CI should be idempotent
  • CI should be quick to start up cold
  • CI should ideally be minimal
  • CI should be sound. Completeness is impossible
  • Testing pipelines should be quick. They should finish by the time someone commits, opens a browser, goes to the repository, and looks at the results of their pipeline. 20-30 seconds, ideally.
    • Not everything needs to be tested every time
    • Not every test needs to run on every code change
    • Compiling can be done incrementally if caches are configured well
    • There are 3 tiers of tests. Each corresponds to the context required to test a falsifiable assertion of a Good Explanation.
      • Warning: This article shows why it’s important to write even the logic tests within the context of Good Explanations.
      • As a general rule, the more stateless an application is, and the less IO it does, the more its total complexity can be tested with pure logic tests.
      • Logic: Stateless, idempotent. No IO or threading. (<1ms) 70%
        • Ex: Unit tests, Generative testing
      • Interface: Stateless, Idempotent. (<1s) 20%
        • Composition of logic preserves semantics
        • Tests composition invariants that can’t be represented in the type system
      • Environment/Context. “The full thing” (no limit) 10%
        • Tests things as fully and interactively as possible from the perspective of the end user (ie their full environment with their full context)
    • Schedule tests and CI runs. Run Env/Context tests nightly.
    • Use branches as control flow. Example: Use CI to auto merge to the “staged” branch. Test the staged branch. Auto merge to master if it passes the long nightly test suite.
  • Compiling should ideally be reproducible
  • CI test results should be ergonomically reproducible in a development environment
  • Properties of CI-CD-CC#
  • Don’t break the build/
  • Many large projets see to follow the optimization of calculating changed files, then calculating the minimal sets of targets to build and then building only those targes. Particularly for monorepos.