Test Suites

  • 70:20:10 ratio

  • 70% sub-1ms tests (no threading, no I/O)

  • 20% medium tests (can TCP, I/O, etc)

  • 10% large tests (full browser emulation, hardware running, etc).

  • https://danluu.com/everything-is-broken/

    • ctrl+f what's the best way to do it?
  • https://danluu.com/testing/

  • https://danluu.com/broken-builds/

  • https://blog.patchgirl.io/haskell/2020/08/02/testing-haskell-with-stack-ghcid-and-hspec.html

  • https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/ “automatically detecting how flaky a test is and checking for regressions in quality of tests”.

    • We set out to find tests that don’t show any unreliable behavior (sometimes passing, sometimes not). We quickly realized that there aren’t any — all end-to-end tests have some degree of flakiness.
    • Every test framework and test environment brings an inherent level of flakiness that cannot be reduced by improving the test. PFS lets us compare the flakiness of a real-world test expressed in a particular test framework with the flakiness of the simplest possible test implemented using the same framework. When these two scores converge, it’s a sign that one cannot make the test any more reliable — unless one decides to improve the framework itself.
  • https://engineering.fb.com/2018/11/21/developer-tools/predictive-test-selection/: https://arxiv.org/abs/1810.05286

    • Why using build dependencies is inefficient: Dependency tracking is not per function/feature/functionality granular
    • In practice, many transitive dependencies are not, in fact, relevant for regression testing.
    • To develop a better method, we consider a different question: What is the likelihood that a given test finds a regression with a particular code change?
    • To ensure that our test selection works well for real-world tests, the system needs to address the problem of test flakiness
      • When collecting training data, aggressively retry failed tests to make sure they’re failing consistently and not just flaky