"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson

5 minutes into this talk and my mind is already blown.

What if the invariant you want to test isn't so invariant after all, but actually depends on what happened during the simulation? Ex: if some message gets dropped, then I wouldn't expect the final state to have key X. Is this feasible? You would need to make the simulator dynamically update the expectation based on what random thing it breaks. It seems like doing that would likely require reimplementing a lot of the state management logic of the system under test itself.

I remember the "spin" network protocol simulator from Bell Labs in the early '90s. It was used at then-AT&T to predict and avoid bugs in telecom equipment systems. Sounds very similar in approach; simulate first, detect protocol problems (deadlocks, livelocks, bad states, etc) and then implement once you get it right.

can u fix the sound?

Also, tools like TLA+ are made to solve these problems. The difference being that TLA+ uses logic to verify all traces exhaustively, without taking millions of years to do so. Running a sim will only explore a tiny percentage of possible event traces.