A switchback test runs the new system in some time blocks and the old one in others, with the schedule drawn at random, often across several sites at once. Each block is compared with the others, so weather, demand and staffing that vary through the week fall evenly on both versions.
It exists for systems that act on everyone in the queue at once, such as routing, pricing, dispatch or stock allocation. Splitting those by user does not work, because the treated and untreated users share the same queue and the effect leaks between them. DoorDash’s engineering team described switchbacks for its dispatch algorithm in 2018 for this reason.
The number of independent time blocks, not the number of transactions, decides how small a change a switchback can detect. Blocks must also be long enough that one block’s effect does not carry into the next. In practice that means weeks of blocks before the answer is confident.