Head-based sampling decides at the root span, before anything interesting has happened. It is cheap and it throws away exactly the traces you wanted. Tail-based sampling buffers the full trace, then decides. It catches errors and slow requests, at the cost of holding spans in memory across the whole trace duration and needing all spans of one trace to land on the same collector. A workable middle ground: head-sample at a low base rate, but propagate a force-keep flag that any service can set when it sees an error or exceeds a latency budget.
Compaction Debt and Why It Comes Due at Night
LSM trees defer work. Writes go to a memtable, flush to L0, and the merging happens later. “Later” is doing a lot of work in that sentence. If the ingest rate exceeds the compaction throughput, debt accumulates silently — read amplification climbs, then one night the write path stalls because L0 hit its file limit. The instrument to watch is not disk usage. It is pending compaction bytes, and it should be flat, not merely bounded.
Backpressure Is Not Rate Limiting
Rate limiting is a policy applied at the edge with a number someone picked. Backpressure is a signal that propagates from the actual bottleneck. The difference shows up under partial degradation. A rate limiter set for healthy capacity keeps admitting traffic when a downstream dependency is at half speed. Backpressure notices, because the queue in front of that dependency grows and the signal travels upstream. Bounded queues everywhere, and a way to refuse work when they fill. That is the whole design.
Three Ways to Kill a Cache Stampede
When a hot key expires, every concurrent request misses at once and the origin gets hammered. Probabilistic early expiration recomputes the value slightly before the TTL, with probability rising as expiry approaches. Only one request typically wins the race. A mutex per key serializes recomputation, but converts the stampede into a queue — fine if the recompute is fast, bad if it is not. Serving stale while revalidating in the background is usually the right default. Correctness cost is bounded staleness, which most read paths can absorb.
The P99 Is Not Your Slow Path
A common mistake is treating the 99th percentile as “the slow requests” and optimizing those specifically. If a page load fans out to 50 backend calls, the probability that at least one hits the P99 is about 39%. The tail is not an edge case — it is the common case for anything with fan-out. Reducing tail latency is therefore a median-user problem, not a minority problem.
Linearizable Reads Without Going Through the Log
Serving reads through the Raft log is correct but wasteful. Two cheaper options exist. ReadIndex: the leader records its current commit index, confirms leadership with a heartbeat round, then waits for the state machine to catch up to that index. One round trip instead of a full log append. Lease reads: skip the heartbeat entirely by relying on a clock-bounded leader lease. Faster, but correctness now depends on bounded clock drift across nodes. Worth it only when you control the hardware.
Why Write-Ahead Logs Fsync Twice
A write-ahead log has to survive two different failure modes, and they need different guarantees. The first is process crash. The second is power loss. Only the second one requires the data to actually be on the platter, which is why a single fsync on the log file is not enough — the directory entry needs one too, otherwise a freshly created segment can vanish while its contents survive. Most engines batch this. Group commit amortizes the syscall across concurrent writers, which turns a per-transaction cost into a per-batch cost. The tradeoff is latency: a transaction now waits for the batch window to close.