Latency is a silent killer.

May 16, 2024

Sometimes, the problem is not only in your system but also in your dependencies.

3 ways to deal with Latency:

Get Timeouts right

Time-outs are easy to overlook, but they are important to get right.

If you wait too long, you can slow the whole system down. Time out too fast, and you'll consider a call that might have worked as failed.

But have no time-outs; a downstream service could hang your whole system.

Look at your system's behavior. If you usually get the response in 1 sec, staying with the default of 30 sec is overkill.

Think about the time-out for the entire operation and adjust your timeouts to fix your budget.

Even if we had the timeouts right, we would have to wait a long time to see the error.

A circuit breaker is blown after a certain number of requests to the resource fail (due either to an error or to a timeout).

All further requests that go through that circuit breaker fail fast while the breaker.

This pattern protects the consumer from the downstream problem and the downstream service from more calls it can’t handle.

In a bulkhead architecture, application elements are isolated into pools so that if one fails, the others will continue to function.

Bulkhead prevents a single component from monopolizing resources and causing a system-wide failure.

In many ways, bulkheads are the most important.

Timeouts and circuit breakers help free up resources when they become a problem, but bulkheads can help avoid the problem in the first place.

Proper Timeouts: Prevent indefinite waits and identify hung services quickly.

Bulkheads: Isolate resources to ensure one failure doesn’t cascade across the system.

Circuit Breakers: Detect problematic services early to prevent wasted retries and allow recovery.

When it is slow, your system ends up waiting for a long time before giving up; the same applies to your users.

What else are you doing to deal with latency?