In the real world, you might need more than a simple Work Queue.

As systems become more complicated, you need to link work queues together. And these 5 Patterns of Event-Driven Processing give you a blueprint.

Sep 04, 2024

In many batch applications, you want to perform more than one action or generate different outputs from a single input.

One common solution is to link work queues together so that the output of one work queue becomes the input to one or more other work queues, and so on, but there are more than this simple approach.

Here, you have 5 patterns beyond the linking work queues.

Thank you to our sponsors who keep this newsletter free:
Multiplayer auto-documents your system, from the high-level logical architecture down to the individual components, APIs, dependencies, and environments. Perfect for teams looking to streamline system design and documentation management without the manual overhead.

The first pattern takes advantage of a distributed system's most powerful features: Parallel processing.

Copier

A copier duplicates a single stream of work items into two or more identical streams.

This pattern is useful when multiple pieces of work are needed on the same work item.

An example of this might be rendering a video. Depending on where the video is intended to be shown, there are many formats that are useful when rendering a video.

A high-resolution format for playing off of a hard drive
A rendering for digital streaming
An animated GIF thumbnail.

These work items can be modeled as separate work queues for each render, but the input to each work item is identical.

Process & Filter

The role of a filter is to reduce a stream of work items to a smaller stream of work items by filtering out work items that don't meet particular criteria.

For example, a batch workflow that handles new users signing up for a service.

Some users checked the option to be contacted via email for promotions and other information.

In this case, you can filter the set of newly signed-up users to only those who have explicitly opted to be contacted.

The original producer provides the complete list of items to be worked on, and the filter then adjusts that list based on the filter criteria and only produces those filtered results to the next work queue.

Process & Split

The role of a splitter is to evaluate some criteria (like a filter), but instead of removing the input, the splitter sends different inputs to different queues based on those criteria.

An example of an application of the splitter pattern is online order processing, where people can receive shipping notifications via email or text message.

Given a work queue of shipped items, the splitter splits it into two queues: one responsible for sending emails and one for sending text messages.

A Copier and two Filters can also implement a Splitter.

Sharder

The role of a shader in a workflow is to divide a single queue into an evenly divided collection of work items based on some sharding function.

One reason for sharding is reliability. If you shard your work queue, then the failure of a single workflow only affects a fraction of your service.

Imagine you push a bad update to your worker container, which causes your workers to crash and your queue to stop processing work items.

If you only have a single work queue processing items, your service will have a complete outage, with all users affected.

Merger

A merger is the opposite of a copier; its job is to combine two different work queues into a single queue.

For example, many different source repositories may add new commits simultaneously.

You want to take each of these commits and perform a build-and-test.

Creating a separate build infrastructure for each source repository is not scalable.

We can model each source repository as a separate work queue source that provides a set of commits.

Using a merger adapter, we can transform these different work queue inputs into a single merged set of inputs.

This merged stream of commits is then the single source of the build system that performs the actual build.

The merger is another great example of the adapter pattern, though in this case, the adapter is adapting multiple running source containers into a single merged source.

Key Takeaways:

These patterns offer a blueprint to manage complex workflows.
They help scale the system by distributing work across multiple queues or consolidating work from different queues into one.
Patterns like Sharder and Merger enhance system reliability by mitigating the impact of failures and consolidating efforts where appropriate.