How to Survive the Recommendation Engine Question

Most engineers focus on the matching algorithm. The real test is separating generation from serving.

The fastest way to fail the recommendation engine question is to spend all your time designing the matching algorithm.

That is the trap.

Most engineers hear “recommendation engine” and immediately think about collaborative filtering, embeddings, similarity scores, ranking models, or machine learning.

Those things matter.

But they are not where the system design answer usually wins.

The recommendation engine question is not asking whether you can invent the perfect algorithm in 45 minutes.

It is asking whether you can design the system around the algorithm.

Can you collect behavior reliably?

Can you turn events into useful signals?

Can you generate recommendations before users need them?

Can you serve those recommendations quickly?

Can you handle stale results, missing data, cold starts, and fallbacks?

That is the real test.

Now let’s get practical. A recommendation engine has two different jobs:

Recommendation generation decides what might be relevant.

Recommendation serving decides how to return it reliably.

Mix those two jobs together, and the design gets slow, expensive, and hard to operate.

Separate them, and the architecture becomes much easier to reason about.

That separation is how you survive the recommendation engine question.

---

Run AI agents in a system you already control

AI coding agents are a great tool when you’re tackling a backlog of tasks, but they can also introduce challenges.

• Most run on vendor servers.

• This can mean specialized software, API keys, and agent harnesses that don’t suit your needs.

• Security and governance aren’t always in your hands.

Coder Agents runs on your own trusted infrastructure, directly within the Coder control plane.

The process is simple: describe the work you want done, and Coder Agents takes care of selecting a template, provisioning a workspace, and executing the task, all for your review. Delegate development and research without losing control.

---

The Trap: Treating Recommendations Like a Query

Imagine you are designing homepage recommendations for an e-commerce app.

A user opens the app.

The backend needs to recommend products based on previous purchases, recent views, similar users, product popularity, inventory, location, price range, and business rules.

A beginner's design often looks like this:

This looks reasonable. The API receives a request, loads user history, compares that history against products, ranks the best matches, and returns the results.

For a small catalog, this works. For a decent catalog, it is a different story.

The system now depends on several databases at request time. The recommendation logic grows from a simple query into a pipeline of joins, scoring rules, user history, item metadata, and business constraints.

The main problem here is treating recommendation generation as something that should happen every time the user asks. That pushes too much work into the serving layer.

A better answer starts by splitting the system into two parts.

The Real Test: Generation vs Serving

A recommendation system has two different responsibilities.

The first responsibility is generation.

Generation answers:

What should we recommend?

This part uses events, features, candidate generation, ranking models, business rules, and nowadays, machine learning.

The second responsibility is serving.

Serving answers:

How do we return those recommendations safely, reliably, and with good enough freshness?

This part uses prepared recommendation lists, low-latency storage, final filters, fallback logic, experiment assignment, and monitoring.

The clean architecture separates those concerns:

This is the core design.

Generation can be complex. Serving should be predictable.

Generation can use large datasets, streaming jobs, vector search, model scoring, and offline training.

Serving should mostly read prepared results, apply final corrections, and return a useful response.

But there is an important nuance.

Separating generation from serving does not mean everything runs in a nightly batch job. That is an outdated mental model.

Modern recommendation systems usually use multiple freshness layers.

Some signals are updated in batches, while others are updated through streaming pipelines. Some get corrected during serving.

That gives you a better mental model than “batch vs real time.”

The bad design computes everything inside the API request. A naive design assumes everything can wait for the next batch job.

The better design lets each signal update at the speed it actually needs.

For example, long-term user taste may not need second-by-second updates.

If someone has watched cooking videos for months, that preference can come from batch or periodic feature jobs.

But session intent may need faster updates.

If the same user suddenly watches three videos about hiking gear, a streaming pipeline can update recent-interest features within seconds or minutes.

Inventory and policy rules need even stricter handling.

If an item goes out of stock or violates a policy rule, the serving layer should remove it before returning the response.

Generation: Prepare good recommendation candidates.

Serving: Return the best valid recommendations available right now.

This is why the question is not primarily about picking the perfect matching algorithm. The heavy lifting of matching and ranking belongs in the generation pipeline, even when some model scoring happens close to serving.

The architecture decides how events become features, how features become candidates, how candidates become ranked lists, and how those lists reach users without falling apart.

Once you explain generation vs serving, the rest of the design becomes easier to discuss.

Now we can walk through the generation side first.

Step 1: Collect User Behavior

Recommendations start with behavior. The system needs to understand what users do.

For an e-commerce product, useful events might include:

The exact events depend on the product, but the architecture follows the same pattern.

You do not want every click, purchase, or view to directly recompute recommendations. That creates tight coupling between user actions and recommendation logic.

Instead, the application emits events.

This gives the recommendation system raw material without making the product experience depend on recommendation computation.

But raw events are still too noisy to use directly.

The next step is turning those events into useful signals.

Step 2: Build Features

A feature is a signal the recommendation system can use.

For a user, features might include:

Top categories
Recently viewed items
Purchase history
Average price range
Preferred brands
Location
Activity frequency
Last active timestamp

For an item, features might include:

Category
Brand
Price
Popularity
Rating
Availability
Freshness
Embedding vector

The feature pipeline transforms raw events into structured signals.

For example, this behavior:

This part of the system may require aggregation, deduplication, cleanup, joins, and historical data.

That is why it belongs in the generation side. But…

Bad events create bad features.

Bad features create bad recommendations.

Late features create stale recommendations.

Missing features create fallback scenarios.

So a strong recommendation starts with the data pipeline that feeds the ranking model.

Where Features Actually Live

Building features is only half the problem. The system also needs to serve them fast and consistently, in the same shape during both training and inference.

That is the job of a feature store (Feast, Tecton, Hopsworks, or a custom equivalent). It keeps an offline store for training and an online store for low-latency lookups, and keeps both in sync. Without it, teams hit training-serving skew: the model trains on one version of a feature and serves a slightly different, staler one at request time, quietly degrading recommendation quality.

You don’t need to design a feature store in an interview. Naming it, and explaining why training and serving need consistent features, is enough.

Once the system has user and item features, it can start building candidate sets.

Step 3: Generate Candidates

Candidate generation answers one question:

Which items are worth considering?

The system should not rank every item in the catalog for every user.

If the product has 10 million items, scoring all 10 million items for every homepage request does not make sense.

The first job is to narrow the search space.

A candidate set might contain a few hundred or a few thousand items. Then the ranking system can do deeper scoring on that smaller set.

Simply put, candidate generation optimizes for recall.

Ranking optimizes for precision.

A practical system usually generates candidates from multiple sources.

Candidate sources:

- Items similar to recently viewed products
- Items bought by similar users
- Trending items in the user’s category
- New items from preferred brands
- Frequently bought together items
- Sponsored or editorial items
- Fresh inventory that needs exposure

Each source brings a different signal.

• Collaborative filtering helps find items based on similar users.

• Content-based filtering helps find items similar to what the user already liked.

• Popularity-based recommendations help when the user has little history.

• Trending recommendations help capture what is changing right now.

• Editorial or sponsored candidates let the business inject controlled inventory.

• Embedding-based retrieval helps the system find semantic similarity at scale.

At scale, every modern recommendation system uses embedding.

A common pattern is a two-tower model.

One tower creates a user embedding. The other tower creates an item embedding.

The idea is simple.

If the user vector sits close to an item vector, that item may be a good recommendation.

But there is an important scaling detail. You do not compare the user vector against every item vector one by one. That would be too slow for millions or billions of items.

Instead, the system uses Approximate Nearest Neighbor search.

ANN search trades a small amount of exactness for a large improvement in speed.

Examples include FAISS, ScaNN, and HNSW-based indexes.

This is how embedding-based candidate generation becomes practical. The system precomputes item embeddings, stores them in a vector index, and queries that index using the user or session embedding.

The result is not the final recommendation list, but only a “candidate set”. Items that might be good.

Ranking decides which ones are actually best for this user, this context, and this business goal.

A simple candidate generation strategy might look like this:

For user u123:

40% embedding-based candidates
25% recently viewed category candidates
15% collaborative filtering candidates
10% trending regional candidates
10% exploration / editorial candidates

That mixture avoids relying on one signal.

• If the user has strong history, personalized sources perform well.

• If the user is new, trending and editorial candidates help.

• If the item catalog changes quickly, fresh inventory candidates help.

• If the model becomes too narrow, exploration candidates give the system a way to learn.

This is also where exploration starts.

A recommendation system cannot only show what it already believes the user will click. That creates a feedback loop.

The system shows running shoes because the user clicked running shoes. The user clicks more running shoes because that is all the system shows. The model becomes more confident that the user only wants running shoes.

That is exploitation.

Exploration deliberately reserves some candidate slots for uncertain items, new categories, new creators, or fresh inventory.

Common approaches include epsilon-greedy, Thompson Sampling, and Upper Confidence Bound.

You do not need to fully implement those in an interview. But mentioning the trade-off shows maturity.

A good recommendation system should not only optimize what it already knows. It should also create opportunities to learn.

The final output of candidate generation is a blended candidate set.

Candidate Set:
[
  items from embeddings,
  items from similar users,
  items from recent categories,
  items from trending,
  items from exploration,
  items from business rules
]

Now the system has a smaller set of possible recommendations.

The next step is ranking them.

Step 4: Rank the Candidates

Candidate generation gives the system a smaller pool of possible recommendations.

Ranking decides the order.

At this point, the system is no longer asking: Which items might be relevant but which items should appear first?

A simple ranking function might look like this:

Full disclosure: this is useful for explaining the idea, but it is not how serious production rankers usually work. A linear score is a teaching tool. It assumes each signal contributes independently. Real user behavior has interactions.

For example, “expensive item” might be bad for one user and good for another.

“Fresh item” might matter more in news, less in furniture, and a lot in short-form video.

“Popular item” might be useful for a new user but boring for a power user.

That is why production systems often use models that capture feature interactions better, such as GBDTs or neural rankers.

The important interview point is not the exact model. The important point is where ranking fits in the system.

Candidate Generation: Reduce millions of items to hundreds or thousands.

Ranking: Score those candidates deeply and order them.

The output is an ordered list.

Candidates:
[p18, p91, p43, p22, p77, p10]

Ranked recommendations:
[p91, p22, p18, p77, p43, p10]

One common idea is to re-rank the final list to reduce redundancy.

For example, Maximal Marginal Relevance balances relevance with novelty by penalizing items that are too similar to items already selected.

That ranked list is the system's best answer for now. The next step is making it fast to retrieve.

Step 5: Store Recommendations for Serving

The recommendation store should match the access pattern.

This store could be Redis, DynamoDB, Cassandra, Elasticsearch, or another low-latency store. The technology is less important than the shape of the data.

You want reads by key. You do not want the recommendation API to join users, products, orders, clicks, and inventory just to build a homepage response.

This is why recommendations often behave like cached decisions.

And Like Cache als expires

TTL is not the whole design. If a recommendation key expires, the serving layer needs a clear cache miss strategy.

A practical flow looks like this:

1. Try personalized recommendations

2. If missing, try stale-but-recent recommendations

3. If unavailable, use category or regional trending

4. Trigger async regeneration

5. Return the best available fallback

The API should not block while a full recommendation pipeline runs. That would turn a cache miss into a slow user request.

A better design returns a fallback immediately and triggers regeneration in the background.

Some systems also keep stale recommendations for a short grace period.

That is better than returning nothing.

The rule is simple:

Cache miss should degrade quality, not availability.

The system does the expensive thinking earlier. Then it stores the answer in a form the product can serve.

Now we can look at the serving side.

Step 6: Serve Recommendations

Serving should be boring.

The API fetches a prepared list, applies final checks, and returns results.

Notice what this call does not do:

1. It does not scan the full product catalog.

2. It does not compute similarity across millions of users.

3. It does not build long-term user features.

4. It does not run the whole ranking pipeline.

It serves a prepared answer and applies final safety checks.

That is the serving mindset.

Serving is not where the whole recommendation system lives. Serving is where the recommendation system meets the product.

But prepared recommendations have one weakness.

They can become stale.

The Freshness Problem

Precomputed recommendations improve serving speed, but they create freshness problems.

• A user might buy a product and still see it recommended.

• A product might go out of stock after the list was generated.

• A user might suddenly show interest in a new category.

• A trending item might spike before the next ranking job runs.

• A policy rule might change after recommendations were computed.

This is the cost of separating generation from serving. You gain simplicity and predictable reads, but you lose perfect real-time accuracy.

The answer is not to make the entire system real-time. That usually makes the system more expensive and harder to operate. The better answer is to separate slow-changing signals from fast-changing constraints.

Generation handles relevance. Serving handles final correctness.

This gives you a hybrid design.

The Hybrid Design

Most realistic systems use precomputed recommendations plus small real-time adjustments.

Example:

The system can also store more recommendations than it needs. For example, it might store the top 200 recommendations for a user but only return 20.

That gives the serving layer room to filter bad items without running out of results.

The same idea applies to fallback chains.

Recommendation serving should assume some generated results will be missing, stale, or invalid.

That leads to the next major piece: fallbacks.

Fallbacks Are Not Optional

Recommendation systems fail in normal ways.

• The user may be new.

• The item may be new.

• The recommendation list may not exist.

• The feature pipeline may lag.

• The ranking job may fail.

• The recommendation store may timeout.

• The final filters may remove too many items.

If the system has no fallback, the product returns empty space or low-quality results.

A practical fallback chain might look like this:

1. Personalized recommendations

2. Recently viewed category recommendations

3. Trending in user region

4. Global trending

5. Editorial/default list

Fallbacks also solve the cold-start problem.

• A new user has no history.

• A new item has no engagement.

• A new category has weak signals.

In those cases, the system can use onboarding preferences, session behavior, trending content, editorial lists, or popularity by region.

A good recommendation engine does not assume perfect data. It assumes missing data and still returns something useful.

That is the difference between a demo and a production system.

Now we can talk about choosing the right recommendation key.

Not Every Recommendation Should Be Personalized

Many engineers assume every recommendation must be user-specific.

That is not true.

Some recommendation surfaces are item-based:

• People also bought this

• Similar products

• Related articles

• More videos like this

These can use item keys:

key: recs:item:p91:also_bought
value: [p12, p88, p43, p71]

This scales well because many users can reuse the same result. If 10,000 users view the same product, they can all read the same related-products list.

Other recommendation surfaces are category-based:

key: recs:category:running_shoes:trending
value: [p8, p31, p72, p19]

Some are region-based:

key: recs:region:florida:popular
value: [p18, p43, p29, p71]

Some are anonymous-session based:

key: recs:anonymous:homepage:mobile:us
value: [p77, p41, p10, p93]

The key matters because the key defines the serving strategy.

• User-based recommendations give better personalization, but they cost more to compute and store.

• Item-based recommendations are easier to reuse, but they know less about the current user.

• Trending recommendations are less precise, but they make great fallbacks.

This is another place where the interview tests architecture.

You are not only choosing how to match users and items. You are choosing how the product will serve recommendations at scale.

That brings us to observability.

What You Should Monitor

A recommendation system can break even when the API still returns 200 OK.

• The response may contain stale items.

• The ranking job may be delayed.

• The feature pipeline may stop processing events.

• The fallback rate may spike.

• The store may return empty lists.

• The API still works, but the product experience gets worse.

That is why monitoring matters.

Fallback rate is especially useful. If fallback rate suddenly increases, something upstream probably broke.

• Maybe the ranking job failed.

• Maybe feature generation is delayed.

• Maybe the recommendation store lost keys.

• Maybe filters removed too many items.

Good observability helps you detect when recommendations are technically available but product quality has degraded.

If CTR drops, was it the new ranking model, a feature pipeline delay, or a seasonal shift? You can’t tell from a dashboard alone.

That is the job of experimentation infrastructure: holdout groups, experiment assignment, and consistent bucketing so a user sees one version of the system, not a mix of several. Without it, every model change is a guess dressed up as a metric.

You don’t need to design an experimentation platform in an interview. Mentioning that ranking changes need controlled rollout, not just deployment, is enough.

Now we can turn this into a clean interview answer.

How to Answer This in an Interview

A strong answer starts by separating generation from serving.

You can say:

I would not design this as one live query.

I would split the system into a generation side and a serving side.

The generation side collects events, builds features, generates candidates, ranks them, and writes recommendation lists to a serving store.

The serving side fetches those prepared lists, applies final filters, handles fallbacks, and returns the response.

That answer shows the right architecture.

Then walk through the components:

Events
  -> Feature pipeline
  -> Candidate generation
  -> Ranking
  -> Recommendation store
  -> Serving API
  -> Filters and fallbacks

After that, mention the main trade-off:

Precomputed recommendations are easier to serve, but they can become stale.

So I would handle fast-changing constraints during serving: inventory, already purchased items, already seen items, regional availability, and policy rules.

Then mention cold starts:

For new users, I would use trending, region-based, editorial, or onboarding-based recommendations.

For new items, I would use content-based similarity, category placement, editorial boosts, or exploration traffic.

Finally, mention monitoring:

I would monitor generation freshness, serving latency, fallback rate, empty result rate, and product metrics like CTR or conversion.

That is a complete system design answer.

It does not ignore algorithms. It puts algorithms in the right place.

That is the whole point.

Final Thoughts

The recommendation engine question looks like an algorithm question, but it is really an architecture question.

The matching algorithm matters, but it is only one piece of the system.

A strong design separates generation from serving.

Generation figures out what might be relevant. Serving returns those recommendations safely, quickly, and reliably.

That separation is the mental model most engineers miss.

The recommendation question is not an algorithm problem. It is an architecture problem disguised as one.

Until next time,
— Raul

---

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.