Your AI Agent Won’t Crash. It Will Happily Pay an Invoice Without Approval

A practical guide to observing AI decisions with OpenTelemetry

At some point, you’ll get the call from above:

“We need an AI agent to handle [fill this with whatever you like].”

A team I know reached out to me for help; they had an acquisition and needed to process invoices from several companies, each with different formats and approval policies.

The idea was to replace 167+ if statements with specialized agents. I got it; the agents could decide which workflow applied.

They already had logs, metrics, and distributed traces. By “traditional standards”, they were in “good shape”.

But does this traditional telemetry tell you if the agents made the right decisions?

Here is the thing: the moment you sprinkle AI magic over a beautifully deterministic workflow, the rules change. The very same input can take a different path, skip a critical step, and still return 200 OK.

And your life as a backend engineer changes with it.

My recommendation was simple: don’t put this thing in production until you have a way to record and observe those decisions.

And that is what this article is about.

---

Once you PR review like this, you’ll never go back (free tool)

AI is writing more and more code than ever before in history.

So why are you still reviewing PRs like it’s 2010: alphabetical order, zero context, no high-level summary of why changes were made?

CodeRabbit’s Change Stack restructures PR reviews into layered walkthroughs, giving you…

• Approachable PR overviews with one-click fixes

• Timeline View to see how the PR evolved over time (and why)

• Semantic Diff to understand real logic changes

And if you need help? Just ask the Agent Chat for context directly where you’re working.

---

From one workflow to five agents

The existing workflow was boring, and that was probably for the best. It extracts fields, matches the purchase order, verifies approvals, and schedules payment.

So when they broke the process into specialized agents, the whole idea ended up looking like this:

Intake reads documents and identifies the legal entity. Validation matches purchase orders. Approval loads a policy and checks approvals. Payment schedules approved invoices. The Supervisor chooses what runs next.

Each agent has one job, but one agent’s output becomes the next agent’s input. And if an early decision is wrong, the workflow can follow the wrong path without crashing.

That’s what we learn as soon as we expose this setup to real data; not production yet, but real data.

Here is an example:

Everything starts with the wrong policy

The parent company and the acquired one use different approval policies:

Parent company:   Below $25,000 requires one approval
Acquired company: Above $10,000 requires two approvals

Invoice INV-1042 belongs to the acquired company and is worth $12,400. The only problem is that a shared invoice template shows the parent company’s brand, so Intake selects the wrong legal entity.

And of course, from there, approval loads the parent company’s policy, finds one approval, and approves the invoice:

{
  "detected_legal_entity": "parent-company",
  "policy_version": "parent-company-v3",
  "required_approvals": 1,
  "recorded_approvals": 1,
  "decision": "approved"
}

Every component follows the context it receives. Payment gets a clean instruction too.

{
  "invoice_id": "invoice-1042",
  "approval_decision": "approved",
  "amount": 12400
}

There wasn’t any big crash here. But the reality is that the acquired company required a second approval, so it should not have scheduled a payment until the second required approval was recorded.

This was only caught because we had all eyes on testing, and it was found days after the problem happened while accounting was matching invoices.

That was exactly the moment we realized that traditional observability wouldn’t help with this; we needed a new standard to understand what happened, how agents interacted, which tools they called, and how the decision was made.

Traditional observability stops at execution

We had logs, metrics, and distributed traces. They could see that the payment API accepted the request, measure latency and errors, and follow a request through the service, database, and payment provider.

But many of these things only explain execution, not the decisions introduced by agents.

For this system, the trace also needed to answer:

• Which agents participated?

• In what order did they run?

• Which tools did each agent select?

• Which legal entity did Intake select?

• Which entity owns the purchase order?

• Which approval policy did Approval load?

• What decision reached the Payment Agent?

• Which model and prompt version influenced the path?

We need the observable decisions: agent and tool calls, classifications, policy selections, handoffs, and the final business outcome.

You probably already use OpenTelemetry for traces; it provides a vendor-neutral telemetry model.

OpenTelemetry has already been working on this, and the GenAI semantic conventions give us a common language for observing models, agents, workflows, and tools.

OpenTelemetry’s vocabulary for GenAI

The GenAI semantic conventions add vocabulary for models, agents, workflows, tools, and evaluations. They are still in progress, and not every library or observability platform supports the same fields yet, but it's still the best I have found so far.

Here, invoke_workflow coordinates the process, invoke_agent runs an agent, chat calls a model, and execute_tool records a tool.

The official agent and workflow conventions define those operations. The model and tool conventions cover inference and tools.

The semantic conventions define many attributes, but here is a list of my favorites.

My favorite attributes and why

You can capture many GenAI attributes, but these are the ones I would start with.

I separate them into four groups. Together, they tell us who participated, what happened, and which version influenced the result.

So far so good, but you have work to do. These conventions define what the attributes mean, but they do not guarantee that your framework will emit them.

So go and choose one owner for model-call spans. If a validated instrumentation library already creates compatible spans, use it. Otherwise, instrument the model-client boundary yourself.

Then add an integration test that makes a single logical inference call and verifies that the trace contains exactly one model span.

One important thing: retries may create several HTTP spans, but each logical inference operation should still have one GenAI model span.

On the other hand, everything under invoice.* must come from our code because only the application understands that business context.

I like to start with automatic instrumentation, inspect a complete trace, and manually add what is missing.

Now, what does this look like in our case?

In simple terms: GenAI attributes describe activity. Application attributes connect it to the business decision.

span.name = invoke_agent intake-agent
gen_ai.operation.name = invoke_agent
gen_ai.agent.name = intake-agent
gen_ai.agent.version = 2.1.0
invoice.entity.detected = parent-company

Approval loads a policy:

span.name = execute_tool get_approval_policy
gen_ai.operation.name = execute_tool
gen_ai.agent.name = approval-agent
gen_ai.tool.name = get_approval_policy
gen_ai.tool.call.id = call-9a42
invoice.approval.policy.entity = parent-company
invoice.approval.required_count = 1
invoice.approval.recorded_count = 1
invoice.approval.decision = approved

Now gen_ai.* identifies the operation, while invoice.* explains why its result matters.

With both layers connected, the team could instrument the rest of the workflow.

Instrumenting the multi-agent workflow

Think of the trace as an execution tree. The workflow owns the run, the supervisor owns routing, and each specialist sits below it, with model and tool calls below the specialist.

In .NET, Activity.Current creates that parent-child relationship. A small StartInternal wrapper around ActivitySource.StartActivity can pass ActivityKind.Internal and the initial tags:

static readonly ActivitySource Source = new("Invoice.AgentWorkflow");

static Activity? StartInternal(
    string name,
    params KeyValuePair<string, object?>[] tags) =>
    Source.StartActivity(
        name,
        ActivityKind.Internal,
        Activity.Current?.Context ?? default,
        tags);

using var workflow = StartInternal(
    "invoke_workflow invoice_processing",
    new("gen_ai.operation.name", "invoke_workflow"),
    new("gen_ai.workflow.name", "invoice_processing"),
    new("invoice.processing.id", processingId));

using var supervisor = StartInternal(
    "invoke_agent supervisor-agent",
    new("gen_ai.operation.name", "invoke_agent"),
    new("gen_ai.agent.name", "supervisor-agent"));

The Supervisor span must record more than “I ran.” You need to capture important decisions:

supervisor?.AddEvent(new ActivityEvent(
    "invoice.supervisor.route",
    tags: new ActivityTagsCollection
    {
        { "invoice.route.target_agent", "approval-agent" },
        { "invoice.route.reason_code", "approval_required" }
    }));

Here we couldn't find a stable GenAI attribute to describe this route, so we have to use the application-owned invoice.* namespace.

Now start Approval while the Supervisor remains active, and start its policy tool inside that scope:

using (var approval = StartInternal(
    "invoke_agent approval-agent",
    new("gen_ai.operation.name", "invoke_agent"),
    new("gen_ai.agent.name", "approval-agent")))
{
    using var policy = StartInternal(
        "execute_tool get_approval_policy",
        new("gen_ai.operation.name", "execute_tool"),
        new("gen_ai.agent.name", "approval-agent"),
        new("gen_ai.tool.name", "get_approval_policy"),
        new("gen_ai.tool.call.id", toolCallId),
        new("gen_ai.tool.type", "datastore"));

    var result = await policyService.Get(entity, amount);
    policy?.SetTag("invoice.approval.policy.entity", result.Entity);
}

First, you add the operation name when you create the span.

The sampler decides whether to keep the trace at that point, so anything added later cannot affect that decision. Once the operation finishes, you add the result.

Some tips:

• If the call crosses a service boundary, pass the trace context so it stays connected.

• Use gen_ai.provider.name only when calling a hosted AI agent.

• A call to a payment provider is just a regular HTTP or RPC call, so instrument it like any other service dependency.

Once the trace mirrors the execution tree, you can define what a safe trace should contain.

Start with the question the trace must answer

Before creating spans, decide what you need to understand from a single invoice run.

In this workflow, the important question is not simply, “Did the payment request succeed?”

We need to know whether every agent used the same business facts:

• Which legal entity did Intake detect?

• Which entity owns the purchase order?

• Which entity owns the approval policy?

• Did the recorded approvals satisfy that policy?

• What did the policy gate decide?

• Did the system create a payment request after that decision?

That gives us a clear safety invariant:

can_pay =
    detected_entity == authoritative_entity
    AND policy_entity == authoritative_entity
    AND approvals_satisfy(policy_version)

The policy gate enforces this invariant synchronously. If any condition fails, it records a denied authorization decision and stops the workflow before it creates a payment intent or contacts the payment provider.

At the same time, the trace has a different job. It records the facts used by the gate, its decision, and a stable reason code:

invoice.authorization.result = "denied"
invoice.authorization.reason_code = "entity_mismatch"
invoice.approval.policy.version = "v42"

A denied payment is not automatically an incident. It may prove that the guardrail worked. We can monitor denial rates and investigate unusual increases, but the condition that should trigger a serious alert is a bypass:

authorization.result != "allowed"
AND payment.intent.created == true

An evaluation can later judge whether the agents kept the entity and policy consistent across the workflow. It helps us test behavior and detect patterns, but it does not replace the synchronous policy gate.

Now that we know what the trace must explain, we can turn those production questions into spans and attributes.

Traces explain. Evaluations judge.

OpenTelemetry defines a gen_ai.evaluation.result event with a name, label, score, and explanation.

For this workflow, emit a deterministic evaluation:

event.name = gen_ai.evaluation.result
gen_ai.evaluation.name = entity-policy-consistency
gen_ai.evaluation.score.label = pass | fail
gen_ai.evaluation.score.value = 1 | 0

This check does not need another model. If a rule can be a boolean, boring, and deterministic, start there.

You can run it in tests, during rollout, and on sampled production traces.

That said, evaluations provide a quality signal, but still observability cannot authorize a payment. The system still needs deterministic enforcement below the agents.

Put guardrails below the agents

Observability can explain why an agent wanted to pay an invoice, but it shouldn’t authorize the payment. That decision must happen synchronously before we create the request to the payment provider.

The Payment Agent can recommend an action, but it must pass a typed contract to a deterministic policy gate:

public sealed record PaymentProposal(
    string InvoiceId,
    string DetectedLegalEntity,
    ApprovalRecommendation Recommendation);

Notice what is missing: the authoritative entity, approval count, and policy version.

The agent does not own those values. The policy gate must load them from the system of record.

We also need to keep the database transaction short. Do not call a model, remote service, or payment provider while the transaction is open.

If the invoice, policy, and approvals live in the same database, the flow can look like this:

await database.ExecuteTransaction(async () =>
{
    var snapshot =
        await database.LoadAuthorizationSnapshot(
            proposal.InvoiceId,
            authorizationTime);

    var allowed =
        proposal.DetectedLegalEntity == snapshot.LegalEntity &&
        snapshot.ApprovalIds.Count >= snapshot.RequiredApprovals;

    var authorization = PaymentAuthorization.Create(
        invoiceId: proposal.InvoiceId,
        policyVersion: snapshot.PolicyVersion,
        policyEffectiveAt: snapshot.PolicyEffectiveAt,
        approvalIds: snapshot.ApprovalIds,
        recommendation: proposal.Recommendation,
        allowed: allowed);

    database.PaymentAuthorizations.Add(authorization);

    if (allowed)
    {
        var intent = PaymentIntent.Create(
            invoiceId: proposal.InvoiceId,
            authorizationId: authorization.Id,
            amount: snapshot.Amount,
            idempotencyKey: $"pay:invoice:{proposal.InvoiceId}:auth:{authorization.Id}");

        database.PaymentIntents.Add(intent);
        database.Outbox.Add(
            PaymentRequested.From(intent));
    }

    await database.SaveChangesWithExpectedVersion(
        snapshot.StateVersion);
});

SaveChangesWithExpectedVersion represents an optimistic concurrency check. If the policy or approvals change while we make the decision, the write fails and the application retries with a fresh snapshot.

In this workflow, any mismatch between the detected and authoritative entity requires review, even if the authoritative record and approvals would otherwise permit payment.

This protects the decision without holding a transaction open across slow network calls. The transaction only reads authoritative database state, evaluates a few rules, and writes the authorization, payment intent, and outbox message.

After the transaction commits, an outbox worker sends the request to the payment provider. It uses the payment intent’s stable idempotency key, so processing the outbox message more than once does not create another payment.

If the policy and approvals live in another service, do not try to hold a transaction open. Let the service that owns that state create the immutable authorization record and publish the authorized action through its own outbox.

The policy-gate span should record the decision code, policy version, and an opaque authorization ID. Keep approval IDs and other evidence in the durable authorization record, not in telemetry.

Simply put:

Agents can interpret invoices and recommend actions, but deterministic code must enforce the business rules, and the database must preserve proof of the decision.

Once that enforcement boundary is in place, metrics can show whether the system’s behavior changes over time.

Measure outcomes, not only activity

The GenAI metric conventions cover duration, inference calls, tools, and tokens.

They reveal whether a prompt tripled model calls, an agent created a retry loop, or tool latency broke the workflow deadline.

But again, activity by itself is not the outcome. Track evaluation pass rate, entity-policy mismatches, blocked payments, human reviews, and cost per safely processed invoice.

Watch each run. More tool calls may indicate retries; fewer may mean skipped checks. A lower pass rate after a version change should stop the rollout.

These signals make the system operable, but detailed AI telemetry can also copy sensitive invoice data into places it does not belong.

Do not turn observability into a data leak

Be careful here; invoices contain vendor, tax, banking, and employee data. Model and tool calls can copy it into telemetry.

And before adding custom attributes, define a small telemetry contract.

I like to use predictable values like the workflow, decision, policy version, and reason code for filtering and metrics.

Also, I like to use an opaque processing ID only to find a specific trace.

I recommend keeping invoice numbers, vendor details, and free-form model output out of metric labels. If you need that context for debugging, record a redacted version in sampled events or logs with stricter access controls.

Safe telemetry leads to the final operating model.

My observability baseline before production

Before these agents process an invoice, I want the team to answer yes to these questions:

• Does one trace connect the workflow, Supervisor, specialists, models, tools, handoffs, policy gate, and outcome?

• Can I identify the versions and verify the business state at each handoff?

• Can I compare latency, token usage, tool calls, and evaluation pass rate by version?

• Will alerts and sampling retain unsafe runs without exposing invoice data?

If the answer to any of these is no, the system is just not ready.

AI does not remove the old system design rules. It adds a decision-maker, so telemetry has to follow the decision.

That is how you survive it. Start with the official documentation below.

Official references

• GenAI conventions

• Agent and workflow spans

• Model and tool spans

• Metrics

• Events and evaluations

Until next time,
— Raul

---

System Design Classroom is a reader-supported publication. To receive new posts and support my work, consider becoming a paid subscriber.