Series II: From infrastructure to agentic projects in production. Episode 14 of 20.
The fictional B2B portal in this series has passed its release checks. Yet a user may still see a stuck request while the server responds normally. For an SME, the next decision is not buying the most elaborate dashboard. It is choosing signals that reveal the impact on users and enable someone to intervene. We propose minimum observability, initially applied to one important workflow.
Observability starts with a question
Observability is the ability to understand a system's internal state from the signals it produces. Telemetry is the data collected about its operation. The useful question is “Why has this request not finished?”, not “How many charts do we have?”.
We extend the requests and documents portal through a hypothetical scenario. An uploaded document enters a processing queue. A worker picks it up and, in a proposed pilot, an agent integrated into the application prepares a summary for human review. This is not the coding agent that modifies the repository, the project's code store. The first performs a product function; the second helps the team build it.
Three signals, one request
A log records an event: processing started, failed, or finished. A metric is a numerical measurement tracked over time. A trace connects operations belonging to the same execution across services. A span is one operation within that trace, with a start, end, and descriptive attributes.
Correlation means associating this evidence through shared identifiers. For the portal, we connect request receipt, queue publication, processing, and calls to the model or other services. Context must also travel through asynchronous messages processed later; otherwise, the path appears broken. A tracing identifier does not replace access checks.
Google's Dapper technical report, published in 2010, is a reference for tracking distributed operations. It does not establish our portal's performance. The practical lesson is that isolated service timings do not always explain the waiting time of the complete workflow.
OpenTelemetry is not the entire system
OpenTelemetry provides tools and conventions for producing, collecting, and exporting telemetry. By itself, it is not the storage, search, and alerting solution. We separately choose where data goes, who can access it, and what retention costs.
Instrumentation means adding mechanisms that emit signals. We start with the web service and document worker, then add stages relevant to users. We record application version and environment to compare behavior before and after the controlled release described in the previous episode.
When checked on 15 October 2026, the OpenTelemetry generative AI conventions in their dedicated repository are marked “Development”. We do not treat every name and field as a finalized interface. We pin instrumentation library versions and test compatibility after upgrades. If an integration does not provide a measurement, we show it as unavailable, not zero.
The portal's minimum dashboard
Our starting proposal combines workflow outcomes with technical explanations. A successful web response may confirm document acceptance, not completed processing.
The team's question | Proposed signal |
|---|---|
Does the user receive the result? | Completed, failed, and pending requests |
Where is time being lost? | Total duration, stage durations, and oldest message age |
What is the AI component doing? | Call duration, errors, retries, and available token usage |
Can we trust our visibility? | Resource utilization, export errors, and unexpected telemetry gaps |
A token is a unit of text processed by a model; reported usage is not automatically an invoice. Cost calculation requires the provider's terms and usage context. For durations, we can track p95, the percentile below which approximately 95% of observations fall within the chosen interval. We also display observation counts: a low-traffic interval does not reliably describe future load.
We avoid email addresses and individual request identifiers in metric labels. They can increase cardinality, the number of distinct label combinations, making collection and analysis more expensive. Details needed for investigation belong in controlled logs and traces.
In our scenario, a delayed summary does not immediately justify switching models. A trace might show that the document waited in the queue while the AI call was brief. We then investigate worker capacity, not prompt optimization. If duration increases after a release, we filter by installed version and compare similar requests, without confusing coincidence with a demonstrated cause.
Before extending coverage, we also trigger a known error in an isolated environment. We check that the event appears in a log, is counted in a metric, and can be linked to its trace. This tests whether instrumentation explains the actual problem, not merely whether it sends data.
An alert must lead to action
As a strictly hypothetical example, the team could test an alert when documents have been pending for more than ten minutes and that condition persists for five minutes. These thresholds illustrate the mechanism; they are neither a service commitment nor universally suitable values. We adjust them to user needs and portal measurements.
The designated operational owner receives the alert with the environment, interval, observed impact, and a link to response instructions. They check whether the worker is running, messages are being consumed, and the external dependency responds. They do not automatically restart every operation: some retries can duplicate actions.
We test the rule in staging, the preproduction verification environment, by stopping the worker under controlled conditions. We confirm notification delivery and alert resolution after recovery. An alert without an available recipient and clear action is merely noise.
Visibility without copying documents
For this workflow, we propose an explicit allowlist: service, version, operation type, duration, outcome, and error category. We do not send documents, prompts, complete responses, credentials, or authentication headers to the observability platform by default.
We also inspect data emitted automatically by libraries. Sensitive information must be removed before transmission to external storage, not merely hidden in a screen. We restrict access and choose retention according to purpose and risk. Replacing an identifier with a hash does not guarantee anonymization.
Sampling retains only some traces to control volume. A missing trace does not prove that an execution never happened. We document what is sampled and separately monitor collection health so that a telemetry failure does not look like perfect operation.
Successful execution is not a correct answer
An agent can finish quickly without technical errors while omitting an obligation from a document. We keep quality evaluation separate, using verified examples and acceptance criteria discussed in the agent evaluation episode. Telemetry supports investigation; it does not independently establish the summary's truth.
The usable outcome is a worksheet describing the monitored workflow, collected signals, prohibited fields, dashboard, and alert owner. The development agent can propose instrumentation and tests; the team checks the data actually exported. The next episode adds backup, recovery, and incident response, for situations where diagnosis must be followed by service restoration.
Sources
Sources checked on 15 October 2026:
- OpenTelemetry, Signals, Traces, and Metrics, for signals, spans, context propagation, and cardinality.
- OpenTelemetry, Generative AI semantic conventions, for their scope and Development status.
- OpenTelemetry, Handling sensitive data, for minimization and the limits of hashing for anonymization.
- Google, Site Reliability Engineering, Monitoring Distributed Systems, for monitoring and useful alerts.
- Sigelman et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, 2010, a research technical report on distributed tracing.
Want to understand where errors and delays arise in your applications? Talk to i8 about observability, protected data, and actionable alerts.
