AI production monitoring means watching a live AI feature so you notice when it gets worse, more expensive or unsafe, before your users tell you. In practice it comes down to a short list of signals (quality, latency, cost, errors, user behavior), an alert on each, and a named person who acts when one fires. You don't need a platform team for this. For most small products, a few well-chosen metrics and a weekly review beat a dashboard nobody reads.
I'm Hanad, a freelance senior engineer in Vienna. I build and rescue software products for founders and owner-operated businesses, and the same pattern keeps showing up: an AI feature ships, it demos well, and nobody knows whether it still works three weeks later. This article is what I would hand a founder or CTO in that spot.
Key takeaways
- Monitoring tracks chosen metrics and alerts on thresholds. Observability lets you investigate why something went wrong. You need the first on day one and the second soon after.
- Pre-launch testing can't predict real use. NIST says controlled evaluations cannot account for dynamic real-world use or nondeterministic outputs, so monitoring after deployment is part of the product.
- Start with five signal groups: quality, latency, cost per request, errors and user behavior. Add drift checks once you have real traffic.
- Every alert needs an owner and a documented next step. A metric without a decision attached is decoration.
- A lean version can be built in a few weeks and doesn't need a vendor suite.
Table of Contents
- What AI production monitoring actually means
- Why a launched AI feature needs watching
- The signals worth tracking
- Monitoring LLM features and agents
- What to do when an alert fires
- A lean setup that works for small teams
- My perspective: monitor less, but act on all of it
- Getting this built
- FAQ
- Sources
What AI production monitoring actually means
Traditional software fails loudly. A server crashes, an endpoint returns a 500, a page won't load. AI features fail quietly. The model answers, the response is fluent, the status code is 200, and the answer is wrong, stale or ten times more costly than last month.
Monitoring is the practice of choosing a set of measurements and alerting when they cross a line. It answers: is something wrong right now?
Observability is broader. Microsoft's AI in Production guide describes it as using logs, metrics, traces and other telemetry to understand the state of a system, and for AI it extends to the model, the data, the infrastructure and the code, so you can find root causes. Monitoring tells you the fire alarm is ringing. Observability lets you walk the building and find the room.
For a small team the practical difference is this: monitoring is a handful of dashboards and alerts. Observability is the stored traces, prompts, retrieved documents and versions that let you replay a bad answer and see what happened. Build the first, and log enough to grow into the second.
Why a launched AI feature needs watching
There are three reasons I give clients.
The real world is not your test set. NIST's 2026 report on monitoring deployed AI systems argues that post-deployment monitoring is needed to validate reliability, detect drift and unforeseen outputs, and identify consequences that only emerge once a system is integrated into changing contexts. Users ask things you never imagined, and your data changes underneath you.
Models and inputs move. Google's guidance notes that a static model can go stale when the relationship between features and labels changes, and that dynamic training needs continuous monitoring and the ability to roll back a faulty job or corrupted input data. For LLM features the equivalent is a provider updating a model version, or your knowledge base drifting out of date.
Cost is a failure mode. A feature that works but burns money on every request will kill a small business's margin. The Singapore government's MLOps Playbook recommends tracking compute or inference cost per request, storage use and cloud spend per model or service, with budgets and alerts as the response.
Australia's Digital Transformation Agency puts the wider case in its technical standard for government AI: monitoring after deployment is critical for reliability, availability, performance, security, safety and compliance. That is a public-sector document, but the list is a good checklist for any product.
The signals worth tracking
The official guides list dozens of metrics. Nobody needs all of them. Here is how I group them for a product with real users and a small team.
| Signal group | What to measure | Why it matters |
|---|---|---|
| Quality | Sampled output scores, human review rate, thumbs up/down, regression test pass rate | Catches silent degradation |
| Latency | p50, p95 and p99 response time, time to first token | Slow answers get abandoned |
| Cost | Tokens and dollars per request, per user and per feature | Protects your margin |
| Reliability | Error rate, timeouts, provider rate-limit hits, retries | Shows outages and capacity limits |
| User behavior | Abandonment, retries, edits, task completion | Shows what quality scores miss |
| Data and drift | Input distribution, missing values, knowledge-base freshness | Early warning before quality drops |
Quality
This is the hardest signal and the one most teams skip. Google's monitoring guidance suggests assessing live quality with human-generated labels, statistical bias checks, real-world metrics such as user-reported problems, and gradual traffic shifts to a new model version. You rarely have perfect ground truth in production, so use proxies.
Google's productionization guidance gives a clear proxy example: if the share of mail users manually move to spam jumps from 0.5% to 3%, the spam model probably has a problem. The same logic works for your feature. If users start regenerating answers, copying less, or escalating to support more, something changed.
I wrote a longer piece on evaluation metrics that stop regressions if you want the offline side of this: a fixed set of test cases you re-run on every prompt or model change, so production monitoring only has to catch what testing couldn't.
Latency and reliability
Google's list of operational signals includes API response-time percentiles and queries per second, plus memory and training speed for teams running their own models. If you call a hosted model, the equivalents are p95 latency, timeouts and the rate-limit errors your provider returns. Rate limits in particular surprise founders at the worst moment, usually right after a successful launch. I covered how to check, handle and raise OpenAI rate limits separately.
Cost
Log input and output tokens on every call, tagged with the feature, the user or account, and the model version. OpenTelemetry's generative AI conventions are working toward standardizing attributes such as model parameters, response metadata and token usage, which is worth following if you want portable instrumentation instead of locking yourself to one vendor's format.
Then set a budget alert per feature. A single runaway loop in an agent can cost more in an afternoon than the feature earns in a month.
User behavior
AWS recommends combining automated quality metrics with explicit feedback and implicit behavioral signals such as abandonment and retries, plus business outcomes like conversion, resolution time and customer satisfaction. I like this because it ties the AI feature to something the business already cares about. If you have no analytics function, my note on AI feature adoption metrics for small teams shows a minimal way to start.
Data and drift
Google recommends validating data against expected schemas and distributions, comparing training and serving data, and tracking model age, alerting when a model passes an age threshold. If you use a hosted LLM with retrieval, translate that into: is the knowledge base still current, are documents failing to parse, and are queries landing on topics your content doesn't cover?
NIST organizes the whole field differently, into functionality, operational, human-factors, security, compliance and large-scale-impact categories. That is a useful lens if you sell to regulated customers and need to show a structured approach.
Monitoring LLM features and agents
A classic ML model gives you a number. An LLM feature gives you text, often after several hidden steps. That changes what you log.
For each request, store:
- The user input, after removing or masking personal data.
- The final prompt, with a version identifier for the template.
- Any retrieved documents and their IDs.
- The model name and version, and its parameters.
- The output, token counts, latency and cost.
- Any user feedback attached later.
With that record you can replay a complaint and see which step failed. Without it, you are guessing.
For agents, AWS's AgentOps guidance says telemetry should expose decision accuracy, latency, cost and reliability, and that version histories for prompts, policies and model interactions support diagnosis and rollback. Its generative AI lifecycle guidance structures monitoring around application health, business metrics, model quality, drift detection, feedback loops and security controls. AWS also recommends continuous agent evaluation against benchmarks specific to your use case, with dashboards, alerts and correlation to change events.
That last point is underrated. When quality drops, the first question is always "what changed?" If you log every prompt, model and config change as an event on the same timeline as your metrics, the answer is often visible in minutes.
Hallucinations deserve their own line item. Retrieval helps, but it doesn't solve the problem alone, and I explain why in reducing AI hallucinations in production. For monitoring, the practical move is to sample a small percentage of answers each week and have a human grade them against the source documents.
What to do when an alert fires
Alerts only help if they lead somewhere. NVIDIA's guide treats monitoring as an ongoing operational responsibility, not a one-time deployment task, and recommends a documented path from alert to troubleshooting to action. The Australian standard says the same in plainer terms: evaluate continuously, detect anomalies, intervene, and promptly remedy outputs that don't match the system's intended purpose.
Here is the runbook I set up for small teams. It fits on one page.
- Name the owner. One person is paged for each alert type. For a small company that is often the founder.
- Set severity. Wrong answers in a medical, legal or financial context are urgent. A slow response is not.
- Contain. Have a switch ready: fall back to a simpler prompt, turn off the feature for one account, or route to a human.
- Diagnose. Pull the stored trace, check the change timeline, and compare against the last good version.
- Fix and validate. Add the failing case to your regression set so it can't return unnoticed.
- Record. Write two lines on what happened and what you changed.
AWS warns that feedback is only useful when linked to specific agent behaviors and versions, routed by severity, and followed through identification, root-cause analysis, improvement, implementation and validation. Step 5 is that loop in miniature. For features where a wrong answer is costly, a review step before the output reaches the customer is often cheaper than any amount of after-the-fact monitoring. I describe how to build one in shipping a human-in-the-loop MVP.
A lean setup that works for small teams
You can get most of the value without buying a full observability suite. This is the order I would build in:
Week 1: log everything per request. Structured logs with prompt version, model, tokens, latency, cost and a request ID. Mask personal data at this point, not later.
Week 2: dashboards and alerts. One page with error rate, p95 latency, daily spend and feedback ratio. Alert on spend per day, error rate and a sudden drop in positive feedback.
Week 3: quality sampling. Pull 20 to 50 random responses a week into a review sheet. Score them against a simple rubric. Track the trend.
Week 4: regression set and release gates. Turn every bad case you've found into a test. Run the set before any prompt or model change goes live.
I've written a more detailed version of this plan for engineers in shipping LLM monitoring in four weeks. If you aren't sure where your existing feature stands, the AI audit checklist is a quicker first step: it shows what is missing before you spend anything.
A few honest notes on tooling. Hosted monitoring platforms are fine when you have a lot of traffic and a team to read them. For a small product, structured logs in your existing database plus a simple dashboard often do the job, and they keep your customer data where you control it. If you serve EU customers, where traces and prompts are stored is a real GDPR question, so decide it up front rather than after the first customer asks.
My perspective: monitor less, but act on all of it
I've seen two failure modes. The first is no monitoring at all: the feature ships, nobody looks, and the first signal is an angry email or a surprise invoice. The second is a dashboard with forty charts, built in a burst of enthusiasm and abandoned within a month.
What works is smaller. I pick a handful of signals tied to what the business cares about, make sure each has an owner and a next step, and review them on a fixed rhythm. I also design the code so the next developer can read it, which matters for monitoring more than most things: logging added as an afterthought is usually the first thing a new engineer rips out.
When I rebuild a prototype from a tool like Lovable or Bolt into real code, the AI calls are usually the least visible part. There are no logs, no cost tracking and no way to replay a bad answer. Adding those during the build costs little. Retrofitting them after an incident costs far more. And if an off-the-shelf tool covers your need, I'll tell you so rather than sell you a build.
— Hanad Kubat
Getting this built
If you have an AI feature in production, or about to be, and want monitoring built in properly, I do this as fixed-price work. I'm one engineer: you speak to the person writing the code, you see working progress before you pay, and you own the code from the first commit. Builds start from €12,000, with a two to four week turnaround. See what I offer and get in touch at hanadkubat.com.
FAQ
What is the difference between AI monitoring and AI observability?
Monitoring watches chosen metrics and alerts when they cross a threshold. Observability keeps enough telemetry, such as logs, traces and versions, to investigate why something happened. Start with monitoring, and log richly enough that observability is possible later.
Which metrics should I start with?
Start with error rate, p95 latency, cost per request, a sampled quality score and a user feedback signal. These five cover most of what goes wrong in an LLM feature. Add drift and data-quality checks once you have steady traffic.
How do I measure quality without ground truth labels?
Use proxies. Track user behavior such as retries, abandonment and edits, collect thumbs up or down, and have a human grade a weekly random sample. Google's own example uses a rise in user-flagged spam as a signal that a model has degraded.
Do I need a dedicated tool for AI monitoring?
Not at the start. Structured logging, a simple dashboard and budget alerts cover a small product. Consider a dedicated platform when traffic, team size or customer requirements make manual review impractical.
Sources
- NIST: Challenges to the monitoring of deployed AI systems
- Microsoft: AI in Production Guide, Observability
- Google: Production ML systems, Monitoring pipelines
- Google: Production ML systems, Static versus dynamic training
- Google: Productionization
- Singapore MLOps Playbook: Model monitoring and observability metrics
- Australian DTA: Technical standard for government use of AI
- OpenTelemetry for Generative AI
- AWS: Performance monitoring for generative AI
- AWS: AgentOps lifecycle focus areas
- AWS: Evaluate and track ongoing agent performance
- AWS: Feedback loops for agents