More teams are trying to bring AI into SRE and operations by building an AI on-call assistant. The engineering goal is clear: when production breaks, the assistant should pull logs, compare code changes, query traces, and locate root causes faster than a tired human on-call engineer. But in systems with high alert volume, where every alert can trigger multi-step investigation, a common risk is that the API bill spikes after a single ordinary incident. The problem is usually not that the model is too weak. It is the default architecture intuition: wire monitoring webhooks straight into agents, one alert one agent task. In real production systems, failures are rarely isolated.
Imagine a common failure scene. You run a microservice system. One day a database connection-pool misconfiguration causes a short timeout. Within the next 90 seconds, that root failure fans out: the API gateway starts returning 502s, the async queue backs up, Kubernetes liveness checks fail, and APM latency alerts fire at the same time. If your system starts one agent investigation for every alert, fifty related alerts can become fifty independent agent tasks in a few minutes.
Two bad outcomes follow. First, tool-call and trajectory explosion: the dominant cost of an agent is often not one prompt, but many tool calls such as shell commands, log queries, and git lookups. Fifty independent agents mean fifty long trajectories burning compute at once, easily producing thousands of API calls in minutes. Second, fragmented diagnosis: those fifty agents do not know about each other. They produce fifty local explanations for gateway 502s, queue backlog, and pod restarts, while missing the real root cause in the database connection pool.
In traditional operations, a Chicago Trading Company case on PagerDuty’s customer page says Event Intelligence automatically grouped alerts so a burst of 50-200 alerts fell to 5-10. That is a customer case, not a universal compression ratio. On 2026-07-28, Datadog published a more extreme version of the same economic structure in its AI Security Detection Pipeline. According to that official blog, Mambark is an approximately 96.9M-parameter Mamba model that selects about 10,000 candidates a day out of about 10 billion security events, then hands the shortlist to a frontier agent for investigation. By candidate count, that is six orders of magnitude; the production scale and cost numbers remain vendor-reported and independently unverified.
If you only read that sentence, it is easy to think this is just another cascade, or a cheap small model doing generic text classification. The design is more specific. Mambark is not answering free-form questions about individual logs. It first learns event sequences with self-supervised next-event prediction and uses negative log-likelihood as a surprise score; Datadog also says it fine-tunes briefly per downstream source to classify which events deserve agent attention. If you try to put a generic text small model on a raw log stream, the first obstacle is task shape, not FLOPs: with no human query, a single log line has no inherent good/bad label, so the model does not know what to look for. The real intuition is to treat the log stream as a system state sequence and use next-event prediction as an unsupervised anomaly score. When the observed event diverges from the normal evolution of the system, it is promoted as high-surprise. That is why low-cost screening at billion-event scale becomes plausible.
This design shrinks the expensive agent’s investigation set from the full stream to a shortlist. It supports a core judgment: under high fan-in streaming events, deciding what deserves investigation matters more, and more structurally, than making each agent thought slightly cheaper. In After AI subsidies retreat, agents start pricing intelligence per dollar and The most expensive model in your agent pipeline may sit in the wrong seat, we discussed model placement and cost control inside the agent. Pre-agent filtering is the gate that sits outside the agent.
When engineers hear information retrieval and two-stage filtering, the first association is often a query-driven dense retriever such as DPR: start from a question, search a vector index for related text. Not every vector method requires a human query. The real distinction is that DPR optimizes query-to-corpus retrieval, while event-stream anomaly detection optimizes item-to-history scoring. In massive logs and security streams, nobody is continuously asking questions, so the system cannot invent a query vector for the unknown next failure.
The fix is to turn the first stage into language-model perplexity or next-event probability. That idea goes back at least to DeepLog at ACM CCS 2017 from the University of Utah. DeepLog treats system logs as language and uses an LSTM to predict the next log event distribution; if the actual log key is outside the model’s top-g candidates, it is treated as a sequence anomaly.
From 2017 to 2026, the paradigm went through three important iterations:
Small language models find a clear engineering role here: not answering questions or calling tools, but scoring next-event probability as a high-throughput streaming anomaly filter.
Once the unsupervised sequence-prediction intuition is clear, a practical pre-agent filter has three implementation pieces.
First, template tokenization and log preprocessing. Do not send raw logs through a generic BPE tokenizer. Timestamps, IPs, and hash variables explode into meaningless fragments and waste compute. Prefer template parsers such as Drain to turn unstructured lines into fixed event templates and event IDs. That turns a log stream into a discrete event sequence and reduces prediction complexity at the source.
Second, base model choice and self-supervised fine-tuning. Parameter
count does not need to be large. A prototype can start from Hugging Face
checkpoints such as state-spaces/mamba-130m-hf;
Hugging Face also documents LoRA examples. If inputs are Drain event
IDs, you still need an event vocabulary and adapted embedding/output
heads. Training time should be measured against data volume, sequence
length, and hardware rather than assumed to be trivial.
Third, choose thresholds by event volume and investigation cost. Treat daily volumes around -, -, and as rough capacity-planning starting points, not hard gates. Whether to add a filter layer, and whether to use rules, shallow classifiers, or a small sequence model, should be decided from per-investigation cost, prevalence, miss cost, and investigation capacity. When volume is low and the strong model only produces one batch digest, calling a frontier model directly is often simpler. When every candidate triggers multi-step agent investigation and the shortlist can be held to about 10%-20%, cheap triage or rule prefiltering starts to have financial room. Only when the cheap API pass itself becomes a major cost and volume stays higher does a dedicated small-model filter become worth the engineering.
Builders may wonder why materials such as Anthropic’s agent guide, OpenAI’s agent developer track, and Cursor’s harness evolution rarely expose this filter as a reusable component. In these three guides, the unit of discussion is mostly a task or session already inside the system. At least these materials do not abstract high fan-in stream admission control as a general component. One possible reason is that filtered false negatives never enter ordinary agent traces, so current evals struggle to observe them. That is an explanation, not a publicly confirmed design motive from those vendors. Security, AIOps, fraud, and content moderation already have isomorphic architectures; they have long lived under vertical names and have not yet become a first-class primitive in horizontal agent engineering.
As model prices fall, it is tempting to believe that admission filtering will matter less. In agent workloads with multi-round tool calls, total cost usually depends on model price, token volume, trajectory length, and external tool fees at once; high fan-in especially multiplies repeated trajectories. Without an entrance filter, unfiltered trajectories still produce oversized bills and fragmented diagnosis. In KV cache hit rate: the first cost lever of agent inference, we discussed prefill reread optimization inside the agent. Pre-agent filtering is the gate outside the agent. Deciding what is worth thinking about before the agent sees the input is the most durable engineering moat under high fan-in event streams.