When Agents Can Ask for More: Dynamic Collection and Root Cause Accuracy
Your AI SRE agent doesn't need you to collect everything. It needs a way to ask for more data when an incident starts, by reconfiguring your collectors at the edge.
+145%
more root causes found with Telflo
34% → 83% of trials(55 → 135 of 162)
Claude Code, HolmesGPT (an open-source AI SRE agent in the CNCF Sandbox) and our custom loop agent together, across all nine incidents.
- Claude Codev2.1.310+213%15 → 47 of 54
- HolmesGPTv0.45.0+105%22 → 45 of 54
- Custom loop agentwritten for this test+139%18 → 43 of 54
432 trials, none excluded or re-run, including two comparison setups.
What we measured
A trial passes only if the agent's top answer names the right service and says why it failed.
Alert
Checkout p95 latency above 5 s
Service
Shipping
Not enough to pass
Root cause
Shipping quotes for international orders take 5 seconds
Passes
Without Telflo, agents usually named the right service but missed the cause
Named the right service
Found the root cause
How it works
- Break something: A fault in the OpenTelemetry Demo web shop, under load.
- Raise the alert: It points at the symptom, not always the cause.
- Investigate (the only difference):
- Without Telflo: Reads what the baseline kept.
- With Telflo: Can also turn collection up, then look again.
- Answer: Names the root cause and explains it.
- Grade: Passes only if the service is right and GPT-5 agrees on the cause.
9 incidents, 6 trials each, with and without Telflo, for each agent.
Both start from the same baseline. It keeps:
- 100% of error traces
- 100% of traces slower than 1.5 s
- 1% of other traces
- 100% of warning and error logs
- Request, error and latency metrics
- Container CPU and memory
Why it's built this way
- A fair starting point
- The baseline still keeps every error trace, every trace slower than 1.5 seconds, and every warning and error log.
- One difference
- Same model, prompt, read tools, faults and backend with and without Telflo. Only with Telflo can the agent change the collector's configuration.
- No shortcut
- No setup's telemetry records which fault was switched on, so no agent can simply read the answer.
- A grader from another vendor
- GPT-5 judges each answer's cause against a written rubric, three votes each.
- Agents people actually run
- Claude Code and HolmesGPT are two widely used agents with their own prompts. Our custom loop agent is deliberately minimal.
Trial by trial
| Claude Code | HolmesGPT | Custom loop agent | ||||
|---|---|---|---|---|---|---|
| Incident | Without Telflo | With Telflo | Without Telflo | With Telflo | Without Telflo | With Telflo |
| International shipping slowdownINFO logs | 0 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 5 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 5 of 6 found the root cause, 6 of 6 named the right service |
| Small international shipping delayINFO logs | 0 of 6 found the root cause, 5 of 6 named the right service | 5 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 5 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 4 of 6 found the root cause, 6 of 6 named the right service |
| Database lock contentionDatabase metrics, INFO logs | 0 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 2 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service |
| Garbage-collection pausesJVM metrics, INFO logs | 5 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 4 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service |
| One slow productProduct ID attribute | 0 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service |
| One CPU-starved replicaHost name attribute | 2 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 4 of 6 found the root cause, 6 of 6 named the right service | 4 of 6 found the root cause, 5 of 6 named the right service | 2 of 6 found the root cause, 4 of 6 named the right service | 4 of 6 found the root cause, 6 of 6 named the right service |
| Payment service unreachableError traces (kept) | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service |
| Cart store unreachableERROR logs (kept) | 2 of 6 found the root cause, 2 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service | 4 of 6 found the root cause, 4 of 6 named the right service | 6 of 6 found the root cause, 6 of 6 named the right service |
| Shipping slowdown, already overINFO logs, never kept | 0 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service | 0 of 6 found the root cause, 6 of 6 named the right service |
| Total | 15/54 | 47/54 | 22/54 | 45/54 | 18/54 | 43/54 |
Show the counts
| Incident | Claude Code: Without Telflo | Claude Code: With Telflo | HolmesGPT: Without Telflo | HolmesGPT: With Telflo | Custom loop agent: Without Telflo | Custom loop agent: With Telflo |
|---|---|---|---|---|---|---|
| International shipping slowdown | 0/6 (6) | 6/6 (6) | 0/6 (6) | 5/6 (6) | 0/6 (6) | 5/6 (6) |
| Small international shipping delay | 0/6 (5) | 5/6 (6) | 0/6 (5) | 6/6 (6) | 0/6 (6) | 4/6 (6) |
| Database lock contention | 0/6 (6) | 6/6 (6) | 2/6 (6) | 6/6 (6) | 0/6 (6) | 6/6 (6) |
| Garbage-collection pauses | 5/6 (6) | 6/6 (6) | 4/6 (6) | 6/6 (6) | 6/6 (6) | 6/6 (6) |
| One slow product | 0/6 (6) | 6/6 (6) | 0/6 (6) | 6/6 (6) | 0/6 (6) | 6/6 (6) |
| One CPU-starved replica | 2/6 (6) | 6/6 (6) | 4/6 (6) | 4/6 (5) | 2/6 (4) | 4/6 (6) |
| Payment service unreachable | 6/6 (6) | 6/6 (6) | 6/6 (6) | 6/6 (6) | 6/6 (6) | 6/6 (6) |
| Cart store unreachable | 2/6 (2) | 6/6 (6) | 6/6 (6) | 6/6 (6) | 4/6 (4) | 6/6 (6) |
| Shipping slowdown, already over | 0/6 (6) | 0/6 (6) | 0/6 (6) | 0/6 (6) | 0/6 (6) | 0/6 (6) |
Found the root cause, out of 6 trials. In parentheses: named the right service.
Cost
82% less
to store telemetry with Telflo than to collect everything all the time
- Collect everything100%
- With Telflo18%
An estimate at Datadog list prices for ingesting, indexing and keeping logs and traces for 15 days, at 20 incidents a month. It assumes extra collection is turned back down after 30 minutes; in the trials, agents never turned it down. Telemetry only: model spend and Telflo's own price aren't included.
Details
All results
| Agent and setup | Root cause found | 95% interval | Right service | All 3 of 3 right | At least 1 of 3 right | Time | Tool calls |
|---|---|---|---|---|---|---|---|
| Claude Codev2.1.310 | |||||||
| Without Telflo | 28%15/54 | 18%–41% | 91%49/54 | 17% | 40% | 41 s | 10.9 |
| With Telflo | 87%47/54 | 76%–94% | 100%54/54 | 83% | 89% | 195 s | 16.9 |
| HolmesGPTv0.45.0 | |||||||
| Without Telflo | 41%22/54 | 29%–54% | 98%53/54 | 27% | 53% | 101 s | 19.1 |
| With Telflo | 83%45/54 | 71%–91% | 98%53/54 | 74% | 89% | 273 s | 26.2 |
| Custom loop agentwritten for this test | |||||||
| Without Telflo | 33%18/54 | 22%–47% | 93%50/54 | 24% | 42% | 65 s | 17.9 |
| With Telflo | 80%43/54 | 67%–88% | 100%54/54 | 66% | 89% | 186 s | 19.6 |
| Fixed rule | 63%34/54 | 50%–75% | 100%54/54 | 52% | 72% | 59 s | 15.9 |
| Collect everything | 98%53/54 | 90%–100% | 98%53/54 | 94% | 100% | 43 s | 11.9 |
All nine incidents. Time and tool calls are means per investigation.
Collect everything: every trace, log and metric, all the time. Fixed rule: when the alert fires, collect everything from the service it names.
Two more setups for comparison
Collect everything. Every trace, log and metric, all the time. Nothing is dropped.
Fixed rule. The baseline, plus one rule: when the alert fires, collect everything from the service it names. No agent decides anything.
- Collect everything98%
- With Telflo80%
- Fixed rule63%
- Without Telflo33%
Collecting everything costs 443% more than with Telflo, and finds the root cause in 98% of trials against 80%. Only collecting everything found the cause of the incident that was already over: 6 of 6.
The fixed rule finds the cause in only 1 of 12 trials when it is one service away from the alert: the two shipping incidents, where the alert fires on checkout.
| Incident, custom loop agent | Collect everything | Fixed rule |
|---|---|---|
| International shipping slowdown | 6/6 (6) | 0/6 (6) |
| Small international shipping delay | 6/6 (6) | 1/6 (6) |
| Database lock contention | 6/6 (6) | 5/6 (6) |
| Garbage-collection pauses | 6/6 (6) | 6/6 (6) |
| One slow product | 6/6 (6) | 6/6 (6) |
| One CPU-starved replica | 6/6 (6) | 4/6 (6) |
| Payment service unreachable | 6/6 (6) | 6/6 (6) |
| Cart store unreachable | 5/6 (5) | 6/6 (6) |
| Shipping slowdown, already over | 6/6 (6) | 0/6 (6) |
Found the root cause, out of 6 trials. In parentheses: named the right service.
Hypotheses we wrote down before the run: 5 of 6 held
Written before the run for the first six incidents, so their numbers differ from the rest of the page. Verdicts are for our custom loop agent.
Held
AI SRE agents with Telflo find the root cause more often
Predicted: With Telflo, the agent finds the root cause more often than without it (one-sided Fisher p < 0.05).
Result: 31 of 36 with Telflo against 8 of 36 without, p < 0.0001.
Claude Code and HolmesGPT: held (35 against 7, and 33 against 10, of 36).
Didn't hold
Telflo is as accurate as collecting everything
Predicted: With Telflo, the agent finds the root cause in at most one fewer trial than collecting everything, on each of the six incidents.
Result: Within one trial on four incidents. Two trials short on the small shipping delay and the CPU-starved replica (4 of 6 against 6 of 6). In those misses the agent made the right change but described the cause too loosely.
Held
Telflo costs at least 30% less than collecting everything
Predicted: Over a month, the baseline plus short bursts of extra collection during incidents costs at least 30% less than collecting everything all month.
Result: 42% less, as written before the run (model spend included). Storing telemetry alone at Datadog list prices: 82% less.
Held
Changes stay narrow
Predicted: In at least 5 of 6 trials per incident, the agent turns collection up only for services it names, and never makes a destructive change.
Result: 6 of 6 on every incident. No destructive change in any trial.
Claude Code and HolmesGPT: held, with one fleet-wide change each.
Held
Telflo can't recover the past
Predicted: On an incident that ended before the investigation, collecting everything beats the baseline, and Telflo doesn't.
Result: Collect everything 6 of 6, without Telflo 0 of 6, with Telflo 0 of 6. By the time collection was turned up, there was nothing left to collect.
Claude Code and HolmesGPT: Telflo didn't help them either (0 of 6 with and without); they didn't run collect everything.
We're experimenting with a small memory store on the agent that may help here, and will benchmark it once it's released.
Held
A fixed rule only helps when the alert names the culprit
Predicted: Collecting everything from the alerting service beats the baseline when the alert names the culprit, and not when the cause is one service away.
Result: 21 of 24 against 8 of 24 when the alert names the culprit (p < 0.001). 1 of 12 against 0 of 12 on the two shipping incidents, where the alert fires on checkout.
The incidents
| Incident | Alert the agent gets | Where the cause shows | What happens |
|---|---|---|---|
| International shipping slowdown | Checkout p95 latency above 5 s | INFO logs | Shipping quotes for international orders take 5 seconds. The alert fires on checkout, one service away, and only an INFO log line says which orders are slow. |
| Small international shipping delay | Checkout p99 latency above 800 ms | INFO logs | The same fault at 1 second, so the slow requests also stay under the 1.5-second slow-trace cutoff. |
| Database lock contention | Product catalog p95 latency above 1 s | Database metrics, INFO logs | Product lookups queue behind an exclusive lock on the products table. |
| Garbage-collection pauses | Ad service p95 latency above 500 ms | JVM metrics, INFO logs | The ad service stalls in repeated full garbage collections. All three agents usually inferred it from the latency pattern even without that evidence (15 of 18 trials without Telflo). |
| One slow product | Product catalog p99 latency above 300 ms | Product ID attribute | Lookups of one product are 0.35 seconds slower than the rest, and stay under the slow-trace cutoff. |
| One CPU-starved replica | Product catalog p99 latency above 1 s | Host name attribute | One of two replicas is cut to 2% of a CPU core. Without host names, the two are hard to tell apart (agents still found it in 8 of 18 trials without Telflo). |
| Payment service unreachable | Checkout error rate above 50% | Error traces (kept) | Checkout can't reach the payment service. |
| Cart store unreachable | Cart error rate above 20% | ERROR logs (kept) | The cart service can't reach its store and logs it at ERROR. The misses ranked the store above the cart service. |
| Shipping slowdown, already over | Checkout latency, resolved 4 minutes ago | INFO logs, never kept | The international shipping fault, switched off before the investigation starts. |
In the first six, the cause is visible when collecting everything and invisible on the baseline; we checked each before the run, with no agent involved. In the next two nothing is hidden, and the last is over before the investigation starts. The shop is the OpenTelemetry Demo 3.1.0.
Agents, grading and statistics
Claude Code 2.1.310. Anthropic's coding agent, with its own system prompt and its built-in tools turned off, so it had the same tools as every other agent.
HolmesGPT 0.45.0. An open-source AI SRE agent built by Robusta, now a CNCF Sandbox project. It ran with its own system prompt and its built-in toolsets and shell turned off, so it had the same tools as every other agent.
Custom loop agent. A deliberately minimal agent we wrote for this test, limited to 40 tool calls or 25 minutes. It's the only agent that ran all four setups, so the headline uses it.
Every agent ran Claude Sonnet 5.5 and got the same task: the alert, the collector fleet it runs on, and the answer format, ranked root causes with how each happened. It read traces, logs and metrics through Grafana's MCP server, and with Telflo it could also use Telflo's.
Grading. A trial passes when the top-ranked root cause is the right service and GPT-5, on a best-of-three vote, finds that the answer states what went wrong. Each incident has a written rubric: a correct cause stated with a hedge passes, and a list of options that doesn't pick one fails. Where nothing is hidden, only the service is scored. The grader agreed with all 67 answers we labeled by hand before the run, and with a fixed keyword check on 157 of the custom loop agent's 168 verdicts.
Statistics. Rates come with 95% Wilson intervals. With and without Telflo are compared with a one-sided Fisher's exact test, per incident and pooled over all nine incidents, with no correction for multiple comparisons. Pooled, p < 0.0001 for each agent and for the three together. Trials ran in random order: Claude Code Oct 25–26, 2026 (108 trials), HolmesGPT Oct 26–27, 2026 (108 trials), the custom loop agent Oct 24–25, 2026 (216 trials).
How we modeled cost
The cost of storing telemetry for one fleet like the benchmark's, 730 hours and 20 incidents a month, at Datadog's annual list prices: every log and span collected is ingested and indexed with 15-day retention. Metrics are priced at a flat rate per sample, since Datadog prices them per host and time series. No plan allowances. Extra collection is priced for 30 minutes per incident at the rate the trials measured; in the trials it stayed on a median of 2.5 to 2.9 minutes before the agent answered. At 10 to 120 minutes per incident, Telflo stays 81% to 82% cheaper than collecting everything. Against the same telemetry without Telflo, it costs 2% more. Model spend and Telflo's own price aren't included.
Limits
- We built what we tested
- Telflo is the setup under test. We designed, ran and graded this benchmark ourselves, and wrote down the hypotheses and analysis before the run. Read the results with that in mind.
- Telflo can't recover the past
- Turning collection up only captures what happens next. When the fault had ended before the investigation began, agents with Telflo found the cause in 0 of 18 trials. Collecting everything found it 6 of 6 times. We're experimenting with a small memory store on the agent that may help here, and will benchmark it once it's released.
- One application
- Every trial ran on the OpenTelemetry Demo. Real systems are bigger and noisier, with more services to suspect.
- One model
- Every agent ran Claude Sonnet 5.5. Other models may use Telflo differently.
- Six trials per incident
- Differences of one or two trials on a single incident are within noise. The pooled results are not.
- Cost is modeled, not billed
- It prices measured telemetry volumes at Datadog list prices and assumes extra collection ends 30 minutes into each incident; in the trials, agents never turned it down. Your volumes, prices and incident rate will differ.
- An AI grader
- The score comes from a model. It agreed with all 67 answers we labeled by hand before the run, but it can still be wrong.
Found an error? Email [email protected] and we'll publish the fix here.
Let your agent turn collection up when it needs to
Telflo's MCP server lets an agent read, change, validate and deploy your collector configuration.