TRANSCRIPT
Ryan Donovan: Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I’m your host, Ryan Donovan. And today we’re talking about agents. What are they building in there? And how to find out. My guest for that is Yanbing Li, who is the chief product officer at Datadog. So, welcome to the show, Yanbing.
Yanbing Li: Thank you for having me, Ryan.
Ryan: My pleasure. Before we figure out what those pesky agents are doing, can you give us a little flyover of how you got into software and technology?
Yanbing: I feel like I’ve been in software forever, for the most part of my life. I would characterize my education as studying until there is no more school to go to. So I have a PhD in electrical and computer engineering. Then I’ve been an engineer ever since, except at Datadog, where I’m far more focused on product than combinations of product and engineering. My introduction to observability started with my time at Google, seven years ago, when I was leading the observability effort at Google — both for powering Google’s infrastructure and services, and as a product within Google Cloud. Then I had an interesting detour into everything AI, where I was building self-driving trucks at a company called Aurora. Aurora today is probably still not just the first, but the only company operating driverless trucks commercially on U.S. public roads. That was the experience of absolutely needing to understand everything an AI agent is doing. Granted, self-driving started before the ChatGPT LLM days, but there are definitely a lot of parallels in thinking about what it takes to fully unleash your agent into the wild.
Ryan: Observability is one of those things that’s really ramped up a lot. Datadog used to be a metrics and dashboard company, but now it’s fully part of the observability space. And everybody’s trying to figure out how to apply observability to agents. How are you thinking about keeping an eye on agents when they’re running?
Yanbing: As you said, observability has existed for a long time for really complex distributed systems. That part remains the same. But with AI agents, certain things are new and different. Starting with AI models playing an instrumental role in bringing reasoning and intelligence into AI agents — now we deal with the non-deterministic nature of AI models. That tends to be the first thing that comes up when we talk about AI observability. We started with LLM observability, really to understand what the LLM is doing. But now that is quickly expanding from just observing the AI models and their interactions to understanding the AI agent as a whole. If we think about an AI agent, it’s still fundamentally a distributed system — it’s just that this distributed system is now powered by intelligent models underneath. Certain things remain the same, but we have to understand the new nuances introduced by AI.
Ryan: The observability of LLMs is still a sort of ongoing research project, as I understand it, particularly around explainability. Is that right?
Yanbing: There are certainly things at the research frontier, but I also think a lot is already in production. The fact that we rely on AI agents in day-to-day use and in enterprise settings is an indication that LLM observability — or agent observability — in combination with other things, is giving us enough confidence to depend on these agents. And it is a journey. I work with lots of enterprises that are still in this early phase: can I trust my AI agent not just to stay up, but to do the right thing? I’ve seen lots of successful agents, but when I speak to our customers, many more are still in the process of building that trust. Given the rapid pace of the AI landscape, how we do this is going to keep evolving.
Ryan: The LLM was: you give it an input, it gives a response. Now there’s multi-step turn taking, there’s tool use. What are the pieces you have to observe in an AI agent?
Yanbing: Certainly we started with understanding prompts and LLM output. But because agents are becoming distributed systems in nature, it’s getting a lot more complicated. We need to understand the reasoning, the tool calls, the prompts and responses, and we need to understand the behavior — why it’s making certain decisions. We also need to understand what the broader implication is across applications and outcomes. We’re definitely evolving from simply observing software health to truly observing software behavior.
Ryan: The comparison to network software is interesting. I’ve been saying that AI agents are sort of speed-running the microservices path. For network software, observability has had three or four pillars: logging, metrics, traces, and events. Do those traditional pillars apply to AI software?
Yanbing: I think they do still apply, but more things are emerging in the realm of agent observability. We’re observing additional characteristics. The traditional pillars are still important — from what I see in our customer base, AI has generated tremendous demand for observability, and a lot of that is through the bread and butter of microservices and distributed system observability: metrics, traces, logs. That remains very foundational. But then you add additional layers to uniquely understand model interactions and agent behavior. The need for observability is growing in both areas.
I was actually looking at some of our own agent observability product statistics to see what kinds of errors it catches. For example, it catches a large chunk of errors related to hitting rate limits on API calls and data access. This has nothing to do with agents per se — it’s a pre-existing problem — but it becomes much more pronounced in agents given the increased speed and scale of data access. A lot of things remain the same, but there are also these new primitives we’ve been discussing.
Ryan: With AI agents, it feels like you can automate more failures.
Yanbing: Certainly there is token limit, but it’s often just regular rate limiting. A lot of software was designed for human or programmatic access, and agents are exposing rate limits at a much higher rate. We also constantly receive requests from customers to raise rate limits in our systems. This is something old, but it’s new in a different way.
Ryan: When you think about observing agents beyond standard failures — incorrect tool calls, PII exposures, things that are largely outside of traditional network systems — are you applying observability to those kinds of issues?
Yanbing: Yes, we definitely are. When we started doing agent observability, the initial focus was much more on the LLM call itself — the prompt and response. That has now expanded into the full trajectory of calls and loops. The sophistication of agents has increased tremendously, and all of those things are now things we observe.
I also think simply having telemetry and signals is not sufficient on its own. The key question is how that translates into an understanding of what isn’t working and where you need to improve. The contrast between a traditional system and an LLM system is significant. In traditional cloud software, you deploy, you trust production, you observe, and you go back and fix bugs as they arise. But for AI agents, there’s a much stronger argument for continuous monitoring, because a lot of what can be verified in a traditional system simply cannot be exhaustively verified in an AI agent system.
In traditional software, that feedback loop is mostly about bug fixing, understanding production load, or understanding customer behavior. In the world of AI, all of those still apply — but no matter how much you test and evaluate during the build phase, the real world will always surprise you in many different ways. That continuous feedback loop and rapid iteration is becoming much more crucial. Agent observability is no longer just about monitoring production; it’s about how production data becomes a flywheel that helps you iterate upstream, constantly improving the outcome of your agent. And with AI now accelerating development itself, the expectation is that this loop becomes significantly faster.
Ryan: A lot of people are talking about loops and agents, and feeding production data back to the build phase. Are you thinking about automating that — mining signals from telemetry, surfacing them as tickets for builders, and then feeding that to an agent to automatically heal the software?
Yanbing: Software development has proven to be the dominant AI use case on the market, where people are spending most of their investment. We’re definitely seeing AI accelerate software development, and that acceleration is applying to the observability feedback loop as well. The vision is exactly what you described: production signals informing automated remediation and faster iteration cycles — closing the gap between detecting a problem and fixing it, with AI participating in that process at every step.
