How to effectively use Application Insights for Copilot Studio agents

Someone in finance tells you the agent gave them the wrong purchase-order limit on Tuesday afternoon. They cannot remember what they typed, they have closed the chat, and the answer looked plausible enough that nobody noticed for two days. The Copilot Studio analytics page will tell you that sessions went up 8 per cent that week. It will not tell you which tool the agent called, what arguments it sent, what came back, or why the model preferred one knowledge source over another.

Application Insights will, provided you turned it on before Tuesday.

That caveat is most of this post. Agent telemetry is not retroactive, and the gap between “we should set that up” and “a user is waiting for an explanation” is where these investigations go wrong. What follows is the configuration I would put in place on a new agent, the way the data is actually shaped once it arrives, and the queries I keep saved because I have needed them more than once.

Application Insights is good at a specific set of questions: where the time went in a slow turn, which tool call failed and what it returned, whether an authentication step was ever reached, which conversations hit a content filter, and how often a given failure happens across a population rather than in the one conversation that got escalated. It is poor at questions about the model’s reasoning. The intermediate search queries the orchestrator generated, the knowledge chunks it retrieved and discarded, the ranking scores behind that choice: none of that is exported.¹ If your root cause lives there, you need conversation transcripts or a different architecture, and knowing that early saves you an afternoon of querying for something that was never written.

Two telemetry scopes, and they are not interchangeable#

This is the part the documentation splits across three pages, and getting it wrong is the single most common reason people conclude their telemetry is broken. Copilot Studio can send telemetry to Application Insights at two different scopes, and they emit completely different shapes into completely different tables.

If you configure agent-level telemetry and then run one of Microsoft’s environment-level example queries against dependencies, you get an empty result set and a bad hypothesis. The reverse is worse: environment-level telemetry does not capture topic events at all, so an investigation that depends on knowing which topic fired will quietly find nothing.

Environment-level is the better model for anything agentic. It follows the OpenTelemetry semantic conventions for GenAI, which means tool calls, models and sub-agent invocations are first-class spans rather than text you have to parse out of a message body. It is also in preview, only available on managed environments, and excludes declarative agents.² Most teams I have seen end up running both, for different audiences.

Turn it on before you need it#

Agent-level takes about two minutes. Open the agent, go to Settings › Advanced, and paste the connection string from your Application Insights resource’s Overview blade. Use the connection string, not the legacy instrumentation key.

Then there are four toggles, and they are not free.

The second row deserves a decision rather than a default. Turning on Log conversation details writes user identities and verbatim message text into a Log Analytics workspace that is probably governed by a different team, under a different retention setting, than the conversation transcripts in Dataverse. That may be entirely fine. It should still be a choice someone made on purpose, because the debugging value is real: without message text, “the agent gave a wrong answer” investigations stall immediately.

Environment-level needs more. It requires a managed environment, an export package created in the Power Platform admin center with the export type set to Copilot Studio, and local authentication left enabled on the target Application Insights resource. If it is disabled, export fails. If you want to query the data programmatically rather than through the portal, you also need an Entra ID app registration with the Data.Read delegated permission on the Application Insights API.

Where each kind of failure leaves a trace#

Application Insights has a fixed vocabulary of telemetry types, and Copilot Studio maps onto it in a way that is not obvious until someone tells you.

Requests are the coarse unit of work. For agent-level telemetry these give you session and message counts and are mostly useful for volume and trend questions, not for diagnosis.

Dependencies are where environment-level telemetry lives, and they are the interesting table. Every exported agent event is a span row with itemType of dependency and a type of GenAI. There are exactly three span names: InvokeAgent, ExecuteTool and OutputMessages.

Traces carry log lines. Real-time voice agents write their whole event stream here, keyed on customDimensions.Subject, and the top-level message field is a useless placeholder. The content is all in the custom dimensions.

Exceptions capture thrown errors. Content filter rejections surface as filtered events containing ContentFiltered, which is how you find responsible-AI blocks after the fact.

Custom dimensions are the part that matters most and the part people skip. Nearly every field worth querying in Copilot Studio telemetry lives inside the customDimensions JSON bag rather than in a native column. Agent-level puts channelId, fromName, text, TopicName, Kind and designMode there. Environment-level puts the entire OpenTelemetry GenAI namespace there: gen_ai.agent.name, gen_ai.conversation.id, gen_ai.request.model, gen_ai.tool.name, gen_ai.tool.call.arguments, gen_ai.tool.call.result, plus Status.code and error.type.

The structural idea underneath environment-level telemetry is worth twenty seconds of attention, because every query in this post depends on it. One agent turn is one trace, identified by a shared operation_Id. The InvokeAgent span is that trace’s root; the ExecuteTool and OutputMessages spans hang beneath it with operation_ParentId set to the root span’s id. A conversation is many turns, and the only thing threading them together is gen_ai.conversation.id.

Sub-agents complicate this. When an agent calls another agent as a tool, the sub-agent gets the parent’s conversation ID with a _<subConversationId> suffix appended, so reconstructing a multi-agent tree means splitting on _ and matching the root portion. Their spans also currently parent to the invoking InvokeAgent span rather than to the root inside their own trace, which makes the end-to-end transaction view look flatter than the real call graph.²

A conversation ID is the only thread worth pulling#

Every successful investigation I have run followed the same path, and the whole thing hinges on getting a conversation ID early. Everything else is filtering.

Start by narrowing time. “Tuesday afternoon” plus a channel and a user name is usually enough to find the session in customEvents, and once you have the session you have the identifier you need for everything downstream.

With an ID in hand, Transaction search is the fastest first look. Filter on the conversation ID and you get every operation in that conversation as a list, ordered in time.

Selecting any row opens the end-to-end transaction view, the Gantt-style timeline that shows each span nested under its parent with its duration as a bar. This is the view that answers “where did the time go” without a single query. A turn that took eleven seconds usually has one bar occupying nine of them, and that bar is your answer.

Then drill into the span itself and expand customDimensions. This is where the actual evidence is: the tool name, the arguments the agent constructed, the payload that came back, the status code and the error type. In a wrong-answer investigation this pane usually ends the argument, because you can see that the tool was called with a customer ID from the wrong field, or that it returned an empty array and the model narrated around it.

The telemetry you did not turn on before the incident is telemetry you do not have.

Six failures and where they show up#

Most of what gets reported falls into a handful of shapes. This is the lookup table I wish someone had handed me.

Slow responses. Compare sibling span durations within one turn before you look at anything else. Agent turns are rarely uniformly slow; one tool, one connector or one model call is usually responsible. If duration is empty, you are looking at a classic agent trace, where it is not populated.²

Failed tool calls. Read the next section carefully, because the obvious approach does not work.

Authentication problems. These are diagnosed by absence. An agent that cannot get a token often never produces an ExecuteTool span at all, so the turn looks suspiciously short and clean. If you only ever query for error rows, authentication failures are invisible to you. Check that the spans you expected are present before concluding nothing went wrong.

Downstream API failures. The status on the span may say the call succeeded while the payload in gen_ai.tool.call.result contains an error object the API returned with a 200. Read the payload, not the verdict.

Unexpected agent behaviour. Put gen_ai.input.messages next to gen_ai.output.messages for the turn. A surprising proportion of “the agent went off the rails” reports turn out to be a user message that genuinely was ambiguous, and seeing the exact input settles it. For agent-level telemetry the equivalent is checking which TopicName fired against which you expected.

Conversation failures. Responsible-AI filtering shows up as events containing ContentFiltered, and Copilot Studio error codes are embedded in the message text of BotMessageSend events rather than in a structured field, so extracting them means string surgery.

Queries worth saving#

Every one of these has earned its place by being needed twice. Save them as queries in the Application Insights workspace so the next person does not rebuild them under pressure.

The everyday entry point. You know the agent name, you want the most recent conversation, and you want every span in it ordered with roots before children.

let Window = 7d;
let AgentName = "Purchasing Assistant";
let LatestConvo = toscalar(
    dependencies
    | where timestamp > ago(Window)
    | where tostring(customDimensions["gen_ai.agent.name"]) == AgentName
    | where isnotempty(tostring(customDimensions["gen_ai.conversation.id"]))
    | top 1 by timestamp desc
    | project tostring(customDimensions["gen_ai.conversation.id"])
);
dependencies
| where timestamp > ago(Window)
| where tostring(customDimensions["gen_ai.conversation.id"]) == LatestConvo
| order by operation_Id asc, iff(name == "InvokeAgent", 0, 1) asc, timestamp asc
| project timestamp, name, id, operation_Id, operation_ParentId,
          duration, resultCode, customDimensions

Read it top to bottom as a story. Each operation_Id block is one turn; within a block, the InvokeAgent row tells you the turn started, each ExecuteTool row is a tool the agent chose, and OutputMessages is what the user saw.

The same thing when you already have an ID from a user report. Substitute and run.

let Convo = "<paste the conversation ID>";
dependencies
| where tostring(customDimensions["gen_ai.conversation.id"]) == Convo
| order by operation_Id asc, iff(name == "InvokeAgent", 0, 1) asc, timestamp asc
| project timestamp, name, id, operation_Id, operation_ParentId,
          duration, resultCode, customDimensions

Raw customDimensions gets unreadable quickly. This flattens the keys you actually look at into columns, which makes the result sortable and scannable.

let Convo = "<paste the conversation ID>";
dependencies
| where tostring(customDimensions["gen_ai.conversation.id"]) == Convo
| extend
    Operation  = tostring(customDimensions["gen_ai.operation.name"]),
    Agent      = tostring(customDimensions["gen_ai.agent.name"]),
    Model      = tostring(customDimensions["gen_ai.request.model"]),
    ToolName   = tostring(customDimensions["gen_ai.tool.name"]),
    ToolType   = tostring(customDimensions["gen_ai.tool.type"]),
    ToolArgs   = tostring(customDimensions["gen_ai.tool.call.arguments"]),
    ToolResult = tostring(customDimensions["gen_ai.tool.call.result"]),
    Channel    = tostring(customDimensions["microsoft.channel.name"])
| extend
    UserInput   = tostring(parse_json(tostring(customDimensions["gen_ai.input.messages"]))[0].parts[0].content),
    AgentOutput = tostring(parse_json(tostring(customDimensions["gen_ai.output.messages"]))[0].parts[0].content)
| order by operation_Id asc, iff(name == "InvokeAgent", 0, 1) asc, timestamp asc
| project timestamp, name, Operation, Agent, Model, ToolName, ToolType,
          ToolArgs, ToolResult, UserInput, AgentOutput, duration, Channel

Now the one that matters most, and the reason this post has a warning in it. Finding failed tool calls looks like it should be | where success == false. It is not.

dependencies
| where timestamp > ago(24h)
| where name == "ExecuteTool"
| extend
    ToolName   = tostring(customDimensions["gen_ai.tool.name"]),
    StatusCode = tostring(customDimensions["Status.code"]),
    ErrorType  = tostring(customDimensions["error.type"]),
    Detail     = tostring(customDimensions["Status.message"])
| where StatusCode == "2" or isnotempty(ErrorType)
| summarize failures = count(), example = take_any(Detail) by ToolName, ErrorType
| order by failures desc

Latency, broken down by span and tool, so you can see which specific tool is dragging the population rather than the one conversation you were shown.

dependencies
| where timestamp > ago(24h)
| where name in ("InvokeAgent", "ExecuteTool")
| extend ToolName = tostring(customDimensions["gen_ai.tool.name"])
| summarize
    calls = count(),
    p50 = percentile(duration, 50),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99)
  by name, ToolName
| order by p95 desc

Look at the gap between p50 and p95 rather than the p50 alone. A tool with a 400 ms median and an 11-second p95 is a timeout problem wearing a disguise, and it will be reported to you as “the agent is sometimes slow”.

Multi-agent setups need the sub-agent tree, which means matching the _-suffixed conversation IDs back to their root.

let Window = 7d;
let RootConvo = "<paste the root conversation ID>";
dependencies
| where timestamp > ago(Window)
| extend ConversationId = tostring(customDimensions["gen_ai.conversation.id"])
| where ConversationId == RootConvo or ConversationId startswith strcat(RootConvo, "_")
| extend
    Role     = iff(ConversationId == RootConvo, "root", "sub-agent"),
    Depth    = countof(ConversationId, "_"),
    Agent    = tostring(customDimensions["gen_ai.agent.name"]),
    ToolName = tostring(customDimensions["gen_ai.tool.name"])
| order by timestamp asc
| project timestamp, Role, Depth, Agent, name, ToolName, duration, operation_Id

Switching to agent-level telemetry: your first filter should almost always exclude the test canvas, because otherwise your own testing pollutes every aggregate you produce.

customEvents
| where timestamp > ago(7d)
| extend isDesignMode = tostring(customDimensions["designMode"])
| where isDesignMode == "False"

Microsoft’s own documentation is inconsistent about the casing of that key. The Copilot Studio page uses designMode, the Dynamics 365 guidance uses DesignMode. KQL bag lookups are case-sensitive, and a mismatch returns zero rows rather than an error, which is the worst possible failure mode for a filter. Check yours before you trust it:

customEvents
| where timestamp > ago(1d)
| mv-expand Key = bag_keys(customDimensions) to typeof(string)
| where Key contains "designmode"
| distinct Key

Copilot Studio error codes are buried in message text rather than exposed as a field, so counting them means substring extraction.

customEvents
| where timestamp > ago(7d)
| where name == "BotMessageSend"
| extend Text = tostring(customDimensions["text"])
| where Text contains "Error code:"
| extend
    errorStart = indexof(Text, "Error code:") + strlen("Error code:"),
    convoStart = indexof(Text, "Conversation ID:")
| extend ErrorCode = trim(" ", substring(Text, errorStart, convoStart - errorStart))
| summarize occurrences = count() by ErrorCode
| order by occurrences desc

And because the environment-level schema is still in preview and will move, the most valuable query in the set is the one that tells you what the schema is right now.

dependencies
| where timestamp > ago(7d)
| mv-expand Key = bag_keys(customDimensions) to typeof(string)
| summarize Events = make_set(name), Sample = take_any(tostring(customDimensions[Key])) by Key
| order by Key asc

Run that before you believe any field list, including the one in this post.

What makes the second investigation faster than the first#

The difference between a team that diagnoses an agent issue in ten minutes and one that spends a day on it is almost never KQL skill. It is preparation.

Correlate on one identifier and write it down. gen_ai.conversation.id is the join key across turns and across sub-agents, and if your support process captures anything from the user, it should capture that. A support ticket template with a conversation ID field is worth more than any dashboard.

Use the built-in views before building your own. Application Insights ships a Copilot Studio Dashboard workbook (Monitoring › Workbooks, then the gallery) covering conversations, latency, exceptions, tool usage and topic analytics. It opens as an editable workbook, so the sensible pattern is to open it, add a tile for the one thing your agent does that the template does not know about, and save it as your own. There are also Agents (preview) blades that read the dependencies table directly and give you agent runs, tools and models without any query at all.

Save queries rather than pasting them into chat. Anything in this post that you have run twice belongs in the workspace’s saved queries, named for the question it answers rather than the table it hits. “Which tools failed today” beats “ExecuteTool query v3”.

Alert on the things that are invisible. Nobody reports a tool that has been failing 4 per cent of the time for a month. Build the alert off Status.code and error.type, for the reason in the warning above.

Decide on sampling and retention deliberately. Node execution events in particular produce a lot of rows, and ingestion is billed by volume. Sampling that discards the exact exception you needed is a genuinely painful way to learn how sampling works, so if you enable it, understand what it drops before an incident rather than during one.

A few things to avoid, collected from watching them happen:

• Trusting success or resultCode on spans.
• Sending both telemetry scopes to one resource, then writing aggregates that double-count.
• Assuming environment-level telemetry replaces agent-level. It has no topic events.
• Debugging the pipeline during the first 24 hours after enabling export.
• Treating export as lossless. It is not transactional, and small gaps can occur during transient service events.

What I would set up on day one#

On a new agent, before it sees a real user: agent-level telemetry with logging and conversation details on, with the data-protection decision written down somewhere and agreed rather than assumed. Environment-level export as well if the environment is managed, because tool arguments and results are the difference between diagnosing a wrong answer and guessing at it. The Copilot Studio Dashboard workbook opened, saved as a copy, and given one extra tile for whatever this agent does that matters most. Four saved queries: latest conversation, trace by conversation ID, failed tool calls, latency percentiles.

The compromise I have not solved is the one at the top of this post. Full conversation logging is the setting that makes investigations possible and the setting that puts customer message text into a second system with its own retention clock. I turn it on, because an agent nobody can debug is worse, but I would not pretend that is a free choice or that it suits every organisation. If your data-protection position rules it out, know in advance that your wrong-answer investigations will run on structure rather than content, and set expectations with whoever will be asking.

The rest is practice. These queries are unremarkable once you have run them a few times, and the fluency is what makes the difference at the point where someone is waiting. Run them on a quiet Thursday against your own agent, while nothing is broken. That is much cheaper than learning the schema at the same time as you are learning what went wrong.