Deploy AI-Powered Business Solutions

An architect does not monitor “the agent” as a single box. Every solution has layers: the conversation experience, the orchestration and tools, the model, the grounding data, and the platform it runs on. Each layer has a different tool that sees it best. Exam scenarios usually describe a symptom and ask which tool or which change gets you to the root cause fastest.

Monitoring tools by layer

Key agent metrics

Know what each metric means and, more importantly, what a bad value usually points to. In Copilot Studio, a session becomes engaged when it triggers a custom topic or enters a system topic such as Escalate, Fallback, or Conversational Boosting. Resolution, escalation, and abandonment are measured against engaged sessions.

Analyze the backlog and user feedback

Feedback is only useful when it becomes prioritized work. A practical loop looks like this:

1. Collect: thumbs up/down reactions, CSAT surveys, free-text comments, transcripts, escalation reasons from human representatives, and support tickets about the agent.
2. Cluster: group issues by theme (missing knowledge, wrong answer, failed action, tone, latency). AI-based tools help here—Copilot Studio surfaces themes and improvement suggestions from analytics, and you can use generative AI to summarize and categorize large volumes of transcripts or comments.
3. Size: weight each cluster by volume and by business impact (for example, revenue-bearing or compliance-sensitive flows first).
4. Fix: add or correct knowledge, refine instructions, add a topic or tool, adjust the model or retrieval settings, or route to a human by design.
5. Verify: rerun your regression test set and evaluations, then watch the same metric after release.

Interpret telemetry: symptom to fix

Generative AI is non-deterministic, so testing shifts from “exact expected output” to “output meets criteria.” You still need traditional functional tests for flows, connectors, and integrations, but you add evaluation sets scored by metrics, often using an AI judge plus human review.

Test process for agents

Test types and metrics

Foundry evaluations

Microsoft Foundry evaluation applies evaluators to a test dataset, to responses generated from test inputs, or to production traces. Results show aggregate scores and per-row explanations so you can compare versions.

Validation criteria for custom AI models

When the solution includes a custom or fine-tuned model (for example a classification model or a fine-tuned small language model), define acceptance criteria before training starts:

• Baseline: compare against the current process, a rules-based approach, or the base model without tuning. A custom model must beat the baseline by enough to justify its cost.
• Holdout data: keep a test set the model never sees during training or tuning, drawn from the same distribution as production data. Never report results on training data.
• Precision: of the items the model flagged, how many were correct. Prioritize when false positives are expensive (for example, blocking legitimate invoices).
• Recall: of the items that should have been flagged, how many the model caught. Prioritize when misses are expensive (for example, fraud or safety cases).
• F1 score: the harmonic mean of precision and recall; a single number when both matter.
• Fairness and robustness: check performance across user segments, languages, and noisy inputs.
• Operational criteria: latency, cost per call, and model size relative to the deployment target.

Copilot prompt best practices

Good test prompts and good production instructions follow the same rules. A clear prompt includes a goal, context, expectations for format and tone, and sources to use. Beyond that:

• Be specific and give examples of the desired output.
• Tell the model what to do, not just what to avoid.
• Break complex work into steps and iterate in conversation.
• For agent instructions, state scope, refusal behavior, and when to hand off to a human.
• Version prompts and instructions with the solution so tests can be rerun against a known state.

End-to-end testing across Dynamics 365 apps

Many business processes cross app boundaries. Lead-to-cash may start in Dynamics 365 Sales, continue to an order in Supply Chain Management, and finish with invoicing in Finance; a service case may trigger a field service work order. An end-to-end test plan should:

• Follow a full business process from trigger to outcome, including the Copilot or agent steps inside each app.
• Use realistic, non-production data that is consistent across the apps (same customers, products, and IDs).
• Test the integration points: dual-write or virtual tables, Power Automate flows, connectors, and agent handoffs.
• Run under real security roles so each step proves the user or agent only sees the data it should.
• Include failure paths: unavailable connector, rejected approval, or a human taking over from the agent.
• Automate regression where possible (for example, the Regression Suite Automation Tool for finance and operations apps and Power Apps Test Engine for model-driven and canvas apps).

Generating test cases with Copilot

Copilot and other generative AI tools speed up test design: ask them to produce varied user utterances for each topic, edge cases, multi-turn scripts, adversarial prompts, and synthetic data records. Copilot Studio can also generate test sets for agent evaluation from the agent’s knowledge and topics. The architect’s strategy should include guardrails:

• Have subject matter experts review and approve generated cases and expected answers.
• Use synthetic or masked data; don’t paste production personal data into prompts.
• Mix generated cases with real (anonymized) user transcripts so coverage reflects how people actually talk.
• Store test sets with the solution so they version alongside the agent.

Application lifecycle management (ALM) governs a solution from idea through development, testing, deployment, operation, and retirement. For AI solutions, ALM has to cover four things together: the agent and app components, the models, the prompts and instructions, and the data that grounds or trains them.

Power Platform and Copilot Studio ALM building blocks

Environment strategy

Managed environments add governance on top: sharing limits, solution checker enforcement, usage insights, IP firewall and other security controls, and the ability to act as pipeline targets. Combine them with environment groups and rules to apply consistent policy across many environments.

ALM for Copilot Studio agents, connectors, and actions

• Create the agent inside a custom solution from day one; add required objects (topics, flows, custom connectors, environment variables) before export.
• Parameterize anything environment-specific: knowledge source URLs, API endpoints, Application Insights connection strings, and channel display name suffixes all belong in environment variables.
• Custom connectors and their connection references travel in the solution; credentials do not. Set up connections in each environment.
• Agent flows and Power Automate cloud flows used as actions should be solution-aware so they deploy with the agent.
• Add a pre-deployment approval or automated review (solution checker, evaluation gate) to the pipeline stage before production.
• After deployment, publish the agent in the target environment and confirm channel configuration and authentication settings.

ALM for Foundry Agent Service and custom models

ALM for data used in AI models and agents

• Treat grounding content and training data as versioned assets: know which version of a knowledge base or dataset an agent or model was tested with.
• Promote reference and configuration data with tools such as the Configuration Migration tool or data packages rather than manual entry.
• Use masked, synthetic, or subset data in non-production environments; respect the same residency and sensitivity rules as production.
• Automate index refresh and data quality checks, and include them in the pipeline so stale data doesn’t silently degrade answers.
• Record lineage so you can trace an answer or prediction back to the source data that produced it.

ALM for AI in Dynamics 365

Microsoft Responsible AI Standard: six principles

Security for agents

Governance with Microsoft Purview

• Sensitivity labels: Microsoft 365 Copilot and agents respect labels and permissions on content; labeled content keeps its protection, and responses can inherit the most restrictive label of their sources.
• Data loss prevention: Purview DLP can restrict how Copilot processes items with specific labels or sensitive information types.
• Audit: Copilot and agent interactions are captured in the unified audit log for investigation.
• DSPM for AI: Data Security Posture Management for AI shows how AI apps and agents use sensitive data, flags oversharing risks, and offers one-click policies.
• Retention, eDiscovery, and Communication Compliance: apply to Copilot prompts and responses for legal and regulatory needs.

Model security and AI vulnerabilities

Data residency and movement

Copilot Studio and Power Platform generative AI features rely on Azure OpenAI and related services that may not be hosted in every geography. For environments outside the United States, a Power Platform or Dynamics 365 administrator may need to allow data movement across regions in the environment’s Generative AI features settings in the Power Platform admin center. Points to know:

• The setting is per environment and requires administrator consent to the terms.
• Data that already moved while the setting was on isn’t pulled back by turning it off.
• Related settings, such as Bing search, depend on data movement being allowed where the checkbox is shown.
• For Foundry, choose the region and deployment type deliberately: global and data zone deployments can process data outside a single region, while regional (standard) deployments keep processing in that region.
• Document residency requirements during planning; they constrain which features and regions you can use.

Access controls on grounding and tuning data

• Ground on sources that enforce user permissions (SharePoint, Dataverse security roles, Azure AI Search with security trimming) so users only receive answers from content they can already access.
• Fix oversharing at the source before broad rollout; an agent surfaces whatever permissions already allow.
• Restrict who can read, change, or export tuning datasets with Azure RBAC, and keep them in storage with private networking and encryption.
• Separate duties: data stewards approve datasets; ML engineers train; release managers promote.

Audit trails for model and data changes

• Version and log every change to prompts, instructions, models, and datasets, including who approved it.
• Use Purview audit for Copilot and agent activity, Dataverse auditing for record and configuration changes, and Azure activity logs for resource changes.
• Keep evaluation results with each release so you can show the evidence behind a go-live decision.
• Route logs to a central workspace (Log Analytics or Microsoft Sentinel) for retention and investigation.

Weight 40–45% through November 15, 2026 Changing 20–25% from November 16, 2026