AI-300 Part 4: GenAI Quality and Observability

Part 4 of 5 — 10–15% of the AI-300 exam. This domain covers how to evaluate the quality and safety of generative AI outputs using Azure AI Foundry's built-in evaluators, how to write custom evaluators, how to gate CI/CD pipelines on evaluation scores, and how to instrument AI applications with OpenTelemetry for distributed tracing and observability.

Exam Objectives

The official skill areas for Domain 4:

Evaluate generative AI applications

  • Run evaluations using Azure AI Foundry built-in evaluators
  • Implement custom evaluators for domain-specific metrics
  • Integrate evaluation runs into CI/CD pipelines

Implement observability for AI applications

  • Instrument applications with OpenTelemetry for distributed tracing
  • Collect and analyse AI-specific metrics in Application Insights
  • Create alerting rules for quality and latency degradation

Evaluation Architecture

The diagram below shows how queries flow through the LLM, are scored by the evaluation layer, and feed into a monitoring dashboard.

Azure AI Foundry Built-in Evaluators

Azure AI Foundry provides a set of built-in evaluators through the azure.ai.evaluation SDK. These evaluators fall into two categories: AI-assisted evaluators that use a GPT-4 judge model to score responses on a 0–5 scale, and NLP-based evaluators that use deterministic text similarity metrics. The AI-assisted evaluators are more semantically accurate but incur additional cost because they make LLM calls to produce scores.

Evaluator Type Scale What it measures
GroundednessEvaluator AI-assisted 1–5 Is the response factually grounded in the provided context (not model knowledge)?
RelevanceEvaluator AI-assisted 1–5 Does the response directly answer the query asked?
CoherenceEvaluator AI-assisted 1–5 Is the response logically structured and easy to follow?
FluencyEvaluator AI-assisted 1–5 Is the language grammatically correct and natural-sounding?
SimilarityEvaluator AI-assisted 1–5 Semantic similarity between response and a ground-truth answer
F1ScoreEvaluator NLP (deterministic) 0.0–1.0 Token-level F1 overlap between response and ground-truth
ViolenceEvaluator Safety (AI-assisted) 0–7 (severity) Presence and severity of violent content in the response
HateUnfairnessEvaluator Safety (AI-assisted) 0–7 (severity) Hate speech or discriminatory content targeting groups
SelfHarmEvaluator Safety (AI-assisted) 0–7 (severity) Self-harm related content in the response
SexualEvaluator Safety (AI-assisted) 0–7 (severity) Sexually explicit content in the response

Running Evaluations with the SDK

The azure.ai.evaluation SDK provides an evaluate() function that accepts a dataset and a list of evaluator instances. The dataset must contain query, response, and (for groundedness) context columns. The function runs each evaluator across the dataset and returns aggregated scores.

from azure.ai.evaluation import (
    evaluate,
    GroundednessEvaluator,
    RelevanceEvaluator,
    CoherenceEvaluator,
    ViolenceEvaluator
)
from azure.identity import DefaultAzureCredential

# Evaluators that use GPT-4o as a judge require a model config
model_config = {
    "azure_endpoint": "https://my-openai.openai.azure.com/",
    "azure_deployment": "gpt-4o",
    "api_version": "2024-08-01-preview"
}

groundedness = GroundednessEvaluator(model_config=model_config)
relevance = RelevanceEvaluator(model_config=model_config)
coherence = CoherenceEvaluator(model_config=model_config)
violence = ViolenceEvaluator(
    azure_ai_project={
        "subscription_id": "<sub-id>",
        "resource_group_name": "rg-genai-prod",
        "project_name": "proj-chatbot-prod"
    },
    credential=DefaultAzureCredential()
)

results = evaluate(
    data="eval_dataset.jsonl",          # query, response, context columns
    evaluators={
        "groundedness": groundedness,
        "relevance": relevance,
        "coherence": coherence,
        "violence": violence
    },
    output_path="eval_results.json"
)

print(f"Groundedness mean: {results['metrics']['groundedness.groundedness']}")
print(f"Relevance mean:    {results['metrics']['relevance.relevance']}")

Exam tip: Groundedness measures whether the answer is factually supported by the provided context — not whether it is factually accurate based on the model's training data. A response can be factually correct but score low on groundedness if it relies on knowledge not present in the retrieved context.

Custom Evaluators

When the built-in evaluators do not capture a domain-specific quality dimension — for example, whether a response correctly cites a source document — you can implement a custom evaluator. A custom evaluator is a Python class with a __call__ method that accepts keyword arguments matching the dataset columns and returns a dictionary containing the metric name and a numeric score.

class CitationAccuracyEvaluator:
    """
    Custom evaluator: checks whether every claim in the response
    is backed by at least one document ID cited inline.
    Returns a score from 0.0 (no citations) to 1.0 (all claims cited).
    """

    def __call__(self, *, response: str, context: str, **kwargs) -> dict:
        import re
        # Count sentences and how many reference a [DocN] citation
        sentences = [s.strip() for s in response.split(".") if s.strip()]
        cited = sum(1 for s in sentences if re.search(r"\[Doc\d+\]", s))
        score = cited / len(sentences) if sentences else 0.0
        return {"citation_accuracy": round(score, 3)}

# Register alongside built-in evaluators in evaluate()
citation = CitationAccuracyEvaluator()

results = evaluate(
    data="eval_dataset.jsonl",
    evaluators={
        "groundedness": groundedness,
        "citation_accuracy": citation
    }
)

Custom evaluator scores are numeric floats in the range 0.0–5.0 (or 0.0–1.0 for proportion-based metrics). The dictionary key becomes the metric name in the evaluation results and can be tracked over time in Foundry's evaluation history view.

Exam tip: Custom evaluators must return a dictionary — the key is the metric name and the value is the numeric score. The __call__ method must accept keyword arguments that match the column names in your evaluation dataset.

Evaluation in CI/CD Pipelines

Quality gates in CI/CD pipelines prevent regressions when prompts or models change. The typical pattern is: a pull request changes a prompt template → CI pipeline runs the evaluation dataset against the new prompt → the pipeline fails if any metric drops below a defined threshold → the PR is blocked until the score recovers.

# GitHub Actions: evaluation gate step
- name: Run AI evaluation
  run: |
    pip install azure-ai-evaluation
    python run_eval.py \
      --dataset eval_dataset.jsonl \
      --threshold-groundedness 3.5 \
      --threshold-relevance 3.5

- name: Check evaluation thresholds
  run: |
    python check_thresholds.py eval_results.json \
      --fail-below groundedness=3.5 \
      --fail-below relevance=3.5
# check_thresholds.py — fail CI if scores drop below threshold
import json, sys, argparse

def check(results_path: str, thresholds: dict):
    with open(results_path) as f:
        results = json.load(f)
    metrics = results.get("metrics", {})
    failures = []
    for metric, threshold in thresholds.items():
        key = f"{metric}.{metric}"   # e.g. "groundedness.groundedness"
        score = metrics.get(key, 0)
        if score < threshold:
            failures.append(f"{metric}={score:.2f} (threshold {threshold})")
    if failures:
        print("EVALUATION FAILED:", ", ".join(failures))
        sys.exit(1)
    print("All evaluation thresholds passed.")

Distributed Tracing with OpenTelemetry

Distributed tracing connects a user request to every downstream service call it triggers — including LLM API calls, RAG retrieval queries, and agent tool steps. Azure Monitor (Application Insights) is the backend for trace storage and querying in Azure. The recommended instrumentation approach uses the Azure Monitor OpenTelemetry Distro for Python, which configures the OpenTelemetry SDK to export spans to Application Insights.

Instrumenting Azure OpenAI Calls

The opentelemetry-instrumentation-openai package automatically creates spans for every call to the Azure OpenAI client library. Each span captures the model name, token counts, latency, and prompt/completion content (if enabled). Spans are linked to the incoming request span via the W3C TraceContext propagation header, creating a full trace tree in Application Insights.

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from openai import AzureOpenAI

# Configure Application Insights as the trace/metric exporter
configure_azure_monitor(
    connection_string="InstrumentationKey=<key>;..."
)

# Auto-instrument all OpenAI client calls
OpenAIInstrumentor().instrument()

client = AzureOpenAI(
    azure_endpoint="https://my-openai.openai.azure.com/",
    api_version="2024-08-01-preview"
)

# All calls below are automatically traced — span created per call
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user",   "content": "Summarise the key points of this document."}
    ]
)

Key Span Types in AI Traces

  • LLM call span — one span per chat.completions.create call; attributes include gen_ai.request.model, gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, duration
  • RAG retrieval span — created manually or by an instrumented vector search client; captures the query, number of results, and retrieval latency
  • Agent step span — one span per tool call in an agent loop; captures tool name, input, and output
  • Root request span — the incoming HTTP request span; all AI spans are children of this root, making end-to-end correlation possible via a single trace ID

Observability Dashboard in Azure Monitor

Once traces flow into Application Insights, you can build Azure Monitor Workbooks that surface AI-specific metrics across your application. Key metrics to track:

  • Latency p50 / p99 — LLM call duration percentiles; alert when p99 exceeds SLA threshold
  • Token usage — prompt and completion tokens per day, segmented by model and deployment
  • Error rate — HTTP 429 (throttled) and HTTP 5xx rates from the LLM endpoint
  • Evaluation scores over time — groundedness and relevance scores from batch evaluation runs, tracked as custom metrics in Application Insights
# Push evaluation scores as custom metrics to Application Insights
from azure.monitor.opentelemetry.exporter import AzureMonitorMetricExporter
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider

exporter = AzureMonitorMetricExporter(connection_string="InstrumentationKey=<key>;...")
provider = MeterProvider()
metrics.set_meter_provider(provider)
meter = metrics.get_meter("ai-evaluation")

groundedness_gauge = meter.create_gauge(
    "ai.evaluation.groundedness",
    description="Mean groundedness score from evaluation run",
    unit="score"
)

# After evaluation run completes
groundedness_gauge.set(mean_groundedness_score, {"model": "gpt-4o", "flow": "support-chat"})

Exam tip: OpenTelemetry traces use correlation IDs (trace IDs) to link requests across services. A single user request that touches the application server, the RAG retrieval service, and the LLM endpoint will appear as one trace tree in Application Insights — this is how you diagnose latency spikes and failures end-to-end.

Exam Tips & Key Takeaways

Critical concepts for Domain 4:

  • Groundedness vs relevance — groundedness checks factual support in the provided context; relevance checks whether the response answers the question. Both are AI-assisted and score 1–5
  • AI-assisted evaluators cost money — they make LLM calls using GPT-4o as a judge; this incurs additional Azure OpenAI charges beyond the application's own LLM calls
  • Custom evaluator contract — must implement __call__ with keyword arguments matching dataset columns; must return a dict with a string key and float value
  • Safety evaluators use a 0–7 severity scale — not the 1–5 scale used by quality evaluators; a score of 0 means no harmful content detected
  • OpenTelemetry distro — configure_azure_monitor() sets up automatic export to Application Insights; OpenAIInstrumentor().instrument() auto-traces all OpenAI client calls
  • CI/CD quality gates — fail the pipeline by calling sys.exit(1) when evaluation scores fall below threshold; this blocks PR merges that regress quality

Exam tip: AI-assisted evaluators require an LLM judge — typically GPT-4o — to be configured in the evaluator's model config. This means running evaluations in an environment that has access to an Azure OpenAI GPT-4o deployment. Safety evaluators additionally require an Azure AI project connection for the content safety service.

Further Learning – Microsoft Learn