AI-300 Part 5: Optimize GenAI Systems

Part 5 of 5 (Final) — 10–15% of the AI-300 exam. This domain covers how to systematically improve the cost, latency, and quality of generative AI systems through RAG pipeline optimisation, fine-tuning, inference cost controls, and latency reduction techniques.

Exam Objectives

The official skill areas for Domain 5:

Optimise retrieval-augmented generation

• Select and configure chunking strategies for document ingestion
• Choose embedding models based on cost and quality requirements
• Implement reranking and hybrid search in Azure AI Search
• Apply query enhancement techniques

Fine-tune foundation models

• Determine when fine-tuning is appropriate vs RAG
• Prepare training data in JSONL format and upload to Azure OpenAI
• Create and monitor fine-tuning jobs in Azure OpenAI
• Use distillation to create efficient student models

Optimise inference cost and latency

• Use the Azure OpenAI Batch API for non-real-time workloads
• Implement semantic caching to avoid redundant LLM calls
• Apply model routing to match request complexity to model tier
• Reduce latency with streaming and parallel tool calls

Optimisation Hierarchy

The diagram below shows the three optimisation layers — RAG pipeline quality, model adaptation through fine-tuning, and inference efficiency — and the levers available at each layer.

RAG Pipeline Optimisation

RAG system quality depends heavily on what text chunks reach the LLM. A well-retrieved, well-sized chunk gives the model exactly the context it needs; a poorly chunked or poorly ranked result leads to hallucination or irrelevant answers. There are four main levers: chunking strategy, embedding model selection, search configuration, and query enhancement.

Chunking Strategies

Chunking determines how source documents are split before being embedded and indexed. The right strategy depends on document structure, typical query length, and the model's context window.

Strategy	How it works	Tradeoff	Best for
Fixed-size	Split every N tokens with optional overlap	Simple; may break sentences mid-thought	Uniform text, quick baseline
Sentence	Split at sentence boundaries, group N sentences per chunk	Preserves meaning; chunk size varies	Q&A over factual prose
Paragraph / heading	Split at Markdown headings or paragraph breaks	Preserves document structure; uneven sizes	Structured documents (policies, manuals)
Semantic	Group sentences with similar embeddings together	Highest semantic coherence; computationally expensive	Long documents with topic shifts

Chunk size is a key parameter: small chunks (128–256 tokens) return precise, targeted snippets but may lose surrounding context. Large chunks (512–1024 tokens) provide more context but consume more of the LLM's context window per retrieved result. A common approach is to retrieve small chunks for precision, then expand each retrieved chunk to include surrounding paragraphs before passing to the model.

Embedding Model Selection

Azure OpenAI offers two embedding model generations. text-embedding-3-small is lower cost and suitable for most scenarios; text-embedding-3-large produces higher-dimensional embeddings with better semantic discrimination at higher cost. Both support dimension reduction — you can request fewer dimensions to reduce index storage and query latency at a modest quality cost.

Hybrid Search and Reranking

Azure AI Search supports hybrid search that combines BM25 keyword scoring with vector similarity scoring. Hybrid search outperforms pure vector search for queries that include specific terms (product codes, names, acronyms) that may not be well-captured by embedding similarity alone.

The semantic reranker in Azure AI Search applies a cross-encoder model as a second pass over the top-N hybrid search results, reordering them by semantic relevance to the query. Enabling the semantic ranker is one of the most impactful single changes you can make to RAG retrieval quality. It is enabled at the index level and adds a small cost per query.

Query Enhancement

Raw user queries are often short and ambiguous. Two techniques improve retrieval quality before the search call:

• Query rewriting — use a small LLM call to expand or clarify the user's query into a richer form that better matches indexed content. For example, "error 404" becomes "HTTP 404 Not Found error handling and troubleshooting steps"
• HyDE (Hypothetical Document Embeddings) — generate a hypothetical answer to the query using the LLM, embed that answer, and use it as the retrieval vector. HyDE often retrieves more relevant passages than embedding the query directly, because the hypothetical answer matches the style and content of indexed documents more closely than a short question does

Exam tip: RAG is always the preferred approach for grounding a model in knowledge that changes over time or is organisation-specific. Fine-tuning cannot inject new knowledge — a fine-tuned model still requires RAG to answer factual questions about content it has never seen.

Fine-tuning Foundation Models

Fine-tuning adapts a pre-trained model's behaviour through supervised training on a curated dataset. It is the right tool when you need to change the model's style, format, or tone consistently — for example, always responding in a specific JSON schema, adopting a brand voice, or following a structured reasoning pattern. It is not the right tool for adding new factual knowledge — the model will not reliably recall facts from training data it has only seen a few hundred times.

Fine-tuning vs RAG Decision

• Use RAG when the knowledge changes frequently, is large, or needs to be auditable (you can inspect the retrieved chunks)
• Use fine-tuning when the issue is output format, tone, style, or following a complex instruction pattern that is difficult to specify in a prompt
• Combine both for best results — fine-tune for format/behaviour, RAG for knowledge grounding

Azure OpenAI Fine-tuning Steps

Azure OpenAI supports fine-tuning of gpt-4o-mini and gpt-4o. The process follows five steps:

from openai import AzureOpenAI
import json

client = AzureOpenAI(
    azure_endpoint="https://my-openai.openai.azure.com/",
    api_version="2024-08-01-preview"
)

# Step 1: Upload JSONL training file
# Each line: {"messages": [{"role":"system",...},{"role":"user",...},{"role":"assistant",...}]}
with open("training_data.jsonl", "rb") as f:
    upload = client.files.create(file=f, purpose="fine-tune")

print(f"File ID: {upload.id}")

# Step 2: Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=upload.id,
    model="gpt-4o-mini",
    hyperparameters={"n_epochs": 3}
)

print(f"Job ID: {job.id}, Status: {job.status}")

# Step 3: Monitor job (poll or use events endpoint)
job_status = client.fine_tuning.jobs.retrieve(job.id)
print(f"Status: {job_status.status}, Model: {job_status.fine_tuned_model}")

# Step 4: Deploy the fine-tuned model via Azure portal or REST API
# Step 5: Evaluate — run evaluation dataset against fine-tuned deployment

Model Distillation

Distillation is a technique where a large, capable teacher model (for example, GPT-4o) generates synthetic training examples — completions, reasoning chains, structured outputs — and a smaller student model (for example, Phi-4) is fine-tuned on those examples. The result is a smaller model that approximates the teacher's behaviour at a fraction of the inference cost. Azure OpenAI supports distillation workflows through the fine-tuning API: generate completions with GPT-4o with store=True, then use those stored completions as the training dataset for a Phi-4 or gpt-4o-mini fine-tuning job.

Exam tip: Fine-tuning cannot add new knowledge to a model — the fine-tuned model still needs RAG to answer questions about facts it was not trained on. Fine-tuning improves format, style, and instruction following; RAG provides knowledge grounding.

Inference Cost Optimisation

LLM inference cost is driven primarily by token volume — input tokens plus output tokens multiplied by per-token price. Several techniques reduce token consumption or shift workloads to cheaper pricing tiers.

Azure OpenAI Batch API

The Batch API accepts a file of requests and processes them asynchronously within a 24-hour window, at a 50% discount compared to real-time pricing. It is designed for workloads where immediate response is not required: document processing, overnight analysis, bulk evaluation runs. Jobs that do not complete within 24 hours are cancelled and return partial results.

import json
from openai import AzureOpenAI
from pathlib import Path

client = AzureOpenAI(
    azure_endpoint="https://my-openai.openai.azure.com/",
    api_version="2024-08-01-preview"
)

# Prepare batch request file (JSONL, one request per line)
requests = [
    {
        "custom_id": f"req-{i}",
        "method": "POST",
        "url": "/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "user", "content": f"Summarise document {i}: {doc_text}"}
            ],
            "max_tokens": 200
        }
    }
    for i, doc_text in enumerate(documents)
]

Path("batch_requests.jsonl").write_text(
    "\n".join(json.dumps(r) for r in requests)
)

# Upload and submit
with open("batch_requests.jsonl", "rb") as f:
    batch_file = client.files.create(file=f, purpose="batch")

batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/chat/completions",
    completion_window="24h"
)

print(f"Batch ID: {batch.id}, Status: {batch.status}")

Token Optimisation

Beyond Batch API, several prompt-level techniques reduce token consumption in real-time calls:

• Concise system prompts — avoid verbose instructions; every token in the system prompt is charged on every call
• Structured output (JSON mode) — request JSON responses to avoid the model writing explanatory prose around the data; parse and discard wrapper text programmatically
• max_tokens limits — set an explicit output token limit appropriate to the task to prevent unexpectedly long completions

Semantic Caching

Semantic caching stores the embedding of a user query alongside the LLM's response. On subsequent requests, the incoming query is embedded and compared (by cosine similarity) against cached embeddings. If a sufficiently similar query has been seen before, the cached response is returned without making an LLM call. This is distinct from context caching (which caches the KV state of a long prefix within a single session) — semantic caching operates across sessions and across users.

Semantic caching is most effective for FAQ-style applications where many users ask semantically equivalent questions. It is typically implemented using a Redis cache with vector search capability (Azure Cache for Redis with vector search preview) or Azure AI Search as the cache store.

Model Routing

Not every request requires the most capable (and expensive) model. Model routing classifies the complexity of an incoming request and directs it to the appropriate model tier. Simple requests — short questions, classification tasks, template filling — go to a smaller, cheaper model like GPT-4o-mini or Phi-4. Complex requests requiring multi-step reasoning, long-form generation, or tool use escalate to GPT-4o.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://my-openai.openai.azure.com/",
    api_version="2024-08-01-preview"
)

def classify_complexity(query: str) -> str:
    """Returns 'simple' or 'complex' based on query characteristics."""
    if len(query.split()) < 15 and "?" in query:
        return "simple"
    return "complex"

def route_request(query: str, messages: list) -> str:
    complexity = classify_complexity(query)
    model = "gpt-4o-mini" if complexity == "simple" else "gpt-4o"

    response = client.chat.completions.create(
        model=model,
        messages=messages
    )
    return response.choices[0].message.content

Latency Optimisation

Perceived latency in AI applications has two components: time-to-first-token (TTFT) — how long before the user sees any output — and total generation time. Streaming and PTU deployments address different parts of this.

Streaming Responses

With stream=True, the Azure OpenAI API returns tokens as they are generated rather than waiting for the full completion. This dramatically improves perceived responsiveness because the user sees output appear progressively. TTFT drops from several seconds (full completion wait) to under a second (first token).

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": user_query}],
    stream=True
)

for chunk in response:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)

Parallel Tool Calls

GPT-4o supports parallel function calling — when a user request requires multiple tool calls (for example, fetching weather and calendar data simultaneously), the model can emit multiple tool call requests in a single response rather than sequentially. This reduces the number of round trips in agent loops and cuts total latency proportionally to the number of parallelisable calls.

Regional Deployment and PTU Benefits

Deploy Azure OpenAI resources in the same Azure region as your application to minimise network round-trip latency. For latency-critical workloads, use PTU deployments: because capacity is reserved, PTU endpoints have no cold-start delay and maintain consistent latency under load. With serverless deployments, latency can spike during regional demand peaks as requests queue behind other tenants' workloads.

Optimisation Technique Summary

Technique	Improves Cost	Improves Latency	Improves Quality
Semantic chunking	—	—	Yes
Hybrid search + semantic reranker	—	—	Yes
Fine-tuning	Possible (smaller model)	Possible (smaller model)	Yes (format/style)
Azure OpenAI Batch API	Yes — 50% discount	No (24hr SLA)	—
Semantic caching	Yes — eliminates redundant calls	Yes — cache hits are fast	—
Model routing	Yes — cheaper model for simple tasks	Yes — smaller models are faster	—
Streaming (stream=True)	—	Yes — lower perceived TTFT	—
PTU deployment	Potentially (high volume only)	Yes — predictable, no cold start	—

Exam Tips & Key Takeaways

Critical concepts for Domain 5:

• RAG before fine-tuning — always prefer RAG for knowledge that changes over time; fine-tuning is for format and style, not factual grounding
• Fine-tuning cannot add new knowledge — a fine-tuned model will not reliably recall facts from its fine-tuning dataset that were not in its original pre-training
• Batch API = 50% cost, 24hr SLA — remember both the discount and the constraint; batch jobs that exceed 24 hours are cancelled
• Semantic caching vs context caching — semantic caching stores complete responses keyed by query similarity, across sessions; context caching (provider-level) reuses the KV cache for a repeated prompt prefix within a session
• Semantic reranker — enabled at the Azure AI Search index level; applies as a second-pass reranking over hybrid BM25 + vector results
• Distillation — teacher (GPT-4o) generates training data; student (Phi-4 or gpt-4o-mini) is fine-tuned on it; smaller model approximates teacher's behaviour at lower cost

Exam tip: When a scenario presents a workload that runs nightly, processes thousands of documents, and does not need real-time responses, the correct answer is the Azure OpenAI Batch API — not PTU and not a streaming serverless deployment.

Further Learning – Microsoft Learn

• How to fine-tune Azure OpenAI models
• Semantic search in Azure AI Search
• Azure OpenAI Batch API
• Model distillation in Azure OpenAI