Part 3 of 5 — 20–25% of the AI-300 exam. This domain covers the infrastructure that underpins generative AI operations in Azure: provisioning Foundry Hubs and Projects with IaC, selecting the right deployment type for LLM endpoints, managing prompts as versioned artifacts, and controlling cost and throughput with APIM and quota management.
Exam Objectives
The official skill area for this domain breaks into three areas:
Design and provision a GenAI infrastructure
- Provision Azure AI Foundry Hubs and Projects using Infrastructure as Code
- Configure shared resources — storage, Key Vault, compute, connections
- Assign appropriate RBAC roles to hubs and projects
Deploy and manage generative AI models
- Choose between serverless API and provisioned throughput deployments
- Deploy models from the Azure AI model catalog
- Manage deployment versions and endpoints
Implement prompt operations
- Store and version prompt templates in source control
- Use Prompt flow for DAG-based orchestration of prompt steps
- Manage rate limits, quotas, and APIM gateway patterns
GenAIOps Architecture Overview
The diagram below shows how Git-managed prompts and Bicep infrastructure combine with Azure AI Foundry deployments, routed through APIM to consuming applications.
Foundry Hub and Project Infrastructure as Code
An Azure AI Foundry Hub is an Azure resource of type
Microsoft.MachineLearningServices/workspaces with the kind
property set to Hub. Projects are child workspaces under the hub — each
project inherits the hub's shared resources (storage account, Key Vault, container
registry, and connections) without needing its own copies. This shared-resource model is
central to how Foundry reduces management overhead at scale.
When you provision a Foundry Hub with Bicep, you reference an existing Azure OpenAI account as a connection so that all child projects can use the same OpenAI deployments without each project holding individual credentials. The hub stores the connection, and projects simply reference it by name.
// Bicep: Azure AI Foundry Hub with connected Azure OpenAI resource
resource hub 'Microsoft.MachineLearningServices/workspaces@2024-04-01' = {
name: 'hub-genai-prod'
location: location
kind: 'Hub'
identity: {
type: 'SystemAssigned'
}
properties: {
friendlyName: 'GenAI Production Hub'
storageAccount: storageAccount.id
keyVault: keyVault.id
applicationInsights: appInsights.id
}
}
// Connection to Azure OpenAI — inherited by all projects under this hub
resource openAiConnection 'Microsoft.MachineLearningServices/workspaces/connections@2024-04-01' = {
parent: hub
name: 'conn-azure-openai'
properties: {
category: 'AzureOpenAI'
target: openAiAccount.properties.endpoint
authType: 'ApiKey'
credentials: {
key: openAiAccount.listKeys().key1
}
}
}
// Child project — inherits hub storage, Key Vault, and connections
resource project 'Microsoft.MachineLearningServices/workspaces@2024-04-01' = {
name: 'proj-chatbot-prod'
location: location
kind: 'Project'
properties: {
hubResourceId: hub.id
friendlyName: 'Chatbot Production Project'
}
}RBAC for Foundry Hub and Projects
Access in Foundry follows a two-tier model. The hub is the management plane — governed by the Azure AI Hub Admin role, which can create projects, manage connections, and configure shared resources. Projects are the working environment — data scientists and ML engineers hold the Azure AI Developer role on the project, giving them permission to run prompt flows, create evaluations, and deploy to project-scoped endpoints.
| Role | Assigned at | Key permissions |
|---|---|---|
| Azure AI Hub Admin | Hub | Create and delete projects, manage hub connections, configure compute |
| Azure AI Developer | Project | Run prompt flows, create evaluations, deploy to project endpoints, read connections |
| Azure AI Inference Deployment Operator | Project | Create and manage model deployments only — no flow authoring |
| Reader | Hub or Project | View resources and outputs, no write access |
Exam tip: Foundry project connections are inherited from the hub — they are not duplicated per project. A developer with the Azure AI Developer role on a project can use the hub's connections without seeing the underlying credentials.
Foundation Model Deployment Decisions
Choose the serving model from the operational requirement. A serverless API deployment minimizes infrastructure management and charges by model usage, making it useful for variable demand and rapid model adoption. Managed compute is appropriate when the model needs dedicated Azure Machine Learning compute, custom containers, specialized GPUs, or control over scaling and the serving environment. Azure OpenAI deployment options add capacity choices: consumption deployments suit elastic token demand, while provisioned throughput reserves model capacity for predictable high-volume workloads.
Treat the model name, model version, deployment type, region, quota, content-filter configuration, and fallback behavior as versioned production configuration. Pin versions when repeatability matters, evaluate a newer version against a regression dataset, and promote it gradually. A model alias or deployment name should remain stable for clients while the implementation behind it moves through test, canary, and production stages.
Model Catalog and Deployment Types
The Azure AI Foundry model catalog surfaces models from Microsoft, OpenAI, Meta, Mistral, and others. Each model can be deployed in one of two ways depending on your throughput needs and budget tolerance.
Serverless API Deployments
Serverless API deployments — also called pay-as-you-go deployments — require no GPU provisioning. You create a deployment and immediately receive an endpoint URL. Billing is per token consumed: input tokens and output tokens are metered and billed monthly. There is no upfront commitment, making serverless ideal for development, low-volume production workloads, or scenarios where traffic is unpredictable. The downside is that latency can vary under high load, and you share quota with other customers in the region.
Provisioned Throughput Units (PTU)
PTU deployments reserve dedicated model capacity measured in Provisioned Throughput Units. You purchase a PTU commitment (monthly or yearly), deploy the model to a provisioned endpoint, and that endpoint has guaranteed throughput and consistent latency regardless of overall Azure load. PTU is the correct choice for high-volume, latency-sensitive production workloads — for example, a customer-facing copilot that must respond within two seconds under sustained load.
PTU deployments have a different endpoint URL format than serverless deployments, and they have their own quota bucket separate from token-per-minute (TPM) quota.
| Dimension | Serverless API | Provisioned (PTU) |
|---|---|---|
| Cost model | Per token consumed | Fixed monthly commitment (PTU hours) |
| Latency | Variable — depends on shared load | Predictable — dedicated capacity |
| Provisioning | Instant — no GPU reservation needed | Requires PTU quota approval and commitment |
| Quota type | Token-per-minute (TPM) quota | PTU quota (separate from TPM) |
| Best for | Dev/test, variable or low volume | High-volume, SLA-bound production |
Models Available in the Catalog
The model catalog includes both Microsoft-hosted OpenAI models (deployed to Azure OpenAI Service) and third-party models deployed as serverless endpoints. Key models to know for the exam:
- GPT-4o / GPT-4o-mini — OpenAI multimodal models; GPT-4o-mini supports fine-tuning
- o1 / o3-mini — OpenAI reasoning models with extended thinking; higher latency by design
- Phi-4 — Microsoft's small language model, optimised for on-device and cost-sensitive scenarios
- Llama 3 — Meta's open model, available as serverless endpoint in Foundry
- Mistral Large — Mistral AI's flagship model, available via serverless API in Foundry
Exam tip: PTU deployments are suited for predictable, high-volume workloads where latency consistency matters. Serverless deployments are better for variable or low-volume traffic where you want to avoid upfront commitment. Know this distinction — the exam will present cost/latency scenarios and ask which deployment type to recommend.
RAG Architecture in Azure AI Foundry
Retrieval-Augmented Generation (RAG) is the dominant pattern for grounding LLM responses in organisational knowledge. The diagram below shows the full RAG flow: user query → Azure AI Search retrieval → context augmentation → LLM generation → response.
Prompt Versioning and Management
In GenAIOps, prompts are treated as first-class artifacts — versioned, tested, and promoted through environments just like application code. The two primary storage formats for prompts in Foundry are .prompty files (a YAML-based format with metadata and message sections) and plain YAML templates stored in Git.
The .prompty File Format
A .prompty file is a YAML document with a front-matter metadata block and a
body containing the prompt template. It captures the model, parameters, and message
structure in a single portable file that can be rendered by the Azure AI Foundry SDK or
run directly in VS Code with the Prompt flow extension.
---
name: SupportAssistant
description: Customer support prompt for product enquiries
model:
api: chat
configuration:
type: azure_openai
azure_deployment: gpt-4o
parameters:
max_tokens: 800
temperature: 0.3
inputs:
customer_query:
type: string
product_name:
type: string
---
system:
You are a helpful support assistant for {{product_name}}.
Answer customer questions accurately and concisely.
If you do not know the answer, say so — do not guess.
user:
{{customer_query}}Prompt Variants and Promotion
A prompt version includes more than the instruction text. Track the system message, examples, template variables, model and version, generation parameters, tool schemas, retrieval settings, content-filter assumptions, and evaluation dataset revision. Otherwise, two runs can use the same prompt file and still behave differently.
Create variants to test one deliberate change at a time, such as wording, few-shot examples, context ordering, or temperature. Run the same mapped evaluation rows against every variant, compare quality, safety, latency, and token cost, and promote only when acceptance thresholds pass. Store the winning prompt and its evaluation evidence in Git; do not edit production prompts directly in a portal without bringing the change back to source control.
Prompt Versioning Workflow
Store prompt files in Git alongside application code. Each prompt change goes through a
pull request, automated evaluation (see Part 4), and merge to main before deployment.
Tag commits that represent tested, production-ready prompt versions — for example,
support-prompt-v2.1. CI/CD pipelines can read the tag and promote the
corresponding prompty file to the production Foundry project.
Prompt Flow in Foundry
Prompt flow is the orchestration tool built into Azure AI Foundry. It lets you build DAG-based (directed acyclic graph) pipelines where each node is a step: an LLM call, a Python function, a retrieval query, or a conditional branch. Prompt flow is not a separate service — it is a feature of Foundry projects. You author flows in the Foundry portal or in VS Code, run them interactively, and publish them as deployable endpoints.
The Azure AI Projects SDK (azure.ai.projects) allows you to trigger flow
runs, retrieve outputs, and manage prompt assets programmatically from CI/CD pipelines.
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
client = AIProjectClient(
subscription_id="<subscription-id>",
resource_group_name="rg-genai-prod",
project_name="proj-chatbot-prod",
credential=DefaultAzureCredential()
)
# Trigger a prompt flow run
run = client.runs.create(
flow="support-chat-flow",
data={
"customer_query": "How do I reset my password?",
"product_name": "Contoso Portal"
}
)
print(f"Run status: {run.status}, output: {run.outputs}")Exam tip: Prompt flow is the orchestration tool within Foundry — it is not a separate Azure service. Flows are authored and deployed within a Foundry project. This is a common distractor on the exam.
Rate Limits and Quota Management
Azure OpenAI enforces quota at the deployment level in two dimensions: tokens per
minute (TPM) — the total volume of tokens the deployment can process per
minute — and requests per minute (RPM) — the number of API calls
allowed per minute. When a deployment is throttled, the API returns HTTP 429 with a
Retry-After header.
Managing Quota
Default TPM and RPM quotas are set per region and per model. You can view and adjust deployment quotas in the Azure portal under Azure OpenAI > Deployments or in the Foundry portal. To increase quota beyond the regional default, submit a quota increase request in the portal. Quota is not shared between deployments — each deployment has its own TPM/RPM allocation drawn from the regional pool.
APIM as an AI Gateway
Azure API Management (APIM) is the recommended pattern for exposing Azure OpenAI deployments to applications. APIM sits between your applications and the Azure OpenAI endpoints and provides rate limiting, request logging, load balancing across multiple deployments, and authentication abstraction (so applications use APIM subscriptions keys rather than OpenAI API keys).
Key APIM policies for AI gateway scenarios:
- Rate limiting —
rate-limit-by-keypolicy restricts calls per subscription key per time window, preventing any single consumer from exhausting the OpenAI quota - Load balancing across deployments — use the
set-backend-servicepolicy with a backend pool of multiple Azure OpenAI endpoints (e.g., deployments in different regions) to distribute traffic - Circuit breaker / retry — on HTTP 429, APIM can automatically route
to a backup deployment using the
retrypolicy combined with backend health checks - Token tracking — the
azure-openai-token-limitpolicy counts tokens in the request and response body, allowing you to enforce token budgets per consumer
# APIM inbound policy: rate limit + backend load balance
<inbound>
<rate-limit-by-key calls="60" renewal-period="60"
counter-key="@(context.Subscription.Id)" />
<set-backend-service backend-id="openai-backend-pool" />
<authentication-managed-identity resource="https://cognitiveservices.azure.com" />
</inbound>Circuit Breaker Pattern
When a deployment returns HTTP 429 (throttled), the circuit breaker pattern routes subsequent requests to a backup deployment rather than queuing them or failing. In APIM, this is implemented using a backend pool with health probes — a deployment that returns 429 is temporarily removed from the rotation until it recovers. This is especially important for PTU deployments, which return 429 when the provisioned throughput is exceeded rather than queuing requests.
Exam tip: Serverless deployments use different endpoint URLs than PTU deployments. A serverless GPT-4o endpoint URL contains
openai.azure.comwith a deployment path, while a PTU endpoint may be in a different region or have a distinct resource name. APIM can abstract both behind a single API URL.
Exam Tips & Key Takeaways
Critical concepts for Domain 3:
- Foundry Hub kind — the resource type is
Microsoft.MachineLearningServices/workspaceswithkind: Hub; projects usekind: Projectand reference the hub viahubResourceId - PTU vs serverless — PTU for predictable latency and sustained high volume; serverless for variable or low-volume traffic with no upfront commitment
- Inherited connections — project connections are inherited from the hub; you do not create separate connections per project
- Prompt flow location — Prompt flow is a feature of Foundry projects, not a standalone Azure service
- APIM for quota management — use APIM's rate-limit-by-key and token limit policies to enforce per-consumer quotas and route around throttled deployments
- RBAC tiers — Azure AI Hub Admin on the hub; Azure AI Developer on the project
Exam tip: When a scenario asks how to prevent a single application from consuming all Azure OpenAI quota, the answer involves APIM with a rate-limit-by-key policy — not Azure OpenAI quota settings alone, which apply at the deployment level rather than per consuming application.