AI-300 Part 3: GenAIOps Infrastructure

Part 3 of 5 — 20–25% of the AI-300 exam. This domain covers the infrastructure that underpins generative AI operations in Azure: provisioning Foundry Hubs and Projects with IaC, selecting the right deployment type for LLM endpoints, managing prompts as versioned artifacts, and controlling cost and throughput with APIM and quota management.

Exam Objectives

The official skill area for this domain breaks into three areas:

Design and provision a GenAI infrastructure

Provision Azure AI Foundry Hubs and Projects using Infrastructure as Code
Configure shared resources — storage, Key Vault, compute, connections
Assign appropriate RBAC roles to hubs and projects

Deploy and manage generative AI models

Choose between serverless API and provisioned throughput deployments
Deploy models from the Azure AI model catalog
Manage deployment versions and endpoints

Implement prompt operations

Store and version prompt templates in source control
Use Prompt flow for DAG-based orchestration of prompt steps
Manage rate limits, quotas, and APIM gateway patterns

GenAIOps Architecture Overview

The diagram below shows how Git-managed prompts and Bicep infrastructure combine with Azure AI Foundry deployments, routed through APIM to consuming applications.

Foundry Hub and Project Infrastructure as Code

An Azure AI Foundry Hub is an Azure resource of type Microsoft.MachineLearningServices/workspaces with the kind property set to Hub. Projects are child workspaces under the hub — each project inherits the hub's shared resources (storage account, Key Vault, container registry, and connections) without needing its own copies. This shared-resource model is central to how Foundry reduces management overhead at scale.

When you provision a Foundry Hub with Bicep, you reference an existing Azure OpenAI account as a connection so that all child projects can use the same OpenAI deployments without each project holding individual credentials. The hub stores the connection, and projects simply reference it by name.

// Bicep: Azure AI Foundry Hub with connected Azure OpenAI resource
resource hub 'Microsoft.MachineLearningServices/workspaces@2024-04-01' = {
  name: 'hub-genai-prod'
  location: location
  kind: 'Hub'
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    friendlyName: 'GenAI Production Hub'
    storageAccount: storageAccount.id
    keyVault: keyVault.id
    applicationInsights: appInsights.id
  }
}

// Connection to Azure OpenAI — inherited by all projects under this hub
resource openAiConnection 'Microsoft.MachineLearningServices/workspaces/connections@2024-04-01' = {
  parent: hub
  name: 'conn-azure-openai'
  properties: {
    category: 'AzureOpenAI'
    target: openAiAccount.properties.endpoint
    authType: 'ApiKey'
    credentials: {
      key: openAiAccount.listKeys().key1
    }
  }
}

// Child project — inherits hub storage, Key Vault, and connections
resource project 'Microsoft.MachineLearningServices/workspaces@2024-04-01' = {
  name: 'proj-chatbot-prod'
  location: location
  kind: 'Project'
  properties: {
    hubResourceId: hub.id
    friendlyName: 'Chatbot Production Project'
  }
}

RBAC for Foundry Hub and Projects

Access in Foundry follows a two-tier model. The hub is the management plane — governed by the Azure AI Hub Admin role, which can create projects, manage connections, and configure shared resources. Projects are the working environment — data scientists and ML engineers hold the Azure AI Developer role on the project, giving them permission to run prompt flows, create evaluations, and deploy to project-scoped endpoints.

Role	Assigned at	Key permissions
Azure AI Hub Admin	Hub	Create and delete projects, manage hub connections, configure compute
Azure AI Developer	Project	Run prompt flows, create evaluations, deploy to project endpoints, read connections
Azure AI Inference Deployment Operator	Project	Create and manage model deployments only — no flow authoring
Reader	Hub or Project	View resources and outputs, no write access

Exam tip: Foundry project connections are inherited from the hub — they are not duplicated per project. A developer with the Azure AI Developer role on a project can use the hub's connections without seeing the underlying credentials.

Model Catalog and Deployment Types

The Azure AI Foundry model catalog surfaces models from Microsoft, OpenAI, Meta, Mistral, and others. Each model can be deployed in one of two ways depending on your throughput needs and budget tolerance.

Serverless API Deployments

Serverless API deployments — also called pay-as-you-go deployments — require no GPU provisioning. You create a deployment and immediately receive an endpoint URL. Billing is per token consumed: input tokens and output tokens are metered and billed monthly. There is no upfront commitment, making serverless ideal for development, low-volume production workloads, or scenarios where traffic is unpredictable. The downside is that latency can vary under high load, and you share quota with other customers in the region.

Provisioned Throughput Units (PTU)

PTU deployments reserve dedicated model capacity measured in Provisioned Throughput Units. You purchase a PTU commitment (monthly or yearly), deploy the model to a provisioned endpoint, and that endpoint has guaranteed throughput and consistent latency regardless of overall Azure load. PTU is the correct choice for high-volume, latency-sensitive production workloads — for example, a customer-facing copilot that must respond within two seconds under sustained load.

PTU deployments have a different endpoint URL format than serverless deployments, and they have their own quota bucket separate from token-per-minute (TPM) quota.

Dimension	Serverless API	Provisioned (PTU)
Cost model	Per token consumed	Fixed monthly commitment (PTU hours)
Latency	Variable — depends on shared load	Predictable — dedicated capacity
Provisioning	Instant — no GPU reservation needed	Requires PTU quota approval and commitment
Quota type	Token-per-minute (TPM) quota	PTU quota (separate from TPM)
Best for	Dev/test, variable or low volume	High-volume, SLA-bound production

Models Available in the Catalog

The model catalog includes both Microsoft-hosted OpenAI models (deployed to Azure OpenAI Service) and third-party models deployed as serverless endpoints. Key models to know for the exam:

GPT-4o / GPT-4o-mini — OpenAI multimodal models; GPT-4o-mini supports fine-tuning
o1 / o3-mini — OpenAI reasoning models with extended thinking; higher latency by design
Phi-4 — Microsoft's small language model, optimised for on-device and cost-sensitive scenarios
Llama 3 — Meta's open model, available as serverless endpoint in Foundry
Mistral Large — Mistral AI's flagship model, available via serverless API in Foundry

Exam tip: PTU deployments are suited for predictable, high-volume workloads where latency consistency matters. Serverless deployments are better for variable or low-volume traffic where you want to avoid upfront commitment. Know this distinction — the exam will present cost/latency scenarios and ask which deployment type to recommend.

RAG Architecture in Azure AI Foundry

Retrieval-Augmented Generation (RAG) is the dominant pattern for grounding LLM responses in organisational knowledge. The diagram below shows the full RAG flow: user query → Azure AI Search retrieval → context augmentation → LLM generation → response.

Prompt Versioning and Management

In GenAIOps, prompts are treated as first-class artifacts — versioned, tested, and promoted through environments just like application code. The two primary storage formats for prompts in Foundry are .prompty files (a YAML-based format with metadata and message sections) and plain YAML templates stored in Git.

The .prompty File Format

A .prompty file is a YAML document with a front-matter metadata block and a body containing the prompt template. It captures the model, parameters, and message structure in a single portable file that can be rendered by the Azure AI Foundry SDK or run directly in VS Code with the Prompt flow extension.

---
name: SupportAssistant
description: Customer support prompt for product enquiries
model:
  api: chat
  configuration:
    type: azure_openai
    azure_deployment: gpt-4o
  parameters:
    max_tokens: 800
    temperature: 0.3
inputs:
  customer_query:
    type: string
  product_name:
    type: string
---
system:
You are a helpful support assistant for {{product_name}}.
Answer customer questions accurately and concisely.
If you do not know the answer, say so — do not guess.

user:
{{customer_query}}

Prompt Versioning Workflow

Store prompt files in Git alongside application code. Each prompt change goes through a pull request, automated evaluation (see Part 4), and merge to main before deployment. Tag commits that represent tested, production-ready prompt versions — for example, support-prompt-v2.1. CI/CD pipelines can read the tag and promote the corresponding prompty file to the production Foundry project.

Prompt Flow in Foundry

Prompt flow is the orchestration tool built into Azure AI Foundry. It lets you build DAG-based (directed acyclic graph) pipelines where each node is a step: an LLM call, a Python function, a retrieval query, or a conditional branch. Prompt flow is not a separate service — it is a feature of Foundry projects. You author flows in the Foundry portal or in VS Code, run them interactively, and publish them as deployable endpoints.

The Azure AI Projects SDK (azure.ai.projects) allows you to trigger flow runs, retrieve outputs, and manage prompt assets programmatically from CI/CD pipelines.

from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

client = AIProjectClient(
    subscription_id="<subscription-id>",
    resource_group_name="rg-genai-prod",
    project_name="proj-chatbot-prod",
    credential=DefaultAzureCredential()
)

# Trigger a prompt flow run
run = client.runs.create(
    flow="support-chat-flow",
    data={
        "customer_query": "How do I reset my password?",
        "product_name": "Contoso Portal"
    }
)
print(f"Run status: {run.status}, output: {run.outputs}")

Exam tip: Prompt flow is the orchestration tool within Foundry — it is not a separate Azure service. Flows are authored and deployed within a Foundry project. This is a common distractor on the exam.

Rate Limits and Quota Management

Azure OpenAI enforces quota at the deployment level in two dimensions: tokens per minute (TPM) — the total volume of tokens the deployment can process per minute — and requests per minute (RPM) — the number of API calls allowed per minute. When a deployment is throttled, the API returns HTTP 429 with a Retry-After header.

Managing Quota

Default TPM and RPM quotas are set per region and per model. You can view and adjust deployment quotas in the Azure portal under Azure OpenAI > Deployments or in the Foundry portal. To increase quota beyond the regional default, submit a quota increase request in the portal. Quota is not shared between deployments — each deployment has its own TPM/RPM allocation drawn from the regional pool.

APIM as an AI Gateway

Azure API Management (APIM) is the recommended pattern for exposing Azure OpenAI deployments to applications. APIM sits between your applications and the Azure OpenAI endpoints and provides rate limiting, request logging, load balancing across multiple deployments, and authentication abstraction (so applications use APIM subscriptions keys rather than OpenAI API keys).

Key APIM policies for AI gateway scenarios:

Rate limiting — rate-limit-by-key policy restricts calls per subscription key per time window, preventing any single consumer from exhausting the OpenAI quota
Load balancing across deployments — use the set-backend-service policy with a backend pool of multiple Azure OpenAI endpoints (e.g., deployments in different regions) to distribute traffic
Circuit breaker / retry — on HTTP 429, APIM can automatically route to a backup deployment using the retry policy combined with backend health checks
Token tracking — the azure-openai-token-limit policy counts tokens in the request and response body, allowing you to enforce token budgets per consumer

# APIM inbound policy: rate limit + backend load balance
<inbound>
    <rate-limit-by-key calls="60" renewal-period="60"
        counter-key="@(context.Subscription.Id)" />
    <set-backend-service backend-id="openai-backend-pool" />
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
</inbound>

Circuit Breaker Pattern

When a deployment returns HTTP 429 (throttled), the circuit breaker pattern routes subsequent requests to a backup deployment rather than queuing them or failing. In APIM, this is implemented using a backend pool with health probes — a deployment that returns 429 is temporarily removed from the rotation until it recovers. This is especially important for PTU deployments, which return 429 when the provisioned throughput is exceeded rather than queuing requests.

Exam tip: Serverless deployments use different endpoint URLs than PTU deployments. A serverless GPT-4o endpoint URL contains openai.azure.com with a deployment path, while a PTU endpoint may be in a different region or have a distinct resource name. APIM can abstract both behind a single API URL.

Exam Tips & Key Takeaways

Critical concepts for Domain 3:

Foundry Hub kind — the resource type is Microsoft.MachineLearningServices/workspaces with kind: Hub; projects use kind: Project and reference the hub via hubResourceId
PTU vs serverless — PTU for predictable latency and sustained high volume; serverless for variable or low-volume traffic with no upfront commitment
Inherited connections — project connections are inherited from the hub; you do not create separate connections per project
Prompt flow location — Prompt flow is a feature of Foundry projects, not a standalone Azure service
APIM for quota management — use APIM's rate-limit-by-key and token limit policies to enforce per-consumer quotas and route around throttled deployments
RBAC tiers — Azure AI Hub Admin on the hub; Azure AI Developer on the project

Exam tip: When a scenario asks how to prevent a single application from consuming all Azure OpenAI quota, the answer involves APIM with a rate-limit-by-key policy — not Azure OpenAI quota settings alone, which apply at the deployment level rather than per consuming application.