Microsoft Sentinel Data Lake Architecture

The Evolution of SIEM Storage: From Log Aggregation to Intelligent Data Lake

If you've been using Microsoft Sentinel for more than a year, you've probably hit a familiar problem:

"We need to query older data, but it's prohibitively expensive. Our queries are getting slower. We can't do the behavioral analysis and threat hunting we need because the data isn't accessible."

That's exactly the problem Sentinel Data Lake solves.

What Is Sentinel Data Lake?

Traditional SIEM deployments store data for fast queries (30-90 days) and archive old data at extremely high costs. Sentinel Data Lake fundamentally changes this:

Instead of choosing between:

  • ❌ Hot storage: Fast but expensive (pay for every query)
  • ❌ Archive storage: Cheap but slow (hours to retrieve data)
  • ❌ Deleted data: Gone forever (can't hunt on it)

You get:

  • ✅ Analytics tier: Hot data for ongoing detections and automated responses (30-90 days)
  • ✅ Search tier: Historical data optimized for hunting (searchable archive)
  • ✅ Restore tier: Long-term archive with efficient cold storage

The business impact: You can now perform sophisticated threat hunting across your entire data history, build behavioral baselines over months, and enable AI agents to analyze patterns that would be invisible in just 90 days of data.

Core Benefits of Sentinel Data Lake

1. Graph-Powered Interactive Visualization (Preview)

Investigate incidents and hunt threats using interactive graphs from your actual data:

Real-world scenario:

  • Incident: Suspicious Azure app accessing Exchange Online
  • Traditional approach: Manual KQL queries across multiple tables, manual correlation
  • Data Lake approach: Visual graph showing attack path → Entity connections → Blast radius
  • Result: Minutes instead of hours

2. Query Over Historical Data Without Cost Explosion

Use KQL to hunt on years of data, not just months. Before Data Lake, a single historical query could cost $100+. After Data Lake, the same query is included in your data lake meter—usually 50-70% cheaper for historical queries.

Capabilities:

  • Behavioral baseline analysis (requires 90+ days of data minimum)
  • Predictive modeling (needs trend analysis across seasons)
  • Anomaly detection training (machine learning needs historical patterns)
  • Long-term threat hunting (attackers often hide for months)

3. Built-In, Flexible Data Tiers

Three storage options purpose-built for different use cases:

Tier Use Case Retention Query Speed
Analytics Real-time detections, automations, SOAR 30-90 days Instant
Search Historical threat hunting, behavioral analysis 30 days - 7 years Fast (seconds)
Restore Long-term archive, compliance retention Indefinite Slow (minutes-hours)

4. Data from Everywhere: 350+ Native Connectors

Consolidate data across your entire infrastructure:

What gets pulled in automatically:

  • Microsoft 365 (Exchange, Teams, SharePoint, etc.)
  • Azure (Activity Logs, diagnostic logs, SQL audit logs)
  • Entra ID (Sign-ins, audit logs, risky users)
  • On-premises (Windows Events, Syslog, Linux audit)
  • Third-party clouds (AWS CloudTrail, GCP audit logs)
  • Security tools (Proxy logs, antivirus, EDR, firewalls)

5. Centralized Cost Management

Track, estimate, and optimize your spending with usage dashboards, cost estimators, retention controls, and budget alerts.

Billing: Understanding the Cost Structure

Sentinel Data Lake introduces new billing meters. Costs typically range from $0.50-1.50/GB for ingestion, $0.40-0.60/GB/month for analytics tier, and significantly less for search and restore tiers.

Real-world cost example: A typical organization with 500 GB/day ingestion, 30-day hot storage, 2-year search tier, and 7-year archive runs approximately $48,600/month. Compared to traditional archive retrieval costs ($100-500 per query × 50+ queries/month), Data Lake often saves 20-40%.

Step 1: Prerequisites Before Setup

Required Permissions:

  • ✅ Azure Subscription Owner or Contributor
  • ✅ Microsoft Sentinel Contributor role on the workspace
  • ✅ Storage Account Contributor
  • ✅ Reader on the resource group

Step 2: Enable Sentinel Data Lake

In the Azure Portal:

  1. Navigate to Microsoft Sentinel → Your workspace
  2. Go to Settings (gear icon) → Workspace settings
  3. Look for "Data Lake" section
  4. Click "Enable data lake"

Post-Enablement Timeline:

  • ✅ Immediate (minutes): Data lake enabled, storage created
  • ⏳ Hours: First asset data starts appearing in data lake
  • ⏳ 1-7 days: Asset inventory fully populated
  • ⏳ Ongoing: Historical data migrated to search tier (if configured)

Key Takeaways: Part 1

  • ✅ Sentinel Data Lake fundamentally changes SIEM economics - Historical queries that cost $500 now cost $5
  • ✅ Three-tier architecture optimizes for both speed and cost - Analytics for real-time, Search for hunting, Restore for archives
  • ✅ 350+ native connectors + asset data = unified security data model - Everything your infrastructure touches in one place
  • ✅ Cost management is critical - Budget alerts and usage dashboards prevent bill shock
  • ✅ Asset inventory is automatic - M365, Entra, Azure assets stored in data lake for relationship mapping
  • ✅ Graph capabilities enable visual investigations (preview) - See attack paths instead of parsing query results
  • ✅ Setup takes ~60 minutes - But planning tier strategy takes more thought
  • ✅ This is prerequisite for AI agents - Part 2 shows how to leverage this with Claude API, ChatGPT, and Copilot Studio

Ready to move beyond traditional log retention to intelligent threat hunting? Part 2 shows you exactly how to connect AI agents to analyze your data lake with natural language queries.

You can do this

Archives