microsoft-iq

# Microsoft IQ — Intelligent Knowledge for Agents, Work, and Data Three hands-on demos showcasing how Microsoft's AI platform makes knowledge accessible to agents, workers, and data pipelines — connected through an end-to-end data cycle. ## What You'll Build | Module | What It Demonstrates | Key Services | |--------|---------------------|--------------| | **[Foundry IQ](foundry-iq/)** | Knowledge bases grounded in Kaggle e-commerce data; writes agent insights to Postgres | Azure AI Foundry, Blob Storage, AI Search, PostgreSQL | | **[Work IQ](work-iq/)** | Extract work intelligence with Copilot CLI + MCP servers | Copilot CLI, Dataverse, MCP Protocol | | **[Fabric IQ](fabric-iq/)** | Analyze e-commerce data from Postgres in Lakehouse; write results back | Microsoft Fabric, Lakehouse, PostgreSQL | ## End-to-End Data Cycle Data flows from Kaggle through three stages — **Transactional → Analytical → Transactional** — with PostgreSQL as the bridge: ``` ┌──────────────────────────────────────────────┐ │ END-TO-END DATA CYCLE │ │ │ Kaggle │ TRANSACTIONAL ANALYTICAL │ (Olist) │ │ │ │ ┌──────────┐ ┌──────────┐ │ ├──► Blob ──► │ │ AI Search│ │ Fabric │ │ │ Storage │ │ Index │ │ Lakehouse│ │ │ │ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ │ ┌────▼─────┐ ┌────▼─────┐ │ │ │ │ Foundry │ │ Fabric │ │ │ │ │ Agent │ │ Agent │ │ │ │ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ │ ▼ ▼ │ └──► Postgres ◄┼── insights ──► ◄── results ──► │ (orders, │ (agent writes) (agent writes) │ items, │ │ │ │ payments)│ └───► sync ◄────────┘ │ │ (Postgres → Fabric Lakehouse) │ └──────────────────────────────────────────────┘ ``` ### Data Flow Steps 1. **Kaggle → Blob Storage**: Download Brazilian E-Commerce dataset, upload CSVs to Azure Blob 2. **Blob → AI Search**: Index products and reviews for Foundry agent knowledge base 3. **Kaggle → PostgreSQL**: Load orders, items, and payments into transactional tables 4. **Foundry Agent** answers questions grounded in indexed data; writes each Q&A as an `agent_insight` to Postgres 5. **Postgres → Fabric Lakehouse**: Sync orders + insights to Lakehouse via REST API 6. **Fabric Notebook**: Creates derived tables (sales_summary, delivery_performance, top_products) 7. **Fabric Agent** queries analytical tables; writes `analytical_result` back to Postgres 8. **Foundry Agent** reads unconsumed analytical results from Postgres to enrich future answers ## Prerequisites - **Python 3.11+** with [uv](https://docs.astral.sh/uv/) package manager - **Azure CLI** (`az`) authenticated to your subscription - **Azure subscription** with: - Azure AI Foundry access - Azure Blob Storage - Azure AI Search - **PostgreSQL 15+** (local or Azure Database for PostgreSQL) - **Kaggle account** with API credentials (`~/.kaggle/kaggle.json`) - **Copilot CLI** installed (for Work IQ module) - **Microsoft Fabric** capacity (for Fabric IQ module) ## Quick Start ```bash # Clone the repo git clone https://github.com/Cataldir/microsoft-iq.git cd microsoft-iq # Install dependencies uv sync # Copy and configure environment cp .env.example .env # Edit .env with your Azure + Postgres + Kaggle credentials # ── Step 1: Download Kaggle dataset ── python scripts/download_kaggle.py # ── Step 2: Upload to Blob Storage ── python scripts/upload_to_blob.py # ── Step 3: Index products/reviews into AI Search (Foundry IQ) ── python scripts/index_blob_data.py # ── Step 4: Initialize Postgres and load order data ── python shared/postgres_client.py --action init python shared/postgres_client.py --action load-kaggle --data-dir data/raw # ── Step 5: Query Foundry Agent (writes insights to Postgres) ── cd foundry-iq && python src/api_server.py # or: python src/query_agent.py "top product categories?" # ── Step 6: Sync Postgres → Fabric Lakehouse ── python scripts/sync_to_fabric.py --workspace microsoft-iq-demo --lakehouse iq-lakehouse # ── Step 7: Run Fabric notebook + query Fabric Agent ── cd fabric-iq python src/fabric_agent.py --action query --workspace microsoft-iq-demo --name iq-analyst \ --question "What are the delivery performance trends?" # → writes analytical result back to Postgres (completing the cycle) ``` ## Module Details ### 1. Foundry IQ — Knowledge Bases for Agents Demonstrates creating knowledge bases in Azure AI Foundry grounded in real e-commerce data from Kaggle (Olist Brazilian E-Commerce). Products and reviews are indexed in AI Search; the agent writes every Q&A interaction to PostgreSQL as a transactional insight that feeds Fabric IQ. **Key highlights:** - Portal walkthrough: knowledge base creation, model selection, retrieval configuration - Kaggle data pipeline: download → blob upload → AI Search indexing - Foundry agent grounded in products and customer reviews - Transactional writeback: agent insights → PostgreSQL → Fabric Lakehouse - Local HTML/CSS/JS demo UI ### 2. Work IQ — Work Intelligence with Copilot CLI Demonstrates extracting work signals using the MCP server pattern connected to Dataverse. Shows how to build a minimal MCP server, query CRM opportunities, classify signals, and generate daily work digests — all sanitized with synthetic data for public demonstration. **Key highlights:** - MCP server tool registration and dispatch - Dataverse Web API integration for CRM data - Signal classification (wins, losses, escalations, compete signals) - Copilot CLI prompt templates for daily digests ### 3. Fabric IQ — Data Pipeline Agents Demonstrates orchestrating analytical pipelines with Microsoft Fabric using real e-commerce data synced from PostgreSQL. The Fabric Agent reasons over Lakehouse tables and writes analytical results back to PostgreSQL, completing the Transactional → Analytical → Transactional cycle. **Key highlights:** - Kaggle e-commerce data flows: orders, items, payments, agent insights - PostgreSQL → Fabric Lakehouse sync via REST API - PySpark notebook: delivery performance, payment analysis, revenue rankings - Fabric Agent for natural-language analytics over Lakehouse - Analytical writeback: Fabric results → PostgreSQL → Foundry Agent enrichment ## License MIT ## Author Ricardo Cataldi — Sao Paulo, Brazil