Skip to content
Nagendra Inuguri
Go back

Building an AI foundation platform on Azure: one home for ML and LLM workloads

Most companies don’t start their AI journey with a platform. They start with a use case. One team builds a classification model, another builds a chatbot, a third experiments with document Q&A. Each picks its own tools, its own cloud services and its own way of getting to production.

That works until there are seven of them.

This post is about the AI foundation platform we designed for an enterprise software company that had reached exactly that point. The goal was a central place to build, deploy and run both traditional ML models and LLM-based copilots, on Azure, with one set of standards instead of seven.

The starting point: seven use cases, many stacks

The company had a roadmap of seven AI use cases, all aimed at productivity and customer experience:

  • A bulk classification tool that maps product descriptions to categories
  • A product copilot that answers questions about the company’s products in a chat interface
  • A smart search feature built on top of the classifier
  • Copilots for test generation, API field mapping, and conversations over data and documents
  • A diagnostics assistant that guides customers to health-check insights

Phase one covered two of them: the classification model and the product copilot. Between them they represented both halves of the problem. One was a classic ML pipeline. The other was a RAG application. If the platform could host both well, it could host the rest.

What we found

The product copilot already ran on Azure. Azure Bot captured the user’s question, Azure OpenAI and AKS did the processing, Azure AI Search held the document index, Redis cached responses and Cosmos DB stored each request, response and piece of user feedback. Everything sat inside a VNet, and secrets lived in Key Vault.

It worked, but the edges were rough. Document ingestion ran as a GitHub Action that had to be restarted by hand when it failed. Small changes to a source file triggered a full index rebuild. Prompt changes had no proper path from experimentation to production.

The classification model lived somewhere else entirely. Airflow on Kubernetes orchestrated the pipelines, SageMaker provided compute, MLflow tracked experiments and registered models, and Snowflake on AWS held the data. CI/CD existed for some supporting services, but not for model training or deployment.

So the company had two cloud providers, two experiment-tracking approaches, two deployment stories, and no shared monitoring. That was with only two of the seven use cases live.

The key decision: two lifecycles, one platform

The first design decision shaped everything else. We split the platform into two components:

  • An ML lifecycle component for traditional, predictive models
  • An LLM lifecycle component for copilots and RAG applications

It’s tempting to build one generic “AI pipeline” that does everything. But the tools, workflows and failure modes of the two are different enough that forcing them together makes both worse. An ML pipeline cares about training data, features, experiments and drift. An LLM application cares about prompts, retrieval, grounding and user feedback. Keeping them separate also kept the views simpler for the teams using each one.

What they share is everything underneath: networking, identity, secrets, storage, CI/CD conventions and monitoring.

The AI foundation platform: separate ML and LLM lifecycles on top of a shared foundation for networking, identity, storage, CI/CD and monitoring

The ML lifecycle component

The aim here was to move the classification workload onto Azure without disrupting the team that ran it.

The ML lifecycle: Snowflake and Azure Storage feed Azure ML datasets, managed Airflow runs training on Azure ML compute, and registered models deploy to AKS

A few choices worth calling out:

  • Keep Airflow, for now. The team already knew Airflow, so we moved them to managed Airflow rather than forcing a rewrite. Azure ML pipelines were the longer-term target, once the platform had settled.
  • Everything in and out goes through Azure Storage. Data from Snowflake lands in storage and is read through Azure ML datasets, and results are written back the same way. That gave us lineage and governance almost for free, and compressed formats like Parquet plus archival tiers kept the cost down.
  • Azure ML experiments replace self-hosted MLflow. The experiments service speaks the MLflow API, so existing tracking code kept working, and nobody had to run MLflow servers any more.
  • CI/CD for models, not just services. GitHub Actions triggers training on centrally provisioned compute and deploys registered models to AKS. Versioned endpoints allow several model versions to run side by side for comparison.

The LLM lifecycle component

For the copilot, the biggest gap was the path from a prompt idea to production. We built that path around Azure ML Prompt Flow.

The LLM lifecycle: prompts move from local Prompt Flow through GitHub to production, the copilot uses Azure OpenAI and Azure AI Search, and feedback lands in Cosmos DB

  • Ingestion became a proper pipeline. Document ingestion moved to Azure ML pipelines, with Logic Apps providing retries. A failed run no longer needed someone to notice it and restart it by hand.
  • Prompts are code. Developers experiment with Prompt Flow locally, then merge to a dev branch. GitHub Actions deploys the updated prompts and flow configuration to dev, and the same trigger promotes from main to production.
  • Feedback is captured from day one. Every response and every thumbs up or down goes to Cosmos DB, which gave the team the raw material for evaluation later.

Security as part of the platform, not an add-on

Because the platform would host every future AI use case, security had to be built in rather than added per project:

  • Private by default. All Azure services talk over VNets and private endpoints. The one exception was HTTPS calls to AWS services during the migration period.
  • No secrets in code. Managed identities and Key Vault everywhere, with Entra ID PIM for just-in-time admin access.
  • Data isolation. Training data for each application comes from its own isolated schema, encrypted at rest and in transit.
  • Defence in depth. Network security groups, API Management in front of services, container image scanning, Azure Policy for governance and Sentinel for incident response.

Monitoring: models drift, platforms do too

The monitoring plan covered three layers:

  • Infrastructure: telemetry from VMs, containers and Azure ML into the company’s existing observability tool
  • Models: version tracking, run history, logged metrics and artifacts for every training run
  • Drift: input data drift through statistical tests on feature distributions, and model drift through performance comparisons over time and across model versions

Capturing the real inputs and outputs at each endpoint was what made the drift monitoring possible. Without that data, you can only guess why a model has started to get worse.

What I’d tell anyone building one

  • Start with two very different use cases. One ML, one LLM. If the platform handles both, the design is probably right.
  • Separate the lifecycles, share the foundation. ML and LLM workloads need different tools. They don’t need different networks, identity or CI/CD.
  • Meet teams where they are. Keeping Airflow and MLflow-compatible tracking made adoption far easier than a big-bang rewrite would have.
  • Route all data through governed storage. Lineage and governance are much cheaper to design in than to retrofit.
  • Treat prompts like code. Version them, review them and deploy them through the same pipeline as everything else.
  • Collect feedback before you need it. You can’t build an evaluation set from data you never stored.

At the time, Azure AI Studio (since renamed Azure AI Foundry) was still in preview, so we listed it as a future-state option rather than building on it. Much of what we assembled by hand from Azure ML, Prompt Flow and AI Search is now available in one place. The principles haven’t changed, though: clear lifecycles, a shared secure foundation, and one well-paved path to production.

If you’re designing something similar, I’d love to compare notes. Find me on LinkedIn.



Previous Post
Underwriter AI engine: reading medical records to triage health insurance proposals
Next Post
Re-platforming intelligence: moving ML and LLM workloads from AWS to Azure