Skip to content
Nagendra Inuguri
Go back

One platform, many agents: designing a hybrid enterprise AI architecture

Most enterprises don’t fail at AI because a model is weak. They fail because every use case becomes its own isolated proof of concept, with its own stack, its own data plumbing and its own way of being trusted (or not). After a year, there are five pilots and no platform.

This post is about the enterprise AI architecture we proposed for a large engineering and construction (EPC) company that wanted to avoid exactly that, during a discovery engagement where I worked on the AI architecture. The brief was to support three very different use cases from day one, and many more later, on one shared platform.

Three use cases, deliberately different

The three use cases were chosen because they stress the platform in different ways:

  • Concrete mix design: optimise mix designs for cost, carbon and compliance with building standards. Mostly a traditional ML and optimisation problem, with a small GenAI assistant for standards.
  • Smart RFP: read very large tender packages, extract requirements, flag risks and contradictions, and help draft the tender study note. Document-heavy and agentic. I’ve written about the experiments behind it in a separate post.
  • Voice AI interviewer: run competency-based assessment interviews by voice, score answers against a rubric and draft the report for human reviewers.

If one architecture can serve an optimiser, a document agent and a voice agent, it can serve most of what comes next.

Three use cases on one shared enterprise AI platform: concrete mix design, Smart RFP and a Voice AI interviewer

Principles: maximise return on AI

Before any boxes and arrows, we agreed on a few principles to get the most return for the effort:

  • Start where the data already is. Prioritise use cases with good-quality data available today, such as past tenders or existing interview transcripts, instead of waiting for large data engineering programmes.
  • Buy or build, per task. Use hosted LLMs where sophisticated reasoning is needed, such as summarisation and comparative analysis. Build custom models where a simple, controllable model does the job better, such as optimisation and regression for concrete mixes.
  • Don’t overuse agents. Agents earn their place where a task needs dynamic reasoning and decisions, like orchestrating clause retrieval and analysis in Smart RFP. Deterministic workflows and simple Q&A don’t need them.
  • Build reusable capabilities, not point solutions. Every component should be usable by the next use case without rework.

The architecture

Enterprise AI architecture: data sources, document and retrieval intelligence, an agentic AI platform, responsible AI, hybrid models and MLOps, under shared API, identity and observability layers

The platform has nine building blocks:

  1. Data sources: enterprise data, document repositories and curated knowledge bases, treated as the single source of truth.
  2. Document and retrieval intelligence: template-aware parsing, OCR for scans and drawings, fused embeddings and a configurable retrieval layer, so every answer can cite its source.
  3. Agentic AI platform: a multi-agent system built from capability-specific agents that can be reused across use cases, plus persona tuning, a prompt library, context and memory management, consistency checks and an evaluation framework.
  4. Responsible AI: guardrails, a scope constrictor that keeps each model inside its intended domain, explainability with citations, and content filtering.
  5. Hybrid models: a plug-and-play model layer that combines traditional ML (discriminative and optimisation models) with GenAI (LLMs, SLMs, vision and voice), chosen per task by complexity and cost.
  6. MLOps: a model and LLM registry, CI/CD/CT, experiment tracking and a feedback loop for continuous learning.
  7. API layer: one way in for applications, users and external systems.
  8. Identity and access: role-based access to data, agents and models.
  9. Observability: latency, token usage and cost per agent, so the platform’s economics are visible from day one.

The hybrid AI configurator is the piece that makes this more than a GenAI platform. For each step of each use case, it decides whether a task is best served by an LLM, a small model, or a classic ML model, and wires the components together.

Designing for trust

An enterprise platform lives or dies on whether people trust its outputs. Four design choices did most of the work:

  • Scope constriction. Each GenAI component gets a narrow, explicit scope and is held to it. In the voice interviewer, for example, the model can’t drift outside the interview domain.
  • Context management. The context each component sees is curated: summarised, windowed and inserted or removed on purpose, not just appended.
  • Evaluation and consistency frameworks. Every component is tested against a controlled set of cases covering its full scope, and the state flowing between components is checked end to end. When something goes wrong in a multi-step pipeline, you can see which step failed.
  • Auditability. Prompts, conversations and rule overrides are versioned, so every AI decision can be traced back.

The Voice AI interviewer, as an example

The voice use case shows how the building blocks combine in the proposed design:

Voice AI interviewer: speech to text and text to speech around an LLM dialogue orchestrator, with RAG, assessment and report agents, producing a transcript and a draft report for human review

Speech is transcribed live, and an LLM-based dialogue orchestrator decides the next question, probe or follow-up. A RAG agent grounds questions in the question bank and the competency rubric, an assessment agent scores answers against that rubric, and a report generator drafts the assessment. The full transcript is kept for audit, and human assessors review and approve every report. The AI prepares the work; people make the call.

Nothing here is specific to voice except the speech components. Retrieval, agents, evaluation and guardrails all come from the shared platform.

Technology choices

The stack favoured managed services for speed, while keeping open-source-compatible components to avoid lock-in:

LayerChoice
LLMs and fine-tuningAzure OpenAI, Azure AI Foundry
Agent orchestrationLangGraph
RetrievalLangChain, Qdrant
Custom ML, experiments and registryDatabricks model serving and MLflow
VoiceAzure Speech
EvaluationCustom framework with Ragas
Prompt logging and observabilityLangFuse, Grafana, ELK
DeliveryGitHub Actions, Terraform

A roadmap that proves value first

The adoption path was deliberately incremental:

  1. Anchor and validate (month 0–1). Pick a small set of use cases with clear business value, KPIs and data that’s already usable. Exclude anything that needs a large data programme first.
  2. Build with a platform mindset (months 1–2). Implement the first use case on shared components: prompts, retrieval and evaluation, built once for reuse.
  3. Validate and deploy (months 2–3). Validate outputs with business users, introduce the evaluation framework with grounding and hallucination metrics, and keep deployment simple.
  4. Monitor, improve and scale (month 4 onwards). Track accuracy, drift, cost per transaction and business KPIs, and add new use cases using the same criteria.

The takeaway

An enterprise AI platform isn’t a GenAI platform with some ML bolted on, or the reverse. It’s a set of reusable capabilities (retrieval, agents, models, guardrails, evaluation and observability) plus a clear rule for when to use which. Use agents where reasoning is needed and simple models where they’re enough, and make every output traceable. Do that, and each new use case starts from a platform, not from scratch.

If you’re designing something similar, I’d love to compare notes. Find me on LinkedIn.



Previous Post
Beyond text RAG: document intelligence for tenders with hundreds of pages, tables and CAD drawings
Next Post
From a handful of markets to global coverage: lessons from scaling demand forecasting