Most enterprises don’t fail at AI because a model is weak. They fail because every use case becomes its own isolated proof of concept, with its own stack, its own data plumbing and its own way of being trusted (or not). After a year, there are five pilots and no platform.
This post is about the enterprise AI architecture we proposed for a large engineering and construction (EPC) company that wanted to avoid exactly that, during a discovery engagement where I worked on the AI architecture. The brief was to support three very different use cases from day one, and many more later, on one shared platform.
Three use cases, deliberately different
The three use cases were chosen because they stress the platform in different ways:
- Concrete mix design: optimise mix designs for cost, carbon and compliance with building standards. Mostly a traditional ML and optimisation problem, with a small GenAI assistant for standards.
- Smart RFP: read very large tender packages, extract requirements, flag risks and contradictions, and help draft the tender study note. Document-heavy and agentic. I’ve written about the experiments behind it in a separate post.
- Voice AI interviewer: run competency-based assessment interviews by voice, score answers against a rubric and draft the report for human reviewers.
If one architecture can serve an optimiser, a document agent and a voice agent, it can serve most of what comes next.
Principles: maximise return on AI
Before any boxes and arrows, we agreed on a few principles to get the most return for the effort:
- Start where the data already is. Prioritise use cases with good-quality data available today, such as past tenders or existing interview transcripts, instead of waiting for large data engineering programmes.
- Buy or build, per task. Use hosted LLMs where sophisticated reasoning is needed, such as summarisation and comparative analysis. Build custom models where a simple, controllable model does the job better, such as optimisation and regression for concrete mixes.
- Don’t overuse agents. Agents earn their place where a task needs dynamic reasoning and decisions, like orchestrating clause retrieval and analysis in Smart RFP. Deterministic workflows and simple Q&A don’t need them.
- Build reusable capabilities, not point solutions. Every component should be usable by the next use case without rework.
The architecture
The platform has nine building blocks:
- Data sources: enterprise data, document repositories and curated knowledge bases, treated as the single source of truth.
- Document and retrieval intelligence: template-aware parsing, OCR for scans and drawings, fused embeddings and a configurable retrieval layer, so every answer can cite its source.
- Agentic AI platform: a multi-agent system built from capability-specific agents that can be reused across use cases, plus persona tuning, a prompt library, context and memory management, consistency checks and an evaluation framework.
- Responsible AI: guardrails, a scope constrictor that keeps each model inside its intended domain, explainability with citations, and content filtering.
- Hybrid models: a plug-and-play model layer that combines traditional ML (discriminative and optimisation models) with GenAI (LLMs, SLMs, vision and voice), chosen per task by complexity and cost.
- MLOps: a model and LLM registry, CI/CD/CT, experiment tracking and a feedback loop for continuous learning.
- API layer: one way in for applications, users and external systems.
- Identity and access: role-based access to data, agents and models.
- Observability: latency, token usage and cost per agent, so the platform’s economics are visible from day one.
The hybrid AI configurator is the piece that makes this more than a GenAI platform. For each step of each use case, it decides whether a task is best served by an LLM, a small model, or a classic ML model, and wires the components together.
Designing for trust
An enterprise platform lives or dies on whether people trust its outputs. Four design choices did most of the work:
- Scope constriction. Each GenAI component gets a narrow, explicit scope and is held to it. In the voice interviewer, for example, the model can’t drift outside the interview domain.
- Context management. The context each component sees is curated: summarised, windowed and inserted or removed on purpose, not just appended.
- Evaluation and consistency frameworks. Every component is tested against a controlled set of cases covering its full scope, and the state flowing between components is checked end to end. When something goes wrong in a multi-step pipeline, you can see which step failed.
- Auditability. Prompts, conversations and rule overrides are versioned, so every AI decision can be traced back.
The Voice AI interviewer, as an example
The voice use case shows how the building blocks combine in the proposed design:
Speech is transcribed live, and an LLM-based dialogue orchestrator decides the next question, probe or follow-up. A RAG agent grounds questions in the question bank and the competency rubric, an assessment agent scores answers against that rubric, and a report generator drafts the assessment. The full transcript is kept for audit, and human assessors review and approve every report. The AI prepares the work; people make the call.
Nothing here is specific to voice except the speech components. Retrieval, agents, evaluation and guardrails all come from the shared platform.
Technology choices
The stack favoured managed services for speed, while keeping open-source-compatible components to avoid lock-in:
| Layer | Choice |
|---|---|
| LLMs and fine-tuning | Azure OpenAI, Azure AI Foundry |
| Agent orchestration | LangGraph |
| Retrieval | LangChain, Qdrant |
| Custom ML, experiments and registry | Databricks model serving and MLflow |
| Voice | Azure Speech |
| Evaluation | Custom framework with Ragas |
| Prompt logging and observability | LangFuse, Grafana, ELK |
| Delivery | GitHub Actions, Terraform |
A roadmap that proves value first
The adoption path was deliberately incremental:
- Anchor and validate (month 0–1). Pick a small set of use cases with clear business value, KPIs and data that’s already usable. Exclude anything that needs a large data programme first.
- Build with a platform mindset (months 1–2). Implement the first use case on shared components: prompts, retrieval and evaluation, built once for reuse.
- Validate and deploy (months 2–3). Validate outputs with business users, introduce the evaluation framework with grounding and hallucination metrics, and keep deployment simple.
- Monitor, improve and scale (month 4 onwards). Track accuracy, drift, cost per transaction and business KPIs, and add new use cases using the same criteria.
The takeaway
An enterprise AI platform isn’t a GenAI platform with some ML bolted on, or the reverse. It’s a set of reusable capabilities (retrieval, agents, models, guardrails, evaluation and observability) plus a clear rule for when to use which. Use agents where reasoning is needed and simple models where they’re enough, and make every output traceable. Do that, and each new use case starts from a platform, not from scratch.
If you’re designing something similar, I’d love to compare notes. Find me on LinkedIn.