AI on your terms: meet the enterprise-ready AI operating model

Many firms have conducted successful Artificial Intelligence (AI) pilot projects, but scaling them across departments and workflows remains a challenge. Inference costs, data silos, talent gaps and poor alignment with business strategy are just some of the issues that leave organisations trapped in pilot purgatory. This inability to scale successful experiments means AI’s potential for improving enterprise efficiency, decision-making and innovation isn’t fully realised. So what’s the solution?

Although it’s not a magic bullet, an AI operating model is really the foundation for scaling pilot projects up to enterprise-wide deployments. Essentially it’s a structured framework that defines how the organisation develops, deploys and governs AI. By bringing together infrastructure, data, people, and governance in a flexible and secure way, it ensures that AI delivers value at scale while remaining ethical and compliant.

“A successful AI proof-of-concept is like building a single race car that can go fast,” says Professor Yu Xiong, chair of business analytics at the UK-based Surrey Business School. “An efficient AI technology operations model, however, is the entire system – the processes, tools, and team structures – for continuously manufacturing, maintaining, and safely operating an entire fleet of cars.”

But while the importance of this framework is clear, how should enterprises establish and embed it?

“It begins with a clear strategy that defines objectives, desired outcomes, and measurable success criteria, such as model performance, bias detection, and regulatory compliance metrics,” says Professor Azadeh Haratiannezhadi, co-founder of generative AI company Taktify and professor of generative AI in cybersecurity at the Open Institute of Technology.

Platforms, tools and MLOps pipelines that enable models to be deployed, monitored and scaled in a safe and efficient way are also essential in practical terms.

“Tools and infrastructure must also be selected with transparency, cost, and governance in mind,” says Efrain Ruh, continental chief technology officer for Europe at Digitate. “Crucially, organisations need to continuously monitor the evolving AI landscape and adapt their models to new capabilities and market offerings.”

An open approach

The most effective AI operating models are also founded on openness, interoperability and modularity. Open source platforms and tools provide greater control over data, deployment environments and costs, for example. These characteristics can help enterprises to avoid vendor lock-in, successfully align AI to business culture and values, and embed it safely into cross-department workflows.

“Modularity and platformisation…avoids building isolated ‘silos’ for each project,” explains professor Xiong. “Instead, it provides a shared, reusable ‘AI platform’ that integrates toolchains for data preparation, model training, deployment, monitoring, and retraining. This drastically improves efficiency and reduces the cost of redundant work.”

A strong data strategy is equally vital for ensuring high-quality performance and reducing bias. Ideally, the AI operating model should be cloud and LLM agnostic too.

“This allows organisations to coordinate and orchestrate AI agents from various sources, whether that’s internal or 3rd party,” says Babak Hodjat, global chief technology officer of AI at Cognizant. “The interoperability also means businesses can adopt an agile iterative process for AI projects that is guided by measuring efficiency, productivity, and quality gains, while guaranteeing trust and safety are built into all elements of design and implementation.”

A robust AI operating model should feature clear objectives for compliance, security and data privacy, as well as accountability structures. Richard Corbridge, chief information officer of Segro, advises organisations to: “Start small with well-scoped pilots that solve real pain points, then bake in repeatable patterns, data contracts, test harnesses, explainability checks and rollback plans, so learning can be scaled without multiplying risk. If you don’t codify how models are approved, deployed, monitored and retired, you won’t get past pilot purgatory.”

Of course, technology alone can’t drive successful AI adoption at scale: the right skills and culture are also essential for embedding AI across the enterprise.

“Multidisciplinary teams that combine technical expertise in AI, security, and governance with deep business knowledge create a foundation for sustainable adoption,” says Professor Haratiannezhadi. “Ongoing training ensures staff acquire advanced AI skills while understanding associated risks and responsibilities.”

Ultimately, an AI operating model is the playbook that enables an enterprise to use AI responsibly and effectively at scale. By drawing together governance, technological infrastructure, cultural change and open collaboration, it supports the shift from isolated experiments to the kind of sustainable AI capability that can drive competitive advantage.

In other words, it’s the foundation for turning ambition into reality, and finally escaping pilot purgatory for good.

Duncan Jefferies

In association with

Artificial intelligence has arguably disrupted the business landscape faster than any previous technology. Within a remarkably short timeframe, it has reshaped everything from strategic planning and customer engagement to service innovation and product development. But there’s a significant gap between AI leaders that have successfully embedded the technology across departments and workflows, and enterprises that have yet to achieve organisation-wide deployments.

The latter group are often held back by several key issues. First, the substantial expense of inference operations (applying trained AI models to fresh data for predictions), particularly when relying on commercial large language models (LLMs) in live environments.

Second, the intricate process of choosing appropriate models and synchronising them with company data, whether through retrieval augmented generation (RAG), which enhances accuracy by connecting models with external information sources, or through costly fine-tuning procedures. And finally, the operational demands of implementing and maintaining AI infrastructure across diverse hybrid environments.

“It can be difficult to deploy AI where you need to – but at the same time businesses need to embrace that kind of flexibility,” says Martin Isaksson, go-to-market lead in the AI business unit at Red Hat.

By taking a collaborative approach to deployment, enterprises can achieve this flexibility and meet the complex demands of AI at scale. The specialised capabilities of an extensive partner network – encompassing hardware manufacturers, software developers, cloud infrastructure providers and systems integrators – can help them to overcome bottlenecks and unlock the full power of the technology. But all of this requires an open platform for co-innovation.

“That sort of foundation is incredibly important, because no single vendor can provide every tool that’s needed,” says Isaksson. “A key benefit of working with open source is that you’re always operating at the same pace as innovation, right where it happens.”

This approach underpins Red Hat’s AI portfolio. It includes Red Hat Enterprise Linux AI (RHEL AI), a foundation model platform for developing, testing and deploying LLMs, Red Hat AI Inference Server, and Red Hat OpenShift AI, a platform for managing the entire lifecycle of AI and machine learning models, from development and training to deployment and monitoring, across hybrid cloud environments.

The company’s partner ecosystem also includes global system integrators, hardware providers like AMD and Nvidia as well as cloud specialists like IBM and Google. By coordinating and enhancing these partnerships, Red Hat ultimately enables organisations to advance their AI programmes while maintaining vendor independence.

“It’s impossible to predict every use case for AI right now – there’s so many out there in the world,” Isaksson explains. “So you can really benefit from working with a modular platform and an ecosystem of partners.”

Any model, any accelerator, any cloud

With so many LLMs, inference server settings and accelerator options available today, businesses need an easy way to navigate them and ensure that tradeoffs between performance, accuracy and cost meet their needs.

“The ability to easily switch between different models is of great importance,” says Isaksson. “We have a pre-selected model catalogue with optimised models, so they’re faster and cheaper to run.”

These curated and validated models are available on Hugging Face, an open-source community for co-innovating models, datasets and applications. Deploying them helps businesses to reduce their dependence on proprietary LLM providers, whose solutions often include ‘black box’ algorithms and training data. “We believe you should be in control of the infrastructure, the data, the model, how it works – and in the end the whole application,” says Isaksson.

Small language models (SLMs) – a category of LLMs that allow rapid, straightforward customisation using organisational data for targeted applications – are another key weapon in the open source armoury. Together with IBM, Red Hat has co-created InstructLab, an open source project designed to lower the barriers to customisation by allowing domain experts with minimal data science expertise to fine-tune SLMs. In fact, when paired with RAG, the right SLM could even outperform a proprietary model.

Cost-effective inference is another crucial component of high-performing, enterprise-scale AI deployments. This is supported by Red Hat AI Inference Server, which optimises model inference across the hybrid cloud to drive down costs. Developed from the open source vLLM project, it provides compatibility with any generative AI model, across any AI accelerator hardware, within any cloud infrastructure.

Scale and trust

OpenShift AI has already enabled data scientists at DenizBank, a private bank based in Turkey, to save hours when selecting and validating data for model training, fine-tuning and validation. Integration with hardware accelerator dashboards has helped to optimise GPU usage, with the platform automatically scaling up the slices of GPU a model has access to, as needed. This allows for more workloads to run simultaneously without the need for additional GPU hardware, cutting costs and improving efficiency.

Such AI solutions can help businesses to optimise the performance of AI deployments across a range of hardware configurations. “If your accelerator infrastructure is fragmented, with resources scattered in different places – some in the cloud, some on-prem – with one platform you can virtually pool all these resources and optimise their usage,” Isaksson explains.

Open source collaboration can also help enterprises to maintain compliance with rapidly changing regulatory and security standards. Proprietary LLMs often generate concerns regarding security, privacy and safety protocols, with ambiguity about training data sources and output reliability potentially increasing an organisation’s legal exposure. Open source projects, on the other hand, provide the transparency needed to identify bias and privacy issues before they become problematic.

“That’s really important,” says Isaksson. “This is why we’re supporting TrustyAI, which is an open-source responsible-AI toolkit that aims to solve AI’s well-documented problems with bias.”

The TrustyAI community maintains a number of responsible AI projects, revolving around model explainability, model monitoring, and responsible model serving. Red Hat engineers involved in the community recently developed a guard railing solution that ensures LLMs behave ethically, safely and within organisational or regulatory boundaries, making them more viable for high-stakes deployments.

The company also plays an active role in Llama Stack, Meta’s open source framework for building generative applications, and supports Anthropic’s Model Context Protocol (MCP), which standardises agent-to-application interactions.

These projects again demonstrate a simple truth: that no individual technology provider is really equipped to address all the complexities, needs and demands of enterprise AI across today’s hybrid environments.

What’s needed is an open source foundational platform that provides access to a diverse partner ecosystem, allowing enterprises to transform pilot projects into production-ready deployments while maintaining control over their information assets and budgets. One that ensures AI works on enterprise terms, not those of a proprietary vendor.

In association with

For more information please visit redhat.com

Concerns about transparency, costs, compliance and vendor lock-in can put the breaks on AI deployments. But sitting on the sidelines while rivals push ahead with transformative projects is hardly a winning strategy.

Modular, open-source platforms and tools offer a solution to the problem, enabling organisations to deploy AI at a pace and scale that aligns with their governance needs. The transparency of open source solutions avoids the data security pitfalls associated with proprietary models, for example, while providing a strong foundation for hybrid deployments.

“The strategic use of open source tools, platforms and components gives organisations maximum flexibility when choosing where to deploy their applications and how their data is managed,” says Dr Rosemary Francis, chief technology officer of CommonAI Compute. “By building intelligently on an open stack they can opt for an on-prem, hybrid, cloud or even multi-cloud approach and switch fluidly as their governance needs evolve.”

Enterprises can also customise their AI applications as much as is needed. “This includes meeting any data protection or cybersecurity requirements they may need to comply with as part of their overall governance strategy,” says Dr Francis.

In fact, with an open source approach “organisations can inspect and modify every aspect of their AI pipeline while scaling at speeds that align with their specific compliance and operational requirements, rather than being constrained by vendor roadmaps or proprietary limitations,” says Simon James, managing director of data science and AI at Publicis Sapient.

Granted, this flexibility may require some additional effort on the part of enterprises. It is arguably more time-intensive – and may require some specialist skills – to deploy and maintain AI applications built with open source components. Some firms may therefore feel it’s quicker and easier to use proprietary LLMs and other services provided by hyperscale AI players.

“However, this strategy becomes much less attractive at medium to long time horizons, since it is likely to stifle innovation, increase institutional capture and ultimately lead to higher costs further down the road,” Francis explains.

Indeed, as James points out: “The most significant benefit [of open source] lies in creating genuine intellectual property – by forking open source models and saving weights, organisations build competitive assets rather than renting technology from vendors.”

Cutting costs and creating independence

Open source solutions also address the fact that inference costs ramp up dramatically at scale. “Self-hosting eliminates per-query charges and creates predictable cost structures, while providing the deployment flexibility to run the same models across on-premises infrastructure, private clouds, and public cloud services without vendor lock-in concerns,” says James.

However, enterprises still need access to proven technologies with active community support. Platforms like Red Hat OpenShift AI, for example, can help firms to deploy AI in a transformative and trusted manner in hybrid cloud environments. In addition, the Red Hat Enterprise Linux AI (RHEL AI) foundation model platform allows them to develop, test, and run IBM’s Granite models for enterprise applications.

A significant advantage of SLMs like the Granite family is that they require less computational power, energy and financial investment than proprietary LLMs. What’s more, their customisability and openness addresses the ‘black box’ issue associated with larger models.

“The choice of SLMs can reduce vendor lock-in, give you full control over data residency and inference costs, and make auditability and explainability easier because you control the stack,” says Richard Corbridge, chief information officer of Segro.

Indeed, the risk mitigation benefits of open-source SLMs could be crucial for many firms as they look to embed AI at scale. “When you can inspect model architecture, training data, and decision-making processes, you create the auditability that’s essential for regulated industries and enterprise governance frameworks,” James continues. “This transparency enables organisations to implement human-in-the-loop designs where they will add the most value, typically in areas requiring judgment or compliance oversight.”

The fact that sensitive data isn’t being sent to a third-party LLM vendor can further reduce security and governance concerns. “Open-source SLMs enable ‘data localisation’, where all processing and fine-tuning occur within the company’s firewall or a trusted cloud environment, significantly reducing data leakage and compliance risks,” says Professor Yu Xiong, chair of business analytics at Surrey Business School in the UK.

Often there’s zero trade-off in terms of performance. “A well-architected system using multiple SLMs for specific tasks often outperforms a single large model attempting to handle everything, while providing better cost control and governance visibility,” says James. “The key is selecting SLMs that match your specific use cases and implementing proper evaluation frameworks to ensure performance meets business requirements.”

Organisations that deploy SLMs must ensure they have the skills and governance policies to manage them responsibly, however. “Patching, provenance, bias testing and secure deployment all move to you,” says Corbridge. But when organisations pair SLMs with strong model lifecycle practices, they can “tap into their flexibility and lower long-term cost without giving up safety.”

Another advantage of open source platforms and models is the ecosystem itself. A global community of contributors is continuously adding new features and sharing best practices. By accessing this collective innovation, enterprises may find a faster path to AI maturity than they would if locked into a single vendor’s vision. In short, when it comes to AI openness can truly be a superpower.