Your AI Ambitions Need Better Foundations
Clients come to us excited to "do AI." We love doing AI. But sometimes, before we can get to the fun part, we need to talk about the bailing wire and bubble gum holding their cloud environment together.
That conversation belongs in the budget meeting. If you are funding an AI application that people will depend on, you also need to fund the infrastructure required to operate it. Otherwise, the project risks exposing weaknesses your existing software has been quietly living with.
You can experiment while you work on those weaknesses. You should. Just understand that a convincing demo tells you very little about whether the house is ready for the party.
Build a foundation for what you want to put on it
There is a minimum amount of structural integrity a house needs before it is safe to occupy. Better foundations give you room to build more. If you want a second story, the answer cannot be to spend the absolute minimum on whatever keeps the first floor standing.
The same reasoning should apply to an AI budget.
A tool that drafts product descriptions from public information, with a person reviewing every draft, has relatively limited authority. An application that reads customer records and issues refunds can cause a different class of problems. These are hypothetical examples, but they illustrate a real budgeting distinction: the second application needs stronger controls over what it can access and do.
The foundation determines which ambitions are reasonable. Getting access control, deployment, and monitoring into good shape lets a team contemplate more consequential applications. It gives them a way to make changes and investigate failures without treating every release as an expedition into the unknown.
So I would push back on an executive asking how little infrastructure they can get away with. Start with what you want the application to accomplish, including what you hope it can do next. Fund a foundation suitable for that ambition.
AI can make an old problem much easier to find
Consider a hypothetical assistant that searches internal documents. The demo uses a small collection of files selected for the occasion. It answers questions well. Everyone can see how this would save time.
Now connect it to a shared repository containing customer agreements, internal plans, and years of miscellaneous documents. Who is allowed to retrieve which files? Does the assistant enforce each user's permissions, or does it search with a service account that can read everything?
If the answer is the latter, you may have built a very convenient way to find information people should never have had access to.
That access problem might predate the AI project. An obscure file with overly broad permissions can remain unnoticed for years. A search assistant can make it easy to discover. The application team still has to deal with the consequences, regardless of which department originally created the mess.
Google Cloud's AI and ML security guidance calls for role-based access controls and least-privilege permissions, including for the service accounts used by AI pipelines and deployed models. The application needs access appropriate to its job. Giving it more access because that makes the demo easier is a poor production plan.
The data itself needs attention too. If the repository contains three conflicting versions of a policy, connecting a model does not establish which one is authoritative. Someone needs to make that decision and maintain the source. A polished answer can conceal a stale document remarkably well.
You need to be able to deploy it again
Deployment and operations would be my first serious discussion after understanding the proposed application.
Can your team reproduce the environment? Can they identify what changed between the working release and the broken one? Is there a tested recovery path, or does recovery mean finding the person who remembers how everything was configured?
These questions are ordinary software engineering. They become particularly useful when an application can change behavior because someone updated a prompt, replaced a model, or changed the documents it retrieves.
AWS's Generative AI Lens recommends infrastructure as code, version control, automated deployment processes, and observability across the application. It also calls for operational controls such as rate limits and workflow tracing. There is a reason this guidance spends time on activities less exciting than asking a model clever questions.
When a user reports a bad answer, the team needs enough information to investigate it. Was the source wrong? Did retrieval miss the relevant document? Did a recent change alter the response? A dashboard showing that the server returned a successful request will not answer those questions.
The amount of evaluation and tracing you need depends on the application. A human-reviewed drafting tool may have a modest release process. Software authorized to take consequential actions needs stronger evidence that changes are acceptable, plus a way to limit or stop those actions when necessary.
For each production service, somebody also needs responsibility for responding when it fails. That can be an existing team. You do not need to invent an AI governance department to decide who gets the alert.
We have been doing this for a while
I'm the lead author of Kubeflow for Machine Learning: From Lab to Production, published by O'Reilly. Getting useful model behavior into an application that can actually be operated is familiar territory.
Generative AI has made some kinds of experimentation much easier. It has not removed the need to control access, manage changes, and recover from failures. The operational customs established around software and machine learning still apply, with additional work where the model's behavior demands it.
This also does not mean every company needs to run Kubeflow. I helped write the book; you are still allowed to solve a problem without installing it.
A managed service may remove a substantial amount of infrastructure work. Using a hosted model can spare you from operating model-serving infrastructure yourself. You still have to decide what your application may send to it, what systems the application can reach, and whether its output is acceptable for the task.
Buying a service changes the division of work. It is worth being explicit about the work that remains yours.
Fix the parts your application depends on
There is a legitimate objection to all this: an infrastructure improvement program can expand until nobody remembers the application that prompted it.
You could spend years cleaning up a cloud environment. Some of that work might be valuable. Requiring all of it before a bounded experiment would be an excellent way to ensure the experiment never happens.
Scope the preparation around the application you intend to operate. Identify the data it reads, the services it depends on, and any changes it is authorized to make. Look at how that path is deployed, monitored, and recovered. Fix the weaknesses that make it unsafe or unmanageable.
A low-risk experiment can proceed in an isolated environment with appropriate data and limited permissions. An application handling sensitive information or changing business records requires more preparation before real users depend on it. There is no useful universal threshold expressed as a particular cloud spend or number of tools installed.
You also do not need to make every historical mistake in your AWS or GCP account disappear. You need a credible account of the dependencies and controls for the thing you are shipping.
Then consider the next version. If the plan is to move from drafting recommendations to executing them, budget for that change in authority. Stronger infrastructure can make the expansion practical. Waiting until the feature is ready to discover that the permissions model cannot support it is an expensive way to plan.
Put the foundation in the proposal
Before approving the next production AI project, ask the team to attach a short infrastructure-readiness plan to the proposal.
It should identify what the application can read and do, the gaps that must be closed before launch, and how the team will deploy, observe, and recover it. It should also explain which additional investments would enable the next level of capability. Give those items costs and owners so they can be considered alongside the application features.
That is the next step I would take. It makes the tradeoffs visible while there is still time to choose what to fund.
Keep the ambition. If you want a second story, budget for a foundation that can support it.
