Why everything we know is only half-right (Part-1)

Enterprise AI spend is rising fast. Average monthly AI costs have crossed $85,000 per organisation* and the trajectory has yet to flatten. When the CFO asks what the organisation is getting for this, teams struggle to give a confident response. The 2026 State of FinOps report captured it plainly: is your AI providing value? For most organisations, the answer remains unclear.

Most organisations are applying a governance model built for cloud infrastructure to a problem that is similar on the surface but meaningfully different underneath. By the time the mismatch becomes obvious, spend has already scaled considerably.

Our work at Microland spans cloud, data centres, AI infrastructure, model platforms, and enterprise applications. That full-stack view shapes the thinking in this two-part series. FinOps for AI (or AI FinOps) is closely related to Cloud FinOps, but it is not the same discipline. Understanding where the two diverge can prevent organisations from optimising the wrong costs, assigning ownership to the wrong teams, and creating significant rework later.

The cost model has shifted

Cloud FinOps was built around waste reduction. You identify idle resources, right size over-provisioned compute, and buy reserved capacity where utilisation justifies it, etc. The discipline rewards spending less. The central question in every review is: where are we spending without corresponding output?

AI FinOps cannot be governed that way. A well-run agentic pipeline that processes twice as many customer queries this quarter should be spending more than it did last quarter. A fine-tuned model that improves support resolution rates by 20% justifies higher inference costs if the business outcome is real. The question is not whether you are spending less but whether you are generating proportionally more value per dollar of AI spend.

This shift from waste reduction to value per unit of cost is the single most important reorientation in AI FinOps.

It changes what you measure, how you set targets, and who you hold accountable. Organisations that apply cloud FinOps thinking directly to AI end up optimising the wrong things, often cutting model quality to reduce token spend while the actual cost waste sits in idle GPU hardware or unused Copilot licences.

What the cost stack actually looks like

Ask most teams where AI costs live and they will say tokens. That is like saying cloud costs live in CPU cycles. Technically correct, practically insufficient as a governance framework.

AI infrastructure costs run across five distinct layers. Each has different cost drivers, different optimisation levers, and different team ownership. This excludes the physical data centre layer.

Layer-1: Silicon & Hardware. The GPUs, AI accelerators, and the power and cooling that support them. Enterprise average GPU utilisation for inference workloads sits at around 5% in most large deployments, which means 95 cents of every silicon dollar generates nothing. This is not a token efficiency problem. It is a workload scheduling, hardware selection, and capacity planning problem that requires infrastructure expertise to address, not FinOps dashboard configuration.

Layer-2: Interconnect, Memory & Storage. Above the hardware sits interconnect, memory, and storage. High-bandwidth networking between GPU nodes, HBM memory, and the KV cache that stores attention state during inference. The KV cache is a significant cost driver at scale that almost never surfaces in FinOps conversations. At high concurrency, per-node KV cache replication becomes extremely expensive. Shared persistent cache architectures can reduce this materially, but implementing them is an infrastructure architecture decision, not a quick configuration fix.

Layer-3: Model Serving & Foundational Software. This is where vLLM, TensorRT-LLM, and Triton live. The choices made here, including quantisation from FP16 to FP8, batching strategy, and speculative decoding, can move cost per token by three to five times in either direction. These are ML engineering decisions that no current FinOps tool surfaces or provides guidance on. They sit entirely below the API surface and are invisible to token-level monitoring.

Layer-4: API and Token Consumption. This is where most AI FinOps conversation begins and ends currently. Provider billing, LLM gateway attribution, model routing, prompt caching, and output token management are all meaningful activities. But they represent one layer of five, and in terms of optimisation potential they offer a narrower range than the layers underneath.

Layer-5: SaaS-bundled AI. The outermost layer where Microsoft 365 Copilot, Salesforce Einstein, ServiceNow Now Assist, GitHub Copilot sits. These products have no token layer visible to the enterprise at all. Governance here is a licence and adoption management problem. The cost levers are seat right-sizing and user enablement programs, not consumption efficiency.

Understanding which layer your AI costs sit in tells you immediately which team should own the governance conversation and which tools are relevant. Conflating all five into a single AI cost category leads to misdiagnosis and misallocated effort.

Consumption patterns need different approaches

A further complication is that AI value and cost behave differently depending on how the organisation is using AI. Three broad patterns exist in most enterprises, and they require genuinely different governance approaches.

Productivity AI: In productivity AI, which covers tools like M365 Copilot and Google Gemini for Workspace, the consumer is an individual knowledge worker. The cost structure is a flat seat fee. You never see a token. Value is measured as adoption rate, time saved per active user, and task completion across targeted use cases. The primary governance lever is an HR-led adoption program, not FinOps tooling. Only 20 to 30% of M365 Copilot seats are active in a typical week in most enterprises., Which means the cost problem is almost entirely an adoption problem, not a consumption efficiency problem.

Embedded AI: In embedded AI, where AI capabilities are built into customer-facing products or internal applications, value can be measured directly. Cost per recommendation accepted, cost per support ticket deflected, cost per transaction processed. Unit economics are calculable at the feature level, but only if applications are instrumented correctly from the start. Attribution at the feature level, through LLM gateway metadata, is what makes this possible.

Agentic AI: In agentic AI, which covers multi-step autonomous workflows that decompose tasks, call tools, and iterate to completion, the economics are genuinely new. The same task can cost ten times more on one run than another depending on the reasoning path the agent takes. Agentic workloads consume five to thirty times more tokens per task than equivalent single-turn interactions. Monthly budget reviews are structurally too slow for this dynamic. Governance needs to operate at the task level with hard budget limits and real-time alerting, not periodic reporting cycles.

Most published AI FinOps guidance treats these three patterns as a single category. They are not, and applying one governance model across all three produces poor results in at least two of them.

Why tagging is no longer enough

Cloud FinOps was built on resource tagging as a foundational element. Tag every resource with a cost centre, owner, and environment, and cost allocation follows. This still applies for IaaS-hosted AI workloads such as GPU training clusters and self-hosted inference endpoints, and it should be applied rigorously.

But tagging loses its foundational role as you move up the stack. SaaS-delivered AI has no taggable infrastructure. Microsoft controls the model, the API, and the billing. Multi-tenant API consumption through OpenAI, Anthropic, or AWS Bedrock allows some attribution through LLM gateway proxies, but attributing cost to a business outcome requires application-level context such as feature name, workflow ID, or user cohort. That context never reaches the billing layer automatically. Agentic frameworks generate costs across multiple models, multiple API calls, and multiple tool invocations within a single task execution. The tag on the agent container tells you almost nothing about the value of what the agent accomplished or how much the task actually cost in total.

The practical shift is toward telemetry at the application layer. LLM gateways that sit in the API call path and inject business context are the functional equivalent of resource tagging for API-consumed workloads. But they require application engineers to become the primary producers of cost attribution data, which is a meaningfully different operating model from traditional FinOps where infrastructure tagging drove everything.

What this means before you read Part-2

Part-1 of this series is about clarifying the current state of affairs. AI FinOps is a discipline that requires understanding which of three deployment models you are operating.  Where is the cost being generated, how is AI being consumed, and what business outcome should that consumption produce. The governance approach differs significantly across these dimensions, and most of the confusion in the market comes from treating them as one problem.

In Part 2, we move from diagnosis to practice: what carries over from your existing FinOps work, how governance differs across deployment models, what the tooling landscape honestly looks like today, who needs to own what, and a practical sequence for getting started.

It will also include a decision map tool that helps you unclutter the complexity of this new, rather enhanced FinOps discipline.

Stay Tuned.