Fact: Anthropic’s Commercial Terms require customers to comply with applicable laws and the Usage Policy (legal and compliance). Customers—not the model—bear responsibility for how outputs are used, including human review where appropriate in high-risk domains (usage policy update).
Interpretation: Vendor compliance is bilateral. Anthropic governs your use of Claude; you govern your use of third-party scholarly content inside Claude. Passing a PDF to an API does not transfer copyright clearance any more than photocopying transfers ownership.
Deployers of AI scholarly workflows should maintain controls across six domains:
| Domain | Question |
|---|---|
| Platform tier | Commercial/API vs consumer? |
| Copyright & license | OA status, license, intended reuse |
| Contractual | Library licenses, publisher ToS, Unpaywall API terms |
| Data protection | GDPR, HIPAA, confidential research data |
| Provenance | Can you reconstruct what was retrieved and why? |
| Governance | Who approves high-risk literature use cases? |
Missing any row produces audit findings—even when every retrieved article was “open access.”
Fact: Anthropic’s consumer terms (Free, Pro, Max) differ from Commercial Terms on data retention and whether inputs may be used to improve models, subject to user settings (consumer terms update). Commercial API and Enterprise paths offer stronger defaults against training on customer content.
Fact: Zero Data Retention (ZDR) arrangements for the API limit how long customer data is stored at rest (ZDR documentation). ZDR is an organizational setting—not automatic on all tiers.
Risk: Employees pasting paywalled or confidential PDFs into consumer Claude for “quick summaries” bypasses enterprise DPA, retention controls, and license restrictions simultaneously. Shadow AI is a scholarly-compliance problem, not only a security problem.
Control: Mandate Commercial or API access for work-related scholarly workflows; block or monitor consumer endpoints where feasible; train researchers that OA discovery does not equal upload permission.
Scholarly articles are not automatically non-personal data. Clinical trials, case reports, genetics studies, and social-science fieldwork may contain personal data subject to GDPR (jurisdiction-specific) or protected health information under HIPAA in the United States.
Fact: Anthropic offers HIPAA-ready API access with a Business Associate Agreement when ZDR is activated for the organization (legal and compliance — BAA section).
Risk analysis: Uploading a patient-enriched oncology PDF to a non-HIPAA, non-ZDR endpoint may constitute unauthorized processing—even if the article is OA. OA addresses copyright access, not data-protection classification.
Deployers should:
Fact: Institutions often subscribe to journals under licenses that restrict systematic download, text mining without addendum, or sharing with third-party processors—including AI vendors.
Interpretation: Unpaywall finding an OA copy does not override a separate subscription agreement for the toll-access version an employee uploaded from a desktop. Conversely, absence of OA does not authorize uploading a subscription PDF to Claude if the license prohibits it.
Library and research offices should publish clear guidance: which retrieval paths are approved (OA resolver, interlibrary loan, licensed platform APIs) and which are not (paste into consumer chat).
Regulators and enterprise auditors increasingly ask not what did the model say but what did it read.
Minimum provenance log per retrieval event:
oa_statushost_typeFor organizations deploying systems that may qualify as high-risk under the EU AI Act, transparency obligations toward deployers appear in Article 13 of Regulation (EU) 2024/1689 (EUR-Lex, AI Act Service Desk Art. 13). Applicability depends on system classification—another jurisdiction-specific analysis requiring counsel.
Anthropic’s Transparency Hub documents platform-side enforcement and legal process handling; it does not replace deployer-side logging of scholarly sources.
NC-licensed corpus in a commercial product. An agent ingests CC BY-NC articles discovered via Unpaywall into a revenue-generating RAG application. License violation regardless of lawful access.
Confidential proposal plus third-party PDFs. A researcher uploads a draft grant containing unpublished results and attaches licensed papers. Data leak plus potential copyright breach.
RAG index retains paywalled content. Scraper misclassifies a toll-access PDF as OA; embeddings persist after takedown. Provenance failure turns a fetch error into a sustained compliance debt.
Consumer-tier clinical summarization. Hospital team uses Pro-tier Claude on OA cancer literature containing trial participant details. Potential HIPAA/GDPR exposure independent of OA status.
A hospital research team uses Claude to summarize recent OA cancer literature. Trial reports include demographic tables with small-cell counts. Staff use consumer-tier paste because “the papers are open access.” Copyright access may be fine; PHI handling is not. Tier selection failed. Provenance is absent—the compliance team cannot reconstruct which articles entered which sessions during an investigation.
“Anthropic’s terms protect us.” They protect the customer–vendor relationship, not customer–publisher relationships.
“We only use OA papers.” OA does not eliminate NC restrictions, bronze ambiguity, PHI in articles, or API tier requirements.
“Provenance is engineering overhead.” It is cheaper than post-incident reconstruction under regulatory inquiry.
User compliance is operational, not theoretical. The tools make retrieval easy; governance makes it defensible. Publishers and rights holders face a mirror-image problem: legal OA increases readership while AI-scale aggregation erodes control they once exercised through access restrictions alone.
Enterprise AI governance for research workflows—tiering, logging, DPIA support. Contact me.
Fact: SPARC defines open access as “the free, immediate, online availability of research articles coupled with the rights to use these articles fully in the digital environment” (SPARC Open Access). That definition intentionally pairs access with reuse rights—it is not synonymous with “free to read behind a registration wall.”
Interpretation: “Open” describes a publishing and rights architecture, not a single revenue model. Payment can flow through article processing charges (APCs), institutional agreements, funder mandates, society subsidies, or unfunded green archiving. AI-mediated discovery and summarization add a new value-capture layer that most legacy OA economics did not anticipate.
| OA model | Who typically pays | Creator payment signal |
|---|---|---|
| Gold OA | Author, institution, or funder (APC) | Publisher receives APC; author may bear cost |
| Hybrid OA | APC for OA articles in subscription journal | Publisher paid per OA article; journal remains hybrid |
| Green OA | Often no per-read payment | Author retains some rights via repository deposit |
| Bronze OA | Reader pays nothing | Publisher may monetize attention; no clear reuse license |
Unpaywall’s oa_status field encodes this taxonomy (oa_status definitions). AI builders who treat all is_oa: true records as economically equivalent misunderstand what creators actually received when they published.
SPARC’s factsheet notes that CC BY is the standard license aligned with full OA reuse (SPARC factsheet PDF)—but CC BY governs permission, not payment.
Many authors transfer copyright to publishers under standard publication agreements, then receive back specific rights—or publish under a Creative Commons license on the OA copy. The SPARC Author Addendum is a legal instrument designed to help authors retain distribution, repository, and derivative-work rights that default contracts would otherwise withhold (addendum text).
Interpretation: “Open” for readers does not mean “paid fairly” for authors. An author may pay a $3,000 APC, publish under CC BY, and still receive zero revenue when a commercial AI product ingests the paper at scale—while remaining license-compliant.
Georgetown University Library’s guidance on negotiating contracts illustrates how publisher-retained rights often restrict which versions authors may share and where (negotiating your contract). AI adds pressure because new uses (embedding, summarization markets, training) may fall outside rights authors thought they retained.
A typical AI-mediated literature workflow looks like this:
Unpaywall discovery → HTTP fetch → parse/chunk → embed or summarize → product output
Value accrues at the summarization and product layers—often to the AI vendor or the deployer’s application—not to the author or publisher. Creators may receive citation uplift without royalty participation.
The U.S. Copyright Office’s pre-publication Part 3 report on generative AI training (Part 3 PDF) examines whether training on copyrighted works requires permission or compensation. Fact: The Office concludes that training may implicate exclusive rights and that fair use must be assessed case-by-case—not categorically. Fact: The report emphasizes that existing or feasible licensing markets weigh against fair use under the fourth factor.
Interpretation: Even if retrieval for summarization and training are legally distinct activities, they share an economic theme: creators may lack payment when AI systems commercialize their scholarship.
Fact: Creative Commons Attribution 4.0 (CC BY) permits sharing and adaptation for any purpose, including commercially, provided you give appropriate credit, link to the license, and indicate changes (CC BY 4.0).
Fact: CC BY does not require payment to authors.
Risk analysis: A commercial literature-RAG vendor that ingests CC BY corpora discovered via Unpaywall may be license-compliant while economically contentious. Authors and funders who paid APCs expecting public benefit may not have expected uncompensated commercial aggregation.
Authors who intend to restrict commercial AI reuse should consider licenses such as CC BY-NC—but funders and journals increasingly mandate CC BY for maximum dissemination (Katina Magazine on CC BY). That tension is policy, not a technical bug in Unpaywall.
United Kingdom: As of late 2025, the UK government had not adopted a broad commercial text-and-data-mining exception; official statements emphasize that copyright material generally cannot be used for AI development without permission (BBC reporting on government position). The existing UK TDM exception is limited to non-commercial research—relevant to training, not merely reading OA PDFs.
European Union: The CDSM Directive provides TDM exceptions with opt-out mechanisms for rightholders—primarily relevant to copying for mining/training, distinct from accessing an OA PDF a human could read. Deployers operating across borders need counsel on which regime applies.
These points are jurisdiction-specific and evolve quickly; do not treat this post as a current law memo.
A professor publishes gold OA in a hybrid journal after paying a $3,000 APC, under CC BY. A commercial AI research tool ingests the PDF via Unpaywall, indexes millions of similar articles, and sells synthesized literature briefs to investors. Access is lawful. Many reuse forms are CC BY-compliant if attribution is correct. The author receives citations but no AI royalty. The funder’s compliance office asks whether the APC purchased only reader access—or also permission for unchecked third-party commercialization. That question has no universal answer in today’s licensing market.
Before publishing OA—or when reviewing an existing corpus exposed to AI retrieval—creators and research offices should verify:
“OA means authors chose to give everything away.” OA chooses a rights architecture—often CC BY—not necessarily uncompensated commercial exploitation by AI intermediaries.
“AI summarization helps authors through visibility.” Citation uplift is real but uneven; it is not a substitute for payment where markets might otherwise exist.
“Funders require CC BY, so authors have no choice.” Policy tension is genuine; advocacy and licensing markets are the venue—not covert paywall bypassing.
Payment and permission diverge in the AI era. Legal OA clears many access barriers; it does not automatically align creator economics with new AI value chains. Organizations that deploy AI tools face their own parallel obligation: staying compliant when they consume scholarly content, regardless of how frictionless retrieval has become.
Publishing teams and research offices: I help map OA licenses to AI use cases and funder compliance. Get in touch.
Compliance teams benefit from a layered model rather than a binary “legal/illegal” flag. Think of four concentric rings:
Ring 1 — Discovery services. Unpaywall API, library link resolvers, discovery layers (EBSCO, Primo, WorldCat). Lowest friction when used as documented. Contractual obligations here are primarily API terms and rate limits (REST API).
Ring 2 — Lawful access endpoints. Publisher OA pages, institutional repositories, preprint servers, subject archives. The bytes live here. Access may be free to humans; automated access may still be constrained.
Ring 3 — Permitted use. License terms (CC BY, CC BY-NC, publisher custom), embargo rules on green OA, citation requirements. This ring governs what you may do after lawful access.
Ring 4 — Prohibited or high-risk conduct. Toll-access content without authorization, credential sharing, DRM or technical protection measure (TPM) circumvention, ignoring rightholder opt-outs, training commercial models on NC-licensed corpora without permission.
An AI pipeline can pass Ring 1 and Ring 2 while failing Ring 3 or Ring 4. That is the compliance boundary this post addresses.
Fact: The Unpaywall browser extension invites users to “skip the paywall on millions of peer-reviewed journal articles” by clicking when a legal free copy exists (extension). The marketing language reflects user experience on toll-access landing pages—not unauthorized access to content publishers have withheld.
Fact: Library link-resolver integrations use Unpaywall data to offer OA copies when the institution lacks a subscription (link resolver integrations). When no OA location exists, redirecting to the toll-access publisher page is described as correct behavior.
Interpretation: These patterns are structurally different from services that obtain content through stolen credentials, exploit kits, or infringing mirrors. Unpaywall is not comparable to those tools—and compliance discussions should not collapse them into one category.
The following patterns sit outside lawful OA discovery even when Unpaywall or similar tools appear somewhere in the toolchain:
closed or missing OA signals and brute-forcing alternate URLs, leaked copies, or third-party infringing hosts.Sci-Hub and similar infringing services are not OA discovery—they are unauthorized distribution. Any AI pipeline that routes through them fails compliance regardless of how the DOI was resolved.
Even when Ring 2 access is free for human readers, Ring 3 and site policies may restrict automation.
Risk analysis: Headless browsers fetching thousands of OA PDFs overnight can trigger bot management, breach-of-contract claims under terms of service, and—in the U.S.—debates about computer fraud and abuse statutes that courts have applied inconsistently (jurisdiction-specific; not legal advice). The Congressional Research Service provides a neutral overview of generative AI and copyright intersections (CRS LSB10922), but site-access questions often turn on contract and computer-access law rather than copyright alone.
The operative distinction for governance:
| Pattern | Typical risk profile |
|---|---|
| User-initiated single fetch for session summarization | Lower |
| Unattended corpus harvest into persistent storage | Higher |
| Commercial RAG index built from publisher domains at scale | Highest without explicit license |
Fact: Unpaywall’s API terms limit daily call volume (REST API). That is a contractual boundary distinct from—but related to—publisher scraping norms.
The Unpaywall extension operates in a human context: one page, one article, one click. An AI agent operates in a machine context: loops, retries, parallel fetches, unpredictable volume, and downstream storage.
Compliance controls for agents should include:
Anthropic’s Usage Policy updates clarify platform-side prohibitions on malicious computer and network compromise. Your retrieval agent’s behavior must comply with both vendor policy and publisher policy—the stricter effective control wins.
Bronze OA. Free on the publisher site but license null in Unpaywall metadata (oa_status definitions). Access may be lawful; reuse for AI products is ambiguous. Default-deny for automated commercial reuse unless counsel approves.
Version mismatch. Green repository copies may be preprints or author-accepted manuscripts, not the version of record. Citation norms, commercial use, and publisher policies may differ (openaccess.nl copyright guide).
Paratext and non-article DOIs. Unpaywall indexes Crossref DOIs broadly; not every DOI is a journal article (support FAQ). Garbage DOIs in, garbage compliance out.
Extension-only bronze detection. The extension may surface bronze when it finds a PDF on-page without going through the API—behavior documented in Unpaywall support materials. Agents relying only on API metadata may classify the same article differently.
A compliance team audits a “research agent” that queries Unpaywall, then fetches ten thousand PDFs overnight from publisher domains into cloud storage for RAG indexing. Ring 1 (API use) may be lawful if rate limits were respected. Ring 2 (many copies are OA) may be lawful for access. Ring 3 (licenses vary; bronze and NC articles mixed in) is unreviewed. Ring 4 (site ToS, scraping norms) is likely violated regardless of OA status. Discovery was not the failure point—automation scale and reuse without license triage was.
Build—or buy—a policy engine that evaluates each candidate document before fetch and again before retain:
Inputs: doi, oa_status, license, host_type, robots_txt, intended_action
Actions: display | summarize_ephemeral | store | embed | train
Decision: ALLOW | REVIEW | DENY
Recommended defaults:
license is null“OA means we can scrape freely.” Bronze and green complicate this. Free to read ≠ free to bulk-ingest for commercial AI.
“We’re just like the browser extension.” Scale, automation, storage, and product embedding change the risk profile. Equivalence is an engineering claim, not a legal defense.
“The API gave us a URL, so we’re covered.” The API gives a location, not a use license for your product.
The compliance boundary is behavioral and contextual, not merely technical. Passing Unpaywall’s API is necessary but never sufficient. On the other side of the access question, creators and publishers must also ask whether open access and AI economics actually pay those who produce the work—a tension this boundary analysis does not resolve on its own.
Need a boundary assessment for your AI retrieval stack? Let’s map discovery, access, and reuse rings to your controls.
| **Browser Extension | Unpaywall** — OurResearch — https://unpaywall.org/products/extension |
| **Integrations | Unpaywall** — OurResearch — https://unpaywall.org/integrations |
| **REST API | Unpaywall** — OurResearch — https://unpaywall.org/products/api |
Every scholarly retrieval workflow has two stakeholders whose incentives do not automatically align.
Side A — the AI user or builder: Can a system like Claude lawfully use Unpaywall-style open-access discovery to reach content?
Side B — the creator, publisher, or rights holder: Even when access is legal, are you sure you get paid, stay compliant, and remain protected when AI systems ingest or repackage your work?
Both sides deserve equal seriousness. This article does not frame the topic as paywall bypassing. Unpaywall is an open-access discovery service, not a piracy tool. Throughout, we distinguish facts (what a system does), interpretation (how law or policy may apply), and risk analysis (what can go wrong operationally).
The focus here is mechanics and first-order questions around discovery and initial retrieval—not model training, bulk redistribution, or commercial exploitation.
Fact: Unpaywall is an open database operated by the nonprofit OurResearch. It indexes open-access locations for scholarly articles identified by Crossref Digital Object Identifiers (DOI), drawing on data from tens of thousands of publishers and repositories.
Fact: When the browser extension detects a DOI on a page, it queries the Unpaywall API to retrieve the best known open-access location for that article (FAQ). The API returns structured JSON including is_oa, oa_status, and a best_oa_location object with fields such as license, host_type, and URLs (data format).
Fact: Unpaywall does not crack paywalls. It points to copies that publishers or repositories have made available. If no open-access location exists, integrations may redirect to the toll-access publisher page—behavior that link-resolver documentation describes as working correctly (link resolver integrations).
Interpretation: Describing Unpaywall as a “piracy tool” mischaracterizes its stated function. Libraries, discovery systems, and link resolvers worldwide integrate it as legitimate scholarly infrastructure (integrations). That said, misuse of URLs returned by Unpaywall—for example, unattended bulk scraping that violates publisher terms—is a separate compliance problem from discovery itself.
Unpaywall assigns each article an oa_status: gold, green, hybrid, bronze, or closed (oa_status definitions). These labels describe where and how openness was achieved—not a blanket grant of rights for every downstream use.
| Status | Plain-language meaning | Compliance note |
|---|---|---|
| Gold | Published in a fully OA journal | Often carries explicit license (e.g., CC BY) |
| Hybrid | OA article in otherwise toll-access journal | License usually present on OA copy |
| Green | Archived in repository | Version and license may differ from publisher VoR |
| Bronze | Free to read on publisher site | license field may be null—reuse rights unclear |
| Closed | No OA location indexed | Discovery may still return publisher toll page |
Risk: Treating is_oa: true as permission for AI ingestion, summarization-for-resale, embedding in a commercial RAG product, or model training without reading the license is one of the most common compliance failures we see in design reviews.
SPARC defines open access as free, immediate, online availability plus the rights to use articles fully in the digital environment (SPARC Open Access). That definition makes clear that OA is not merely “free to read”—but the extent of reuse rights still depends on the specific license attached to each copy.
The Unpaywall browser extension is a human-in-the-loop pattern: a researcher on a publisher page clicks a tab when a legal OA copy exists (extension). An AI agent workflow is structurally different:
GET /v2/{doi}?email=...)Each step can invoke different obligations: API terms, copyright reproduction, database rights (jurisdiction-specific), site terms of service, and license conditions on reuse.
Provenance requirement: At fetch time, log the DOI, oa_status, license string, host_type, URL fetched, timestamp, and document version where identifiable. Without this record, later audit—regulatory, contractual, or internal—is guesswork.
Think of compliance as three stacked layers, not one switch:
Layer 1 — Discovery compliance: Are you using the Unpaywall API as intended? Requests require a valid email parameter; the documented limit is 100,000 calls per day (REST API). High-volume or commercial deployments should contact OurResearch about data feeds rather than assuming anonymous bulk access is appropriate. Agent loops that hammer the API invite HTTP 429 responses and reputational harm.
Layer 2 — Access compliance: Is the copy at the returned URL lawfully available for your fetch? For hybrid or gold OA with a clear license, access is typically uncontroversial. For bronze OA or green repository copies, access may be lawful while reuse remains ambiguous.
Layer 3 — Reuse compliance: Does your intended use—summarize, store, adapt, train, redistribute—fall within the license, publisher policy, and applicable law? Access does not imply reuse rights. A CC BY article permits broad reuse with attribution (CC BY 4.0); a bronze article with no license metadata does not give you a safe default.
Anthropic’s legal and compliance documentation makes an important distinction: your relationship with the AI vendor is governed by Commercial Terms (API, Team, Enterprise) or Consumer Terms (Free, Pro, Max). These terms regulate data retention, training on your inputs, and acceptable use—they do not grant copyright permissions for third-party scholarly works.
Fact: Under Commercial Terms, Anthropic does not use customer API content for model training by default. Consumer tiers have different retention and opt-in training settings (consumer terms update).
Interpretation: Choosing the right tier is a compliance control for your organization’s data, not a substitute for publisher licenses. The Usage Policy also restricts certain high-risk uses and malicious automation; it does not replace Creative Commons or publisher terms.
There is no universal yes/no answer—outcomes depend on use case, license, jurisdiction, and scale.
Likely lower-risk pattern (interpretation, not legal conclusion): A human-supervised research assistant on Commercial/API terms queries Unpaywall for a known DOI, fetches a hybrid-OA article licensed CC BY, summarizes it with proper attribution for an internal literature review, and does not retain full text beyond the session—provided publisher and repository terms of service do not prohibit the automated fetch.
Higher-risk patterns:
license: null) for commercial RAGis_oa aloneUncertain and jurisdiction-specific: Whether transient copies made solely for summarization qualify as fair use or fair dealing; how EU CDSM text-and-data-mining exceptions apply; the UK’s narrow TDM exception limited to non-commercial research. The U.S. Copyright Office AI initiative is actively examining training and reuse questions but does not resolve retrieval-for-summarization in every scenario.
A biotech startup wires a Claude agent to Unpaywall for competitive intelligence. The agent fetches a hybrid-OA oncology paper tagged CC BY in Unpaywall’s API response. Discovery is consistent with Unpaywall’s design. Access to the publisher-hosted OA PDF is likely lawful. Reuse becomes the crux: embedding chunks in a commercial product requires CC BY attribution compliance, human oversight for medical claims, and a decision about whether persistent vector storage counts as “adaptation” under the license. None of that is settled by the green tab alone.
“Unpaywall says it’s legal, so our pipeline is legal.” Unpaywall locates OA copies authorized by publishers and repositories. It does not warrant your use case, jurisdiction, volume, or downstream product.
“If it’s on the web, Claude can read it.” Visibility is not permission. Robots.txt, terms of service, and license terms may restrict automated access even when a human could read the same page for free.
Conflating extension UX with headless scraping. The extension operates at human scale with single-article intent. Agents operating at corpus scale change the risk profile even when the underlying OA copies are the same.
oa_status, license, and host_type for every retrieved documentlicense is null (bronze) unless legal review approvesis_oa aloneLegal open-access discovery is a legitimate scholarly infrastructure pattern—and for many research workflows, Unpaywall is an appropriate starting point. It is the start of compliance, not the end. The boundary between lawful OA access and circumvention, scraping, and terms-of-service violations is a separate layer—governed by scale, automation, and license scope as much as by whether a copy was discoverable in the first place.
Designing agentic research workflows? If you need a compliance architecture review—discovery, provenance, data retention, and license-aware retrieval—contact me directly to discuss a focused engagement.
| **FAQ | Unpaywall** — OurResearch — https://unpaywall.org/faq — Defines extension/API discovery model and DOI-based lookup. |
| **REST API | Unpaywall** — OurResearch — https://unpaywall.org/products/api — API requirements, email parameter, rate limits. |
| **Data Format | Unpaywall** — OurResearch — https://unpaywall.org/data-format — Schema for license, host_type, OA locations. |
Tokenmaxxing is the implicit strategy of the 2023–2025 AI adoption wave: use the most capable model available, maximise context windows, deploy agents liberally, and treat AI costs as a growth investment that will be justified by productivity gains. The rationale was defensible in the early adoption phase — when AI capabilities were advancing rapidly, the cost of under-investing in capability was higher than the cost of over-spending on tokens.
That calculus has inverted. Frontier model capabilities have plateaued relative to the cost curve. The gap between a Haiku-class model and a Sonnet-class model on most production tasks is far smaller than the 10x pricing gap between them. The gap between a 4-bit quantised Mistral 7B and a frontier API model on classification and extraction is operationally negligible for the majority of enterprise workloads. Meanwhile, the budget consequences of unconstrained API spend have compounded — as Uber discovered — to the point where AI tooling is consuming budget at a rate that is not sustainable without explicit governance.
Economic governance does not mean restricting AI usage. It means ensuring that every dollar of AI spend is allocated to the workload tier that requires it, that usage is instrumented and attributed, and that the organisation has the data to make rational decisions about where to invest and where to optimise.
The synthesis of this series yields a framework organised across seven domains, each corresponding to a specific cost vector addressed in the preceding posts.
Layer 1: Token Metering and Attribution Every API call must be attributed to a cost centre, a team, a product, and a workload type. This is the prerequisite for all other governance. Without attribution, optimisation is guesswork. Instrument at the API gateway layer — not the application layer — so that attribution is infrastructure-enforced rather than developer-maintained.
Layer 2: Context Window Discipline The context tax is real and compounding. Establish organisational standards for what belongs in a context window: what is a required system prompt, what is optional, what is never appropriate. Measure average input token counts per workload. Any workload averaging more than 10,000 input tokens per call without a clear quality justification is a candidate for context pruning.
Layer 3: Agent Step Budgets Every agentic workflow must have an explicit step budget — a maximum number of tool calls or LLM invocations per task execution. No agent should run unbounded. The budget should be set empirically: measure the step distribution for a sample of successful task executions, set the cap at the 95th percentile, and implement circuit-breaker logic to halt and report on runs that hit the cap.
Layer 4: Billing Model Routing Not all workloads should use the same billing model. Token-per-call pricing is optimal for low-token-density text workloads. Per-minute pricing (Gemini Live API) is optimal for real-time audio and high-token-density multimodal workloads. Build routing logic that assigns each workload class to its economically optimal billing model, and re-evaluate the routing quarterly as pricing evolves.
Layer 5: Prompt Cache Architecture Every production prompt must be audited for its static and dynamic components. The static portion — system prompts, knowledge bases, fixed instructions — should be cached. Target a cache hit rate of 80% or higher for high-volume workloads. Treat the cache hit rate as a production SLO, alongside latency and error rate.
Layer 6: Infrastructure and Make/Buy Policy Establish a volume threshold policy for each workload type: below the threshold, API inference is the default; above the threshold, local inference on SLMs is evaluated. For most enterprise workloads, this threshold is in the range of 3–6 million documents or API calls per month at Haiku-tier pricing. Review the policy annually as hardware costs, model quality, and API pricing evolve.
Layer 7: Upstream Risk Management Monitor provider infrastructure constraints — data centre capacity, power availability, hardware supply chains — as leading indicators of API price trajectories. Maintain multi-provider capability (not just multi-provider integration, but tested and production-ready fallback routes) for all critical AI workloads. Provider concentration risk is infrastructure risk, not just vendor risk.
Organisations fall into four tiers of AI economic maturity:
| Tier | Description | Indicators |
|---|---|---|
| Tier 0: Unaware | No cost instrumentation, no governance | Budget surprises, post-hoc reconciliation |
| Tier 1: Measured | API costs attributed and visible | Monthly spend dashboards, per-team visibility |
| Tier 2: Managed | Costs instrumented and governed | Caps enforced, caching implemented, model tiers assigned |
| Tier 3: Optimised | Continuous cost/quality optimisation | Routing logic, SLM deployment, cache hit SLOs |
| Tier 4: Governed | Economic governance as org capability | AI FinOps function, policy framework, audit trail |
Most organisations that have been running AI in production for 12+ months are at Tier 1 or early Tier 2. The Uber case is a Tier 0 organisation that scaled to high spend without progressing through the tiers. The organisations that will define the AI economics of 2027 are moving from Tier 2 to Tier 3 now.
If you are a CTO reading this series and wondering where to start, the sequence that delivers the highest ROI per unit of engineering effort:
Instrument first. Deploy API gateway-level token metering with team and workload attribution. If you cannot see the spend, you cannot govern it. Time to value: 2–4 weeks.
Cache your system prompts. Enable prompt caching for every production workload with a stable system prompt longer than 1,000 tokens. This requires only a prompt restructuring and a cache header — no infrastructure change. Expected savings: 40–70% of input token costs for affected workloads. Time to value: 1 week.
Assign model tiers. Audit every production workload against the model you are using. For classification, extraction, and structured generation workloads on Opus or Sonnet, migrate to Haiku or Flash. Expected savings: 60–90% on those workloads with minimal quality impact. Time to value: 2–4 weeks per workload.
Budget your agents. For every agentic workflow, add a step counter and a circuit breaker. Cap at 95th-percentile empirical step counts. Time to value: 1–2 weeks per agent.
Evaluate local inference. For your top two or three high-volume workloads, run the break-even calculation from Post 7. If any are above the threshold, pilot a local SLM deployment. Time to value: 4–8 weeks for pilot.
Build the routing layer. Once tiers and models are assigned, invest in routing infrastructure that enforces the assignments programmatically. This is the foundation of Tier 3 maturity. Time to value: 4–8 weeks.
A fully governed AI stack does not pay the full token price for its workloads. It pays:
\[C_{governed} = \sum_{w \in W} V_w \cdot \left( f_{cache}(w) \cdot P_{cache} + (1 - f_{cache}(w)) \cdot P_{tier}(w) \right) + C_{local}\]Where:
In practice, a governed organisation pays 30–50% of what an equivalent ungoverned organisation pays for the same AI output quality. At the scale of a mid-to-large enterprise, this difference is measured in millions of dollars annually — and it is the difference between an AI budget that is defensible in a CFO review and one that produces the next Uber headline.
The AI capability curve will continue upward. Models will become more capable, context windows will expand further, and agentic workflows will become more autonomous and more deeply embedded in engineering processes. Every one of those trends increases both the value of AI investment and the potential cost of ungoverned AI consumption.
The organisations that build economic governance now — when the costs are measurable and the frameworks are available — will enter 2027 with a structural advantage. They will be able to absorb new capabilities without budget crises because they have the instrumentation to understand what they are buying and the architecture to route spend appropriately.
The organisations that do not will face the same reckoning Uber faced, at higher absolute costs, with more entrenched dependencies and less flexibility to change course.
Don’t let your 2027 budget evaporate. The guardrails Uber missed are not complex — they are the governance primitives this series has described. If your organisation is at Tier 0 or Tier 1 and you need to close the gap before your next planning cycle, contact me directly to discuss a focused AI FinOps engagement: instrumentation, model tier audit, caching architecture, and a prioritised roadmap calibrated to your specific stack and spend profile.
This post concludes the 2026 AI Token Economy series. The full series:
The past 18 months have seen a qualitative shift in what small models can do. The models that define the current frontier of efficient inference:
| Model | Parameters | Context Window | Notable Strength |
|---|---|---|---|
| Microsoft Phi-3 Mini | 3.8B | 128K | Reasoning, code generation |
| Microsoft Phi-3 Medium | 14B | 128K | Broad task coverage |
| Mistral 7B v0.3 | 7B | 32K | Instruction following, multilingual |
| Mistral Nemo | 12B | 128K | Long-context, function calling |
| Meta Llama-3.1 8B | 8B | 128K | General purpose, strong benchmarks |
| Meta Llama-3.1 70B | 70B | 128K | Near-frontier quality at smaller footprint |
| Google Gemma 2 9B | 9B | 8K | Efficient, strong for size |
| Qwen-2.5 7B | 7B | 128K | Code, maths, Chinese-English |
These are not toy models. Phi-3 Mini at 3.8 billion parameters scores above GPT-3.5-Turbo on MMLU (Massive Multitask Language Understanding) benchmarks. Llama-3.1 8B approaches GPT-4-level performance on structured reasoning tasks when deployed with careful prompt engineering. For well-scoped enterprise tasks — classification, extraction, summarisation, code review, structured data generation — models in the 7–14B range meet quality bars that would have required frontier models as recently as 2023.
A full-precision (FP16 or BF16) 7B parameter model requires approximately 14GB of GPU VRAM — the full model weights loaded into memory for inference. This demands at minimum a single A10G GPU (24GB VRAM) for comfortable operation, or two consumer-grade RTX 4090s (24GB each).
4-bit quantisation reduces each weight from a 16-bit floating-point number to a 4-bit integer representation, shrinking the memory footprint by 75%. A 7B model in 4-bit (GGUF format via llama.cpp, or GPTQ/AWQ for GPU inference) fits in approximately 4–5GB of memory — within the VRAM budget of a single RTX 4090, or on CPU RAM with acceptable throughput for moderate-volume workloads.
The quality trade-off is measurable but, for most production tasks, acceptable:
| Task Category | FP16 Quality | 4-bit Quality | Degradation |
|---|---|---|---|
| Text classification | Baseline | ~97% of baseline | ~3% |
| Named entity extraction | Baseline | ~95% of baseline | ~5% |
| Code generation (simple) | Baseline | ~93% of baseline | ~7% |
| Complex multi-step reasoning | Baseline | ~85% of baseline | ~15% |
| Creative generation | Baseline | ~88% of baseline | ~12% |
For classification and extraction workloads — which represent the majority of enterprise AI deployments — 4-bit quality is operationally indistinguishable from full precision. The degradation concentrates in complex reasoning and creative tasks, which are also the workloads that most justify using a frontier model in the first place.
The break-even analysis between local inference and API inference requires modelling the full TCO of a local deployment against the API spend it replaces. Consider a mid-sized enterprise with the following workload profile:
\(C_{API/month} = 500{,}000 \times \left( \frac{800}{1{,}000{,}000} \times 0.25 + \frac{200}{1{,}000{,}000} \times 1.25 \right)\) \(= 500{,}000 \times (0.0002 + 0.00025) = 500{,}000 \times 0.00045 = \$225/\text{month}\)
At this volume, the API cost is modest — $225/month, or $2,700 annually. A local deployment would not be justified on cost grounds alone.
Now scale to a high-volume deployment:
Local deployment TCO (4-bit Mistral 7B on single A100 80GB server):
| Cost Category | One-Time | Monthly | Annual |
|---|---|---|---|
| A100 80GB server (used/leased) | $18,000 | — | — |
| Co-location power + rack | — | $400 | $4,800 |
| Engineering setup (one-time) | $8,000 | — | — |
| Maintenance + monitoring | — | $200 | $2,400 |
| Total Year 1 | $33,200 | ||
| Total Year 2+ | $7,200 |
At 10M documents/month, the API spend is $54,000/year. Local deployment costs $33,200 in year one and $7,200 in year two. Break-even occurs during year one; year two savings are $46,800.
The formula for the volume threshold at which local inference becomes economically superior:
\[V_{break-even} = \frac{C_{capex} + C_{opex\_year1}}{C_{API\_per\_doc}}\]Where \(C_{API\_per\_doc} = 0.00045\) per document in this example, and \(C_{capex} + C_{opex\_year1} = \$26,200\) (excluding engineering):
\[V_{break-even} = \frac{26{,}200}{0.00045} \approx 58{,}200{,}000 \text{ documents/year} \approx 4{,}850{,}000 \text{ documents/month}\]Below approximately 5 million documents per month at Haiku pricing, the API wins on economics. Above that threshold, local inference wins in year one and dominates from year two onward.
The tooling for local SLM deployment has matured significantly. llama.cpp provides CPU and GPU inference for GGUF-quantised models with minimal dependencies — a single binary, a model file, and an OpenAI-compatible API endpoint that drops in as a replacement for cloud API calls with no application code changes.
Ollama wraps llama.cpp with a model registry, CLI management, and an HTTP API, reducing deployment to a three-command sequence:
# Pull the model
ollama pull mistral:7b-instruct-q4_K_M
# Serve it
ollama serve
# Call it via OpenAI-compatible endpoint
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "mistral:7b-instruct-q4_K_M", "messages": [{"role": "user", "content": "Classify this document..."}]}'
The OpenAI-compatible endpoint means that any application using the standard OpenAI SDK can switch to local inference by changing the base_url parameter — no business logic changes required. This is the migration path that makes the break-even calculation actionable rather than theoretical.
The economically optimal architecture for most enterprises is not a binary choice between API and local inference — it is intelligent routing based on task complexity and volume:
This tiered architecture combines the cost efficiency of local inference for the high-volume base with the capability ceiling of frontier models for the tasks that genuinely require them. In practice, for most enterprise workloads, 60–70% of API spend concentrates in the high-volume bounded-complexity tier — exactly the tier where local SLMs are most competitive.
Data sovereignty is an additional non-financial justification for local deployment that applies in regulated industries. Prompts, context, and outputs that never leave your infrastructure cannot appear in a provider’s training data, cannot be subject to a provider’s data retention policy change, and cannot be disclosed in a provider security incident. For healthcare, legal, and financial services workloads, this consideration may justify local inference even when the pure cost calculus is marginal.
Next in the series: From Tokenmaxxing to Economic Governance: The 2026 AI Roadmap for CTOs Who Want a 2027 Budget — synthesising the full series into a governance framework, a prioritised action list, and the consulting CTA for teams that want the guardrails Uber missed.
]]>Gartner’s AI spending forecast projects global AI-related spend reaching $2.52 trillion by 2026, with approximately 54% — roughly $1.37 trillion — allocated to infrastructure: data centre construction, GPU and TPU hardware procurement, networking, and the power and cooling systems required to operate them.
This is an extraordinary figure. For comparison, global cloud infrastructure spending (excluding AI-specific builds) was approximately $270 billion in 2023. The AI infrastructure build-out represents a 5x acceleration over the baseline cloud expansion rate, compressed into a 24-month window.
The composition of that spend:
| Category | Estimated Share | 2026 Spend Estimate |
|---|---|---|
| GPU / accelerator hardware | 38% | $521B |
| Data centre construction | 22% | $301B |
| Power & cooling systems | 18% | $247B |
| Networking & interconnect | 12% | $164B |
| Storage infrastructure | 10% | $137B |
Hardware dominates, but the power and cooling line — $247 billion — is the constraint that does not scale with capital. You can order more H100s. You cannot order more megawatts from a grid that is already at capacity.
A single NVIDIA H100 GPU, the dominant training accelerator in 2025–2026, has a thermal design power (TDP) of 700 watts. A standard AI training cluster uses 8 H100s per server node. A modest training run for a frontier model might use 1,024 such nodes:
\[P_{cluster} = 1{,}024 \text{ nodes} \times 8 \text{ GPUs/node} \times 700\text{W} = 5.73\text{ MW}\]That is 5.73 megawatts for the GPUs alone — before accounting for the additional 30–40% overhead of CPU, networking, storage, and crucially, cooling. A data centre’s Power Usage Effectiveness (PUE) ratio — the ratio of total facility power to IT equipment power — typically ranges from 1.2 to 1.5 for modern hyperscale facilities. A PUE of 1.35 means the full facility draw for that cluster is:
\[P_{facility} = 5.73\text{ MW} \times 1.35 = 7.74\text{ MW}\]A frontier model training run lasting 90 days at this cluster size consumes:
\[E_{training} = 7.74\text{ MW} \times 90 \times 24\text{ h} = 16{,}718\text{ MWh}\]That is 16.7 gigawatt-hours for a single training run — equivalent to the annual electricity consumption of approximately 1,500 average US households.
Inference at scale is a separate problem. Unlike training, which occurs once per model version, inference runs continuously, serving every user query, every API call, every agentic loop in production. The world’s frontier AI providers are now operating thousands of nodes in continuous inference mode, 24 hours a day, 365 days a year. The inference power demand is arguably more consequential than training, because it does not stop.
The data centre industry’s power demand has grown to the point where grid capacity is the primary constraint on hyperscaler expansion in several major markets. Virginia’s Northern Virginia data centre corridor — historically the world’s largest concentration of data centre capacity — is facing utility interconnection queues measured in years, not months. Power delivery for new facilities signed in 2024 is not expected until 2027 or later in several jurisdictions.
Microsoft, Google, Amazon, and Meta have collectively committed to construction projects representing over 40 gigawatts of new data centre capacity globally through 2030. The announced capacity significantly exceeds current grid availability in the target markets, driving investment into:
Each of these strategies has a cost premium relative to standard grid electricity. That premium is embedded in the price you pay per token.
The relationship between infrastructure investment and API pricing is not immediate — providers absorb infrastructure costs over multi-year depreciation schedules and use scale economics to suppress per-unit costs. But the directional pressure is clear.
As grid constraints extend build timelines, as hardware procurement costs remain elevated due to accelerator supply chain concentration (TSMC’s advanced node capacity is the ultimate upstream constraint), and as cooling overhead increases with thermal density, the floor price for inference — the minimum cost at which a provider can run a frontier model without operating at a loss — rises.
The providers with the most efficient infrastructure (lowest PUE, best hardware utilisation, most aggressive custom silicon investment) will hold the pricing advantage. This is why Google’s custom TPU v5 fleet and its investment in 24/7 carbon-free energy procurement are not just sustainability initiatives — they are cost moats.
| Provider | Primary Accelerator | Custom Silicon | Noted Power Strategy |
|---|---|---|---|
| TPU v5 / v6 | Yes (in-house) | 24/7 CFE matching, on-site generation | |
| Microsoft / OpenAI | NVIDIA H100/H200 + Maia | Partial (Maia) | Nuclear PPA (TMI restart) |
| Amazon / AWS | Trainium 2 / Inferentia | Yes (in-house) | Renewable PPAs, on-site generation |
| Anthropic | NVIDIA H100/H200 | No | AWS-hosted, inherits AWS power strategy |
| Meta | NVIDIA H100 + MTIA | Partial (MTIA) | On-site solar, long-term wind PPAs |
For engineering leaders and CTOs, the infrastructure bottleneck has two strategic implications that extend beyond watching API prices.
First, provider concentration risk is amplified. If your entire AI stack runs through a single API provider, you are exposed not just to that provider’s pricing decisions but to their infrastructure constraints. A provider that cannot expand data centre capacity in your required region may respond with higher prices, lower rate limits, or degraded performance during peak demand. Diversification across providers is not just a pricing hedge — it is an infrastructure risk hedge.
Second, the economics of on-premises inference improve relative to API. When API prices embed a premium for constrained infrastructure, the total cost of ownership for dedicated inference hardware — leased or owned — becomes more competitive. A team running 10 million daily inferences on a dedicated A100 cluster in a co-location facility with reliable power access may, under certain workload profiles, achieve lower per-inference costs than the equivalent API spend. Post 7 in this series examines this calculation in detail through the lens of Small Language Models and quantised inference.
The organisations that model infrastructure risk as a first-class budget variable — not just today’s API price, but the trajectory of that price given the constraints upstream — will be better positioned to make rational make-vs-buy decisions for AI infrastructure over the next 24 months.
Next in the series: The Local Inference ROI: 4-Bit Quantization, SLMs, and the Case for Bypassing the API — the real numbers behind running Phi-3, Mistral 7B, and Llama-3 on your own hardware, and when the economics of local inference decisively outperform the API.
]]>When a transformer processes an input sequence, it computes key-value (KV) pairs for each token in the attention layers. These KV pairs are the intermediate representation that enables the model to contextualise each token against all preceding tokens. Computing them is the expensive part of processing input — it is the work that makes long-context models computationally intensive and expensive to run.
Server-side prompt caching stores these pre-computed KV pairs on the inference provider’s servers, keyed to a specific prefix of your input prompt. When a subsequent request begins with that same prefix, the provider skips the KV computation for the cached portion and begins processing only from the cache boundary forward.
The cost structure shifts accordingly:
\[C_{cached} = T_{cached} \cdot P_{cache\_read} + T_{uncached} \cdot P_{in} + T_{out} \cdot P_{out}\]Where \(P_{cache\_read}\) is the discounted rate for reading from cache, and \(T_{cached}\) is the portion of input tokens that hit the cache. For Anthropic’s implementation:
| Token Type | Price (per 1M tokens) |
|---|---|
| Standard input | $3.00 |
| Cache write (first call) | $3.75 |
| Cache read (subsequent calls) | $0.30 |
| Output | $15.00 |
The cache read price is 90% cheaper than the standard input price. On the first call, you pay a 25% premium to write to cache. On every subsequent call that hits the cache, you pay 10 cents on the dollar. The break-even point is two calls: if a cached prefix is read at least twice, you recover the write premium and begin generating savings.
S3 changed infrastructure economics by separating the cost of storing data from the cost of generating it. You generate data once (expensively), store it cheaply, and serve it repeatedly at marginal cost. The pattern is so fundamental that it now underlies every CDN, every database read replica, and every API response cache in modern infrastructure.
KV cache in AI inference follows the identical economic logic. You compute the KV representation of your prompt once (at the write price), store it in the provider’s inference cluster, and retrieve it repeatedly at marginal cost (the read price). The “content” being cached is not bytes in a file — it is the model’s internal representation of your text. But the economic structure is identical.
The analogy extends to management principles:
| S3 Concept | KV Cache Equivalent |
|---|---|
| Object key | Cache prefix hash |
| TTL / expiration policy | Cache invalidation on prompt change |
| Cache-Control headers | Prompt structure discipline |
| CDN edge caching | Provider-side KV storage |
| Cache hit rate | Cost reduction multiplier |
| Cache miss cost | Full input token cost |
| Write-through vs write-back | Single vs batched cache priming |
Just as S3 economics improve with higher cache hit rates, KV cache economics improve with higher prefix stability. The engineering challenge in both cases is the same: design your data structures (prompts or files) to maximise the portion that is stable and reusable.
The critical insight for maximising KV cache ROI is that cache hits are a function of prompt architecture, not just prompt length. A provider’s caching implementation stores and retrieves KV pairs based on a prefix match — the cached prefix must be bit-for-bit identical to the beginning of the new request. Any modification to the prefix, however small, invalidates the cache entry.
This means prompt engineering and cost engineering are the same activity. Practices that maximise cache hit rates:
Place stable content at the beginning. System prompts, role definitions, static knowledge bases, and fixed instructions should come first in your prompt structure. Dynamic content — user inputs, session-specific context, real-time data — should come last. The cache boundary falls immediately before the first dynamic element.
Separate system prompts from user prompts architecturally. Many frameworks co-mingle system instructions with per-request context. Restructuring to maintain a clean boundary between the static system layer and the dynamic user layer is the single most impactful structural change for cache performance.
Avoid timestamp and UUID injection in system prompts. A common anti-pattern is including a current timestamp or session ID in the system prompt for logging purposes. This guarantees a cache miss on every request because the prefix changes every second. Move dynamic identifiers to user-turn messages or metadata fields instead.
Version and manage your system prompts explicitly. Treat system prompts as code. Store them in version control. Use feature flags to roll them out. Avoid ad-hoc modifications that create new cache entries and orphan old ones, consuming cache storage without delivering hits.
Consider an enterprise deployment of a customer service AI with the following characteristics:
Without caching: \(C_{daily} = 100{,}000 \times \left( \frac{4{,}650}{1{,}000{,}000} \times 3.00 + \frac{400}{1{,}000{,}000} \times 15.00 \right)\) \(= 100{,}000 \times (0.01395 + 0.006) = 100{,}000 \times 0.01995 = \$1{,}995/\text{day}\)
With caching (4,500-token system prompt cached; 150-token user query uncached): \(C_{daily} = 100{,}000 \times \left( \frac{4{,}500}{1{,}000{,}000} \times 0.30 + \frac{150}{1{,}000{,}000} \times 3.00 + \frac{400}{1{,}000{,}000} \times 15.00 \right)\) \(= 100{,}000 \times (0.00135 + 0.00045 + 0.006) = 100{,}000 \times 0.00780 = \$780/\text{day}\)
Daily savings: $1,215. Annual savings: approximately $443,000 — for a single prompt caching configuration change, with zero change to model quality or user experience.
The ROI is immediate and compounding. At 100,000 calls per day, the cache write cost on day one (paying $3.75/1M instead of $3.00/1M for the 4,500 cached tokens) is fully recovered within the first 45 minutes of operation.
The single-system-prompt case is the simplest illustration, but the pattern scales to more complex architectures. A RAG system with a tiered knowledge base can implement layered caching:
This architecture maximises cache hit rate for the most expensive stable tokens while preserving the flexibility to inject dynamic context at the tail of each prompt. The engineering investment is primarily in prompt structure discipline — the infrastructure change is configuration, not code.
Prompt caching is not an advanced optimisation for teams with sophisticated ML platforms. It is table-stakes cost management for any production AI deployment running more than a few hundred daily calls. The AI software development costs that continue to escalate in 2026 are disproportionately concentrated in organisations that have not yet adopted this practice.
The governance requirement is simple: audit every production prompt for its stable and dynamic components. Measure the token split. Calculate the savings from caching the stable prefix. Implement the structural change. The investment is measured in engineering hours; the return is measured in budget lines.
The teams that treat their KV cache hit rate as a production metric — alongside latency, error rate, and throughput — will find that AI infrastructure begins to behave like cloud infrastructure: optimisable, predictable, and defensible to finance.
Next in the series: Power as the New Token: Gartner’s $1.37 Trillion Infrastructure Bet and the Physics of AI at Scale — why the constraint on AI growth in 2026 is not model capability or API pricing, but electricity and the data centre supply chain.
]]>The Gemini Live API is designed for continuous, real-time, multimodal interactions: voice conversations, live video analysis, streaming audio transcription paired with language model response. In these workloads, the concept of a discrete “token” begins to break down as a billing unit.
Audio inference presents a specific challenge. A one-minute audio clip, when tokenised for an audio-capable model, produces a token count that varies with the audio’s information density — silence, background noise, speech rate, and speaker count all affect tokenisation. The variance in token counts for semantically equivalent audio makes per-token pricing difficult to plan against. A user who speaks slowly on a quiet line might generate 800 tokens per minute; a rapid speaker in a noisy environment might generate 3,000 tokens per minute for a conversation of equivalent informational value.
Per-minute pricing eliminates this variance. The billing unit is time, which is constant regardless of audio characteristics. From Google’s perspective, it also aligns billing with the most predictable dimension of server-side resource consumption: wall-clock inference time.
As of 2026, the Gemini family pricing for real-time and standard workloads:
| Model / Mode | Pricing Unit | Rate |
|---|---|---|
| Gemini 2.0 Flash Live (audio) | Per minute | $0.005 |
| Gemini 2.0 Flash Live (video) | Per minute | $0.005 |
| Gemini 2.0 Flash (text/image input) | Per 1M tokens | $0.075 |
| Gemini 2.0 Flash (text output) | Per 1M tokens | $0.30 |
| Gemini 1.5 Pro (input) | Per 1M tokens | $1.25 |
| Gemini 1.5 Pro (output) | Per 1M tokens | $5.00 |
| Gemini 2.0 Pro (input) | Per 1M tokens | $1.25 |
| Gemini 2.0 Pro (output) | Per 1M tokens | $10.00 |
The per-minute rate for Live API is notably aggressive. At $0.005/minute, a 10-minute voice interaction costs $0.05 — five cents. A 1,000-call daily volume of 10-minute interactions costs $50 per day, or approximately $18,000 per year.
The economic comparison between per-minute and per-token pricing depends on the token density of your specific workload. The crossover point can be derived from the token cost formula:
\[C_{token} = T_{in} \cdot P_{in} + T_{out} \cdot P_{out}\]For a per-minute model, cost is simply:
\[C_{minute} = t \cdot P_{min}\]Where \(t\) is the duration in minutes and \(P_{min}\) is the per-minute rate. The per-minute model is more economical when:
\[t \cdot P_{min} < T_{in} \cdot P_{in} + T_{out} \cdot P_{out}\]For a Gemini Flash text comparison: at $0.075/1M input and $0.30/1M output, a one-minute interaction that generates 2,500 input tokens and 400 output tokens costs:
\[C_{token} = \frac{2{,}500}{1{,}000{,}000} \times 0.075 + \frac{400}{1{,}000{,}000} \times 0.30 = 0.000188 + 0.00012 = \$0.000308\]The per-minute rate for the same one-minute interaction: \(\$0.005\). The token rate is 16x cheaper for a text-only workload of this density.
However, the economics reverse for multimodal workloads. A one-minute audio clip at 150 words per minute, transcribed and processed with a system prompt and conversation history, can generate 20,000–40,000 input tokens depending on the audio model’s tokenisation. At Gemini Pro rates ($1.25/1M input):
\[C_{token} = \frac{30{,}000}{1{,}000{,}000} \times 1.25 = \$0.0375\]The per-minute rate ($0.005) is now 7.5x cheaper. This is the workload class for which per-minute pricing was designed, and it represents a genuine economic advantage for real-time voice and video applications.
The practical challenge for engineering leaders is that most real-world AI applications are not purely text or purely audio — they are hybrid workflows in which the optimal billing model differs by feature. A customer service platform might use:
An architecture that routes each workload component to the appropriate billing model and provider can reduce total inference costs by 40–60% compared to a uniform model selection. This is multi-provider inference routing, and it is becoming a standard capability in mature AI platform architectures.
The routing logic is straightforward in principle but requires empirical calibration:
Google’s per-minute pricing is not just a billing convenience — it is a market positioning decision. By making Gemini Live API economical for high-volume real-time applications, Google is targeting the workload category where it has the strongest infrastructure advantage: streaming inference at scale, backed by TPU v5 hardware optimised for throughput rather than latency.
The competitive pressure on Anthropic and OpenAI is real. Neither currently offers a per-minute pricing tier for real-time audio. Both are priced on token-based models that make extended voice interactions disproportionately expensive relative to Google’s offering. If per-minute pricing proves to be the preferred billing model for the voice assistant category — which represents one of the highest-volume AI deployment patterns in consumer and enterprise markets — the pricing structure advantage could translate into significant market share over a 12–18 month horizon.
For operators, this creates a short-term opportunity to capture genuine cost savings by adopting Gemini Live for qualifying workloads, while maintaining existing Anthropic or OpenAI integrations for text-heavy tasks where the token economics remain superior.
Per-minute billing requires different monitoring primitives than per-token billing. The relevant metrics shift from:
An organisation accustomed to token-level spend instrumentation will need to extend its observability stack to capture session duration metrics for Live API workloads. The risk profile also changes: rather than a single call that unexpectedly balloons to 200K tokens, the equivalent risk in per-minute billing is a session that runs for an unexpected duration — a user who leaves an audio session open, an agent that fails to terminate a real-time task, or a voice interface that enters a conversational loop.
Duration limits, idle session timeouts, and session cost caps are the per-minute equivalents of context window limits and token budgets. They require explicit implementation. The meter runs whether or not the session is producing value.
Next in the series: KV Cache Optimization: Why Server-Side Cache Is the New S3 of AI Infrastructure — the technical mechanics of prompt caching, why it is the single highest-ROI optimisation available to most teams today, and how to architect your prompts to maximise cache hit rates.
]]>A standard LLM completion is a bounded transaction: a prompt goes in, a response comes out, and the API call closes. Cost is deterministic at the time of the request, given the context window size.
An agentic loop is structurally different. It is a control flow in which the model’s output determines the next action, which in turn generates new input, which feeds another model call. The loop terminates when the model decides it has completed its objective — or when an external circuit breaker intervenes. If neither condition is met cleanly, the loop continues.
The cost of a single agent task is therefore not the cost of one API call. It is the sum of all API calls across the entire task execution:
\[C_{agent} = \sum_{k=1}^{K} \left( T_{in}^{(k)} \cdot P_{in} + T_{out}^{(k)} \cdot P_{out} \right)\]Where \(K\) is the number of steps the agent takes to complete (or fail to complete) the task. In a well-designed agent, \(K\) is bounded. In a poorly designed one, \(K\) is determined by the model’s own assessment of task completion — which can be confused, misdirected, or pathologically optimistic.
Consider an agent tasked with “audit the security posture of this repository and generate a remediation plan.” The intended execution path might be:
In practice, an under-specified agent with access to search, read, and web-browsing tools may:
This is not a theoretical failure mode. It is a documented pattern in every agentic framework that allows models to self-direct their tool use without explicit step budgets. The model is behaving rationally from its own perspective — gathering more information to produce a more comprehensive output. The operator never specified a limit, so the model imposed none.
Anthropic’s 2026 shift to metered agent billing across its subscription tiers reflects a recognition that flat-rate pricing for agentic workloads is structurally unsustainable — for both the operator running the agent and for Anthropic managing server capacity.
Under the new model, agent runs within Claude’s built-in tools (computer use, web search, file operations) are billed against a token meter that tracks both input and output across the entire agent session. Subscription tiers receive a monthly token allocation; overages are billed at standard API rates. The architecture creates a natural pressure toward efficient agent design: organisations that burn their allocation on runaway loops face real financial consequences, not just degraded performance.
The implications for teams building on the Claude API rather than the consumer product are equally significant. The metered model validates a set of engineering practices that had previously been regarded as optional:
| Practice | Pre-Metering Status | Post-Metering Status |
|---|---|---|
| Step count limits per agent run | Optional best practice | Financial necessity |
| Token budget per task type | Rarely implemented | Standard requirement |
| Agent run cost attribution | Difficult to instrument | Critical for billing accuracy |
| Circuit breakers on tool calls | Advanced implementations only | Baseline engineering requirement |
| Task scope specification in prompts | Improved output quality | Also cost control |
Agentic cost overruns concentrate around three patterns:
1. Scope creep without bounds. The agent is given access to tools that allow it to expand its own task definition. Web search is the most common vector — a task that starts as “summarise this document” becomes “research all claims in this document” becomes “verify every cited source” becomes a multi-hour research engagement. Mitigation: define tool access narrowly, and include explicit scope boundaries in system prompts (“do not follow external links; work only from the provided document”).
2. Retry loops on tool failure. When a tool call fails — a web page returns a 404, an API is rate-limited, a file cannot be parsed — a poorly calibrated agent will retry. Without a maximum retry count, transient failures become infinite loops. Mitigation: implement explicit retry budgets (maximum 3 retries per tool call) and instruct the model to report failures rather than loop on them.
3. Verification spirals. Some models, when instructed to be thorough, enter a pattern of self-verification — generating a result, evaluating it, finding it inadequate, regenerating, re-evaluating. Each iteration is a full generation cycle. Mitigation: separate generation from evaluation into distinct agent steps with independent token budgets, and cap the number of evaluation passes.
The engineering pattern that prevents runaway loops is the explicit budget constraint, embedded at three levels:
Prompt-level constraints. The system prompt specifies maximum steps, maximum tool calls, and acceptable scope boundaries. Example: “Complete this task in no more than 10 steps. If you cannot complete it within 10 steps, return a partial result with an explanation of what remains.”
Framework-level guardrails. Agent orchestration frameworks (LangGraph, AutoGen, CrewAI) support maximum iteration counts and token budgets as first-class configuration. These should always be set explicitly — the default in most frameworks is unbounded.
Infrastructure-level circuit breakers. API gateway or proxy layers can enforce hard token limits per agent session, killing runs that exceed the budget regardless of model state. This is the safety net when prompt-level and framework-level constraints fail.
The cost model for a well-governed agent run is:
\[C_{bounded} = \min\left( C_{agent},\ K_{max} \cdot \bar{C}_{step} \right)\]Where \(K_{max}\) is the maximum permitted step count and \(\bar{C}_{step}\) is the average cost per agent step — a number that can be calibrated empirically during development and used to set \(K_{max}\) against a per-task budget constraint.
Anthropic’s move to metered agent billing is not an isolated vendor decision. It reflects a broader industry direction. As agents become the primary interface through which organisations interact with AI — replacing both chat UIs and batch pipelines — the cost per unit of value delivered will increasingly be measured in agent steps rather than tokens.
This changes the economic calculus for every AI architecture decision. The question is no longer only “which model produces the best output?” It becomes “which model produces sufficient output at the lowest step count?” A smaller, faster model that completes a task in four steps is economically superior to a more capable model that reaches the same outcome in twelve — even if the per-token price of the smaller model is higher.
The organisations that tune their agents for step efficiency now will have a durable cost advantage as the agentic paradigm matures. Those that optimise only for output quality, without constraining step count, will find that the meter runs whether or not the agent produces useful work.
Next in the series: Beyond the Token: Google’s Per-Minute Pricing and What It Means for the Economics of Real-Time AI — how Gemini Live API’s $0.005/minute rate is disrupting the token-per-call pricing model, and when it makes economic sense to pay for time rather than tokens.
]]>