The Web Is Rebuilding Itself for AI Agents, and Your Organisation Needs to Pay Attention

Paul Bratcher, Prosper AI Consulting | June 2026

The web was built for humans clicking links. Across the first half of 2026, the biggest infrastructure and AI companies on the planet (Cloudflare, Shopify, Google, Anthropic, Microsoft, and OpenAI) have been quietly rebuilding it for AI agents. Not as an experiment. Not as a beta feature buried in developer documentation. As production infrastructure, available now, reshaping how content is delivered, commerce is conducted, and information is discovered.

I first connected these threads in the spring, when five developments landed in a six-week window. Six months on, none of them has stalled. Every one has shipped further, and a sixth shift has joined them: the agents themselves have moved in. They now have a clock and a key: they run on schedules, in isolated cloud environments, with their own credentials, driving real browsers without a human at the keyboard.

Together these developments tell a story every business leader needs to understand: the web now has two front doors, most organisations have only built one of them, and the visitors coming through the second door no longer wait to be invited.

The Language Is Changing: From SEO to AEO to GEO

For two decades, digital strategy meant one thing: SEO. Get your pages ranking. Drive clicks. Measure traffic.

That vocabulary is expanding. Two newer terms have settled into the conversation through 2026:

AEO (Answer Engine Optimisation): optimising content so AI answer engines (ChatGPT, Perplexity, Google AI Overviews) surface it directly in their responses
GEO (Generative Engine Optimisation): the broader discipline of ensuring your content is cited, referenced, and used by AI systems across platforms

The data behind this shift is striking. AI Overviews now reduce traditional clicks by 58% (Ahrefs, February 2026). But here is the counterpoint that matters: AI-driven sessions convert at 14.2% compared to 2.8% for traditional organic search (Conductor, Q1 2026). Fewer clicks, but dramatically better ones.

This creates what the industry is calling the "Crocodile Mouth Effect": impressions are up 31%, but organic clicks are down 18% year-over-year. The mouth is opening. If you are still measuring success by clicks alone, the picture looks dire. If you measure by citations and conversions, the picture looks different.

The practical implication: success in 2026 is measured by whether AI systems cite your content, not whether humans click on it. SEO is not dead, but it is no longer sufficient on its own.

What Google said at I/O: in May 2026, Google used its I/O conference to make what it called the biggest Search overhaul in 25 years, and to plant a flag in the vocabulary debate. Alongside the news that AI Mode had passed one billion monthly users a year after launch, Google published its first official AI search optimisation guide and took a deliberately deflationary line: AEO and GEO are "still SEO". Its argument is that there is no separate discipline to buy into. Strong foundational SEO, plus content only your team can produce, is what earns visibility across every AI surface. The guide names llms.txt and "chunking" directly, and pours cold water on both, stating there is no need to break content into small pieces because its systems already understand multiple topics on a page. For business leaders being pitched expensive "AEO services", that is a useful piece of cover: the fundamentals still matter most, and the acronyms are mostly new packaging around them. The caveat is that this is Google describing its own surfaces; it says nothing about how ChatGPT, Perplexity, or Claude choose what to cite, and those engines do not share Google's playbook.

Cloudflare: Serving Two Versions of the Web

On 12 February 2026, Cloudflare launched Markdown for Agents. The concept is elegant. When an AI agent requests a web page, it sends an Accept: text/markdown header. Cloudflare intercepts the request at the CDN layer and converts the HTML to clean markdown before returning it. Human visitors see normal HTML. AI agents get content in their native format.

The economics are significant. A blog post that requires 16,180 tokens in HTML shrinks to approximately 3,150 tokens in markdown, a reduction of over 80%. When AI agents are processing thousands of pages to answer a query, that difference translates directly into speed, cost, and quality. The converted response even carries an x-markdown-tokens header so the agent knows the context cost before it reads.

But the more important feature is Content Signals. Alongside Markdown for Agents, Cloudflare introduced a mechanism that lets you declare how AI systems may use your content. Through the Content-Signal HTTP header, you can signal whether your content is available for AI training, search indexing, agentic use, or any combination. This is not robots.txt, which blocks crawlers. This is content negotiation, serving the same content in different ways to different consumers, with explicit publisher control over what AI can do with it.

The feature shipped in beta at no additional cost to Pro and Business plans. Claude Code, OpenCode, and several other AI development tools already send the Accept: text/markdown header when fetching web content.

The debate: Google's John Mueller called converting pages to markdown "such a stupid idea" in January 2026. The counter-argument from Cloudflare and supporters: this is standard HTTP content negotiation, the same mechanism that has served images in different formats for years. It delivers the same content in a different format. It is not cloaking. Both perspectives have merit, and the debate is far from settled.

What has happened since: Cloudflare did not stop at content delivery. At its Agents Week in June, it shipped an entire runtime stack for agents: a single durable browser_execute tool that lets an agent drive a real Chrome session, inspect pages, and capture screenshots, plus Temporary Accounts (announced 19 June) that let a coding agent deploy a live service with one command and a 60-minute claim window, no OAuth or API token required. It also added AI-agent verification as a first-class signal in Bot Management on 2 June, launching with 19 verified agents including Claude in Chrome, ChatGPT Atlas, Perplexity Browser, and Gemini Agent Mode. The same company building the agent web is now also building the controls to identify and govern the agents arriving through it.

llms.txt: A Curated Front Door for AI

While Cloudflare handles content delivery, a complementary standard addresses content discovery. Jeremy Howard's llms.txt specification provides a markdown-formatted map of a site's most important resources, placed at the root directory, like robots.txt, but designed to guide AI rather than block it.

The format is simple: a markdown file with a title, a summary blockquote, and structured links to the pages the site owner considers most important. Where robots.txt says "do not go here", llms.txt says "start here, and here is what matters."

Adoption has grown. Anthropic, Cloudflare, Docker, and HubSpot publish llms.txt files, and by spring 2026 the convention had moved from niche proposal to near-mainstream, with partial support across every major Western AI platform.

The honest assessment, now with evidence: when I first wrote about llms.txt the impact was simply unproven. The evidence has since come in, and it is sobering. An Ahrefs study of 137,000 sites found that 97% of llms.txt files are never read, and the file appeared in under 1% of cited sources. The major answer engines (Google, OpenAI, Anthropic, Meta) do not yet fetch it for citation. So if your goal is to be quoted in AI answers, llms.txt is not the lever.

There is one real and growing exception: developer tooling. IDE agents such as Cursor, Windsurf, Claude Code, and GitHub Copilot routinely fetch /llms.txt when pointed at a documentation site. If your audience includes developers consuming your docs through an AI assistant, the file earns its keep. For everyone else, it remains a near-zero-cost, low-priority hedge rather than the citation win some vendors implied. Google's own AI search guide, published at I/O in May, addresses llms.txt directly and declines to lean on it. Implement it if you can do it in an afternoon; do not expect it to move your numbers on its own.

Shopify and Google: AI Agents That Buy Things

On 11 January 2026 at the National Retail Federation conference, Google and Shopify announced the Universal Commerce Protocol (UCP), an open standard that defines how AI agents discover products, negotiate terms, and complete purchases across merchants.

The ambition was significant from the start. UCP is not a Shopify feature or a Google product. It is an open protocol, publicly available at ucp.dev, co-developed with Etsy, Wayfair, Target, and Walmart, and endorsed by more than 20 partners including Visa, Mastercard, Stripe, American Express, Best Buy, and The Home Depot. The architecture is layered (shopping service, capabilities, and extensions), each independently versioned, deliberately reminiscent of how TCP/IP structured the internet itself.

The practical mechanism is a machine-readable manifest, a "passport" for your store that broadcasts its capabilities to AI agents. What products are available. What payment methods are accepted. What checkout flows are supported. An AI agent reading this manifest can discover, compare, and purchase products without ever rendering a web page.

From ambition to rollout: the half-year since launch is where UCP stopped looking speculative. On 19 March Google added Cart, Catalog, and Identity Linking capabilities plus a simpler onboarding path through Merchant Center. On 24 April the UCP Tech Council expanded to ten members as Amazon, Meta, Microsoft, Salesforce, and Stripe joined Google, Shopify, Etsy, Target, and Wayfair, a remarkable alignment of rivals around a single standard. At Google Marketing Live on 20 May, Google confirmed UCP-powered checkout going live with major US retailers, with Canada and Australia following in the coming months and the UK after that. Most non-Shopify merchants will inherit UCP support through Shopify's Agentic Storefronts feature, which manages the integration centrally.

The reality check still holds: only 17% of consumers currently say they are comfortable letting AI complete a purchase on their behalf (ChannelEngine, 2026). The infrastructure is being built for full adoption while consumer trust catches up. That gap is a timing opportunity, not a reason to wait. The merchants who are machine-readable when comfort arrives will have a structural advantage over those who start building then. With UCP checkout now reaching the UK, that window is measured in months, not years.

Google AI Mode: The End of Ten Blue Links

Google's AI Mode, powered by Gemini, has rolled out globally and, as of I/O in May 2026, passed one billion monthly users a year after launch. It replaces the traditional search results page with an AI-synthesised answer that includes citations to source material.

The distinction from AI Overviews matters. AI Overviews appear alongside traditional search results; you might lose the featured snippet, but your organic listing still exists. AI Mode has no organic results alongside. You are cited in the AI answer, or you are invisible.

Research from Ahrefs found only 13.7% citation overlap between AI Mode and AI Overviews. They cite different sources. This means optimising for one does not automatically cover the other.

Some sites report losing 20 to 60% of organic traffic. The traffic that remains behaves differently: when AI Mode does send a visitor, that visitor has been pre-qualified by the AI. They know what they are looking for. They convert better.

Google reports that AI Mode queries are two to three times longer than traditional searches. Users are asking complex, multi-part questions that would have required multiple searches before. The AI synthesises across sources. If your content is authoritative, well-structured, and citable, you benefit. If it is thin, duplicative, or poorly structured, you disappear.

The New Layer: The Agents Now Have a Clock and a Home

Every development above assumes an agent shows up, reads or buys something, and leaves. The biggest shift of the year is that agents have stopped being things you summon in a chat window and become things that run on their own: on a schedule, in a persistent environment, with stored credentials and a real browser. Three of the largest AI providers shipped this within weeks of each other.

Anthropic put Claude Managed Agents on autopilot. On 9 June it moved scheduled agents into public beta: an agent attached to a cron schedule that wakes up, starts a fresh session, completes its task, and can be paused, resumed, or triggered on demand. Paired with it are credential vaults: register an API key against an environment variable and a list of allowed domains, and command-line tools inside the agent's sandbox can make authenticated calls without a human present. The scheduler, the sandbox, and the secret store are all part of the managed platform. Anthropic also previewed "dreaming", a scheduled process that reviews an agent's past sessions and curates its memory so it improves over time.

Microsoft built the same capability into the enterprise stack. Copilot Studio now supports scheduled prompts that run daily, weekly, or monthly and deliver their output proactively, a Monday-morning pipeline check or an end-of-month digest on autopilot. At Build 2026 Microsoft went further with Autopilots, long-running autonomous agents led by one called Scout, and made Agent 365 generally available as a control plane for managing every agent's inventory, permissions, and activity in one place. Computer-using agents that drive websites and desktop apps through the interface are now generally available too.

OpenAI rounded out the picture. Scheduled Tasks in ChatGPT let anyone set a prompt to run once or on a recurring basis, managed from a dedicated schedules page. Its Workspace Agents can be schedule-triggered or fired through an API from any internal system, and the April 2026 Agents SDK added native sandbox execution so an agent gets a real workspace to read and write files, install dependencies, and run code, using built-in support for E2B, Modal, Daytona, Cloudflare, Vercel, and others.

Behind all of this sits a fast-growing market of sandbox and browser-infrastructure startups giving agents somewhere safe to run. General-purpose code sandboxes like E2B (Firecracker microVMs) and Daytona (sub-90ms startup, a $24M Series A in February) handle execution. For agents that need to drive a browser, purpose-built platforms have emerged: Browserbase and Kernel (kernel.sh), backed by a $22M Series A from Accel, whose unikernel architecture spins up a browser in under 325 milliseconds against the three to five seconds typical of container-based rivals. Cloudflare, Vercel, Ramp, and Modal have all shipped sandbox features. An entire infrastructure layer now exists for the sole purpose of giving autonomous agents a computer of their own.

Why a business leader should care. The optimisation question, "can an agent read and buy from us?", was about being found. This new layer raises a sharper governance question: agents are now acting on your behalf, repeatedly, unattended, holding credentials, at three in the morning. Who owns the schedule. What can the credential vault reach. What happens when a scheduled agent runs against stale instructions. The same shift that makes agents genuinely useful also makes them a new category of standing access to manage. The organisations that get value here will be the ones that treat a scheduled agent like a member of staff with a login, not like a feature they switched on.

The Pattern: A Parallel Web for Agents

These developments share a common thread. The web is building dedicated infrastructure for AI agents, not as an afterthought, but as a parallel system, and that system now has its own runtime as well as its own content and commerce layers.

Layer	Human Web	Agent Web
Content delivery	HTML via browser	Markdown via `Accept: text/markdown` (Cloudflare)
Site map	`sitemap.xml`	`llms.txt`
Commerce	Shopping cart + checkout page	UCP manifest + agent negotiation (Shopify/Google)
Search	Ten blue links	AI-synthesised answer with citations (Google AI Mode)
Runtime	A person at a keyboard, on demand	Scheduled agents in sandboxes, on a cron, with credential vaults (Anthropic/Microsoft/OpenAI)
Success metric	Clicks, rankings	Citations, AI visibility
Optimisation discipline	SEO	AEO / GEO

Taken together, these are a single architectural shift, happening at once across content delivery, discovery, commerce, search, and now execution. The infrastructure companies are not waiting for standards bodies or industry consensus. They are building the agent web now.

What This Means for Your Organisation

The response depends on what you do.

If You Own a Website

Your content now serves two audiences. For AI agents:

Enable Cloudflare Markdown for Agents if you use Cloudflare. Free, opt-in, immediate token savings for agents consuming your content.
Review your Content Signals: decide explicitly how AI may use your content rather than leaving it to default.
Structure your content for citation: clear headings, factual claims with evidence, authoritative sourcing. This is what AI systems cite, and it matters far more than any single file.
Publish an llms.txt file only if it is cheap: useful if developers consume your docs through AI tools, but the evidence says it will not lift your citation numbers on its own. Do not over-invest.
Measure AI visibility alongside traditional SEO metrics. Tools are emerging. Start tracking now.

If You Sell Online

The "ask and buy" channel is growing, and UCP checkout is now reaching the UK.

Understand UCP and evaluate whether your commerce platform supports it. Shopify merchants have a head start through Agentic Storefronts.
Make your product data machine-readable: structured data, clear specifications, competitive pricing in parseable formats.
Monitor AI-driven commerce metrics as a distinct channel, not a footnote to organic search.
Prepare for the consumer trust inflection: when the 17% becomes 50%, the merchants who are already machine-readable will capture disproportionate share.

If You Deploy AI Agents

The infrastructure your agents interact with, and run on, is changing fast.

Send Accept: text/markdown headers when fetching web content. The token savings are immediate and significant.
Use scheduled and managed agents deliberately: Anthropic, Microsoft, and OpenAI now offer crons, sandboxes, and credential vaults out of the box. Start with low-risk, repeatable tasks.
Govern standing agent access like staff access: inventory your scheduled agents, scope what each credential can reach, and review them on a cadence. Microsoft's Agent 365 and Anthropic's vaults exist because this is now a real control problem.
Choose your runtime substrate: if agents need to execute code or drive browsers, the sandbox market (E2B, Daytona, Modal, Browserbase, Kernel) is mature enough to buy rather than build.

The Window

The web did not ask permission to split in two. The infrastructure companies built it, and the six weeks of activity I first wrote about have turned into a sustained, accelerating build-out. The parallel web for agents is not coming. It is live, it is shipping new layers every month, and the agents using it now run on their own schedule.

Organisations that understand this shift still have a window. Not to panic, not to overhaul everything overnight, but to make deliberate, low-cost moves (structured content, a Content Signals header, machine-readable product data, a governed approach to the scheduled agents you deploy) that position them for the agent-first world that has already arrived.

Those that wait will find their content invisible and their products undiscoverable to the fastest-growing channel on the internet. Not because they were blocked. Because they never opened the second door, and never noticed who had already walked through it.

Sources and further reading:

Paul Bratcher is a Partner at Prosper AI Consulting, specialising in outcome-driven technology adoption for organisations navigating AI transformation.

Article | Prosper AI Consulting, UK