Agentic web

What is the agentic web?

The agentic web is the web as it looks once a real share of your visitors is software acting for a person instead of the person. Someone tells an assistant "find a supplier of stainless steel worktops near Hasselt, compare delivery times and order the 180 centimetre one". The assistant opens your site, reads it and tries to finish the job. Your site got a visit and no human ever looked at it.

That visitor ignores your banner, does not scroll past the hero photo, and will not find a price that appears only after a click. It reads text, follows links, and when it cannot find what it needs it moves to the next result. It also does not convert the way a person converts: it may finish the job on a competitor's site, or copy your phone number and hand it back to its human, and your funnel records neither.

Two neighbouring entries cover the parts around this one. Generative engine optimisation is about getting named inside an AI answer, so about being found. The agentic browser entry is about the tool your own staff run. This one is about your website and the visitor that arrives with a task to complete.

Who is visiting your site now

Cloudflare reported in 2026 that more than half of internet traffic is now non-human, the first time it crossed that line. Not all of it is AI: uptime monitors, price scrapers and search crawlers have been in your logs for twenty years. What changed is the mix, and Cloudflare now splits AI traffic by purpose.

• Training. A crawler taking your content to train a model. Cloudflare measured 52 percent of crawler requests as training in June 2026, up from 22 percent in spring 2025. It sends you nothing back.

• Search. A crawler indexing your pages so an answer engine can cite you later. This one still produces referrals.

• Agent. Automated activity in real time on behalf of a person. This is the agentic web in the narrow sense, and the one that can turn into an order.

Three signals separate them, in rising order of how much you can trust them: the user agent string, which any script can copy; the IP address, checked against the list the operator publishes or against reverse DNS; and a signature on the request itself. Web Bot Auth is that last one. It uses HTTP message signatures so a bot proves per request which operator it belongs to. The IETF chartered a working group for it with first specifications due in 2026, and Cloudflare has classified traffic this way since August 2025, starting with OpenAI's ChatGPT agent, Block's Goose, Browserbase and Anchor Browser.

Your analytics tool shows almost none of this. Google Analytics excludes known bots automatically using Google's own research plus the Interactive Advertising Bureau's list, and its documentation says you cannot switch that off or see how much was excluded. An agent that fetches your HTML without running JavaScript never fires the tag at all. The raw server or CDN log is the only place you find out whether this is happening to you yet.

How a site talks to an agent, and who you let in

Four ways to be readable, from the one that works everywhere to the one that works nowhere yet. Plain HTML that says what it means: the price as text, not baked into an image or fetched after a click, and stock as words rather than a green dot. Structured data, the same schema.org markup that has fed search results for a decade, as long as it agrees with the page, because a stale price in the markup is worse than no markup. A file describing the site for machines: the llms.txt proposal from Jeremy Howard, published in September 2024 and updated to a second version on 10 August 2026, which remains a proposal no engine has been shown to depend on. And WebMCP, which lets a page publish its own actions as tools an agent can call through a document.modelContext object; it is a draft report of a W3C community group, it states on its face that it is not a W3C standard, and Chrome has run it only as an origin trial. The unglamorous fifth option is a documented public API, with an MCP server in front of it if you want assistants to call it directly.

Access is the genuinely unsettled part. robots.txt, standardised as RFC 9309, was written for crawlers fetching pages to index. It is a request rather than access control, and it has no vocabulary for an agent acting for one named person who wants to spend money with you. A rule that keeps out a training crawler keeps out that customer's assistant too.

Three pieces exist as of September 2026. Identification is Web Bot Auth, still a draft but already checked in production by several networks; it tells you which operator an agent belongs to, not which person sent it. Blocking by purpose is live: Cloudflare lets you treat search, agent and training traffic separately and, for each, block everywhere, block only on pages carrying ads, or allow. From 15 September 2026 new domains get training and agent traffic blocked on ad pages by default. Charging is the thin one: pay per crawl revives HTTP status 402, Payment Required, so a crawler either presents payment intent in a header or gets a price back. It was announced on 1 July 2025 and the documentation still called it a closed beta in July 2026.

The hard question is not technical. Blocking a training crawler costs you nothing you can measure. Blocking a shopping agent costs you the order, and you never see the order that did not happen.

When an agent tries to buy

A customer asks their assistant to reorder two boxes of A4 paper and a toner cartridge from the office supplier in Genk they have used for years. A typical webshop, nothing unusual about it.

1. The agent fetches the product page. The price is written in by JavaScript after the page loads, so the raw HTML holds the word "from" and an empty element. There is no number to compare.

2. It falls back to a browser that renders the page. A cookie banner covers the content and the accept button sits in a third-party frame. Sometimes it clicks the right thing, sometimes it accepts marketing cookies on the customer's behalf.

3. Stock is a green dot with no text. In stock and back-ordered look identical to anything reading the page as text.

4. The request comes from a data centre address with an unfamiliar user agent and no signature, so bot protection serves a challenge. Most agents stop here, and your log records a blocked bot rather than a lost customer.

5. Suppose it gets through. Checkout needs an account, and creating one sends a confirmation link to the customer's mailbox, which the agent cannot open.

6. Payment. Bancontact, or a card with 3-D Secure. Under PSD2 the payment needs strong customer authentication, and the second factor is a push to the customer's banking app. The agent cannot supply it, and it should not be able to.

Six steps, and only the last is a rule you have to respect. The other five you could fix this quarter, and four of them would help a human on a phone as well.

The protocols meant to solve that last step exist at different stages. Google announced the Agent Payments Protocol (AP2) on 16 September 2025 with more than sixty partners including Mastercard, PayPal and Adyen. It uses signed mandates carried as verifiable credentials: an intent mandate for what the person asked for, a cart mandate for the exact items and price they approved, so a merchant can check the agent's request against what the human authorised. The Agentic Commerce Protocol from OpenAI and Stripe covers the checkout handover and the tokenised payment credential while the merchant stays merchant of record. It powered Instant Checkout inside ChatGPT, and in March 2026 OpenAI moved that feature into ChatGPT Apps and said it was prioritising search and product discovery, with the protocol as the plumbing underneath. The protocol survived; buying straight out of the chat window, in that first form, did not. Underneath both sits European law: the firm Osborne Clarke wrote in March 2026 that agentic payments stay subject to PSD2 and its authentication rules with no separate regime, and that it is still unresolved who provides the payment service and what counts as valid authorisation when an agent presses the button.

A site for a person versus a site an agent can transact on

Compare them on what the visitor needs in order to finish a task. A person needs to be oriented and reassured: a photo, a review, a phone number, a sense that the company is real. They forgive a price that lives inside a picture because they can read pictures, they solve a captcha, they wait for a confirmation mail and they pick up their phone for the bank's push notification. They can be persuaded, and they try again tomorrow if the site is down.

An agent needs the facts retrievable as text, one stable URL per product, an action it can perform without guessing, an identity it can prove, and a clean point at which it hands payment back to its human. It cannot be persuaded, it has no patience and it does not come back. The two lists overlap more than they conflict: nearly everything on the second one also makes a faster site for the person.

What to do now, in order

1. Read your logs before you spend anything. Ask whoever runs your hosting or CDN for a breakdown of requests by user agent over the last month, and how many were challenged or blocked. That tells you whether agent traffic is a real number for you or a slide in someone's deck.

2. Make your core facts machine-readable and correct. What you sell, where you are, what you charge, when you deliver, in text on the page and mirrored in structured data. The common failure is not missing markup, it is markup that contradicts the page.

3. Make prices and availability retrievable without JavaScript. Load the page with scripting turned off and look at what is left. If the price and stock status survive, an agent can read them. If they do not, that is work measured in hours, not months.

4. Decide deliberately whether agents may buy, per product group if it differs, then check what your CDN or bot protection is doing today, because a default was set for you. A wholesaler with a negotiated price per customer may reasonably want agents out of the ordering flow. A retailer of standard goods almost certainly does not.

5. Design the handover. The realistic outcome today is an agent that gathers the facts, fills a basket and hands its human a link to finish. A shareable cart URL, a quote request that works without an account, and a checkout that survives being opened on a phone twenty minutes later catch that business now, with no new protocol involved.

What to watch out for with the agentic web

Separate what runs from what is a proposal. Bot classification by purpose, blocking and challenging, signed agents, structured data and plain HTML are deployed and usable this week. llms.txt, WebMCP, AP2 and agent-initiated checkout in Europe are proposals or early pilots. Both sets get sold in the same slide deck.

Ask what will read it. When someone offers to make your site agent-ready, ask which mechanism they will implement and what evidence they have that any agent reads it. Your server log answers that within a week.

Your numbers get stranger, not clearer. Session counts, bounce rate and time on page assume a human. As agent visits grow, those metrics describe a population that is partly machine, and an agent-mediated sale that ends in a phone call shows up in your funnel as nothing at all. Asking new customers how they found you is becoming more reliable than the dashboard.

The access question belongs to the owner, not to IT. Whether an assistant acting for a customer may see your prices is a commercial decision with the same weight as deciding whether to publish prices at all. Right now it is being made by default settings.