AI agents won't scrape your site - they'll skip it

There’s a comforting story leaders tell themselves about AI agents. The agent will visit our website, read it like a careful customer, fill in our forms, and get its answer. Our site already works for people, so it’ll work for the robot too.

It won’t. Not for long, anyway.

An autonomous agent doesn’t want to read your page. Reading a rendered page is slow, expensive, and breaks the moment you ship a redesign. What the agent wants is a structured surface it can call directly, the same way it calls any other tool. When Google shipped WebMCP in Chrome, the Hacker News headline said it out loud: the feature turns “every website into a tool for AI agents.” A tool, not a brochure. If your site gives an agent a clean way to act, it acts. If it doesn’t, the agent has two fallbacks, and you’ll like neither of them: cobble together a brittle scraping layer that snaps on your next release, or quietly route the user to a competitor who exposed a real surface.

The thing is, either way you lose, and you lose in a way that barely shows up in your analytics. This is one corner of a broader shift in how AI is changing operations: the people using your software are no longer the only ones, and the non-human users have sharp preferences about how they want to be served. Ignore them and they don’t complain. They just go elsewhere, silently, and you see a number drift down with no obvious cause.

Why agents won’t browse your site like people do

A person lands on your page, scans it, and figures out where to click. An agent could do the same by parsing pixels, and the first wave of browser agents did exactly that. It’s a transition technology, not the destination. Pixel-parsing is slow, clunky, costs tokens on every screen, and is famously fragile. Move a button and the agent that learned your old layout is suddenly lost.

Picture the agent trying to book a demo on your site the pixel way. It loads the page, waits for everything to render, scans for something that looks like a form, guesses which field is the email, guesses which is the date, and clicks what it hopes is the submit button. Each guess is a chance to fail, and the whole sequence shatters the day your marketing team ships a new layout.

Now picture the same agent finding a declared “schedule a demo” tool with named inputs. It calls the tool. Done. One of these the agent will keep doing. The other it abandons the first time it breaks, and it remembers which sites wasted its time.

Structured access fixes all of that, which is why the industry is racing toward it. The Chrome WebMCP docs describe a “proposed web standard” that lets a site “expose structured tools for AI agents” instead of leaving them to interpret the interface. Give the agent a declared “search” or “book” tool and it basically stops guessing. So the strategic question isn’t whether agents will reach your software. It’s whether they’ll reach it through a door you built or a window they pried open. One of those you control. The other one breaks on your schedule and blames you for it.

There’s a second-order effect leaders miss. On that same Chrome thread, a commenter asked, “How would this not create backend load and abuse?” It’s a fair worry, and it points at the real shift: an agent is a new kind of visitor. It doesn’t behave like a browser session or a tidy API client. It does whatever the prompt steering it happens to ask for, at machine speed. Designing for that visitor isn’t optional once your customers’ assistants start showing up.

An agent finds three doors at your site

When an AI agent arrives wanting to do something with your software, it’s really choosing among three doors. Knowing which one you’ve left open is the whole game.

Door one is WebMCP, the in-browser surface. It works when a person is already on your page with an AI assistant running, and the assistant can see and call the tools you registered. Door two is a Model Context Protocol server: an authenticated endpoint a connected AI client calls directly, no browser involved, with a login and permissions in front of it. Door three is the one too many sites leave as the only option: nothing structured at all, so the agent either scrapes you or skips you.

Door three deserves a harder look, because it’s where most sites stand right now and most leaders assume it’s a neutral place to stand. It isn’t. “No structured surface” used to mean “agents can’t really use us yet,” which felt safe enough. Today it means “agents will use someone else.” And the backend-load worry from that Chrome thread cuts both ways: even when an agent does scrape you, it pounds your pages in patterns no human ever would, so you pay to serve a visitor you didn’t design for and can’t bill. Leaving door three as your only option isn’t a decision to wait and see. It’s a decision to be the slow, costly option that agents quietly route around on their way to a competitor.

Standing still is the one move here that isn’t actually neutral.

Most people conflate the first two doors, and the confusion is costly. They’re different surfaces for different moments. WebMCP serves the human-plus-assistant case on your actual website. The MCP server serves the headless case, where an agent in someone’s chat tool needs your data and never opens a browser. Tallyfy runs both.

The in-browser layer is four read-only tools; the authenticated server exposes 100+ tools to clients that log in. They are not the same door, and building one when you needed the other is a common, expensive mistake.

Do you need MCP, WebMCP, or both?

Here’s the part where leaders want a straight answer, so here’s mine. If your customers reach you mostly through your website, and they’re starting to use browser-based AI assistants, you’ll want WebMCP on the pages where they act. When your product is something other AI clients need to pull from or push into programmatically, like a status, a record, or a task, you want an authenticated MCP server. Both true at once? You want both, and that’s increasingly the normal answer for any serious operations tool.

A question we field from operations leads keeps getting sharper: which of our vendors will our team’s AI assistant actually be able to use? That’s the real test, and it cuts the other way too. Your procurement portal, your ticketing system, your onboarding app, all of them are about to be judged by whether an agent can act through them. The vendors who exposed a structured surface get used. The ones who didn’t get worked around, and “worked around” usually means replaced.

Run that test on your own product for a second. A customer asks their AI assistant to “check the status of my onboarding with that vendor.” If you expose an MCP server, the assistant pulls the status and the customer never logs in, never files a ticket, never emails you. If you don’t, the assistant gives up, the customer does it the slow way, and your product quietly starts feeling like friction next to the one that answered in a sentence. Now multiply that across every tiny interaction in a week. You can see which direction the switching pressure points, and it isn’t toward the vendor who made the agent do all the work.

Friction an agent hits is still friction, even when no human ever files a complaint about it.

But, and this is the part the tooling conversation skips, a door is not a destination. Exposing a “create order” tool to an agent is reckless unless that call lands inside a process with the right checks. That’s the difference between AI mediated by a structured layer and an agent with a raw key to your systems. The surface gets the agent in. The workflow decides what it’s allowed to do once it’s inside.

What to do before your tools become targets

You don’t get to opt out of this. You only get to decide whether you’re ready when it arrives. So treat it like the operational change it is, not a developer side-quest.

Pick your highest-traffic customer-facing process and ask a blunt question: if an agent tried to complete this for someone today, what would it do? If the honest answer is “scrape the page and hope,” you’ve found your first project. Then decide the surface based on how the agent actually shows up, on your site (WebMCP) or in someone else’s AI client (an MCP server), and build the structured access there.

Keep the scope tiny at first. One process, a structured surface in front of it, and a human gate on anything that spends money or touches a customer record. You’ll learn more from two weeks of real agent traffic than from a quarter of planning slides, because the edge cases turn up fast and they’re never the ones you’d have guessed in a meeting. Widen only once the narrow version has earned it. That’s the same discipline that keeps any automation safe, and agents don’t get a special exemption from it just because they’re new.

Whatever you do, don’t expose the tools before you’ve defined the process behind them. We’ve made the case that an agent needs a workflow to follow and that the safest pattern is to bind it to a defined workflow rather than hand it the run of the building. A clean surface with no process behind it doesn’t make you AI-ready. It just makes you faster to break. The agents are coming to your software whether you prepared a door or not. The only choice you control is whether they find a tracked process waiting for them, or a button with nothing behind it.