Why AI Agents on the Web Keep Hitting Access Barriers

AI agents on the web can only do useful work where they’re allowed to act—and where a site’s defenses let them through. That makes access, not just model capability, a practical constraint for anyone building an agent that needs to browse, book, or shop.

Why AI agents on the web hit access barriers

Websites including Yelp, Delta Air Lines, and financial institutions have restricted AI agents over licensing or security concerns. Those decisions are understandable from the site’s side: an automated agent can raise questions about how content is used or whether an action is authorized. But the result for an agent builder is straightforward: a workflow that depends on those sites may stop at the point where it needs to do real work.

The source article also notes that anti-bot systems can block agents without a site specifically targeting AI. These systems are designed to catch automated activity, and “These AI agents trigger those same safeguards.” In practice, an agent can be denied even when the developer has no obvious way to distinguish a general bot defense from a deliberate block.

That changes how I’d evaluate a browser-using agent. A clean demo on an accessible site says little about whether the same workflow can complete reliably across a user’s actual mix of services. The integration problem is not only handling page changes; it is whether the agent is permitted to reach the page at all.

Retail partnerships don’t solve the whole web

There are competing approaches. Meta has partnered with retailers including Walmart and Gap to let its Muse assistant shop on their sites. Amazon, meanwhile, has challenged Perplexity’s agentic shopping feature. The source reports that OpenAI, Meta, Google, and others have rolled out agents, while access to websites remains uneven.

Partnerships may create a workable route for specific shopping tasks, but they are not evidence of general access across the web. For a product team, that distinction matters: a supported retailer flow can be valuable, yet it does not guarantee an agent can complete the same task on another site. The quote “Turning away a personal agent means turning away the customer behind it” captures the commercial argument for access, but it does not settle the site’s licensing or security concerns.

Build around permission, not an assumed open web

Cloudflare has introduced tools to block AI crawlers and says its efforts respond to the strain AI bots place on web infrastructure. That makes the conflict more than a one-off dispute between a particular agent and a particular site: bot defenses can affect access even when the immediate goal is a user-requested task.

My read is that agent reliability will depend partly on the access arrangements around each workflow, not just the agent’s ability to interpret a page. In 2025, the practical question for a developer is whether a task has a supported path—such as a retailer partnership—or whether it relies on browsing that may be restricted or caught by anti-bot controls. I would treat broad web access as an assumption to test, not a capability to promise. The signal worth watching is whether agents can complete useful tasks across more sites without running into access restrictions or triggering the defenses those sites already use.