If you’re one of the shrinking majority who still uses a search engine rather than an AI tool, you may have noticed a small but telling shift: more and more websites now stop you at the door with a single blunt question, “Are you a human?”
It sounds like a minor bit of bot-filtering. It’s actually a symptom of a much bigger tension playing out across the web.
How search used to work
Type “what are the responsibilities of an executor of a will” into a search engine, and you’ll get back a list of websites that might contain the answer. There could be hundreds of them. It’s then your job, not the search engine’s, to work through them and decide which one actually answers your question well.
In practice, most people just trust the order the results appear in, using ranking criteria the search engine keeps deliberately secret. That secrecy spawned an entire industry: Search Engine Optimisation, or SEO. Its purpose isn’t really to help you find the best information; it’s to get a website in front of you in the hope that you’ll find something worth buying while you’re there. To do that job, SEO providers have to reverse-engineer the rules a search engine is using: rules that were never built around the actual value of the information being ranked.
The result, often, is that the answer you actually wanted, the “nugget,” is buried thirty pages deep in a site optimised more for visibility than usefulness. You end up reading far more than you needed to, just to extract one fact.
How AI tools changed the deal
AI tools remove nearly all of that friction. Instead of handing you a list of possible sources and leaving you to do the work, they read the relevant sites themselves and return a specific, plain-text answer to your specific question, in a fraction of the time it would take a human to do the same search manually.
On the surface, that looks like a pure improvement. What’s not to like?
The catch is that producing the original information isn’t free. Someone researched it, wrote it, tested it, published it, usually a human, usually a business, often with the hope that a reader who found their content useful would explore the rest of the site, get in touch, or eventually buy something. AI answers can bypass that entirely: the reader gets the nugget of information without ever being exposed to the person or company who created it, or to anything else they offer.
That’s the elephant in the room. And it’s why more and more information providers are starting to ask AI systems to identify themselves, and, increasingly, to decline them entry.
It isn’t a universal reaction
Not every business is closing the door, though, and it wouldn’t necessarily make sense for all of them to. Some information providers are genuinely happy for AI tools to draw on their knowledge. It extends their reach at effectively no cost to them.
The value of staying open often depends entirely on the nature of the question being asked. Take a query like “who can help me with the drafting of a will?” An AI tool answering that question has no way of knowing about, let alone recommending, a firm that has chosen to restrict itself to human visitors only. By closing its doors to AI, that firm hasn’t just kept its content private; it’s made itself invisible to an entire and fast-growing category of search behaviour.
Where this leaves us
There’s no single, definitive answer to whether a company should open its content to AI or shut it out. Reasonable businesses will land in different places depending on their model, their audience, and what they’re actually trying to achieve by publishing in the first place.
What does seem clear is that the current arrangement, near-unlimited AI access to the web’s accumulated knowledge, at essentially no cost to the systems doing the accessing, isn’t a stable end state. AI itself isn’t new. What changed recently is cheap, fast access to petabytes of data. That kind of access doesn’t stay free indefinitely.
Someone, eventually, has to pay for the information being used, whether through subscription models, advertising, licensing deals between publishers and AI providers, or some combination that hasn’t been invented yet. The economics of who pays, and how, are still being worked out in real time. Getting that right, building a fair way to administer it, may turn out to be one of the more consequential problems of the next few years.