Tech strategy is difficult AND fascinating because it’s unpredictable. In addition to worrying about the actions of direct competitors (i.e. Samsung vs Apple), companies need to also worry about the actions of ecosystem players (i.e. smartphones and AI vendors) who may make moves that were intended for something else but have far-reaching consequences.
In the competition between frontier AI models, it is no surprise that Google, where the Transformer architecture virtually all LLMs are based on was created, was caught off-guard by the rapid rise of OpenAI and AI-powered search vendors like Perplexity and Chinese participants like DeepSeek and Alibaba/Qwen. While Google (and its subsidiary DeepMind) have doubled down on their own impressive AI efforts, the general perception in the tech industry has been that Google is on defense.
But, as I started, tech strategy is not just about your direct competition. It’s also about the ecosystem. Cloudflare, which offers distributed internet security solutions (which protect this blog and let me access my home server remotely) recently announced that it would start blocking the webscrapers that AI companies use due to concerns from publishers of websites that their content is being used without compensation.
However, because search is still a key source of traffic for most websites, this “default block” is almost certainly not turned on (at least by most website owners) for Google’s own scrapers, giving Google’s internal AI efforts a unique data advantage over it’s non-search-engine rivals.
Time will tell how the major AI vendors will adapt to this, but judging by the announcement this morning that Cloudflare is now actively flagging AI-powered search engine Perplexity as a bad agent, Cloudflare may have just given Google a powerful new weapon in it’s AI competition.
The Internet as we have known it for the past three decades is rapidly changing, but one thing remains constant: it is built on trust. There are clear preferences that crawlers should be transparent, serve a clear purpose, perform a specific activity, and, most importantly, follow website directives and preferences. Based on Perplexity’s observed behavior, which is incompatible with those preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling.
Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Gabriel Corral, Vaibhav Singal, Brian Mitchell, Reid Tatoris
| Cloudflare Blog