Cloudflare has accused AI startup Perplexity of scraping websites that explicitly opted out of data collection, raising fresh concerns over AI companies’ data practices. The internet infrastructure giant says Perplexity bypassed protections, ignored industry standards, and disguised its identity to collect online content.
Cloudflare Detects Suspicious Activity
Cloudflare’s research team launched an investigation after multiple customers reported unauthorized scraping from Perplexity’s systems. The company monitored its network and identified patterns of behavior that matched large-scale, stealthy web crawling.
The team says Perplexity ignored Robots.txt directives, a long-standing web standard that informs automated bots which pages they should avoid. Robots.txt files give publishers the ability to opt out of data collection, but Cloudflare claims Perplexity disregarded these rules entirely.
According to Cloudflare, the AI startup also hid its identity by:
- Switching user-agent strings to mimic regular browsers like Google Chrome on macOS.
- Rotating autonomous system numbers (ASNs) to appear as different sources and avoid consistent tracking.
Cloudflare says these methods allowed Perplexity to bypass filters and access pages that site owners had explicitly restricted.
“This activity was observed across tens of thousands of domains and millions of requests per day,” Cloudflare stated. The company used machine learning and network-level signals to detect, fingerprint, and confirm the bot traffic as originating from Perplexity.
How AI Models Depend on Scraping
Modern AI models rely heavily on large-scale web data for training and response generation. Startups like Perplexity aggregate vast volumes of text and images from the internet to power their conversational AI and search tools.
However, web publishers increasingly restrict automated scraping to protect content from unauthorized use. They deploy Robots.txt files, firewall rules, and bot-blocking tools to prevent AI companies from repurposing their material without consent or compensation.
Despite these defenses, Cloudflare’s findings suggest AI crawlers can still bypass rules if they deliberately disguise themselves. This behavior heightens tension between content creators and AI firms, as many publishers rely on ads, subscriptions, and licensing deals that depend on controlled access to their work.
Perplexity Denies the Allegations
Perplexity pushed back against Cloudflare’s accusations. Company spokesperson Jesse Dwyer dismissed the blog post as a sales tactic, claiming Cloudflare provided screenshots that didn’t prove actual content access.
In a follow-up email to TechCrunch, Dwyer denied ownership of the crawler named in Cloudflare’s report. “The crawler isn’t even ours,” he said, insisting that Perplexity follows ethical data practices.
Despite the denial, Cloudflare stood by its investigation. The company says it ran controlled tests that confirmed the scraper’s identity and activity, including instances where the bot impersonated Chrome on macOS to circumvent blocks.
By switching user-agents and hiding ASNs, Perplexity allegedly evaded detection and bypassed publisher protections. Cloudflare calls this a deliberate tactic to mislead websites into treating the traffic as human visitors.
Cloudflare Responds with New Enforcement
Cloudflare has removed Perplexity from its list of verified crawlers. Verified crawlers receive preferential treatment on websites using Cloudflare because site owners can distinguish them from malicious bots.
By revoking that status, Cloudflare ensures its security tools can automatically block or throttle Perplexity’s traffic. The company has also launched new tools to help publishers control AI scraping:
- AI Scraper Marketplace:
Website owners can choose to charge AI companies for content access, creating a potential revenue stream. - Free AI Bot Blocker:
A new free service allows any publisher to block known AI crawlers attempting to train large language models on their content.
Cloudflare CEO Matthew Prince has warned repeatedly about AI’s threat to the web’s economic model. He argues that unchecked scraping undermines advertising revenue, erodes incentives for content creation, and places publishers at a disadvantage.
A Pattern of Controversy for Perplexity
Perplexity already faced scrutiny last year when critics accused the company of plagiarizing content. Analysts said its AI-generated summaries occasionally mirrored original sources without proper attribution.
With Cloudflare now publicly challenging its data practices, Perplexity enters deeper regulatory and reputational risk territory. AI ethics and copyright concerns have intensified in recent months, and lawsuits targeting AI scraping continue to emerge worldwide.
If regulators investigate Cloudflare’s findings, Perplexity may face legal and commercial challenges, particularly if publishers begin blocking its services en masse.
The Broader Implications for AI and the Web
This conflict highlights the growing tension between AI startups and web publishers. AI companies need massive amounts of data to train models, while publishers seek control and compensation for how their content gets used.
Several industry trends emerge from this standoff:
- Enforcement Will Get Stricter:
Cloudflare and other infrastructure providers will introduce advanced detection methods to identify stealth scrapers. - Publisher Monetization Will Rise:
As Cloudflare’s AI Scraper Marketplace suggests, pay-to-train models may become a new revenue channel for content owners. - AI Companies May Face Reputation Risk:
Perceived scraping misconduct can damage trust, especially as regulators and copyright holders take action against unauthorized content use. - Ethics and Transparency Will Become Essential:
AI firms must clearly disclose their data sources and honor opt-out mechanisms like Robots.txt to avoid backlash.
The Road Ahead
Cloudflare’s accusations against Perplexity spark a larger debate about how AI interacts with the open web. If AI companies continue to bypass publisher controls, the internet’s content ecosystem may fracture, with increased paywalls, private APIs, and stricter bot-blocking measures.
Meanwhile, Cloudflare positions itself as a defender of publisher rights, giving web owners the tools to fight unauthorized AI scraping. If more companies adopt these protections, AI firms may need to negotiate access rather than take data without permission.
Perplexity now faces a critical choice: work with publishers to rebuild trust or risk escalating conflicts that could limit its data access. As the AI industry scales rapidly, these early battles will shape the norms and rules of online content use for years to come.
Also Read – SSC Exam Mayhem 2025: Chaos, Protests, and Lost Dreams