UA Spoofing Is Free Scraping: What Fake GPTBot Traffic Means for Your Site

In August 2026, a security researcher named Can Bölük posted a table that made the rounds. He had tested a list of AI crawler user-agent strings — GPTBot, ClaudeBot, and obscure internal ones like OpenAI File Downloader — against major websites, changing nothing else. Same requests, same machine, same TLS stack. Only the User-Agent header changed.

The result: sites that throw challenges and paywalls at a normal browser served clean, complete content to strings that claimed to be AI crawlers. LinkedIn, the post claimed, even dropped its engagement-bait and login-wall layers for a UA identifying as Claude-User. The post crossed 619,000 views in three days.

His one-line summary: “UA spoofing is back, baby. For only $0.00, you too can be OpenAI File Downloader.”

UA spoofing is back on baby, for only $0.00 you too can be OpenAI File Downloader, XaiImageApiFetch/1.0. Some sites like LinkedIn even remove their click-bait/paywall garbage if you’re Claude-User. — Can Bölük (@_can1357), August 21, 2026

UA spoofing is back on baby, for only $0.00 you too can be OpenAI File Downloader, XaiImageApiFetch/1.0

Some sites like LinkedIn even remove their click-bait/paywall garbage if you're Claude-User pic.twitter.com/ti2VbZetVW

— Can Bölük (@_can1357) August 21, 2026

This is not a hacking story. There is no exploit, no zero-day, no stolen credentials. It is a story about incentives — and if you run a website, it is quietly your problem.

Why would anyone spoof an AI bot?

To understand why this works, you have to understand what changed in the last two years of web economics.

When ChatGPT launched, AI crawlers were treated as parasites. Sites blocked GPTBot in robots.txt because they did not want their content used as free training data. Cloudflare’s Q1 2026 analysis of robots.txt files across its network found GPTBot to be the single most-blocked crawler on the web.

Then the incentive flipped. AI search — ChatGPT, Perplexity, AI Overviews — started sending real traffic. A citation inside a ChatGPT answer became worth as much as a Google ranking. So the same site owners who blocked AI crawlers in 2024 started whitelisting them in 2026: let GPTBot in, let ClaudeBot in, serve them everything, because being quoted by an AI is the new SEO.

The whitelisting is not just a robots.txt line. It became infrastructure. Sites now serve AI-branded user-agents from the same fast paths, skip interstitials for them, and route them around cookie walls and login nudges. The reasoning is rational: an AI crawler that hits a “sign in to continue” wall sees nothing, cites nothing, and your competitor gets the citation instead.

That is the door Bölük walked through. If the VIP lane is keyed to a string that says “I am OpenAI,” then anyone who types that string gets the VIP lane.

What actually gets bypassed

Three layers, in increasing order of embarrassment for the sites involved.

Layer one: robots.txt and rate limits. Many sites that aggressively limit unknown crawlers let known AI bots through with generous or unlimited crawl rates. A scraper claiming to be GPTBot inherits that budget.

Layer two: paywalls and interstitials. This is the LinkedIn case. A site decides that AI crawlers should see full article content — otherwise the AI cannot quote it — but humans should see a login wall after paragraph three. The gate is implemented per user-agent. A human with a Claude-User UA string is, to that logic, a crawler. Full article, no wall.

Layer three: bot protection itself. Some edge networks that challenge suspicious traffic exempt user-agents on the AI allowlist. A request that would get a CAPTCHA as curl sails through as ClaudeBot. The security industry has a name for traffic like this — spoofed verified bots — and it has been a known signature of hostile scanning for years, precisely because pretending to be a trusted bot is the cheapest way in.

Each layer exists for a defensible reason. Stacked together and keyed to an unsigned string, they turn “I claim to be Claude” into a master key.

The oldest trick on the web, wearing a new coat

If any of this sounds familiar, it should. User-agent spoofing is not a novel AI-era invention — it is a recurring pattern, and the industry has fought this exact war before.

In the 2010s, scrapers discovered that pretending to be Googlebot unlocked the same VIP treatment from thousands of sites. The response became standard practice: check the reverse DNS of the connecting IP. A real Googlebot request comes from *.googlebot.com and resolves back to a Google IP. A fake one comes from a random VPS. Within a few years, “verify the IP, not the string” was table stakes for anyone running a serious site.

The AI crawler wave reset that institutional knowledge. New generations of bots arrived faster than verification habits spread. Site owners copied allowlist snippets from SEO blogs. Middleware libraries shipped AI-bot detection keyed to UA strings because that is a ten-line function. The person who configured it moved on. Nobody went back to ask “wait, do we check where these requests actually come from?”

There is a second, uglier variant worth knowing about: some scrapers do not bother claiming to be an AI bot at all. They rotate through real browser user-agents and residential proxies, which defeats IP checks entirely. Bölük’s table did not need that machinery, which is exactly why it spread — a plain string substitution is something anyone with curl can do in ten seconds. When the cheap attack works, you do not need the expensive one.

The numbers behind the incentive flip

It helps to see how fast the ground moved.

Cloudflare’s agentic internet report put AI training-related crawler requests at roughly 52% of all crawler traffic by June 2026, up from 22% in spring 2025. Radar data showed AI crawlers at about 20% of verified bot traffic in May 2026, with AI-search crawlers growing month over month. Meanwhile GPTBot remained the most-blocked crawler in robots.txt — meaning the web is simultaneously blocking AI bots and courting them, often on the same domain, for different purposes.

That contradiction is where spoofers live. A site blocking GPTBot for training but allowing it for search, or blocking it in robots.txt while exempting it at the WAF, creates the exact ambiguity a forged header exploits. The bots do not need to defeat your defenses. They need your defenses to disagree with each other.

And the traffic stakes are no longer theoretical. Referrals from AI answers are climbing across every publisher dashboard that tracks them. A site that decides “we want ChatGPT to cite us” is making a rational bet — one that happens to be implemented, in most stacks, by trusting a header a child could edit.

Who is actually doing the spoofing

It is worth separating the actors, because “scrapers” is doing a lot of work in this story and the motivations differ.

Data brokers and training-data suppliers. Companies that sell web datasets face the same anti-bot walls as everyone else. An AI-branded UA that unlocks the fast lane is not a hack to them; it is a cheaper line item than a residential proxy pool. These are the high-volume, patient spoofers, and the ones most likely to already be inside your “AI traffic” numbers.

Competitors and price watchers. E-commerce and SaaS sites grant AI crawlers generous access to product and pricing pages — they want to be cited when someone asks an AI “what does X cost.” A competitor’s scraper claiming the same identity gets the identical clean feed, at scale, with no JavaScript rendering cost. For any business where pricing data is competitive intelligence, an AI allowlist is an intelligence leak.

Security researchers and hobbyists. The Bölük table itself came from this bucket. Researchers test these things because the gap between claimed security and actual security is the finding. This group publishes, which is how site owners hear about the problem at all.

Copycats. The uncomfortable part of a 619,000-view post demonstrating a free bypass is that it does not stay a research finding. Within days of publicity like this, exploit snippets circulate in scraping communities, get wrapped into commercial scraping APIs, and become a checkbox in data-acquisition tools. Whatever was true before August 2026 is now more true.

The common thread: none of these actors need to defeat cryptography or beat Cloudflare’s ML. They only need origins that key trust to a header. As long as those exist, the supply of takers is unlimited.

What this means if you run a site

You are probably not being scraped this way today. The spoofers going after high-value targets — paywalled publishers, LinkedIn, data-rich platforms. But the pattern tells you three things about the web you are publishing into.

One: your robots.txt AI policy is now a security surface. If you allow GPTBot with special permissions, understand that the permission is being granted to a string, not to OpenAI. Behind Cloudflare with verified-bot checking enabled, spoofed traffic gets caught. On origin servers, WAFs, and homegrown middleware, it usually does not. Assume your AI allowlist is public knowledge, because it is — it is served as a plain-text file from your own domain.

Two: the “AI-ready” checklist items compound. Serving clean HTML to AI agents, skipping interstitials for them, opening llms.txt — each is a small hole. Together they are a wide-open side door for anyone who reads your robots.txt and owns a HTTP client. The more aggressively you court AI crawlers, the more you should verify who is actually crawling.

Three: verify, do not trust. The practical playbook, in order of effort:

• Put AI bot handling behind a reverse proxy that checks reverse DNS against the official OpenAI and Anthropic published IP ranges, not just the UA string.
• If you are on Cloudflare, enable verified-bot enforcement rather than manual UA rules.
• Decide deliberately what AI-facing paths expose. If ClaudeBot sees the full article and a human sees a paywall, that difference is a product decision that now leaks — price it accordingly.
• Watch your logs for AI user-agents arriving from IPs outside the official ranges. That is not OpenAI discovering your site. That is someone pretending.

How to read your own logs for fake AI bots

OpenAI and Anthropic both publish the IP ranges their crawlers use, and both document reverse-DNS patterns (*.openai.com, *.anthropic.com). That gives you a concrete audit you can run this week:

1. Pull a month of access logs and filter for requests whose user-agent contains GPTBot, OAI-SearchBot, ClaudeBot, or Claude-User.
2. For each, resolve the connecting IP’s reverse DNS. Anything that does not resolve to the official domains is either a proxy-fronted real crawler (rare, and the companies say they do not do it) or a spoofer (common).
3. Sort spoofed requests by volume and by path. High-volume spoofing concentrated on product pages, pricing pages, or paywalled content is a competitor or a data buyer, not a security researcher.
4. Feed the confirmed spoofed IPs into your WAF or firewall. This is the one case where IP blocking is genuinely effective — spoofers are lazy by construction; anyone maintaining a residential proxy farm uses browser UAs instead.

If you find nothing, you have still gained something rare: you now know your AI traffic is real, which means your AI visibility numbers — citations, crawl frequency, answer share — are measuring OpenAI and Anthropic, not an anonymous scraper wearing their name.

The deeper story: identity on the agent web

Zoom out and this is one battle in a bigger war: the web is being rebuilt around agents — AI crawlers, AI shoppers, AI researchers — and nobody agrees on how an agent proves who it is.

Cloudflare’s bot report put AI-related crawler requests at over half of all crawler traffic by mid-2026, up from about a fifth a year before. Pay-per-crawl schemes, where sites charge AI companies for content access, are in beta. Every one of those systems needs the same primitive: a way to know that a request claiming to be from OpenAI actually is.

User-agent strings cannot carry that weight. They were designed in 1993 to describe browsers, not to authenticate billion-dollar companies. The OpenAI File Downloader string working as a skeleton key is just the oldest lesson on the web re-learning itself: whatever field the client controls, the client will eventually abuse.

For site owners, the actionable version is short. Whitelisting AI crawlers is now a traffic strategy, and like every traffic strategy it has a cost. The cost used to be “your content trains someone’s model.” In 2026 it is also “anyone can claim to be that model.” Verify who you are letting in, or accept that the VIP lane has no bouncer.

---

This is the same trust gap our agent readiness audit checks from the other side: when we audit a site as an AI agent would, we read its robots.txt and llms.txt as signals — and what an agent can read, a spoofer can read too. If your AI policy and your bot defenses disagree with each other, that gap is measurable.

FAQ

Is spoofing an AI user-agent illegal?

Almost never criminal on its own, but it routinely violates terms of service and computer-fraud statutes depending on jurisdiction and what the scraper does with the access. LinkedIn in particular has a long litigation history around unauthorized scraping. The practical risk is IP bans, account termination, and building a business on data that can be legally yanked away.

Does this mean I should block AI crawlers?

No. Blocking and allowing are both decisions you should implement with verification instead of trust. Blocking GPTBot in robots.txt stops the honest crawler and does nothing to the spoofer. Allowing it opens the citation channel and the side door. Neither choice is wrong; making either choice with UA-string logic alone is.

How do I know if a GPTBot request is real?

Check the IP. OpenAI publishes its crawler ranges and legitimate requests reverse-resolve to openai.com domains; the same applies to Anthropic's ClaudeBot and Google's Google-Extended. About ten lines of edge middleware can compare against those ranges before granting any AI-specific treatment.

Is Cloudflare's signed-bots program the fix?

It is the best available answer for sites on Cloudflare that enable it. The open question is coverage: signing requires bot operators to cooperate, and spoofing stops being free only when every origin that grants AI privileges verifies signatures. A mixed web — some sites verifying, most not — keeps the cheap attack alive.

I run a small site. Nobody is spoofing bots to reach it, right?

Almost certainly true today. The pattern matters for a different reason: the same identity-and-trust logic will govern the next wave of AI shopping and research agents. Build the verification habit now while the stakes are low, and it will already be there when they are not.