When GPTBot and ClaudeBot Ate My Quota: How I Cut AI Bot Traffic by 90% on My Free Site (and Why I Ended Up Paying USD 5 Anyway)

One Saturday in October, at 9:49 in the morning (Pacific Time), I got an email from Cloudflare: my account had already used 75% of the 100,000 daily requests on the Workers free tier. There were still more than seven hours left until the reset. At that rate, my site was going to go down by mid-afternoon.

The site was Ultimate Passport Guide: a visa requirements guide with 39,402 passport-and-destination combinations across six languages. Roughly 236,000 pages. It had only been online for a couple of days. It didn't have enough traffic to burn through 75,000 visits before lunch.

My visitors weren't human.

How I Figured It Out

First rule: don't guess. I asked my bot team to check in real time who was coming in. In a one-minute sample, there were 138 invocations, all of them hitting passport-destination pair pages: 94 from GPTBot (OpenAI's crawler) and 44 from ClaudeBot (Anthropic's). Not a single human. Neither Googlebot nor Bingbot. The pace had been steady since the previous day: around 150 invocations per minute.

The technical detail matters. Every pair page is built on the fly by a Cloudflare Worker. On the free tier, every time a bot requests a page, it counts as a Worker invocation. With 236,000 pages and two tireless crawlers, the math just doesn't work out.

It's like opening a store with a turnstile that only lets 100,000 people through a day, only to discover two tour buses have been driving in circles outside your front door all morning.

The Hard Decision

The easy way out was to pay. The Workers Paid plan costs USD 5 a month and includes 10 million requests a month. But the site was brand new, and my rule was clear: no new spending without proven necessity. I also didn't want the opposite solution—blocking all AI bots forever. If someone asks ChatGPT or Claude what visa they need, I want my site to be the answer.

So the decision was this: don't kick the crawlers out, put them on a schedule.

What We Did (for Free)

Three layers, in this order:

1. A scheduled 429. The Worker started replying with a 429 Too Many Requests status and a Retry-After header to GPTBot, ClaudeBot, and other known AI crawlers (CCBot, Bytespider, Amazonbot, among others). For the rest of that day, until the quota reset. Starting the next day, they were only allowed in during minutes 0 through 19 of every hour. The other 40 minutes, the store is closed for them.
2. Never touch what I actually want. Googlebot, Bingbot, humans, and ChatGPT-User (when a person asks ChatGPT to open a specific page) always get through. And robots.txt and llms.txt always respond to anyone. In robots.txt, we added a Crawl-delay: 10 for those crawlers as a courtesy.
3. A bouncer before the door. A 429 from the Worker slows the bot down, but the Worker still runs and the invocation still counts. That's why, an hour later, we added a rate-limiting rule in the Cloudflare firewall (WAF), which runs before the Worker. On the free tier, it has hard limits: a single rule, a 10-second window, a 10-second block, and it can't filter by user-agent. The rule targets the verified "AI Crawler" bot category and applies only to pair pages: more than 2 requests in 10 seconds from the same IP means a block.

The Result

Within four minutes, invocations dropped from 138–163 per minute to 46, 16, 15, and 8: roughly −90%. Tests with curl gave us what we expected: Googlebot, Bingbot, and a standard browser got 200; GPTBot and ClaudeBot got 429.

That day wrapped up at 82% of the quota, according to the Cloudflare alert. It never hit 100%. Cost: USD 0.

What Broke

Three days later, on a Tuesday at 6:16 in the evening, another bot showed up. It identified itself as ShapBot, came from Google Cloud servers, and Cloudflare classified it as "AI Search" rather than an "AI Crawler." My rule didn't catch it. It hammered random pair pages with about 159,000 requests in 30 minutes.

At 6:39 we blocked it using a free firewall rule (based on its user-agent and IPs), and invocations dropped from 7,000–9,000 per minute down to 13–15. But the damage was already done: Cloudflare's quota day was already at 169,863 requests, well above the 100,000 limit. The next morning at 4:05, my automated watchdog found the site responding with "This website has been temporarily rate limited". At 6:04 it was still the same.

At 6:35 I approved the paid plan. At 6:52 the site was back to returning 200. Since then, I've been paying USD 5 a month.

The Principle: The Free Plan Is a Dam, Not a River

I don't regret starting for free. The three layers worked against the problem we had. What I learned is that on a free plan, you don't decide when you hit the limit—the most aggressive bot you don't know about yet does. A river distributes its flow. A dam, when it fills up, overflows all at once.

Today I separate two questions I used to mix up. The first is technical: how do I stop the bots I don't want without kicking out the ones I do? The second is leadership: how much is it worth to me to keep the site from going down? If the answer is more than USD 5 a month, the decision isn't about engineering. It's common sense, and it's best made before 4 in the morning.

What You Can Do on Your Own Free Site

1. Look before you blame. Check your logs or Cloudflare analytics to see who is visiting you. In my case, it was two named crawlers; in yours, it might be a different one.
2. Serve everything static you possibly can. On Cloudflare, requests to static assets are free and unlimited, while those that trigger a Worker count against the 100,000 daily free tier limit. In my case, that wasn't an option: the free plan supports 20,000 static files per version, and I had over 236,000 pages.
3. Put the bouncer before the door. If you filter inside your code, every blocked request still counts. Use access policies for AI crawlers and agents or a rate-limiting rule, which run before your code does.
4. Use robots.txt as a courtesy, not as a wall. It's a request; bots can ignore it.
5. Don't block what brings you customers. Decide beforehand what to do with search engines, AI crawlers, and agents that open a page because a person asked them to.
6. Turn on the usage alert and decide your price ahead of time. The 75% alert gave me a seven-hour buffer. Knowing beforehand that I'd pay USD 5 rather than let the site go down would have saved me an early morning.

Frequently Asked Questions

What are GPTBot and ClaudeBot?

They are OpenAI's and Anthropic's crawlers. They crawl the web and gather content for their AI models. They identify themselves via their user-agent, and Cloudflare classifies them as verified AI bots.

Does blocking AI crawlers get me kicked out of Google?

No, if you do it right. In my case, Googlebot and Bingbot continued to receive 200 on all pages. Whatever you block or limit should target AI crawlers, not all bots.

Is robots.txt enough?

No. robots.txt is a voluntary request. It works as a courtesy, but if the bot ignores it, or if it's a new one you don't have on your list, your quota keeps dropping.

Why respond with 429 instead of blocking?

A 429 Too Many Requests with Retry-After tells the bot "not now, come back later." You aren't kicking it out forever; you're putting it on a schedule. Warning: if that 429 is served by your Worker, the invocation still counts. That's why you need a layer upstream.

How much does it cost to stop worrying?

On Cloudflare, the Workers Paid plan starts at USD 5 a month with 10 million requests included. I activated it after the site was already down. The best move is to decide ahead of time.