Source: I posted to HN, and then my website was flooded with AI BOT traffic, by sam
A developer posted a feed reader to Hacker News and the site slowed down. The cause was not the readers. Two counts over the server logs, one per path and one per client, showed it was mostly bots.
The developer posts sprinklz.io, a feed reader that can push topics down a feed. The post gets a hundred or so upvotes. The site becomes painfully slow.
The developer puts the slowness down to a popular post. It continues, and gets worse several days later.
They do not add up: a good handful of active, logged-in users, and a server under very high load.
Most requests never try to fetch an article. The site is a single-page app, so most requests never ran the JavaScript that renders the page.
Nearly half the traffic comes from one service. It watches sites for changes and summarises them into an AI newsletter, polling many times per minute. It is blocked.
Several others are known crawlers that name themselves: Claude bot, OpenAI bot, Google’s search indexer and more. A few vulnerability scanners probe WordPress paths on a site that is not WordPress.
robots.txt now discourages most AI crawlers and allows search crawlers. Rate limits are added. The developer replays the scanner paths to check nothing leaks, then works on making the site faster.
One log, two counts, two different answers.
Each count answers a different question. Run both before you blame the launch.
| Signal | Lesson | Rule now |
|---|---|---|
| Slow site after a popular post | Attention was the wrong suspect | Check load against real users first |
| Few logged-in users, high load | The load is not people | Count requests per path |
| Most requests skip articles and JS | Clients are not browsers | Count requests per client |
| One client, nearly half the traffic | One poller can swamp a small site | Block it; add rate limits |
I posted to HN, and then my website was flooded with AI bot traffic, by Sammy Straus.
Sources: every fact and hedge on this page (“a hundred or so”, “nearly half”, “several”, “most”) comes from the post above. The post gives no exact request counts.