Why
Your code leaves home.
Public
Private
"An agent does not wait for the next training run."
It can scrape and steal the shape of your product today.
"The next training run is going to consume your value anyway."
Keep work where crawlers cannot reach it.
Agents and live crawling
Cloudflare
Perplexity is using stealth undeclared crawlers to evade no-crawl directives
When declared bots get blocked, undeclared crawlers keep fetching pages anyway. Live scraping, not a quiet training archive.
↗
Ars Technica
AI site Perplexity uses stealth tactics to flout no-crawl edicts
Rotated IPs, spoofed browser identities, and millions of requests a day across tens of thousands of domains.
↗
The Verge
OpenAI's Operator agent can do things on the web for you
An agent with its own browser can open pages, click through them, and pull what it needs without waiting for a crawl farm.
↗
Anthropic
Developing a computer use model
Claude can move a cursor, click, type, and use the same apps people use, including browsers that can reach your code.
↗
BleepingComputer
FakeGit uses 7,600 GitHub repos and baiting AI agents
Agents now search public repos and registries for skills and tools on their own. Public code is part of that hunt.
↗
Training data and crawlers
The Verge
The lawsuit that could rewrite the rules of AI copyright
Copilot was trained on public GitHub code, including licensed code. The scale of it triggered a class action against Microsoft, GitHub and OpenAI.
↗
Ars Technica
AI crawlers dominate traffic, forcing blocks on entire countries
Some open source projects now see 97 percent of their traffic coming from AI company bots. The bots spoof user agents and ignore robots.txt.
↗
TechCrunch
Open source devs are fighting AI crawlers with cleverness and vengeance
Maintainers are building proof of work walls and tarpits just to keep scrapers off their infrastructure.
↗
Cloudflare
Cloudflare blocks AI crawlers by default across its network
Scraping got so routine that the biggest infrastructure provider on the web flipped its default to deny.
↗
If your code had value, why did you just do that.....