ChatGPT, Claude, Perplexity, Gemini Fetch Sites
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- ChatGPT fetched pages from five Azure IP ranges (23.98.x.x, 20.215.x.x, 40.67.x.x, 51.8.x.x, 51.107.x.x) in a single burst, pulling several candidate URLs before deciding which to cite.
- Claude always requested /robots.txt first from Anthropic‑owned IPs (216.73.216.0/24) before fetching the target page, following redirects and normalizing URLs.
- Perplexity made its request without an Accept header, unlike ChatGPT’s Chrome‑style Accept string and Claude’s wildcard Accept.
- Gemini retrieved the page with a distinct user‑agent signal but did not negotiate for text/markdown content.
- OpenAI’s documentation confirms the multi‑IP fetch behavior, meaning rate‑limiting on a single IP will under‑count ChatGPT traffic.
Why it matters: Site operators lose when they under‑estimate AI traffic; ChatGPT’s multi‑IP bursts can overwhelm per‑IP rate limits, forcing them to redesign throttling and risking higher bandwidth costs.
