Should your law firm block AI crawlers? (and the robots.txt to do it right)

"Should we block AI from scraping our site?" is a fair question, and the instinct behind it is understandable. You've read that AI engines answer questions without sending a click, that they train on other people's writing, that they're built on scraped content. Blocking the crawlers feels like taking a stand. For a publisher whose words are the product, it sometimes is one.

For a law firm trying to be found by potential clients, it's almost always working against yourself. The honest answer is that most firms should not block the crawlers that matter — and the whole decision comes down to one distinction that most "block AI" advice skips right over.

The distinction that decides everything: answer-engine crawlers vs training crawlers

Not all AI crawlers do the same job, and lumping them together is what leads firms to the wrong choice. There are two kinds, and they matter to you for opposite reasons.

Answer-engine crawlers fetch your pages to answer a live question. When someone asks ChatGPT or Perplexity "who's a good estate planning lawyer in Tucson," these are the crawlers that go read the web and come back with an answer that can name you and link you. This is the moment you care about. A crawler you've blocked can't cite you in that answer, full stop — it isn't allowed to read the page it would have quoted.

• GPTBot and OAI-SearchBot — OpenAI's crawlers, feeding ChatGPT and its search.
• ClaudeBot — Anthropic's crawler, feeding Claude.
• PerplexityBot — Perplexity's crawler.
• Google-Extended — Google's control for AI use (Gemini and AI Overviews), separate from ordinary Googlebot. You can rank in Google Search while being closed to this one.

Training-only crawlers collect text in bulk to train models. There's no live answer attached and no citation — your pages become a few drops in a training set, and nothing links back. CCBot (Common Crawl, which many models train on) and Bytespider (ByteDance) are the usual names here.

Here's the trap: a lot of the "block AI" robots.txt templates floating around cover only the training crawlers. They feel protective, and they do nothing about the crawlers that actually cite — you've blocked the bulk-collection bots and left the answer engines untouched, or, worse, the reverse. Cloudflare's own managed template has historically leaned this way, which is part of why so many sites end up with a policy nobody actually chose. If you want to see exactly which bots your site allows and blocks today, line by line, run our AI crawler access checker — it reads your live robots.txt and reports each crawler by name.

So should you block them?

Take the two kinds in turn, because the answer is different for each.

For the answer-engine crawlers: no. If you want to be named when someone asks an engine for a lawyer like you, these have to be allowed in. Blocking them to "protect" your content also removes any chance of being the source an engine quotes, which for a firm trying to get found is the opposite of protection. This is the same trade we lay out in how to check if ChatGPT can read your website: an engine that can't reach your pages can't cite them, and that's worse than being cited without a click.

For the training-only crawlers: this is the one defensible block, and it's a values choice, not a visibility one. Some firms would rather their writing not feed model training on principle, and blocking CCBot and Bytespider costs them nothing in AI answers, because those crawlers were never going to cite anyone. If that's a line you want to hold, hold it with clear eyes: it's a statement about training data, not a lever that changes whether clients find you. It won't move your visibility either direction.

The one thing everyone gets backwards

Letting the citation crawlers in is necessary. It is not sufficient. This is the sentence to tape to your monitor before you spend a weekend on robots.txt: opening the door removes a blocker; it doesn't manufacture a citation.

An open robots.txt gets you eligible. Whether an engine actually names you depends on things no robots.txt can create — answers written so they can be lifted cleanly, an identity consistent enough across the web that an engine can tell which firm you are, and corroboration from sources you don't control. That's the real work, and it's covered in how AI engines decide which law firm to cite. If a vendor is selling you crawler access as the thing that gets you cited, they're overselling a floor as a finish line — the same move behind the llms.txt pitch we take apart in whether llms.txt matters for law firms.

Check what your robots.txt already says before you touch it

Most firms have never read their own robots.txt, which is how the blocks that are there got there: a "privacy" or "security" plugin that turned AI crawlers off without saying so, a hosting default, or a CDN injecting its own policy on your behalf. So the first move isn't to write a new file. It's to see what the current one does.

If competitors keep coming up and you never do, an accidental crawler block is one of the cheaper explanations to rule out — and one of the cheapest to fix if that's what it is. Work through this in order:

1. Read your live robots.txt. Type your domain followed by /robots.txt into a browser and look for any Disallow: / under an AI crawler's User-agent line. Read the live file, not the one in your editor — platforms and CDNs generate their own.
2. Sort what you find into the two kinds. A block on GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, or Google-Extended is costing you AI citations. A block on CCBot or Bytespider is a training-data choice and costs you nothing in answers.
3. Decide the training-crawler question on purpose. Feed the models, or don't — either is fine, as long as it's a decision and not a default you inherited.
4. Keep the answer-engine crawlers allowed unless you have a specific, deliberate reason not to. For almost every firm, that reason doesn't exist.
5. Write the file to match, then verify the live version reflects it after you deploy.

Writing the robots.txt without guessing

You can hand-write a robots.txt — it's plain text, one User-agent block per crawler with an Allow or Disallow under it. But the two things that trip firms up are getting every current bot name right and not accidentally clobbering the rest of the file (a sitemap line, disallowed admin paths). Our AI-crawler robots.txt generator handles both: you pick a stance for the answer engines and a separate stance for the training-only crawlers, and it writes a valid file with every line explained. It runs in your browser, keeps the two decisions separate on purpose, and produces text you paste at your site root.

One caution the tool won't say for you: a site has exactly one robots.txt. If you already have rules — a Sitemap: directive, disallowed /wp-admin/ paths — merge the crawler groups into your existing file rather than pasting over the whole thing. On WordPress an SEO plugin or your host's file manager can edit it. Some hosted builders manage robots.txt for you and won't let you set it by hand, which is its own thing to find out before you plan around it.

And remember it's a request, not a lock. robots.txt asks a well-behaved crawler not to fetch a path, and the major AI crawlers honor it. It doesn't physically stop anything — truly forcing a bot out is server- or firewall-level work. For a firm's purposes, though, the polite sign is what the reputable engines read, and it's the file that decides whether they can cite you.

The honest default

For almost every firm: let the answer-engine crawlers in, make a real decision about the training-only ones, and then go do the work that earns the citation, because access alone won't. Blocking is a legitimate choice only when it's a choice — made on purpose, aimed at the right crawlers, and not left in a config file by a plugin you forgot you installed. To see exactly which crawlers can and can't reach your site and what's keeping you out of AI answers, run the free review, or build the whole thing in deliberately with our AEO service.