Allow or Block AI Crawlers on Your Cafe Site?
You have seen a warning somewhere about AI crawlers hitting your website, and now you are staring at a robots.txt file wondering whether to block something, allow it, or just leave it alone and hope for the best. Most owners never touch this file at all, which means whatever default their platform set is quietly making the decision for them right now.
Short answer: for most independent cafes, the decision is small. The goal is to be described accurately when someone asks an AI assistant about coffee nearby, so the sensible default is to allow the documented search and answer crawlers, and decide separately, on its own terms, about crawlers whose stated purpose is training a model rather than answering a live question.
This post lays out the common bot names and what each vendor's own documentation says its crawler does, the honest trade offs between blocking and allowing, and where to actually make the change on the platforms most cafes use. Nobody, including this post, can promise that allowing a crawler earns a citation, or that blocking one removes a site from training data entirely. Those are the honest limits, and they are worth spelling out before anything else, rather than glossed over to make the decision sound simpler than it is.
The Crawlers, By Name and Stated Purpose
A handful of named crawlers show up repeatedly in this conversation, and each vendor documents its own purpose differently. Google operates crawlers under its common crawlers documentation, including its general search crawler and a separately named crawler, Google Extended, which Google's own documentation describes as a control specifically for AI training uses distinct from search indexing, checked September 21, 2026.
OpenAI publishes an overview of its own crawlers, distinguishing between crawlers used to fetch pages in response to a live user request and crawlers used to gather training data, with separate user agent tokens for each purpose named in its documentation. This distinction matters because a site owner who wants to appear in a live AI answer but does not want to contribute to a general training set can, according to the documentation, treat these two purposes differently in a robots.txt file.
Anthropic and Perplexity each publish their own crawler documentation describing their respective bots and stated purposes, generally following the same pattern of naming a specific user agent token tied to a specific stated purpose, whether that is answering a live query or building a broader index or training set.
The practical takeaway from reading these pages directly is that not every AI related crawler does the same thing, and a blanket rule of allow all or block all treats meaningfully different purposes as if they were identical. Reading the actual vendor page for each bot named in this post, rather than relying on secondhand summaries, is worth the ten minutes it takes before making any change to a robots.txt file.
It is also worth noting that vendor documentation for these crawlers changes over time, sometimes adding a new named token or clarifying a stated purpose. A decision made today about a specific bot name should be treated as current as of the date it was checked, not as a permanent fact, and an owner revisiting this topic later should confirm the token names and stated purposes against the current version of each vendor's page rather than relying on what was true when this post was written.
The Honest Trade Offs
Blocking a training focused crawler can reduce the odds that a site's specific text is used to help train a future model, at least for the vendors that respect robots.txt for that purpose, which is what their own documentation states they do. That is a real, if modest, form of control an owner has over how their content is used going forward.
But blocking broadly, especially blocking a crawler whose stated purpose includes answering live user questions, carries a real cost too: it can mean the shop simply is not described when someone asks an AI assistant about coffee nearby, since a crawler that cannot fetch the page cannot use anything on it to answer a question about the shop. No vendor promises that allowing a crawler results in a citation or a mention, and this post does not claim otherwise. Allowing access is necessary for a chance to be included, not a guarantee of one.
There is no single right answer that fits every cafe. A shop that primarily wants to be found and described accurately when someone searches or asks an assistant about it has a reasonable case for allowing the documented search and answer crawlers by name. A shop that is more concerned about its specific written content, such as original blog writing, being used for model training has a reasonable case for blocking training specific crawlers separately, using the distinct tokens each vendor documents, without necessarily blocking the search and answer crawlers at the same time.
It also helps to think about what content actually lives on a cafe website. For most independent shops, the pages that matter most, hours, location, menu description, story, are pages the owner generally wants seen and described accurately by as many channels as possible, since that description is effectively free advertising when it happens. The calculation shifts somewhat for a shop publishing a large amount of original, distinctive writing it considers a meaningful asset on its own, where the training question deserves more individual weight.
What This Decision Does Not Control
It is worth being clear about what a robots.txt change cannot do. It cannot retroactively remove content that has already been used for training by a model built before the file was changed, since robots.txt only governs future crawling behavior for vendors that choose to respect it. It also cannot force any AI system to mention a shop, since inclusion in an answer depends on many factors beyond simple crawl access, most of which are not published by any vendor.
It also does not replace the more foundational work of having an accurate, complete Google Business Profile and a website that clearly states what the shop is and where it is located. A perfectly configured robots.txt file on a site with thin, generic, or outdated content is not going to produce accurate AI descriptions, because there is little accurate content there to draw from in the first place. Robots.txt settings are a gate, not a substitute for what is on the other side of it.
Where to Actually Change This
How a cafe edits its robots.txt depends entirely on the platform the site is built on, and the honest answer varies by platform rather than being universal. On platforms that expose a direct robots.txt editor or a settings panel for it, an owner or whoever manages the site can typically add or remove specific user agent rules directly, following that platform's own help documentation for the exact menu location, since these interfaces change over time and this post is not going to describe a specific path that may be out of date by the time it is read.
On platforms that do not expose robots.txt for direct editing, the available control is more limited, and an owner in that position should check that platform's own help pages to confirm exactly what, if anything, can be adjusted, rather than assuming a workaround exists. This is not a criticism of any platform; some website builders are simply built to manage this file automatically rather than exposing it for manual editing.
Whatever the platform, the practical first step is the same: fetch the site's current robots.txt directly, by visiting the file's address in a browser, and see what it currently says. A file that says nothing about any AI crawler by name is currently allowing all of them by default, since a crawler not explicitly disallowed is generally permitted. A file that already blocks something may have been set that way by a platform default rather than a deliberate choice, which is worth knowing before assuming the current setting reflects an actual decision.
Making the Decision
A five minute version of this decision looks like this: first, decide whether being described accurately by AI assistants is something the shop wants, which for nearly every independent cafe trying to be found by new customers, the answer is yes. If so, the documented search and answer crawlers named by each major vendor should generally be allowed rather than blocked.
Second, decide separately whether training specific crawlers are a concern, understanding that this is a distinct question from the first one and uses a distinct, separately documented user agent token for most major vendors. There is no wrong answer here; it is a genuine values based decision about how a shop's original writing should be used, and reasonable owners can land in different places.
Third, whatever is decided, write it down somewhere the site's current robots.txt file can be checked against later, since platforms and vendor documentation both change over time, and a decision made once is worth revisiting occasionally rather than assumed to be permanent.
For context on why this matters at all for a small, newly launched site specifically: Hillcane's own site recorded roughly 2,000 AI bot hits out of 7,738 total requests in its first twelve days after launching on September 9, 2026, a single new site's dated observation, not a norm every cafe should expect to match. That case shows AI crawler traffic is a real, measurable share of activity on a new small business site from very early on, which is part of why this decision is worth making deliberately rather than by default.
It is worth adding one more caveat to that case specifically. A single new site's twelve day bot traffic count is shaped by many factors particular to that site, including when it launched, how it was linked to elsewhere, and what content it carried at the time, so this figure should not be read as a prediction for any other specific cafe's own traffic. It is offered here only as evidence that the phenomenon is real and measurable, not as a benchmark to compare against.
A Simple Way to Write the Settings
Once the decision is made, the actual robots.txt entries can stay simple. A shop choosing to allow the documented search and answer crawlers while blocking training specific ones would name each training token explicitly with a disallow rule, leaving every other path open by default, rather than trying to write one broad rule that tries to cover every case at once.
Whatever wording is used, it is worth testing the result in a robots.txt testing tool where one is available, checking that the intended crawlers are shown as allowed and the intended ones are shown as blocked, rather than assuming the syntax was written correctly on the first attempt. A small syntax mistake in this file can unintentionally block far more than intended, so a quick test after any change is a cheap way to avoid an accidental, broader block.
Related Reading
More from Hillcane on this topic, plus the pages on the site that sit next to the work.
Frequently Asked Questions
Should I block AI crawlers on my cafe website?
For most independent cafes, not broadly. If the goal is to be described accurately when someone asks an AI assistant about coffee nearby, allow the documented search and answer crawlers that Google, OpenAI, Anthropic, and Perplexity each name in their own documentation, checked September 21, 2026. Training crawlers, which use separately documented tokens, are a distinct decision to make on its own terms. A robots.txt file that names no AI crawler at all is already allowing all of them by default.
What is the difference between a search crawler and a training crawler?
A search or answer crawler fetches a page in response to a live user question, while a training crawler gathers content to help build a model. OpenAI's own crawler overview draws this line and gives each purpose its own user agent token, and Google separates its general search crawler from the Google Extended control for AI training. Anthropic and Perplexity document their bots in a similar pattern. Because the tokens differ, one robots.txt file can allow live answers while disallowing training.
What does Google Extended control?
According to Google's common crawlers documentation, checked September 21, 2026, Google Extended is a control specifically for AI training uses, kept separate from Google's general search crawling and indexing. Blocking it, by Google's own stated purpose, does not affect whether a site appears in ordinary Google search results. For a cafe that wants to stay findable in search but hold back its original writing from training, a disallow rule for Google Extended alone is the narrow, documented way to do that.
Will blocking training crawlers hurt my search ranking?
This post makes no claim linking robots.txt choices about training crawlers to ranking, because no vendor's documentation states that relationship. Google's own documentation, checked September 21, 2026, describes Google Extended as a training control separate from search indexing, and OpenAI, Anthropic, and Perplexity likewise document training and live answer purposes under different tokens. Search indexing and AI training are documented as separate purposes, so a narrow block on a named training token is a different act from blocking a search crawler.
Does allowing AI crawlers guarantee my cafe gets mentioned in AI answers?
No. No vendor, whether Google, OpenAI, Anthropic, or Perplexity, promises a citation or mention in exchange for allowing its crawler, and this post does not claim otherwise. Allowing access is only necessary for a chance at inclusion, since a crawler that cannot fetch a page cannot use it. Accurate content matters just as much: a perfectly configured robots.txt on a site with thin or outdated hours, location, and menu pages gives an assistant little correct information to draw from. Robots.txt is a gate, not a substitute.
Can every website platform edit robots.txt directly?
No, it depends on the platform. Some website builders expose a direct robots.txt editor or a settings panel where you can add or remove specific user agent rules, while others manage the file automatically and do not allow manual edits. Neither approach is a flaw. Check your own platform's help documentation for the current menu location, since those interfaces change over time, and confirm what can be adjusted before assuming a workaround exists. Whatever you change, test it afterward with a robots.txt testing tool.
How do I check what my site's robots.txt currently says?
Type your domain followed by /robots.txt into a browser and read the rules that come back. If no AI crawler is named there, every one of them is currently allowed by default, since a crawler not explicitly disallowed is generally permitted. If something is already blocked, it may reflect a platform default rather than a deliberate choice, which is worth knowing before you assume the setting was intended. Compare what you see with the named tokens in each vendor's own documentation.
How much AI bot traffic does a small cafe site get?
One dated case: Hillcane's own site recorded roughly 2,000 AI bot hits out of 7,738 total requests in the first twelve days after it launched on September 9, 2026. That shows AI crawler activity is a real, measurable share of traffic on a small new business site from very early on. It is a single site's observation, shaped by its launch timing, links, and content, so it is not a norm or a prediction for any other cafe's traffic.
Is there a single correct decision every cafe should make?
No. Allowing the documented search and answer crawlers makes sense for most cafes that want new customers to find them, since hours, location, and menu pages are content an owner usually wants described widely. The training question is a genuine values based choice, and it weighs more for a shop publishing a lot of original, distinctive writing. Reasonable owners land in different places. Whatever you decide, write it down so you can check your robots.txt against it later.
Should I revisit this decision later?
Yes. Vendor documentation changes, sometimes adding a newly named token or clarifying a stated purpose, and platform robots.txt tools change too, so treat any decision as current only as of the date you checked it. This post's sources from Google, OpenAI, Anthropic, and Perplexity were checked September 21, 2026. Revisit occasionally: reload your robots.txt in a browser, compare it with your written decision, confirm the token names on each vendor's current page, and retest the file after any change.
Work with Hillcane
A warning about AI crawlers pops up. The real decision underneath it is smaller than it looks, and it splits into two separate questions.
If you want help reviewing your site's current robots.txt and deciding what fits your cafe, reach Hillcane at hillcane.co/contact or call (256) 384-2449.
Comments