Should You Block Anything in Your Cafe Site Robots File?
Short answer: for most cafe websites, the safest default is allowing search engine crawlers and the major named AI crawlers to read your public pages, and blocking only genuine junk, cart pages, account pages, internal search result pages, none of which a visitor or a search engine needs to see. Leave your platform's default robots.txt alone unless you have a specific reason to change it.
This post covers one file, robots.txt, and one decision, what to allow and what to block. It explains what the file actually does and does not do, why blocking a page there does not remove it from Google, and how to think about the specific named AI crawlers that have become part of this decision. Blocking a crawler is a business choice, not a right or wrong answer, and this post aims to give you enough to make that choice deliberately rather than out of fear.
What Robots.txt Actually Does
Robots.txt is a plain text file sitting at the root of your website that tells well behaved crawlers which parts of your site you would prefer they not access. Google's own documentation on robots.txt, checked September 21, 2026, is explicit and worth quoting in substance rather than paraphrasing loosely: a robots.txt rule is a request respected by cooperating crawlers, not an enforcement mechanism, and it is explicitly not a way to keep a page out of Google search results entirely.
That last point trips up a lot of owners. A page blocked in robots.txt can still show up in Google search results, sometimes with a listing that simply lacks a description, if Google finds it linked from elsewhere on the web even without crawling the page's actual content directly. If your real goal is keeping a specific page out of search results entirely, the documented method for that is a noindex directive on the page itself, not a robots.txt block, which is a meaningfully different tool for a meaningfully different job.
It is also worth understanding what robots.txt is not built to do at all. It is not a security measure, and it does not hide a page from anyone who has the actual URL or finds a link to it. A private or sensitive page should never rely on robots.txt as its only protection, since the file itself is publicly readable by anyone who checks it, meaning it can actually advertise the existence of a path you would rather keep unlisted, rather than concealing it.
It helps to think of robots.txt as a note left for polite, cooperating visitors rather than a lock on a door. A well behaved search crawler reads the note and follows it. A crawler that does not bother respecting robots.txt at all simply ignores the note entirely, and the file has no mechanism to force compliance either way. That distinction, a request rather than an enforced rule, is the single most important thing to understand about this file before deciding anything about what to put in it.
Safe Defaults for a Cafe Site
For the large majority of small cafe websites, the safest approach is leaving your platform's default robots.txt file essentially untouched. Wix, Squarespace and Shopify all ship with sensible default rules that already block the genuinely useless paths, cart and checkout flows, account and login pages, internal site search result pages, none of which serve any purpose being crawled or shown in search results.
If you do want to make a deliberate change, block only paths that meet a clear test: would a stranger ever want to find this page through a search engine. A cart page, no. An account settings page, no. Your menu, your homepage, your blog posts, your location page, all clearly yes, and none of these should ever be blocked.
A mistake worth naming specifically, because it is more common than it should be and hides an entire site rather than one page: a stray rule blocking an entire directory or even the whole site, sometimes left over from a development or staging version of a site that never got corrected before launch. This single mistake can quietly keep a whole cafe site out of search results, and it is worth checking your own live robots.txt file directly, not just assuming it matches whatever the platform's documented default is, since a well meaning edit at some point in the past can leave a rule nobody remembers adding.
It is worth adding one more category to the safe default list: blocking duplicate or parameter heavy versions of the same page, the kind of address a shopping cart or filtering system sometimes generates automatically with extra tracking text appended to an otherwise normal URL. Most platform defaults already handle this correctly without any manual input, but it is a reasonable, low risk thing to confirm if you notice unusual looking addresses showing up anywhere in your own site's structure.
What Robots.txt Cannot Fix by Itself
It is worth being blunt about a pattern this project has seen across audited cafe sites generally: robots.txt related notes turn up across the shops.csv audit files at a meaningful rate, checked September 21, 2026, roughly 134 of 1,109 audited shops have auditor notes mentioning robots in some form. Those notes describe a range of situations, from a normal, unremarkable default file to an overly broad block, and they are recorded as general auditor observations rather than a structured, precisely classified count, so treat that number as a lower bound on how often robots.txt comes up as a topic worth a note, not a precise breakdown of how many shops have each specific kind of problem.
What that pattern does suggest is that robots.txt is a file owners rarely look at closely, for better or worse. A default file left alone is usually fine. A file that was edited once, for a reason nobody currently remembers, and never revisited, is the more common source of an actual problem than a file that was never touched at all.
This also means robots.txt is not something a busy owner needs to check constantly. A quick look once, when first setting up a site or when handing site management off to someone new, followed by a periodic glance during a broader yearly review, is enough for the overwhelming majority of small cafe sites. The goal is confirming nothing broke by accident, not treating the file as an ongoing project requiring frequent attention.
The Named AI Crawlers, and What They Are For
Several AI companies publish their own crawler documentation naming specific bots and describing what each one is used for. OpenAI documents GPTBot, used for training data collection, alongside OAI SearchBot and ChatGPT User, which serve live search and browsing features rather than training. Anthropic publishes its own crawler documentation for bots used in similar training and retrieval contexts. Perplexity publishes documentation for its own crawler used to support its search and answer product. Google separately documents Google Extended as a token site owners can use to control whether their content is used for certain AI training purposes, distinct from the crawler Google uses for its main search index.
Each of these crawlers is documented, by its own publisher, to respect robots.txt rules the same way a standard search crawler does, meaning a specific disallow rule aimed at a named crawler's specific token will generally be honored by that crawler, assuming it is implemented correctly and the crawler continues to behave as documented at the time you read this. Crawler names, tokens and behaviors are the kind of detail vendors update periodically, so this list is a snapshot as of the date checked, not a permanent reference, and it is worth a quick check of each vendor's current documentation again before making a final decision.
The decision about whether to allow or block any specific one of these crawlers is a business choice, not a technical requirement. Allowing a crawler used for AI search and answer features means your public content can potentially be read and referenced by that system. Blocking it means that specific system will not read your site, for whatever reason you have decided matters to your business, neither choice is inherently correct for every cafe.
What Hillcane's Own Site Actually Saw
As one small, dated case, hillcane.co, launched September 9, 2026, recorded roughly 2,000 hits identified as AI bot traffic out of about 7,738 total hits across its first twelve days. That is one new site's early bot traffic pattern, a small sample over a short window, not a claim about what any other cafe site should expect to see in a comparable period.
What that case does illustrate concretely is that AI bot traffic is not a hypothetical or rare category for a new site, it showed up quickly and made up a meaningful share of total traffic from the very first days the site existed. Whether that fact argues for allowing or blocking any specific crawler depends entirely on what a given business wants from that traffic, documented citation potential from an AI search product being read as a plausible upside, unwanted resource use or a preference to opt out of AI training being a plausible reason to restrict it. This post does not take a position on which choice is correct for your specific cafe, since that is a business decision resting on your own priorities, not a technical one this data alone can settle.
Reading Your Own File Line by Line
You can read your own site's robots.txt directly by adding slash robots dot txt to the end of your homepage address in a browser. What you should expect to see, for a normal, healthy default file, is a short set of rules disallowing a small number of specific paths, cart, account, internal search, while leaving everything else open by omission.
Read each line and ask the same question of every rule you find: does this block something a stranger would ever want to find through search, or does it block something genuinely internal that serves no purpose being crawled or indexed. Anything that fails that test, a rule blocking your blog, your menu, your homepage, or your entire site, is worth investigating immediately, since it likely represents an accident rather than a deliberate choice.
The One Mistake That Hides a Whole Site
If this post leaves you with only one thing to check, make it this: confirm that nothing in your robots.txt file disallows your entire site or a directory containing your main pages. A single overly broad rule, sometimes just a leftover disallow slash from a staging environment that was never removed before launch, is the one mistake capable of quietly keeping an entire cafe website out of search results while everything else about the site looks completely normal.
This single check takes under a minute and catches the one serious risk hiding in an otherwise low stakes file. Everything else described in this post, which specific AI crawlers to allow or block, whether to leave the platform default entirely alone, is a matter of preference and business judgment. This one is worth treating as a hard requirement: whatever else your robots.txt does, it should never accidentally disallow the pages you want people and search engines to find.
If you find that mistake on your own site right now, correcting it is usually a simple edit, removing or narrowing the overly broad rule so it only covers the genuine junk paths it was meant to, cart, account, internal search. After making that correction, use Search Console's URL Inspection tool to confirm your key pages are recognized as available for indexing again, closing the loop on a mistake that, left alone, could have kept an otherwise healthy cafe site invisible in search results indefinitely.
Related Reading
More from Hillcane on this topic, plus the pages on the site that sit next to the work.
Frequently Asked Questions
Should I block AI bots from my cafe website?
That is a business choice, not a technical requirement, and this post does not take a position on the right answer for your specific cafe. Allowing a documented AI crawler means your public content can potentially be read and referenced by that system's search or answer product. Blocking it opts your site out of that system entirely, which is a reasonable choice if that is what you prefer.
What is robots.txt?
Robots.txt is a plain text file at the root of your website that tells cooperating crawlers which parts of your site you would prefer they not access. Google's own documentation is explicit that it is a request, not an enforcement mechanism, and that it is not a way to keep a page out of Google search results entirely, which is a common misunderstanding.
Does blocking a page in robots.txt remove it from Google search results?
No, and this is one of the most common misunderstandings about the file. A page blocked in robots.txt can still appear in search results, sometimes with a listing lacking a description, if Google finds it linked from elsewhere on the web. The documented method for keeping a page out of search results is a noindex directive on the page itself, a different tool entirely.
What should a cafe site block in robots.txt?
Genuine junk only: cart and checkout flows, account and login pages, and internal site search result pages, none of which a stranger would look for through a search engine. Your menu, homepage, blog posts and location pages should never be blocked. Wix, Squarespace and Shopify already handle this by default. If the real goal is keeping a page out of Google, Search Central documents a noindex directive for that, not robots.txt.
What is the most dangerous robots.txt mistake?
A single overly broad rule disallowing an entire directory or the whole site, sometimes a leftover from a staging or development version that was never removed before launch. This one mistake can quietly keep an entire cafe website out of search results while everything else about the site looks completely normal, and it is worth checking directly rather than assuming your file matches the platform default.
How many audited cafes have robots.txt issues?
Across the census, checked September 21, 2026, roughly 134 of 1,109 audited shops have auditor notes mentioning robots in some form, recorded as general observations covering a range of situations rather than a precisely classified count. This should be read as a lower bound on how often robots.txt comes up as a topic worth noting, not an exact breakdown of specific problem types.
What AI crawlers exist, and do they respect robots.txt?
OpenAI documents GPTBot, OAI SearchBot and ChatGPT User for different purposes, Anthropic and Perplexity each publish their own crawler documentation, and Google documents Google Extended as a separate AI training control token distinct from its main search crawler. Each is documented by its publisher to respect a correctly implemented robots.txt disallow rule aimed at its specific token.
How much AI bot traffic did hillcane.co see early on?
As one small, dated case, hillcane.co, launched September 9, 2026, recorded roughly 2,000 hits identified as AI bot traffic out of about 7,738 total hits in its first twelve days. This describes one new site's early bot traffic over a short window, not a claim about what any other cafe site should expect to see in a similar period.
Should I leave my platform's default robots.txt alone?
For most small cafe websites, yes. Wix, Squarespace and Shopify ship with sensible defaults that block cart, account and internal search paths while leaving everything else open. Still read the live file once by adding slash robots dot txt to your homepage address, because a leftover rule from a staging version can hide the whole site. Change it only for a deliberate reason, such as opting out of a named AI crawler.
Can robots.txt protect private or sensitive information?
No, and it should never be relied on for that purpose. The file itself is publicly readable by anyone who checks it, which means listing a sensitive path in robots.txt can advertise that the path exists rather than hiding it. A private or sensitive page needs real access controls, not a robots.txt entry, to be protected.
Work with Hillcane
Robots.txt does not remove a page from Google, and blocking every AI crawler you have heard of does not automatically protect anything. Here are safe defaults for one file, one decision.
If you want a second look at your cafe site's robots.txt file, reach Hillcane at hillcane.co/contact or call (256) 384-2449.
Comments