- Data source
- HTTP Archive, October 2026 run of mobile home pages, with sites grouped by words in their domain name
- Collected
- October 2026 data, analyzed November 2026
- Sample
- 13,303 real estate agent and brokerage home pages, identified by real estate words in their domain name, against 15,349,591 other home pages; plus 2,430 home builder and 918 property manager sites; robots.txt rules for nine named AI bots
Key findings
- 14.1% of 13,303 real estate agent and brokerage websites have robots.txt rules for AI search and browsing bots, against 2.6% of other websites.
- 212.7% of agent websites have robots.txt rules for AI training bots, more than twice the 5.9% of other websites.
- 398.9% of the 189 agent sites on Squarespace have rules for AI training bots, a platform default, and they make up 11% of all agent sites with such rules.
- 4Without Squarespace, 11.5% of agent sites have training bot rules, against 4.4% of other websites without Squarespace.
- 5GPTBot is the AI bot agent sites name most often, on 9.2% of agent sites, against 5% of other websites.
- 6Agent sites on WordPress rarely name AI bots, 3.8% for training bots, against 16.6% of other and custom-built agent sites.
Summary
A buyer asks ChatGPT which agents in her town specialize in waterfront homes. To answer, the assistant fetches a few local agent websites. One of them has a line in its robots.txt file asking AI search bots to stay out, so that agent's site is not read, and the answer is built from the sites that were.
Robots.txt is the small file at the root of a website that tells automated visitors which pages they may read. AI companies run two kinds of bots: training bots that collect pages for building models, and search and browsing bots that fetch pages to answer a question in real time. Whether to block AI bots is now a real choice for agents, because the second kind decides whether an agent's site can be read when a buyer asks an AI assistant for help. This study measures how often agent websites have robots.txt rules for nine named AI bots.
Few agent sites shut out AI search. 4.1% have rules for AI search and browsing bots, close to the 2.6% of other websites. Training bots are named more often, on 12.7% of agent sites, and part of that comes from one platform: nearly every agent site on Squarespace names training bots by default. Without Squarespace the share is 11.5%, still more than twice the rate for other websites.
What we measured
HTTP Archive is a public data set that loads the home pages of millions of websites every month and records their code, files and the technology they use. We used its October 2026 run of mobile home pages: more than 15 million sites. HTTP Archive covers sites worldwide that real visitors use, so it is not limited to the United States.
A site counts as a real estate agent or brokerage site when its domain name contains realty, realtor, realestate, real-estate, brokerage, homeseller, sellmyhome or homevalue: 13,303 sites. Home builder sites are those whose domain contains homebuilder, builders, customhomes or newhomes, with or without hyphens (2,430 sites), and property manager sites those whose domain contains propertymanagement, propertymgmt, pmgroup or rentalmanagement (918 sites). Every other home page, 15,349,591 sites, is the comparison group.
For each home page, HTTP Archive records the site's robots.txt file. We checked it for rules that name nine AI bots by their user-agent token. The training group covers bots that collect pages for model training, such as GPTBot for ChatGPT, ClaudeBot for Claude, Google-Extended for Gemini, CCBot, which builds an open web archive widely used for training, and Bytespider. The search group covers bots that fetch pages to answer questions or browse on a user's behalf, which are OAI-SearchBot and the ChatGPT browsing bot for ChatGPT, PerplexityBot for Perplexity and Claude-SearchBot for Claude. A site counts when its robots.txt has a rule group for at least one bot in the group.
Platforms come from HTTP Archive's own technology detection. WordPress, Wix and Squarespace are named; two smaller site builders are shown without names, and every other site, including custom-built sites and agent website platforms, is grouped as other or custom. We report the training share with and without Squarespace because Squarespace's standard robots.txt names AI training bots, so its sites count as blocking whether or not the owner chose to.
Limits: robots data shows that a site has rules for a bot, not whether the whole site is blocked. A rule can close the entire site to that bot or only some folders, and this edition does not separate the two. Robots.txt is also a request that a bot may or may not honor, not a lock. The domain rule finds sites with a real estate word in their name and misses agents whose domain is their own name, so it covers only part of all agent websites, and it can include listing portals and other real estate businesses with those words in their name. The data covers one month.
Only 4.1% of agent sites have rules for AI search bots
of agent websites have robots.txt rules for AI search and browsing bots
AI search and browsing bots are the ones that fetch a page when someone asks an assistant a question. 4.1% of agent and brokerage websites have robots.txt rules for at least one of them, against 2.6% of other websites. Removing Squarespace barely moves the figure, to 4.2%, because Squarespace's default rules name training bots, not search bots.
Agent sites name AI bots more often than other websites, mostly training bots.
So about 96 agent sites in 100 have no robots.txt rules for AI search bots at all, and most agents who want to block AI bots for training have not shut out AI search at the same time. 91.1% of agent home pages have a robots.txt file at all, so the low share is not for lack of a file: most files simply do not mention AI bots.
Agent sites name AI training bots twice as often, at 12.7%
of agent websites have robots.txt rules for AI training bots
12.7% of agent websites have robots.txt rules for at least one AI training bot, more than twice the 5.9% of other websites. Real estate stands out among related trades too: 6.7% of home builder sites and 3.9% of property manager sites have such rules.
| Kind of site | Sites | AI training bots | AI search bots |
|---|---|---|---|
| Real estate agents and brokerages | 13,303 | 12.7% | 4.1% |
| Home builders | 2,430 | 6.7% | 2.1% |
| Property managers | 918 | 3.9% | 0.8% |
| All other websites | 15,349,591 | 5.9% | 2.6% |
Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026
Training bots take pages to build models and send no visitors back, which is one likely reason they are named more often than search bots. For an agent, the trade-off is different for the two kinds: blocking training bots keeps listing descriptions and blog posts out of future models, while blocking search bots keeps the site out of AI answers today.
Squarespace's default explains 11% of agent sites with these rules
of agent sites on Squarespace have rules for AI training bots, a platform default
The 189 agent sites on Squarespace show the effect of a platform default. 98.9% of them have robots.txt rules for AI training bots, because Squarespace's standard file includes them, and they make up 11% of all agent sites with training bot rules. Without Squarespace, the agent share falls from 12.7% to 11.5%, and the share for other websites falls from 5.9% to 4.4%.
Without Squarespace, agent sites are about two and a half times as likely as other sites to name training bots.
With Squarespace removed, the gap with other websites widens rather than shrinks: 11.5% of agent sites against 4.4% of other sites. So the higher rate on agent sites is not a Squarespace effect. It comes mostly from the other and custom-built group, where 16.6% of agent sites have training bot rules.
GPTBot is the AI bot agent sites name most, at 9.2%
GPTBot, which collects pages for training ChatGPT's models, is named on 9.2% of agent sites, against 5% of other websites. Bytespider follows at 7.5%, then ClaudeBot at 6.3%, CCBot at 5% and Google-Extended, the token that controls Gemini training, at 4.7%.
Training bots lead; each bot behind AI search answers is named on about 3% of agent sites or fewer.
The bots behind AI answers are named least: the ChatGPT browsing bot on 3.2% of agent sites, PerplexityBot on 2.9%, OAI-SearchBot on 2.8% and Claude-SearchBot on 2.5%. Every one of the nine is named more often on agent sites than on other websites, with the widest relative gap for Claude-SearchBot, 2.5% against 0.8%.
WordPress agent sites rarely name AI bots, at 3.8%
Platform shapes the rate. Among the 4,466 agent sites on WordPress, 3.8% have rules for training bots and 1.3% for search bots. On Wix, 0% of 269 sites do. The 8,036 other and custom-built sites, which include agent website platforms, have the highest rates, 16.6% for training bots and 6.1% for search bots.
| Platform | Sites | AI training bots | AI search bots |
|---|---|---|---|
| WordPress | 4,466 | 3.8% | 1.3% |
| Wix | 269 | 0% | 0% |
| Squarespace | 189 | 98.9% | 0% |
| Site builder A | 220 | 0% | 0% |
| Site builder B | 90 | 3.3% | 2.2% |
| Other or custom | 8,036 | 16.6% | 6.1% |
Site builders A and B are two smaller website builders, not named here. Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026
On hosted platforms the robots.txt file is usually generated by the platform, so these rates mostly reflect platform settings rather than choices by individual agents. That matters for an agent deciding whether to block AI bots: on many platforms the decision has already been made, in one direction or the other, and it is worth checking the file rather than assuming.
What this means for agents
Check your robots.txt file. It sits at the root of your site, and it takes a minute to see which bots it names. 4.1% of agent sites have rules for AI search bots, and an agent who blocks AI bots of that kind keeps the site out of the pages AI assistants read when a buyer asks for help.
Treat the two kinds of bot separately. Blocking training bots, which 12.7% of agent sites do, keeps your pages out of future models. Blocking search and browsing bots keeps you out of AI answers now. Most agents who want to be found will leave the search bots alone, whatever they decide about training.
What an AI assistant reads on your site is your words and your listing photos, so both should show the homes at their best. Agents who stage their own photos usually start with virtual staging software rather than a per-photo service.
To compare tools side by side, our guide to the best virtual staging software compares self-serve AI apps and editing services on speed, price per image and realism.
Cite and download
Stagingmate Research. (2026). 4.1% of Agent Sites Use Robots.txt to Block AI Search Bots. Stagingmate. https://stagingmate.com/research/agent-websites-in-the-ai-era-who-blocks-ai-crawlers
Free to use and republish, as long as you credit Stagingmate Research with a link to this page. Need a local cut or a quote? Email support@stagingmate.com.
Frequently asked questions
It depends on the bot. Blocking AI search and browsing bots keeps a site out of AI answers; only 4.1% of agent sites have rules for them. Blocking training bots keeps pages out of model training; 12.7% of agent sites have rules for those.
9.2% of agent and brokerage websites have robots.txt rules for GPTBot, the ChatGPT training bot, 2.8% for OAI-SearchBot, used for ChatGPT search, and 3.2% for the bot ChatGPT uses when it browses for a user.
Its standard robots.txt names AI training bots. 98.9% of the 189 agent sites on Squarespace have rules for training bots, and none have rules for AI search bots.
Yes. 12.7% of agent sites have rules for AI training bots against 5.9% of other websites, and 11.5% against 4.4% without Squarespace. For AI search bots it is 4.1% against 2.6%.
Not always. A rule can close the entire site to a bot or only some folders, and bots may or may not honor it. This study counts sites with rules for a bot, not full blocks.



