stagingmate
Data study · Agents

4.1% of Agent Sites Use Robots.txt to Block AI Search Bots

Only 4.1% of 13,303 real estate agent and brokerage websites have robots.txt rules that block AI bots used for search answers, such as ChatGPT search, while 12.7% have rules for AI training bots. Without Squarespace sites, which name training bots by platform default, the training share is 11.5%, still more than twice the 4.4% of other websites.

Analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages, with their robots.txt rules · HTTP Archive October 2026 data, analyzed November 2026

Stagingmate Research
Published Nov 11, 2026 · 8 min read
A closed laptop and a small wooden model house on a desk at dusk
Data source
HTTP Archive, October 2026 run of mobile home pages, with sites grouped by words in their domain name
Collected
October 2026 data, analyzed November 2026
Sample
13,303 real estate agent and brokerage home pages, identified by real estate words in their domain name, against 15,349,591 other home pages; plus 2,430 home builder and 918 property manager sites; robots.txt rules for nine named AI bots

Key findings

  1. 14.1% of 13,303 real estate agent and brokerage websites have robots.txt rules for AI search and browsing bots, against 2.6% of other websites.
  2. 212.7% of agent websites have robots.txt rules for AI training bots, more than twice the 5.9% of other websites.
  3. 398.9% of the 189 agent sites on Squarespace have rules for AI training bots, a platform default, and they make up 11% of all agent sites with such rules.
  4. 4Without Squarespace, 11.5% of agent sites have training bot rules, against 4.4% of other websites without Squarespace.
  5. 5GPTBot is the AI bot agent sites name most often, on 9.2% of agent sites, against 5% of other websites.
  6. 6Agent sites on WordPress rarely name AI bots, 3.8% for training bots, against 16.6% of other and custom-built agent sites.

Summary

A buyer asks ChatGPT which agents in her town specialize in waterfront homes. To answer, the assistant fetches a few local agent websites. One of them has a line in its robots.txt file asking AI search bots to stay out, so that agent's site is not read, and the answer is built from the sites that were.

Robots.txt is the small file at the root of a website that tells automated visitors which pages they may read. AI companies run two kinds of bots: training bots that collect pages for building models, and search and browsing bots that fetch pages to answer a question in real time. Whether to block AI bots is now a real choice for agents, because the second kind decides whether an agent's site can be read when a buyer asks an AI assistant for help. This study measures how often agent websites have robots.txt rules for nine named AI bots.

Few agent sites shut out AI search. 4.1% have rules for AI search and browsing bots, close to the 2.6% of other websites. Training bots are named more often, on 12.7% of agent sites, and part of that comes from one platform: nearly every agent site on Squarespace names training bots by default. Without Squarespace the share is 11.5%, still more than twice the rate for other websites.

What we measured

HTTP Archive is a public data set that loads the home pages of millions of websites every month and records their code, files and the technology they use. We used its October 2026 run of mobile home pages: more than 15 million sites. HTTP Archive covers sites worldwide that real visitors use, so it is not limited to the United States.

A site counts as a real estate agent or brokerage site when its domain name contains realty, realtor, realestate, real-estate, brokerage, homeseller, sellmyhome or homevalue: 13,303 sites. Home builder sites are those whose domain contains homebuilder, builders, customhomes or newhomes, with or without hyphens (2,430 sites), and property manager sites those whose domain contains propertymanagement, propertymgmt, pmgroup or rentalmanagement (918 sites). Every other home page, 15,349,591 sites, is the comparison group.

For each home page, HTTP Archive records the site's robots.txt file. We checked it for rules that name nine AI bots by their user-agent token. The training group covers bots that collect pages for model training, such as GPTBot for ChatGPT, ClaudeBot for Claude, Google-Extended for Gemini, CCBot, which builds an open web archive widely used for training, and Bytespider. The search group covers bots that fetch pages to answer questions or browse on a user's behalf, which are OAI-SearchBot and the ChatGPT browsing bot for ChatGPT, PerplexityBot for Perplexity and Claude-SearchBot for Claude. A site counts when its robots.txt has a rule group for at least one bot in the group.

Platforms come from HTTP Archive's own technology detection. WordPress, Wix and Squarespace are named; two smaller site builders are shown without names, and every other site, including custom-built sites and agent website platforms, is grouped as other or custom. We report the training share with and without Squarespace because Squarespace's standard robots.txt names AI training bots, so its sites count as blocking whether or not the owner chose to.

Limits: robots data shows that a site has rules for a bot, not whether the whole site is blocked. A rule can close the entire site to that bot or only some folders, and this edition does not separate the two. Robots.txt is also a request that a bot may or may not honor, not a lock. The domain rule finds sites with a real estate word in their name and misses agents whose domain is their own name, so it covers only part of all agent websites, and it can include listing portals and other real estate businesses with those words in their name. The data covers one month.

Only 4.1% of agent sites have rules for AI search bots

4.1%

of agent websites have robots.txt rules for AI search and browsing bots

AI search and browsing bots are the ones that fetch a page when someone asks an assistant a question. 4.1% of agent and brokerage websites have robots.txt rules for at least one of them, against 2.6% of other websites. Removing Squarespace barely moves the figure, to 4.2%, because Squarespace's default rules name training bots, not search bots.

Share of home pages with robots.txt rules for AI bots, agent sites against all other sites
Agent sites, AI search bots
4.1%
Other sites, AI search bots
2.6%
Agent sites, AI training bots
12.7%
Other sites, AI training bots
5.9%
Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026

Agent sites name AI bots more often than other websites, mostly training bots.

So about 96 agent sites in 100 have no robots.txt rules for AI search bots at all, and most agents who want to block AI bots for training have not shut out AI search at the same time. 91.1% of agent home pages have a robots.txt file at all, so the low share is not for lack of a file: most files simply do not mention AI bots.

Agent sites name AI training bots twice as often, at 12.7%

12.7%

of agent websites have robots.txt rules for AI training bots

12.7% of agent websites have robots.txt rules for at least one AI training bot, more than twice the 5.9% of other websites. Real estate stands out among related trades too: 6.7% of home builder sites and 3.9% of property manager sites have such rules.

Robots.txt rules for AI bots, by kind of site
Kind of siteSitesAI training botsAI search bots
Real estate agents and brokerages13,30312.7%4.1%
Home builders2,4306.7%2.1%
Property managers9183.9%0.8%
All other websites15,349,5915.9%2.6%

Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026

Training bots take pages to build models and send no visitors back, which is one likely reason they are named more often than search bots. For an agent, the trade-off is different for the two kinds: blocking training bots keeps listing descriptions and blog posts out of future models, while blocking search bots keeps the site out of AI answers today.

Squarespace's default explains 11% of agent sites with these rules

98.9%

of agent sites on Squarespace have rules for AI training bots, a platform default

The 189 agent sites on Squarespace show the effect of a platform default. 98.9% of them have robots.txt rules for AI training bots, because Squarespace's standard file includes them, and they make up 11% of all agent sites with training bot rules. Without Squarespace, the agent share falls from 12.7% to 11.5%, and the share for other websites falls from 5.9% to 4.4%.

Share of home pages with robots.txt rules for AI training bots, with and without Squarespace
Agent sites, all
12.7%
Agent sites, without Squarespace
11.5%
Other sites, all
5.9%
Other sites, without Squarespace
4.4%
Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026

Without Squarespace, agent sites are about two and a half times as likely as other sites to name training bots.

With Squarespace removed, the gap with other websites widens rather than shrinks: 11.5% of agent sites against 4.4% of other sites. So the higher rate on agent sites is not a Squarespace effect. It comes mostly from the other and custom-built group, where 16.6% of agent sites have training bot rules.

GPTBot is the AI bot agent sites name most, at 9.2%

GPTBot, which collects pages for training ChatGPT's models, is named on 9.2% of agent sites, against 5% of other websites. Bytespider follows at 7.5%, then ClaudeBot at 6.3%, CCBot at 5% and Google-Extended, the token that controls Gemini training, at 4.7%.

Share of agent home pages with robots.txt rules for each AI bot
GPTBot (ChatGPT training)
9.2%
ClaudeBot (Claude training)
6.3%
CCBot (open web archive)
5%
Google-Extended (Gemini training)
4.7%
Bytespider (AI training)
7.5%
OAI-SearchBot (ChatGPT search)
2.8%
ChatGPT browsing bot
3.2%
PerplexityBot (Perplexity search)
2.9%
Claude-SearchBot (Claude search)
2.5%
Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026

Training bots lead; each bot behind AI search answers is named on about 3% of agent sites or fewer.

The bots behind AI answers are named least: the ChatGPT browsing bot on 3.2% of agent sites, PerplexityBot on 2.9%, OAI-SearchBot on 2.8% and Claude-SearchBot on 2.5%. Every one of the nine is named more often on agent sites than on other websites, with the widest relative gap for Claude-SearchBot, 2.5% against 0.8%.

WordPress agent sites rarely name AI bots, at 3.8%

Platform shapes the rate. Among the 4,466 agent sites on WordPress, 3.8% have rules for training bots and 1.3% for search bots. On Wix, 0% of 269 sites do. The 8,036 other and custom-built sites, which include agent website platforms, have the highest rates, 16.6% for training bots and 6.1% for search bots.

Agent home pages with robots.txt rules for AI bots, by platform
PlatformSitesAI training botsAI search bots
WordPress4,4663.8%1.3%
Wix2690%0%
Squarespace18998.9%0%
Site builder A2200%0%
Site builder B903.3%2.2%
Other or custom8,03616.6%6.1%

Site builders A and B are two smaller website builders, not named here. Source: Stagingmate analysis of 13,303 real estate agent and brokerage home pages and 15,349,591 other home pages · HTTP Archive, October 2026

On hosted platforms the robots.txt file is usually generated by the platform, so these rates mostly reflect platform settings rather than choices by individual agents. That matters for an agent deciding whether to block AI bots: on many platforms the decision has already been made, in one direction or the other, and it is worth checking the file rather than assuming.

What this means for agents

Check your robots.txt file. It sits at the root of your site, and it takes a minute to see which bots it names. 4.1% of agent sites have rules for AI search bots, and an agent who blocks AI bots of that kind keeps the site out of the pages AI assistants read when a buyer asks for help.

Treat the two kinds of bot separately. Blocking training bots, which 12.7% of agent sites do, keeps your pages out of future models. Blocking search and browsing bots keeps you out of AI answers now. Most agents who want to be found will leave the search bots alone, whatever they decide about training.

What an AI assistant reads on your site is your words and your listing photos, so both should show the homes at their best. Agents who stage their own photos usually start with virtual staging software rather than a per-photo service.

To compare tools side by side, our guide to the best virtual staging software compares self-serve AI apps and editing services on speed, price per image and realism.

Cite and download

Cite this study

Stagingmate Research. (2026). 4.1% of Agent Sites Use Robots.txt to Block AI Search Bots. Stagingmate. https://stagingmate.com/research/agent-websites-in-the-ai-era-who-blocks-ai-crawlers

Use this data

Free to use and republish, as long as you credit Stagingmate Research with a link to this page. Need a local cut or a quote? Email support@stagingmate.com.

↓ Download the data (CSV)

Frequently asked questions

It depends on the bot. Blocking AI search and browsing bots keeps a site out of AI answers; only 4.1% of agent sites have rules for them. Blocking training bots keeps pages out of model training; 12.7% of agent sites have rules for those.

Related studies