The short answer
robots.txt tells search engine crawlers which parts of a site they may fetch. It does not keep a page out of search results, and it is not a security tool. Its main risk is a single line that blocks pages you want found.
What the file is
robots.txt is a small text file at the root of a website, at an address such as yoursite.com/robots.txt. It gives instructions to crawlers, the automated programs search engines use to fetch pages. Each instruction says which crawler it applies to and which paths it may or may not fetch.
Well-behaved crawlers, including Google's, read the file before fetching pages and follow it. Others may ignore it entirely.
What it controls
robots.txt controls crawling, meaning whether a crawler fetches an address. It is useful for keeping crawlers away from areas that waste their time or should not be fetched at all, such as internal search results, admin areas, shopping basket pages or endless filtered versions of the same listing.
It can also point crawlers to the XML sitemap, which is a sensible line to include. Our answer on what an XML sitemap is covers what that file should contain.
What it does not control
This is where most confusion sits:
- It does not remove pages from search. If other sites link to a blocked page, Google can still list the address in results, usually without a description, because it knows the page exists but cannot read it.
- It does not keep anything private. The file itself is public, so listing a secret area in it tells everyone where to look. Private content needs a login.
- It does not override noindex. In fact, blocking a page in robots.txt stops Google from seeing a noindex tag on it, which can keep an unwanted page in results longer.
If the goal is to keep a page out of search results, the right tool is a noindex tag on a page Google is allowed to crawl. Google's own introduction to robots.txt makes the same distinction.
The mistakes that cost traffic
The most damaging error is a line that blocks the whole site, often left over from a staging copy. It looks like this: a line naming all crawlers, followed by a line disallowing the root path. One forgotten line can keep an entire new site out of Google.
Other common problems are blocking the folders that hold the site's styles and scripts, which stops Google from rendering pages properly, and blocking a section with a pattern that also catches pages you wanted crawled.
How to check yours
Open your robots.txt address in a browser and read it. Then, in Search Console, use URL inspection on a few important pages and check that crawling is allowed. The Page indexing report will also show any pages under Blocked by robots.txt. If an important page appears there, the file needs changing. Our SEO audit includes this check as a first step.