Directives and Syntax
The robots.txt file is the first thing a bot checks. Advanced usage goes beyond simple Disallow directives. It includes utilizing wildcards (*) for pattern matching, the end-of-string anchor ($) to block specific file types (e.g., Disallow: /*.pdf$), and targeted Allow directives to override broader blocks for specific, high-value subdirectories.
Crawling vs. Indexing
A fundamental SEO misconception is that robots.txt prevents indexing. It does not. robots.txt prevents crawling. If a URL is blocked in robots.txt, Googlebot will not fetch it. However, if that URL is linked externally, Google may still index the URL based purely on the anchor text, resulting in a search result that says "No information is available for this page". To properly remove a page from the index, you must allow it to be crawled so the bot can read the noindex meta tag.
The Danger of Blocking Resources
Modern Googlebot relies on rendering the page to understand its content. If your robots.txt blocks critical CSS, JavaScript, or API endpoints that build the page's layout or load primary content, the Web Rendering Service will fail to render the page correctly. This often results in mobile-usability errors, layout shifts, or missing content being indexed. You must always allow crawling of necessary rendering assets.