The robots Meta Tag (noindex)
The definitive way to keep a page out of the search index is the <meta name="robots" content="noindex"> tag in the HTML head. Unlike a canonical tag (which is a strong hint), a noindex tag is a strict directive. Note that long-term use of noindex, follow will eventually be treated as noindex, nofollow by Google, as it stops crawling pages it knows it shouldn't index.
Canonicalization
The <link rel="canonical" href="..."> tag consolidates indexing signals for duplicate or highly similar content. It passes link equity from the duplicate page to the master version. Canonicals must be absolute URLs, must point to a 200 OK page, and should be self-referencing on the master version to prevent hijacking.
X-Robots-Tag HTTP Header
HTML meta tags only work for HTML documents. To control the indexing of non-HTML assets—like PDFs, images, or API JSON endpoints—you must use the HTTP response header. Adding X-Robots-Tag: noindex at the server level (via Apache/Nginx or a CDN worker) is the only reliable way to keep these file types out of the SERPs.
The Indexing API
Google's Indexing API allows site owners to directly notify Google when pages are added or removed, prompting near-instant crawling. However, Google strictly limits its intended use cases to short-lived content like job postings (JobPosting schema) or live broadcast videos. Misusing the Indexing API to push standard articles or product pages is a violation of guidelines and often results in the API requests being ignored.