Skip to main content

Web and SEO terms made simple

Clear definitions. Related terms. Reliable resources. Updated as I learn.

About this glossary

Robots.txt – Glossary

Robots.txt tells crawlers which paths they can request. Use it to manage crawl access, protect staging with appropriate disallows, and avoid rules that block important resources (CSS, JS, images).

Note: robots.txt controls crawling, not indexing. To keep pages out of results, use a noindex directive on the page, and allow crawlers to fetch it.

Further reading

Reference

robots.txt, Google Search Central

How Google interprets robots.txt, writing and updating files, common pitfalls.

https://developers.google.com/search/docs/crawling-indexing/robots/intro

Related terms

Indexing

Indexing is how search engines discover, select, and store your pages so they can appear in results. Ensure correct d...

Noindex and Nofollow

Use noindex to keep a page out of search results after crawlers fetch it. Use nofollow on links to avoid passing link...

Category:
Category
Technical SEO
· Crawl and index control

Last updated:

About this glossary

I’m building this glossary to become a better developer and website strategist, in my ongoing quest to build clearer, better websites. It maps how terms relate, adds practical examples, and highlights best practices so I can communicate more clearly with clients. If it helps others too, great.

It’s a living resource, I aim to cite sources clearly. If you spot errors or omissions, please contact me so I can update it. Drafts may be assisted by AI, I review and fact check before publishing. For details on how I use AI, see AI Use at Webmarks.