Robots.Txt: What Is Robots.Txt & Why It Matters for SEO

A robots.txt is a file that tells search engine robots which pages they should and shouldn’t crawl.