Sitecore Robots.txt Template

Sitecore is another content management system, whose 38% of brands earn more than $1 billion annually. Sitecore isn't a CMS I've had too much experience in, but I did find the below robots.txt on Andrew Burns' blog. I performed a crawl on a couple of Sitecore sites and did find that Andew's file made a…

Edwin Romero

December 1, 2020

Sitecore is another content management system, whose 38% of brands earn more than $1 billion annually.

Sitecore isn’t a CMS I’ve had too much experience in, but I did find the below robots.txt on Andrew Burns’ blog. I performed a crawl on a couple of Sitecore sites and did find that Andew’s file made a ton of sense. See his file below.

User-agent: *
Disallow: /App_Browsers/
Disallow: /App_config/
Disallow: /App_Data/
Disallow: /sitecore
Disallow: /Sitecore
Disallow: /sitecore_files/
Disallow: /sitecore modules/
Disallow: /temp/
Disallow: /upload/
Disallow: /xsl/

Sitemap: http://www.example.com/sitemap.xml

Edwin Romero

Edwin Romero is an SEO consultant specializing in website migrations. He has 10 years of agency experience & 1 as an independent SEO consultant. He's worked with Fortunate 500 and mid-size e-commerce brands and has supported over 200 site migrations. Edwin loves knowledge-sharing and lending an ear.

All author posts