Build a valid robots.txt file for your website. Choose presets for common CMS platforms or customise every rule manually.
# Generating your robots.txt...
robots.txt — accessible at yoursite.com/robots.txt
Don't block CSS and JS: Google needs to crawl your CSS and JavaScript to render and understand your pages. Blocking them can hurt your rankings significantly. Only block them if you have a specific reason.
Include your sitemap URL: Always add a Sitemap directive pointing to your XML sitemap. This helps Google discover all your pages efficiently.
Test before uploading: Use Google Search Console's robots.txt tester to verify your rules before going live. A single typo can accidentally block your entire site.
robots.txt is not private: Your robots.txt file is publicly visible to anyone. Never put sensitive URLs in Disallow rules expecting them to stay private — use proper authentication instead.