robots.txt is a small text file at the root of your website, at yourdomain.lk/robots.txt, that tells search engine crawlers which parts of your site they may crawl. One wrong line can hide your whole site from Google, so it's worth understanding.
What robots.txt does
It controls crawling: whether a bot may request a URL. That's all.
What robots.txt does NOT do
- It doesn't remove pages from Google. A blocked URL can still be indexed if other pages link to it, just without its content. To keep a page out of search results, use a
noindexmeta tag, and don't block it in robots.txt, otherwise Google can't see the noindex. - It isn't security. Anyone can read your robots.txt. Never use it to "hide" private areas; protect those with a login.
The basic syntax
User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://yourbusiness.lk/sitemap.xml
User-agentnames the crawler.*means all crawlers.Disallowlists paths not to crawl.Allowmakes exceptions.Sitemaptells crawlers where your sitemap is.
A sensible robots.txt for WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourbusiness.lk/sitemap_index.xml
A sensible robots.txt for an online store
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?add-to-cart=
Sitemap: https://yourstore.lk/sitemap.xml
Blocking cart, checkout and sorting parameters stops Google wasting crawling on pages that shouldn't rank.
Common mistakes
Blocking the whole site
User-agent: *
Disallow: /
This is often left over from a staging site or a "discourage search engines" setting. It's the most damaging robots.txt mistake. Check yours now.
Blocking CSS and JavaScript
Google renders pages like a browser. If it can't load your CSS and JS, it may think your site isn't mobile-friendly or miss content entirely.
Using robots.txt to remove pages
As above, use noindex instead, or remove the page and return a 404 or 410.
Wrong file location
It must be at the root: yourdomain.lk/robots.txt, not /blog/robots.txt. Each subdomain needs its own.
AI crawlers
You can also allow or block AI crawlers in robots.txt, using their user-agent names, such as GPTBot or Google-Extended. Blocking Google-Extended stops your content being used to train Google's AI models, but doesn't affect normal Google Search. Think carefully before blocking AI crawlers if you want to be cited in AI answers. See generative engine optimisation.
How to test
- Visit yourdomain.lk/robots.txt in a browser
- Check the robots.txt report in Google Search Console
- Use the URL Inspection tool on important pages to confirm they're crawlable
Our free SEO health check checks whether your robots.txt and sitemap exist. Next, read the XML sitemap guide, or see the full technical SEO checklist.