Glossary · technical-seo
What is Robots.txt?
Robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may and may not request. It works by naming a user-agent, such as Googlebot or GPTBot, and listing paths to allow or disallow. It controls crawling, not indexing: a page blocked in robots.txt can still appear in Google if other sites link to it, just without a description. To keep a page out of search results, use a noindex tag and leave the page crawlable so Google can see that tag. One wrong line here can hide an entire site.
What does Robots.txt mean for a business in Pakistan?
A recurring problem on Pakistani sites is a robots.txt left set to Disallow: / from the development stage, so the finished site never appears in Google. Developers block crawling while building, then forget at launch. It is the first file we open when an Islamabad business says its new website does not show up at all.
A worked example
A Rawalpindi school launches a new website, and three months later it still does not appear for its own name. Its robots.txt contains User-agent: * followed by Disallow: /. Changing that to allow crawling, keeping only /admin/ blocked, and adding a Sitemap line pointing to the sitemap fixes it. The homepage appears in results within days.
Terms that go with this one
These come up in the same conversations and are worth reading together.
- CrawlingcrawlingCrawling is a search engine's bot fetching your pages by following links — the same way a person clicks around a site, but automated and far faster.…
- Noindexnoindex tagNoindex is an instruction, placed in a page's robots meta tag or sent as an HTTP header, telling search engines not to include that page in their…
- SitemapsitemapA sitemap is an XML file listing the URLs on your site that you want search engines to know about, usually sitting at /sitemap.xml. It is a…
- AI crawlerAI botAn AI crawler is a bot run by an AI company that visits websites to collect pages, either to train models or to fetch fresh information when a user…
- Crawl budgetcrawl budgetCrawl budget is the number of pages Googlebot is willing and able to crawl on a site in a given period. It depends on how fast and stable the server…
Robots.txt — quick answers
What does Robots.txt mean?
Robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may and may not request. It works by naming a user-agent, such as Googlebot or GPTBot, and listing paths to allow or disallow.
Robots.txt ko Urdu mein kya kehte hain?
روبوٹس ٹی ایکس ٹی — though in practice almost everyone in Pakistan says "robots file". A recurring problem on Pakistani sites is a robots.txt left set to Disallow: / from the development stage, so the finished site never appears in Google.
Why does Robots.txt matter for a business in Pakistan?
A recurring problem on Pakistani sites is a robots.txt left set to Disallow: / from the development stage, so the finished site never appears in Google. Developers block crawling while building, then forget at launch. It is the first file we open when an Islamabad business says its new website does not show up at all.
Checked and updated
Next step
Would you rather someone just handled this?
Understanding the terms is useful. Doing the work every day is the hard part — and that is what we do, from PKR 15,000 a month.
WhatsApp us