Robots File: Do I Need one for My Site?

AI, SE Bots and Robots.txt

With artificial intelligence systems (ChatGPT, Claude, Copilot, Gemini, Jasper, etc.) does my website needs a robots.txt file? Not exactly ; that document is not specifically designed to deal with AI. However it is very useful to manage and control how search engine (SE) bots interact with your website. Its text content guides the web crawlers on which parts of your site are allowed to be visited and indexed, which can help improve your site’s SEO and performance.

Table of Contents

• • AI, SE Bots and Your Site
  • Robots.txt File Introduction
  • Virtual vs Physical
  • How to Create (and Edit) one
  • Any Block Instructions?
  • Any Crawl Extra Instructions?
  • Any Content Examples?
  • Additional Information

Introduction to the Robots.txt

Virtual vs Physical Robots.txt

WordPress comes with a built-in feature that generates a robots.txt file dynamically – which isn’t a physical file stored on your server. This means that WP automatically creates a virtual file upon a crawl request from a search engine bot.

The default document generated by WordPress is very basic. Its text content tells all bots to crawl all of your website except one folder. It looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Once you create a physical one in your site’s root folder, you can customise it to your liking and it will override the virtual one that WP had generated.

How to Create (and Edit) a Robots.txt File

1. Via a Plugin
   You can create and access your robots.txt file via plugins such as All in One SEO, Rank Math SEO and Yoast SEO. This method is more accessible for site owners who may not be as tech-savvy or who prefer a centralised approach to their site’s optimisation for SEs.
2. Manually, on Your PC, laptop
   1. Create a plain text file using any text editor such as Notepad.
   2. Name it “robots.txt” and save it.
   3. Upload it to the root folder of your site (public_html) using FTP, cPanel, hPanel, SG Site Tools, etc.
3. Manually, directly on Your Website Server
   1. Log in to your hosting account’s File Manager to use its file editing feature.
   2. Go to public_html, click “New File”, name it as above and save it.

Any Block Instructions?

Yes, you should consider restricting bots from accessing specific folders:

Disallow: /wp-admin/
Disallow: /wp-content/plugins/

And you should consider restricting bots from crawling WordPress search (box) results, since those pages can create duplicate content and eat up your crawl budget, offering no value for your search rankings.

Disallow: /?s=

Any Crawl Extra Instructions?

Yes, you want to allow bots to access a specific file in a disallowed folder:

Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/

And you should consider specifying the location of your sitemap. This helps SEs to index your site faster and more efficiently.

Sitemap: https://mydomain/sitemap_index.xml

You can also include multiple sitemaps in your robots.txt to help search engines index different content areas on your site. Add lines such as:

Sitemap: https://mydomain/sitemap-pages.xml
Sitemap: https://mydomain/sitemap-posts.xml

Any Robots.txt Content Examples?

1. Visit any website – ex “facebook.com”
2. Add /robots.txt to the URL
3. And look at the result > https://www.facebook.com/robots.txt

Btw, we recommend a short one ; here is ours. And of course, optimising your file is included in our hosting packs. Read Website Running Simplified to lean more.

Additional Information

• Introduction to … – Google Search Central.
• How to write and submit … – GSC.
• Update your … – Google Search Central.
• How Google interprets the … specification – GSC.
• How to use … – Best Practices and Examples – SiteGround Knowledge Base.