Tag: robots txt file

Unlock the power of robots.txt files with insights from the Mc Starters Blog. Explore our articles to learn how to create, optimize, and leverage robots.txt files to control search engine crawlers’ access to your website, empowering you to enhance your site’s visibility and SEO performance effectively.

Four Steps to Create a Robots txt File for Your Website

Most newcomers don’t know how to create a robots txt for the website. Today, We will show you four steps to Create a Robots txt File.

Robots.txt is a simple text file to guide google robots on how to crawl site posts and pages on their website. The robots.txt file is also known as the robots exclusion protocol.

You Might Also Like: Free Keyword Research Tools

Four Steps to Create a Robots txt File

Step 1: Copy the below lines of snippet

User-Agent: *
Allow: /?display=wide
Disallow: /readme.html
Disallow: /refer/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /cgi-bin/
Disallow: /trackback/
Disallow: /comments/
Disallow: /xmlrpc.php
Allow: /wp-admin/admin-ajax.php

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /
 
User-agent: Googlebot-Mobile
Allow: /

Sitemap: https://example.com/post-sitemap.xml

Step 2: Open notepad and paste the above code and save the file with the name “robots.txt”. Last, you must just change the sitemap URL according to your own website sitemap URL.

Create a Robots txt File

Step 3: Install and Activate File Manager plugin

Install and Activate File Manager

Step 4: Now you should go to file manager and upload the robots.txt file into the root directory. Make sure this is placed outside the “wp-content” if you are using WordPress.

File Manager Upload Robots txt File
upload robots.txt file into root directory

Watch Video

4 Steps to Create a Robots txt File for Your Website

How do I Create a Robots txt File?

Step 1: Copy the below lines of snippet
Step 2: Open notepad and paste
Step 3: Upload robots.txt file into the root directory

Is robot txt necessary?

No, a robots.txt file is not required for a website.

You Might Also Like Cheap WordPress Design services for business.

Get Professional Websites