{"id":22107,"date":"2022-01-15T06:00:00","date_gmt":"2022-01-15T01:00:00","guid":{"rendered":"https:\/\/mcstarters.com\/blog\/?p=22107"},"modified":"2025-07-12T19:41:24","modified_gmt":"2025-07-12T14:41:24","slug":"create-a-robots-txt-file","status":"publish","type":"post","link":"https:\/\/mcstarters.com\/blog\/create-a-robots-txt-file\/","title":{"rendered":"Four Steps to Create a Robots txt File for Your Website"},"content":{"rendered":"\n<p>Most newcomers don&#8217;t know how to create a <a href=\"https:\/\/developers.google.com\/search\/docs\/advanced\/robots\/intro\" target=\"_blank\" data-type=\"URL\" data-id=\"https:\/\/developers.google.com\/search\/docs\/advanced\/robots\/intro\" rel=\"noreferrer noopener nofollow\">robots txt<\/a> for the website. Today, We will show you <strong>four steps to Create a Robots txt File<\/strong>.<\/p>\n\n\n\n<p><em>Robots<\/em>.<em>txt<\/em>&nbsp;is a simple text file to guide google&nbsp;<em>robots<\/em> on how to crawl site posts and pages on their website. The&nbsp;robots.txt&nbsp;file&nbsp;is also known as the robots exclusion protocol.<\/p>\n\n\n\n<p class=\"bg-light\">You Might Also Like: <a href=\"https:\/\/mcstarters.com\/blog\/free-keyword-research-tools\/\" data-type=\"URL\" data-id=\"https:\/\/mcstarters.com\/blog\/free-keyword-research-tools\/\"><strong>Free Keyword Research Tools<\/strong><\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Four Steps to Create a Robots txt File<\/h2>\n\n\n\n<p class=\"step\"><strong>Step 1: <\/strong>Copy the below lines of snippet<\/p>\n\n\n\n<pre class=\"wp-block-code bg-code\"><code>User-Agent: *\nAllow: \/?display=wide\nDisallow: \/readme.html\nDisallow: \/refer\/\nDisallow: \/wp-admin\/\nDisallow: \/wp-login.php\nDisallow: \/cgi-bin\/\nDisallow: \/trackback\/\nDisallow: \/comments\/\nDisallow: \/xmlrpc.php\nAllow: \/wp-admin\/admin-ajax.php\n\nUser-agent: NinjaBot\nAllow: \/\n\nUser-agent: Mediapartners-Google*\nAllow: \/\n\nUser-agent: Googlebot-Image\nAllow: \/wp-content\/uploads\/\n\nUser-agent: Adsbot-Google\nAllow: \/\n \nUser-agent: Googlebot-Mobile\nAllow: \/\n\nSitemap: https:\/\/example.com\/post-sitemap.xml<\/code><\/pre>\n\n\n\n<p class=\"step\"><strong>Step 2: <\/strong>Open notepad and paste the above code and save the file with the name &#8220;robots.txt&#8221;. Last, you must just change the <strong>sitemap URL<\/strong> according to your own website sitemap URL.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"700\" height=\"372\" src=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/Create-a-Robots.txt-File.webp\" alt=\"Create a Robots txt File\" class=\"wp-image-22117\" title=\"\"><\/figure>\n\n\n\n<p class=\"step\"><strong>Step 3: <\/strong>Install and Activate File Manager plugin<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1328\" height=\"438\" src=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/Install-and-Activate-File-Manager.webp\" alt=\"Install and Activate File Manager\" class=\"wp-image-22283\" title=\"\" srcset=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/Install-and-Activate-File-Manager.webp 1328w, https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/Install-and-Activate-File-Manager-768x253.webp 768w\" sizes=\"auto, (max-width: 1328px) 100vw, 1328px\" \/><\/figure>\n\n\n\n<p class=\"step\"><strong>Step 4:<\/strong> Now you should go to file manager and upload the robots.txt file into the root directory. Make sure this is placed outside the &#8220;wp-content&#8221; if you are using WordPress.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1350\" height=\"614\" src=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/File-Manager-Upload-Rebots-txt-File.webp\" alt=\"File Manager Upload Robots txt File\" class=\"wp-image-22282\" title=\"\" srcset=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/File-Manager-Upload-Rebots-txt-File.webp 1350w, https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/File-Manager-Upload-Rebots-txt-File-768x349.webp 768w\" sizes=\"auto, (max-width: 1350px) 100vw, 1350px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"93\" src=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/upload-robots.txt-file-into-root-directory.webp\" alt=\"\" class=\"wp-image-22124\" title=\"\" srcset=\"https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/upload-robots.txt-file-into-root-directory.webp 1024w, https:\/\/mcstarters.com\/blog\/wp-content\/uploads\/2022\/01\/upload-robots.txt-file-into-root-directory-768x70.webp 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Watch Video<\/h2>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"4 Steps to Create a Robots txt File for Your Website\" width=\"720\" height=\"405\" src=\"https:\/\/www.youtube.com\/embed\/EDKK12DZXVE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n<div id=\"rank-math-faq\" class=\"rank-math-block\">\n<div class=\"rank-math-list \">\n<div id=\"faq-question-1641926458548\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">How do I Create a Robots txt File?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p><strong>Step 1:\u00a0<\/strong>Copy the below lines of snippet<br \/><strong>Step 2:\u00a0<\/strong>Open notepad and paste<br \/><strong>Step 3:<\/strong>\u00a0Upload\u00a0<strong>robots.txt file<\/strong>\u00a0into the root directory<\/p>\n\n<\/div>\n<\/div>\n<div id=\"faq-question-1641926542531\" class=\"rank-math-list-item\">\n<h3 class=\"rank-math-question \">Is robot txt necessary?<\/h3>\n<div class=\"rank-math-answer \">\n\n<p><strong>No<\/strong>, a\u00a0<strong>robots.txt file <\/strong>is not required for a website.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<p class=\"bg-blue\">You Might Also Like <strong><a href=\"https:\/\/mcstarters.com\/services\/wordpress-web-design-services\/\" target=\"_blank\" data-type=\"URL\" data-id=\"https:\/\/mcstarters.com\/services\/wordpress-web-design-services\/\" rel=\"noreferrer noopener\">Cheap WordPress Design<\/a> <\/strong>services for business.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Most newcomers don&#8217;t know how to create a robots txt&#8230;<\/p>\n","protected":false},"author":5,"featured_media":22323,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","footnotes":""},"categories":[8],"tags":[465,212],"class_list":["post-22107","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-seo","tag-robots-txt-file","tag-website-builder"],"_links":{"self":[{"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/posts\/22107","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/comments?post=22107"}],"version-history":[{"count":1,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/posts\/22107\/revisions"}],"predecessor-version":[{"id":37372,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/posts\/22107\/revisions\/37372"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/media\/22323"}],"wp:attachment":[{"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/media?parent=22107"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/categories?post=22107"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mcstarters.com\/blog\/wp-json\/wp\/v2\/tags?post=22107"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}