{"id":2229,"date":"2021-06-25T11:24:09","date_gmt":"2021-06-25T08:24:09","guid":{"rendered":"https:\/\/www.scalahosting.com\/kb\/?p=2229"},"modified":"2025-11-27T13:55:44","modified_gmt":"2025-11-27T11:55:44","slug":"how-to-use-the-robots-txt-file-to-improve-the-way-search-bots-crawl-your-website","status":"publish","type":"post","link":"https:\/\/www.scalahosting.com\/kb\/how-to-use-the-robots-txt-file-to-improve-the-way-search-bots-crawl-your-website\/","title":{"rendered":"Create robots.txt File Best Practices"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Organic searches have become an integral part of our daily lives. Recent data shows that close to <\/span><b>30 percent of global web traffic <\/b><span style=\"font-weight: 400;\">is from online searches.\u00a0\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Search engines crawl and index billions of web content every day, ranking them in search results according to relevancy\u2014<\/span><b>how relevant they are to search queries\u2014making them available to the public.<\/b><\/p>\n<p><span style=\"font-weight: 400;\">You could set up directives on<\/span><b> how you wish search engines to crawl and show your web content to the public using the robot.txt file<\/b><span style=\"font-weight: 400;\">.\u00a0 This article takes you through everything you need to know about the robots.txt file.<\/span><\/p>\n<h2><b>Understanding the Robot.txt File<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Search indexation begins with a simple search engine crawl. The robots.txt file, also known as the <\/span><b>Robots Exclusion Protocol<\/b><span style=\"font-weight: 400;\">, instructs search bots on crawling a website\u2014where and where not to go. Users often use the file to <\/span><b>specify the pages search engines shouldn\u2019t crawl<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">When a<\/span><b> search engine discovers a website through links or a sitemap<\/b><span style=\"font-weight: 400;\">, it opens the website\u2019s robots.txt file to learn the page to crawl and the ones it shouldn\u2019t.\u00a0 The crawler caches the robots.txt file to save it from opening it each time it visits the website. The cached file <\/span><b>auto-refreshes several times each time<\/b><span style=\"font-weight: 400;\">, regularly keeping it updated.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The robots.txt is case sensitive and sits at the domain\u2019s root, for example, <\/span><b>www.domain.com\/robots.txt.<\/b><\/p>\n<h2><b>Why a Robots.txt File Matters<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Creating a robot.txt file for your website comes with many benefits; for instance, you could use it to <\/span><b>manage your crawl budget<\/b><span style=\"font-weight: 400;\">.\u00a0 Search spiders often have a <\/span><b>predetermined number of pages <\/b><span style=\"font-weight: 400;\">they can crawl on a website or the amount of time to spend on a website. If you manage a website with thousands of pages, you could <\/span><b>block unimportant pages to maximize the crawl budget<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><b>The other benefits of using a robots.txt file include:<\/b><\/p>\n<ul>\n<li><i><span style=\"font-weight: 400;\">It helps<\/span><\/i><b><i> web admins to control the web pages<\/i><\/b><i><span style=\"font-weight: 400;\"> search engines can visit.<\/span><\/i><\/li>\n<li><i><span style=\"font-weight: 400;\">The file gives users complete freedom to<\/span><\/i><b><i> block specific bots from crawling their websites.<\/i><\/b><\/li>\n<li><i><span style=\"font-weight: 400;\">The file helps prevent sensitive content from getting indexed.<\/span><\/i><\/li>\n<li><i><span style=\"font-weight: 400;\">You could use it to block indexing of unnecessary files, like images, PDF and videos.<\/span><\/i><\/li>\n<\/ul>\n<h2><b>Improving Crawlability With Robots.txt File<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Now, how do you<\/span><b> improve your website crawlability <\/b><span style=\"font-weight: 400;\">with a robots.txt file? Of course, let\u2019s find out.<\/span><\/p>\n<h3><b>Robots.txt Syntax\u00a0<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">A robot file contains one or more blocks of directives to search engines, with the first line specifying the user agent\u2014the name of the search spider to which you give the crawl directive.<\/span><\/p>\n<p><b>Here\u2019s how a basic robots.txt file looks:<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Sitemap: https:\/\/yourdomain.com\/sitemap_index.xml<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: *<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/*?comments=all<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/wp-content\/themes\/user\/js\/script-comments.js<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/wp-comments-post.php<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/go\/<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: Googlebot<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/login<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: bingbot<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/photo<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The above robots.txt file contains three blocks of directives\u2014the first directive is to all user-agents, the second directive is to <\/span><i><span style=\"font-weight: 400;\">Google crawlers<\/span><\/i><span style=\"font-weight: 400;\">, while the third is for Bing bots.<\/span><\/p>\n<p><b>Here\u2019s what the terms mean:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Sitemap<\/b><span style=\"font-weight: 400;\"> specifies the location of the website sitemap, which lists all the pages in a website, making it easier for crawlers to find and crawl them. You could also place the sitemap at the end of the robots.txt file.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>User-agent<\/b><span style=\"font-weight: 400;\"> refers to the search bot(s) you wish to address the directives to, as explained earlier. Using asterisks (*) wildcard assigns the directive to all user-agents, but you could specify a user-agent using its correct name.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Disallow<\/b><span style=\"font-weight: 400;\"> directs the user-agents not to crawl the specified URL. You could leave the line empty to specify you\u2019re not disallowing anything.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">The <\/span><b>Allow<\/b><span style=\"font-weight: 400;\"> directive instructs the bots to crawl the specified URL, even if a prior instruction disallowed its directory, and here\u2019s an example.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: *<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/wp-admin\/<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Allow: \/wp-admin\/admin-ajax.php<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The robots.txt file blocks the <\/span><b>wp-admin<\/b><span style=\"font-weight: 400;\"> directory, which contains sensitive WordPress files including plugins and themes but permits the spiders to crawl and index the <\/span><b>admin-ajax.php<\/b><span style=\"font-weight: 400;\"> file in the directory.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The <\/span><b>crawl-delay<\/b><span style=\"font-weight: 400;\"> directive (<\/span><span style=\"font-weight: 400;\">crawl-delay: 10<\/span><span style=\"font-weight: 400;\">) tells the user-agents to wait for the specified number of seconds (for example, ten seconds) before crawling the page.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The directive tells search engines to change how frequently they crawl a page, helping you save bandwidth. Unfortunately, Google doesn&#8217;t recognize this directive again, but yahoo and Bing still do.<\/span><\/p>\n<h3><b>User-Agents Directives<\/b><\/h3>\n<p><b>Most search engines have different crawlers for different purposes. <\/b><span style=\"font-weight: 400;\">For example, some search engines have spiders for normal indexing, for images and videos, while some like Bing even have spiders for their ads program.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">So, we\u2019ve put together a table of all the <\/span><b>common user-agents currently available today in alphabetical order.<\/b><\/p>\n<p><span style=\"font-weight: 400;\">Let\u2019s take a look.<\/span><\/p>\n<table>\n<tbody>\n<tr>\n<td><b>S\/N<\/b><\/td>\n<td><b>Search Engine<\/b><\/td>\n<td><b>Bots Type<\/b><\/td>\n<td><b>User-agent<\/b><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">1<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Baidu<\/span><\/td>\n<td><span style=\"font-weight: 400;\">General Indexing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">baiduspider<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">2<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Baidu<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Image\u00a0<\/span><\/td>\n<td><span style=\"font-weight: 400;\">baiduspider-image<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">3<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Baidu<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Mobile indexing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">baiduspider-mobile<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">4<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Baidu<\/span><\/td>\n<td><span style=\"font-weight: 400;\">News<\/span><\/td>\n<td><span style=\"font-weight: 400;\">baiduspider-news<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">5<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Baidu<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Videos<\/span><\/td>\n<td><span style=\"font-weight: 400;\">baiduspider-video<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">6<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Bing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">General<\/span><\/td>\n<td><span style=\"font-weight: 400;\">bingbot<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">7<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Bing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">General<\/span><\/td>\n<td><span style=\"font-weight: 400;\">msnbot<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">8<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Bing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Images and Videos<\/span><\/td>\n<td><span style=\"font-weight: 400;\">msnbot-media<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">9<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Bing<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Ads<\/span><\/td>\n<td><span style=\"font-weight: 400;\">adidxbot<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">10<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">General<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Googlebot<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">11<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Images<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Googlebot-Image<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">12<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Mobile<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Googlebot-Mobile<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">13<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">News<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Googlebot-News<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">14<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Video<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Googlebot-Video<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">15<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">AdSense<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Mediapartners-Google<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">16<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Google<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Ads<\/span><\/td>\n<td><span style=\"font-weight: 400;\">AdsBot-Google<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">17<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Yahoo<\/span><\/td>\n<td><span style=\"font-weight: 400;\">General<\/span><\/td>\n<td><span style=\"font-weight: 400;\">slurp<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-weight: 400;\">18<\/span><\/td>\n<td><span style=\"font-weight: 400;\">Yandex<\/span><\/td>\n<td><span style=\"font-weight: 400;\">General<\/span><\/td>\n<td><span style=\"font-weight: 400;\">yandex<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span style=\"font-weight: 400;\">The user-agents are case sensitive, so use the name correctly <\/span><b>when setting up your robots.txt file.<\/b><\/p>\n<h3><b>Setting Up Crawl Directives<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Let\u2019s explore some of the ways you could use the robots.txt file to crawl your website seamlessly.<\/span><\/p>\n<h4><b>Crawling the Entire Website<\/b><\/h4>\n<p><span style=\"font-weight: 400;\">You<\/span><b> could set up the robots.txt file to allow all search bots to crawl and index your entire website.<\/b><span style=\"font-weight: 400;\"> We don\u2019t recommend this if you\u2019ve private or sensitive files on your website.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">However, to give this directive, add the below lines to your robots.txt file.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: *<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow:\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">But<\/span><b> if you wish to allow only selected spiders to crawl and index the entire website<\/b><span style=\"font-weight: 400;\">, then specify the user agents, of course, one directive block per user-agent.<\/span><\/p>\n<h4><b>Blocking the entire Website<\/b><\/h4>\n<p><span style=\"font-weight: 400;\">To<\/span><b> prevent search engines from crawling and indexing<\/b><span style=\"font-weight: 400;\"> your website, especially if you\u2019re redesigning the website, you could block the entire website from getting indexed.\u00a0 <\/span><b>Add this directive to your robots.txt file to get it done.<\/b><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: *<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/<\/span><\/p>\n<p><span style=\"font-weight: 400;\">To prevent a bot from crawling your website, then specify the user-agent.<\/span><\/p>\n<h4><b>Blocking Selected Sections(s)<\/b><\/h4>\n<p><span style=\"font-weight: 400;\">To block specific sections of the website, set up a disallow directive for the folder or page, and here\u2019s an example.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: *<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/Videos<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The<\/span><b> directive blocks all spiders <\/b><span style=\"font-weight: 400;\">from crawling the video directory and everything in it. You could also use regular expressions like<\/span><b> wildcard (*) and ($) to block groups of files<\/b><span style=\"font-weight: 400;\">. Unfortunately, most search engines don&#8217;t recognize the latter, including Google.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">But here\u2019s how to use regular expressions to block a group of files.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: images\/*.jpg<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/*php<\/span><span style=\"font-weight: 400;\">$<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The wildcard (*) blocks files in the <\/span><b>image directory<\/b><span style=\"font-weight: 400;\"> containing <\/span><span style=\"font-weight: 400;\">.jpg<\/span><span style=\"font-weight: 400;\"> in its filename, while ($) blocks all files that end with <\/span><span style=\"font-weight: 400;\">.php<\/span><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><b>Please do note that the disallow, allow, and user-agent values are case-sensitive. In our two examples above, search spiders will block:<\/b><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\">\u00a0<\/span><b>Videos directory <\/b><span style=\"font-weight: 400;\">but will not block <\/span><b>\/videos<\/b><\/li>\n<li><b>\/images\/beach.jpg <\/b><span style=\"font-weight: 400;\">but will crawl<\/span><b> \/images\/beach.JPG<\/b><\/li>\n<\/ul>\n<h2><b>Robot.txt File Vs. NoIndex Tag<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">The robots.txt file directs spiders not to crawl a page but might not stop search engines from indexing the page if many websites link it. If a<\/span><b> search engine discovers enough external links to the page<\/b><span style=\"font-weight: 400;\">, it will index the page without knowing its content, giving you a search result that looks thus:<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2230\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image8-5.png\" alt=\"Create robots.txt File Best Practices, Robot.txt File Vs. NoIndex Tag\" width=\"518\" height=\"44\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image8-5.png 518w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image8-5-300x25.png 300w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">But you could add the <\/span><b>Noindex directive to your robots.txt file<\/b><span style=\"font-weight: 400;\"> to prevent the files from showing up in the search result.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">User-agent: *<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Disallow: \/Videos<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Noindex: \/Videos<\/span><\/p>\n<p><span style=\"font-weight: 400;\">You could also add a meta robots <\/span><span style=\"font-weight: 400;\">noindex<\/span><span style=\"font-weight: 400;\"> tag to the page\u2019s header to reliably prevent search engines from indexing it. If you use this option,<\/span><b> avoid blocking the page with the robots.txt to enable the spiders to find the tag.<\/b><\/p>\n<h2><b>Generating a Robot.txt File<\/b><\/h2>\n<p><b>You can generate a robots.txt file for your website using some intuitive online tools, and here are just five:<\/b><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><i><span style=\"font-weight: 400;\">Ryte Robots.txt Generator<\/span><\/i><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><i><span style=\"font-weight: 400;\">SureOak Robots.txt File Generator<\/span><\/i><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><i><span style=\"font-weight: 400;\">SEOptimer Free Robots.txt Generator<\/span><\/i><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><i><span style=\"font-weight: 400;\">SEO PowerSuite Robots.txt Generator Tool<\/span><\/i><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><i><span style=\"font-weight: 400;\">SEOBook Robots.txt File Generator<\/span><\/i><\/li>\n<\/ul>\n<h2><b>Adding a Robots.txt File to Your Domain<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">You can<\/span><b> add your newly created robots.txt<\/b><span style=\"font-weight: 400;\"> to your domain via your account control panel, and here\u2019s how.<\/span><\/p>\n<h3><b>Step 1: Access Your Account Control Panel<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Access your account\u2019s control panel by logging in to SPanel. Visit <\/span><b>www.domain.com\/spanel\/login<\/b><span style=\"font-weight: 400;\">, replacing <\/span><i><span style=\"font-weight: 400;\">domain.com<\/span><\/i><span style=\"font-weight: 400;\"> with your domain name.<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2234\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image3-6.jpg\" alt=\"Create robots.txt File Best Practices, Step 1: Access Your Account Control Panel\" width=\"627\" height=\"500\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image3-6.jpg 627w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image3-6-300x239.jpg 300w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Input your login credentials to log in.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you logged in as an<\/span><b> admin, <\/b><a href=\"https:\/\/www.scalahosting.com\/spanel.html\"><b>SPanel<\/b><\/a> <span style=\"font-weight: 400;\">takes you to your admin dashboard, but user access logs you to the control panel.\u00a0 On the admin dashboard, scroll to <\/span><b>QUICK LINKS <\/b><span style=\"font-weight: 400;\">and click <\/span><b>List Accounts<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-large wp-image-2231\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image9-1024x547.jpg\" alt=\"Create robots.txt File Best Practices, Step 1: Access Your Account Control Panel 2\" width=\"1024\" height=\"547\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image9-1024x547.jpg 1024w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image9-300x160.jpg 300w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image9-768x410.jpg 768w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image9.jpg 1032w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Click the <\/span><b>Actions <\/b><span style=\"font-weight: 400;\">button of the account you wish to access its control panel and choose<\/span><b> Login<\/b><span style=\"font-weight: 400;\"> from the <\/span><b>pull-up menu to get access.<\/b><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2233\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image2-6.jpg\" alt=\"Create robots.txt File Best Practices, Step 1: Access Your Account Control Panel 3\" width=\"453\" height=\"486\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image2-6.jpg 453w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image2-6-280x300.jpg 280w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<h3><b>Step 2: Open the File Manager<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">On the control panel, click <\/span><b>File manager<\/b><span style=\"font-weight: 400;\"> under the <\/span><b>FILES<\/b><span style=\"font-weight: 400;\"> section.<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2236\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image5-7.png\" alt=\"Create robots.txt File Best Practices, Step 2: Open the File Manager\" width=\"676\" height=\"175\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image5-7.png 676w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image5-7-300x78.png 300w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Open your website\u2019s base or root directory. The root domain uses the <\/span><b>public_html<\/b><span style=\"font-weight: 400;\"> folder as its root directory.<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2232\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image1-9.png\" alt=\"Create robots.txt File Best Practices, Step 2: Open the File Manager 2\" width=\"400\" height=\"489\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image1-9.png 400w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image1-9-245x300.png 245w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<h3><b>Step 3: Create the Robots.txt File<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">In the root directory, click the <\/span><b>New File\/Folder<\/b><span style=\"font-weight: 400;\"> icon and select <\/span><b>New File<\/b><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2237\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image6-9.png\" alt=\"Create robots.txt File Best Practices, Step 3: Create the Robots.txt File\" width=\"657\" height=\"562\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image6-9.png 657w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image6-9-300x257.png 300w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Name the new file <\/span><i><span style=\"font-weight: 400;\">robots.txt<\/span><\/i><span style=\"font-weight: 400;\"> without caps and click <\/span><b>OK<\/b><span style=\"font-weight: 400;\"> to save<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2238\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image7-6.png\" alt=\"Create robots.txt File Best Practices, Step 3: Create the Robots.txt File 2\" width=\"527\" height=\"166\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image7-6.png 527w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image7-6-300x94.png 300w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Write your crawl directives or Copy and paste them into the blank file and save.<\/span><\/p>\n<p><img decoding=\"async\" class=\"aligncenter size-full wp-image-2235\" src=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image4-11.png\" alt=\"Create robots.txt File Best Practices, Step 3: Create the Robots.txt File 3\" width=\"387\" height=\"95\" srcset=\"https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image4-11.png 387w, https:\/\/www.scalahosting.com\/kb\/wp-content\/uploads\/2021\/06\/image4-11-300x74.png 300w\" sizes=\"(max-width: 361px) 660px, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 910px, 1140px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">That\u2019s it.<\/span><\/p>\n<h2><b>Wrapping It Up<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">When you publish your robots.txt file, use the <\/span><b>Google robots.txt Tester tool<\/b><span style=\"font-weight: 400;\"> to validate the crawl directives to ensure you don\u2019t mistakenly disallow pages you don\u2019t intend to block.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">And you can select any <\/span><b>Google user-agent you wish to simulate<\/b><span style=\"font-weight: 400;\">. If you have questions related to robots.txt, do contact <\/span><a href=\"https:\/\/www.scalahosting.com\/kb\/\"><span style=\"font-weight: 400;\">our support<\/span><\/a><span style=\"font-weight: 400;\"> for quick assistance. We\u2019re always available and ready to help.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Organic searches have become an integral part of our daily lives. Recent data shows that close to 30 percent of global web traffic is from online searches.\u00a0\u00a0 Search engines crawl and index billions of web content every day, ranking them in search results according to relevancy\u2014how relevant they are to search queries\u2014making them available to [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_seopress_robots_primary_cat":"none","_seopress_titles_title":"Create robots.txt File Best Practices | ScalaHosting KB","_seopress_titles_desc":"","_seopress_robots_index":"","footnotes":""},"categories":[27],"tags":[],"class_list":["post-2229","post","type-post","status-publish","format-standard","hentry","category-vps-hosting"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/posts\/2229","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/comments?post=2229"}],"version-history":[{"count":6,"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/posts\/2229\/revisions"}],"predecessor-version":[{"id":5870,"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/posts\/2229\/revisions\/5870"}],"wp:attachment":[{"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/media?parent=2229"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/categories?post=2229"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.scalahosting.com\/kb\/wp-json\/wp\/v2\/tags?post=2229"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}