.Google.com's John Mueller addressed an inquiry concerning why Google.com marks pages that are forbidden from creeping by robots.txt and also why the it's secure to overlook the associated Explore Console files concerning those crawls.Crawler Traffic To Question Parameter URLs.The person asking the inquiry documented that crawlers were making links to non-existent concern specification URLs (? q= xyz) to pages with noindex meta tags that are actually likewise blocked in robots.txt. What caused the inquiry is actually that Google is actually creeping the hyperlinks to those webpages, acquiring blocked by robots.txt (without noticing a noindex robots meta tag) at that point acquiring turned up in Google.com Browse Console as "Indexed, though shut out by robots.txt.".The person talked to the adhering to concern:." Yet listed here is actually the significant question: why would certainly Google.com mark webpages when they can't even observe the content? What's the benefit in that?".Google's John Mueller validated that if they can't crawl the web page they can not view the noindex meta tag. He additionally creates an intriguing acknowledgment of the internet site: search driver, suggesting to overlook the end results considering that the "average" customers won't observe those outcomes.He created:." Yes, you're appropriate: if our experts can not crawl the web page, we can't find the noindex. That pointed out, if our company can't crawl the pages, at that point there is actually certainly not a whole lot for us to index. So while you may observe a number of those web pages with a targeted internet site:- query, the normal individual will not view them, so I would not fuss over it. Noindex is actually likewise great (without robots.txt disallow), it simply indicates the URLs will wind up being actually crept (and end up in the Look Console document for crawled/not listed-- neither of these conditions trigger problems to the remainder of the site). The vital part is actually that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the constraints in using the Site: search evolved hunt driver for diagnostic reasons. One of those reasons is given that it is actually certainly not linked to the frequent search mark, it is actually a separate point completely.Google's John Mueller talked about the site hunt driver in 2021:." The short solution is that an internet site: question is not meant to be total, neither utilized for diagnostics objectives.A site question is actually a details sort of hunt that limits the results to a particular site. It's basically merely the word web site, a colon, and afterwards the website's domain name.This inquiry confines the results to a specific site. It's not indicated to be a detailed selection of all the pages from that site.".2. Noindex tag without making use of a robots.txt is actually great for these type of conditions where a robot is connecting to non-existent pages that are receiving found out by Googlebot.3. Links with the noindex tag will produce a "crawled/not recorded" item in Search Console which those won't possess an unfavorable effect on the rest of the site.Read through the concern as well as address on LinkedIn:.Why would Google.com mark web pages when they can not even view the material?Featured Image through Shutterstock/Krakenimages. com.