Search the Community
Showing results for tags 'robots.txt'.
Site URL: https://www.5theis.com/distributor-login I've got a password-protected page that is getting flagged in SEMrush as 80 broken internal links because of it being in the main menu. The page comes up fine, but is returning a 401 Unauthorized error. So I'd simply like to add a line to robots.txt to have it ignore that page in its site crawler. Saw a comment in an online forum saying we can't edit your robots.txt. Any suggestion on fixing this issue?
I want to determine which individual photos can be found on google ... is this possible? Google says: add a "robots.txt" file to the root directory of your server... I already did use this code: <meta name="robots" content="noindex"> BUT: I want to know how this work only for certain photos :)
Site URL: https://klikblicvijesti.info/ Hello everybody I have a problem with a "mysterious" robots.txt file. It’s mysterious for the reason that I didn’t even create it however it appears and creates a problem for me in indexing my page by Google. When I enter the File Manager of my site I can't find robots.txt anywhere. It blocks my ads and I don't know how to remove it. Here is the contents of that robots.txt: User-Agent: * Allow: / ads / preferences / Allow: / gpt / Allow: /pagead/show_ads.js Allow: /pagead/js/adsbygoogle.js Allow: /pagead/js/*/show_ads_impl.js Allow: /stat
Site URL: http://www.baotris.com Hello! I am just learning about SEO and Marketing so apologies for lack of knowledge/ jargon. We have a webpage and when I ran the robots.txt file, (https://baotris.com/robots.txt) - it shows that everything is marked as "disallow", does this mean our site isn't able to be crawled by search engines and won't come up with searches? If so, how do I fix this? Thanks in advance!