G-man Posted October 19, 2020 Share Posted October 19, 2020 (edited) Site URL: https://www.5theis.com/distributor-login I've got a password-protected page that is getting flagged in SEMrush as 80 broken internal links because of it being in the main menu. The page comes up fine, but is returning a 401 Unauthorized error. So I'd simply like to add a line to robots.txt to have it ignore that page in its site crawler. Saw a comment in an online forum saying we can't edit your robots.txt. Any suggestion on fixing this issue? Edited October 19, 2020 by G-man Shorten title Link to comment
modsquare Posted October 20, 2020 Share Posted October 20, 2020 Yeah, you do have a few options here. This page outlines them pretty well: https://support.squarespace.com/hc/en-us/articles/360022347072-Hiding-pages-from-search-engine-results The easiest option is to flip the switch on the SEO settings tab of the page to tell it to not be indexed. This option is below the fold on the pop-up screen which is kinda annoying and may be why you didn't see it. The other option is to index some code into the page like this: <meta name="robots" content="noindex"> Hopefully the SEMrush crawler will obey these directives despite what the robots.txt file says. The code injection option often confuses Google Search Console because the page still appears in the automated sitemap.xml file but you're also telling them to not index it — so it'll keep sending you warnings. Link to comment
G-man Posted October 20, 2020 Author Share Posted October 20, 2020 Many thanks for replying. The very first thing I did was set it in the SEO module to not include it in search results. Didn't try the meta tag thinking that would be redundant setting that in the SEO module. Just added the meta tag and SEMrush still says "80 internal links are broken", all from that one page being in the main menu. The only way SEMrush will avoid crawling it is adding it to robots.txt. Guess I'm SOL. Since we can't edit robots.txt, do you think SS Support would consider adding it? Link to comment
ArtByPino Posted October 21, 2020 Share Posted October 21, 2020 17 hours ago, G-man said: Many thanks for replying. The very first thing I did was set it in the SEO module to not include it in search results. Didn't try the meta tag thinking that would be redundant setting that in the SEO module. Just added the meta tag and SEMrush still says "80 internal links are broken", all from that one page being in the main menu. The only way SEMrush will avoid crawling it is adding it to robots.txt. Guess I'm SOL. Since we can't edit robots.txt, do you think SS Support would consider adding it? When you type out your [domain]/sitemap.xml, does it show the page you are trying to hide/noindex? If it does then your turning it off is (obviously) not working and SSS should be able to resolve. My experience with some 3rd party tools is that despite turning off SEO (for example, for blog, turning off categories and tags), it still gives me duplication and canonical errors and I just ignore those. Additionally, some pages also are displayed with AMP errors by the crawlers of the 3rd party bots which I also ignore. https://www.google.com/webmasters/tools/robots-testing-tool?hl=en will show how Google reads your robots.txt and I would go by that rather than what SEMRush says. Link to comment
mirandakelton Posted September 16, 2023 Share Posted September 16, 2023 @ArtByPino I am having this same issue with my member site & it is driving me crazy! Did you ever find a solution to this? Link to comment
Adrian_CH Posted June 6 Share Posted June 6 (edited) This is an actual problem. I too have it now because I linked it in the menu. It's especially bad because I'm an inhouse-SEO and it is hard to sell to my boss that this is nothing to worry about. In fact Google Search Console shows that it's not indexed so it's actually no problem. But it just looks bad in the report. How on earth is it still not possible to edit the robots.txt ... 🙄 I'm an advocate for easy solutions like Squarespace instead of WordPress because of the security risks and the maintenance volume. But this is actually a good argument for WordPress and other CMS. It would be easy af to fix this in WordPress. Edited June 6 by Adrian_CH Link to comment
Collaborada Posted June 8 Share Posted June 8 Hi @Adrian_CH I wouldn’t recommend making a reactive move to WP based on a Semrush report. It is a common misconception that every issue flagged impacts performance. As an in-house SEO, your role includes educating leadership about optimization and tailoring reports as needed. Have you tried disallowing URLs: https://www.semrush.com/kb/539-configuring-site-audit#urls Building high-performance websites with expert SEO for local and national reach. Have an SEO question? Google to find our advice or book a Zoom session for tailored help. ★ Official Squarespace Experts since 2014 ★ Link to comment
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment