G-man
-
Posts
2 -
Joined
-
Last visited
Content Type
Forums
Gallery
Blogs
Events
Store
Downloads
Profiles
Posts posted by G-man
-
-
Site URL: https://www.5theis.com/distributor-login
I've got a password-protected page that is getting flagged in SEMrush as 80 broken internal links because of it being in the main menu. The page comes up fine, but is returning a 401 Unauthorized error.
So I'd simply like to add a line to robots.txt to have it ignore that page in its site crawler. Saw a comment in an online forum saying we can't edit your robots.txt. Any suggestion on fixing this issue?
Password-protected page showing as a broken internal link
in SEO
Posted
Many thanks for replying. The very first thing I did was set it in the SEO module to not include it in search results. Didn't try the meta tag thinking that would be redundant setting that in the SEO module. Just added the meta tag and SEMrush still says "80 internal links are broken", all from that one page being in the main menu.
The only way SEMrush will avoid crawling it is adding it to robots.txt. Guess I'm SOL.
Since we can't edit robots.txt, do you think SS Support would consider adding it?