Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Advanced Search

Homepage blocked in Google crawl (401 error?), but all other pages indexed?


Go to solution Solved by Shanhou,

Recommended Posts

Site URL: https://www.dwbehavioralhealth.com

Hi all,

This site went live about a week ago and I immediately connected it with Google Search Console, submitted the sitemap and requested indexing. Within a few days, all of the site's pages EXCEPT the homepage were showing up in search and show in Google Search Console as successfully submitted and indexed.  Only the homepage shows up as excluded and the reason given is "Blocked due to unauthorized request (401)." When I click on the help link within Search Console, it brings me to instructions about verifying GoogleBot that unfortunately are of no help to me as a newbie at this stuff (the first step is to run a reverse DNS lookup on the accessing IP addresses using a host command? What is that?). 😭😞

Does anyone have any Squarespace-specific instructions about how I can try to verify GoogleBot to see if that is indeed the problem here?

OR, is this all something that might just be normal in the first few weeks after a site goes live, and I should stop worrying and just hope Google finds everything eventually? 

As a side note, one thing I noticed while going through Google Search Console is that there are results for both the main homepage, www.dwbehavioralhealth.com, and for www.dwbehavioralhealth.com/home.  Could the fact that the homepage has a URL slug be messing with Google somehow? Or does that have nothing to do with anything?  For example, www.dwbehavioralhealth.com/home is listed in search console as "Discovered - currently not indexed" while the main URL (no slug) is coming up as blocked with the 401 message above.  🤔

Any help would be so greatly appreciated. Please let me know if more details are needed in order to help. Thanks so much in advance!

Link to comment
  • Solution

Well, this appears to be solved after I resubmitted the site map and requested reindexing again - there really was no robots.txt error, but for some reason after resubmitting it just worked! 
Just leaving this here in case it happens to anyone else and they find this 😄

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

×
×
  • Create New...