My Blogs are blocked from SEO/Robots.txt?

Site URL: https://onyxhive.com.au/blog

Hi,

So looking at Ahrefs, all my blog links are blocked from robots.txt. Is this normal? Cause when I googled my blogs, nothing shows. Not even the keywords.

I checked the blog SEO settings and nothing is blocked.

How do I fix this?

Thanks

Hi @hasher22 - I think what your Ahrefs screenshot shows is actually just that the page that shows blog posts filtered by author is blocked from search results, not the blog posts themselves. The URL listed in the "Target URL" column in the Ahrefs report is the one that is being reported as blocked in the robots.txt file, not the blog posts themselves.

However if you want to double check the contents of the robots.txt file you can use Google's Robots Testing Tool to view the contents of the file: https://www.google.com/webmasters/tools/robots-testing-tool You will need to have your domain set up as a verified property in Google Search Console to use the tool.

Hope that helps!

- Ed

--

Blue Hills Digital
Support with website optimization, SEO and finding a digital marketing strategy that works.

Latest resource: Squarespace SEO Services

41 minutes ago, edharris said:

Hope that helps!

Thank you, it does.

Quote

The URL listed in the "Target URL" column in the Ahrefs report is the one that is being reported as blocked in the robots.txt file, not the blog posts themselves.

Any reason why my blog pages and even /blog is not even showing in Google when I search specifically for my blog with all different keywords?

Keywords such as:
- onyx hive blog
- onyx hive blogs
- onyx hive *insert article heading*
- onyve hive *insert link keywords*

None of them shows up. But searching for my products, about, pictures etc... they all show up as normal in google.

This is my robots.txt file but are you able to help me decipher it and see if my /blog is actually blocked?

Quote

# Squarespace Robots Txt

User-agent: AdsBot-Google
Disallow: /config
Disallow: /search
Disallow: /account$
Disallow: /account/
Disallow: /commerce/digital-download/
Disallow: /api/
Allow: /api/ui-extensions/
Disallow: /static/
Disallow:/*?author=*
Disallow:/*&author=*
Disallow:/*?tag=*
Disallow:/*&tag=*
Disallow:/*?month=*
Disallow:/*&month=*
Disallow:/*?view=*
Disallow:/*&view=*
Disallow:/*?format=json
Disallow:/*&format=json
Disallow:/*?format=page-context
Disallow:/*&format=page-context
Disallow:/*?format=main-content
Disallow:/*&format=main-content
Disallow:/*?format=json-pretty
Disallow:/*&format=json-pretty
Disallow:/*?format=ical
Disallow:/*&format=ical
Disallow:/*?reversePaginate=*
Disallow:/*&reversePaginate=*

User-agent: AdsBot-Google-Mobile
Disallow: /config
Disallow: /search
Disallow: /account$
Disallow: /account/
Disallow: /commerce/digital-download/
Disallow: /api/
Allow: /api/ui-extensions/
Disallow: /static/
Disallow:/*?author=*
Disallow:/*&author=*
Disallow:/*?tag=*
Disallow:/*&tag=*
Disallow:/*?month=*
Disallow:/*&month=*
Disallow:/*?view=*
Disallow:/*&view=*
Disallow:/*?format=json
Disallow:/*&format=json
Disallow:/*?format=page-context
Disallow:/*&format=page-context
Disallow:/*?format=main-content
Disallow:/*&format=main-content
Disallow:/*?format=json-pretty
Disallow:/*&format=json-pretty
Disallow:/*?format=ical
Disallow:/*&format=ical
Disallow:/*?reversePaginate=*
Disallow:/*&reversePaginate=*

User-agent: AdsBot-Google-Mobile-Apps
Disallow: /config
Disallow: /search
Disallow: /account$
Disallow: /account/
Disallow: /commerce/digital-download/
Disallow: /api/
Allow: /api/ui-extensions/
Disallow: /static/
Disallow:/*?author=*
Disallow:/*&author=*
Disallow:/*?tag=*
Disallow:/*&tag=*
Disallow:/*?month=*
Disallow:/*&month=*
Disallow:/*?view=*
Disallow:/*&view=*
Disallow:/*?format=json
Disallow:/*&format=json
Disallow:/*?format=page-context
Disallow:/*&format=page-context
Disallow:/*?format=main-content
Disallow:/*&format=main-content
Disallow:/*?format=json-pretty
Disallow:/*&format=json-pretty
Disallow:/*?format=ical
Disallow:/*&format=ical
Disallow:/*?reversePaginate=*
Disallow:/*&reversePaginate=*

User-agent: *
Disallow: /config
Disallow: /search
Disallow: /account$
Disallow: /account/
Disallow: /commerce/digital-download/
Disallow: /api/
Allow: /api/ui-extensions/
Disallow: /static/
Disallow:/*?author=*
Disallow:/*&author=*
Disallow:/*?tag=*
Disallow:/*&tag=*
Disallow:/*?month=*
Disallow:/*&month=*
Disallow:/*?view=*
Disallow:/*&view=*
Disallow:/*?format=json
Disallow:/*&format=json
Disallow:/*?format=page-context
Disallow:/*&format=page-context
Disallow:/*?format=main-content
Disallow:/*&format=main-content
Disallow:/*?format=json-pretty
Disallow:/*&format=json-pretty
Disallow:/*?format=ical
Disallow:/*&format=ical
Disallow:/*?reversePaginate=*
Disallow:/*&reversePaginate=*

Sitemap: https://onyxhive.com.au/sitemap.xml

Hard to say without direct Google Search Console access, but I'm not seeing anything that would prevent Google from crawling and potentially indexing the blog URLs. It may just be that the site hasn't been crawled recently, or if it has, the crawl bots didn't "notice" the blog posts.

A few suggestions that might help get the blog posts crawled and indexed in future:

• Add the Blog to the main/header navigation somewhere
• Add a summary block to the footer that will include links to the three most recent blog posts or something like that (that will result in more internal linking to your posts)
• If you own your Google Business profile, try creating a post or two there pointing directly to your blog posts
• Use the URL inspection / request indexing feature in Search Console to submit a blog post URL to the priority queue (doesn't guarantee indexing but doesn't hurt)
• Make sure your sitemap URL is added to the property in Search Console

Good luck!

- Ed

--

Blue Hills Digital
Support with website optimization, SEO and finding a digital marketing strategy that works.

Latest resource: Squarespace SEO Services

10 hours ago, edharris said:

A few suggestions that might help get the blog posts crawled and indexed in future:

Thanks!
 

1. Sitemap to google search console done a few months ago
2. I have requested to 'URL inspection' for "https://onyxhive.com.au/blog" last night and it showed "URL not on Google", I checked it this morning and now state "URL is on Google". Yay!

Question: Do I need to 'request indexing' for all my article links? Only /blog is showing up in google. I know it may take a few weeks but does the 'URL inspection or request indexing' looks at all the links within that URL? For example all the article links in my blog to show up in google. Cause right now it's only showing onyxhive.com.au/blog in google and not the articles.

Edit: It seems google found my individual blog links and put them in the category "Discovered – currently not indexed". Hmmmmm

Thanks

On 2/1/2022 at 8:53 PM, hasher22 said:

Edit: It seems google found my individual blog links and put them in the category "Discovered – currently not indexed". Hmmmmm

That's good that those individual blog URLs are "discovered" now. Next step is for them to be crawled, and then potentially indexed and ranked. Outside of the actions I suggested in my earlier comment there's not much else you can do to speed this process up – just make sure the quality of the content is good and keep up the focus on internal linking on the site!

You can "request indexing" for each individual URL (but don't submit more than once for each one). It may not necessarily make any difference in the speed that they are prioritized though.

– Ed

--

Blue Hills Digital
Support with website optimization, SEO and finding a digital marketing strategy that works.

Latest resource: Squarespace SEO Services