Product Page Crawl/Index blocked by Robots.txt

Site URL: https://www.speedysearch.ca/

Hi, getting the below error on 86 product pages and I can't figure out why it would be happening. Ive read over the robots.txt file and nothing about product pages are in there. The site has had its sitemap regenerated and has been reindexed and recrawled. Any ideas? Any help would be greatly appreciated. 

Hi – I suspect these product URLs might be getting blocked because they are in the .../search-all/... directory, and .../search is blocked by default in the Squarespace robots.txt file. Obviously the intention there is to block indexing of URLs generated by the built-in website search functionality, but I suspect your products are getting caught up in that too.

I'm not aware of a way to modify the robots.txt file contents on Squarespace, so if that was the case and it's important for your business to have these product URLs indexed by Google, you might need to adjust your URL structure. If you need support there, let me know.

Hope that helps!

- Ed

--

Blue Hills Digital
Support with website optimization, SEO and finding a digital marketing strategy that works.

Latest resource: Squarespace SEO Services

Thanks for the reply. I believe the "/search" in the robots.txt file refers to the actually search function that squarespace provides. Since it doesn't have any indicators that there can be any leading or following characters. As well, its only happening with some of the products, not all, some of the products with  "/search-all" in the URL are just fine...

I think I understand what you are saying now. Thank you again for your reply! 🙂