Jump to content

How do I find and edit my robots.txt file?

Go to solution Solved by paul2009,

Recommended Posts

Site URL: https://www.stevemaskery.com/

Hi, I'm new to SS and not a lot of previous experience with HTML.

I have two domains, workshopessentials.com and stevemaskery.com, both of  which are forwarded to:

https://amphibian-celery-b7en.squarespace.com

I'm getting a nag from Google saying that I need to add a couple of lines to my robots.txt file so that it can index my images properly. How do I find and access this file please?

Many thanks
Steve

 

Link to comment
  • Solution
On 3/16/2022 at 10:47 AM, SteveMaskery said:

How do I find and access [my robots.txt] file please?

All Squarespace sites use the same robots.txt file and as a Squarespace user you cannot access or edit it. Squarespace have already specified the pages that should not be crawled by search engines because they’re for internal use only or display duplicate content. For example, /config/ is your Admin login page, and /api/ blocks the Analytics tracking cookie.

Here are some pages that Squarespace ask search engines not to crawl. These pages organize content that exists elsewhere on your site.

/search

/*?author=*

/*&author=*

/*?tag=*

/*&tag=*

/*?month=*

/*&month=*

/*?view=*

/*&view=*

/*?format=*

/*&format=*

/*?reversePaginate=*

/*&reversePaginate=*

For a complete list of the excluded pages, view the robots.txt file on any Squarespace website.

Edited by paul2009
Added additional information

Improve your online store with our extensions.
About: Squarespace Circle Leader since 2017. I value honesty, transparency, appreciation and great design ♥.
Work: Squarespace Developer and founder of SF Digital, building the features Squarespace didn't include™.
Content: Links in my posts may refer to SF Digital products or may be affiliate links.

Buy me a coffee

Link to comment

Ah, OK, thank you.

Google is telling me this:

Quote

 

Dear Google Merchant Center user,

Merchant Center Account:  xxxxxxxxx

We have detected a recent configuration change of the site hosting your images that result in the disapproval of some of your items in your Merchant Center account.

Since images are an important part of the rich product information shown in Shopping ads, we require that all items include a valid image that can be indexed by Google. We crawl the images you submit to Merchant Center every few weeks to ensure that users always see the most recent version. Our ability to crawl your images can be restricted with a robots.txt file, which might lead to the disapproval of affected items. Learn more about robots.txt by visiting https://support.google.com/webmasters/answer/6062608.

What's the issue?
We have detected that a robots.txt file that controls the indexing of some of your provided images has been updated recently. As a result of these updates we aren't able to index the images of some of your items. This will result in the disapproval of the affected items.

Details and impact:
Estimated percentage of offers affected: 100
File: http://www.workshopessentials.com/robots.txt

In order for us to access these images, please modify the robots.txt file mentioned above to allow the user-agent Googlebot-Image to index these images. You can do this by adding the following lines to the robots.txt file:

User-agent:       Googlebot-Image
Disallow:

If modifying this robots.txt file is not feasible you might want to consider hosting your images on a different hosting service that allows images to be indexed by Google.

 

Is this something I should be worried about or can I safely ignore it?

Many thanks
Steve

 

 

Edited by SteveMaskery
Remove sensitive info
Link to comment
  • 1 month later...

@burmdoge What issue are you trying to address? 

Improve your online store with our extensions.
About: Squarespace Circle Leader since 2017. I value honesty, transparency, appreciation and great design ♥.
Work: Squarespace Developer and founder of SF Digital, building the features Squarespace didn't include™.
Content: Links in my posts may refer to SF Digital products or may be affiliate links.

Buy me a coffee

Link to comment

Hi Paul,

I'm trying to get the index page on our Squarespace website indexed, but in the Google Search Console, URL inspection it's showing:

URL is not on Google: Indexing errors

Failed: Redirect error  

 https://mesaio.com.au

Looks like it's related to the robots.txt file, could you please help resolve this issue?

Thank you,

Dion

Link to comment
  • 3 weeks later...
  • 1 month later...
  • 1 month later...
  • 1 month later...
  • 2 weeks later...
On 9/5/2022 at 5:29 PM, MJB1923 said:

Squarespace robots.txt is blocking Google from indexing my blog posts.

How do I edit the robots.txt to stop this? Totally ridiculous. Checking before I have to move my site off of Squarespace as it will not allow blog posts to be indexed. Regular pages are indexed.

Screen Shot 2022-09-05 at 11.27.31 AM.png

I experience the exact same!! Can someone from Squarespace please help? Because blogposts sure are NOT pages that google should not crawl. Tell us how to fix it or a lot of people will move their business away from squarespace. 

Link to comment
On 11/15/2022 at 11:14 AM, Chiara1234 said:

Squarespace robots.txt is blocking Google from indexing my blog posts. Can someone from Squarespace please help? Because blogposts sure are NOT pages that google should not crawl.

@Chiara1234 I believe you have misunderstood the message from Google.

Squarespace is not blocking Google from indexing your blog posts. Google is indexing your blog but Squarespace is correctly preventing Google from indexing the category links on your blog page, for example /blog/category/Reviews.

This link leads to duplicate content (the same blog posts that have already been indexed individually) and so this link should not be indexed, as I mentioned in my answer above.

Did this help? Please give feedback by clicking an icon below  ⬇️

Improve your online store with our extensions.
About: Squarespace Circle Leader since 2017. I value honesty, transparency, appreciation and great design ♥.
Work: Squarespace Developer and founder of SF Digital, building the features Squarespace didn't include™.
Content: Links in my posts may refer to SF Digital products or may be affiliate links.

Buy me a coffee

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

×
×
  • Create New...

Squarespace Webinars

Free online sessions where you’ll learn the basics and refine your Squarespace skills.

Hire a Designer

Stand out online with the help of an experienced designer or developer.