Jump to content

Robots.txt File

Recommended Posts

Hi,

Could I get some advice please on something.
I know literally nothing about robots and have seen that my robots.txt file reads as below. 
Is this normal and allowing Google to crawl my entire site correctly?

Thank you in advance for any support and advice.

Tim 

 

 

# Squarespace Robots Txt

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: CCBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: FacebookBot
User-agent: Claude-Web
User-agent: cohere-ai
Disallow: /

User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AdsBot-Google-Mobile-Apps
User-agent: *
Disallow: /config
Disallow: /search
Disallow: /account$
Disallow: /account/
Disallow: /commerce/digital-download/
Disallow: /api/
Allow: /api/ui-extensions/
Disallow: /static/
Disallow:/*?author=*
Disallow:/*&author=*
Disallow:/*?tag=*
Disallow:/*&tag=*
Disallow:/*?month=*
Disallow:/*&month=*
Disallow:/*?view=*
Disallow:/*&view=*
Disallow:/*?format=json
Disallow:/*&format=json
Disallow:/*?format=page-context
Disallow:/*&format=page-context
Disallow:/*?format=main-content
Disallow:/*&format=main-content
Disallow:/*?format=json-pretty
Disallow:/*&format=json-pretty
Disallow:/*?format=ical
Disallow:/*&format=ical
Disallow:/*?reversePaginate=*
Disallow:/*&reversePaginate=*

Sitemap: https://www.ambientlife.co.uk/sitemap.xml

 

Link to comment
Posted (edited)
On 7/1/2024 at 7:46 AM, AmbientLife said:

I know literally nothing about robots and have seen that my robots.txt file reads as below. 
Is this normal and allowing Google to crawl my entire site correctly?

Yes, it is correct.

The robots file shown above is the standard Squarespace robots file when Artificial Intelligence (AI) crawlers are prevented from scanning your site for content. 

Squarespace manage your robots file for you. Other than deciding whether you want to allow AI to crawl your site, you do not need to worry about this file. You cannot make any changes to the file yourself.

Did this help? Please give feedback by clicking an icon below  ⬇️

Edited by paul2009

Me: I'm Paul, a SQSP user for >18 yrs & Circle Leader since 2017. I value honesty, transparency, diversity and good design ♥.
Work: Founder of SF.DIGITAL. We provide high quality original extensions to supercharge your Squarespace website. 
Content: Views and opinions are my own. Links in my posts may refer to my own SF.DIGITAL products or may be affiliate links.
Forum advice is completely free. You can thank me by selecting a feedback emoji. Buying a coffee is generous but optional.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

×
×
  • Create New...

Squarespace Webinars

Free online sessions where you’ll learn the basics and refine your Squarespace skills.

Hire a Designer

Stand out online with the help of an experienced designer or developer.