Jump to content

How do I find and edit my robots.txt file?

Go to solution Solved by paul2009,

Recommended Posts

1 hour ago, creedon said:

I suspect crawlers in Settings > Website > Crawlers has been disabled.

Screenshot2024-06-29at11_02_21AM.thumb.png.3bb624117ed7f9ddebad84aae38984f7.png

Looking at your robots.txt we can see the disallow.

Screenshot2024-06-29at11_00_09AM.thumb.png.8f60c35e2ea11a367c56e17d8ba78ba4.png

Please see Excluding your site from AI scans.

Let us know how it goes.

Thank you for the reply! Ive only disabled AI crawlers. My understanding was that this isn't supposed to prohibit you from showing up on goole ...?

I did toggle the AI crawler back on again, did a refresh, and Im still not showing up

Screenshot 2024-06-29 at 20.09.22.png

Edited by ilseS
Link to comment
23 minutes ago, ilseS said:

Ive only disabled AI crawlers. My understanding was that this isn't supposed to prohibit you from showing up on goole ...?

I did toggle the AI crawler back on again, did a refresh, and Im still not showing up

The key phrase in the settings is "If turned off, your site won't be indexed by search engines."

So you need to reverse the settings from what you've show in your screenshot.

Also it may take some amount of time before your pages begin to show up again in Google results. Google crawls when they decide to crawl and you can't force them to update now.

Find my contributions useful? Please like, upvote, mark my answer as the best ( solution ), and see my profile. Thanks for your support! I am a Squarespace ( and other technological things ) consultant open for new projects.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

×
×
  • Create New...

Squarespace Webinars

Free online sessions where you’ll learn the basics and refine your Squarespace skills.

Hire a Designer

Stand out online with the help of an experienced designer or developer.