Jump to content

How to crawl my website with Robot.txt .

Recommended Posts

I have had my website here for several years. if the robot.txt is not showing up after my domain name in the google search bar on my HOME page or BLOG page . Would someone please look at this page in tools . Screenshot I provided and is this the correct spot ? Line 1  ? to add robots.txt on this line . but then It will show up on every page on my website and that may not be allowed with google or bing microsoft search engines to crawl.  So do I add it on the HOME page after opening up the circle tool icon and go to "advanced" and add it after <h1> as line number 2 <robot.txt>    ?  please help . I will attach files . 

Screenshot 2024-01-24 144454.png

Screenshot 2024-01-24 152740.png

Link to comment
  • Replies 1
  • Views 1.1k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Create an account or sign in to comment

You need to be a member in order to leave a comment

×
×
  • Create New...

Squarespace Webinars

Free online sessions where you’ll learn the basics and refine your Squarespace skills.

Hire a Designer

Stand out online with the help of an experienced designer or developer.