Jump to content

How to block nefarious web spiders

Recommended Posts

Posted

Site URL: https://www.simonmetzwoodworking.com

I have an issue where a scam site from China has a page that appears to link to my site but in actuality links to their home page. I need to block this as it is some kind of scan financial services site and don't want to get dinged by Google. Normally I would block spiders in robots.txt file but that's not available in SS. Anyone have any ideas on how to block spiders. I used to bloc Baidu when I coded my own site as their bot spent hours indexing my site from what my logs told me.

 

The URL I'm trying to block is https://m.pushade.com/by/simonmetzwoodworking/

 

Thanks!

Screen Shot 2021-09-19 at 3.04.18 PM.png

  • Replies 2
  • Views 778
  • Created
  • Last Reply
Posted

I don't think you can block a url on someone else's page. In reality however the url you posted doesn't actually exist on the site, its an empty URL redirect. You could substitute any website domain for yours and it will still go to the sites home page, and then your broswer cache remembers that redirect as if it were a real url. For example you can substitute facebook for simonwetz.... and the site behavior is the same. 

Posted

Thanks, didn't see that. I was trying to block the chinese spiders so that cannot even see my site. When I coded my own site my logs showed that their spiders would spend hours on my sites which keeps the servers working for no good reason, so I just blocked them wholesale 🙂

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Squarespace Webinars

Free online sessions where you’ll learn the basics and refine your Squarespace skills.

Hire a Designer

Stand out online with the help of an experienced designer or developer.