If you pay attention to your webstats, you’ve likely come across regular hits coming from a crawler/bot called “Semalt”. According to their site and some other blog sites, the bot is not hitting your site with any bad intentions, but others have stated that they seem to be associated with spamming and scamming of some kind. Either way, its best to keep your site free from these bots as they can screw up your web statistics by inflating your visitor and bounce rates. The majority of sites are recommending to block this bot – and there are some great tips on doing so.
In order to block them via the server level, you need to add the following code to the BOTTOM of your .htaccess file:
SetEnvIfNoCase Via evil-spam-proxy spammer=yes
SetEnvIfNoCase Referer evil-spam-domain.com spammer=yes
SetEnvIfNoCase Referer evil-spam-keyword spammer=yes
SetEnvIfNoCase Via pinappleproxy spammer=yes
SetEnvIfNoCase Referer semalt.com spammer=yes
SetEnvIfNoCase Referer poker spammer=yes
Allow from all
Deny from env=spammer
They also created a tool where you can remove your website from their crawl list. A lot of sites are saying “why should we give them our domain information, when they shouldn’t be crawling our site in the first place – and there is a point there. But we did test their ignore list with a few domains, and a month later, none of those sites had been hit by semalt – so it does appear, at least in theory, to work.
To add your sites to their block list, simply visit this link, and type all of your URLs into the list, and hit submit.