I have 13 Bots/Crawlers all over my websites that are causing me server issues which is one of the reasons I got this lovely component.
I have blocked four that I managed to work out but can anyone tell me how to identify the others to block them
Last edit: 5 years 2 months ago by HBAGG. Reason: Folder not attached
As you can see every line has info like the IP, date, resource accesed and the user-agent used. Most web crawlers have its description in the user-agent field (although this field can be modified to hide the real bot, or even leave it empty), so you can block unwanted bots using that info.
In the next version of SCP, that will be released in a few days, I have updated the list of predefined malicious bots with more than 40 new bots.