If you use our site frequently, you might have noticed that the servers have been less than reliable lately. I’ve been monitoring the logs (httpd, mysqld etc) to try and track the source of the server crashes down; it has been a hard slog as I’ve never done server maintenance regularly before. Several issues have come to light, and I’m not totally sure if the problems have been resolved. I therefore apologise for any further outages we suffer (always seems to be when I am away!) We have the servers monitored via pingdom (thanks to Andrew Larcombe).
Things that came out of the investigation were:
- Yahoo’s search spider was hammering our server constantly with a huge amount of activity which seemed to generate a slow query, send mysqld or httpd mad and then crash the server. This has infuriated our staff no end as the site is a live working tool via which they input their data about archaeological objects. I’ve blocked a couple of their spiders (via iptables) which seemed to do the most damage and then added User-agent: slurp Crawl-delay: 120 to the robots file on our server. It only seemed to have this issue with the findsdatabase URL. This month alone, their spider has taken 2.5GB of bandwidth for that site, and generated over 300,000 page requests for just finds.org.uk Is that wrong? I dunno….. 5% of your overall bandwidth in spider traffic?
Although I did find that someone from the Pentagon seems to frequent our findsdatabase rather a lot.
- We’re getting probed for XSS vulnerabilities constantly by domains from Latvia and Russia, but their IP addresses seem to be hosted out of Marina del Rey, CA, America! (For example: 220.127.116.11 or 18.104.22.168)They try and see if they can redirect variables on your site by adding their URL to a query string. For example: http://www.iamamoron.com/?ID=http://holegirl.eclub.lv/.images/pictureofme?
Now I’m not really interested in looking at the picture of someone from Latvia, with the domain name of holegirl. Haven’t you got better things to do?
If you look around the web, there’s not much information about the eclub.lv domain hacking attempts; or at least not that I could find that was worthwhile reading.
Their URL just goes to a 404 page with Cyrillic script which contains no malicious scripting that I can determine. However, I think I have sanitised all areas where there might be XSS vulnerabilities. I of course might well be wrong about the above. I guess you can use your .htaccess file to prevent http:// query strings, but I haven’t had time yet to figure this out. If anyone can give me a pointer, would be most grateful!
- We’ve also got a problem with a couple of SQL queries that were written by our former suppliers OAD, that constantly run slow. I’ve removed the functions that generate these on the website where ever I can. Hopefully that fixes it.
- There’s also been a couple of instances where we had intrusions via the old wordpress systems. These holes were well documented and have been cleared up. If you run wordpress and haven’t updated yet, more fool you. Their founder Matt, writes a good piece about why you need to upgrade over at his blog.
I’m not a server guru, so no doubt I’ve done it wrong.
Update to this: I’m also seeing the following IP address 22.214.171.124, which guess what is from the same USA address, checking the wp-cron.php file anyone know what they’re up to?