OT - Tracking IP's and Usage?

[koltz]

Since I have lived my OSC-MAX site, I have caught 2 IP's using spiders or ??? on my site. One of them was from MS which for a while, I left, then got tired seeing all the bandwidth and open sessions they left open, so I blocked the IP address with IPTABLE. The problem though was I wasn't sure if it was a legit spider (search engine, commerce engine, etc.). Yesterday I caught 66.82.9.49 which is a direcpc IP address. So I blocked that one also.

I am wondering, if something is built into Linux (Red Hat) that I can see traffic for a given hour, day, week? Or is it possible to mod the who's online to store it into a sql table? I just don't want to see good bandwidth used up by somebody trying to hack into my server or spider it for some other purpose.

Corey

[jonthewebb]

you can put the following in your .htaccess file:

deny from xxx.xxx.xxx.xxx

or more correctly

Order Deny, Allow
Allow from all
Deny from xxx.xxx.xxx.xxx

you can leave off the last one or two sets of numbers and that will block a range of addresses.

Personally I don't find that unwanted spiders take up much bandwidth anyway. Google visits my pages regularly but that's good, right?

As to keeping the records in your db, you could always change the clean-up time in the "who's online" functions:

includes/functions/whos_online.php

change line 33-ish to:

$xx_mins_ago = ($current_time - 900);

(something other than 900)

If you left it very long it would keep all the records. But you'd get a rapidly filling db if you left it too long.

In my experience it's mostly a waste of time running around trying to track and stop spiders. Easiest just to get a list of ips from the various sites available, eg http://www.iplists.com/ , the following is probably all the ones you'll want to bother with:

http://www.iplists.com/non_engines.txt

hope this helps



jon

[koltz]

But this specific one has been almost straight for 3 weeks from Microsoft. Could very well be a good spider, will look at the site, but when one translates to direcpc, I have to be concerned a little looking at the non_engines list. Thanks for the post.

[jonthewebb]

what is the ip? I would be interested to know and investigate a little.


jon

[koltz]

66.82.9.49 for the direcpc one, the other IP that is from Microsoft was listed, and went online, it is currently in the iplist.com under Misc. I just find it rather weird that when it is doing a spider search, that it needs to look at the same object sometimes 8 times. Really, it gets annoying, when you are looking at what is being looked at for most interest by real users.

[jonthewebb]

With a dynamic site like oscommerce the spider can appear to be looking at the same thing many times, while in fact there could be subtle differences, for example it could have "clicked" on a different language.

I see what you mean about not being able to tell what "real" users are looking at the most, I hadn't thought of that.


jon

[koltz]

That makes sense, just not the different languages part since I only have english activated. If this specific spider is still annoying me, I will drop it again. On just 3 items, last night, those had been looked at around 15 times each, cleared history, and again, I think those items again were looked at between 15 and 17 times since last night.

[jonthewebb]

maybe it's interested in buying your products......

[koltz]

It is a msnbot, which I believe is helping with the new msn search engine, but it isn't integrated for it to use yet. I will leave it, but gets quite annoying that it does it daily.