Nexia’s Spiders Tracker
Stats, always stats, this is good, but hey, how can we track spiders without just pointing stats ?!
My Spiders Tracker is simple: each time a spider hit a page, the engine store that info in the dB… one hit, one entry. Ok, this add one query per page hit, on the other side, spiders have no access to most of the features of the vBulletin engine itself and they have 30% less queries for each page loaded than the regular visitors, so the server is not requested that much.
Why i do that tracker ?!… comparing to Google Analytics, my tracker will trace all activities from the spiders, making all their moves useful to see what pages are indexed, what pages are avoided, and where there is possible threats (because spiders usually visit any link they can, so a no-follow could be useful sometimes!)
This can also be a good engine for SEO… i suppose Lizard King can give more info on that..
…
So, what the system can track ?!
- each pages hit by a spider, with date and time
- each search queries coming from google, yahoo etc, and their page hit
What can we do with these:
- Verify the periods in days/weeks where spiders are visiting, making it easier to see where they are consuming server-side resources and what they search
- Verify how your sitemap is exploited by spiders (if you have one)
- Verify what pages are most visited (you can even put a page tracker displaying info like « this page was indexed X times by spiders »)