MegaEpix Enormous Presents
(with help from our friends @)
The Crazy World Under the Surface of the Internet
by the webmaster @ membrane.com
psehelp@membrane.com
I'm placing this page under 'Business' because I think that is the audience that will be most hungry for it. But, what I'm about to say holds true of for any webpage.
The Way You Search the Web
There are basically three different ways to find information in an orderly fashion over the Internet: - Locally Within a Given Website
- An Organized & Reviewed Directory (Like Yahoo)
- Spiders, Robots and Crawlers
Though the bulk of the traffic that our customers receive is from traffic we send them 'locally within our website', search engines that use robots are still very important. This page is devoted to discussing a recent experiment involving the robots of InfoSeek, Alta Vista, Excite, Lycos, Webcrawler, HotBot, Open Text, Northern Light, and Planet Search.
Robots, Spiders and Other Searchers of the Web
A robot (also known as a spider) goes out on the Internet... usually, at night... and crawls around on the world wide web... retrieving some (or all) information that it finds.
If you wanted to try to search for 'everything that is out there' on a given subject, then your best bet is to go to a search engine that uses a robot. The main disadvantage of this type of search is that you often get too much information. And, you are forced to sift through it.
These robots also have implications for a website that wants to be found, such as a commercial website. Of course, most of our clients want more than to just show up on these search engines... they want to show up first on every search. We've been conducting a variety of experiments on achieving this goal.
Every year since 1994, I've set up situations to watch the behavior of the robots. Every year I find out a wealth of new information. The details of our findings are too lengthy (and confidential) to publish here, but I will go over several of the general findings:
The 1998 tests show that the robots have become much less responsive and much less aggressive. Of the search engines tested in the initial survey, the only one to impress me was Northern Light. Their "Taz" robot tore our webtree to shreds. Which brings up another important point, robots can be hard on our systems. They tax our resources. Imagine a robot that comes and pulls every file that we have... hehehe... we'd choke robot, or robot might cause us to crash. (I point this out because many people that I speak with seem to think that search engines offer only advantages.)
Northern Lights had the only robot to truly index our test site. InfoSeek, though not aggressive, did prove to be very responsive. Excite was disappointing... in that last year it had proven to be the most aggressive. This year they performed with an 'average' showing.
© The Philadelphia Spirit Experiment Publishing Company
These graphics, images, text copy, sights or sounds may not be used without expressed written consent of the Glistening Web Communications Corporation.