Friday, January 1, 1999

Search engines: bring ’em on!


By James D. Cimino

In my last article, we talked about some of the "tools of the trade"; ways to help drive traffic to your Web site. In this article, we'll talk about how to make your Domino Web site Internet search engine "friendly" and how to tweak your pages so they will get indexed.

Search engines

Also called "spiders" or "crawlers", search engines constantly visit Web sites on the Internet in order to create catalogs of Web pages. It usually requires no special work on the part of Web masters for their sites to be included in the catalog created by search engines (although, without any special attention, quality, position, and description all is left up to the whims of the gods). Alta Vista is an example of a search engine. Table A lists the leading search engines, and some of their characteristics.

Table A: Leading Search Engines and their Characteristics
ServiceContentMeta Tag SupportIndex DatePage Popularity Used?Catalog UpdateSpam Penalty
Alta Vistafull-textYesURLNo4 -6 WeeksYes
Excitefull-textNoNoSome4 -6 WeeksYes
HotBotfull-textYesYesNo4 -6 WeeksYes
InfoSeekfull-textYesURLNo4 -8 WeeksYes
LycosabstractNoNoYes2-4 WeeksYes
WebCrawlerfull-textNoNoYes6 -7 WeeksYes

Full text or abstract

The content entry in Table A shows whether the engine catalogs the full-text of a page or instead creates a description or abstract based on a page's text. Full-text may be better because it ensures that every word from your Web site will be available to match keywords entered by those consulting search engines. "Mostly full-text" generally means that an engine may exclude certain common words.

Index date

The search engines may go out nightly to find new Web pages, but that work means nothing if the search engine catalogs aren't updated with this new information. From the index date, you can get an idea of how long it might take for your site to show up.

Page popularity used

Some search engines determine the popularity of a page by analyzing how many links there are to it from other pages. Pages that are more popular may turn up earlier in a search over less popular pages, or popularity may be used in other ways. [There are some engines, like, that list sites that pay a fee higher in the search results than those who don't. This is a misleading practice that can prove very disappointing to users. --DG]