Search strategy to Search that elusive Web page

We all have had times on the internet when we don't find the thing that we are looking for on the net.It is not only time consuming but also very frustrating. With the web growing every minute the number of pages is increasing to incredible levels. From 54,000 odd pages in 1994 to about 1.347 billion pages in 2001 to 8.058 billion pages in 2005 the growth has been phenomenal. How often we wished that someone could have classified things for us like our office secretary and kept them in small filing cabinet, subject wise so that without wasting time we could reach for it straightway.

Directories: Search engines crawl the net and store information about every site they visit. They don't categorise or differentiate. Humans all along have been making the effort to simplify and classify the information available on the Internet. This classifications have been called directories.

How are directories created: The Directory researchers work constantly to examine new web pages one-at-a-time. They classify all the pages they find to categories. These start with one main category and then keep going down to sub categories and sub-sub categories .e.g. The Home page of An American Museum of Contemporary Art may thus be classified as :

ARTS & ARCHITECTURE > ARTS > CONTEMPORARY ART > AMERICAN > MUSEUMS

The people working on the directory projects browse the new pages, make value judgements concerning the appropriateness of each page for listing, find a suitable category for listing, and exclude inappropriate content. This whole process is very much like what a librarian does when he selects, acquires, catalogs, and weeds books.

What is an open search: The most common way of searching the Internet is by using keywords. Type the keywords for the topic you are looking for in the search box of the search engine and see the relevant search results. e.g. searching for biography of Robert Ludlum you may type biography, Robert Ludlum. This type of search is called open search. (here we are not discussing the results of this search)

What is a directory search: In directory search we go form one main category to a category to a sub-category to a sub-sub category of the search directory, till we reach the desired level of the topic we searched e.g. searching for the same biography the path would be;

Arts > Literature > Authors > biography > (alphabet) L > Ludlum > topic

When to use open search:
1. When you want to quickly reach to relevant web pages and you have the keywords

2.When searching for all the freshly published content.

3.When you want all the pages on the topic without excluding anything.

When to use Directory Search:
1.When you dont have the keywords.

2.When keywords are not yielding the results.

3.When the open search is giving too many results.

4.When seeking for quality article/ web site on a specific subject.

5.When you want to exclude all junky pages from the search.

What are the top of the line directory projects: Compiling a directory is a time consuming job. Not every web site or Search Engine owner starts a simultaneous project of creating a directory. Looksmart was one of the early pioneers in this field and it subsequently tied with Altavista. Now they together claim to have the Largest directory on the Internet with 350 million indexed web pages. AOL.com is another big player which has a Human compiled directory. DMOZ is also in the field and attempting to create one of the largest open directories on the net. Wikipedia is the latest entrant and going very rapidly everyday.

We can only say in the end that search engines are essential but Directories are also important.Because the search engines will search and search and create chaos, it is only the humans who will bring sanity to this data.