The Invisible Web

I have been working on the net for last several years. I always thought that when I needed to find anything on the net I could use Google, Altavista, Hotbot, Yahoo, Excite, Snap, About, AOL, MSN, Looksmart, Web crawler, Ask Jeeves, Northern Light and covered almost the entire gamut of the Internet for my searches. It was one day when going through a job advertisement on Web Researcher, in qualifications column, I saw ...."knowledge of Invisible web required....". Now what was that..Invisible Web...? I had never heard of it before. Research on the topic produced startling information which I share with you.

What exactly is Invisible web: A vast majority of information available on the Internet does not reside directly on the World Wide Web. It is found in hidden databases that cannot be seen or searched by common Internet search engines. There are many many such databases This vast Ocean of databases is commonly referred to as the "The Invisible Web". It is called Invisible as it is not searched through commonly used search engines. It is also called "Hidden Web" as this part of the Internet hides inside the Data Base. Some People call it the "Deeper Web" because most of the search Engines only search (spider/crawl) the surface of the Internet.

Why is this part of the Internet Hidden/Invisible: To understand this we must understand how search engines work. Search engines visit (crawl, spider) the web servers registering the addresses of Web pages they discover. When they come across a database, they read and index the home page of a database. They do not go inside that database to find out what is stored therein. Thus search engines don't distinguish between the index page of a huge data base and a simple Web page. Although information is there in the Database it is invisible to the search engine because of its technology. Another reason is that the HTML pages we normally post on the Internet are fixed or static pages. Pages of the databases are more active and offer content that is put together from many parts of the database. These are called dynamic pages. Commonly used Search Engines cannot index dynamic pages and so the databases remain hidden.What is the extent of the Invisible web: The Google Search Engine mentions indexing 28.0547 billion pages. This at the moment is considered to be one of the biggest indexing of Web pages. Bright Planet a company which has brought out a white paper on the subject estimates this to be 800 billion to 950 billion web pages. This translates to up to 500 times of the visible web. The Invisible Web contains 7,500 terabytes of information, compared to 19 terabytes of information in the commonly searched Web. There are an estimated 1,00,000 Invisible web sites and growing.

What is the significance of this concept to Web Masters: As web masters especially those working with the data bases it is very essential to know that a conventional search engine will not index their site/ web page. More likely it will only index the Index page of the entire data base once. There is, as yet, no solution to this problem of data base not being found through a conventional search engine.

Why is Invisible Web important: Most of the Invisible Web contains information that can be publicly accessed and available for free. This information is extremely relevant for domain names, market research,and is commercially relevant information.

Gateways to Invisible web: Now the question arises how to search the Invisible web. One way is through gateways, which are the interfaces giving direct access to the thousands of searchable databases on the Web that the major search engines ignore. Some of these are Alpha Search, The Big Hub, WebData.com, Digital Librarian etc.

How to search the Invisible Web: Besides gateways there are specific search engines which search the Invisible web. Most significant amongst these are Lexibot, Infomine, InvisibleWeb

contributed by:Kainaat creations