Home » Writing & Publishing » Library/Information Science » The Invisible Web

The Invisible Web

© Gillian Davis

If you're an information professional or a fairly experienced Web user, you’ve no doubt heard about the Invisible Web. But what exactly is the Invisible Web, and how concerned do you need to be about it?

Many Web researchers have been lulled into thinking that search engines will find whatever we need. Most of us now know that each search engine indexes only a small portion of sites available on the Web, and that in order to do a thorough search, we need to repeat our search in a number of different search engines. But even some of the most experienced among us don’t realize that despite this search strategy, we still may be missing out on a significant portion of information available on the Web. That's because general search engines - no matter how many you use and how great they are individually - simply cannot find everything on the Web. As surprising as it might seem to some, there are vast amounts of quality material that you're not getting at with your usual methods of Web searching.

The Invisible Web has been defined as that part of the Web that is comprised of specialized, searchable databases. Because the pages of data in these databases are created dynamically at the time of the database search, (or they are .pdf documents or login restricted), they are not stored anywhere on the Web on a Webpage. And since they are not Webpages, they aren't picked up by search engine spiders and therefore aren't retrieved by general search engines. Simply put, the Invisible Web is made up of unindexable Web content that is not accessible via common Web search tools. It's a very large part of the Web (some say as much as one third of the entire Web), and it’s growing quickly.

How concerned do you need to be about the data contained in the Invisible Web? The answer to this question depends on your perspective and your research goals. If you're a casual Web user and searcher, or if you only need a broad overview of a topic, it may not be so important to delve into the Invisible Web. It's quite likely that all the information you'll need for your purposes is readily available in the "regular" Web. But if you're an information professional, someone who needs to do serious and thorough research, or someone seeking an answer to a very specialized and specific question, then you definitely should spend some time learning about the Invisible Web.

The copyright of the article The Invisible Web in Library/Information Science is owned by . Permission to republish The Invisible Web in print or online must be granted by the author in writing.