Parts of the web are invisible to search engines — they may be pages that are created as a result of database searches, those that require users to register to view them, those located too deeply in large sites, ones that are explicitly blocked from search engine "spiders," ones prepared in unreadable file types or those that are password protected. They also include the portions of those that are simply too long to be completely indexed by the spiders.

Much of this content is freely available if you can find it. Even more important, much of it is more specific and professional than what is available on the visible web.

Start with What You Know

Consider possible resources. Go to their web sites and search directly. For example, if you need information about a president, going to that individual's Presidential Library is likely to turn up useful resources that may be too deeply buried within the site to appear in a regular search engine's results.

Go local. Often regional resources will provide the best information on a topic, but it may be buried deeply within these web sites. Check out the local newspaper sources and hometown pages. Many resources like local newspapers or publications will require a membership or password for access. Often these forms of access are free, after you register, however. Set up an extra free email account with a service such as Hotmail or Yahoo, and use it to register for services such as these, because they often generate unwanted email or advertising. Or you can take advantage of the services of BugMeNot to access a site you are unlikely to access again.

Search Engines

Add words like "database," "archives" and "repository" to a regular keyword search for your topic. A search for "presidents" and "database" provides access to a number of resources that do not turn up in a straight search on the term alone, including the government archive of all public presidential papers.

Specialized Search Engines