Enterprise content is kept out of the surface web by design (not always all that well) so it is never going to be “surfaced” One could argue that the for-fee content of the publishers and aggregators is part of an enterprise data store, but it is really a “product” designed to be sold eventually. What I am talking about are the corporate internal resources - anything from HR records to secret recipes. They will stay hidden (or deep).

The volatile data is only deep because of its nature. Here today, gone in a flash. This data by its nature will never become part of the surface web. Although any one piece of information has a fleeting existence (the current temperature, a stock price, etc.) the data item is there long term, so it sort of flickers in and out of existence (not to get overly philosophical…) Interestingly one of the categories of these are search results where they happen to be displayed at the exact instant one of the search engines crawls that results page. A little cunning Google research will find a few interesting OPAC pages captured and indexed in various states. And, of course the advent of AJAX and friends means that some pages are partly static and partly volatile. And so it goes on.

One final question. Are Federated Search engines really associated only with the Deep Web? And if so should they be? Or should they range more widely?

What has been bothering me is how to frame discussing the places where Deep Web rubs up against Semantic Web.

Even though by far Google is the most popular free public search engine, it shouldn’t be measured based only on what _Google_ has indexed, should it?

But yeah, one way or another, I think more and more of the web will become indexed by free public search engines. For the portions that aren’t — and keeping in mind that this is not an ‘existential’ category, but just an ‘accidental’ one — perhaps ‘hidden web’? With some parts of it more hidden than others?

One part of the ’somewhat hidden web’ whose hidden nature is as much policy-based as technically-limited is for-pay licensed content.

In the academic meta-search market, providing access to this for-pay licensed material is one of the main motivators.

On the one hand, even vendors of for-pay licensed content and search indexes are trying to figure out how to expose their content to Google without giving away the cow for free. Sometimes this leads to odd arrangements where they are willing to share more with Google (for free) than they are with even their own paying customers, or to special arrangements with Google that they don’t allow other search engines to do. Or, on the other end, I’m guessing these special arrangements with Google sometimes involve Google giving them special dispensaton to violate a rule Google normally applies to the general public — allowing them to present a different version of a page to the google search engine that isn’t presented to the general public.

I’m not sure how long these kind of weird special arrangements can go on, but they’re trying to figure out their business model, same as everyone else.

The next time we are in a very interesting conversation (to me), please remember that I would be happy to search the “shallow” web for a nearby carwash so you don’t have to rush off so quickly! (smile)

I look forward to our next conversation. Perhaps we could ponder your proposed koan?

Deep Web Darcy