First of all I agree with the comments/observations that federated search operates with incomplete, “regurgitated” information as Tom points out in an earlier comment.

The problem as discussed by Sol and commenters to his post is two-fold: first a federated search engine may only bring back and analyze/rank a small subset of available results from a large information source which has many results for a given query. Note, that this is more of a problem with the user’s query than a federated search problem. For example if a user goes to PubMed (U.S. Government’s most popular database which also happens to return its results in chronological order) and searches for “cancer” the search is not going to bring back very useful results.

A federated search engine, ours included, relies to a great extent on the relevance ranking capabilities of the information sources being queried. We at Deep Web Technologies do a number of things to significantly improve the chances that we’ll bring back and find the most relevant documents that the user is searching for. We ensure that each of the connectors that we create is optimized (supports all search operators of the source, and supports advanced fielded search capabilities) for the information source that is searched and we bring back a larger number of results (at least 100 where possible) from each source being federated.

The second challenge of federated search is ranking of the results that have been brought back. How does the federated search engine know that the first result returned by one information source is more relevant than the fifth result returned by another information source? Almost 5 years ago I implemented the first of our relevance ranking algorithms, QuickRank, to an initially skeptical group of my customers. QuickRank which ranks results based on the occurrence of search terms within titles and snippets within a result has proven to work extremely well. No, it doesn’t ensure that the most relevant results are always returned to a user but much more often than not the best results are found and returned within the first page of results.

Results which might be highly relevant but don’t include the search terms in a title, author or snippet are returned as unranked results. Note that Google suffers from a similar type of problem in that a highly relevant web page, a “gem”, that hasn’t yet been discovered (i.e. doesn’t have a many links to it) is not likely to be found by someone searching Google.

Finally, as I discussed in one of my earlier posts, federated search is a great discovery tool for students and researchers who don’t know or don’t want to know where to search for information. Yes, if one could create one very large index of all the information that one might be interested in searching for, and have it indexed by a highly capable search engine, then this option would be preferable to federated search, but since this is not possible and is going to become less possible as more and more information sources become available what alternative is there to federated search?

Both Sol and Peter provide caveats that I agree with, but I will state here as I did in the presentation (although using different words) that a federated search engine in the context in which I was talking is performing whatever it does on regurgitated data from the databases it points to. If this data has been already ranked by the pointed to database, that will be the order in which the federated search engine receives (and in most cases) displays to the user. If additional processing is done on that data to provide some different type of relevancy ranking, the federated search engine is potentially operating with incomplete information (part of Sol’s point). If the federated search engine is sending the user’s search to several databases simultaneously, the outcome is even more problematic because:
1. The federated search engine may not be dealing with a complete result set from the database searched (Peter’s point); and
2. Each database may be performing relevancy ranking using different criteria for relevance.

It is not so much that it is impossible to conceptually design a metasearch engine that scours all of the results from the target databases and other resources and itself applies a relevancy algorithm to (potentially) re-rank the new uber-result set as it is extremely difficult to operationalize the concept given the constraints mentioned above and the fact that users want an immediate response.

Further and perhaps more to the point, the audience for this presentation was people working in a library context in which mega-bucks are spent on licensing access to non-crawlable resources, the breadth and depth of published literature. We may be talking about two very different types of federated searching targets: a) those which can be harvested in toto; and b) those which cannot.

The way I understand it, the individual database results may be ranked, but there are no standards to allow federated search engines to re-rank that information.

Since the federated search engine can’t see all 10,000 hits, the usability of relevance ranking is further impaired.