via Federated Search – (Daniel Tunkelang):[ Editor’s Note: This is a guest article by Daniel Tunkelang. (See his bio below.) Daniel is passionate about designing search systems that improve users’ experience with information retrieval. This passion …

Facets are an attempt to extract the semantics of a set of results. And they work pretty well whether pre-coordinated against a vocabulary of some sort, or are just the natural outcome of the retrieved documents. Normalizing these semantics is the big problem as Daniel points out.

Returning not just the facet values, but also the terms which have been used to derive them, does indeed provide for the very useful possibility that the federated search system could derive common facet values across a number of Sources. They would of necessity be “fuzzy” in that the supplied terms would not be co-extensive across the facets across the Sources, but they would probably be fairly good approximations. And much better than nothing at all.

Processing would be increased all round, but probably not by an unacceptable amount, and the result could be not only a “normalized” set of facet values for the user, but also a set of documents which could be de-duped or clustered on those values. This should lead to a richer set of documents for the user.

I await the data so we can process it and see.

Daniel states: “In particular, this approach to federation necessarily assumes a lowest common denominator of search functionality–a consequence of the requirement to evenhandedly broker among a variety of search applications that vary in the richness of their APIs”

This assumption of LCD searching is taken as an obvious truism. But why?

If federated search systems are presumed to be capable of handling multiple record formats for data extraction from retrieved records, then why should they not be considered capable of generating Source specific search statements?

The reason (as for federated search itself) seems to be that most don’t because it is yet another messy thing to deal with on a Source by Source basis. Note that here I am talking about more than adding blanks and quotes to a search statement.

The next step is to be more aware of the actual search syntax. Is the index for an author search represented by “au=” or “/au” or some other string? This is a fairly common, but not universal, capability – many systems restrict their interaction to the language of standards such as RPN for Z39.50, or a commonly implemented search language such as Open Search.

Moving to proprietary APIs, and also to web search interfaces, provides the much larger variety Daniel posits as beyond the reach of federated search systems.

It ain’t necessarily so. There are federated search systems which match the search to the capabilities of the Source engine. Applying limits where they exist, using indices where they exist, mapping to alternatives where the requested function is not available. All under user control for the desired strictness of the query. Acting very much like a user would in the same circumstances – adapting the query to what the Source can handle.

This addresses a second point in the quote above: that the desire is to have an identical query sent to all Sources, rather than the “best” one each can handle. Why? It is contrary to the way users act. They attempt to get the best results from each Source adapting to what it offers in the way of search tools. Surely federated search systems should try to do no less.

Admittedly, very few federated search systems do go to these lengths, but some do and, of course, we believe that Muse provides one of the more advanced capabilities of this type, or I wouldn’t be mentioning it.

The bottom line is that some federated search systems do adapt the searches to the richness, or otherwise, of the Sources they access. The LCD approach is not a necessary evil, but a chosen one.