As an operation I agree with the comments by Dave, Sol and Jonathan. LCD to me is literally where the Fed Search system issues a search of the same functionality (but possibly different syntax) to each Source. That implies that the functionality of the search is limited to that available from the least functional Source. I have deliberately used “functionality” to be as inclusive in the definition as possible. Indices, operators, relations, limits, even vocabularies are all examples of “functionality”.

Thus Dave’s A, B, C case is a perfect example of what I would call an LCD situation – in the second instance. Here the search is issued only as a full text search, because that is the only functionality supported by all three Sources. The other instance (where only A & B are searched) is dealt with below.

A couple of comments need to attach to this. Firstly it is interesting to ponder that the Fed search system must know enough about the different Sources to be able to determine what the “common” functions are. If it can do this, it is halfway(-ish) towards being able to handle Source Specific Searches (SSS). So why dumb down? Who knows? I don’t.

The second point is that the first of Dave’s instances (search A&B only) uses what we call a “strict” mode for the search. That is: if the Source can’t handle the search in it’s entirety, then fail for that Source, and return 0 results. This is not LCD, but rather a strange form of “HCF” (if we want to stick with basic arithmetic acronyms) where only those Sources which meet ALL the requirements (support all the functionality) are sorted. Different category, but a problem none-the-less for certain types of searches.

Where non-LCD searching is possible (SSS as used above) then it is possible to switch to a “relaxed” mode and allow some portion of the search to be processed and produce at least some results. (You want details – I knew you would. OK, the most obvious example is where a Source does not support a particular index and any terms for that Source are mapped to one it does support – in 99% of cases this is “keyword” or its functional equivalent. )

This mapped and relaxed operation does allow results from Sources which would be “0″s to a strict search and, in our experience, most people would rather get something back which is somewhat like what they asked for, rather than nothing. Often because they weren’t too sure about the query in the first place. Again a good result for some case, but not all. However take the union of the two cases and you cover what virtually everybody would want – and what you would get from the native Sources – always an important touchstone. (And this whole comment raise a host more issues for Sol to tease out into other posts – like the issue of notifying users of what the search has done.)

I think of the myth as the blanket statement that federated search engines rollover/suck because some sources are not easy to search well.

But, I’m interested to know what you all think LCD means. Abe and I discussed it for a while and realized that we weren’t quite sure what it means. So, please tell me.

At the same time there’s a different discussion about how some federated search engines handle tricky sources better than others. I’ll do what I can to find examples of smart connectors searching sources particularly well.

I am also curious to see any particular examples you can point to of federated search providers that manage to do well what you describe in this post.

Providing the kind of advanced features you require is exceedingly difficult (meaning expensive), and if you want to cover a large range of sources will take constant (expensive) maintenance.

And then, even when you’ve done your best, searchers can still receive unexpected results — even in the simplest case of the author search you give, both options are somewhat undesirable. Leaving a source out, the searcher may not realize the source has been left out and may think she has searched a source that in fact remains un-examined. Searching full text when the user asked for an ‘author’ search may produce results that in fact don’t meet the specifications of the search, frustrating the user. And then there’s sophisticated searching criteria like Dave mentions, a more common example might be various kinds of controlled vocabularies, from MeSH terms to molecular designations.

I’m a big believer in federated search as an important service to scholarly researchers, because the convenience justifies the flaws. But there certain inherent flaws that are difficult or impossible to get around.

Let’s say I federate full-text queries to sources A, B, and C. Sources A and B can filter on a geographic bounding box, but source C cannot. If a searcher gives me a full-text query term and a bounding box, I can only include results from A and B. If the searcher needs to query all three, she’s limited to full-text.

Federated search may still be very useful in this scenario, but the LCD problem needs to be recognized and understood.