Federated search in 2008: The year in review

[ Editor's note: Deep Web Technologies President, founder, CEO, and blog sponsor, Abe Lederman, continues a tradition he began last year of reviewing key events of the year. ]

I can’t believe that a whole year has gone by since my first Year in Review post. A lot has happened in 2008. In this post I’d like to highlight a few of the most significant events of the past year.

First of all I would be remiss if I didn’t mention that in early December the Federated Search blog turned a year old. The blog is going strong and has become the most authoritative, comprehensive resource on federated search on the Web. Sol has written a wonderful article detailing his experiences authoring the blog. Sol’s article is being published by FUMSI this month.

One of the more significant events of 2008 was the acquisition in February of WebFeat by ProQuest, parent company of Serials Solutions federated search vendor. I remember arriving at my hotel in Boston on Thursday evening, February 14 (yes, I was traveling on business on Valentine’s Day) and starting up my email and seeing a flurry of messages about the acquisition of WebFeat. I quickly got on the phone with Sol and we brainstormed on the significance of the announcement. A couple of hours later Sol published his assessment in the federated search blog.

Today, the WebFeat and Serials Solutions products are yet to be fully integrated although I hear that’s coming later this year. This event was significant because it decreased the number of low-cost options for federated search to be deployed in lots of libraries to mostly just one vendor. The acquisition of WebFeat by ProQuest also meant that a major federated search vendor (i.e. unaffiliated with a content publisher) was no longer independent. Sol’s post on the acquisition speculated that we might see federated search costs rise following the acquisition. This hasn’t been the case as far as I can tell.

Microsoft recently introduced federated search in Windows Vista and in Windows Server 2008. Before that, Microsoft announced federated search features in Microsoft Search Server 2008. I wrote about this last year. I believe that Microsoft is trying to further position itself as having the federated search of choice (in the enterprise), in direct competition with Google’s Search Appliance. For the industry, I think it’s a good thing that Microsoft aims to make “federated search” a household term. A potential downside, however, is that those not familiar with the technology may view federated search as something that should be much cheaper than it is. Neither Microsoft nor Google’s federated search capabilities are much of a threat in the library.

This year saw renewed interest in the “deep web”. In April Google announced that they were unlocking the “deep web” by filling out forms and harvesting retrieved content into their index. Since the announcement came from Google my inbox was inundated for the next several weeks with copies of the Google press release and/or articles commenting on this Google effort. The reality as explained by Sol’s blog post is that there is little chance of Google being able to systematically and comprehensively add significant portions of the “deep web” to their index. Later in the year Infovell, which soon afterwards became DeepDyve, announced that they were unleashing the “deep web”. For some reason DeepDyve marketing folks claimed that they were the first to make deep web content available to researchers. Neither Google nor DeepDyve are using federated search technology to access the deep web. However, federated search is the best technology available to unlock the treasure troves of content in the deep web, so this renewed interest in the deep web is really positive.

This fall we saw an unprecedented meltdown of the U.S. economy. It is not clear yet what the impact of the economy will be to the federated search industry in 2009. We have seen public libraries being shut down (or attempted to be shutdown). It is unfortunate that federated search is viewed by some as a nice to have, and by others as not even necessary. Carl Grant, in one of my favorite blog posts of 2008, argues against the fallacy that federated search (or metasearch as referred by Carl) is not really needed and that users would be better served by learning to use the native interfaces of the information sources they are searching.

I believe that we in the federated search industry, particularly in this economic climate, must make a stronger case for the value of federated search. We must show that federated search provides a strong ROI to organizations that acquire federated search solutions. We must show that federated search significantly reduces the amount of time that researchers spend looking for information. Not only will researchers spend less time looking for information but they are much less likely to miss finding important results, either because they didn’t search the source containing the critical item of information, or because this critical item was buried in one of the databases being searched.

2008 was a good year for Deep Web Technologies. Software engineering was kept busy making significant enhancements to our technology, including developing a new Ajax-based User Interface, adding a clustering capability to Explorit, and releasing a major upgrade to our Alerts product. Major upgrades were made to a number of the portals that we’ve developed including Science.gov, WorldWideScience.org, Science Accelerator, Scitopia.org and DTIC MultiSearch. In June, Sol and I attended a ceremony in Seoul, Korea where WorldWideScience.org was turned over to ICSTI. Late in the year we launched the first two of our vertical portals, Biznar and Mednar. I was excited to learn while on vacation, just last week, that Mednar was chosen as the best new health search engine by AltSearchEngines.com.

I wish you all a happy, healthy and prosperous 2009.

If you enjoyed this post, make sure you subscribe to the RSS feed!

Tags: federated search