[ Editor’s note: This post is by Abe. Information overload is a subject that’s near and dear to his heart. Technology can simplify life or it can complicate it, or both. Federated search is no different. ]
An email that I recently received from Basex, a knowledge management market research consulting company, got me thinking about the role that federated search plays in addressing information overload.
Basex has created a non-profit organization, the Information Overload Research Group (IORG), whose mission in part is to “conduct research, help define best practices, contribute to the creation of solutions, share information and resources, offer guidance and facilitation, and help make the business case for fighting information overload.”
Tonight, July 14th, IORG is kicking off a conference on Information Overload at the Penn Club in New York City.
I was curious by what others have been saying about federated search and information overload so I entered these two phrases into Google and got back about 1700 results. Quite a few of the results (some referred to the same vendor) on the first 2-3 pages of results were from federated search vendors who proposed solutions that mitigate what otherwise might be the worsening of information overload that can be caused by federated search. I found a datasheet from Groxis titled “Information overload is good …” that emphasized clustering and visualization of results as a way to reduce information overload and a reference to an article published in Computers in Libraries that talks about SwetsWise (employs MuseGlobal technology) using content mining to help the user build more targeted queries that reduce the number of results returned by their federated search engine.
So what do I think about federated search and information overload? Knowledge workers are drowning in the ever increasing number of information sources and amount of information that is available.
On the one hand federated search can clearly reduce the amount of time and effort that a knowledge worker spends looking for information. Federated search can provide one stop access to quality information that reduces the level of information overload by reducing the amount of non-quality information the user has to review.
On the other hand federated search, by facilitating the search of more information sources than a user might otherwise search, could be seen as exacerbating information overload.
Deep Web’s Explorit federated search engine returns more results, typically 100, from information sources searched, more than most other federated search products. A typical broad search such as for “global warming” might return 1000 or more results.
So, is Explorit contributing to information overload? We don’t think so. We have spent a great deal of effort designing and implementing relevance ranking algorithms that display first the best of the best results returned by each search engine we search. We have also implemented a smart clustering capability (where results within a cluster are displayed in rank order) as another approach for reducing information overload. I expect that in the near future we will be implementing visualization technologies that also help to mitigate information overload.
Readers, please let me know your thoughts on this subject. I would love to hear from you.
Tags: federated search