Jul
[ Editor's note: This post is by Abe. Information overload is a subject that's near and dear to his heart. Technology can simplify life or it can complicate it, or both. Federated search is no different. ]
An email that I recently received from Basex, a knowledge management market research consulting company, got me thinking about the role that federated search plays in addressing information overload.
Basex has created a non-profit organization, the Information Overload Research Group (IORG), whose mission in part is to “conduct research, help define best practices, contribute to the creation of solutions, share information and resources, offer guidance and facilitation, and help make the business case for fighting information overload.”
Tonight, July 14th, IORG is kicking off a conference on Information Overload at the Penn Club in New York City.
I was curious by what others have been saying about federated search and information overload so I entered these two phrases into Google and got back about 1700 results. Quite a few of the results (some referred to the same vendor) on the first 2-3 pages of results were from federated search vendors who proposed solutions that mitigate what otherwise might be the worsening of information overload that can be caused by federated search. I found a datasheet from Groxis titled “Information overload is good …” that emphasized clustering and visualization of results as a way to reduce information overload and a reference to an article published in Computers in Libraries that talks about SwetsWise (employs MuseGlobal technology) using content mining to help the user build more targeted queries that reduce the number of results returned by their federated search engine.
So what do I think about federated search and information overload? Knowledge workers are drowning in the ever increasing number of information sources and amount of information that is available.
On the one hand federated search can clearly reduce the amount of time and effort that a knowledge worker spends looking for information. Federated search can provide one stop access to quality information that reduces the level of information overload by reducing the amount of non-quality information the user has to review.
On the other hand federated search, by facilitating the search of more information sources than a user might otherwise search, could be seen as exacerbating information overload.
Deep Web’s Explorit federated search engine returns more results, typically 100, from information sources searched, more than most other federated search products. A typical broad search such as for “global warming” might return 1000 or more results.
So, is Explorit contributing to information overload? We don’t think so. We have spent a great deal of effort designing and implementing relevance ranking algorithms that display first the best of the best results returned by each search engine we search. We have also implemented a smart clustering capability (where results within a cluster are displayed in rank order) as another approach for reducing information overload. I expect that in the near future we will be implementing visualization technologies that also help to mitigate information overload.
Readers, please let me know your thoughts on this subject. I would love to hear from you.
If you enjoyed this post, make sure you subscribe to the RSS feed!
Tags: federated search
4 Responses so far to "Federated Search: Does it Mitigate or Exacerbate Information Overload?"
July 14th, 2008 at 6:47 am
Great to see your comments in information overload and IORG. One point of clarification: Basex is one of several companies that is forming IORG, an independent industry consortium. Others include IBM, Microsoft, and Intel, just to name a few.
July 15th, 2008 at 11:48 pm
As described, federated search does nothing to reduce information overload, just the opposite as Abe says. The mitigating factor for all federated search systems is that they all employ some form of results analysis to give the user the “best of the best’ right there up front.
In reality this is no different from an individual search engine producing its own secret sauce ranking. but…
The very fact that the federated search engines draw from multiple sources is their strength here. The universe of results from which they are extracting the best is broader than any single search engine. The number of results they analyse is probably smaller (even Deep Web’s 1,000 is pretty small compared to a search engine with millions of records in its database, and a non-specific query), but it covers a wider set original sources (journals, countries, choose your favourite), and points of view and time periods, and so on. The user still gets 10 records on page 1, but they are selected from a much broader basic range of sources.
So the chances are higher that a “good” record will be found first time. And that is an important federated search overload reduction - less second searches.
July 16th, 2008 at 9:44 am
Search is one of the critical technologies in finding information but the price for using search is information overload. A practical addition to any search results would be adding summaries permitting at a glance to see the essence of the text without the clutter of details.
I’m working with summarization technology and when I do Google searches, the results are automatically summarized and presented according to the keywords of my interest. Such search + summarization approach lets the user much quicker find out the relevant documents.
In fact summarization is a good filtering technology that can be a real asset in reducing information overload while taking better advantage of search engines.
July 10th, 2009 at 1:40 pm
In my job as a CIO, I’ve been working on tackling information overload with mixed results. My company, a professional services firm, suffers more than most because of a couple of infrastructure problems that arose from a couple of mergers.
I’ve been trying to get my colleagues to acknowledge that attacking our information overload problem will improve our overall knowledge sharing collaboration efforts and also contribute to our bottom line. But some people here just don’t understand the extent of the problem.
I just read about information overload awarenesss day and I’ve signed up our company as a participant and designated site - I hope this will get my point across to my colleagues and help them understand what we can do to improve our overall position relative to information overload. For others in my position (and I’m sure there are many of you) I encourage you to do the same, Information is available at http://www.informationoverloadday.com