Kosmix is the search engine that produces “topic pages” on millions of subjects. Kosmix creates these topic pages by searching APIs of deep web sources (in real time). In other words, Kosmix relies heavily on federated search for the content of their topic pages. (Actually, Kosmix combines federated search with crawling technology. More about this later in this article.)

Kosmix co-founder, Anand Rajaraman, recently spoke at PARC, the prestigious Palo Alto Research Center. Rajaraman’s talk: “What lies beneath: harnessing the deep web.” A video of the hour-long talk is available at the PARC web-site. The slides are available at the Kosmix Blog.

Rajaraman has very impressive credentials. He is also co-founder of the VC firm Cambrian Ventures, he teaches a class for Stanford’s Computer Science department, and he is former Director of Technology at Amazon.com where:

he was responsible for technology strategy. Anand helped launch the transformation of Amazon.com from a retailer into a retail platform, enabling third-party retailers to sell on Amazon.com’s website. Third-party transactions now account for over 25% of all US transactions, and represent Amazon’s fastest-growing and most profitable business segment.

Rajamaran has his own Kosmix topic page.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

I was absolutely delighted to read a recent article by Barbara Quint, editor-in-chief of Information Today’s Searcher magazine. Federated Searching: Good Ideas Never Die, They Just Change Their Names reminds us that federated search existed before the term became popular:

Even back in the days when only professional searchers accessed online databases, searchers wanted some way to find answers in multiple files without having to slog through each database one at a time. In those days, the solution was called multi-file or cross-file searching, e.g. Dialog OneSearch or files linked via Z39.50 (ANSI/NISO standard for data exchange).

A little sidebar: I heard about this article from my brother Abe (founder, President, and CTO of blog sponsor Deep Web Technologies) before it came onto my Google Alerts radar. Abe was at the NFAIS Conference where he had gone to deliver a presentation on multilingual federated search. At the conference, Abe had a conversation with Iris Hanney, President of Unlimited Priorities, a support services company for businesses. It turns out that Barbara Quint is a member of the Unlimited Priorities team and produced this article for one of their publications, DCLNews. And, that’s how Abe heard about the article, in which he’s mentioned. Small world!

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

Check out this article at the Concordia University Library News Blog: Goodbye, BearHunt. BearHunt is (soon to be ‘was’) their federated search system. They didn’t like it and they’re not shy about telling you how much they didn’t like it. I’ve never seen so many negative comments about a federated search solution. I counted ten complaints. I’ve broken them out in my reproduction of Concordia’s article here:

Goodbye, BearHunt

QuickSearch via BearHunt will no longer be available beginning March 1st. There are a number of reasons that the library chose to discontinue this federated search service. Most important, as everyone is tightening their belts,

BearHunt came at a substantial cost while offering no new content.

Also important,

usage was low compared to cost.

When it first became available, we were excited to try a federated search, but along with other CLIC libraries and the rest of the library world,

we’ve realized that the technology needed to support the idea isn’t quite there yet.

The relevancy rankings don’t seem to work correctly or consistently,

links were frequently broken,

searches executed slowly, and

the interface left a lot to be desired.

We also felt

there was a possibility that it was misleading searchers to think they were searching ‘everything’, while in fact there are a number of both purchased and freely available resources that aren’t included in a BearHunt search.

Also,

the search is not complete in that it stops at a predetermined time interval instead of continuing until all searches have had a chance to complete.

In the end,

BearHunt and federated search as it now exists seems at odds with our mission to teach academic research skills tailored to specific situations and inquiries, using the full breadth of our resources.

We will look forward to new federated search products with anticipation, especially if they meet our criteria of completeness and have added-value in their presentation.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

David Walker received third place in the Federated Search Blog contest. David, library web services manager at California State University, produced a video titled “Using Metasearch to create a journal table of contents alerting service.” The video describes the work his library is doing to connect researchers to journal articles. The challenge is that while many publishers have alerting services to notify subscribers of new content, procedures for accessing the services vary greatly between publishers. Additionally, these publisher-provided services typically provide links to content that a researcher may not have permission to access due to authentication and location issues. David explains how combining a number of existing technologies overcomes these hurdles.

Here’s a link to David’s video.

If you enjoyed this post, make sure you subscribe to the RSS feed!

If you’ve got a half hour to spare, maybe in your car via iTunes, then you might enjoy this blogtalkradio interview at Friday Traffic Report: Exploring the Deep Web.

Friday Traffic Report host Jack Humphrey interviewed Bill Wardell about the deep Web. Wardell’s site, The CyberHood Watch Blog, aims to keep families and especially children safe on the Web.

While I know quite a bit about the deep Web, I enjoyed the conversational style within which a basic introduction was provided. I recommend this interview to those of you new to the concept of the deep Web and to new LIS students.

If you enjoyed this post, make sure you subscribe to the RSS feed!

[ Editor's note: Charles Knight received honorable mention in the second annual Federated Search Blog contest. In recognition of this honor, the Blog is publishing his essay and contest sponsor Deep Web Technologies is awarding Charles a $100 prize.

Charles Knight is a blogger and leading authority on alternative search engines. Recently the editor of AltSearchEngines, he now blogs about "All things Search" for TheNextWeb at http://thenextweb.com/search. He lives in Charlottesville, VA. ]

How Federated Search can Make You RICH! by Charles Knight

Unlike all of the other entries to this contest, this is the ONLY one that will MAKE YOU RICH.

Yes, you heard me right, this is a COMMERCIAL application of Federated Search technology.

Feel free to PATENT, BUILD, and SELL this product and then WATCH THE MONEY ROLL IN!

The FIRST STEP is to print out and tape in front of you this image of a beautiful Gemstone Globe.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

My radar (Google Alerts) pointed me this morning to this article by Barbara Quint at Information Today. My first response to “EBSCO Exclusives Trigger Turmoil” was “What a mess!” Quint shares the saga of EBSCO and Gale lobbing volleys at each other during the ALA Midwinter meeting. EBSCO announces new acquisitions that were ‘exclusive to EBSCO for the library “marketspace.”‘ Major competitor Gale issued a letter to the library community urging “librarians to get involved in opposing publishers granting exclusives, at least to EBSCO.” Read Quint’s article for all the gory details.

If you’re a librarian running or contemplating a discovery service, how do you feel? EBSCO has some new content I assume is going to become available via their EBSCO Discovery Service and some content is going to disappear from Gale’s holding which I assume means it will disappear from Serials Solutions’ Summon discovery service which includes Gale as a major participant.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

Blog sponsor Deep Web Technologies built a federated search tool for Stanford University. I was involved with the first prototype and I’m proud of what the Stanford/Deep Web Technologies partnership has accomplished. Stanford’s Grace Baysinger - head librarian and bibliographer for the Swain Library of Chemistry and Chemical Engineering - provides a nice overview of the search tool in the “Speaking of Computers” newsletter. And, Deep Web Technologies’ account manager, Debbie Taylor, shares a little bit of her involvement with the project as well on the Deep Web Technologies Blog.

My involvement with the Stanford federated search tool was multi-faceted. I worked with Stanford to understand their requirements. I also worked with Deep Web Technologies’ connector team to build the source interfaces Stanford wanted. I even used my limited web design skills to create the HTML that was used in the very first prototype. There were many conversations and many details that were worked out during a number of phone conversations. When I left my full time position of engineer and manager at Deep Web Technologies I lost touch with the folks at Stanford who led the way in introducing federated search to their university. I hear that they’re very happy with their search tool. I’m proud to have had a role in the building of that tool.

If you enjoyed this post, make sure you subscribe to the RSS feed!

I recently purchased a copy of How UK academic libraries choose metasearch systems from Emerald Insight. I was reading the article on my way to writing a review of it when I was struck by this statement:

Advice comes from successful, as well as unsuccessful implementations. One of the most common pieces of advice is that libraries should remember to treat metasearch as a service, not a product, and to ensure that there is adequate post-installation vendor support.

What struck me was that the statement, which seemed so obvious to me, was deemed important enough by the authors to state explicitly. Of course, federated search is a service. Whether or not you host federated search on your own servers with your own staff managing the server hardware, software, and application there’s plenty of service that someone will have to perform, and it’s on an ongoing basis. Off the top of my head, here are some service aspects to federated search:

• Building connectors - the more work that goes into building connectors the better the search experience
• Monitoring/disabling/re-enabling connectors when sources become unavailable and when they come back
• Updating or rewriting connectors when sources change their search interface
• Updating the application when the vendor provides new maintenance or feature releases
• Customizing the application to suit your needs

None of these tasks are easy. Each requires vendor support or time from your staff.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

Abe pointed me to an article, “An info island or the kitchen sink: when you think about federated search engines, think about what you’ll include in the searches.” While the article is a bit dated (2007) it has this air of timelessness to it.

The author, Richard Todd, considers three paths to source selection:

In the course of implementing a federated search engine at my organization, I eventually recognized three fundamental approaches for determining which of the available sources to include in our system. I call these three guiding concepts the kitchen sink approach, the Easter Island approach, and the gatekeeper approach.

Without reading further can you guess what these three approaches are? Abe told me that he was contacted by someone who threw out these terms as if Abe should be familiar with them. He wasn’t. Neither was I.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!