Comments on: Discovering discovery services

By: Eric Tull

Eric Tull — Fri, 31 Jul 2009 03:00:26 +0000

I would distinguish two types of sources to be pre-indexed by a discovery service. First are the full-text aggregators and other vendors who are in the business of directing users to their full-text materials. To do so, they would probably not see a problem with providing a discovery service with their index terms so that these terms can be incorporated into the discovery service pre-index, as the user will eventually be directed to their full-text.

However, database vendors who only provide indexing or abstracts, such as classics like Bio Abs, Sociological Abs, or PsychLit, are unlikely to want to provide this material to discovery services if it can then be provided to users whose libraries do not have subscriptions to these products.

To get around this problem I think discovery services will need to keep their pre-indexing separate for each database, with duplicates between databases tagged somehow. The results would then be merged on receiving a search request, with users only receiving results from the databases that their library subscribes to. The discovery services would need to keep up-to-date on a library’s subscriptions, presumably through information received from the database producers.

I would suspect that discovery service vendors will want to do this in any case, particularly as some are offering to include your library’s own databases – something they are likely to want to restrict to the one library rather than delivering them to everyone.

Such a system would also permit subject-specific subsets of databases, so that users could specify that the search cover all the databases their library subscribes to, but only for a specific subject.

I am not sure if developers of discovery services will discover this need on their own, but I think it is something libraries should push on them when these products are promoted to libraries.

By: Sebastian Hammer

Sebastian Hammer — Thu, 23 Jul 2009 22:52:50 +0000

I’ve seen the term Discovery Service used several times over the past couple of years, but this is the first time I have seen it used to denote something exclusively distinct from the kind of federated searching you discuss in this blog. I see it more frequently used in library environments to distinguish from the other, important piece of the puzzle: Delivery. We’re building solutions at the moment which are described by our customers as discovery services although they make use of a mix of locally indexed metasearching and broadcast searching — whatever best meets the needs… and at least one of the services you mention, WorldCat Local, is moving quickly towards adding a federated search capability (using some of our technology). If you’d asked me, I would have characterized a discovery service in library-land as a service that seeks to create a unified view of an organization’s information resources for the purpose of discovery. How it’s implemented under the hood is a different issue.

I think we need to be a little careful with these ‘fuzzy’ terms that mean different things to different people.

By: Mads Villadsen

Mads Villadsen — Mon, 20 Jul 2009 22:25:48 +0000

After reading this post I felt the need to place the product you mention into two different categories.

1) The ones where the unified search index is hosted at the local institution

2) The ones where the unified search index is hosted at a single service provider and the web interface (possibly) runs at the local institution

It may not seem like an important distinction but it does impact how much material can be made available.

It seems to me that the publishers are more likely to make deals about “handing over” their metadata (and fulltext) to the big players.

That is why the Summon sites search across 400 million records (much of it at the article level) and the Blacklight and LibraryFind sites only appear to search across a few million records.

I think one of the major problems integrated search has to overcome in the next few years to be successful is how do the local institutions get hold of the metadata and fulltext needed to create their own unified search index.

You are absolutely correct when you talk about the problems of source lock-in, and I agree with you when you say that federating discovery services can help overcome some of these issues. Both doing so will not give you all the advantages of a unified index, and for that reason I think it is important to keep working on improving the content of local integrated search solutions.

When a library signs up for a subscription at a given database or publisher they should try hard to get an agreement where they get access to the metadata and the fulltext – this will give them the greatest flexibility when it comes to deciding what they want to search, how they want to rank it, how they want to present it, and more.

I do realize that currently many publishers are resisting agreements like this since they probably see them as a threat to their core business. But hopefully over the coming years more and more libraries will want to be in control over how they present the data they have bought access to, and this will help pave the way for better agreements between publishers and libraries.

And I would of course also like to provide you with a link to a discovery service – the Search system at the State and University Library, Denmark:

http://www.statsbiblioteket.dk/search/

I don’t really like calling it a demo since it has been our main search interface since 2006.

It is a locally developed web interface on top of an integrated search system called Summa. More information about Summa can be found at these sites:

http://www.statsbiblioteket.dk/summa/
https://wiki.statsbiblioteket.dk/summa/

In short Summa is an open source integrated search system based on Lucene.

Disclaimer: I am the project lead of the Summa project.