Steven Bell on the future of federated search

Steven J. Bell is the first runner up in the federated search writing contest. The aim of the contest was to predict the future of federated search. Below is Steven’s bio and his essay, in its entirety.

Steven J. Bell is Associate University Librarian for Research and Instructional Services at Temple University. Previously he was Director of the Library at Philadelphia University and Assistant Director at Penn’s Wharton School Library, where he also earned his Ed.D. He writes and speaks about academic librarianship, learning technologies and library management. An Adjunct Professor at Drexel University’s College of Information Science and Technology, he teaches the academic librarianship course. His website and blog, “Steven Bell’s Keeping Up Web Site” and “The Kept-Up Academic Librarian” promote current awareness skills and resources. Steven is a co-founder of the Blended Librarian’s Online Learning Community on the Learning Times Network and has participated in numerous virtual presentations. He blogs for ACRLog, ACRL’s official Weblog, and Designing Better Libraries, a blog about design thinking and library user experiences. He is co-author of the book “Academic Librarianship by Design”. For additional information about Steven J. Bell or links to his projects, point your browser to http://stevenbell.info.

Finding is Finally Here: Federated Search and the End of Searching

In 2008 I was just another clueless library school student. I first got interested in information retrieval as a college undergraduate. As a computer science major I spent a fair amount of time in my academic library, but I could never quite understand the search systems our librarians constantly tried to get us to use. For one thing there were way too many of them so it was a constant struggle to find the right one I needed. Then when I finally figured out how to use one of them my next research project on a different subject meant using another database – with a completely different interface. That meant learning an entirely new system, and wasting the time invested in that process. And no matter which one I used it was never as easy as Google. It was often an exercise in frustration. The only thing worse was when one of the librarians came to my class to explain how it all worked. Talk about confusion. One of my Google searches once turned up a quote by an old library techie who said something like “Only librarians want to search, everyone else wants to find.” I think that librarian was onto something.

So how did I go from being totally frustrated with our library systems and librarians to becoming one of them myself? It all started when the library rolled out a new technology that really captured my attention; a search system that could search nearly all of the library’s databases at once. My library called it SuperSearch. Others called it MultiSearch or even FindIt. I think if they could have called it Google they would have, but the name was already taken. In fact, the idea of an engine that searches many different sites and portals and compiles the results into one easy-to-follow listing sounds quite like Google. The big difference is that one worked rather well while the other didn’t. You know which is which. I wanted to be more involved in the future of federated searching.

By 2018 standards it’s almost humorous to look back at the early and rather primitive federated search systems we had in 2008, and even those were vastly improved from the systems of just a few years before. Here are a few of the features of those old engines compared to the ones we use in 2018.

Early on, convincing students to use federated search was no easy task. For too long these complacent searchers limited themselves to the one database they knew well or the one some professor told them to use. What’s the point of having 400 or 500 databases with hundreds of thousands of full-text articles if you have no idea what they represent, when to use them, why they are useful or how to get at them? Federated search was supposed to eliminate this what, when, where, why and how conundrum, and replace it with who cares – just get good results - fast. The point was to focus energies on the results and engaging with it for research and learning. But in 2008 federated search was often less effective than what it was supposed to replace or supplement. It was clunky, slow and limited. Who could have expected it would improve so vastly over the past 10 years.

The first big improvement for end-users came in 2010 with predictive database selection. Librarians were always challenged to devise preset categories with just the right databases so that a search of the “sociology” category would hit all the right databases. The only problem was that end users rarely thought the same way as librarians. Choosing the right category was quite a challenge. Once it became possible to set preferences in the federated search software the system would track every end-user search. Once users identified themselves to the system all they needed to do was begin the search. Selecting the right mix of databases was left up to the system. Sure, this meant giving up a bit of privacy but realistically, by 2010 privacy was a fairly obsolete concept anyway.

Touchscreen had been around for a number of years, but by 2011 the makers of federated search had adapted and perfected it for their main interface. One of the things that made early federated searching so painful was what happened after you finally hit the search button. The result displays were anything but intuitive. The confusion centered around the systems’ inability to clearly distinguish where the results were coming from, why they were retrieved in the order displayed and how to begin customizing the results from the initial display. For example, let’s say I just searched eight databases. Then I realize databases number 3 and 7 have great results. Now I need to delete all the other results, and I want to use subject headings with just my two top choices. Completing this sequence as described is possible, but to do so would require the sort of clumsiness and pain that would leave most searchers running for Google. But touchscreen made it possible for the software makers to visually and tactilely move around the results and resources to their optimal configuration as defined by the searcher. If I wanted the results from databases two and six then I just swiped away all the others with a glide of my hand. Then I touched the screen to integrate more search techniques from the native mode interfaces. It was really quite easy and intuitive for the users.

For hardcore librarians the big knock against federated searching was always the loss of features found in the native databases. Even something as simple as a limit to any group of database’s controlled vocabulary was exceedingly difficult. These losses were just a tradeoff one made for the ease of federated searching. It took until 2014 to eliminate this barrier to creative search. Of course it helps that in Web 5.0 the translators are no longer limited to the most basic fields, but can now rapidly process searches targeting multiple fields across databases. With the addition of predictive search in 2015 it became even less necessary for the end user to even know what search fields were or how they contributed to better search results. They could simply enter their search, using either hand or voice input, and the system would visually present multiple search field options. Want to search by author, image or words in article titles? With just a touch of the screen or voice command any combination of search fields, across any number of different search systems, could be combined to produce precise results. We were finally reaching the stage where consistently reliable and accurate federated search results were on the cusp of becoming reality.

Federated searching was really coming into its own by 2016 although only two companies remained that sold the product. It was ultimately the Google subsidiary, a firm in the Library Division called NCP, that brought federated search to its apex with the addition of artificial intelligence technology that all but eliminated the need for anyone to think at all about their search. Users only needed to provide a description of the information they wanted. They could dictate a sentence or paragraph, type it in or even upload it from a chip to the system we called FindAll at my library. We were finally getting to the point where the people using our library could find what they needed, with a good degree of accuracy, without the need for the thinking or planning that library database searching required back in 2008. Federated search could do the thinking that went into the searching.

Companies like NCP did well at adding snazzy features to their federated search system, for example, the ability to wirelessly transmit the documents directly to our skin-implanted storage chips. The bottom line is that people only care about getting back fast and accurate results without having to think much about the process. You think people were busy multitaskers in 2008? We weren’t even allowed to drive and text message at the same time back then. That seems primitive now. Back in 2008 when it came to finding information people wanted more finding and less searching and that is what the federated search of 2018 delivers. All along we wanted to eliminate real thinking from the process and now the technology makes that possible. I’m actually proud to be a librarian now. Come to think of it, it’s not really federated searching anymore. It’s time we started calling it Federated Finding.

If you enjoyed this post, make sure you subscribe to the RSS feed!

Tags: federated search