Feb
Back in December ACRLog, the blog for ACRL, ran a post titled Pay Some Attention To The Research. The blog post refers to two research articles. The first article referenced is titled “Portals for Undergraduate Subject Searching: Are They Worth It?” While that article is certainly worthy of attention, what hooked me about this blog post was the reference to and discussion of the second article, “Undergraduate Use of Federated Searching: A Survey of Preferences and Perceptions of Value-Added Functionality.” This article, by Belliston, et al, was published in the November 2007 issue of “College & Research Libraries.” While the full-text is only available from ALA to ACRL members, through the wonders of Google one can find a free copy of the article at Brigham Young University, where the research for this article was conducted.
Here is the abstract of the article:
Randomly selected undergraduates at Brigham Young University, Brigham Young University-Idaho, and Brigham Young University-Hawaii, all private universities sponsored by The Church of Jesus Christ of Latter-day Saints, participated in a study that investigated four questions regarding federated searching: (1) Does it save time? (2) Do undergraduates prefer it? (3) Are undergraduates satisfied with the results they get from it? (4) Does it yield higher-quality results than nonfederated searching? Federated searching was, on average, 11 percent faster than nonfederated searching. Undergraduates rated their satisfaction with the citations gathered by federated searching 17 percent higher than their satisfaction using nonfederated search methods. A majority of undergraduates, 70 percent, preferred federated searching to the alternative. This study could not ultimately determine which of the two search methods yielded higher citation quality. The study does shed light on assumptions about federated searching and will interest librarians in different types of academic institutions, given the diversity of the three institutions studied.
Much of what was in the report was of no surprise to me. What caught my attention, however, was the discussion of quality of federated search results vs. those found by searching the databases directly. As expected, only 11% of the audience at the ACRL conference where the findings were discussed indicated that they expected federated search to yield higher quality results. However, if you examine Table 3 (Comparison of Results Between Schools) in Appendix D: Librarian Created Quality Rubric, you’ll find a couple of interesting things:
- The differences in quality scores between federated and non-federated assigned by faculty and staff was actually pretty small. I’m not a statistician but the differences look pretty small to me.
- Faculty, in two out of four of the Brigham Young schools, assigned a higher score to federated search results.
Having been exposed to federated search technology for some time I am keenly aware that a number of factors influence the quality of results from a federated search engine. The one that is most often overlooked is the quality of the connectors that search the databases. I’ve heard stories of federated search users claiming that they actually got better results from a federated search engine searching a particular database than from searching the database directly. How is that possible? It is very possible when connectors are designed to do more effective advanced searches on the user’s behalf than the user would do for himself.
When a user is faced with searching a number of databases directly it becomes tiring to learn the idiosyncrasies of the advanced search form for each database searched. When he or she searches these same databases from the advanced search page of the federated search engine the user only needs to learn one search syntax so the user is more likely to fill that form out correctly. Then, when the connector performs a search, it can translate wild cards, Boolean expression, mapping of fields, phrasing, and other elements of the expression to suit the target database’s search engine.
Other factors influence quality of search results; a detailed discussion of these merits its own post. In terms of this study one needs to consider the quality of relevance ranking provided by the individual databases as this is often very poor, and the number of results analyzed by both the underlying search engines and the federated search engine. Also, clustering can impact perceived value of federated search results. And, you need to consider that not all federated search engines are created equal; some will return more relevant results than others.
Given all the variables that play into determining the quality of federated search engine search results I hope that everyone can appreciate my push to get vendors to produce publicly available demos and my desire that customers ask vendors for public demos that they can use to compare applications. If enough vendors played along, wouldn’t it be interesting to redo Belliston’s study with different federated search applications and see how the perceived quality of search results varied by product?
Read What is a connector to get a better understanding of what connectors do and how critical they are to obtaining good search results. And read the Brigham Young article to reinforce what you already know and to learn more about perception of quality of results.
If you enjoyed this post, make sure you subscribe to the RSS feed!
Tags: belliston, federated search
