Aug
This is the second installment of my interview with federated search luminary, Todd Miller. Part I is here. Today, Todd answers questions about the early days of WebFeat, about WebFeat and libraries, and about WebFeat’s contributions to federated search.
Todd is the second luminary that I recognize in this blog. You can find future and past luminary interviews in the luminary category. I invite you to nominate people who deserve to hold the federated search luminary distinction.
2. What motivated you to start WebFeat?
I wish I could claim that WebFeat was the result of divine inspiration, but the reality is that it was an accident. Several of us Knight-Ridder Information expatriates were trying to make a living doing IT consulting in NYC when one of our clients asked me one day if it was possible to search the New York Times and Wall Street Journal websites simultaneously. I asked our CTO (who was all of 24 years old at the time) if it was possible to search multiple websites in tandem. That was a Friday afternoon in April 1998. Our CTO went home for the weekend and came back the following Monday morning with the first prototype of WebFeat. I had been around the industry long enough to recognize the potential of the invention. From that point, we dropped all our other business in pursuit of federated search. I should mention that we later found that federated search was not nearly as easy as we had initially believed. It actually took us several years to develop a technology that was robust and fast and economical. Some of the most challenging technologies were those involving authentication and session management, for which we were granted a number of patents. I think most of the other companies that got into the federated search business went through the same curve we went through: premature elation at the apparent ease of federated search, followed by the sobering realization of the actual challenges involved.
3. Is there an interesting story as to how you came up with the the very catchy name WebFeat?
Not that I can recall, though there are many other interesting stories — most stemming from the fact that we were a 100% virtual office. We rarely met employees prior to their joining WebFeat, so it was always interesting when we would eventually match faces with names and voices. One of my current projects is a book about our virtual company.
4. What were the early days of WebFeat like?
Very tough. WebFeat was a bootstrap start-up. We had no venture funding whatsoever, so we had to fund our operations entirely from sales. The early days were very lean and pretty scary at times. There were a handful of dedicated employees that stuck with it through the tough times — they enabled WebFeat to survive and grow. Thankfully, their hard work and dedication paid off in the end.
5. In 2002, Library Journal published this article about you. The article says that you didn’t feel that libraries were anywhere near their potential. Is that still true today? If so, what challenges remain and what technologies can help libraries to meet those challenges?
I feel I could write volumes about this subject. The quick answer is a qualified “yes.” The qualification is a big one though — it really depends on one’s vantage to the mission statement of libraries. I am a businessperson, not a librarian, so I approach the mission statement from that perspective. As such, I think in terms of the potential opportunity that exists for libraries, stemming primarily from their capacity to fill huge voids in information access and, in the process, to become more of an imperative within our society. Right now, library budgets are typically on the short list to be cut during economic downturns, such as the one we’re experiencing today. If libraries played the role currently played by Google, that would be much less likely to happen. Libraries offer an extraordinary information resource to the public, yet most people are completely unaware of this treasure. I see tremendous opportunities to bridge the gap between what you can find on Google, and what you can’t. Libraries are ideally positioned to fill that gap and, by doing such, should become essential to society rather than tertiary. That is much of the unrealized potential, in my view. Having said this, over time, I’ve come to understand and appreciate that my mission statement for the library might not be the same as that of the information professionals who actually do the work. Additionally, I’ve gained a much better appreciation of the challenges facing any government funded organization. In summary, I think whether or not libraries have fully realized their potential is entirely in the eye of the beholder, and there are many views to that particular question.
6. What is WebFeat’s greatest contribution to federated search?
That’s a difficult question for me to answer, primarily because I think federated search is largely a bridge to somewhere else. That is not to say federated search is not important or unnecessary — it’s the only game in town right now when it comes to efficient and economical search of a large number of resources in tandem. It’s just that the real goal is, and has been, to enable fast economical searching of many (preferably all) sources of information in an elegant way. While federated search achieves the goal of searching many sources in tandem, there are some hard limits to its ability to do this elegantly. Interestingly, there are still many information professionals, particularly in academia, that question the value of searching many sources in tandem, regardless of how it’s accomplished. The view is that students’ brains will atrophy if you make it too easy to find information. To me, this thinking is similar to the 15th century reaction to the printing press — that minds would liquify because they no longer needed to memorize and regurgitate all the world’s knowledge. I think one of the key problems with this argument is a failure to take opportunity cost into consideration. The cost of our time is enormous and increasing, whether you’re a soccer mom juggling an overpacked schedule, or a student making choices between the value of study time, work time, and leisure time. To me the debate as to whether or not federated or aggregated search is of value is moot. The technology that enables more efficient and elegant access to information is an integral aspect of our increasingly rapid and more productive lives. If libraries don’t provide this capability, someone else will (like Google).
With that preface, let me try to answer your question. I think WebFeat’s contributions to federated search include:
- Advances in translator (or connector) technology. These include authentication and session management technology, as well as the ability to rapidly and economically build and update translators. Translators are the very heart of federated search, and a particular offering succeeds or fails based on the success of its translator service. This is a fact of life which is often overlooked in an industry which is sometimes distracted by bells and whistles. The bottom line is that if you can’t search the resources, the bells and whistles aren’t terribly useful.
- Hosted solution. We recognized early on that locally-hosted “shrink wrapped” federated search solutions were not a good way to go. In the early days of federated search, there seemed to be a preoccupation with federated search as a product instead of federated search as a service. Many libraries insisted on hosting their own solutions, even though there was no benefit to doing so — it complicated and delayed translator maintenance and did not boost performance due to the fact that the libraries did not host the content they were searching. Eventually, the library-hosted model gave way to the current vendor-hosted solution.
- Usage tracking. We recognized the importance of usage tracking early on. One of libraries’ goals for federated search was to drive up usage of the resources to which they subscribed. It was necessary to provide a tool that enabled libraries to track this progress. WebFeat’s SMART usage tracker provided a fast and easy solution.
- Advances in proxy technology. This enabled us to display native full records, preserving all native functionality intact within a vendor-hosted solution. It had the nifty side-benefit of enabling remote users to launch native search interfaces from the federated search menu without re-authenticating or requiring the use of a locally-hosted proxy.
- Self-service technology. WebFeat Express was a revolution in federated search, enabling clients to quickly configure their own federated search systems. It was a paradigm shift within federated searching, dramatically reducing its cost, thus putting it within reach of thousands of small libraries around the world.
- Enterprise technology. Designed for multi-library systems, this enabled library networks of all sizes to configure and maintain federated search systems with a great degree of granularity.
I’ll publish part III on Friday.
If you enjoyed this post, make sure you subscribe to the RSS feed!
Tags: federated search, Todd Miller, WebFeat
