Introducing Erik Selberg: federated search luminary #4

In 1994, Erik Selberg created MetaCrawler, one of the earliest metasearch engines. Erik’s MetaCrawler was truly revolutionary in doing something we all take for granted today: querying and aggregating results from a number of search engines. Today, the work that Erik pioneered lives on in modern metasearch and federated search engines. For his vision and for his important contributions to and influence of today’s federated search technology, I honor Erik Selberg as the fourth luminary, standing together with Kate Noerr, Todd Miller, and Michael Bergman.

Erik was gracious enough to entertain a number of questions. Here’s the list:

1. In 1996, you co-authored a visionary paper: The MetaCrawler Architecture for Resource Aggregation on the Web. The architecture you describe looks very much like federated search engines of today although I should note that your work was focused on metasearch engines, which federate search engines that crawl and index web sites rather than federating content that lives in databases. What background did you have that motivated that paper and enabled you to write one of the first metasearch applications?
2. What inspired you and Oren Etzioni to create MetaCrawler in 1994?
3. Whose idea was MetaCrawler?
4. What role did you and Etzioni each have in creating MetaCrawler?
5. How was MetaCrawler different than its predecessor, SavvySearch?
6. As far as I can tell, MetaCrawler was the second metasearch engine (after SavvySearch). Am I right?
7. What would you say were the most important features of MetaCrawler?
8. How long did it take you to build MetaCrawler?
9. What were your biggest challenges in implementing MetaCrawler?
10. What was the business model for MetaCrawler when you first developed it?
11. Wikipedia makes this interesting statement about MetaCrawler that I bet few people realize:
    

    “Originally, MetaCrawler] was created in order to provide a reliable abstraction layer to early Web search engines such as WebCrawler, Lycos, and InfoSeek in order to study semantic structure on the Web.”

    Can you say more about the abstraction layer and about your study of the semantic structure on the web?

12. What do you see as the major differences between MetaCrawler and today’s federated search applications?
13. Which are the best metasearch engines today?
14. Is there a relationship between MetaCrawler and Dogpile?
15. How would you compare MetaCrawler and Dogpile?
16. What do you think are the major challenges of federated search today?
17. What do you think will be different about federated search in ten years?
18. University of Washington has a web page about MetaCrawler and two applications based on MetaCrawler: HuskySearch and Grouper. The page explains that HuskySearch was developed to study query refinement and Grouper implemented clustering. What became of those two applications?
19. According to the article about you in Wikipedia, you’re a senior manager now at Amazon.com. What kinds of projects are you involved in?
20. Before Amazon.com, Wikipedia says you were a senior software developer at Microsoft working on Live Search. What was your contribution to search there?
21. Does the current search engine at MetaCrawler.com bear any resemblance to your original MetaCrawler?

The interview begins Friday.

If you enjoyed this post, make sure you subscribe to the RSS feed!

Tags: erik selberg, federated search, metasearch