Dr. Karl Kochendorfer makes a compelling case for federated search in the healthcare industry. As a family physician and leader in the effort to connect healthcare workers to the information they need, Dr. Kochendorfer acknowledges what those of us in the federated search world already know - Google and the surface web contain so little of the critical information your doctor and his staff need to support important medical decision-making.
Dr. Kochendorfer delivered a TEDX talk in April: “Seek and Ye Shall Find,” explaining the problem and solution:
Some highlights from the talk:
There are 3 billion terabytes of information out there.
There are 700,000 articles added to the medical literature every year.
Information overload was described 140 years ago by a German surgeon: “It has become increasingly difficult to keep abreast of the reports which accumulate day after day … one suffocates through exposure to the massive body of rapidly growing information.”
With better search tools, 275 million improved decisions could be made.
Clinicians spend 1/3 of their time looking for information.
And, the most compelling reason to get federated search into healthcare is the sobering thought by Dr. Kochendorfer that doctors are now starting to use Wikipedia to get answers to their questions instead of the best evidence-based sources out there just because Wikipedia is so easy for them to use. Scary.
Kosmix is the search engine that produces “topic pages” on millions of subjects. Kosmix creates these topic pages by searching APIs of deep web sources (in real time). In other words, Kosmix relies heavily on federated search for the content of their topic pages. (Actually, Kosmix combines federated search with crawling technology. More about this later in this article.)
Kosmix co-founder, Anand Rajaraman, recently spoke at PARC, the prestigious Palo Alto Research Center. Rajaraman’s talk: “What lies beneath: harnessing the deep web.” A video of the hour-long talk is available at the PARC web-site. The slides are available at the Kosmix Blog.
Rajaraman has very impressive credentials. He is also co-founder of the VC firm Cambrian Ventures, he teaches a class for Stanford’s Computer Science department, and he is former Director of Technology at Amazon.com where:
he was responsible for technology strategy. Anand helped launch the transformation of Amazon.com from a retailer into a retail platform, enabling third-party retailers to sell on Amazon.com’s website. Third-party transactions now account for over 25% of all US transactions, and represent Amazon’s fastest-growing and most profitable business segment.
Here’s a nice video interview. John Meyer of Silicon Prairie News interviews Will Bushee, VP of Development of BrightPlanet. I also found a link to a recent text interview with Mr. Bushee by Stephen Arnold.
BrightPlanet is a Deep Web search company. They’re best known for the seminal work that federated search luminary and BrightPlanet co-founder Michael Bergman did in sizing the Deep Web. The video provides an understandable introduction to the Deep Web plus discusses, at a very high level, what BrightPlanet does.
OSTI (the US Department of Energy Office of Scientific and Technical Information) recently joined YouTube and released a couple of short videos pertaining to the Deep Web. Disclosure: I consult for OSTI.
Deep Web Video
To make the web work better for science, OSTI has developed state-of-the-art technologies and services including a deep web search capability. The deep web includes content in searchable databases available to web users but not accessible by popular search engines, such as Google. This video provides an introduction to the deep web search engine.
I recently discovered a video at the Federal Library & Information Center Committee (FLICC) web-site titled Federated Search Technologies. The video (divided into two pieces) is over two hours long. It includes three presentations from members of organizations who have implemented federated search solutions:
• Betty Day, University of Maryland—MetaLib (ExLibris)
• Karen Huffman, National Geographic Society—AGent (AutoGraphics)
• Tamas Doszkocs, National Library of Medicine—ToxSeek
Abe discovered “Searching the Deep Web” on YouTube yesterday. This is a nice professionally produced 6-minute introduction to the Deep Web made by the US Department of Energy Office of Scientific and Technical Information (OSTI). Deep Web Technologies is mentioned in the video as DWT has created the search technology for a number of major OSTI applications.
What’s very cool about this video hitting You Tube is that Abe and I think of You Tube as hosting very mainstream videos. We like the idea of the public being exposed to federated search in such a venue.
For your viewing pleasure, here’s the video. Make yourself some popcorn, relax, and enjoy the show!
