Six years ago I moved from the Bay Area to New Mexico to be closer to my brother Abe and to his family. For five of those years I was an employee of his at Deep Web Technologies. Now, I just write for this blog and do some project work for him. Ever since starting to work for Deep Web, and to this day, I’ve supported DOE OSTI (The US Department of Energy Office of Scientific and Technical Information) in a number of capacities. OSTI is chartered to disseminate scientific and technical information to the public, especially as it pertains to DOE’s interests. OSTI has built a number of highly visible applications for this purpose, and some of these perform federated search and use technology developed by Deep Web.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

The first thing that most people notice when they use a federated search application is that it’s not nearly as fast as Google. We’ve all gotten spoiled. This is not only the information age, it’s the age of quick information; we all want every search to be as fast as a Google search. However, by its very nature, federated search can’t be as fast as Google. Federated search is at the mercy of the sources it federates. If a source is slow to return results to the federated search application, then there’s nothing the federated search application can do, or is there?

Deep Web Technologies has been displaying incremental results for some time now. The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results. While there are challenges to this approach, there are some significant benefits as well. The aim of displaying incremental results is to minimize the time the user has to wait to see some results. In the show-something-quick department, incremental results works well. The major challenge arises when you try to figure out what to do with the rest of the results as they come in.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

The U.S. Department of Energy (DOE) Office of Scientific and Technical Information (OSTI) has, ever since the Manhattan project, been responsible for stewardship of DOE-related research results, which it makes available for free to scientists, researchers, and the public. The OSTI blog was started last November to share personal perspectives of OSTI employees. Recently, the blog was expanded to include a technology thread. OSTI’s use of technology, much of it based on federated search, should be of interest to readers of this blog.

Due to my familiarity with OSTI technology (from five years of helping to develop and support OSTI products through my relationship with this blog’s sponsor, Deep Web Technologies), I was asked to write for the technology thread, being the sole author of some articles and collaborating author on others.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!