This week the Science.gov Alliance released Science.gov 5.0. The release got a good amount of press from a couple of press releases (from the US Department of Energy and Deep Web Technologies) and from a number of bloggers, including Valerie Allen (Product Manager for Science.gov) on OSTI’s own blog and the SLA Government Information Division blog. When I worked for Deep Web, I supported the application and have enjoyed watching it evolve and grow.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

We’ve all heard the old adage, “Don’t believe everything you read.” The Internet is full of stuff to read; how do we know what to believe? While there are numerous search engines that present us with documents in response to our queries, how do we know if the information presented in these documents is accurate? Granted, much of what’s in the Internet is personal opinion and sometimes all we want is someone’s viewpoint. There are times, however, when we need to know that the information we are reading is of high quality. We may be researching product features to make a purchase decision, company information to form competitive intelligence strategy, or medical information to address a medical concern.

A major part of the answer to the question of whether information is accurate or not is to examine its source. This is where federated search engines really shine. By their nature, federated search applications usually query deep web database sources. The databases can’t be crawled. There are no links for Google to follow to extract all documents in such a database. Now, let’s consider the type of content that lives in these non-crawlable databases. Publishers who specialize in scientific, technical, and business research articles are most likely to store their documents in databases and to make their content searchable by federated search engines. Geological, geographic, demographic data lives in databases. Much political data lives in databases as well.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

The first thing that most people notice when they use a federated search application is that it’s not nearly as fast as Google. We’ve all gotten spoiled. This is not only the information age, it’s the age of quick information; we all want every search to be as fast as a Google search. However, by its very nature, federated search can’t be as fast as Google. Federated search is at the mercy of the sources it federates. If a source is slow to return results to the federated search application, then there’s nothing the federated search application can do, or is there?

Deep Web Technologies has been displaying incremental results for some time now. The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results. While there are challenges to this approach, there are some significant benefits as well. The aim of displaying incremental results is to minimize the time the user has to wait to see some results. In the show-something-quick department, incremental results works well. The major challenge arises when you try to figure out what to do with the rest of the results as they come in.

Read the rest of this entry »

If you enjoyed this post, make sure you subscribe to the RSS feed!

Abe discovered “Searching the Deep Web” on YouTube yesterday. This is a nice professionally produced 6-minute introduction to the Deep Web made by the US Department of Energy Office of Scientific and Technical Information (OSTI). Deep Web Technologies is mentioned in the video as DWT has created the search technology for a number of major OSTI applications.

What’s very cool about this video hitting You Tube is that Abe and I think of You Tube as hosting very mainstream videos. We like the idea of the public being exposed to federated search in such a venue.

For your viewing pleasure, here’s the video. Make yourself some popcorn, relax, and enjoy the show!

If you enjoyed this post, make sure you subscribe to the RSS feed!

I remember well waking up early one morning, November 18, 2004 (no, I didn’t remember the exact date but Outlook did), to a flurry of emails from some of my East Coast customers.

They had seen a story in the New York Times announcing the birth of Google Scholar. A number of questions were raised – were federated search applications such as Science.gov going to become obsolete? Should we federate Google Scholar?

A few months later there was a brief article in Digital Librarian (this article is no longer available but here’s a summary) announcing that “2005 is the year that will be remembered (in the library world) as the year that federated search became obsolete.”

2007 is coming to a close, Google Scholar is still in Beta, and federated search is alive and doing well. In the last few years we’ve seen tremendous improvements in federated search and I expect that the years ahead will be an exciting time for Deep Web Technologies and others in our industry. I have high hopes that this blog can become “the place” where all kinds of information about federated search can be shared and openly discussed.

If you enjoyed this post, make sure you subscribe to the RSS feed!