Comments on: Divide and conquer: federating many sources

By: Stephan Schmid

Stephan Schmid — Tue, 23 Sep 2008 08:07:14 +0000

In my opinion, the scalability is mainly a matter of the software architecture, and if it scales nearly linearly, it can grow as needed. With my experience, in general I agree with Jonathan that it’s mostly simpler to go for a single tier approach. A multi-server distribution can make sense for sources that are geographically remote and build a topical unit.

Some years ago I made a test with 250 test sources (HTTP and JDBC) that were queried multithreaded (on a rather old P4 machine running Linux). Up to four concurrent users worked pretty well – today with fast multicore CPU’s it should run even better. For high volume solutions I would anyway try multiplexed, non-blocking I/O.

By: Sol

Sol — Mon, 22 Sep 2008 16:01:01 +0000

Jonathan – I have a draft of a response to your comment that I’m fine tuning and will publish soon as a blog article. I’ve not forgotten you.

By: Sol

Sol — Wed, 17 Sep 2008 02:54:05 +0000

Jonathan – I do have some thoughts in response to your comments but I’d like to wait a day or two and see what other comments people post and then I’ll respond to all of them.

By: Jonathan Rochkind

Jonathan Rochkind — Wed, 17 Sep 2008 01:42:59 +0000

It’s not clear to me why that multi-tiered federated approach would be neccesary with modern hardware and OSs. What’s the difference between doing that, and just having lots of thread/forks? Is your merging of result sets REALLY so CPU intensive that you need more than one machine? And if you do need more than one machine, wouldn’t some other method of “transparent” multi-server distribution (single tier, but distributed across various servers) under the covers be easier than a multi-tiered approach?

It also continues to be curious to me that SerialSolutions federated searching product claims to be able to search across 200 sources at once. I haven’t investigated it enough to know if there’s a hidden gotcha to that claim, or if it really is what it says it is.