Jan
Carl Grant, President of CARE Affiliates, left a comment on my recent OpenTranslators announcement raises questions post. The comment refers to an in-depth response that Grant posted on his blog. I’m writing this post to bring attention to Grant’s response, to encourage everyone to read it and join in the conversation, and to respond to the response.
The response is well thought out and well articulated. Writing many posts myself I appreciate the time Mr. Grant spent crafting his response, in particular on a Saturday night. This shows quite a dedication to his company and to this industry. I agree with the majority of the content of the response and I still have several concerns.
My first concern was, and still is, about the openness of the OpenTranslators. I am very well aware of the difficulty of creating and maintaining translators/connectors. While not active in connector development, during my five years of full-time employment with Deep Web Technologies I developed some connectors myself. And, I had numerous intimate conversations with the connector development staff about idiosyncrasies of many search interfaces and the challenges it brought to the developers. In my mind an open translator is one that I have the source code for. If I can modify and maintain a connector myself then it is open to me.
I completely get that the current business model is to keep connectors proprietary because connectors are a significant part of the intellectual property of the federated search vendor. I’m not criticizing WebFeat’s business model; Deep Web Technologies follows this model. My objection is about calling them open when in the spirit of open source they are absolutely NOT open.
Perhaps someday the model will change to one where anyone can produce their own connectors and contribute them as true open source connectors. The business model would then become one of charging for software, support, integration, consulting, and maintenance of connectors for those customers who didn’t want to wait for a volunteer to provide upgrades.
My second concern is with the response to my question of whether there really are 10,000 translators. Doesn’t Elsevier have something like 1,000 journals? I’d like to know how many publishers comprise the 10,000 translators. Read the response to my question, item #4 in the response post and see if you’re satisfied with the response.
My final concern, which I addressed in my original post, is with scalability. I’m not so much concerned with the scalability of the federated search engine; I’m concerned with the ability of the individual content providers to handle a large and permanent increase in user queries. I’m delighted that CARE and WebFeat can increase capacity on short notice but neither party can control how much capacity the content provider has. If the service becomes hugely successful, and I sincerely hope it does, then CARE and WebFeat may need to implement some kind of flow control mechanism to throttle back the number of requests they submit to a particular content provider if the source’s ability to respond is seriously degraded. My intent is not to target CARE and WebFeat with this concern. It’s a problem for the entire industry. As vendors find ways of making more connectors available to more users we all become more vulnerable to the ability of individual sources to handle the load.
Read Carl Grant’s reply. What do you agree or disagree with? Join the conversation.
If you enjoyed this post, make sure you subscribe to the RSS feed!
4 Responses so far to "Response to OpenTranslators post"
January 20th, 2008 at 4:03 pm
Actually, in my mind, the ‘open’ refers both to ‘open systems’ in the old, revered ISO sense of the word, as software systems that support well-defined, portable interfaces, and ‘open standards’, generally taken to mean vendor-neutral specifications, again meant to support portability and vendor neutrality. At Index Data, we have based our entire business model upon these concepts for the past 14 years or so, leveraging open source software to support the uptake of open standards and vice versa. To me, these are all pieces of the same puzzle, and the very selfish goal for Index Data is to help foster a dynamic business environment where small companies can compete and play on an even footing (actually often with an advantage) with the older, established companies. I feel quite strongly about this, and one of my biggest regrets is that I didn’t find some way to make ‘open’ a part of our company name when we started. Even as a dedicated open source developer, with more code lines released than many, I find it ridiculous and a little offensive to see the word ‘open’ co-opted into such a narrow interpretation.
January 21st, 2008 at 7:31 pm
As another of the federated search developers I would like to add to this that I think the service CARE/ID/Webfeat are planning to offer is a very valid one. Even if they are competitors.
From years of experience in this industry I appreciate the historical timeline Carl gave, viewing the introduction and evolution of federated search as a way to insulate the user from the vagaries of the interface and search language of the content vendor, then from those of the federated search engine, so that with this sort of service the customer can use a front end system of their choice to provide a user interface of their choice, whether it be “google-like” or an OPAC or a “power searcher” or some other form. This modular, ‘plumbing’, standards based, architecture does promote an openness of choice for the people who matter - the users. (An aside for those who care about these things; this is another use of the term ‘open’ to those mentioned by Sebastian. all valid and all part of the modern experience - open choice.)
I would like to say that I believe this sort of service will work very well. I feel I can say this with some confidence as we at MuseGlobal have been offering just this division of labor from day one when Muse was launched to the world nearly 7 years ago now. In fact one reason why the Muse name is not well recognised in the federated search world is that all our user interfaces are designed and branded by our partners. So interfaces as diverse as those from III(research pro), SirsiDynix (singlesearch), Swets(Swetssearcher), and Ovid (searchsolver) through to mainstream media products with no user interface at all are possible with this architecture.
We welcome this new entrant to the world of the front-middle-back information delivery architecture and wish them well.
January 23rd, 2008 at 9:38 pm
I’m really heartened to see the comments that the recent posts on the CARE announcement and the questions raised by Sol have gathered. I hope that in the future this blog does become the place where all of us interested in federated search can come and have a good exchange of ideas.
In response to the comment that Sebastian Hammer from IndexData made I want to say that I had a good discussion with one of your colleagues, David Dorman, at ALA and understand that OpenTranslators refers to the translators being accessible through open standard protocols (i.e. Z39.50 and SR/W). We at Deep Web Technologies are big believers in open standards and have built our federated search engine so that it is accessible through standards-based web services.
I believe that if IndexData were able to make the connectors themselves available as open source it would do so. At ALA I listened to a good presentation by David Dorman and I’m intrigued by his idea of creating a clearinghouse for connectors that could be shared among federated search vendors. We would welcome a post by David expanding on this concept of his.
Let me also say that as Sol indicated in his original post I believe that the CARE announcement is significant and will be overall good for our industry.
Finally, I’d like to welcome Peter Noerr to our blog, someone that I met in Santa Fe, not long after he and Kate started MuseGlobal. I find it pretty amazing that two federated search companies were started in Northern New Mexico.
May 20th, 2008 at 12:26 am
i have the problem in developing a connector for BIOONE in mapping plz do reply…..