Carl Grant, President of CARE Affiliates, left a comment on my recent OpenTranslators announcement raises questions post. The comment refers to an in-depth response that Grant posted on his blog. I’m writing this post to bring attention to Grant’s response, to encourage everyone to read it and join in the conversation, and to respond to the response.
The response is well thought out and well articulated. Writing many posts myself I appreciate the time Mr. Grant spent crafting his response, in particular on a Saturday night. This shows quite a dedication to his company and to this industry. I agree with the majority of the content of the response and I still have several concerns.
My first concern was, and still is, about the openness of the OpenTranslators. I am very well aware of the difficulty of creating and maintaining translators/connectors. While not active in connector development, during my five years of full-time employment with Deep Web Technologies I developed some connectors myself. And, I had numerous intimate conversations with the connector development staff about idiosyncrasies of many search interfaces and the challenges it brought to the developers. In my mind an open translator is one that I have the source code for. If I can modify and maintain a connector myself then it is open to me.
I completely get that the current business model is to keep connectors proprietary because connectors are a significant part of the intellectual property of the federated search vendor. I’m not criticizing WebFeat’s business model; Deep Web Technologies follows this model. My objection is about calling them open when in the spirit of open source they are absolutely NOT open.
Perhaps someday the model will change to one where anyone can produce their own connectors and contribute them as true open source connectors. The business model would then become one of charging for software, support, integration, consulting, and maintenance of connectors for those customers who didn’t want to wait for a volunteer to provide upgrades.
My second concern is with the response to my question of whether there really are 10,000 translators. Doesn’t Elsevier have something like 1,000 journals? I’d like to know how many publishers comprise the 10,000 translators. Read the response to my question, item #4 in the response post and see if you’re satisfied with the response.
My final concern, which I addressed in my original post, is with scalability. I’m not so much concerned with the scalability of the federated search engine; I’m concerned with the ability of the individual content providers to handle a large and permanent increase in user queries. I’m delighted that CARE and WebFeat can increase capacity on short notice but neither party can control how much capacity the content provider has. If the service becomes hugely successful, and I sincerely hope it does, then CARE and WebFeat may need to implement some kind of flow control mechanism to throttle back the number of requests they submit to a particular content provider if the source’s ability to respond is seriously degraded. My intent is not to target CARE and WebFeat with this concern. It’s a problem for the entire industry. As vendors find ways of making more connectors available to more users we all become more vulnerable to the ability of individual sources to handle the load.
Read Carl Grant’s reply. What do you agree or disagree with? Join the conversation.