Real time RSS feeds sent over HTTP using a Comet server and the painful process of consuming them. It all started because of an entry into a competition to win Le-Web tickets hosted by Superfeedr. Superfeedr is one of the first companies out there to offer RSS feeds in real time. Though this is subject to interpretation, due to the fact that they cannot possibly monitor every feed source from moment to moment the feeds they push to the web are not really real time, but close enough.
Superfeedr offers a sample of their XMPP feed via a port on their HTTP server. This is where the Fun-stration started. I initially thought the API for dealing with such feeds would be as straight forward as REST API. Unfortunately, these Atom Stream RSS feeds are quite new and only a select few web technologies have wrappers to handle their data… the 2 main technologies I found that could handle these feeds were Python and Perl.
Developers for both Python and Perl have a clear advantage here with direct access to command line compiling. In the run-time environments of PHP, Flash, Javascript, etc. It is much more difficult to manage an HTTP connection with a stream of data being sent through it.
It’s a great idea, and I love the idea. It will open the web up to some really great and much need possibilities. It is still too new, and people have yet to figure out a real purpose for it. This fact was proven to me by Superfeedr’s complete lack of documentation and no sample code at all. In fact, the Superfeedr website, in my opinion, was completely useless for this competition.
Superfeedr Founder and I believe creator / programmer Julien Genestoux tried his best to help me. I’m sure he became frustrated at times with what probably appeared to be a lack of programming ability on my part, but I assure you, these RSS Streams cannot be consumed by PHP and Flash alone. They require a much more robust combination of technologies. Namely, a comet server.
He claims to have a nice line-up of apps built for this competition, but I’ll believe it when I see it. He did mention some apps built in Objective C. This means there will be some iPhone apps and some other mobile technologies.
In the end, I learned a lot about this up and coming XMPP and HTTP Streaming technology. I think it will serve nGn and my development well. These cutting edge technologies just add fuel to the creative fires. I hope others can see the good aspects of Superfeedr, even if its useless. It won’t be useless forever.
In the end, he announced one of the apps over Twitter. I looked into the architecture behind this app and found something surprising. To consume these HTTP Atom Streams of RSS feeds, you need to have a comet server installed. The server used in the example was Meteor. I’m sure there are others. This server in conjunction with classic web technologies allows us to run a mock-realtime blog or micro blog. The only thing that bothers me about this is in the app he announced they use long polling to get the output from Meteor. To me, this totally defeats the purpose of the real time web. Real time and comet streams were designed to eliminate the need for long polling. Long polling is the process by which a script will ping a server at a specified interval to see if any new material has been published. This is the classic Pull model of data retrieval used until real time Push was implemented. Not very real time if you ask me.
Nice job on that one. I’m sure there will be other cool apps built with Superfeedr. Looking forward to see what people come up with.
If I have time I will post a short tutorial on how to connect to a comet stream and output it in PHP. I think this type of documentation will be valuable.
