Large-scale Message-broker Federation

During the second half of 2009, I led a RabbitMQ/LShift project for the Ocean Observatory Initiative (OOI) where we were asked to design and prototype a messaging system based on AMQP and RabbitMQ that could grow smoothly all the way from LAN-sized networks up to large data-centers and the cross-continental networks connecting them.

The first phase of the research was completed toward the end of January, 2010. I released a design document and project report that describe the design approach and the implementation of the prototype system, respectively.1 The design document briefly describes the "exchange space" and "exchange point" ideas that gave the project its direction, and the project report includes a list of starting points for future research.

I left RabbitMQ shortly thereafter to start preparation for my PhD research, but the project is continuing at the OOI.

About OOI

The Ocean Observatory Initiative (OOI) is a multi-year project run by a number of American and international universities and other organisations to build a globe-spanning sensing, data collection, data transport and data processing system for observing the world's oceans.

Links

  • The design document and project report describing the software that resulted from the project.

  • OOI-CI, the Cyberinfrastructure project within OOI. Our research contributed to this area of the overall project.

  • I gave a talk at Thoughtworks's London Geek Night in May 2009 which describes some of this project's research into message distribution patterns.


  1. Not mentioned in the texts are the names of the reviewers and contributors, to whom many thanks are due: Matthew Arrott, Michael Bridgen, Marek Majkowski, Matthias Radestock, and Alexis Richardson.