一个简单的Ruby分布式处理示例

A Simple Distributed Queue in Ruby



One of the cooler yet little used gems in the Ruby standard library is DRb - Distributed Ruby. DRb makes it really easy to share objects between different processes. Today, Darcy Laycock will show, how to build a simple (and relatively naive) shared queue with DRb.
A Single Process Example

First off, we’re going to reuse as much code as possible. Ruby provides a built-in thread-safe queue class which we can leverage to build it. So, for our first step, we’re going to start by requiring thread (to access) it and adding a few objects to it / manipulating it to show the results.

  require 'thread'
  queue = Queue.new
  [:a, :b, :c, :d].each do |item|
    queue.push item
    puts "Pushed #{item.inspect} onto the queue - It now has #{queue.size} items"
  end
  until queue.empty?
    item = queue.pop
    puts "Popped #{item.inspect} off the queue, size now is #{queue.size}"
  end

Copying to a file, queue_test.rb, and running it, you should now see the following output:

引用

  Pushed :a onto the queue - It now has 1 items
  Pushed :b onto the queue - It now has 2 items
  Pushed :c onto the queue - It now has 3 items
  Pushed :d onto the queue - It now has 4 items
  Popped :a off the queue, size now is 3
  Popped :b off the queue, size now is 2
  Popped :c off the queue, size now is 1
  Popped :d off the queue, size now is 0

Making it Distributed

The next step is to use DRb to make a simple distributed queue. The first thing we need to do, is to set DRb to make it available. For the moment, we’re going to run it on port 3491. Inside a new file, queue_server.rb, write the following:

  require 'thread'
  require 'drb'
  queue = Queue.new
  DRb.start_service("druby://:3491", queue)
  DRb.thread.join

The DRb.start_service takes a uri which tells it what address and port to bind to (if the port is nil / absent, it will randomly choose a port) and then DRb will make the object available to other processes at said location. We need to call DRb.thread.join so that the process exits when the DRb thread does - otherwise, as soon as lines above had run, the program would exit.

Now, we’ll create two more files - queue_consumer.rb and queue_provider.rb First, opening up queue_consumer.rb we’ll write the following:

  require 'thread'
  require 'drb'
  DRb.start_service
  queue = DRbObject.new(nil, "druby://localhost:3491")
  loop do
    queue.push "Hello from #{Process.pid} at #{Time.now}"
    sleep 2
  end

Which will connect to the remote queue (that we wrote above) and every 5 seconds will push a string onto it. Now, inside queue_consumer.rb we’re simply going to do the following:

  require 'thread'
  require 'drb'
  DRb.start_service
  queue = DRbObject.new(nil, "druby://localhost:3491")
  loop do
    unless queue.empty?
      puts "#{Process.pid} got #{queue.pop.inspect} from the remote queue"
    end
    sleep 1
  end

All this does is poll the queue every second, checking if it has items. If it does, it will print out the processes pid and the message it got. Starting up queue_server.rb, a couple of queue_provider.rb's (I did 2) and a queue_consumer.rb in seperate terminals you should get output something like:

引用

  22703 got "Hello from 22690 at Sun Dec 07 22:01:27 +0900 2008" from the remote queue
  22703 got "Hello from 22691 at Sun Dec 07 22:01:28 +0900 2008" from the remote queue
  22703 got "Hello from 22690 at Sun Dec 07 22:01:29 +0900 2008" from the remote queue
  22703 got "Hello from 22691 at Sun Dec 07 22:01:30 +0900 2008" from the remote queue
  22703 got "Hello from 22690 at Sun Dec 07 22:01:31 +0900 2008" from the remote queue

Of course, your pid’s are going to be different.
Conclusion

In just a few minutes of work you can see we’ve written a basic distributed ruby queue in very few lines of code (5 lines of code + 4 requires) and whilst it may not be the best performing implementation out their, it’s incredibly simple and it hopefully introduced the power of DRb to you.

Lastly, there is a wealth of other stuff you can do with DRb - Ruby’s Rinda classes provide automatic discovery of services and tuplespaces whilst DRb provides simple access control and a bunch of other nifty stuff. If you want to learn more, I suggest checking out Mark Bates’ rubyconf talk which shows off a bunch of cool stuff that can be done and Eric Hodel’s fantastic set of articles.