Distributed RDoc and a DRb tip
Noobkit is powered at its core by RDoc. Obviously it’s not real time, far from it. In fact, full refresh of Noobkit’s database which includes 26 Gems together with Rails and Ruby Core takes about 25 minutes on a single Core 2 Duo 2.4Ghz with 4GB ram box.
About 20 minutes of that time is taken by RDoc parsing the source code. This is what I’m currently working on making better. I have used this article as a starting point and wrote a client/server script on top of customized RDoc.
The end result is having 4 threads running across 2 boxes each parsing a separate source file fed to it by the server.
This being the first time I’ve worked with DRb or Ruby threads in general, one thing that kept annoying me was inability to break with Ctrl+C when using DRb.thread.join. I’ve devised a simple waiting loop instead:
def wait_for_it(&block)
return if block.nil?
loop do
sleep(0.1)
break unless yield
end
end
# ...do DRb magic...
# This will stall the current thread until DRb thread is finished.
wait_for_it { DRb.thread.alive? }
The same function can also be used to insure that all workers have finished their job before doing something else:
# When a worked is done, it's added back to the queue
wait_for_it { @available_workers.size != $workers.size }

No comments yet, be the first one!
Leave a Reply