Last Updated: February 25, 2016

Ruby thread in simple rake task

An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group https://www.facebook.com/groups/rubydevs/)

namespace :somestuffs do
 desc 'Importing some big stuff that requies lotta network stuff'
 task :import
 file, tasks = ENV['FILE'], Queue.new

 # Parse out CSV file and retrieve each row and queue them up 
 # for further processing. 
 #
 # Keeping it a thread allows us to let this process continue while other 
 # threads are not in RUNNING state.
 Thread.new do
 CSV.open(file) { |row| tasks << row }
 end

 # Use something that matters in your envrionment, 
 # if you are using some network call which takes quite long time, 
 # then you can use 10/20 threads / CPU core
 4.times do
 Thread.new do
 while (row = tasks.pop)
 # Do network transfer related blocking task.
 end
 end
 end

 # Control thread which keeps current process blocked until queue becomes zero
 # In our case we are expecting tasks.size could get zero whenever no more data to process.
 Thread.new do
 sleep 1 until tasks.size.zero?
 end.join 

 # Calling "join" allows current process to wait until the JOINED thread is finished. 
 # Thus we can ensure our parent process doesn't exit until all tasks are done.
 end
end

#ruby

#rake

#thread