Reading a file in chunks

Question

I currently have the following class that takes a custom file object as an argument and makes a copy of the file locally.

class FileCopier
 def initialize(file_info_object)
 @file_info_object = file_info_object
 end
 def copy
 File.open(local_file_path, 'wb') do |f|
 read_file_in_chunks do |chunk|
 f.write chunk
 end
 end
 end
private
 def local_file_path
 File.join(Rails.root,'tmp/blah.txt')
 end
 def read_file_in_chunks(chunksize=10343)
 index = 0
 size = @file_info_object.size
 path = @file_info_object.path
 while index < filesize
 yield Client.read(path, offset: index, length: chunksize)
 index += chunksize
 end
 end
end

The copy method seems to be fairly straight forward but I was wondering if there is a way to simplify it. It seems difficult to test to me at the moment.

• 200_success: Could you tell us more about CustomFileReader?
• Jackson: I should have been more specific. My apologies. The CustomFileReader has now been renamed to 'Client'. Basically the client is an HDFS client that is reading from an HDFS directory. I am using the webHDFS gem for the client. webHDFS gem
• Jackson: One constraint I have is that I can't connect to HDFS locally. What would be a good way to test the export method? (Rspec 3)

Answer 1

Some notes:

• Use 2-space indentation.
• f.write chunk. You use parens in other places, be consistent.

Other than these minor issues, I think your code is pretty good. I'd just refactor read_file_in_chunks to use a enumerator instead (more versatile) and a range.

def read_file_in_chunks(chunksize=10343)
 Enumerator.new do |yielder|
 (0...@file_info_object.size).step(chunksize).each do |offset|
 data = Client.read(@file_info_object.path, offset: offset, length: chunksize)
 yielder.yield(data)
 end
 end
end

Using lazy collections (Ruby>= 2.0) it can be simplified even more:

def read_file_in_chunks(chunksize=10343)
 (0...@file_info_object.size).step(chunksize).lazy.map do |offset|
 Client.read(@file_info_object.path, offset: offset, length: chunksize)
 end
end