How can I make this CouchDB view code more "correct"?

Question

I have a fairly simple app set up with couchrest_model to capture data pulled in from vendors to couchdb for later searching and archiving. Each document has a show (the name of the vendor), a datatype (which is limited to seven or eight unique strings), and a pulled_at timestamp field.

The core method on this model needs to return a Hash where each key is a show and the value is an array of the most recent document for each datatype. Below, I have my (simplified) model code which defines this method. It works, but my code in self.most_recent_by_type_and_show() which loops over the list of _ids and calls get() on each one (thus generating n HTTP requests on each method call) will not scale.

class Snapshot < CouchRest::Model::Base
 property :show, String
 property :sources, [String]
 property :pulled_at, DateTime
 property :datatype, String
 property :data, [Hash]
 design do
 view :_most_recent_by_type_and_show, :map => <<-MAP, :reduce => <<-REDUCE
 function(doc) {
 if (doc.type == "Snapshot") {
 emit(doc.show, [doc.datatype, new Date(doc.pulled_at).valueOf()]);
 }
 }
 MAP
 function(keys, values, rereduce) {
 var maxes = {},
 out = [];
 if (!rereduce) {
 for (var i in keys) {
 var show = keys[i][0];
 var doc_id = keys[i][1];
 var datatype = values[i][0];
 var timestamp = values[i][1];
 if (typeof maxes[show] == 'undefined') { maxes[show] = {}; }
 if (typeof maxes[show][datatype] == 'undefined') { maxes[show][datatype] = [0, null]; }
 if (timestamp > maxes[show][datatype][0]) {
 maxes[show][datatype] = [timestamp, doc_id];
 }
 }
 }
 if (Object.keys(maxes).length == 0) { maxes = values; }
 for (var show in maxes) {
 for (var datatype in maxes[show]) {
 out.push(maxes[show][datatype][1]);
 }
 }
 return out;
 }
 REDUCE
 end
 class << self
 def most_recent_by_type_and_show opts = {}
 options = opts.merge({ reduce: true, })
 raw = _most_recent_by_type_and_show(options).group.rows
 raw.reduce({}) do |hash, row|
 hash[row['key']] = row['value'].map { |id| get id }.sort_by(&:pulled_at).reverse
 hash
 end
 end
 end
end

This feels like I'm overcomplicating the view code, and I'm also not sure that my reduce function is written correctly for rereduce. I initially tried emitting [show, datatype, pulled_at] as the key and the entire doc itself as the value in the map function, but I've read that this is Bad Practice as it will cause the size of my view index to balloon over time. And for some reason, couchrest_model will not allow me to include_docs=true if I'm reducing a view, which seems to be just a limitation of the gem as I can't find anything saying this should be the case in the CouchDB docs.

Would this be better written as a different data structure altogether? Should I maybe be using a _list function? I'm not sure where to go from here.

Answer 1

You can use CouchRest::Database.get_bulk API:

hash[row['key']] = database.get_bulk(row['value']).sort_by(&:pulled_at).reverse