Clojure MapReduce Reducer

Question

This program forms the reducer of a Hadoop MapReduce job. It reads data in from stdin that is tab delimited.

foo 1
foo 1
bar 1

and outputs

foo 2
bar 1

Any suggestions for improvements?

(use '[clojure.string :only [split]])
(def reducer (atom {}))
(defn update-map [map key]
 (merge-with + map {key 1}))
(doseq [line (line-seq (java.io.BufferedReader. *in*))]
 (let [k (first (split line #"\t"))]
 (swap! reducer update-map k)))
(doseq [kv @reducer]
 (println (format "%s\t%s" (first kv) (second kv))))

Answer 1

probably a bit too late to help OP, but in case anyone else stumbles upon this question, here's a nice succinct way of doing it, using the frequencies function:

(doseq [[word freq] (frequencies
 (map
 #(re-find #"^[^\t]+" %) ;; just get the first non-tab characters
 (line-seq (java.io.BufferedReader. *in*))))]
 (println (str word "\t" freq)))

Answer 2

Why don't you use reduce instead of the first doseq? Something along the lines (untested, entered directly here):

(def response
 (reduce (fn [map line]
 (let [k (fist (split line #"\t"))]
 (update-map map k)))
 {} (line-seq (java.io.BufferedReader. *in*)))
(doseq [kv response]
 (println (format "%s\t%s" (first kv) (second kv))))

Then you won't need the atom either.

Answer 3

Can ouput contain numbers other than 1? Like:

foo 1
foo 3
bar 10

If so, then:

(use '[clojure.string :only [split]])
(def parsed-input
 (for [line (line-seq (java.io.BufferedReader. *in*))
 :let [[k v] (split line #"\t")]]
 {k (Double/parseDouble v)}))
(def table (apply (partial merge-with + {}) parsed-input))
(doseq [[k v] table]
 (println (str k "\t" v)))

Outputs:

bar 10.0
foo 4.0

If it's just 1's frequencies will do as suggested.