How to merge lists of Map with Lists values using Java Streams API?

Question

How can I reduce the Map<X, List<String>> grouping by the X.p and join all the list values at the same time, so that I have Map<Integer, List<String>> at the end?

This is what I've tried so far:

class X {
 int p;
 int q;
 public X(int p, int q) { this.p = p; this.q = q; }
}
Map<X, List<String>> x = new HashMap<>();
x.put(new X(123,5), Arrays.asList("A","B"));
x.put(new X(123,6), Arrays.asList("C","D"));
x.put(new X(124,7), Arrays.asList("E","F"));
Map<Integer, List<String>> z = x.entrySet().stream().collect(Collectors.groupingBy(
 entry -> entry.getKey().p, 
 mapping(Map.Entry::getValue, 
 reducing(new ArrayList<>(), (a, b) -> { a.addAll(b); return a; }))));
System.out.println("z="+z);

But the result is: z={123=[E, F, A, B, C, D], 124=[E, F, A, B, C, D]}.

I want to have z={123=[A, B, C, D], 124=[E, F]}

Answer 1

Here's one way to do it using two Stream pipelines :

Map<Integer, List<String>> z = 
// first process the entries of the original Map and produce a 
// Map<Integer,List<List<String>>>
 x.entrySet()
 .stream()
 .collect(Collectors.groupingBy(entry -> entry.getKey().p, 
 mapping(Map.Entry::getValue,
 toList())))
// then process the entries of the intermediate Map and produce a 
// Map<Integer,List<String>>
 .entrySet()
 .stream()
 .collect (toMap (Map.Entry::getKey,
 e -> e.getValue()
 .stream()
 .flatMap(List::stream)
 .collect(toList())));

Java 9 is supposed to add a flatMapping Collector, that would make your life easier (I learned about this new feature thanks to Holger).

Output :

z={123=[A, B, C, D], 124=[E, F]}

• tafit3: Do you think it can't be done with a stream being collected by a single groupingBy or toMap (having some combination of mapping, reducing, flatMap...) in Java 8?
• Eran: @tafit3 Currently I can't think of a way to do it in a single Stream pipeline.
• Holger: It’s actually not that hard to do; there is only one mistake in the OP’s code...

Answer 2

You are using the reducing collector incorrectly. The first argument must be an identity value to the reduction operation. But you are modifying it by adding values to it, which perfectly explains the result: all values are added to the same ArrayList which is expected to be the invariant identity value.

What you want to do is a Mutable reduction and Collectors.reducing is not appropriate for that. You may create an appropriate collector using the method Collector.of(...):

Map<Integer, List<String>> z = x.entrySet().stream().collect(groupingBy(
 entry -> entry.getKey().p, Collector.of(
 ArrayList::new, (l,e)->l.addAll(e.getValue()), (a,b)->{a.addAll(b);return a;})));

Answer 3

There is a way to achieve that in one run by writing your own Collector:

Map<Integer, List<String>> z = x.entrySet().stream().collect(
 Collectors.groupingBy(entry -> entry.getKey().p,
 Collectors.mapping(Entry::getValue, 
 Collector.of(ArrayList::new, (a, b) -> a.addAll(b), (a, b) -> {
 a.addAll(b);
 return a;
 })
 )
 )
);

Answer 4

Using the EntryStream class of my StreamEx library such tasks can be solved quite easily:

Map<Integer, List<String>> z = EntryStream.of(x)
 .mapKeys(k -> k.p)
 .flatMapValues(List::stream)
 .grouping();

Internally it's transformed to something like this:

Map<Integer, List<String>> z = x.entrySet().stream()
 .map(e -> new AbstractMap.SimpleImmutableEntry<>(e.getKey().p, e.getValue()))
 .<Entry<Integer, String>>flatMap(e -> e.getValue().stream()
 .map(s -> new AbstractMap.SimpleImmutableEntry<>(e.getKey(), s)))
 .collect(Collectors.groupingBy(e -> e.getKey(), 
 Collectors.mapping(e -> e.getValue(), Collectors.toList())));

So it's actually a single stream pipeline.

If you don't want to use the third-party code, you can simplify the above version a little:

Map<Integer, List<String>> z = x.entrySet().stream()
 .<Entry<Integer, String>>flatMap(e -> e.getValue().stream()
 .map(s -> new AbstractMap.SimpleEntry<>(e.getKey().p, s)))
 .collect(Collectors.groupingBy(e -> e.getKey(), 
 Collectors.mapping(e -> e.getValue(), Collectors.toList())));

Though it still looks ugly.

Finally please note that in JDK9 there's new standard collector called flatMapping which can be implemented in the following way:

public static <T, U, A, R>
Collector<T, ?, R> flatMapping(Function<? super T, ? extends Stream<? extends U>> mapper,
 Collector<? super U, A, R> downstream) {
 BiConsumer<A, ? super U> downstreamAccumulator = downstream.accumulator();
 return Collector.of(downstream.supplier(),
 (r, t) -> {
 try (Stream<? extends U> result = mapper.apply(t)) {
 if (result != null)
 result.sequential().forEach(u -> downstreamAccumulator.accept(r, u));
 }
 },
 downstream.combiner(), downstream.finisher(),
 downstream.characteristics().toArray(new Collector.Characteristics[0]));
}

Using this collector, your task can be solved simpler without additional libraries:

Map<Integer, List<String>> z = x.entrySet().stream()
 .map(e -> new AbstractMap.SimpleImmutableEntry<>(e.getKey().p, e.getValue()))
 .collect(Collectors.groupingBy(e -> e.getKey(), 
 flatMapping(e -> e.getValue().stream(), Collectors.toList())));