Secondary Sort in Hadoop

I write this stuff since it confused me a lot.

First, the input Iterable values for reduce method is not sorted in any
order. It is mentioned on Page 277 in Hadoop: The Definitive Guide,
Third Edition.

Job has three methods related to second sort:

  1) setPartitionerClass
  2) setSortComparatorClass
  3) setGroupingComparatorClass

For 0.18.3 API, JobConf's following three methods are used:
  1) setPartitionerClass
  2) setOutputKeyComparatorClass
  3) setOutputValueGroupingComparator

The confusing part is setGroupingComparatorClass. For the following
input for reduce:

<k1, v1> <k2, v2> // sorted by key from map phase

If setGroupingComparatorClass decides that k1 is equivalent to k2 even
thought k1 and k2 are different, the output from reduce method will be:

<k1, <v1, v2>> // <v1, v2> are in the original key order, k2 is lost.

Now let's talk about MaxTemperatureUsingSecondarySort. Input is like:

1901 99
1901 98
1902 80
1902 10

The output from map is:

<<1901, 99>, null>
<<1901, 98>, null>
<<1902, 80>, null>
<<1902, 10>, null>

The reduce method will be invoked twice. The output is:

<1901, 99>
<1902, 80>

Now turns to SecondarySort in Hadoop examples. For the same input, the
output from map is:

<<1901, 98>, 98>
<<1901, 99>, 99>
<<1902, 10>, 10>
<<1902, 80>, 80>

The reduce method will also be invoked twice. The output is:

<1901, 98>
<1901, 99>
<1902, 10>
<1902, 80>