Shouldn't the reduce_zero_label=True be set in the COCO-Stuff 164k base config? The docstring for the COCOStuffDataset also mentions that it should.

The index zero is for the "unlabeled" class.