Problem
Running maven tests with NonDex tool would discover flaky tests.
NonDex is a tool for detecting wrong assumptions on undeterministic Java APIs. It could help developers fix the assumption before it becomes a problem far in the future and more difficult to fix.
Reproduce step
- Test Environment
Apache Maven: 3.6.3
Java: 11.0.21
OS: Ubuntu 20.04.6 LTS
Linux kernel: 5.4.0-167-generic
mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex
Root cause
EagerTest related test
Take com.hubspot.jinjava.EagerTest#itHandlesImportInDeferredIf as an example, the test fails at assertion in ExpectedTemplateInterpreter.java L45
public String assertExpectedOutputNonIdempotent(String name) {
String template = getFixtureTemplate(name);
String output = JinjavaInterpreter.getCurrent().render(template);
assertThat(JinjavaInterpreter.getCurrent().getContext().getDeferredNodes())
.as("Ensure no deferred nodes were created")
.isEmpty();
assertThat(output.trim()).isEqualTo(expected(name).trim());
return output;
The reason why it fails is that the rendered string is determined by the order of HashMap.entrySet(), which is called at AliasedEagerImportingStrategy.java L255
for (Map.Entry<String, Object> entry : currentAliasMap.entrySet()) {
However, the order of HashMap.entrySet() is not guaranteed. It might change from time to time.
NonDex would shuffle the map on each invocation of methods.
Therefore, the rendered string might change, and make the tests fail.
PyishObjectMapperTest
For PyishObjectMapperTest test, it is the similar reason.
In L25, HashMap has been used. However, the assertion in L29 is relied on the order of HashMap.entrySet()
@Test
public void itSerializesMapWithNullKeysAsEmptyString() {
Map<String, Object> map = new SizeLimitingPyMap(new HashMap<>(), 10);
map.put("foo", "bar");
map.put(null, "null");
assertThat(PyishObjectMapper.getAsPyishString(map))
.isEqualTo("{'': 'null', 'foo': 'bar'} ");
}
Code change
To fix the test flakiness, I replaced HashMap with LinkedHashMap in AliasedEagerImportingStrategy.java and PyishObjectMapperTest.java, and exchange the order of string to aligned with the order of `LinkedHashMap
Failed test
com.hubspot.jinjava.EagerTest#itHandlesImportInDeferredIf
com.hubspot.jinjava.EagerTest#itHandlesDoubleImportModification
com.hubspot.jinjava.EagerTest#itHandlesSameNameImportVar
com.hubspot.jinjava.EagerTest#itDoesNotOverrideImportModificationInFor
com.hubspot.jinjava.EagerTest#itRreconstructsValueUsedInDeferredImportedMacro
com.hubspot.jinjava.EagerTest#itAllowsVariableSharingAliasName
com.hubspot.jinjava.NonRevertingEagerTest#itHandlesImportInDeferredIf
com.hubspot.jinjava.NonRevertingEagerTest#itHandlesDoubleImportModification
com.hubspot.jinjava.NonRevertingEagerTest#itHandlesSameNameImportVar
com.hubspot.jinjava.NonRevertingEagerTest#itDoesNotOverrideImportModificationInFor
com.hubspot.jinjava.NonRevertingEagerTest#itRreconstructsValueUsedInDeferredImportedMacro
com.hubspot.jinjava.NonRevertingEagerTest#itAllowsVariableSharingAliasName
com.hubspot.jinjava.lib.tag.eager.EagerImportTagTest#itDefersTripleLayer
com.hubspot.jinjava.lib.tag.eager.EagerImportTagTest#itHandlesQuadLayerInDeferredIf
com.hubspot.jinjava.lib.tag.eager.EagerImportTagTest#itDoesNotSilentlyOverrideVariable
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapWithNullKeysAsEmptyString
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapEntrySet
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapEntrySetWithLimit
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapWithNullValues
Problem
Running maven tests with NonDex tool would discover flaky tests.
NonDex is a tool for detecting wrong assumptions on undeterministic Java APIs. It could help developers fix the assumption before it becomes a problem far in the future and more difficult to fix.
Reproduce step
Root cause
EagerTest related test
Take com.hubspot.jinjava.EagerTest#itHandlesImportInDeferredIf as an example, the test fails at assertion in
ExpectedTemplateInterpreter.javaL45jinjava/src/test/java/com/hubspot/jinjava/ExpectedTemplateInterpreter.java
Lines 39 to 46 in ddd35c0
The reason why it fails is that the rendered string is determined by the order of
HashMap.entrySet(), which is called atAliasedEagerImportingStrategy.javaL255jinjava/src/main/java/com/hubspot/jinjava/lib/tag/eager/importing/AliasedEagerImportingStrategy.java
Line 255 in ddd35c0
However, the order of
HashMap.entrySet()is not guaranteed. It might change from time to time.NonDex would shuffle the map on each invocation of methods.
Therefore, the rendered string might change, and make the tests fail.
PyishObjectMapperTest
For PyishObjectMapperTest test, it is the similar reason.
In L25,
HashMaphas been used. However, the assertion in L29 is relied on the order ofHashMap.entrySet()jinjava/src/test/java/com/hubspot/jinjava/objects/serialization/PyishObjectMapperTest.java
Lines 24 to 31 in ddd35c0
Code change
To fix the test flakiness, I replaced
HashMapwithLinkedHashMapin AliasedEagerImportingStrategy.java and PyishObjectMapperTest.java, and exchange the order of string to aligned with the order of `LinkedHashMapFailed test
com.hubspot.jinjava.EagerTest#itHandlesImportInDeferredIf
com.hubspot.jinjava.EagerTest#itHandlesDoubleImportModification
com.hubspot.jinjava.EagerTest#itHandlesSameNameImportVar
com.hubspot.jinjava.EagerTest#itDoesNotOverrideImportModificationInFor
com.hubspot.jinjava.EagerTest#itRreconstructsValueUsedInDeferredImportedMacro
com.hubspot.jinjava.EagerTest#itAllowsVariableSharingAliasName
com.hubspot.jinjava.NonRevertingEagerTest#itHandlesImportInDeferredIf
com.hubspot.jinjava.NonRevertingEagerTest#itHandlesDoubleImportModification
com.hubspot.jinjava.NonRevertingEagerTest#itHandlesSameNameImportVar
com.hubspot.jinjava.NonRevertingEagerTest#itDoesNotOverrideImportModificationInFor
com.hubspot.jinjava.NonRevertingEagerTest#itRreconstructsValueUsedInDeferredImportedMacro
com.hubspot.jinjava.NonRevertingEagerTest#itAllowsVariableSharingAliasName
com.hubspot.jinjava.lib.tag.eager.EagerImportTagTest#itDefersTripleLayer
com.hubspot.jinjava.lib.tag.eager.EagerImportTagTest#itHandlesQuadLayerInDeferredIf
com.hubspot.jinjava.lib.tag.eager.EagerImportTagTest#itDoesNotSilentlyOverrideVariable
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapWithNullKeysAsEmptyString
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapEntrySet
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapEntrySetWithLimit
com.hubspot.jinjava.objects.serialization.PyishObjectMapperTest#itSerializesMapWithNullValues