homepage

Creating an enum subclass (ie: defining an enum) is slow. This dramatically impacts startup time of Python programs that import a bunch of potentially needed constant definitions at startup before any proper code executes.
How slow? So slow that a module defining a ~300 enums takes nearly 100ms just to import from its pyc file.
Example code: https://github.com/googleads/google-ads-python/blob/96fd08bb62435f1930df4871033ba8689333b67f/google/ads/google_ads/v2/services/enums.py
We've known this, we should do something about it. (Even if it means implementing the guts of the magic enum machinery in C.) ie, it came up in https://bugs.python.org/issue28637 as a stdlib startup time regression and is likely to come up in similar contexts elsewhere.

I was just looking at this problem, and creating a bare-bones, no safety belts version for use in the stdlib (no patch yet) which decreases Enum creation from 14x slower to only 6x slower. (Comparing to a class with simple attributes.)
Not sure if that's enough improvement, though.
If it needs to be even faster, a C version of that simplified Enum shouldn't be too hard. Anyone that uses the _simple_enum, though, should have a test that uses the full Enum and compares the two to make sure nothing got lost in translation.

New changeset dbac8f40e81eb0a29dc833e6409a1abf47467da6 by Ethan Furman in branch 'master':
bpo-38659: [Enum] add _simple_enum decorator (GH-25285)
https://github.com/python/cpython/commit/dbac8f40e81eb0a29dc833e6409a1abf47467da6

New changeset 503cdc7c124cebbd777008bdf7bd9aa666b25f07 by Ethan Furman in branch 'master':
Revert "bpo-38659: [Enum] add _simple_enum decorator (GH-25285)" (GH-25476)
https://github.com/python/cpython/commit/503cdc7c124cebbd777008bdf7bd9aa666b25f07

New changeset a02cb474f9c097c83cd444a47e9fb5f99b4aaf45 by Ethan Furman in branch 'master':
bpo-38659: [Enum] add _simple_enum decorator (GH-25497)
https://github.com/python/cpython/commit/a02cb474f9c097c83cd444a47e9fb5f99b4aaf45

Commit a02cb474f9c097c83cd444a47e9fb5f99b4aaf45 fails to build in the refleak buildbots:
https://buildbot.python.org/all/#/builders/75/builds/2/steps/5/logs/stdio
Example failure:
❯ ./python -m test test_enum -R :
0:00:00 load avg: 1.81 Run tests sequentially
0:00:00 load avg: 1.81 [1/1] test_enum
beginning 9 repetitions
123456789
.test test_enum failed -- Traceback (most recent call last):
 File "/home/pablogsal/github/python/master/Lib/test/test_enum.py", line 3700, in test_convert_repr_and_str
 self.assertEqual(format(test_type.CONVERT_STRING_TEST_NAME_A), '5')
AssertionError: 'CONVERT_STRING_TEST_NAME_A' != '5'
- CONVERT_STRING_TEST_NAME_A
+ 5
test_enum failed
== Tests result: FAILURE ==
1 test failed:
 test_enum
Total duration: 586 ms
Tests result: FAILURE

Can someone take a look?
As per the buildbot policy (https://discuss.python.org/t/policy-to-revert-commits-on-buildbot-failure/404) we may need to revert it if is not fixed in 24 because of the risk of masking errors.

Looks like this is the issue described in the comment here: https://github.com/python/cpython/blob/master/Lib/test/test_enum.py#L3691-L3692
On the first run you have the correct
 ('CONVERT_STRING_TEST_NAME_A', 5)
but later it turns into
 ('CONVERT_STRING_TEST_NAME_A', test.test_enum.CONVERT_STRING_TEST_NAME_A)
causing double-conversions of the enum elements. This causes the format(x) test to fail. You can re-create the same issue outside of the refleak by adding a simple:
 def test_convert_repr_and_str_again(self):
 self.test_convert_repr_and_str()
method.

`format` issue resolved, but not the refleak issue.

New changeset 37b173c5231b7b202859af97c17bc3d04b1e4c75 by Ammar Askar in branch 'master':
bpo-38659: Properly re-intialize module variables in test_enum (GH-25516)
https://github.com/python/cpython/commit/37b173c5231b7b202859af97c17bc3d04b1e4c75

Actually, I think that fixed the refleak issue as well.
Thanks, Ammar!

def setUp(self):
 # Reset the module-level test variables to their original integer
 # values, otherwise the already created enum values get converted
 # instead.
Why not doing that in a tearDown() method instead? What if you run explicitly a single test method?

Thanks Ammar for the fix. Unfortunately there is still some failures related to this on test_socker:
======================================================================
ERROR: test_msgflag_enum (test.test_socket.GeneralModuleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.refleak/build/Lib/test/test_socket.py", line 1969, in test_msgflag_enum
 enum._test_simple_enum(CheckedMsgFlag, socket.MsgFlag)
 File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-ppc64le.refleak/build/Lib/enum.py", line 1664, in _test_simple_enum
 raise TypeError('enum mismatch:\n %s' % '\n '.join(failed))
TypeError: enum mismatch:
 'MSG_TRUNC' member mismatch:
 extra key '_inverted_' in simple enum member 'MSG_TRUNC'
 'MSG_CTRUNC' member mismatch:
 extra key '_inverted_' in simple enum member 'MSG_CTRUNC'
----------------------------------------------------------------------

Gentle ping

Commenting out the enum tests reveals that test_socket has additional problems:
𓋹 ./python.exe -m test test_socket -R 3:3
0:00:00 load avg: 2.32 Run tests sequentially
0:00:00 load avg: 2.32 [1/1] test_socket
beginning 6 repetitions
123456
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg_into([buf], *args)
./Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2701: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg(bufsize, *args)
/Users/pgalindo3/github/python/master/Lib/test/test_socket.py:2792: RuntimeWarning: received malformed or improperly-truncated ancillary data
 result = sock.recvmsg_into([buf], *args)
.
test_socket leaked [20, 20, 20] file descriptors, sum=60
test_socket failed in 2 min 25 sec
== Tests result: FAILURE ==
1 test failed:
 test_socket
Total duration: 2 min 25 sec
Tests result: FAILURE

Unfortunately, I am being forced to revert commit a02cb474f9c097c83cd444a47e9fb5f99b4aaf45 due to have it failing of all refleak buildbots for more than two days.

My apologies, I was having hardware issues.
Checking it out now.

Thanks a lot Ethan. I will wait then for the investigation.

New changeset 6c681e1a4aa2dbca61be9a26c9257d7d25fa29a7 by Ethan Furman in branch 'master':
bpo-38659: [Enum] do not check '_inverted_' during simple test (GH-25566)
https://github.com/python/cpython/commit/6c681e1a4aa2dbca61be9a26c9257d7d25fa29a7

Pablo, did my latest patch resolved the errors?

Seems that the buildbots are going back to green so I will close the revert PR. THanks a lot, Ethan for the fix and the investigation!

Nobody seemed to mention it so I might as well: defining a regular Enum class takes an amount of time that is clearly quadratic in the number of attributes. That means that the problem is not Python-versus-C or small speed-ups or adding secret APIs to do the simple case faster. The problem is in the algorithm which needs to be fixed somewhere. My timings:
number of attributes time
1500 0.24s
3000 0.94s
6000 3.74s
12000 15.57s

The reason for that quadratic behavior is that for each new member (aka attribute), all the previous members must be checked to see if the new member is a duplicate.
In practice I wouldn't expect this to be a problem as most enums should be fairly small -- are there any real-world examples where there are more than, say, a hundred?