homepage

There is a number of issues with "async for".
1. When assigning to the target raises StopAsyncIteration (in custom __setitem__, __setattr__ or __iter__) it will be silenced and will cause to stop iteration.
2. StopAsyncIteration is dynamically looked up in globals. If set the global StopAsyncIteration or delete it from builtins (for example at the shutdown stage), this will break any "async for".
3. The f_lineno setter doesn't handle jumping into or out of the "async for" block. Jumping into is not forbidden, and jumping out doesn't update the stack correctly. This can cause a crash or incorrect behavior (like iterating wrong loop).
4. The compiler doesn't check all errors when creating new blocks. Some blocks are not used. And the resulting bytecode is suboptimal.
I'll create a series of pull request for fixing all this issue. Some of them can be backported. Others require changes in bytecode or are too hard for implementing in 3.7 and earlier versions (the related code was changed in issue17611). Some of them depend on other PRs or other issues (like issue33026) and need to wait until their be merged.

PR 6052 fixes issue 1. I don't know what is the best place for tests. There are two files with tests for "async for": test_coroutines.py and test_asyncgen.py. I'm not sure that new tests use the simplest way for testing this behavior. Could you please look at them Yury?
PR 6053 fixes issue 4. Adds missed error checks, removes unused variables and removes generating redundant bytecode.

Thanks so much for looking into this, Serhiy!
> 2. StopAsyncIteration is dynamically looked up in globals. If set the global StopAsyncIteration or delete it from builtins (for example at the shutdown stage), this will break any "async for".
IIRC I adapted the approach from some other place in compile.c. Quick looking at it reveals that the `assert` statement is compiled in a similar way w.r.t. how AssertionError is looked up at runtime. You might want to check if there are other places in compile.c that need to be fixed.
> PR 6052 fixes issue 1. I don't know what is the best place for tests. There are two files with tests for "async for": test_coroutines.py and test_asyncgen.py. I'm not sure that new tests use the simplest way for testing this behavior. Could you please look at them Yury?
I think the new tests are fine.

New changeset 24d3201eb7f0b39a7eaf2a5b2a2ceca10ad1f8eb by Serhiy Storchaka in branch 'master':
bpo-33041: Fixed bytecode generation for "async for" with a complex target. (#6052)
https://github.com/python/cpython/commit/24d3201eb7f0b39a7eaf2a5b2a2ceca10ad1f8eb

New changeset 67ee07795bcd84b679c000780212d4d81a1490a3 by Serhiy Storchaka in branch 'master':
bpo-33041: Add missed error checks when compile "async for" (#6053)
https://github.com/python/cpython/commit/67ee07795bcd84b679c000780212d4d81a1490a3

New changeset 9e94c0d3c78d1bc582c865240ed9353fe9689b2a by Serhiy Storchaka in branch '3.7':
[3.7] bpo-33041: Add missed error checks when compile "async for" (GH-6053) (GH-6060)
https://github.com/python/cpython/commit/9e94c0d3c78d1bc582c865240ed9353fe9689b2a

New changeset d0826340d96e0953793b86d0b8475d2f43a280b6 by Miss Islington (bot) in branch '3.6':
[3.7] bpo-33041: Add missed error checks when compile "async for" (GH-6053) (GH-6060)
https://github.com/python/cpython/commit/d0826340d96e0953793b86d0b8475d2f43a280b6

PR 6110 adds tests for jumping in/out of 'async with' blocks. It isn't directly related to 'async for', but it adds helpers that will be used also in tests for 'async for'.
I'm not sure this is a correct and good way of writing such tests.

New changeset bc300ce205f99acb1ef92c37de06dc76147e073b by Serhiy Storchaka in branch 'master':
bpo-33041: Add tests for jumps in/out of 'async with' blocks. (#6110)
https://github.com/python/cpython/commit/bc300ce205f99acb1ef92c37de06dc76147e073b

New changeset 773573e9ac654d4b5c6682e70360f75864acd80e by Serhiy Storchaka in branch '3.7':
[3.7] bpo-33041: Add tests for jumps in/out of 'async with' blocks. (GH-6110). (GH-6140)
https://github.com/python/cpython/commit/773573e9ac654d4b5c6682e70360f75864acd80e

New changeset 193760f00aacb122698674ed15327dba412653a5 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-33041: Add tests for jumps in/out of 'async with' blocks. (GH-6110). (GH-6141)
https://github.com/python/cpython/commit/193760f00aacb122698674ed15327dba412653a5

PR 6142 fixes issues 2 and 4. It adds a new opcode END_ASYNC_FOR and therefore can't be backported. END_ASYNC_FOR combines a number of other opcodes and guaranties using the true StopAsyncIteration. The new opcode is neccessary also for detecting an "async for" loop when jump. Besides introducing the new opcode the code of an "async for" loop was changed in a way that allowed to disallow jumping into an "async for" loop (this is the only part that can be backported). The final bytecode is much simpler. The compiler has been cleaned up and its code is now much simpler too. I expect also a performance boost, but don't know how to benchmark this.
Perhaps only the half of issue 4 (disallowing jumps into an "async for" loop) can be solved in 3.7 and 3.6.

I'm surprised, but seems PR 6154 fixes the whole issue 3 in 3.7.
It keeps only one POP_BLOCK corresponding to SETUP_LOOP. It also make the generated bytecode a tiny bit more efficient (less jumps).

New changeset 702f8f3611bc49b73772cce2b9b041bd11ff9b35 by Serhiy Storchaka in branch 'master':
bpo-33041: Rework compiling an "async for" loop. (#6142)
https://github.com/python/cpython/commit/702f8f3611bc49b73772cce2b9b041bd11ff9b35

New changeset b9744e924ca07ba7db977e5958b91cd8db565632 by Serhiy Storchaka in branch '3.7':
bpo-33041: Fixed jumping if the function contains an "async for" loop. (GH-6154)
https://github.com/python/cpython/commit/b9744e924ca07ba7db977e5958b91cd8db565632

New changeset 18d7edf32e6832a818621cb8cb3d144928eca232 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-33041: Fixed jumping if the function contains an "async for" loop. (GH-6154). (GH-6199)
https://github.com/python/cpython/commit/18d7edf32e6832a818621cb8cb3d144928eca232

New changeset 078c4e3519deeef8014541925da057bb064eb5a8 by Victor Stinner in branch 'master':
bpo-33041: Fix downcast warning on Windows (#6595)
https://github.com/python/cpython/commit/078c4e3519deeef8014541925da057bb064eb5a8

I didn't check if the new warning, fixed by my PR-6595 in the master branch, exists on Windows in 3.6 and 3.7. If it does, you might want to request a backport of my PR. Usually, I only fix compiler warnings in the master branch.