How to properly handle indefinite generators in Python

Question

Let's say we have a generator that is indefinite, where new elements can arrive at any moment with significant (up to indefinite) delay.

An example of such generator is tail -F command. In python (omitting various edge cases) it could be implemented as the following:

def tail_follow(file):
 while True:
 line = file.readline()
 if line:
 yield line
 else:
 sleep(1.0)

Obvious problem with this generator is that it may cause caller's thread to sleep forever. Therefore it should provide caller's side a way to break iteration.

The solution I came up is the following:

def tail_follow(file, on_delay_callback):
 while True:
 line = file.readline()
 if line:
 yield line
 else:
 if on_delay_callback():
 break
 else:
 sleep(1.0)

Is this the only way to get this behavior with Python? I know that there is a send function that allows 2-way data transfer, can it be used to make solution more pythonic?

Answer 1

I don't see the problem with your approach. I think you are almost there. If you change your generator to yield None when there is nothing to return, then you can simply test for that.

See the following example:

#!/usr/bin/python
import sys
import time
tailpath = "./tailfile"
fd = open(tailpath)
def tail_follow(file):
 while True:
 line = file.readline()
 if line:
 yield line
 else:
 yield None
if __name__ == "__main__":
 for line in tail_follow(fd):
 if line:
 print ("found: %s" % (line))
 else:
 # do something useful
 time.sleep(1.0)

Or alternatively:

if __name__ == "__main__":
 mytail = tail_follow(fd)
 while True:
 # do something useful
 time.sleep(1.0)
 # anything to tail ?
 line = next(mytail)
 if line:
 print ("found: %s" % (line))

Jeff Knupp has a nice post about infinite generators.

• Kentzo: I guess if line: should be if line is None: in __main__.
• Kentzo: Can you think of any example in standard library (maybe asyncio?) that does it?
• NZD: In both examples the if line: is correct because you want to do something with the result if tail_follow yields something and ignore it when it yields None. You can easily see that it works if you execute touch ./tailfile to create the file to tail and then echo abc>>./tailfile to add characters to it. In a second shell you then call the script and every time you echo something in the first shell to .tailfile, it is printed in the second shell. I'm not familiar with asyncio.
• Kentzo: I meant if line is not None, sorry. if line will fail if line is empty which isn't desired behavior.
• NZD: Both if line: and if line is not None: behave the same here (1). If the file your tailing contains an empty line, there is still data: line will contain \n and the examples will print an empty line. If you don't want empty lines, then your application can check for them and ignore them. This leaves the generator generic i.e. contains no behaviour or design decisions. (1) There are cases in which the behaviour is different e.g. if the variable would be a number.