Add subprocess.iter_output() convenience function

[ncoghlan]

BPO	13578
Nosy	@ncoghlan, @giampaolo

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-12-11.05:05:54.306>
labels = ['type-feature', 'library']
title = 'Add subprocess.iter_output() convenience function'
updated_at = <Date 2012-06-12.12:00:10.774>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2012-06-12.12:00:10.774>
actor = 'ncoghlan'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2011-12-11.05:05:54.306>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 13578
keywords = []
message_count = 3.0
messages = ['149210', '149215', '152794']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'giampaolo.rodola', 'rosslagerwall']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13578'
versions = ['Python 3.4']

[ncoghlan]

subprocess.check_output() is nice, but doesn't help if you want to process the piped data line-by-line.

Currently, that means you have to do the full Popen dance if you want access to each line of output as it becomes available.

This RFE is for a subprocess.iter_output() module level helper that:

1. Starts the subprocess
2. Yield the individual lines of output as they are produced by the subprocess
3. Cleans up (including checking for errors) at the end

This biggest challenge I have noticed so far in exploring this is how to handle timeouts on Windows - on Unix, select.select() can do the job, but that won't handle pipes in the Windows case.

[ncoghlan]

You can see a version of this here:

https://bitbucket.org/ncoghlan/shell_command/src/2b1988b072aa/shell_command.py#cl-157

[giampaolo]

This biggest challenge I have noticed so far in exploring
this is how to handle timeouts on Windows

I haven't actually looked into it but this somewhat recalls:
http://bugs.python.org/issue1191964
Since bpo-1191964 is supposed to provide async subprocess I/O I presume Windows timeouts are also involved somewhere in the provided patch/recipe.

[devdanzin]

Does this one still make sense?

[ncoghlan]

Unfortunately, it does. Even subprocess.run only readily supports "live output to standard streams, bypassing the parent process" and "pipe output to parent process, hide output from the standard streams while the child process is running".

It's far from clear what a good convenience API for this would actually look like, though, given that things get potentially complicated fast:

• if stdout & stderr remain split, some form of parallel execution is definitely necessary
• even if they're not split, the parent process needs to choose between waiting for the pipes to close before waiting for the process, or waiting for the process to terminate, and dealing with the pipes closing
• child processes may turn off their line buffering if they detect their standard streams are non-interactive

Dropping down to working with Popen directly provides the tools needed to deal with those questions, it's just irritating that we're not able to provide a nicer abstraction over all those details.