pyo's are not overwritten by different optimization levels

[abadger1999]

BPO	1538778
Nosy	@loewis, @warsaw, @brettcannon, @devdanzin, @merwok, @abadger

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-09-13.16:42:49.135>
created_at = <Date 2006-08-11.15:23:37.000>
labels = ['interpreter-core', 'type-feature']
title = "pyo's are not overwritten by different optimization levels"
updated_at = <Date 2010-09-13.18:12:21.008>
user = 'https://bugs.python.org/abadger1999'

bugs.python.org fields:

activity = <Date 2010-09-13.18:12:21.008>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-13.16:42:49.135>
closer = 'barry'
components = ['Interpreter Core']
creation = <Date 2006-08-11.15:23:37.000>
creator = 'abadger1999'
dependencies = []
files = []
hgrepos = []
issue_num = 1538778
keywords = []
message_count = 10.0
messages = ['54874', '54875', '54876', '82217', '116312', '116324', '116325', '116326', '116328', '116337']
nosy_count = 8.0
nosy_names = ['loewis', 'barry', 'brett.cannon', 'ajaksu2', 'abadger1999', 'eric.araujo', 'a.badger', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1538778'
versions = ['Python 3.2']

[abadger1999]

If I invoke "python -OO" on a set of files, python
creates a set of .pyo files. If I then realize this is
an incredibly stupid thing to do because it removes the
docstrings from the output and this program requires
those docstrings python does not give me a method of
overwriting the generated .pyos.

I anticipated that running "python -O" would detect
that the already generated .pyos were created in a
different optimixation level and overwrite them with
new .pyos. Barring that, I expected there to be a
command line switch to tell python to overwrite the
previously generated .pyo cache files. Finding
neither, I realized that the only way to recover
docstrings for people using python -O was to rm all the
generated .pyo files. This seems like a bug or a
missing feature.

[loewis]

Logged In: YES
user_id=21627

This is a feature request; classifying it as such. See also
PEP-304 (notice it has been withdrawn).

[abadger1999]

Logged In: YES
user_id=944205

Thanks. PEP-304 is a bit different. It is specifying the
ability to read and write bytecode to a different directory
structure while this is specifying that bytecode, once
written is taken as canoncical even when options on the
python commandline would change the bytecode that is written.

Another solution in addition to those above would be writing
files with different extensions for different optimization
levels although this seems worse than either of the other
two changes.

[devdanzin]

Confirmed in trunk.

[warsaw]

How is this different than overwriting pyc files, except that there's probably less of a use case for pycs? IOW, if we were to do something about this, it should probably be an option to ignore any existing pyc (or pyo if -O is used) files and write new ones. It *could* be a command line switch, but honestly I think "find ... -exec rm {}" is just as good.

I'd probably prefer adding a switch to Lib/py_compile's main to remove pyo/pyc's first or some such, but I'd still be -0 on supporting this directly in Python.

[abadger]

Here's a usage where this matters. It's a simplification of the bug report that I got that prompted me to open this. Let's say I have the following code:

/usr/lib/python2.7/site-packages/foo.py::

    def help():
        """I'm a little docstring, short and sweet"""
        print help.__doc__

/usr/bin/bar.py::
#!/usr/bin/python -tt
import sys
import foo

    if "--help" in sys.argv:
        foo.help()
    else:
        print "if you type --help, it's me you'll meet"

The system administrator on this box comes along and runs::

    $ PYTHONOPTIMIZE=2 bar.py
    if you type --help, it's me you'll meet

No problems apparent there but then, the user comes along later and runs::
$ PYTHONOPTIMIZE=1 ./bar.py --help
None

At this point the end user opens a bug against my software telling me that --help is broken which confuses everyone.

Not sure the best way to fix this -- ideas that pop into my head:

These solutions don't lead to a bug report as python does the right thing on its own:

• python reads and write separate pyo files for the different optimization levels (foo.pyo1 foo.pyo2)
• python autodetects whether the cached .pyo was written for the current optimization level and disregards its existence if there's a mismatch (leading to rewriting if the user has permissions, otherwise, recompilation but no writing to disk).

The following solution leads to a bug report but can be diagnosed with just the bug reporter. Note that PYTHONOPTIMIZE= /usr/bin/bar.py is almost as good in this situation:

• python has a commandline switch to disregard current .pyo files.

These solutions lead to a bug report but can be diagnosed with the cooperation of the system administrator in addition to the bug reporter:

• Command line switch that overwrites the .pyo files
• rm -rf *.pyo

[Arfrever]

compileall module supports -f (force) option, so you can use:
python3.2 -O -m compileall -f ${files}