Python 3.5.2 non-segfaulting bugs (from PyPy)

[arigo]

BPO	28884
Nosy	@arigo, @ncoghlan, @serhiy-storchaka, @mattip, @iritkatriel
Dependencies	bpo-5322: object.new argument calling autodetection faulty bpo-18961: Non-UTF8 encoding line bpo-25937: DIfference between utf8 and utf-8 when i define python source code encoding. bpo-28891: Standardise more behaviours for zero-argument super() class and classcell

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-12-06.11:55:03.305>
labels = ['type-bug']
title = 'Python 3.5.2 non-segfaulting bugs (from PyPy)'
updated_at = <Date 2021-12-12.19:21:22.599>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2021-12-12.19:21:22.599>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2016-12-06.11:55:03.305>
creator = 'arigo'
dependencies = ['5322', '18961', '25937', '28891']
files = []
hgrepos = []
issue_num = 28884
keywords = []
message_count = 18.0
messages = ['282526', '282527', '282528', '282529', '282530', '282531', '282532', '282533', '282534', '282535', '282536', '282544', '282545', '282586', '282613', '382587', '382588', '408406']
nosy_count = 5.0
nosy_names = ['arigo', 'ncoghlan', 'serhiy.storchaka', 'mattip', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28884'
versions = ['Python 3.5']

[arigo]

As discussed on python-dev, I am creating omnibus issues from the lists of crashers, of wrong-according-to-the-docs, and of strange-behavior-only issues that I found while developing Python 3.5.2 support for PyPy. These occur with CPython 3.5.2 but most of them are likely still here in trunk.

This is the issue containing the bugs that are wrong according to documentation, or at least clearly (imo) unexpected behavior.

[arigo]

(B1) on modern Linux: if the first call in the process to
socketpair() ends in a EINVAL, then cpython will (possibly wrongly)
assume it was caused by SOCK_CLOEXEC and not use SOCK_CLOEXEC at all
in the future

[arigo]

(B2) fcntl.ioctl(x, y, buf, mutate_flag): mutate_flag is there for the case
of buf being a read-write buffer, which is then mutated in-place.
But if we call with a read-only buffer, mutate_flag is ignored (instead
of rejecting a True value)---ioctl(x, y, "foo", True) will not actually
mutate the string "foo", but the True is completely ignored. (I think
this is a bug introduced during the Argument Clinic refactoring.)

[arigo]

(B3) re.sub(b'y', bytearray(b'a'), bytearray(b'xyz')) -> b'xaz'

re.sub(b'y', bytearray(b'\\n'), bytearray(b'xyz')) -> internal TypeError

[arigo]

(B4) if you have a stack of generators where each is in 'yield from' from
the next one, and you call '.next()' on the outermost, then it enters
and leaves all intermediate frames. This is costly but may be
required to get the sys.settrace()/setprofile() hooks called.
However, if you call '.throw()' or '.close()' instead, then it uses a
much more efficient way to go from the outermost to the innermost
frame---as a result, the enter/leave of the intermediate frames is not
invoked. This can confuse coverage tools and profilers. For example,
in a stack f1()->f2()->f3(), vmprof would show f3() as usually
called via f2() from f1() but occasionally called directly from f1().

[arigo]

(B5) this is an old issue that was forgotten twice on the
issue tracker: class C: __new__=int.__new__ and class C(int): __new__=object.__new__ can each be instantiated, even though they
shouldn't. This is because __new__ is completely ignored if it is
set to any built-in function that uses tp_new_wrapper as its C code
(many of the built-in types' __new__ are like that).
http://bugs.python.org/issue1694663#msg75957,
http://bugs.python.org/issue5322#msg84112. In (at least) CPython 3.5,
a few classes work only thanks to abuse of this bug: for example,
io.UnsupportedOperation.__new__(io.UnsupportedOperation) doesn't
work, but that was not noticed because io.UnsupportedOperation()
mistakenly works.

[arigo]

(B6) this program fails the check for no sys.exc_info(), even though at
the point this assert runs (called from the <== line) we are not in
any except/finally block. This is a generalization of
test_exceptions:test_generator_doesnt_retain_old_exc::

    import sys

    def g():
        try:
            raise ValueError
        except ValueError:
            yield 1
        assert sys.exc_info() == (None, None, None)
        yield 2

    gen = g()

try:
    raise IndexError
except IndexError:
    assert next(gen) is 1
assert next(gen) is 2    # <==

[arigo]

(B7) frame.clear() does not clear f_locals, unlike what a test says
(Lib/test/test_frame.py)::

    def test_locals_clear_locals(self):
        # Test f_locals before and after clear() (to exercise caching)
        f, outer, inner = self.make_frames()
        outer.f_locals
        inner.f_locals
        outer.clear()
        inner.clear()
        self.assertEqual(outer.f_locals, {})
        self.assertEqual(inner.f_locals, {})

This test passes, but the C-level PyFrameObject has got a strong
reference to f_locals, which is only updated (to be empty) if the
Python code tries to read this attribute. In the normal case,
code that calls clear() but doesn't read f_locals afterwards will
still leak everything contained in the C-level f_locals field. This
can be shown by this failing test::

    import sys

    def g():
        x = 42
        return sys._getframe()

    frame = g()
    d = frame.f_locals
    frame.clear()
    print(d)
    assert d == {}   # fails!  but 'assert d is frame.f_locals' passes,
                     # which shows that this dict is kept alive by
                     # 'frame'; and we've seen that it is non-empty
                     # as long as we don't read frame.f_locals.

[arigo]

(B8) also discussed in connection with https://bugs.python.org/issue28427

weak dicts (both kinds) and weak sets have an implementation of
__len__ which doesn't give the "expected" result on PyPy, and in some
cases on CPython too. I'm not sure what is expected and what is not.
Here is an example on CPython 3.5.2+ (using a thread to run the weakref
callbacks only, not to explicitly inspect or modify 'd')::

    import weakref, _thread
    from queue import Queue

    queue = Queue()
    def subthread(queue):
        while True:
            queue.get()
    _thread.start_new_thread(subthread, (queue,))

    class X:
        pass
    d = weakref.WeakValueDictionary()
    while True:
        x = X()
        d[52] = x
        queue.put(x)
        del x
        while list(d) != []:
            pass
        assert len(d) == 0  # we've checked that list(d)==[], but this may fail

On CPython I've seen the assert fail only after editing the function
WeakValueDictionary.init.remove() to add time.sleep(0.01) as
the first line. Otherwise I guess the timings happen to make that test
pass.

[arigo]

(B9) CPython 3.5.2: this nonlocal seems not to have a reasonable
effect (note that if we use a different name instead of __class__,
this example correctly complain that there is no binding in the outer
scope of Y)::

    class Y:
        class X:
            nonlocal __class__
            __class__ = 42
        print(locals()['__class__'])     # 42
        print(__class__)                 # but this is a NameError

[arigo]

(B10) Follow-up on issue bpo-25388: running python x.py if x.py contains
the following bytes...

• b"#\xfd\n" => we get a SyntaxError: Non-UTF-8 code
• b"# coding: utf-8\n#\xfd\n" => we get no error!

[serhiy-storchaka]

About (B10) see bpo-18961 and bpo-25937.

[serhiy-storchaka]

About (B3): Only str and bytes are supported. The support of bytearray templates is not documented and is not guarantied.

[ncoghlan]

I broke B9 for the "nonlocal __class__" misbehaviour out to a separate issue: http://bugs.python.org/issue28891

I love zero-argument super() as a user, but as an interpreter maintainer... ugh :P