Pure Python operator.index doesn't match the C version.

[mdickinson]

BPO	18712
Nosy	@arigo, @mdickinson, @ericsnowcurrently, @zware, @serhiy-storchaka, @corona10, @iritkatriel
Dependencies	bpo-17576: PyNumber_Index() is not int-subclass friendly (or operator.index() docs lie)
Files	operator_index.patch x.py operator_index_2.patch y.py operator_index_3.patch operator_index_4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-08-12.14:09:38.011>
labels = ['extension-modules', 'type-bug', '3.11']
title = "Pure Python operator.index doesn't match the C version."
updated_at = <Date 2021-10-02.15:33:51.696>
user = 'https://github.com/mdickinson'

bugs.python.org fields:

activity = <Date 2021-10-02.15:33:51.696>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2013-08-12.14:09:38.011>
creator = 'mark.dickinson'
dependencies = ['17576']
files = ['31273', '31334', '31335', '31336', '31344', '31353']
hgrepos = []
issue_num = 18712
keywords = ['patch']
message_count = 21.0
messages = ['194966', '195000', '195006', '195064', '195451', '195454', '195457', '195458', '195474', '195476', '195489', '195490', '195510', '195513', '195545', '195546', '195547', '195548', '195549', '195665', '401603']
nosy_count = 7.0
nosy_names = ['arigo', 'mark.dickinson', 'eric.snow', 'zach.ware', 'serhiy.storchaka', 'corona10', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18712'
versions = ['Python 3.11']

Linked PRs

• gh-62912: fix pure-Python operator.index() #148023
• gh-62912: return correct exception in python versions for operator.concat/iconcat #148025

[mdickinson]

Nitpick: the pure Python version of operator.index (new in Python 3.4, introduced in issue bpo-16694) doesn't match the C version, in that it looks up __index__ on the object rather than the class.

iwasawa:cpython mdickinson$ ./python.exe
Python 3.4.0a1+ (default:9e61563edb67+, Aug 12 2013, 14:45:12) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import support
>>> py_operator = support.import_fresh_module('operator', blocked=['_operator'])
>>> c_operator = support.import_fresh_module('operator', fresh=['_operator'])
>>> class A(int): pass
... 
>>> a = A(42); a.__index__ = lambda: 1729
>>> 
>>> py_operator.index(a)
1729
>>> c_operator.index(a)
42

[serhiy-storchaka]

We can use type(a).__index__(a). Should we also correct the documentation for operator.index() and operator.length_hint()?

[mdickinson]

Yes, I think it would make sense to fix the docs as well, at least for Python 3.4. Probably not worth it for the maintenance releases.

[serhiy-storchaka]

Here is a patch. Now the code of operator.index() becomes even more complicated. Perhaps you want suggest other wording for documentation?

Some code in stdlib (pyio.py, bz2.py, connection.py) uses a.__index_() instead of type(a).__index__(a) (with replacing AttributeError to TypeError). Is it worth to change?

[arigo]

Just mentioning it here again, but "type(a).__index__(a)" is still not perfectly correct. Attached is a case where it differs.

I think you get always the correct answer by evaluating "range(a).stop". It's admittedly obscure... For example:

    class A:
        def __index__(self):
            return -42**100

    a = A()
    print(range(a).stop)

[serhiy-storchaka]

The difference doesn't look significant. In all cases the TypeError is raised.

But there was other differences between C and Python versions -- when __index__() returns non-integer. Updated patch fixes this.

[mdickinson]

Just mentioning it here again, but "type(a).__index__(a)" is still not perfectly correct.

Hmm. "type(a).__dict__'__index__'" ? (With suitable error checks, as in Serhiy's patch.)

[arigo]

The difference doesn't look significant. In all cases the TypeError is raised.

Ok, so here is another case. (I won't go to great lengths trying to convince you that there is a problem, because the discussion already occurred several times; google for example for "_PyType_Lookup pure Python")

[serhiy-storchaka]

Hmm. "type(a).__dict__'__index__'" ?

This variant fails on:

    class A(int):
        @staticmethod
        def __index__():
            return 42

[mdickinson]

Serhiy: Yep, or even on bool. Thanks.

Armin: I don't think either of us thinks there isn't a problem here. :-)
The Google search you suggested didn't turn up a whole lot of useful information for me. Was there a discussion of this on python-dev at some point? (And if not, should there be?)

For this *particular* issue, it seems we can't exactly reproduce. type(a).__index__(a) seems like the best practical approximation to the true behaviour. I'm guessing that it's fairly rare to have useful definitions of __index__ on a metaclass.

[arigo]

This may have been the most recent discussion of this idea (as far as I can tell):
http://mail.python.org/pipermail//python-ideas/2012-August/016036.html

Basically, it seems to be still unresolved in the trunk Python; sorry, I thought by now it would have been resolved e.g. by the addition of a method on types or a function in the operator module. In the absence of either, you need either to simulate its behavior by doing this:

    for t in type(a).__mro__:
        if '__index__' in t.__dict__:
            return t.__dict__['__index__'](a)

Or you can piggyback on an unrelated call that simply causes the C-level PyNumber_Index() to be called:

    return range(a).stop

[arigo]

Sorry, realized that my pure Python algorithm isn't equivalent to _PyType_Lookup() --- it fails the staticmethod example of Serhiy. A closer one would be:

    for t in type(a).__mro__:
        if '__index__' in t.__dict__:
            return t.__dict__['__index__'].__get__(a)()

But it's still not a perfect match. I think right now the only "perfect" answer is given by workarounds like "range(a).stop"...

[serhiy-storchaka]

Here is updated patch which uses Armin's algorithm for lookup special methods and adds special case for int subclasses in index().

I have no idea how the documentation should look.