Run the CPython typing test suite against typing-extensions

I propose to add a workflow that:

Grabs test_typing.py from CPython
Replaces all typing imports with typing_extensions through some hackery (either sed or some cleverness with importlib)
Runs the test suite

This would help us catch issues where something got changed in CPython and we didn't reapply it to our variant. For example, this would allow us to catch issues in our TypedDict implementation, which is currently different from CPython's on all versions.

We'd have to run this only on matching versions (e.g., run the CPython tests from the 3.14 branch on Python 3.14) because the CPython tests may rely on new syntax or standard library features.

Feel free to pick this up if you see this and are interested! I may do it eventually when I find time.

Note that we already run test_typing after typing_extensions has been installed, but that's not as comprehensive as what's being proposed here:

typing_extensions/.github/workflows/ci.yml

Lines 74 to 81 in 44de568

    
                 - name: Test CPython typing test suite 
        
                   # Test suite fails on PyPy even without typing_extensions 
        
                   if: ${{ !startsWith(matrix.python-version, 'pypy') }} 
        
                   run: | 
        
                     cd src 
        
                     # Run the typing test suite from CPython with typing_extensions installed, 
        
                     # because we monkeypatch typing under some circumstances. 
        
                     python -c 'import typing_extensions; import test.__main__' test_typing -v

Right, that just tests that the monkeypatches we do to typing don't break things.

I decided to take this issue, and here are the errors that were received as a result of running it (some of them are related to the fact that there are __module__ checks):
https://github.com/donBarbos/typing_extensions/actions/runs/16691671387/
https://github.com/donBarbos/typing_extensions/actions/runs/16691671387/job/47250262252 (example job on 3.14)

Thanks, looks like a bunch of them are the result of typing tests testing some private objects in the module. Possibly fixable in the test by monkeypatching typing_extensions to have a module __getattr__ method that delegates to typing.

@donbarbos

I am trying to have a look at the errors that come up with @donbarbos currenty test runs over the CPython test (in 3.14 for now) but running into the problem where the typing_extensions tests explicitely expect the opposite.

test_typing.py:

    def test_cannot_make_mutable_key_readonly(self):
        class Base(TypedDict):
            a: int

        with self.assertRaises(TypeError):
            class Child(Base):
                a: ReadOnly[int]

test_typing_extensions.py:

    def test_make_mutable_key_readonly(self):
        class Base(TypedDict):
            a: int

        self.assertEqual(Base.__readonly_keys__, frozenset())
        self.assertEqual(Base.__mutable_keys__, frozenset({'a'}))

        class Child(Base):
            a: ReadOnly[int]  # type checker error, but allowed at runtime

        self.assertEqual(Child.__readonly_keys__, frozenset({'a'}))
        self.assertEqual(Child.__mutable_keys__, frozenset())

Very simple example showing this:

from typing_extensions import TypedDict as TypedDictExt
from typing_extensions import ReadOnly as ReadOnlyExt
class Base(TypedDictExt):
        a: int

class Child(Base):
    a: ReadOnlyExt[int]  # type checker error, but allowed at runtime

from typing import TypedDict, ReadOnly
class Base(TypedDict):
        a: int

class Child(Base):
    a: ReadOnly[int]  # type checker error, but allowed at runtime

$ uv run --python 3.14 my_test.py
Traceback (most recent call last):
  File "D:\Programming\Projects\typing_extensions\src\my_test.py", line 14, in <module>
    class Child(Base):
    ^^^^^^^^^^^^^^^^^^
        a: ReadOnly[int]  # type checker error, but allowed at runtime
        ^^^^^^^^^^^^^^^^
  File "C:\Users\Jan-Eric\AppData\Roaming\uv\python\cpython-3.14.0a4-windows-x86_64-none\Lib\typing.py", line 3206, in __new__
    raise TypeError(
    ...<2 lines>...
    )
TypeError: Cannot override mutable key 'a' with read-only key

I couldnt find anything about the TypeError in the docs, but it has been present in the code since the introduction: https://github.com/python/cpython/blob/3.13/Lib/typing.py#L3208

If i change this to match CPythons i get 3.14 and 3.13 tests to pass:

https://github.com/JanEricNitschke/typing_extensions/actions/runs/17633620675
main...JanEricNitschke:typing_extensions:run-patched-test_typing

I found this commit: d6c50f5 which linked to this dicussion: https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867/40 but the point was not raised on the PR into CPython which included the TypeError: python/cpython#116350

We should making typing-extensions match the typing behavior. (I might have preferred not erroring at runtime but it's too late now.) Finding this kind of discrepancy is exactly why I opened this issue.

Nice, ok. Actually currently going through these to try to address them. Probably create individual PRs for the first two issues tomorrow.

@JelleZijlstra

@JelleZijlstra sorry, but I want to clarify that we did decide to revert typing_extensions behavior related to the runtime error in PEP 705 that I mentioned here?

It seems that, although changing the typing does break backward compatibility, this option is still preferable. The drop of runtime error was intentional, and it looks like the same behavior was meant to be introduced into typing as well.
Maybe it would be worth opening a separate issue to discuss the implications of this backward incompatibility (or perhaps this isn’t really something that needs discussion, I'm not sure :).

I don't feel too strongly about which of typing and typing-extensions should change, but it's easier to change typing-extensions. Probably best to open a new issue to lay out the options and discuss.

Is that something fro here, the CPython repo or discuss?

Let's start with here.

From what i understand the TypeError in CPython for this behaviour is pretty unique, as i dont think any other such attribute is runtime enforced.

typing.Final and typing.ReadOnly even specify "There is no runtime checking for this property." in the docs. (Although that obviously refers them only being read, but still).

Additionally the typing docs in general state "The Python runtime does not enforce function and variable type annotations. They can be used by third party tools such as type checkers, IDEs, linters, etc.". Lastly, neither the docs nor the PEP say anything about such a type error existing.

So i definitely feel like the optimal thing here would be to not have a runtime error.

On top of that i feel like removing a TypeError is probably a less intrusive change than adding one, even though changes in CPython are obviously more impactful than ones in typing_extensions.

I tried to see what tests exactly are failing when we run the CPython tests:

Some i think are actual potential issues and i have created PRs for most of those. Although some are just error messages.
The only one that doesnt have a PR yet is this one.

Otherwise there are some tests that explicitly assert behavior we are changing with backports. These can be silenced/fixed with some seds that fix the line or skip the test.

And there are a lot of tests that check reprs, assert identity or do pickling. For these i dont have the experience yet to know which are actual issues we want/need to address and which tests we should just fix/skip.

And the last set are where __module__ gets set wrong on exec and becomes "main" instead of None.

I tried to collect some information in this issue: JanEricNitschke#1

Here are the changes i have made so far: main...JanEricNitschke:typing_extensions:run-patched-test_typing

And here is the latest run: https://github.com/JanEricNitschke/typing_extensions/actions/runs/17707749820/job/50322109340

Navigation Menu

Repository navigation

Run the CPython typing test suite against typing-extensions #606

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	- name: Test CPython typing test suite
	# Test suite fails on PyPy even without typing_extensions
	if: ${{ !startsWith(matrix.python-version, 'pypy') }}
	run: \|
	cd src
	# Run the typing test suite from CPython with typing_extensions installed,
	# because we monkeypatch typing under some circumstances.
	python -c 'import typing_extensions; import test.__main__' test_typing -v

Run the CPython typing test suite against typing-extensions #606

Description

Activity

AlexWaygood commented on May 25, 2025

JelleZijlstra commented on May 25, 2025

donbarbos commented on Aug 2, 2025

JelleZijlstra commented on Aug 2, 2025

donbarbos commented on Aug 4, 2025

JanEricNitschke commented on Sep 11, 2025

JelleZijlstra commented on Sep 11, 2025

JanEricNitschke commented on Sep 11, 2025

donbarbos commented on Sep 11, 2025

JelleZijlstra commented on Sep 11, 2025

JanEricNitschke commented on Sep 12, 2025

JelleZijlstra commented on Sep 12, 2025

JanEricNitschke commented on Sep 12, 2025

JanEricNitschke commented on Sep 14, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions