Skip to content

Instantly share code, notes, and snippets.

@Gobot1234

Gobot1234/type-defaults-for-typevar-likes.md Secret

Last active July 14, 2022 19:51
Show Gist options
  • Save Gobot1234/8c9bfe8eb88f5ad42bf69b6f118033a7 to your computer and use it in GitHub Desktop.
Save Gobot1234/8c9bfe8eb88f5ad42bf69b6f118033a7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
type-defaults-for-typevar-likes.md

PEP: 9999 Title: Type defaults for TypeVars Version: $Revision$ Last-Modified: $Date$ Author: James Hilton-Balfe [email protected] Sponsor: Jelle Zijlstra [email protected] Discussions-To: [email protected] Status: Draft Type: Standards Track Topic: Typing Content-Type: text/x-rst Created: 14-Jul-2022 Python-Version: 3.12

Type defaults for TypeVars

Abstract

This PEP introduces the concept of type defaults for TypeVars, which act as defaults for a type parameter when none is specified.

Motivation

T = TypeVar("T", default=int)  # This means that if no type is specified T = int

@dataclass
class Box(Generic[T]):
    value: T | None = None

reveal_type(Box())                      # type is Box[int]
reveal_type(Box(value="Hello World!"))  # type is Box[str]

One place this regularly comes up is Generator. I propose changing the stub definition to something like:

YieldT = TypeVar("YieldT")
SendT = TypeVar("SendT", default=None)
ReturnT = TypeVar("ReturnT", default=None)

class Generator(Generic[YieldT, SendT, ReturnT]): ...

Generator[int] == Generator[int, None] == Generator[int, None, None]

This is also useful for a Generic that is commonly over one type.

class Bot: ...

BotT = TypeVar("BotT", bound=Bot, default=Bot)

class Context(Generic[BotT]):
    bot: BotT

class MyBot(Bot): ...

reveal_type(Context().bot)         # type is Bot  # notice this is not Any which is what it would be currently
reveal_type(Context[MyBot]().bot)  # type is MyBot

Not only does this improve typing for those who explicitly use it. It also helps non-typing users who rely on auto-complete to speed up their development.

This design pattern is common in projects like:

  • discord.py - where the example above was taken from.
  • NumPy - the default for types like ndarray's dtype would be float64. Currently it's Unkwown or Any.
  • TensorFlow (this could be used for Tensor similarly to numpy.ndarray and would be useful to simplify the definition of Layer)

This proposal could also be used on builtins.slice where the parameter should start default to int, stop default to start and step default to int | None

StartT = TypeVar("StartT", default=int)
StopT = TypeVar("StopT", default=StartT)
StepT = TypeVar("StepT", default=int | None)

class slice(Generic[StartT, StopT, StepT]): ...

Specification

Because having the default be the same as the bound is so common, the default argument defaults to bound. i.e.

TypeVar("T", bound=int) == TypeVar("T", bound=int, default=int)

The logic for determining the default is as follows:

  1. An explicit default
  2. The bound if it is not None
  3. A "missing" sentinel

Default ordering and subscription rules

The order for defaults should follow the standard function parameter rules, so a TypeVar with no default cannot follow one with a default value. Doing so should ideally raise a TypeError in typing._GenericAlias/types.GenericAlias, and a type checker should flag this an error.

DefaultStrT = TypeVar("DefaultStrT", default=str)
DefaultIntT = TypeVar("DefaultIntT", default=int)
DefaultBoolT = TypeVar("DefaultBoolT", default=bool)
T = TypeVar("T")
T2 = TypeVar("T2")

class NonDefaultFollowsDefault(Generic[DefaultStrT, T]): ...  # Invalid: non-default TypeVars cannot follow ones with defaults


class NoNonDefaults(Generic[DefaultStrT, DefaultIntT]): ...

(
    NoNoneDefaults ==
    NoNoneDefaults[str] ==
    NoNoneDefaults[str, int]
)  # All valid


class OneDefault(Generic[T, DefaultBoolT]): ...

OneDefault[float] == OneDefault[float, bool]  # Valid


class AllTheDefaults(Generic[T1, T2, DefaultStrT, DefaultIntT, DefaultBoolT]): ...

AllTheDefaults[int]  # Invalid: expected 2 arguments to AllTheDefaults
(
    AllTheDefaults[int, complex] ==
    AllTheDefaults[int, complex, str] ==
    AllTheDefaults[int, complex, str, int] ==
    AllTheDefaults[int, complex, str, int, bool]
)  # All valid

This cannot be enforce at runtime for functions, for now, but in the future, this might be possible (see Interaction with PEP 695).

Generic TypeAliases

Generic TypeAliases should be able to be further subscripted following normal subscription rules. If a TypeVar with a default which hasn't been overridden it should be treated like it was substituted into the TypeAlias. However, it can be specialised further down the line.

class SomethingWithNoDefaults(Generic[T, T2]): ...

MyAlias: TypeAlias = SomethingWithNoDefaults[int, DefaultStrT]  # valid
reveal_type(MyAlias)        # type is SomethingWithNoDefaults[int, str]
reveal_type(MyAlias[bool])  # type is SomethingWithNoDefaults[int, bool]

MyAlias[bool, int]  # Invalid: too many arguments passed to MyAlias

Subclassing

Subclasses of Generics with TypeVars that have defaults behave similarly to Generic TypeAliases.

class SubclassMe(Generic[T, DefaultStrT]): ...

class Bar(SubclassMe[int, DefaultStrT]): ...
reveal_type(Bar)        # type is Bar[str]
reveal_type(Bar[bool])  # type is Bar[bool]

class Foo(SubclassMe[int]): ...

reveal_type(Spam)  # type is <subclass of SubclassMe[int, int]>
Foo[str]  # Invalid: Foo cannot be further subscripted

class Baz(Generic[DefaultIntT, DefaultStrT]): ...

class Spam(Baz): ...
reveal_type(Spam)  # type is <subclass of Baz[int, str]>

Using bound and default

If both bound and default are passed default must be a subtype of bound.

TypeVar("Ok", bound=float, default=int)     # Valid
TypeVar("Invalid", bound=str, default=int)  # Invalid: the bound and default are incompatible

Constraints

For constrained TypeVars, the default needs to be one of the constraints. It would be an error even if it is a subtype of one of the constraints.

TypeVar("Ok", float, str, default=float)     # Valid
TypeVar("Invalid", float, str, default=int)  # Invalid: excpected one of float or str got int

Function Defaults

If a TypeVar with a default annotates a function parameter: the parameter's runtime default must be specified if it only shows up once in the input parameters.

DefaultIntT = TypeVar("DefaultIntT", default=int)

def bar(t: DefaultIntT) -> DefaultIntT: ...  # Invalid: `t` needs a default argument

The TypeVar's default should also be compatible with the parameter's runtime default, this desugars to a series of overloads, but this implementation would be much cleaner.

def foo(x: DefaultIntT | None = None) -> DefaultIntT:
    if x is None:
        return 1234
    else:
        return t

# Would be equivalent to

@overload
def foo() -> int: ...
@overload
def foo(x: T) -> T: ...
def foo(x=None) -> Any:
    if x is None:
        return 1234
    else:
        return t

Defaults for parameters aren't required if other parameters annotated with the same TypeVar already have defaults.

DefaultFloatT = TypeVar("DefaultFloatT", default=float)

def bar(a: DefaultFloatT, b: DefaultFloatT = 0) -> DefaultFloatT: ...  # Valid

bar(3.14)

def foo(a: DefaultFloatT, b: list[DefaultFloatT] = [0.0]) -> DefaultFloatT: ...

reveal_type(foo())                 # type is float
reveal_type(foo("hello"))          # type is str
reveal_type(foo("hello", ["hi"]))  # type is str

In classes defaults are erased if the method only uses the class's type parameters.

class MyList(Generic[DefaultFloatT]):
    def __init__(self, values: list[DefaultFloatT] | None = None):
        self.values = values or []
    
    # Would be equivalent to
    # @overload
    # def __init__(self): ...
    # @overload
    # def __init__(self, values: list[T] | None): ...
    # def __init__(self, values: list[T] | None = None):
    #     self.values = values or []

    def append(self, value: DefaultFloatT):  
        # `value` here doesn't need a default argument because List is transformed into:
        #
        # class MyList(Generic[T]):
        #     values: list[T]
        #     def append(self, value: T): ...
        #
        # where T is probably float
        self.values.append(value)

Implementation

At runtime, this would involve the following changes to typing.TypeVar:

  • the type passed to default would be available as a __default__ attribute.

The following changes would be required to both GenericAliases:

  • logic to determine the defaults required for a subscription.

    • potentially a way construct types.GenericAliases using a classmethod to allow for defaults in __class_getitem__ = classmethod(GenericAlias) i.e. GenericAlias.with_type_var_likes().

      # _collections_abc.py

_sentinel = object()

# NOTE: this is not actually typing.TypeVar, that's in typing.py,
#       this is just to trick is_typevar() in genericaliasobject.c
class TypeVar:
    __module__ = "typing"

    def __init__(self, name, *, default=_sentinel):
        self.__name__ = name
        self.__default__ = default

YieldT = TypeVar("YieldT")
SendT = TypeVar("SendT", default=None)
ReturnT = TypeVar("ReturnT", default=None)

class Generator(Iterable):
    __class_getitem__ = GenericAlias.with_type_var_likes(YieldT, SendT, ReturnT)

  • ideally, logic to determine if subscription (like Generic[T, DefaultT]) would be valid.

Interaction with PEP 695

If this PEP were to be accepted, amendments to PEP 695 could be made to allow for specifying defaults for type parameters using the new syntax. Specifying a default should be done using the "=" operator inside of the square brackets like so:

class Foo[T = str]: ...

def bar[U = int](): ...

This functionality was included in the initial draft of PEP 695 but was removed due to scope creep.

Grammar Changes

        type_param:
            | a=NAME b=[type_param_bound] d=[type_param_default]
            | a=NAME c=[type_param_constraint] d=[type_param_default]
            | '*' a=NAME d=[type_param_default]
            | '**' a=NAME d=[type_param_default]

        type_param_default: '=' e=expression

This would mean that TypeVarLikes with defaults proceeding those with non-defaults can be checked at compile time. Although this version of the PEP does not define behaviour for TypeVarTuple and ParamSpec defaults, this would mean they can be added easily in the future.

Rejected Alternatives

Specification for TypeVarTuple and ParamSpec

An older version of this PEP included a specification for TypeVarTuple and ParamSpec defaults. However, this has been removed as few practical use cases for the two were found. Maybe this can be revisited.

Allowing the TypeVar defaults to be passed to type.__new__'s **kwargs

T = TypeVar("T")

@dataclass
class Box(Generic[T], T=int):
    value: T | None = None

While this is much easier to read and follows a similar rationale to the TypeVar unary syntax, it would not be backwards compatible as T might already be passed to a metaclass/superclass or support classes that don't subclass Generic at runtime.

Ideally, if PEP 637 wasn't rejected, the following would be acceptable:

T = TypeVar("T")

@dataclass
class Box(Generic[T = int]):
    value: T | None = None

Allowing non-defaults to follow defaults

YieldT = TypeVar("YieldT", default=Any)
SendT = TypeVar("SendT", default=Any)
ReturnT = TypeVar("ReturnT")

class Coroutine(Generic[YieldT, SendT, ReturnT]): ...

Coroutine[int] == Coroutine[Any, Any, int]

Allowing non-defaults to follow defaults would alleviate the issues with returning types like Coroutine from functions where the most used type argument is the last (the return). Allowing non-defaults to follow defaults is too confusing and potentially ambiguous, even if only the above two forms were valid. Changing the argument order now would also break a lot of codebases. This is also solvable in most cases using a TypeAlias.

Coro: TypeAlias = Coroutine[Any, Any, T]
Coro[int] == Coroutine[Any, Any, int]

Thanks to the following people for their feedback on the PEP:

Eric Traut, Jelle Zijlstra, Joshua Butt, Danny Yamamoto, Kaylynn Morgan and Jakub Kuczys

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment