[HN Gopher] Python Type Hints - *args and **kwargs (2021) ___________________________________________________________________ Python Type Hints - *args and **kwargs (2021) Author : ekiauhce Score : 189 points Date : 2023-08-27 13:11 UTC (9 hours ago) (HTM) web link (adamj.eu) (TXT) w3m dump (adamj.eu) | curiousgal wrote: | Is it just me or are Python type hints like..goofy? | PheonixPharts wrote: | As someone who has written Python for nearly 20 years now, and | also has plenty of experience with strongly and statically | typed languages (including a fair bit of Haskell), I think type | hints in Python should at most remain just that, _hints_. | | A language being statically typed or dynamically typed is a | _design_ decision with implications for what the language can | do. There are benefits to each method of programming. | | Trying to strap type checking on to Python is born out of some | misplaced belief that static type is just better. Using Python | as a dynamically typed language allows you for certain | programming patterns that cannot be done in a statically typed | language. There are some great examples in SICP of Scheme | programs that could not exist (at least with as much elegance) | in a typed language. Dynamic typing allows a language to do | things that you can't do (as easily/elegantly) in a statically | typed language. | | Some may argue that these type of programming patterns are | _bad_ for production systems. For most of these arguments I | strongly _agree_. But that means for those systems Python is | probably a poor choice. I also think metaprogramming is very | powerful, but also a real potential footgun. It would be | ridiculous to attempt to strip metaprogramming from Ruby to | make it "better", just use a language that depends less on | metaprogramming if you don't like it. | | This is extra frustrating because in the last decade we've seen | the options for well designed, statically typed languages | explode. It's no longer Python vs Java/C++. TypeScript and Go | exist, have great support and are well suited for most of the | domains that Python is. Want types? Use those languages. | mirsadm wrote: | Have to disagree with this. Choosing a language is not just | about its features but also its ecosystem. I chose Python for | my current project because it has great libraries that don't | exist in other languages. | PheonixPharts wrote: | "my current project" type problems are where Python is | great. Types remain "nice to have" (if you like them) and | aren't really essential compared to the ease of prototyping | new ideas and building a PoC. You're choosing Python | because the benefit of libraries outweighs your personal | preference for types. | | Most of my work is in machine learning/numeric computing, | so I'm very familiar with the benefits of Python's | ecosystem. Basically all of AI/ML work is about prototyping | ideas rapidly, where access to libraries and iterating fast | greatly trumps the need for type safety. | | At nearly every place I've worked, Python is the tool for | building models quickly but shipping them to production and | integrating them with the core product almost always | involves another language, typically with types, better | suited for large engineering teams working on a large code | base where you _really_ want some sort of type checking in | place. Most of the companies I know that do serious ML in | production typically take models from python and then | implement them in either C++ or Scala for the actual | production serving. | | It's worth pointing out that the vast majority of those | libraries you use were initially developed without any | consideration, or need, for types. Great, reliable, | software can be written without types. Dynamic typing is a | great language choice, and there's no need to fight the | language itself by trying to bolt types on. | | Where types are important is where you have a complex, | rapidly changing code base with a large number of | developers of differing skill levels releasing frequently. | If that's the environment you're in, I would _strongly_ | recommend against using Python in prod, even if it means | you have to implement the features of some libraries | internally. | insanitybit wrote: | They're quite limited in some ways, obscenely powerful in | others, and have a fairly strange syntax, yeah. | jcalvinowens wrote: | I agree. It adds all the inconvenience of static typing with | none of the benefits. | lijok wrote: | Big time. Getting better very quickly however | b5n wrote: | Call me crazy, but I just use a statically typed language where | static types are required. | m3047 wrote: | That article promulgates a misunderstanding about immutability. | For my way of thinking, python is already an interpreted language | and I can enforce tropes in code more cleanly and effectively | than people taking something five levels up at face value and | trying to figure out what sticks when they throw it against the | wall: no wonder they end up frustrated, and it's a frustrating | situation. | | Given: def foo(*args): print(args) | return class Thing(object): def | __init__(self,a,b): self.a = a | self.b = b return def | foo_style(self): return (self.a, self.b) | | _args is not required to refer to a tuple: >>> | foo(*[31,42]) (31, 42) | | I can have objects construct parameters conforming to the | specifications for a signature: >>> | foo(*Thing(3,91).foo_style()) (3, 91) | | Consider that a counterexample._ | adamchainz wrote: | Within the function, args is a tuple, as your output | demonstrates. | [deleted] | dfee wrote: | I actually created a library for this! | | Forge: forge (python signatures) for fun and profit | | https://python-forge.readthedocs.io/ | | https://github.com/dfee/forge | assbuttbuttass wrote: | This does restrict all of your keyword arguments to the same | type. If you have keyword arguments of different types, you're | right back to no type safety. | actualwitch wrote: | Well, if you want to type your kwargs and use newer versions of | python, you can use Unpack with typed dicts to achieve that. | But the footgun there is that you can't redefine fields when | extending them, so no Partial<SomeType> for you. | zbentley wrote: | True, but there are a couple of mitigations available: you can | express the types of selected kwargs (by leaving them out of | the * residual), and you can use typing.Union/| to express | product types for values in the residual as well. | masklinn wrote: | That seems obvious? If you want a variable number of arguments | of arbitrary type you have to specify the common supertype, | commonly top itself. | | To do otherwise would require some form of vararg generics | which is uncommon. | IshKebab wrote: | It's extremely common for Python programmers to write code | with kwargs of different types. Look at subprocess.run() for | example. | PartiallyTyped wrote: | Alternatively, use an `@overload` in a `.pyi` file and specify | your types there. | | This means that you will have 2^N combinations and doubling every | time you accept a new argument. | | If that is not good enough, then simply use a `TypedDict` with | everything optional instead of `**kwargs`. Your call will then | become `foo(SomeTypedDict(p1=p2,...))`. | amelius wrote: | What if the second argument is a float? | vorticalbox wrote: | Why do people not just type everything they want passed? | | def variable(n:str, nn:str, nnn:str, *, a:int, b:int, c:int) | | Anything after,*, is a kwarg. | Frotag wrote: | It's pretty common when wrapping a function that has a large | number of config options. | | The wrapper is usually some shorthand for building a handful of | those args or adding some side-effect, while still allowing the | caller access to the remaining config options via kwargs. | | Here's one example of that in the wild | https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.... | IshKebab wrote: | In my experience it's generally because Python developers make | functions with an insane number of keyword arguments, and then | wrap those functions. They don't want to type them all out | again so they use kwargs. | | subprocess.run() is an example of that. Also check out the | functions in manim. | | The inability to properly static type kwargs using TypedDict is | probably the biggest flaw in Python's type hint system (after | the fact that hardly anyone uses it of course). | jpc0 wrote: | If you have enough arguments that the signature becomes obscure | to read you need a dataclass to pass into the function instead. | | I would rather: @dataclass(frozen=True, | slots=True) class VarThings: n: int | ... def variable(a: VarThings): ... | | Than a million args | b5n wrote: | I usually start with a namedtuple unless I need the | additional features provided by a dataclass. | joshuamorton wrote: | Why? Dataclasses are vastly better: more typesafe, less | hacky, etc. | masklinn wrote: | Your signature requires exactly 3 positional[0] and 3 keyword | arguments. The OP allows any number of either. | | [0] actually 3 positional-or-keyword which is even more widely | divergent | vorticalbox wrote: | But why would you want that doesn't that make for a more | confusing api? Would it not be better to just have everything | as a kwarg? You would get better types that way | masklinn wrote: | I genuinely don't understand what you are asking. | zbentley wrote: | I think what GP is saying is that with explicit kwargs you | can't express _variadic_ signatures, i.e. "this function | takes one int positional, and then _any number of_ key | /value pairs where the values are lists". The variable | length is the important bit. | | It's certainly debatable whether doing that is better than | passing a single argument whose value is a dict with that | same type, but many people do prefer the variadic | args/kwargs style. | Znafon wrote: | It is used when the number of argument can vary, like: | def sum(*args: int) -> int: if len(args) == 0: | return 0 return args[0] + sum(*args[1:]) | zoomablemind wrote: | It seems altogether surprising that with an empty list or | tuple a, a[1] results in index error, yet a[1:] quietly | returns an empty list or tuple. | hk__2 wrote: | > It seems altogether surprising that with an empty list or | tuple a, a[1] results in index error, yet a[1:] quietly | returns an empty list or tuple. | | `a[1:]` returns the sequence of elements that start at | index 1. If there is no such element, the list is empty. I | don't see any good reason why this should throw an error. | macintux wrote: | Then why doesn't a[1] return None? | | I understand the logic behind both decisions, but it's | not surprising that people find it inconsistent and | unintuitive. | hk__2 wrote: | > Then why doesn't a[1] return None? | | Because there would be no way to distinguish between | "a[1] contains None" and "a[1]" doesn't exist. | macintux wrote: | And with a[1:] returning the empty list there's no way to | distinguish between a is empty and a only has one | element. | | These are, in the end, relatively arbitrary language | design decisions. | akasakahakada wrote: | When you slice a list, you get a list. When you see there | is nothing inside the returning list, you know that means | end of list, contains zero element. Slicing and indexing | return object at different level. | macintux wrote: | Slicing a list, when the first index is invalid for that | list, could easily throw an exception instead. | zoomablemind wrote: | This should signal an explicit error, which invalid index | is indeed. If user believes for some reason the invalid | indexing is ok, then it could be caught and handled. No | ambiguity. | Znafon wrote: | I think it is consistent, it works a bit like filtering | an element from a mathematical set. | | Given a set of sheeps, let x be the five-legged sheep is | inconsistent because we know neither the existence or | uniqueness of shuch sheep, so it raises an exception. | | Given a set of sheeps, let x be the subset of five legged | sheeps is the empty set because there is no such sheep. | | but this may also just be because I internalised Python's | behavior. | | Some language have a specific value to denote the first | thing, for example: ["a", "b", "c"][4] | | gives `undefined` in JavaScript but it differs from | `null` which would be the equivalent to `None` in Python | (and I don't think Python has such concept). | zoomablemind wrote: | Both cases are an index error. It's just for some other | reasons in case of the section, the error is represented | by an empty object and it's left to user to handle the | result. | | This could easily conceal the indexing error unless the | caller code explicitly checks the length of the returned | section. | js2 wrote: | a[1] has to raise an IndexError because there's no return | value it could use to otherwise communicate the item | doesn't exist. Any such value could itself be a member of | the sequence. To behave otherwise, Python would have to | define a sentinel value that isn't allowed to be a member | of a sequence. | | When using slice notation, the return value is a sequence, | so returning a zero-length sequence is sufficient to | communicate you asked for more items than exist. | | It may be surprising, but it almost always leads to more | ergonomic code. | | https://discuss.python.org/t/why-isnt-slicing-out-of-range/ | esafak wrote: | You should use `Iterable` | Znafon wrote: | I'm not sure print(firstname, lastname) | | for example is more readable than | print((firstname, lastname)) | | especially since I would then have to write | print((surname,)) | | to just print a single string. | | Variadic functions are rather classic, I think Go, Rust, C | and JavaScript also have them. | uxp8u61q wrote: | How is it more "readable"? The two are just as readable. | | What do you do with your first example if you have a list | (generated at runtime, not a static one) to pass to the | function? This wouldn't work (imagine the first line is | more complicated): l = (1,2,3) | print(l) | insanitybit wrote: | FWIW Rust does not have variadic functions. The closest | thing would be either macros, which are variadic, or | trait methods, which are not variadic but can look like | they are. | Znafon wrote: | Oh yeah, that's right! Thanks for the correction | esafak wrote: | Your example has a fixed number of names. What if you | wanted to accept any number of names, like _Pablo Diego | Jose Francisco de Paula Juan Nepomuceno Maria de los | Remedios Cipriano de la Santisima Trinidad Ruiz y | Picasso_? Really, though, Iterables make more sense for | monadic types. | sdenton4 wrote: | We would force broad changes in human society to conform | to the assumptions of our database scheme, same as we | always have. | dragonwriter wrote: | > Anything after,*, is a kwarg. | | A required positional OR kwarg as you've done it. Its closer to | an optional kwarg if you expand the type declaration to also | allow None and set a None default. | | But there are times when you want to leave the number and names | of kwargs open (one example is for a dynamic wrapper--a | function that wraps another function that can be different | across invocations.) | [deleted] | hk__2 wrote: | > In the function body, args will be a tuple, and kwargs a dict | with string keys. | | This always bugs me: why is `args` immutable (tuple) but `kwargs` | mutable (dict)? In my experience it's much more common to have to | extend or modify `kwargs` rather than `args`, but I would find | more natural having an immutable dict for `kwargs`. | adamchainz wrote: | Yeah, that is odd. Python still has no immutable dict type, | except it kinda does: https://adamj.eu/tech/2022/01/05/how-to- | make-immutable-dict-... | dragonwriter wrote: | > This always bugs me: why is `args` immutable (tuple) but | `kwargs` mutable (dict)? | | Because python didn't (still doesn't, but at this point even if | it did backward compatibility would mean it wouldn't be used | for this purpose) have a basic immutable mapping type to use. | | (Note, yes, MappingProxyType exists, but that's a proxy without | mutation operations, not a basic type, so it costs a level of | indirection.) | blibble wrote: | now try typing a decorator | | https://stackoverflow.com/questions/47060133/python-3-type-h... | | what a disaster | amethyst wrote: | PEP 612 made this much better FWIW. | | https://peps.python.org/pep-0612/ | akasakahakada wrote: | Although these two comes in handly, people have been using them | wrong. Often in scientific open source package, they slap *kwargs | in function definition without documentation. How am I suppose to | know what to pass in? | | https://qiskit.org/ecosystem/aer/stubs/qiskit_aer.primitives... | tomn wrote: | OT, but this is my number one peeve with code documentation: | going to the effort to write a doc comment, taking up at least | 6 lines, probably using some special syntax, cluttering up the | code, but then adding no information that can't be derived from | the signature. | | If you're not going to document something (which I totally | respect), at least don't make the code worse while doing it. | [deleted] | toxik wrote: | Sadly a problem with any wrapper function is that it nullifies | this kind of information. Use functools.wraps. | akasakahakada wrote: | My question is that can @warps warp more than 1 function? | | Maybe in some use case people need to merge 2 functions into | 1, I don't know if it can handle this situation. | zbentley wrote: | I'm not sure what it means to "merge two functions into | one", can you elaborate? | | If you are referring to a type signature for a function | that passes through it's arguments to one of two inner | functions, each of which has different signatures, such | that the outer signature accepts the union of the two inner | signatures, well ... you _could_ achieve that with | ParamSpecs or similar, but it would be pretty hard to read | and indirected. Better, I 'd say, to manually express | appropriate typing.Union (|) annotations on the outer | function, even if that is a little less DRY. | cbarrick wrote: | > I'm not sure what it means to "merge two functions into | one", can you elaborate? | | I'm not OP, but I see this pattern often enough: | def foo(**kwargs): pass der | bar(**kwargs): pass def | wrapper(**kwargs): foo(**kwargs) | bar(**kwargs) | akasakahakada wrote: | Yup, this exactly. | franga2000 wrote: | PyCharm usually figured this out if it's not too complex. I | often wrap session.request() with some defaults/overrides and | autocomplete usually shows me the base arguments as well. | Syntaf wrote: | Especially when they don't even leave a doc string so you're | forced to track down the packages documentation online just to | interact with certain interfaces. | | I work in a large python codebase, we have almost no usage of | `*kwargs` beyond proxy methods because of the nature of how | they obfuscate the real interface for other developers. | nerdponx wrote: | The worst is when someone puts **kwargs at the _base_ of a | class hierarchy, not only necessitating its use in subclasses | (if you want to be strict about types) but also swallowing | errors for no good reason. Fortunately I think this style is | fading out as type hints become more popular. | hqudsi wrote: | When I was first starting out, a then senior engineer told | me: "friends don't let other friends use kwargs". | | That always stuck with me. | icedchai wrote: | I once worked on a code base where we had *kwargs passed | down 4 or 5 layers deep (not my idea.) It was a true joy. | akasakahakada wrote: | This is literally me. It is a math program that can | evaluate equations and generate code. 6 layers of | heterogeneous data structure which the math operation | being act on 1st layer has its effect down to 6th layer. | Temporarily using *kwargs to make it works but still | thinking what is the proper way to do it right. | crazydoggers wrote: | Can you organize the data structures into classes or | dataclasses? | akasakahakada wrote: | Already doing this. The problem is there are 5 layers in | between. Copy and paste the same docstring into all | layers is doable but do not seem smart. | c32c33429009ed6 wrote: | Out of interest, what sort of company/industry do you | work in where you're able to work on this kind of thing? | dr_kiszonka wrote: | I have been annoyed by this too! I like how seaborn handles it | now in documentation: | https://seaborn.pydata.org/generated/seaborn.barplot.html?hi... | itissid wrote: | #TIL. Also cool to know is pydantic's @validate decorator: | https://docs.pydantic.dev/latest/usage/validation_decorator/... | and in case you were thinking its not superflous to mypy(https:// | docs.pydantic.dev/latest/usage/validation_decorator/...). | SushiHippie wrote: | For typing **kwargs there are TypedDicts | https://peps.python.org/pep-0692/ | | If your function just wraps another you can use the same type | hints as the other function with functools.wraps | https://docs.python.org/3/library/functools.html#functools.w... | awinter-py wrote: | I think pep 612 is trying to make the ergonomics better for the | 'forwarding' / pass-through case (when .wraps isn't | appropriate) | | https://peps.python.org/pep-0612/ | zbentley wrote: | While functools.wraps does propagate __annotations__ by | default, be aware that not all IDE-integrated type checkers | handle that properly. It's easy in PyCharm, for example, to use | functools.wraps such that the wrapper function is treated by | the IDE as untyped. | | Underneath, this is because many (most?) type checkers for | Python aren't actually running the code in order to access | annotation information, and are instead parsing it "from the | outside" using complex and fallible techniques of variable | reliability. That said, it's a testament to JetBrains' | excellent work that PyCharm's checker works as well as it does, | given how crazily metaprogrammed even simple Python often turns | out to be. | veber-alex wrote: | Pycharm has the worst type checker that exists today. It may | have been the best a few years back but others have | suppressed it considerably. | | I recently switched from Pycharm to vscode which uses pyright | and it's night and day on the amount of type errors it | catches, it considerably improved the quality of my code and | confidence during refactoring. | | And to add insult to injury Pycharm doesn't even have a | pyright plugin and the mypy plugin is extremely slow and | buggy. | dalf wrote: | There is also typing.ParamSpec when the purpose is to write a | generic wrapper: | | https://docs.python.org/3/library/typing.html#typing.ParamSp... | ehsankia wrote: | Interesting, looks like they ended up having to introduce | typing.Unpack, to differentiate the ambiguity with the the | TypedDict referring to the type of all the kwargs, vs just | Mapping[str, TypedDict] | | Not ideal but not too bad either. | refactor_master wrote: | The ability of **kwargs to leave behind no proper documentation | and silently swallow any invalid arguments has made us remove | them entirely from our codebase. They're almost entirely | redundant when you have dataclasses. | liquidpele wrote: | Yea, really only useful imho for proxy functions that then just | pass the arguments along to something that DOES properly type | every arg. | qwertox wrote: | But doesn't this break type checking for the users of the | proxy functions? | flakes wrote: | You can write the proxy/decorator to preserve typing info | using a typevar. F = TypeVar("F", | bound=Callable) def wrapper(f: F) -> F: ... | zbentley wrote: | What about decorators, or wrappers around third-party code | whose contracts change frequently (or even second party code | when interacting with functions provided by teams that don't | follow explicit argument typing guidelines, if you have that | sort of culture)? | refactor_master wrote: | Usually the solutions range from a culture of "just don't" to | tests/mypy that have become increasingly stricter over the | years, every time we've come a step further up the ladder. | But I admit, it has taken quite some bridging to get there. | | Moving to static Python in most places has dramatically | improved the code and language. | voz_ wrote: | As someone that works on a Python compiler, this is a very | limited view of reality... | plonk wrote: | Those are better handled by typing.ParamSpec, it should keep | track of the unwrapped function's arguments. | hooloovoo_zoo wrote: | Seems pretty important for something like a plotting function | where you want to be able to pass any tweaks to any subplots. | jerpint wrote: | What do you do when inheriting from a base class with a defined | __init__ ? | yayachiken wrote: | For everybody reading this and scratching their head why this | is relevant: Python subclassing is strange. | | Essentially super().__init__() will resolve to a statically | unknowable class at run-time because super() refers to the | next class in the MRO. Knowing what class you will call is | essentially unknowable as soon as you accept that either your | provider class hierarchy may change or you have consumers you | do not control. And probably even worse, you aren't even | guaranteed that the class calling your constructor will be | one of your subclasses. | | Which is why for example super().__init__() is pretty much | mandatory to have as soon as you expect that your class will | be inherited from. That applies even if your class inherits | only from object, which has an __init__() that is guaranteed | to be a nop. Because you may not even be calling | object.__init__() but rather some sibling. | | So the easiest way to solve this is: Declare everything you | need as keyword argument, but then only give **kwargs in your | function signature to allow your __init__() to handle any set | of arguments your children or siblings may throw at you. Then | remove all of "your" arguments via kwargs.pop('argname') | before calling super().__init__() in case your parent or | uncle does not use this kwargs trick and would complain about | unknown arguments. Only then pass on the cleaned kwargs to | your MRO foster parent. | | So while using **kwargs seems kind of lazy, there is good | arguments, why you cannot completely avoid it in all | codebases without major rework to pre-existing class | hierarchies. | | For the obvious question "Why on earth?" These semantics | allow us to resolve diamond dependencies without forcing the | user to use interfaces or traits or throwing runtime errors | as soon as something does not resolve cleanly (which would | all not fit well into the Python typing philosophy.) | bowsamic wrote: | This is why I hate Python, absolutely none of this is | obvious from the design of the language | sbrother wrote: | Thank you for explaining this; there are a lot of comments | here suggesting trivial code style improvements for use | cases where *kwargs wasn't actually needed. The more | interesting question is how to improve the use case you | describe -- which is how I've usually seen *kwargs used. | Izkata wrote: | > So the easiest way to solve this is: Declare everything | you need as keyword argument, but then only give *kwargs in | your function signature to allow your __init__() to handle | any set of arguments your children or siblings may throw at | you. Then remove all of "your" arguments via | kwargs.pop('argname') before calling super().__init__() in | case your parent or uncle does not use this kwargs trick | and would complain about unknown arguments. Only then pass | on the cleaned kwargs to your MRO foster parent. | | The easiest way is to not put "your" arguments into kwargs | in the first place. If you put them as regular function | arguments (probably give them a default value so they look | like they're related to kwargs), then the python runtime | separates them from the rest when it generates kwargs and | you don't have to do the ".pop()" part at all. | dontlaugh wrote: | Having used Python a lot, I was never glad for multiple | inheritance. I'd prefer traits. | patrickkidger wrote: | FWIW, I've come to regard this (cooperative multiple | inheritance) as a failed experiment. It's just been too | confusing, and hasn't seen adoption. | | Instead, I've come to prefer a style I took from Julia: | every class is either (a) abstract, or (b) concrete and | final. | | Abstract classes exist to declare interfaces. | | __init__ methods only exist on concrete classes. After that | it should be thought of as unsubclassable, and concerns | about inheritance and diamond dependencies etc just don't | exist. | | (If you do need to extend some functionality: prefer | composition over inheritance.) | [deleted] | roland35 wrote: | I agree - it is convenient to use at first but it sure makes it | hard to use an unfamiliar codebase! | codexb wrote: | They are a necessity for subclasses though, especially when | subclassing from an external library that will likely change | underneath you. ___________________________________________________________________ (page generated 2023-08-27 23:00 UTC)