[HN Gopher] Show HN: Python Source Code Refactoring Toolkit via AST
___________________________________________________________________
Show HN: Python Source Code Refactoring Toolkit via AST
Author : treesciencebot
Score : 76 points
Date : 2021-08-01 15:34 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| TekMol wrote:
| The one thing I hope for when it comes to Python is that PyPy
| becomes the default runtime everywhere.
|
| CPython is so unbelievably slow when compared to languages like
| PHP and Javascript.
|
| When using PyPy, I keep encountering road bumps because many
| tools are still expecting that one uses CPython.
|
| Dear world - please accelerate the conversion to PyPy!
| heavyset_go wrote:
| Projects that depend on CPython C extensions should consider
| migrating to HPy[1] for C extension compatibility across Python
| implementations.
|
| [1] https://github.com/hpyproject/hpy
| sitkack wrote:
| This looks interesting, but it also looks ambitious and not
| anywhere ready. Everyone would be well served for extensions
| to not be aware of Python internals and expose a clean handle
| based api to all users. For this, cffi is sufficient and
| already available across nearly all Python implementations.
|
| https://cffi.readthedocs.io/en/latest/
|
| I do agree that a standardized Python should have an object
| model that is a clean public contract, but given that Python
| is CPython and there is no standardization, that the library
| runtime is married to the core also makes this incredibly
| difficult politically.
| treesciencebot wrote:
| Author of this project here (and also a CPython core developer
| FWIW), for many aspects (including the AST) PyPy follows
| CPython closely even on implementation details so `refactor`
| would probably work out of the box once PyPy releases 3.9 (the
| latest stable version for PyPy is 3.7, and 3.8 is on the way!).
| keithasaurus wrote:
| I think it's more likely that people start using mypyc for
| performance-critical code.
| https://mypyc.readthedocs.io/en/latest/
| rattray wrote:
| Wow, this looks exciting. Looks like Cython without any extra
| work/syntax, for those using mypy.
|
| I wish their readme gave some kind of estimate of how the
| speedups might compare to Cython and PyPy. But it's currently
| alpha, so there may be big differences by the time it's ready
| for production use.
|
| Similar approach to the Sorbet Compiler for Ruby (albeit
| targeting C instead of llvm... I wonder how that might impact
| optimizability).
| formerly_proven wrote:
| I assume the performance profile is similar to compiling
| regular Python code with Cython (which it can do). There is a
| decent but not world-changing speed-up; this makes sense,
| because pretty much the only thing you are removing from the
| equation is the instruction and stack overhead of the
| interpreter, you still have to perform all the equivalent
| work that CPython does, otherwise you would end up with
| different semantics. And this work is substantial.
|
| In contrast, a tracing JIT can dynamically elide most of this
| work without changing the semantics.
| xapata wrote:
| After switching to Cython, you can add Cython syntax, like
| type definitions, to gain big speed improvements.
| milkbikis wrote:
| Nice, I've always thought Python could do better by adopting
| source transformation tools like babel. I implemented something
| similar using lib2to3, which preserves whitespace and comments
| more accurately (than the ast module):
| https://github.com/banga/prefactor
| treesciencebot wrote:
| Yes! I love to do transformations with lib2to3, but
| unfortunately it is now deprecated. As I've stated in the
| README, for complex refactorings I also would personally prefer
| working with CST (probably through parso instead of regular
| lib2to3), but for refactoring small fragments I wasn't able to
| observe major differences in formatting in big codebases.
| Thanks for sharing this btw!
| milkbikis wrote:
| Ah I didn't know it was deprecated, but good to see new work
| happening in this area!
| graton wrote:
| A similar type project. Though I haven't seen much activity
| recently:
|
| https://github.com/facebookincubator/Bowler
| treesciencebot wrote:
| I haven't used bowler, but from what I can see it is using
| lib2to3 (through fissix), which can't parse newer Python code
| (parenthesized context managers, new pattern matching statement
| etc.) due to it being using a LL(1) parser. The regular ast on
| the other hand is always compatible with the current grammar,
| since that is what CPython internally consumes. It is also more
| handy to work with since majority of the linters already using
| it, so it would be very easy to port rules.
| jreese wrote:
| (Author of Bowler/maintainer of fissix)
|
| The main concern with using the ast module from stdlib is
| that you lose all formatting/whitespace/comments in your
| syntax tree, so writing any changes back to the original
| sources requires doing a lot more extra work to preserve the
| original formatting, put back comments, etc. This is entire
| point of lib2to3/fissix and LibCST, allowing large scale CST
| manipulation while preserving all of those comments and
| formatting. We do recognize the limitations of
| lib2to3/fissix, though, so there have been some backburner
| plans to move Bowler onto LibCST, as well as building a PEG-
| based parser for LibCST specifically to enable support for
| 3.10 and future syntax/grammar changes. But of course, this
| is very difficult to give any ETA or target for release.
___________________________________________________________________
(page generated 2021-08-01 23:01 UTC)