[HN Gopher] A Faster Lexer in Go ___________________________________________________________________ A Faster Lexer in Go Author : sophiewang Score : 48 points Date : 2022-05-05 06:13 UTC (1 days ago) (HTM) web link (eli.thegreenplace.net) (TXT) w3m dump (eli.thegreenplace.net) | mhh__ wrote: | Free speedups are always appreciated but when benchmarking some | not-optimized-at-all lexers, they were still so fast that I | couldn't really justify optimizing them. | | Unless you're parsing genuinely enormous amounts of code (or | latency bound) I find that most compilers are far too slow in | other areas to justify optimizing the lexer versus fixing | architectural issues in (say) semantic analysis. | | A lot of parts of compilers are non-local, and involve lots of | pointer chasing, which means avoiding repeated work is very | important. In the D compiler there a bunch of checks where we | stop you shooting yourself in the foot, I think generating all of | these tree-walkers from a specification (so you can walk it all | in one go) would probably be a decent speedup. | woodruffw wrote: | In this case, the speedup would probably be during LLVM's build | itself: that's when TableGen inputs are parsed as part of | codegen for big blobs of C++. Improvements to lexing speed here | _could_ actually produce some build time improvements, since | TableGen 's outputs tend to have a high dependent degree in the | dependency graph. | | (That being said, I have no idea if LLVM would merge a TableGen | lexer written in Go.) | mhh__ wrote: | The question prior to any of that is how many lines of | tablegen LLVM has | | According to a cloc definition I just did: | LLVM Table Gen files: 1041 blank: | 75684 comment: 69272 code: | 480047 ___________________________________________________________________ (page generated 2022-05-06 23:00 UTC)