[HN Gopher] Learning Common Lisp to beat Java and Rust on a phon...
___________________________________________________________________
Learning Common Lisp to beat Java and Rust on a phone encoding
problem
Author : medo-bear
Score : 165 points
Date : 2021-10-01 17:42 UTC (5 hours ago)
(HTM) web link (renato.athaydes.com)
(TXT) w3m dump (renato.athaydes.com)
| tyingq wrote:
| Not the same phone encoding challenge, but an interesting feature
| of the bsd globbing built into bash. The map is the typical map
| of letters on a touch-tone phone..where "2" can be a, b, or c.
|
| All letter combos for the number "6397":
| #!/usr/bin/bash m[0]=0;m[1]=1;m[2]="{a,b,c}";m[3]="{d,e,f}"
| ;m[4]="{g,h,i}";m[5]="{j,k,l}";
| m[6]="{m,n,o}";m[7]="{p,q,r,s}";m[8]="{t,u,v}";m[9]="{w,x,y,z}"
| var=$(echo ${m[6]}${m[3]}${m[9]}${m[7]}) eval echo $var
| eggy wrote:
| Before Python became ML's darling, there was Lush: Lisp Universal
| SHell by none other than Yann LeCun and Leon Bottouk[1], and I
| thought my Lisp knowledge was going to finally pay off ;) I've
| since moved on to J and APL. SBCL is amazing and aside from
| calling C directly, it is still very fast. Python is slow without
| its specialized libraries from the C-world, but SBCL produces
| short, readable code that is pretty efficient. There ECL[2] for
| an embedded Lisp, or uLisp[32] for really small processors like
| the ESP chips.
|
| [1] http://lush.sourceforge.net/ [2] https://common-
| lisp.net/project/ecl/ [3] http://www.ulisp.com/
| pvitz wrote:
| I am using J on and off, but I am not aware of a ML package for
| it. Are you writing all algorithms yourself or could you
| recommend a package?
| ruste wrote:
| I'd also like to know. I'm considering writing some for fun
| in GNU APL right now.
| liveranga wrote:
| There's a handful of resources around that look fun though I
| haven't dug into them yet.
|
| There's this on the J wiki: https://code.jsoftware.com/wiki/U
| ser:Brian_Schott/code/feedf...
|
| And there's this YouTube series for APL: https://youtube.com/
| playlist?list=PLgTqamKi1MS3p-O0QAgjv5vt4...
| kragen wrote:
| This is a pretty introductory CL article, mostly a commentary on
| Norvig's solution to the problem. Still, I learned about the #.
| readmacro from it. The conclusion: "[The Lisp implementation] was
| the fastest implementation for all input sizes except the largest
| one, where it performed just slightly worse than my best Java
| implementation." GH repo at
| https://github.com/renatoathaydes/prechelt-phone-number-enco....
| Sounds like he was mostly measuring the performance of the SBCL
| bignum implementation.
| xxs wrote:
| The tests are just few seconds, it's measuring bootstap,
| compilation time for the most of the part.
| kragen wrote:
| It's true that they're just a few seconds, but if he were
| measuring compilation time then none of the Java tests would
| come in under 10 000 ms, and (although the graph isn't
| usefully labeled) it looks like they're around 100 ms.
| tehjoker wrote:
| I tried experimenting with Common LISP but never understood how
| one was supposed to deploy a finished binary to the end user. It
| seemed like you were supposed to load up an environment? It made
| me leery of continuing.
| vindarel wrote:
| See this recipe: https://lispcookbook.github.io/cl-
| cookbook/scripting.html The core of it is to call "sb-ext:save-
| lisp-and-die" after you loaded the system, and the portable way
| across implementations is to use asdf:make, with 2 lines on
| your system declaration (.asd).
| aidenn0 wrote:
| For most common-lisp implementations you can just save the
| current state as an executable, along with specifying an entry-
| point.
|
| Obviously doing this from your dev environment is a bad idea,
| so most people write a script that loads the system and dumps
| the image. Or you can use one that someone else has already
| written.
|
| Not all lisps work this way (most notably ECL does not), and
| you can use ASDF's program-op to create an executable directly
| from a package definition, which should work on any
| implemntation supported by ASDF.
| tehjoker wrote:
| Thanks, the saving the dev environment part was the part that
| scared me. It certainly helps to have someone tell you you're
| not crazy. Maybe I'll give it another try sometime!
| aidenn0 wrote:
| One of the things I've learned with powerful tools is that,
| by their nature, they empower you to do bad things almost
| as much as they empower you to do good things.
|
| It's even alluded to in the ruby bare-words example in this
| famous talk[1]
|
| 1: https://www.youtube.com/watch?v=3se2-thqf-A
| gibsonf1 wrote:
| We have a makefile and: sudo make it creates an executable.
|
| Its slightly tricky in that the whole make process can only
| happen on a single thread, so you have to turn off all parallel
| threads during make - so we have a key on many functions :make
| that turns off parallel for using make. Ultimatel the make file
| uses (asdf:make :package-name) to compile to an exe. The other
| tricky part is to get a web server in the exe to work, and
| thats handled like this:
|
| (handler-case (bt:join-thread (find-if (lambda (th)
| #+os-unix (search "clack-handler-woo" (bt:thread-name th))
| #+os-windows (search "hunchentoot" (bt:thread-name th))
| ) (bt:all-threads))) ;; Catch a
| user's C-c (#+sbcl sb-sys:interactive-interrupt
| #+ccl ccl:interrupt-signal-condition #+clisp
| system::simple-interrupt-condition #+ecl
| ext:interactive-interrupt #+allegro
| excl:interrupt-signal () (progn
| (format *error-output* "Aborting.~&") (stop)
| (uiop:quit))) (error (c) (progn (format t "Woops,
| an unknown error occured:~&~a~& - restarting service..." c)
| #+os-unix (trivial-shell:shell-command "service trinity
| restart")) ))
| xondono wrote:
| Some weeks ago I tried to start learning Lisp too.
|
| I found it odd, but I've seen weirder. Then I tried the simple
| example of writing a function to compute the nth fibonacci number
| (it's almost the first example). After testing some numbers, I
| thought it seemed slow, so I quickly implemented the same
| function in Rust (with recursion too).
|
| Rust took less time in compiling, and computing fib(n) for n
| 1..50 than Lisp to compute fib(50). Maybe I did something very
| wrong, but for now I'd rather learn Rust better.
| xondono wrote:
| Maybe it came out wrong (English is not my first language).
|
| I did not mean to say that I believe it inferior or anything,
| just that I felt writing performant code in Lisp is non trivial
| to learn, while rust feels pretty natural (to me).
|
| I'll try again at some point, but I don't think it's a wise
| investment of time at this point, I'd rather be "fluent" in
| rust first, then maybe try lisp.
| rscho wrote:
| Maybe not knowing how to use a tool efficiently is a good
| reason to refrain from strong criticism?
| Jtsummers wrote:
| The only way anyone but you can know if you did anything wrong
| is if you include your code, and your common lisp
| implementation. They don't all have the same performance.
| remexre wrote:
| I believe that unless you put the inline pragma on the Lisp
| function, the compiler isn't allowed to optimize the recursion.
|
| In general, Common Lisp doesn't favor recursive functions (as
| opposed to Scheme), and as far as I know, most implementations
| don't put huge amounts of effort into optimizing it.
|
| I'd be interested to see whether the same written using the
| loop (or iterate:iter) macros would still be as slow.
| gibsonf1 wrote:
| If you make it tail recursive, its insanely fast in common
| lisp. (sbcl)
| jonsen wrote:
| How would you code a tail recursive Fibonacci in LISP?
| gibsonf1 wrote:
| I haven't looked at trying and don't really have time to,
| so I don't know, but with lisp, you can make anything
| fast as a general rule, very very fast. (at least with
| compiled sbcl)
| lowbloodsugar wrote:
| If you were a non LISP programmer and thought "Fibonacci
| using recursion", you might think you'd have something
| like fib(a)=fib(a-1)+fib(a-2), and scoff at its possible
| performance, and also wonder how that would be tail call
| optimized. But as a LISP programmer, you'll think of this
| in the way a Java programmer would think of a loop, and
| just treat this as a count over two values n times,
| adding along the way. And this is why I don't like LISP:
| because I have programmed assembly and built up from
| there, and doing loops using tail call recursion seems to
| be a lie. It is a faulty abstraction. It is a loop. You'd
| be better off doing it in a loop, in any sane language,
| and if you do it right in LISP, the code will generate a
| _loop_ in assembly language. Just the fact that this
| thread devolved into _how to specify compiler options_ so
| that it actually generates that loop shows how absurd the
| whole thing is.
| oconnor663 wrote:
| I think the appeal of expressing loops with recursion is
| that it lets you to avoid mutation. But I agree with the
| point that Rust folks frequently make, about how being
| able to mutate things safely is a lot nicer than avoiding
| all mutation.
| emptybits wrote:
| There may be a faster or cleverer way to do it but here's
| a basic tail recursive fibonacci:
| (defun nth-fibonacci (n &optional (a 0) (b 1))
| (if (= n 0) a (nth-
| fibonacci (- n 1) b (+ a b))))
| gibsonf1 wrote:
| WEB> (time (nth-fibonacci 9999)) Evaluation took: 0.005
| seconds of real time 0.004520 seconds of total run time
| (0.000168 user, 0.004352 system) 100.00% CPU 13,120,881
| processor cycles 4,716,256 bytes consed
|
| 207936082371334980721126489886428368250870360940159031196
| 829458665285014234556866489274560343052265155917573432971
| 901580106247942672509731761338101799027380382317897483462
| 355564831914315919245323944200280678103204087244146934628
| 490626683870833080482509206544933408787332263775808474463
| 248737976037347946482581138586315504040810172603812029199
| 438923709428526016473982135544790818235937154295669451493
| 129936648467790904377992847736753792842706601751346648332
| 663776986420121068913557911418727769340808035049567940946
| 482928805660563647181876626689707585373833526774208355741
| 559456585420036347653245410061210124467856891714948032624
| 086026930912116019739382294466360499015319632861596990778
| 804277202892355393296718771829156434190791865251186788568
| 216008975201710704994376570673424008710839088118009762597
| 274318205395542568694608153559184582533982343823604357627
| 598231798961167484242695459246332046141379928508143520187
| 384809235815539889908971514694061316956144977837207434613
| 737562186851068568260906963398154909212537145372418669116
| 042505973537478237332681781821985092402269558264160166900
| 847498160728435824886131848299053831501800478443537515542
| 015738331055219809981238332532612286898240517778465884610
| 797908078283671323847984517940110765690575221586803789615
| 321608583872238829743804839319295412221008003135806885850
| 025988795664632214278204484925650731065958088374016489964
| 235633861097820456341224678729218456064091743606356182168
| 838125623216644428229525375774927153653211342045306867424
| 354545051032697681443701184949063902549349423589040315098
| 773697224370533831653603885951169802459279352259015376349
| 256548723808771830083010745694440024264364147569050945350
| 728047646844921056800247399144905559043913692186963870929
| 181892461571034503870502293006032416114107074539600801709
| 282779518347632167052424858208014238665266338160829214428
| 830954632590804718193292017101478280252213856563402074897
| 963176632788722076077910344317001127535588134788887275038
| 253890668230986833556957181378678829821117107964227067785
| 369131923427333645567279280189539891531060473797412807940
| 91639429908796650294603536651238230626 WEB>
| Jtsummers wrote:
| You may want to put extra spaces in front of those lines,
| and also manually break up that massive number.
| weavie wrote:
| SBCL doesn't optimize tail calls by default.
|
| If I recall you have to set the optimization level prior
| to compiling the function using `(declaim (optimize
| xxx))` - where xxx is something I've forgotten. Perhaps
| someone can come along and point out what xxx should be?
| gibsonf1 wrote:
| Oh thats interesting!, we don't use that but use tail
| recursion extensively
|
| https://0branch.com/notes/tco-cl.html#sec-2-2
| Jtsummers wrote:
| ; disassembly for NTH-FIBONACCI ; Size: 94 bytes.
| Origin: #x2264E531 ; NTH-
| FIBONACCI ; 31: 498B4510 MOV RAX,
| [R13+16] ; thread.binding-stack-pointer
| ; 35: 488945F8 MOV [RBP-8], RAX ; 39:
| 488B55F0 MOV RDX, [RBP-16] ; 3D: 31FF
| XOR EDI, EDI ; 3F: E8AC343BFF CALL
| #x21A019F0 ; GENERIC-= ; 44:
| 750A JNE L0 ; 46: 488B55E8
| MOV RDX, [RBP-24] ; 4A: 488BE5 MOV
| RSP, RBP ; 4D: F8 CLC ;
| 4E: 5D POP RBP ; 4F: C3
| RET ; 50: L0: 488B55F0 MOV RDX, [RBP-16]
| ; 54: BF02000000 MOV EDI, 2 ; 59:
| E8C2323BFF CALL #x21A01820 ;
| GENERIC-- ; 5E: 488BC2 MOV RAX, RDX
| ; 61: 488945D8 MOV [RBP-40], RAX ;
| 65: 488B55E8 MOV RDX, [RBP-24] ; 69:
| 488B7DE0 MOV RDI, [RBP-32] ; 6D:
| E84E323BFF CALL #x21A017C0 ;
| GENERIC-+ ; 72: 488BF2 MOV RSI, RDX
| ; 75: 488B45D8 MOV RAX, [RBP-40] ;
| 79: 488BD0 MOV RDX, RAX ; 7C:
| 488B7DE0 MOV RDI, [RBP-32] ; 80:
| B906000000 MOV ECX, 6 ; 85: FF7508
| PUSH QWORD PTR [RBP+8] ; 88: E99507DBFD
| JMP #x203FED22 ; #<FDEFN NTH-FIBONACCI>
| ; 8D: CC10 INT3 16
| ; Invalid argument count trap
|
| Looks like it's doing tail call optimization to me, this
| is without doing anything special with declaim. Note that
| where it returns to the top is with _JMP_ not _CALL_.
|
| http://www.sbcl.org/manual/index.html#Debug-Tail-
| Recursion
| weavie wrote:
| Ah, so as long as optimize is 2 or less you should get
| tail call recursion. I found when I tried it (several
| years ago) the default was debug optimize.
|
| I wonder if different installs (or perhaps it's Slime)
| sets this value to different levels.
|
| Something to bear in mind anyway..
| gibsonf1 wrote:
| Wow, that worked, thanks!!!
|
| WEB> (declaim (optimize (debug 0) (safety 0) (speed 3)))
|
| NIL
|
| WEB> (defun nth-fibonacci (n &optional (a 0) (b 1))
| (if (= n 0) a
| (nth-fibonacci (- n 1) b (+ a b))))
|
| WARNING: redefining LOBE/SRC/WEB::NTH-FIBONACCI in DEFUN
| NTH-FIBONACCI
|
| WEB> (time (nth-fibonacci 9999))
|
| Evaluation took: 0.002 seconds of real time 0.001887
| seconds of total run time (0.001887 user, 0.000000
| system) 100.00% CPU 5,476,446 processor cycles 4,715,760
| bytes consed
| Jtsummers wrote:
| Yep, that's how you'd do it. So long as your CL
| implementation supports tail call optimization, that will
| be on par with the same algorithm using _loop_ or another
| looping construct.
| aidenn0 wrote:
| That's pretty darn close to the single worst benchmark you
| could use for CL vs. rust.
|
| It's going to make a lot of memory allocations and make a lot
| of uninlinable function calls, both of which are places that I
| would expect rust to have an advantage. That being said, I'd be
| curious to see your rust version, as just the number of bignum
| operations done for fib(50) is going to take a long time.
| fsckboy wrote:
| I don't know for certain at all, but I'd propose that you are
| being downvoted for not going far enough with lisp to grasp the
| point of lisp, which has more to do with recursive data
| structures--and data that is code--than just recursive
| procedure calling.
|
| Factorial and Fibonacci as a pair are good introductions to the
| issue of recursion--and lisp had recursion back in the dark
| ages before other languages did--but they're aren't what lisp
| is about.
|
| so, were some other newbie to read your comment, it might
| really put them off learning lisp for entirely the wrong
| reasons.
|
| cheers :)
| get52 wrote:
| Common Lisp? Yuu mean the language that got horsefucked like 30
| years ago and barely anyone uses? I'll pass lol
| Decabytes wrote:
| Is it weird that I don't like Common Lisp at all but I like
| Scheme a lot? I just never liked Lisp 2s and separate name spaces
| for functions and variables. But really that is the biggest issue
| for me. I'm sure if only Common Lisp existed it wouldn't bother
| me at all.
|
| That being said, I think CL is a fantastic language and there is
| a lot more libraries out there to do useful things than in
| scheme. My C programming is really weak so I find it challenging
| whenever I come across a library in c that isn't already wrapped
| aidenn0 wrote:
| Not at all weird. I'm the opposite; I never liked Lisp 1s, and
| whenever I use one I always end up shadowing function or macro
| bindings by accident.
| sleibrock wrote:
| I'm a bit the same. I've been writing Racket for a number of
| years now and looking back at Lisp I see a lot of ugliness that
| I don't really think I enjoy.
|
| Racket has a nice package manager and module system that kind
| of works for me, and the documentation is honestly some of the
| best I've ever used, if not my favorite. Comparatively, I've
| tried using GNU Guile and found the online documentation to be
| horrendous, and trying to find online documentation for what's
| considered to be the "standard library" in Common Lisp still
| confuses me.
|
| I love seeing people use CL and other Lisp-likes in the wild,
| and Norvig was a big inspiration for me.
| bitwize wrote:
| Not really. I'm a diehard Schemer as well, for the same
| reasons: the Lisp-1 nature helps you treat procedures as true
| first-class values and fits in with Scheme's more functional
| style. Something you can do in CL also, just with a bit more
| ceremony. And CL programmers are more "haha, setf go brrrr"
| anyway.
|
| That said, I'd rather use CL by far than any other language
| _except_ Scheme, and there are cases where CL is the right
| thing and Scheme is not. The most brilliant, beautiful, joy to
| maintain enterprise system I 've ever seen was written in CL.
| orthecreedence wrote:
| I miss my common lisp days, and I think rust being able to export
| C ABIs makes it a really great companion language for CL. I also
| think common lisp (or really, any _fast_ lisp) is a really great
| tool for game development. The fact that you can redefine a
| program as it 's running really helps iterate without having to
| set up your state over and over. Pair that with a fast, low-level
| language that can handle a lot of the lower level stuff (been
| trying to find time to look at Bevy on rust), and you have a
| killer combo.
|
| The main reason I stopped using lisp was because of the
| community. There were some amazing people that helped out with a
| lot of the stuff I was building, but not a critical mass of them
| and things just kind of stagnated. Then it seemed like for every
| project I poured my soul into, someone would write a one-off copy
| of it "just because" instead of contributing. It's definitely a
| culture of lone-wolf programmers.
|
| CL is a decent language with some _really_ good implementations,
| and everywhere I go I miss the macros. I definitely want to pick
| it up again sometime, but probably will just let the async stuff
| I used to work on rest in peace.
| kjgkjhfkjf wrote:
| It's not quicker to write code in Lisp when you don't know Lisp,
| and you have to learn it to make a change to some Lisp code that
| someone randomly added to the codebase.
|
| This is true of all niche languages that are supposedly quicker
| to use than regular mainstream languages.
| [deleted]
| Jtsummers wrote:
| Your first part is tautological but worthless. It's never
| faster to write in a language you don't know versus one you do
| know.
|
| EDIT: To amend the preceding sentence, it's _almost_ never
| faster. There are probably languages which people know that are
| sufficiently in conflict with the problem domain that they
| could be faster in some other new-to-them language versus the
| one they know for specific problems.
| fsckboy wrote:
| APL comes to mind as a language that could be faster to learn
| and use for a particular task, if your particular task
| involves a lot of pure matrix combination and manipulation.
|
| though, the fact that it wants its own keyboard is a bit of a
| hurdle :)
| wedesoft wrote:
| If you're coming from Java, you might want to look at Clojure. It
| has immutable datastructures and Java interop.
| afandian wrote:
| I just used a Clojure library from a Kotlin project via Java
| interop. And it wasn't even weird.
| medo-bear wrote:
| From the author:
|
| > Even though Clojure might be a more obvious choice for
| someone, like me, who is used to working with the JVM, I
| actually wanted to try using something else with a lighter
| runtime and great performance.
| facelessuser wrote:
| Unless you go out of your way in writing non-idiomatic Clojure
| like in Java with parenthesis Clojure, Clojure will the slowest
| of the three by far.
| kubb wrote:
| I found Common Lisp to be surprisingly ahead of its time in many
| regards (debugging, repl, compilation and execution speed,
| metaprogramming), but unfortunately it doesn't have a large
| community, and it's showing its age (no standard package
| management, threading not built into the language). It's also
| dynamically typed which disqualifies it for large collaborative
| projects.
| gibsonf1 wrote:
| It has great package management with
| https://www.quicklisp.org/beta/ and some truly great and high
| quality libraries, especially Fukamachi's suite of web
| libraries and so many others. Woo, for example, is the fastest
| web server. https://github.com/fukamachi/woo (Faster then the
| runner up Go by quite a bit)
|
| For parallel computing, we use: https://lparallel.org/ Its been
| great at handling massive loads accross all processors
| elegantly. And then for locking against overwrites on highly
| parallel database transactions we use mutex locks that are
| built into the http://sbcl.org/ compiler with very handy
| macros.
| Mikeb85 wrote:
| Slightly off-topic but I'm in awe of Fukamachi's repos. That
| one person has built so much incredible stuff is amazing to
| me. Not sure it's enough to get me using CL all the time, but
| it's very impressive.
| gibsonf1 wrote:
| The math library we use is incredibly fast with quaternion
| matrix transformations: https://github.com/cbaggers/rtg-
| math/
|
| The only gaps we've had with our production code and lisp
| is PDF (we use the java pdfbox), translating between RDF
| formats (also a java lib) and encrypting JWP tokens for
| PKCE dPop authentication (also java)
|
| The complete conceputal AI system and space/time causal
| systems digital twin technology is all in common lisp
| (sbcl)
|
| Also fantastic is the sb-profile library in sbcl that lets
| you profile any number of functions and see number of
| iterations and time used as well as consing all ordered by
| slowest cummulative time. That feature has been key on
| finding those functions that are slow and optimizing
| leading to orders of magnitude speed improvements.
| medo-bear wrote:
| are you able to go into detail as to what sort of AI
| technology you are using ? when you mention causal
| systems do you mean causal inference ?
| gibsonf1 wrote:
| We basically build a space/time model of the world, where
| systems are performing functions in events that take
| input states and change them into output states such that
| those events either causally trigger each other or the
| causal link that the input states to an event means that
| the event outputting that states is the cause of the
| current event.
|
| The conceptual AI models operational concepts based on an
| understanding on how human concepts work, inference and
| automatic classification using those concepts, and then
| learning new concepts. The operational side of the
| digital twin uses functional specifications held
| elsewhere, which is also true of the operational concepts
| which use specifications in the form of conceptual
| definitions.
|
| And the technology takes in RDF graph as data for input,
| builds the digital twin model from that data with
| extensive infererence, then expresses itself back out
| with RDF graph data. (Making https://solidproject.org/
| the ideal protocol for us where each pod is a digital-
| twin of something)
| dunefox wrote:
| Do you have links to more information?
| gibsonf1 wrote:
| We have a beta running on that framework:
| https://graphmetrix.com The live app and Pod Server is at
| https://trinpod.us
|
| We are working toward commercial launch in the coming
| weeks. (We are adding Project Pods, Business Pods, Site
| Pods with harvesting the sematic parse we do of PDFs into
| the pod, so we handle very big data)
| formerly_proven wrote:
| > It's also dynamically typed which disqualifies it for large
| collaborative projects.
|
| I've been around the block for long enough to see how far the
| pendulum swings on this one. I'm guessing that it starts going
| the other way soon.
| ballpark wrote:
| I'm not following, could you elaborate?
| varjag wrote:
| Dynamic/static typing came in/out of fashion several times
| already. Any trend is temporary; neither kind of type
| system is a help or impediment in collaboration.
| brundolf wrote:
| I think it originally tanked as a backlash against how
| verbose and painful it was in Java and friends (as well
| as the limited benefits because of things like
| nullability)
|
| Modern static type systems are a totally different beast:
| inference and duck-typing cut down on noise/boilerplate,
| and the number of bugs that can be caught is dramatically
| higher thanks to maybe-types, tagged
| unions/exhaustiveness checking, etc. I think we've
| circled around to a happy best-of-both-worlds and the
| pendulum is settling there.
| blacktriangle wrote:
| Line noise is a red herring in the static/dynamic
| comparison. You will still run into serious problems
| trying to shove ugly human-generated data into your nice
| clean type system.
|
| For mechanical things where you the programmer are
| building the abstractions (compilers, operating systems,
| drivers) this is a non-issue, but for dealing with the
| ugly real world dynamic is still the way to go.
| bitwize wrote:
| Alexis King's "Parse, don't validate" is pretty much the
| final word on using type systems to deal with "messy real
| world data": https://lexi-
| lambda.github.io/blog/2019/11/05/parse-don-t-va...
|
| tl;dr when used properly, static type systems are an
| enormous advantage when dealing with data from the real
| world because you can write a total function that accepts
| unstructured data like a string or byte stream and
| returns either a successful result with a parsed data
| structure, a partial result, or a value that indicates
| failure, without having to do a separate validation step
| at all -- and the type system will check that all your
| intermediate results are correct, type-wise.
| blacktriangle wrote:
| You're not the first, nor fourth for that matter, person
| to respond to dynamic typing advocation with that blog
| post, and it's an interesting post but it misses the
| whole point. The problem is not enforcing rules on data
| coming in and out of the system. The problem is that I
| have perfectly valid collections of data that I want to
| shove through an information processing pipeline while
| preserving much of the original data and static typing
| systems make this very powerful and legitimate task a
| nightmare.
| brundolf wrote:
| They don't, though.
|
| For example: a technique I've used to work with
| arbitrary, unknown JSON values, is to type them as a
| union of primitives + arrays of json values + objects of
| json values. And then I can pick these values apart in a
| way that's totally safe while making no dangerous
| assumptions about their contents.
|
| Of course this opens the door for lots of potential
| mistakes (though runtime errors at least are impossible),
| but it's 100% compatible with any statically-typed
| language that has unions.
| rewma wrote:
| > The problem is that I have perfectly valid collections
| of data (...) and static typing systems make this very
| powerful and legitimate task a nightmare.
|
| What leads you to believe that static typing turns a task
| that essencially boils down to input validation "a
| nightmare"?
|
| From my perspective, with static typing that task is a
| treat and all headaches that come with dynamic typing
| simply vanish.
|
| Take for example Typescript. Between type assertion
| functions, type guards, optional types and union types,
| inferring types from any object is a trivial task with
| clean code enforced by the compiler itself.
| brundolf wrote:
| Presumably the GP's data is external and therefore not
| checkable or inferrable by typescript. This makes the
| task less ideal, but still perfectly doable via
| validation code or highly agnostic typing
| rewma wrote:
| > Presumably the GP's data is external and therefore not
| checkable or inferrable by typescript.
|
| There is no such thing as external data that is not
| checkable or inferable by typescript. That's what type
| assertion functions and type guards are for.
|
| With typescript, you can take in an instance of type any,
| pass it to a type assertion function or a type guard, and
| depending on the outcome either narrow it to a specific
| type or throw an error.
| roca wrote:
| Not a nightmare at all. For example, if you're doing JSON
| processing in Rust, serde_json gives you great tools for
| this. The serde_json::Value type can represent any JSON
| value. You parse JSON to specific typed structures when
| you want to parse-and-validate and detect errors (using
| #[derive(Serialize, Deserialize)] to autogenerate the
| code), but those structures can include islands of
| arbitrary serde_json::Values. You can also parse JSON
| objects into structs that contain a set of typed fields
| with all other "unknown" fields collected into a dynamic
| structure, so those extra fields are reserialized when
| necessary --- see https://serde.rs/attr-flatten.html for
| an example.
| ballpark wrote:
| I've found this to be a non-issue in Clojure with
| specification validation. Some call this gradual typing
| olau wrote:
| People seem to interpret blacktriangle's post in a parser
| setting. I don't know why, but if you're writing parsers,
| you're explicitly falling into the category he's
| mentioning where static types make a lot of sense.
|
| GP's claim was that Java was too verbose. But verbosity
| isn't really the problem. There are tools for dealing
| with it. The problem is a proliferation of concepts.
|
| A lot of business applications goes like this: Take a
| myriad of input through a complicated UI, transform it a
| bit and send it to somewhere else. With very accurate
| typing and a very messy setting (say, there's a basic
| model with a few concepts in it, and then 27 exceptions),
| you may end up modeling snowflakes with your types
| instead of thinking about how to actually solve the
| problem.
| brundolf wrote:
| If you're referring to the "parse, don't validate"
| article, it's using the word in a different sense. The
| idea is that you make a data model that can only
| represent valid values, and then the code that handles
| foreign data transforms it into that type for downstream
| code to handle instead of just returning a boolean that
| says "yep, this data's fine"
| mwcampbell wrote:
| I'm not sure I understand what makes dynamic typing
| better for handling real-world data. Yes, the data is
| messy, but your program still has to be prepared to
| handle data in a specific shape or shapes. If you need to
| handle multiple shapes of data, e.g. varying JSON
| structures, you can still do that with static types,
| using sum types and pattern matching.
| [deleted]
| lamontcg wrote:
| Most modern static languages have some way to hold onto a
| dynamically typed object, stuff them into containers of
| that dynamic type and do some basic type reflection to
| dispatch messy dynamic stuff off into the rest of the
| statically typed program. Sometimes it does feel fairly
| bolted on, but the problem of JSON parsing tends to force
| them all to have some reasonably productive way of
| handling that kind of data.
| brundolf wrote:
| This has not been my experience.
| dtech wrote:
| Can you elaborate on that? As I see it dynamic was
| popular in the 90's (Python, JS, Ruby), but outside of
| that it's always been pretty much dynamic for scripting
| and static for everything else.
| varjag wrote:
| Consider that first Fortran (statically typed) and Lisp
| (dynamic) implementations date back to late 1950s. Since
| then there was a constant tug of war between these camps,
| including BASIC, COBOL, Smalltalk, Pascal, and trends
| falling in and out of favour.
|
| All this however is rather orthogonal to the strengths of
| type systems. CL type system, for instance, is stronger
| than one of Java or C.
| bcrosby95 wrote:
| None of those languages were popular in the 90s.
| [deleted]
| AlchemistCamp wrote:
| > It's also dynamically typed which disqualifies it for large
| collaborative projects.
|
| Like Github or WordPress?
| jcelerier wrote:
| .. are those considered "good" ? github is meh at best
| considering it's 13yo and the billions of dollars poured into
| it, and wordpress, I don't think anyone can reasonably say
| that it's a sane software. They are both good arguments
| against dynamic typing imho (especially the latter).
| medo-bear wrote:
| common lisp supports type annotations. there is even an ML-type
| language impletmented in it (see coalton). quick lisp [1] is
| used for package managment, bordeaux-threads [2] for threading.
|
| 1. https://www.quicklisp.org/index.html
|
| 2. https://github.com/sionescu/bordeaux-threads
| lkey wrote:
| As with all of these forays, I applaud the author for learning
| new things, but these benchmarks are primarily testing reading
| from stdin and format printing to stdout, followed by testing
| bignum impls as mentioned elsewhere. For this reason, the title
| and benchmarks are a bit misleading.
| xxs wrote:
| ...also testing half-compiled code. JIT compiled tests that run
| few seconds just discredit their authors. It's yet another
| example of how not to perform microbenchmarks.
| twicetwice wrote:
| Worth pointing out for those who don't know that the Java
| runtime does JIT optimizations of bytecode based on observed
| runtime patterns. Blew my mind when engineers were talking
| about "warming up the code paths" during startup of a large
| Java service and I found out it wasn't joke/superstition.
| xxs wrote:
| Warming is a tiny part of microbenchmarking actually.
| Knowing how and what the compiler optimizes, like purpose
| lattice checks to ensure no extra array boundaries in the
| code. Call site optimizations (class hierarchy analysis) -
| single site (direct calls, inclines), dual site - guarded
| checks, vs 3-5 - inline caches vs. full virtual calls.
| Card-marking of stores for concurrent GC. L2 and L3 CPU
| cache sizes awareness (certain dataset may fit and the
| results are misleading)... There are a lot other techniques
| to consider, including looking directly at the assembly
| code.
|
| Microbenchmarking requires rather deep knowledge how the
| JIT works, how the hardware - CPU/cache/memory operates,
| the costs of calls certain system calls and what not.
| mhh__ wrote:
| I just tentatively found a 10% speedup in the D compiler
| frontend by enabling profile guided optimization.
| xxs wrote:
| Java effectively always have guided compilation with
| perf. counters and all.
| eigenhombre wrote:
| I write Clojure for food, and Common Lisp for fun. One reason for
| the latter is CL's speed -- awhile back I compared a bit of (non-
| optimized) Clojure code I wrote for a blog post with a rough
| equivalent in CL, and was stunned that the Common Lisp code ran
| about 10x faster. This made me curious as to how fast it could be
| made if I really tried, and was able to get nearly 30x more[1] by
| optimizing it.
|
| Clojure is definitely fast enough for everything I've done
| professionally for six years. But Common Lisp, while having
| plenty of rough edges, intrigues on the basis of performance
| alone. (This is on SBCL -- I have yet to play with a commercial
| implementation.)
|
| [1] http://johnj.com/from-elegance-to-speed.html
| NoahTheDuke wrote:
| Your post is a lot of fun! I have a fondness for these kinds of
| deep dives. That being said, I feel like comparing the initial
| unoptimized Clojure code to highly optimized Common Lisp is
| kind of unfair. I wonder how fast the Clojure code could run if
| given the same care. Maybe I'll give that a try tonight!
| vindarel wrote:
| It would be cool if you updated the link to the Cookbook in
| your article from the old one (on Sourceforge) to the newer one
| (on Github): https://lispcookbook.github.io/cl-cookbook/ Best,
| lmilcin wrote:
| I have implemented real algotrading framework in Common Lisp
| (SBCL) that connected _directly_ to Warsaw Stock Exchange.
|
| The application was receiving binary messages from the exchange
| over multicast, rebuilding state of the market, running various
| (simple) algorithms and responding with orders within _5
| microseconds_ of the original message, at up to 10k messages
| per second.
|
| With SBCL you can write a DSL and have ability to fully control
| the resulting instructions (through vops). It is just the
| question how dedicated you are to writing macros upon macros
| upon macros.
|
| I used this to the fullest extent and the result was as good as
| any hand optimized C/assembly.
|
| For example, the binary message parser would receive a stack of
| complicated specifications for the messages in the form of XML
| files (see here if you are curious: https://www.gpw.pl/pub/GPW/
| files/WSE_CDE_XDP_Message_Specifi...), converted XML to DSL and
| then, through magic of macros, the DSL was converted to
| accessors that allowed optimal access to the fields. Optimal
| here mans the assembly could not be improved upon any further.
|
| Large parts of the application (especially any communication
| and device control) was written in ANSI C but the Foreign
| Function Interface makes integrating it into the rest of
| application a breeze.
|
| I write all of this, because I frequently meet complete
| disbelief from people that Lisp can be used in production.
|
| I personally think it is exactly the opposite. Lisps offer
| fantastic development environment. The problem are developers
| who are mostly unable to use Lisp for work effectively, I guess
| due to too much power, freedom and complete lack of direction
| on how to structure your application.
___________________________________________________________________
(page generated 2021-10-01 23:00 UTC)