[HN Gopher] Julia 1.6: what has changed since Julia 1.0?
___________________________________________________________________
Julia 1.6: what has changed since Julia 1.0?
Author : Sukera
Score : 202 points
Date : 2021-02-14 14:46 UTC (8 hours ago)
(HTM) web link (www.oxinabox.net)
(TXT) w3m dump (www.oxinabox.net)
| tomrod wrote:
| I come back every so often to check out Julia again. I have hopes
| for it.
|
| Some questions still in mind since I reviewed previously
|
| (1) How is its database connectivity?
|
| (2) Is there something like python's `requests` lib?
|
| (3) Are the features mature enough that I don't anticipate major
| rewrites for code each year?
| leephillips wrote:
| I don't know about (1) and (2), but since v. 1.0 there has been
| practically no need to rewrite. I believe there is a commitment
| to not introducing breaking changes, or at least a strong
| reluctance to do so.
|
| EDIT after actually looking at the article: the "no breaking
| changes" commitment is right at the top.
| oxinabox wrote:
| > (1) How is its database connectivity?
|
| Databases are not really my thing but: From what I hear: It is
| ok. Not amazing. But decent.
|
| LibPQ.jl is very mature (we run it in production). MySQL.jl
| exists, I hear about SQLite.jl being used pretty often.
|
| I know people use ODBC.jl, and JDBC.jl, though only because
| they complaint about things. I suspect there are a fair few
| people using them without complain that i never hear from.
| Though I haven't heard any mention really of JDBC in a while.
|
| While there is nothing like SQLAlchemy, a nice thing about
| DataBases in julia is they all conform to Tables.jl tables. So
| very easy to take your DataFrame library of choice (or CSV
| reader, or Arrow.jl or a dozen other formats), and use that is
| the input or output from a database query.
| enriquto wrote:
| > (3) Are the features mature enough that I don't anticipate
| major rewrites for code each year?
|
| There is a very good heuristic for that. Write a non-trivial
| program using the last version of the language and according to
| current conventions. Then look how far can you go into past
| versions of the language so that your program runs correctly.
|
| If the oldest version of the language that runs your program is
| X years old, then you can expect your program to stop running
| after X years in the future.
| DNF2 wrote:
| No, this is incorrect, you are confusing backwards and
| forwards compatibility. Running your new code on old Julia
| versions could break _immediately_ , just like in every
| programming language.
|
| Backwards and forwards compatibility have very different
| horizons.
|
| Running old code on new Julia versions should not break until
| the next major version. Packages, on the other hand, are
| different, and could break your code sooner, but that's the
| same in any language.
| eigenspace wrote:
| Yeah, I just feel compelled to echo you.
|
| What that person said is very wrong.
|
| I guess that in a language without a good version bounding
| and manifest system like julia, it could be a semi-valid
| point because you might end up updating your packages and
| breaking your code that way, but julia has reproducible
| package environments, so you can get very strong guarantees
| about backwards compatibility even when you're updating
| packages.
| enriquto wrote:
| Dudes, relax, it's just a _heuristic_ , a rough zeroth-
| order estimate to measure the pace of evolution of a
| language.
| eigenspace wrote:
| Even to zeroth order, this is a bad way to estimate
| backwards compatability. It's a fine way to estimate pace
| of language evolution, but that wasn't your claim.
|
| For instance, Fortran 2018 has new features that would
| fail if you tried to use them in Fortran 2015. However,
| Fortran 2018 is still backwards compatible with Fortran
| 2015, and indeed is backwards compatible with Fortran
| 1977.
|
| That is, in this example Fortran maintains 42 years of
| backwards compatability, yet only 3 years of forwards
| compatability.
|
| The two things are effectively decoupled from eachother.
| swagonomixxx wrote:
| For (2), it seems like HTTP.jl [0] is the equivalent.
|
| [0]: https://github.com/JuliaWeb/HTTP.jl
| pkphilip wrote:
| There is a pretty decent framework (Genie) which can be used
| for building web apps, API backends with database connectivity
| etc. It is also quite well documented.
|
| https://genieframework.github.io/Genie.jl/dev/guides/Working...
| oxinabox wrote:
| > (2) Is there something like python's `requests` lib?
|
| For HTTP (etc) requests?
|
| There is HTTP.jl which I have never had problems with; tons of
| packages use it. I have used it to wrap a ton of different REST
| APIs etc.
|
| And in 1.6, as mentioned, there is the new Downloads standard
| library, based on libcurl. Despide the name, I believe it can
| be used more generally than simply downloading things. Can also
| be used as a normal library in Julia 1.3+
| https://github.com/JuliaLang/Downloads.jl
|
| And there are several other projects
| Sukera wrote:
| About (2), I've extensively used HTTP.jl and am pretty happy
| with it. I don't know about exact differences to e.g. requests,
| but I've found it sufficient for my uses.
|
| Regarding (3), there's a daily CI job called "PkgEval" (which
| also runs before a new release is made) checking for
| regressions of julia vs. all registered packages, seeing if any
| break. This identifies misuse of internal APIs (which are
| allowed to break under semver) and actually breaking changes
| (which are then either undone or the packages are fixed).
| Additionally, you can [compat] bound julia itself in the
| Project.toml of your code.
|
| Those two combined should mak sure you don't have to rewrite
| your code. All 1.x versions are backwards compatible after all.
| Nosferican wrote:
| (1) I have used Julia packages for connecting to PostgreSQL,
| mongoDB, and SQLite. It has been extremely solid. I still wish
| for the GIS components to be more feature extensive (e.g.,
| writing multilayers features directly).
|
| (2) The HTTP package (HTTP.jl) is great. You also have you HTML
| and CSS selectors (Gumbo/Cascadia). Some of the the best JSON
| parsers across languages too. I also developed WebDriver.jl
| (you can use it with Selenium). Diana.jl is a solid GraphQL
| client/server package. Genie.jl is a comprehensive web
| framework.
|
| (3) Those packages have been stable for years. Web and
| databases are quite straightforward. The least matured one
| would be the web framework which published its current major
| version last summer.
|
| Those two ecosystems might be the most matured ones in all of
| Julia and in most programming languages. I use have been using
| them extensively in Julia for several years.
| the__alchemist wrote:
| > Plotting, it turns out, is basically a really hard thing for a
| compiler. It is many, many, small methods, most of which are only
| called once. And unlike most Julia code, it doesn't actually
| benefit all that much from Julia's JIT. Julia's JIT is normally
| specializing code, and running a ton of optimizations. But
| plotting itself isn't in the hot-loop - optimizing the code takes
| longer than running it the few dozen times it might be used
| unoptimized. To make a long-story short, plotting is the poster
| child example for Julia needing to compile things before it can
| run them.
|
| This is misleading. There's no reason you need to to compile the
| plotting library every time you load a REPL or program. Python, a
| "slow" interpreted language handles this quickly, as does Rust, a
| slow-to-compile Lang - You can compile and run a Rust program
| that plots more quickly than in Julia, since it doesn't need to
| compile the plotting lib after it's initially installed.
| oxinabox wrote:
| > You don't need to compile the plotting library every time.
|
| You don't need to but Julia does. Its definately a issue, and I
| know it is being worked on.
|
| Julia doesn't store compiled binary code between sessions.
| Unless you compile it into a sysimage.
|
| There are apparently reasons why caching compiled binary like
| this is complicated in Julia. But once that is solved, wow
| things are going to be nice.
| CyberDildonics wrote:
| > There are apparently reasons why caching compiled binary
| like this is complicated in Julia. But once that is solved,
| wow things are going to be nice.
|
| I put a lot of hours into julia five years ago and all the
| same things were being said. I don't know why the compilation
| is so slow or why the caching is so bad, but it was the main
| complaint then and still is. The solutions are all 'just
| around the corner'. It reminds me of java two decades ago.
|
| Actually most languages that get a lot of use seem to go
| through this. The big problems for some reason have solutions
| "just around the corner" but they remain giant problems.
|
| I think what really happens is that people work on what they
| want. Solving their hard problems is not fun and no one holds
| anyone's feet to the flames. C++ has had problems with
| compile time and template errors, but there has been real
| commercial pressure to making progress on those. Julia's
| problems are the same as they were half a decade ago. Start
| working and wait an enormous amount of time for the exact
| things to compile that you compiled yesterday when you
| started it up.
| Certhas wrote:
| I don't think that's fair, simply because things have
| actually gotten a lot better. This is shown in this blog
| post and it also is evident to people who use Julia:
|
| https://i.redd.it/ik4uymvb28k51.png
|
| Also, compiling a sysimage used to be an arcane art, and
| now works reasonably simply/well. It's easy to imagine a
| future where, with the tooling that is already there and
| without a magic breakthrough in caching Julia code, we
| simply get a per project sys-image in VSC that is
| recompiled when needed.
| DNF2 wrote:
| Well, a lot of resources have been put into reducing
| compilation times, and large improvements have been
| achieved. It's not just perennially 'around the corner',
| the improvements are tangible and happening _right now_.
| lhn wrote:
| I don't think it's fair to say Julia's problems haven't
| changed at all. While compilation latency is still an
| ongoing issue, it has consistently and noticeably improved
| over the years. Package caching is much better, you can
| save compilation results you depend on in your workflow
| with PackageCompiler, etc. There are now incredible tools
| (e.g., SnoopCompile.jl) for package developers to inspect
| closely where the compiler might have difficulty and fix
| the issues.
|
| The major source of improvements in 1.6 is eliminating
| method invalidations. Julia's flexibility makes it
| vulnerable to invalidating already compiled code as new
| packages are loaded and new methods are defined. This
| triggers a disastrous cascade of recompiling a bunch of
| things, and is the main conceptual reason why nothing lower
| than type-inferred code is cached. If your method will be
| recompiled anyway, then what good is it to save the native
| code in the first place? Now that invalidations can be
| efficiently diagnosed and patched, there is definitely
| interest into caching lower levels of code in the
| compilation process, potentially even native machine code.
|
| All of these progress however requires labor and care. I'd
| say the Julia community has spent an admirable amount of
| efforts into its latency issue, but there's a limit to how
| fast you can address these problems through open-source
| development without backing from major tech companies.
| Imagine the improvements to the Julia compiler had Google
| chose Julia for its S4TF project, for instance.
| CoolGuySteve wrote:
| Yeah, I got fed up with Julia's plotting library and wrote a
| C++ Qt plot function that forks and plots arrays of doubles.
|
| It runs instantly even with millions of points.
| find wrote:
| You can certainly achieve the Rust plotting solution for Julia
| by compiling a plotting package with PackageCompiler. I use
| this for day-to-day research tasks.
|
| However, Julia users are greedy! They want their cake
| (composability, portability, dynamic language features) and eat
| it too (performance). Lots of effort has thus been put into the
| language towards not needing solutions like PackageCompiler.
| baldfat wrote:
| I loved the idea of Julia but R and specifically the tiddyverse
| https://www.tidyverse.org/ Just makes everything else seem not as
| elegant to my humble eyes.
| tfehring wrote:
| I use R and the Tidyverse extensively. Exploratory data
| analysis is definitely clunkier in Julia - you need the `@pipe`
| macro to patch up some limitations in native pipes, there's no
| `dbplyr` equivalent that I know of, and despite Julia's better
| metaprogramming in general, the lack of built-in equivalents to
| scoped `select`/`mutate`/`summarize` is a real drag. But
| Julia's type system, substantially better date/time system and
| utilities, explicit vectorization with `.`, and the use of
| functions instead of scoped expressions in functions like
| filter are all real benefits over R.
|
| If you write a lot of Rcpp, Julia's performance without
| dropping down into a lower-level language is also a significant
| advantage. It's easy, bordering on trivial, to performantly
| implement a generic join (i.e. `join(f, df1, df2)`) in Julia;
| `dplyr` still doesn't have those at all, `data.table` only sort
| of does, and I believe the canonical R implementation (AFAIK)
| in the `fuzzyjoin` package requires holding the Cartesian
| product of the dataframes in memory, which is obviously not
| great.
| phillc73 wrote:
| Have you tried Query.jl or DataFramesMeta.jl?
| cwyers wrote:
| Very much not the parent, but as a heavy R user, I don't
| think either of them quite nail the way dplyr and the
| tidyverse work. The thing about dplyr is... it's just
| functions. Okay, so, it's functions that leverage features R
| has (notably lazy evaluation and non-standard evaluation).
| But it's just functions. All you need is a function that
| takes a data frame and returns a data frame. So you can take
| a function out of the R standard library, you can take a
| function from a package written before dplyr came around, you
| can take a function from a recent non-tidyverse package, you
| can write your own function... it's all just functions.
|
| In DataFramesMeta.jl, though, you have a macro, and
| everything runs inside that macro. So if you want to take
| something that isn't a part of DataFramesMeta.jl... here's an
| example. Let's say you want to take the popular mtcars
| dataset, and get the five cars with the best gas milage. In
| dplyr, that goes
|
| mtcars %>% arrange(mpg) %>% head(5)
|
| arrange is a function from the dplyr package, head is a
| function from the standard library, but they both work
| seamlessly together.
|
| DataFramesMeta.jl lets you work in a pipe-forward fashion,
| but (at last I knew, at least, it's been a while since I
| played with it), you couldn't use the Julia head function
| within a DataFramesMeta.jl pipeline. You have to do your data
| transformations, assign to a variable, and then get the head
| of that variable.
|
| Which, okay, probably doesn't sound like a big deal. But I
| think it gets at the heart of what efforts to do something
| Tidyverse-like in other languages (Python and Julia, mostly)
| really miss. The key value proposition of the Tidyverse in R
| is that it is very composable and very extensible. That
| means, if you are trying to solve something in a Tidyverse
| way, you can probably find something that works for you. If
| you are doing financial analysis? Get tidyquant. If you're
| doing time series analysis, the tidyverts packages are for
| you. And it all works because there is so little friction
| involved in writing your own functions that extend the
| functionality of Tidyverse packages. Yes, dplyr is a useful
| querying DSL in its own right, but you can find a bunch of
| SQLish query languages, and they're all some degree of fine.
| Query.jl or DataFramesMeta.jl might expose a useful querying
| DSL for data frames, but they don't seem to me to be built to
| support building a whole ecosystem like dplyr and the
| Tidyverse are.
| phillc73 wrote:
| That's a really good point that I'd not really thought
| about. I'd never really considered the difference between
| calling just functions versus macros.
|
| Thinking about Query.jl and DataFramesMeta.jl, and I am for
| sure not an expert in either, I can't specifically speak to
| your `head` example, but other base functions can be
| combined with macros. For example, see the LINQ examples
| from DataFramesMeta.jl[1] where `mean` is being used. Or
| again the LINQ style examples in Query.jl[2], where
| `descending` is used in the first example, or `length`
| later in the Grouping examples.
|
| Is that the kind of thing you meant?
|
| For whatever reason, with the way my brain is wired, the
| LINQ style of query just works for me. I have never
| directly used LINQ, but do have some SQL experience. In
| fact, I wrote some dinky little wrapper functions[3] around
| duckdb[4] so I could directly query R dataframes and
| datatables with SQL using that backend, rather than
| sqldf[5].
|
| [1] https://juliadata.github.io/DataFramesMeta.jl/stable/#@
| linq-...
|
| [2] https://www.queryverse.org/Query.jl/stable/linqquerycom
| mands...
|
| [3] https://github.com/phillc73/duckdf
|
| [4] https://duckdb.org/
|
| [5]
| https://cran.r-project.org/web/packages/sqldf/index.html
| baldfat wrote:
| I don't like working with DATA TABLES UNLESS it is a HUGE
| data frames. Then if it is huge I'll go towards sparks. I
| normally am working with under a million objects which with
| today's computers is not that big.
|
| Edit I meant to say that DATA TABLES library in R reminds me
| more of Query.jl then tiddyverse
| phillc73 wrote:
| Query.jl supports two different paradigms, one inspired
| directly by LINQ, the other by dplyr.[1] I actually prefer
| data.table in R, over dplyr, and Query.jl is really quite
| different.
|
| [1] http://www.queryverse.org/Query.jl/stable/
| wodenokoto wrote:
| What are you doing in the tidyverse that is not related to
| dataframes (or tibbles as the subclass of data.frame, that
| tidyverse uses is called)?
| diarrhea wrote:
| > tiddyverse
|
| That is not what you meant to say.
| fishmaster wrote:
| You can use RCall to use R from Julia:
| https://github.com/JuliaInterop/RCall.jl
| ku-man wrote:
| Good idea, and since I am on it I should use rpython to, in
| turn, call python from R. Julia provides awesome solutions.
| celrod wrote:
| > (* Technically not all mutable objects live on the heap,
| because some never live at all, as they are optimized away so are
| never allocated in the first place.)
|
| The compiler will often stack allocate mutable objects in Julia.
| This is not the same as "never existed in the first place",
| because the stack pointer gets incremented and underlying data
| layout is you load from it is the same as that of the mutable
| object you allocated.
|
| Here is one example where that's very obviously what's happening:
| https://discourse.julialang.org/t/why-is-svector-faster-than...
| But it can happen now generally with mutable structures that
| can't practically just live in registers.
| oxinabox wrote:
| fixed, to be more limitted in the claim
| celrod wrote:
| Cool, and fantastic summary! I enjoyed reading it.
| 295310e0 wrote:
| Can someone comment on the ease of distributing Julia code? Can I
| easily take a bit of code I write (w/o external libraries) and
| produce a single binary I can ship to someone or do they require
| a full Julia environment to run it?
| jakobnissen wrote:
| They require the full Julia environment to run it - and it's a
| heavy environment. It is possible to compile a binary that
| includes the environment and the compiler, but IIRC, that will
| result in a >400 MB hello-world script taking 150 MB of RAM to
| run.
|
| The core devs have mentioned they are going to add the capacity
| to compile to actual static binaries, but that does not seem to
| be a top priority, so I wouldn't hold my breadth waiting for
| it.
| adgjlsfhk1 wrote:
| Note that this feature doesn't necessary require work by the
| core devs. It is completely feasible (at least in theory) to
| write a library that can output static exectuables.
| ChrisRackauckas wrote:
| It's actually done all of the time. GPUCompiler.jl, the
| core of the CUDA and AMD GPU stack, builds static binaries
| (compiled to .ptx by LLVM for CUDA for example, but the
| choice is just a switch), then stashes those binaries to
| use with a ccall in a Julia function. You could in theory
| use that stack to statically-compile anything that's GPU
| compliable, and it's really well-tested.
| eigenspace wrote:
| Doing this without the user having Julia installed would
| require using PackageCompiler.jl to make a standalone
| executable.
| https://julialang.github.io/PackageCompiler.jl/dev/apps/
|
| This is a pretty stable well established process now, but the
| binaries it produces are huge because they actually have the
| full Julia runtime in them.
|
| Active work is happening on small binary static compilation. We
| already do it for GPUs, we just have to repurpose our GPU AOT
| compilation pipeline for the CPU. There are proofs of concept
| that currently work, but something more polished is feeling
| like it'll probably be another year or so.
| swagonomixxx wrote:
| As someone who knows very little but wants to learn: what's the
| best way to get up and running with Julia? On the website, they
| link a lot of videos, but I prefer textual formats. Is there
| something like the Rust Book for Julia?
| nextos wrote:
| The official manual is very good, and reads more or less like a
| textbook: https://docs.julialang.org/en/v1/
| cbkeller wrote:
| Yeah, this is probably actually the closest equivalent to the
| Rust Book, but I'll also second the suggestion from the other
| comment of "Think Julia" (beginner) and "Design Patterns and
| Best Practices with Julia" (intermediate /advanced).
|
| For me, the most important thing to grasp when coming from
| another language was that Julia's multiple dispatch brings
| with it effectively a whole paradigm of "dispatch-centric
| programming" that you have to embrace to really get the most
| out of Julia, including the c-like speed (have to strictly
| avoid type-instability for that) and the composability that
| everyone talks about.
| thetwentyone wrote:
| - [JuliaLang.org](https://julialang.org/), the home site with
| the downloads to get started, and links to learning resources.
|
| - [JuliaHub](https://juliahub.com/ui/Home) indexes open-source
| Julia packages and makes the entire ecosystem and documentation
| searchable from one place.
|
| - [JuliaAcademy](https://juliaacademy.com/courses), which has
| free short courses in Data Science, Introduction to Julia,
| DataFrames.jl, Machine Learning, and more.
|
| - [Data Science Tutorials](https://alan-turing-
| institute.github.io/DataScienceTutorials...) from the Alan
| Turing Institute.
|
| - [Learn Julia in Y
| minutes](https://learnxinyminutes.com/docs/julia/), a great
| quick-start if you are already comfortable with coding.
|
| - [Think Julia](https://benlauwens.github.io/ThinkJulia.jl/late
| st/book.html), a free e-book (or paid print edition) book which
| introduces programming from the start and teaches you valuable
| ways of thinking.
|
| - [Design Patterns and Best
| Practices](https://www.packtpub.com/application-
| development/hands-desig...), a book that will help you as you
| transition from smaller, one-off scripts to designing larger
| packages and projects.
|
| - Lots more topical books (Statistics, Optimization, etc) if
| looking for a Julia-oriented subject matter
| aliceryhl wrote:
| I really wish the language doesn't force me to use the REPL.
| npr11 wrote:
| Really nice to see so many little useability improvements -- like
| easy temporary envs, syntax highlighting in dependency conflict
| errors, and more partially-applied functions -- as well as more
| significant language development on threading, stack allocations,
| and reducing latency.
| oscardssmith wrote:
| In a lot of ways, I feel like 1.0 was a backend LTS while 1.6
| is more of a front end one. 1.0 had most of the basics nailed
| down, but it's taken a while for it to become as seemless as it
| now is.
| klmadfejno wrote:
| > People often complain about the "Time To First Plot" (TTFP) in
| Julia. I personally have never minded it - by the time I am
| plotting something, I have done minutes of thinking so 20 seconds
| of compilation is nothing.
|
| This amuses me. I hadn't really considered the author's
| perspective, and now I think it aligns with my take on it pretty
| well.
| linspace wrote:
| It is nevertheless im portant because a simple plot is a nice
| test if you are considering using a language for scientific
| computing. Criticism about plotting libraries quality is even
| more valid. I cannot believe that plotting an scatter plot of a
| few million points is so slow while AAA videogames render
| millions of pixels in real time.
|
| On the long term? I think Julia is a better language than
| Python, Matlab or R for several reasons like modularity,
| package management and performance. But these are things that
| require at least 10 hours of use (to say something) instead of
| 10 minutes. With so many languages promising enlightent and
| transcendence to a next power level you cannot expect people to
| make that kind of investment.
| eigenspace wrote:
| Makie.jl [1] does it's plotting on the GPU (like a video game
| engine), so can handle millions of datapoints just fine.
|
| Also, note that for people to whom plotting is really
| important, it's quite easy nowadays to just AOT compile your
| plotting library to your sysimage with PackageCompiler.jl [2]
| for instant plots.
|
| [1] https://github.com/JuliaPlots/Makie.jl
|
| [2] https://github.com/JuliaLang/PackageCompiler.jl
| vanderZwan wrote:
| If that's what you're used to it's fine probably, you'll often
| have no choice but to adjust your workflow to it and make it
| work. But if you've ever experienced an environment that
| instantly shows you results and lets you get into a fast
| iterative loop to explore your data, it can be hard to give
| that up again.
|
| Source: my current and previous job were basically data viz
| programming jobs which were all about optimizing said iterative
| loop for scientists. Going from minute-long to sub-second
| rendering speeds is a game-changer for many.
|
| _EDIT:_ having said that, this kind of reminds me of what the
| biggest difference between analog and digital photography is
| for me, namely whether or not you get instant feedback. I do
| remember from my art school days that in my experience film was
| a much better option for training the skill of observation and
| composition than digital, because it forces you to essentially
| picture the photograph before you take it. However, once you
| get somewhat decent at that... I 'd switch to digital and reap
| all the benefits it has ;). The same logic might apply to
| learning how to plot your data.
| stjohnswarts wrote:
| I used to thing c++/rust was bad. Then I did some FPGA work
| and it was quite painful waiting for rebuilds :) . I even
| appreciated c++/rust build times a bit more after that.
| Waiting an hour before being able to test your code will
| definitely make you cognizant of dotting your i's and
| crossing your t's in new code. We had a design that filled
| almost 80% of the chip and had pretty strict timing (near the
| limit of the chip) and it took it quite a while for the
| fitter to meet the timing requirements.
| cbkeller wrote:
| Totally fair, though note that these slow times are only for
| _first_ plot - I leave my repl open all day, and all but the
| first plot are sub-second
| socialdemocrat wrote:
| It is literally "first" time, not every time. Thus if your
| worry is that you cannot do fast iterations in Julia where
| you plot multiple times, then your worries are misplaced.
|
| Second, third, forth etc plots in Julia are fast. Likely
| faster than any of the competition as it is running highly
| optimized native code at that point.
| oblio wrote:
| That won't stop these people. It's the same discussion as the
| interpreted languages vs compiled languages or the editor flame
| wars.
|
| On one hand you have people that say "thinking takes a lot
| longer than waiting a bit for compilation or actually editing
| source code" (I'm in this camp) and people that go "I don't
| want to wait for compilation and I want my editing to be hyper-
| efficient even if I have to invest hundreds and thousands of
| hours into it, so that I'm always in the zone/in the flow".
|
| People are just different but every camp thinks They're Right
| and The Others Are Dumb and Stupid And Dangerous.
|
| My personal guess is that besides the split in
| personalities/workflows, there's also a difference in projects.
| People who work on existing projects tend to read a ton more
| code/docs/team comms/architectural diagrams and edit/compile
| less so they care less about these issues. People who
| constantly create tons of mini-projects with short lifecycles
| care more about them.
| simias wrote:
| I don't know how Julia fares but personally what bothers me
| with long compile times is when I can't context switch,
| waiting for the compiler's output.
|
| What I mean in practice is that if you take Rust for
| instance, the compile times can be fairly long but the type
| checking occurs early on and is quite fast. Therefore once I
| know that this step succeeded I can usually let the
| compilation continue in the background while I focus my
| attention elsewhere.
|
| If on the other hand if I need to wait a lot longer to
| confirm that my code is actually valid I find myself just
| staring at the output window, not willing to let go of my
| short term memory until I get a confirmation that my code was
| accepted.
|
| The problem is not adding 20s to your overall dev time, it's
| to have a 20s interruption while you're "in the zone".
| dm3 wrote:
| That's right. However, the 20s delay only happens once
| during the first plot action in the fresh REPL. That's when
| Julia compiles all of the functions not present in the
| system image. All of the repeated plotting will not require
| recompilation of the "base" plotting libraries.
| DNF2 wrote:
| If it were 20 seconds for _every_ plot, this would be a
| major problem for me, as I tend to make _lots_ of plots.
| But it 's only the _first_ plot where this is an issue.
| Surely you 're not in 'the zone' that soon?
|
| Seems to me like people are making a mountain out of a
| molehill.
| jampekka wrote:
| If you don't use REPL (and for many good reasons you
| shouldn't) or some other such horror, every plot is the
| first plot. And it's pain. And not just plots really.
| Doing anything in Julia is pain if you try to use it as a
| programming language instead of an app for buggy,
| unreproducible and misunderstood ad-hoc analyses.
|
| Seeing these answers makes me think Julia will never be
| fixed. I forecast Julia will be back in a niche within
| five years if they don't get their act together. And it's
| sad, because the alternatives are fundamentally broken.
| Julia isn't fundamentally broken, but the devs and the
| community seem to insist on superficial breakage.
| eigenspace wrote:
| > (and for many good reasons you shouldn't)
|
| Could you elaborate on this? I'd say repl based
| interactive programming is one of julia's greatest
| strengths, and avoiding the repl is probably setting
| yourself up for pain.
|
| That said, if you do find yourself running lots of
| scripts and paying this penalty all the time, I'd suggest
| https://github.com/dmolina/DaemonMode.jl as a great way
| around these pains.
| jampekka wrote:
| > Could you elaborate on this? I'd say repl based
| interactive programming is one of julia's greatest
| strengths, and avoiding the repl is probably setting
| yourself up for pain.
|
| With REPL you have an invisible global state, can't
| reproduce what you have done, changes earlier in code
| path don't propagate to results, you don't have
| documentation of what you did.
|
| It's for me really like trying to write a book by
| dictating. Except that you're dictating to somebody who's
| gonna give an independent summary of it to a third party
| and never gonna write down what you dictated. It boggles
| my mind how people can work like this, but they probably
| get hooked to REPL from the first tutorials and just
| don't know better.
|
| I'll look into DaemonMode.jl. Not a fan of using a daemon
| (and I'm guessing there will be problems with e.g.
| interactive plots), but in the short term I'll take
| anything that could make Julia programming tolerable.
| eigenspace wrote:
| > With REPL you have an invisible global state, can't
| reproduce what you have done, changes earlier in code
| path don't propagate to results, you don't have
| documentation of what you did.
|
| Mhm, that's fair. I think Pluto.jl has a really neat
| approach to this, using reactivity (and technically even
| more state) to actually eliminate that experienced state.
|
| If I could use it from emacs it might even be my goto way
| to interact with julia, but I also don't mind the
| statefulnes and find it manageable.
|
| For me, the most important thing is that when I'm writing
| serious code, I create a local package. Then, in the REPL
| I load that package and have Revise.jl active so that it
| can watch the the package source ode and constantly do
| hot code reloading for me so that I'm never stuck with
| old versions of code running.
|
| Then I do all my interactive analysis in the REPL, and
| plumbing in the package module. This eliminates a lot of
| statefulness, but keeps restarts to a minimum.
| jampekka wrote:
| I briefly looked at Pluto.jl, and I think it's probably a
| good way. As I understood it it's a "notebook" that
| always runs the whole file. Like e.g. RMarkdown or
| sweave. All good. The state is fine too if it's explicit.
| But I'm fine with just CLI and print and occasional plot,
| which should be a lot simpler use case for development.
|
| I create a "local package", meaning a file from which I
| relatively import. During development/analysis it's hard
| to foresee what the package structure is gonna be, so
| it's quite pointless to go through the whole packaging
| ceremony at this point. FromFile works fine for this.
|
| As a temporary hack I could use REPL to call my "main"
| function and let Revise.jl update automatically. (In long
| term this is bad for interoperability with rest of the
| system). But in my experience Revise.jl tends to break a
| lot. Julia breakage is hard to analyze by itself, and
| Revise.jl often makes this more or less impossible.
|
| I have to repeat that I really don't see how caching of
| the compilation results is even close the complications
| that Revise.jl or Pluto.jl have to do.
| eigenspace wrote:
| > I briefly looked at Pluto.jl, and I think it's probably
| a good way. As I understood it it's a "notebook" that
| always runs the whole file.
|
| Not quite. It builds a dependancy graph of your code and
| can figure out what definitions depend on others. So
| depending on what you change, maybe only one or two cells
| need to be rerun. Or in other circumstances, the whole
| notebook will have to re-run. It just depends on what
| changes.
|
| > I have to repeat that I really don't see how caching of
| the compilation results is even close the complications
| that Revise.jl or Pluto.jl have to do.
|
| I think the main trouble with the caching is that the
| native code you cache can depend very strongly on the
| exact combination of packages you have loaded. This means
| you can hit a combinatorial explosion of different
| methods to cache pretty quickly, so you'd need to find a
| very clever way to find the _right_ methods to keep and
| which ones to delete once the cache gets too big.
|
| I think there's also other potential issues that I
| understand less. This is being actively worked on though.
| jampekka wrote:
| > Not quite. It builds a dependancy graph of your code
| and can figure out what definitions depend on others. So
| depending on what you change, maybe only one or two cells
| need to be rerun. Or in other circumstances, the whole
| notebook will have to re-run. It just depends on what
| changes.
|
| But the effect is still that any changes up-file will be
| always reflected down-file? If so, I don't care how it's
| implemented (given it's fast enough and doesn't break),
| the semantics is the point.
|
| > I think the main trouble with the caching is that the
| native code you cache can depend very strongly on the
| exact combination of packages you have loaded. This means
| you can hit a combinatorial explosion of different
| methods to cache pretty quickly, so you'd need to find a
| very clever way to find the right methods to keep and
| which ones to delete once the cache gets too big.
|
| Yes, I think this is a problem for a clean solution. But
| for a big fat ugly hack that isn't too picky on wasting
| disk space or occasionally recompiling stuff needlessly
| it's probably less so.
|
| For a lot of cases very rough invalidation would probably
| suffice. E.g. invalidate all definitions from all files
| that are changed from the last run (i.e. like Make does).
| And invalidate all definitions for any name that gets any
| definition. I'd guess accomplishing this would cut the
| startup time greatly; the end-user code rarely redefines
| (at least intentionally) anything that's in the packages,
| and vast majority of time is spent (re)compiling the
| packages themselves.
|
| I'm sure there are complications with type inference. But
| I'd be willing to pepper some explicit typing in my code
| if it means I don't have to recompile it every time I run
| it. Binary of a method with concrete types should at
| least be trivially cacheable (given no library changes
| between runs).
|
| > This is being actively worked on though.
|
| It's been worked on for as long as I've known of Julia.
| AFAIK there's still absolutely zero logic on caching
| compilations of "end-user-stuff" (as opposed to stuff
| like package precompilation). I don't think this is
| necessarily due to technical issues, but because the
| community says that REPL (or notebook) is the only way of
| using Julia, and those don't suffer from the problem that
| much (Revise.jl breakage notwithstanding).
|
| Technically it's probably very difficult to do
| "perfectly", and I'm thinking this is how the compiler
| devs want to do it. I'm not sure they even mean
| persisting-between-runs caching when they say "caching"
| in compiler related discussions. It may well be just some
| run-time caching of some compilation artefacts that are
| now compiled multiple times. And that would probably not
| have that dramatic performance gains for the re-run case.
|
| For an AOT compiler Julia is clearly fast enough. There
| are probably no easy tricks left to make it a lot faster.
| But re-run performance doesn't need faster AOT, it just
| needs the compiler not to recompile the same identical
| stuff every time.
| eigenspace wrote:
| > But the effect is still that any changes up-file will
| be always reflected down-file? If so, I don't care how
| it's implemented (given it's fast enough and doesn't
| break), the semantics is the point.
|
| Yes, I was just bringing this up because it means that
| various things can be significantly faster, causing you
| to experience less latency than you normally would by re-
| running a whole file.
|
| As to the rest of your most, I agree it'd be interesting
| to see a more quick and dirty solution. It appears that
| everyone who has the know-how to do this wants to 'do it
| right', so on the public facing side there's very little
| visible progress.
|
| > It's been worked on for as long as I've known of Julia.
| AFAIK there's still absolutely zero logic on caching
| compilations of "end-user-stuff" (as opposed to stuff
| like package precompilation)
|
| This is not really true. E.g. there's PackageCompiler.jl
| which does sysimage based caching and works quite well
| (at the expense of slow compilation and large binaries),
| and briefly there was StaticCompiler.jl which did good
| small binary compilation but then bitrotted quite fast.
|
| All of our CPU compliation stuff is built using a small
| binary, static, AOT compiler (currently hosted in
| GPUCompiler.jl) and it's quite reliable. There's active
| work being done to make this work on the CPU again
| (basically a modern version of StaticCompiler.jl). So
| while I feel your frustration that this has been 'coming
| soon!' for a long time, progress _has_ been made. The new
| compiler hooks for version 1.6 are partially designed to
| make this whole process less hacky and easier to iterate
| on.
| stjohnswarts wrote:
| I'm relegated to that it's mostly the size and complexity of
| the project as a whole. If you ask me to write small
| automation scripts that will quickly become obsolete I will
| choose bash/python every time. If you ask me to make
| something that has to last a while and requirements won't
| change a lot and is of a decent size I might write those same
| scripts/apps in rust or c++ (modern! 17+) for maintainability
| and disciplined structure. I think a lot of people are in one
| camp or the other, but I like to be in the middle somewhere,
| hopefully seeing the advantage of both.
| cmeacham98 wrote:
| You seem to be falling into the same trap you lament in your
| comment:
|
| - The other side is "these people" that "won't stop"
|
| - They're striving for "hyper-efficiency" at the cost of
| "hundreds and thousands of hours"
|
| - People who complain about this issue don't (or do
| significantly less of) reading documentation/code/etc
|
| People talk about TTFP because it is a real issue that is
| off-putting for many programmers that would otherwise love to
| use Julia. Julia is roughly an order of magnitude slower than
| python in this instance on my computer, and that's not a good
| first impression.
|
| That doesn't mean that everyone is going to be impacted by
| this issue (obvious ex: you aren't), but this isn't akin to a
| flamewar because unlike editor choice (which is opinion),
| Julia would be better for everyone if TTFP was improved.
| Whether or not it should be prioritized as a development goal
| is up to the Julia team, but it's not just some difference of
| opinion like interpreted vs compiled or functional vs OO.
| DNF2 wrote:
| I cannot fathom how ttfp is important. It's time to _first_
| plot, not to _every_ plot. After it 's down to ~10 seconds,
| why on earth does anyone care?
| saiojd wrote:
| You can't possibly be serious. As things stand, even just
| redefining a struct field requires restarting the
| interpreter, which forces you to wait ~10-30 seconds for
| everything to compile, over and over again. Maybe you
| only use Julia for small scripts, and can afford to never
| restart the interpreter?
| the-smug-one wrote:
| Dang, why can Common Lisp do this so well and Julia so
| poorly?
| eigenspace wrote:
| The assumptions our compiler is able to make about types
| never changing (and a few other restrictions) allows for
| a lot of optimizations in julia that are not possible in
| Common Lisp. Julia's compiler is able to do a few things
| significantly more aggressively than CL.
|
| That said, there is some very interesting work happening
| on getting around these restrictions. There was a PR from
| Tim Holy a while ago that could have allowed it, but
| there were some problems with the PR, and there were also
| some associated costs that were deemed too steep to pay.
|
| That said, there's other great work on other ways around
| this. For instance, you can dynamically redefine structs
| all you want in Pluto.jl notebooks and there's no
| performance penalty!
|
| This is something that kinda just fell out as a natural
| consequence of it's reactive design.
| fishmaster wrote:
| > even just redefining a struct field requires restarting
| the interpreter
|
| It really doesn't if you use a module.
| whateverwhynot wrote:
| Well, with Julia it is time to first plot for every new
| session. So if your typical workflow is to start Julia
| and just do a single plot of the recently acquired data
| (quite common workflow in my field), Julia is an order of
| magnitude slower than python or scilab.
| cbkeller wrote:
| If that's your workflow it would probably be highly
| advisable to compile plots.jl into your sysimg
| enriquto wrote:
| > I cannot fathom how ttfp is important.
|
| I personally do care for my concrete usage pattern.
|
| This is my use case: A long shell script that does a lot
| of things. At some point, inside a loop that runs
| hundreds of times, it needs to solve a couple of small
| linear systems and plot a simple graph. There's hundreds
| of png graphs, that are then combined into a video
| sequence. Right now, the computation is done by calling
| octave (inside the loop) and then gnuplot (to create the
| actual graph from the octave computed data points). I
| would like to replace each call to octave+gnuplot to a
| single call to julia. Yet, this would make my script run
| in a few hours instead of a few seconds, because for this
| usage pattern _all plots are first plots_
|
| Before you suggest that I should rewrite the whole thing
| in julia, maybe you are right but
|
| 1) it would take me a few weeks that I don't have
|
| 2) that's not my point. A good tool is a tool that can be
| used for purposes that it was not intended to, like this.
| If the time to first plot in julia was a millisecond
| instead of 10 seconds, then julia would be a much better
| tool.
| DNF2 wrote:
| But this is really the worst conceivable use case for
| Julia. Why are you interested in making this change, when
| Julia offers no advantage in this scenario?
| whateverwhynot wrote:
| I'm not the parent, but I faced exactly the same issue
| when I tried to switch to Julia from scilab. As for why
| I'd like to switch: Julia is a better language, it is way
| easy to run your script on a cluster and I've been bitten
| more than once by the two language problem... So Julia
| seems better for everything but for one of my most common
| use cases...
| aliceryhl wrote:
| It would be a perfectly fine use-case for Julia if it
| wasn't so damn slow.
| enriquto wrote:
| > Why are you interested in making this change, when
| Julia offers no advantage in this scenario?
|
| Well I love julia the language. It's the interpreter
| quirks that I find annoying. If julia had something lean
| and superfast like luajit it would be incredible!
| stillyslalom wrote:
| You may want to look at [DaemonMode.jl](https://github.co
| m/dmolina/DaemonMode.jl), which spins up a persistent
| Julia process in the background so you don't have to pay
| the TTFP penalty more than once, even when shell-
| scripting.
| stjohnswarts wrote:
| I still have a soft spot for octave. It was my first
| matrix oriented program. I've moved on to pandas and c++
| but I still pull up octave for signal analysis because
| it's just plain easier and has some great functions and I
| think it may have created folds in my brain as an EE
| undergrad that will always be there :)
| sgt101 wrote:
| Could you cache the requests or write the parameters into
| a file and then make the call to Juila, and ask it to
| loop over the requests?
| jarvist wrote:
| As you already have the gnuplot commands, perhaps using
| Julia + https://github.com/gcalderone/Gnuplot.jl might
| work? It's a very lightweight package, so even using it
| directly should be OK.
|
| I do all my plotting with Gnuplot.jl, as gnuplot is fast,
| and I can save .gpt files which reproduce the plots for
| later reference and making publication-quality.
| cbkeller wrote:
| It's not trivial to do, but compiling plots into your
| sysimg might actually fix that
| enriquto wrote:
| Thanks! Is there a way to compile all installed packages
| into the sysimage? If so, why isn't that the default?
| cbkeller wrote:
| Technically yes, but then IIRC you'd have to recompile
| the whole sysimg any time you update any single package,
| which could get to be a pain. The usual compromise seems
| to be to include just a handful of your most used
| packages in the sysimg (say Plots + Revise)
| enriquto wrote:
| > you'd have to recompile the whole sysimg any time you
| update any single package
|
| Are there any downsides to this? You never care how long
| does a system update take. You _always_ care how long do
| your programs run.
| Sukera wrote:
| The downside is that PackageCompiler (or SnoopCompile,
| for that matter) can only pack into the sysimage what it
| sees. This is usually done by "recording" which methods
| get called during a session and putting the compiled
| native code for those into the sysimage. If it's not
| called during setup, you'll still have dynamic
| compilation (as for any code that hasn't ever run).
| cbkeller wrote:
| I've never tried it myself, but can't think of any reason
| why it wouldn't work in principle if you do want to go
| that route
| walshemj wrote:
| Me too not to go all four yorkshire man here, but I can
| remember when you would have to write your own plotting
| programs in Fortran and then wait for the pen plotter to
| slowly draw your plot.
|
| And not that long before that you would print the results
| out and manually plot them on graph paper.
| dnautics wrote:
| > People are just different but every camp thinks They're
| Right and The Others Are Dumb and Stupid And Dangerous.
|
| That's not quite right, I think. People are looking for
| excuses to not use Julia (r new technology X) because it
| serves as confirmation bias that their choice of <blub
| language> is still good and there is no need to start
| thinking of their extensive training and investment in blub
| is sunk.
| jampekka wrote:
| I've tried it dozens of times. I tried it this week and
| gave up again after several hours.
|
| As a language and technology it's way better than the
| alternatives, but usability of Julia as a programming
| language is broken by the ridiculous startup latency. I'm
| sure it's not even really hard to fix (at least by some
| caching hacks), but for some reason the Julia community is
| actively resisting such fixes.
|
| And don't give me REPL. REPL is a fundamentally broken
| approach to programming, and REPL people just keep looking
| for excuses to keep using it.
|
| Edit: And if you give me REPL, I can answer that I have
| tried it too. It's broken as well. Revise.jl breaks
| constantly with anything non-trivial. And with the effort
| going to horrible hacks like Revise.jl, I'm sure a simple
| caching of compilation results between calls would be
| nothing. It seems to be something ideological.
| socialdemocrat wrote:
| People have different style and preferences. I have
| programmed for over 30 years and I find that REPL based
| development in Julia beats anything else I have tried in
| term of productivity, and I have tried a ton of tools,
| IDEs and languages.
|
| But sure it may not fit your particular preference or it
| may be that you have simply not learned to use it
| effectively. It takes some time to work effectively in a
| REPL style. It took me some years.
|
| Not every language is suited for REPL development. Julia,
| LISP and Haskell seem quite well suited.
|
| I don't have quite the same good experience with Python
| e.g.
| jampekka wrote:
| It probably depends also on what you program. If the task
| is simple enough and doesn't need much revisiting, REPL
| is probably fine. But OTOH, just writing the code for a
| simple case and running it isn't too bad either.
|
| How do you persist and document your code with REPL-
| development? Do you log the commands to some separate
| file? How do you recreate the REPL state if it crashes or
| you have to reboot? How do you make sure the REPLs state
| is what you think it is?
|
| These are (some of) the concrete problems that I see with
| REPL, and that don't exist for program based workflow.
| And I think these are fundamentally impossible to solve
| for REPL, and very important for e.g. reproducibility
| (and IMHO sanity).
| gugagore wrote:
| What do you prefer as an alternative to a REPL? How do
| you interact with your programs?
|
| What seems broken to me about REPLs is how text-centric
| they usually are. But I want to be able to easily
| introspect and play with my programs, and REPLs are one
| way to do that. Really good debuggers and environments
| for static languages are "another" way.
| jampekka wrote:
| I just make programs that have an "entry point" (e.g.
| main() in C or if __name__ == "__main__" in python) and
| run them off the shell. Oftentimes with some arguments.
| This is how I interact with most of my programs, and it
| works fine for my own ones too.
|
| I agree that this is probably not for everybody, i.e. if
| you're not used to the CLI workflow. And admittedly it
| would be sometimes nice to have e.g. embedded graphics,
| but unfortunately the troubles usually outweigh the
| benefits (looking here at emulating damn 70's terminals
| too...).
|
| I mostly do the introspection with print, dir and help
| straight in the code. Not ideal, but rarely fails you,
| and I've yet to find a debugger GUI or IDE that isn't
| more trouble than it's worth.
|
| Something like autoupdating RMarkdown/Sweave/Pweave/etc
| would probably work often as well. I sometimes do use
| Pweave, although it tends to be a bit buggy as well, and
| doesn't have any caching logic (although it's easy to use
| your own).
|
| Sadly most efforts seem to go to Jupyter notebooks and
| such, whose state/code inconsistency are simply a non-
| starter if one wants to keep some sanity.
| gugagore wrote:
| > I mostly do the introspection with print, dir and help
| straight in the code.
|
| This means you have to change your code in order to debug
| it. And you have to know what you're debugging before you
| change your code. This is really, in my opinion, much
| less than ideal, because you have to iteratively
| instrument your code while you figure out what is wrong.
| It's a cycle of You print out the first suspect thing,
| then that produces 5 potential suspects, and you have to
| decide which one to print next, or print all of them.
|
| You're absolutely right that it rarely fails you, and so
| I surely want that facility to be at my finger tips. But
| getting a text representation of a value is literally 33%
| of what a REPL is for.
| cbkeller wrote:
| I'm always amused to see these "I'm sure caching wouldn't
| even be a hard fix, the community must be resisting it"
| takes.
|
| Feel free to show me the PR's that have have been
| rejected that would have solved the problem. Or if you
| think it's easy, feel free to make that PR yourself.
|
| As the quote goes, "There are two hard problems in
| programming: cache invalidation, naming things, and off-
| by-one errors". This is cache invalidation.
| jampekka wrote:
| I will probably give it a go when I have some time to
| spare. I don't expect to get a PR through, or even want
| one. Perhaps I'll release it as a package if it works at
| all.
|
| Cache invalidation is not always that hard. For example
| pure functions are more or less trivial to cache, and you
| don't even have to do any explicit invalidation. Perhaps
| do some LRU type pruning if the disk starts to fill up.
|
| I know next to nothing about Julia's internals, but given
| packages like Revise.jl, PackageCompiler.jl and
| SnoopCompile.jl are even possible, I don't think it can
| be that hard. Dumb caching should be a lot easier than
| any of these.
|
| I may well be wrong, and that nobody has done this yet is
| a hint to me being wrong. But I think another scenario
| may be that Julia ecosystem is so hung up on REPLs and
| notebooks that this case just gets no attention. And very
| few non REPL-or-notebook people hang around long enough
| to get to know the internals at all. Maybe I'm just
| desperate enough?
|
| There's also another possibility, which may sound bizarre
| but I think is possible. At some level people who come
| from scripting language background think that long
| compile times is a sign of a "real language". This is
| somewhat prevalent in e.g. Javascript scene, where more
| and more byzantine compilation systems are introduced for
| a language (or platform) that works just fine without
| compilation (or can do very fast on-the-fly "AOT" if
| needed).
| Sukera wrote:
| If you don't mind me asking, how are you using julia? Are
| you using modules to wrap your code and functions to
| structure it, or are you just running one big script in
| top level scope?
|
| Aside from compiler improvements, most caching related
| optimizations are happening on the module level, because
| that's where namespaces are seperated.
| jampekka wrote:
| I'm not using modules. I usually start with one file with
| a demo or similarly named function that is called if the
| file is called as an entry point (like if __name__ ==
| '__main__', except Julia makes it even worse). First the
| "actual" code is in separate functions in that file. No
| global state.
|
| I tend to refactor code out of there to separate files,
| and then somehow import it. An ugly way is include, and
| I've tried Revise.jl with includet.
|
| But I think the least ugly approach is the @from macro
| from here: https://github.com/Roger-luo/FromFile.jl
| Judging from some opinion in bug trackers, this is
| probably gonna get totally shunned by core devs and
| they'll keep on bikeshedding about the import stuff
| forever.
|
| With this setup I have about 400 lines of code in three
| files. It compiles for 15 seconds. After every single
| change, and actually without any changes too.
|
| I think performance wise this should be equivalent to
| using modules, but saving some pointless ceremony.
| Sukera wrote:
| I'm using `!isinteractive() && main()` as my `if __name__
| == '__main__'` equivalent, not sure how that's even
| worse?
|
| It's not equivalent no - include doesn't introduce a
| namespace and neither does includet. Compiled stuff from
| packages (=modules with a Project.toml) is cached between
| runs, scripts just don't have that luxury of seperation.
| @from doesn't look into the files you're including and
| (somewhat simplified) verbatim pastes the code into your
| "main" file.
|
| I don't think it's a lot of "pointless ceremony",
| especially since it keeps dependency management on a per
| project basis easy, is just a `]generate MyPkg` away and
| allows compiled code to be cached
|
| If you don't want to use projects, that's fine - but
| please do so in a constructive manner and don't be
| surprised that the most common workflow (wrapping things
| in a package) gets more attention sooner. That just
| signals some disregard for other peoples' needs & wants,
| even if that's not intended.
| jampekka wrote:
| !isinteractive() && main() doesn't probably work for my
| case. Oftentimes my files may have a main of their own,
| but that isn't called if the file is just imported. I
| don't use Julia interactively anyway (I've explained why
| many times in this thread). (Edit: the equivalent Julia
| chant is `abspath(PROGRAM_FILE) == @__FILE__,` IMHO
| slightly even more obtuse than Python's, but this is a
| minor detail).
|
| If a "package" is used only by me and only from files
| controlled by me, Project.toml is clearly pointless
| ceremony. And `]generate MyPkg` too, and assumes REPL on
| top. Python manages this (albeit with some stupid
| arbitrary restrictions) fine, Node manages this fine. The
| compiler doesn't need that stuff for anything.
|
| I didn't look into the implementation of @from, but I
| picked it up from a huge bikeshedding bug (still open,
| from 2013...) about local module imports, and assumed
| it's doing imports instead of including, as it also has a
| separate namespace. From the code [2] it's not clear to
| me exactly what it does when, but one branch seems to be
| generating a module with the code imported on the fly.
| Not sure this should be any different than any other
| module for the compiler. Are you sure you're not talking
| out of your ass on this one?
|
| I don't care if people for some reason want to write
| their pointless ceremony, but what I don't understand is
| that people are so jealous of it that they insist of
| pushing it on everybody else too. I just want to somehow
| get access to those symbols defined in another file, why
| does this need more than the path of the file? I'm sure
| using just files-as-modules would probably be less work
| for the compiler, and it's easy to have a byzantine
| package ceremony on top if you want (Python has dozen or
| so available, so lots to draw from).
|
| [1] https://github.com/JuliaLang/julia/issues/4600 [2]
| https://github.com/Roger-
| luo/FromFile.jl/blob/master/src/Fro...
| saiojd wrote:
| I've tried Julia and really liked it, but the user experience was
| pretty bad. The language really needs faster interactivity or a
| strong type checker. I found myself waiting after than compiler a
| lot more than in some AOT compiled language like Rust... I fear
| the language suffers from being overused by people who are
| familiar with Matlab and Python and draws too much inspiration
| from them, much like how Rust draws too much inspiration from
| C++.
| cambalache wrote:
| What boggles my mind is that a language oriented to scientific
| programming has such a lousy time to first plot. I know it has
| been improving, it is still not acceptable. Not for me as a
| user, not acceptable for a language who wants to become
| mainstream.
| oscardssmith wrote:
| It's a better time to first plot than matlab, which is one of
| the other major contenders. On my computer it is about 3
| seconds, which is noticable, but far from disqualifing.
| systemvoltage wrote:
| Julia oversells itself as a general purpose language which
| I find absolutely out of line. Their marketing needs to be
| a lot more humble until they figure out the kinks.
|
| Also, my guess would be Python and not Matlab as it's main
| competitors.
| adgjlsfhk1 wrote:
| Julia has a lot of main competitors. I would consider
| Julia a competitor to Fortran, C, Matlab, R, and python.
| If you look at DifferentialEquations.jl or the clima.jl
| package, these are packages that are competing with low
| level libraries that would traditionally be written in C
| or Fortran. It competes with these by offering comparable
| performance, while having much better quality of life
| features (like automatic PGO, a package system,
| metaprogramming, and not having to deal with make files).
| It competes with matlab by having an incredibly rich
| linear algebra library, while being free and not making
| you do dumb stuff like 1 function per file.
| eigenspace wrote:
| In what way is marketing julia as a general purpose
| programming language 'way out of line'?
|
| People use julia to make webservers, write programming
| languages, create plotting libraries, do scientific
| analysis, do compiler research, make video games, do HPC,
| etc.
|
| Julia has a design that's indeed strongly informed by
| scientific computing, but in order to actually meet the
| needs of the various people using it for scientific and
| technical purposes, it ended up needing to become a
| flexible enough language to be useful for anything.
| ku-man wrote:
| Video games in Julia?, come on. No one is going to make a
| video game in Julia (unless you consider the hang-man or
| sudoku a video game). Julia is already being oversold as
| a scientific programming language, so absurd claims that
| Julia can be used to develop video games are just going
| to damage even more its credibility.
| systemvoltage wrote:
| Do you think R or Matlab is a general purpopse language?
| Sure you could do all these things but should you?
|
| Julia is clearly positioned as a scientific computing
| language. Let's be clear.
| eigenspace wrote:
| Doing these things in julia is very different from doing
| these things in R or Matlab. The tooling and ecosystem
| for non-scientific applications in Julia is growing
| rapidly and is quite competent.
|
| Julia is absolutely a general purpose language. It's user
| base skews heavily towards scientific computing, but the
| demographics and ecosystem are broadening daily.
| systemvoltage wrote:
| Having used Julia for 2+ years, I couldn't disagree more.
| Productionizing Julia code has been a total nightmare.
| The community library support has been growing but hasn't
| gone through the wringer. Just because things are
| improving doesn't provide a meaningful understanding
| against its competitors.
|
| I don't see any reason to use Julia over Go for backend
| webservers. Rust or C++ for systems programming. And
| frankly, I prefer Python for scientific computing.
|
| Julia also has a tiny standard library and lots of flaky
| external libs which make productionization of code a
| risky adventure which I have personally been bitten by.
|
| Most people are allured by Julia's overhyped marketing
| which is a shame because the original paper by Stefan is
| pretty impressive. We're seeing some criticisms of Julia
| in this thread, rightfully so.
|
| My advise to people who are subscribed to Julia's
| marketing is to listen to people that are complaining. No
| one wants to just complain, they're saying that because
| of many reasons. Be humble and try to listen, accept
| Julia's many shortcomings (error messages and stack
| traces, library support, startup time, IDE, debugging,
| etc.). Julia has many shortcomings that are only
| apparently after using it outside of the Jupyter
| Notebooks. Not accepting those makes you an annoying
| fanboy.
| chrispeel wrote:
| > Most people are allured by Julia's overhyped marketing
| ...
|
| Wikipedia says "Marketing refers to activities a company
| undertakes to promote the buying or selling of a product,
| service, or good." Julia is not a company; I think what
| you're calling "marketing" would better be labeled "user
| enthusiasm" :-)
| systemvoltage wrote:
| The founders of Julia _do_ have financial interest
| though! I think they 're part of Julia Foundation. Same
| people wrote the marketing material I presume.
| ChrisRackauckas wrote:
| > The mission of The Julia Foundation is to provide
| assistance to those in need while creating awareness of
| the power of art to heal and inspire.
|
| http://www.thejuliafoundation.org/
|
| I didn't think they were, but now it makes sense why Jeff
| Bezanson's voice is so soothing.
| eigenspace wrote:
| > I don't see any reason to use Julia over Go for backend
| webservers. Rust or C++ for systems programming. And
| frankly, I prefer Python for scientific computing.
|
| Sure, I would never claim Julia is being the best
| language for webservers or systems programming. If
| someone came to me saying they wanted to do this in
| Julia, I'd probably tell them "if this is important, I'd
| probably look at a more established language for this
| purpose unless you have a good reason to want to use
| julia for this"
|
| That doesn't make julia not a general purpose programming
| language. It just means it's not the best language for
| every imaginable purpose (no language is).
|
| I personally prefer Julia very strongly for scientific
| computing to it's competitors, and because of the amount
| of time I've invested in it for that, I also do many
| other things in it and I find it quite nice for this.
|
| It's totally fair that you prefer Python for scientific
| computing. Python has a great ecosystem and huge
| community with tonnes of investment! It's an incredibly
| stiff competitor. I prefer Julia, and think I have strong
| reasons to do so, but everyone's needs and desires and
| different.
|
| > No one wants to just complain
|
| This is an empirical claim about human psychology and
| it's false. But regardless, yes there are a lot of
| totally valid criticisms of julia in this thread! Just
| because these criticisms exist and some of them have good
| points doesn't make julia a bad language though.
|
| Please consider the fact that not everybody has the same
| needs, desires and temperament as you. Every language has
| major probelms with it, but different people feel these
| problems differently.
|
| For many people (for example, me), Julia is a gigantic
| breath of fresh air! For others, it's painful and clunky.
| I think there's a lot of good here that people should see
| and check out and think about, even if they decide it's
| not for them. Especially because these things improve
| every day.
|
| ___________________________________________________
|
| Just a disclaimer in case anyone is suspicious about my
| affiliations: I have absolutely zero financial stake in
| Julia's success. I am not employed by anyone who would
| benefit from more people switching to julia. I'm a
| physics PhD student. I simply find julia very useful and
| pleasant to use and want to share that with others.
| systemvoltage wrote:
| > I'm a physics PhD student. I simply find julia very
| useful and pleasant to use and want to share that with
| others.
|
| User base tends to be scientists and not seasoned
| Software Engineers. No offense to either one, just that
| the community inspires the language and its mechanics.
| This is exactly the reason _it is not a general purpose
| language_. You just proved my point.
|
| Glad you find it useful for your endeavors. I reckon
| DiffEq and other hardcore math is great in Julia.
| sgt101 wrote:
| >Not accepting those makes you an annoying fanboy.
|
| Thanks for the constructive input to the debate.
| systemvoltage wrote:
| I am allowed for a bit of unconstructivism after
| providing a pretty wide take on the productionization of
| Julia code. Spare me? :)
| rightbyte wrote:
| The plot is also very simple. No zoom, brush, save to file,
| label, title, regression etc that you need to process it in a
| GUI and save for a report.
|
| I would really like to use Julia as a "Matlab or Octave but
| with nice string concatenation" but the UI is just lacking
| for one off calculations and data processing.
| enriquto wrote:
| > Octave but with nice string concatenation
|
| As a heavy octave user, I never felt a need to concatenate
| strings in any way. But I'd be happy if julia was an
| _Octave but with fast loops_ , which it sort of is; but
| still not really there.
| rightbyte wrote:
| When you use Matlab/Simulink for c-code generation and
| use it for a lot of purposes Matlab was not really made
| for you might run into processing strings.
|
| But ye for normal use it is not really a problem.
| thebooktocome wrote:
| There are absolutely labels, titles, and the ability to
| save figures to files.
|
| I used to deliver project reports to customers using a
| combination of Julia and LaTeX. It was perfectly suitable
| for that application.
| oxinabox wrote:
| I think they mean no GUI for that. To manually (rather
| than programatically) do this.
| ForHackernews wrote:
| ...it's a programming language, not an Excel competitor.
|
| You could use Julia to build some kind of GUI plot-making
| tool.
| Sukera wrote:
| I'm afraid I don't understand your comment - both Plots.jl
| as well as Makie.jl (the two most commonly used plotting
| packages, as far as I know) support all of those things.
| Makie.jl does so natively, Plots.jl does so if the backend
| (e.g. Plotly, PyPlot) supports interactivity (the rest is
| available by default). Do you mind giving an example, such
| that this could be improved further?
| rightbyte wrote:
| Ye sorry I meant in the plot window (GUI), not the
| scripting. For data processing of lab measurements
| zooming and panning, brushing, "drag to select" etc in an
| easy way is really convenient since you don't know where
| in the plot interesting stuff will be in advance. Adding
| titles and labels, text arrows etc is a nice extra.
|
| Matlab have quite good such capabilities, Octave is more
| limited (you can't add titles, labels, regression lines
| or brush away data points, get simple data statistics
| like sums or std devs like in Matlab, but you can zoom
| and pan, save to file etc).
|
| Julia seems to launch a Qt-window with plot, so adding
| some menu bar with zoom and pan shouldn't add too much
| bloat.
|
| E.g. exploring roots benefits from zooming a lot.
|
| EDIT:
|
| You seem to be able to switch "backend" of Plots to eg.
| PyPlot for some functionality I didn't know that.
| Sukera wrote:
| Yes precisely - it's not the frontend that's doing the
| lifting here. The interactivity as you describe it
| depends on if the backend supports it, which not all do.
|
| I agree that there's room for improvement here, but I
| also think that it won't happen in Plots.jl (and if it
| does, it'll depend on a backend that already provides
| interactivity). I think it'll be more likely to see
| something like this plop out of Makie.jl or something
| built on top of Makie.jl, as that has a lot of primitves
| for interactivity available already.
| Sukera wrote:
| The most important part about the releases since 1.0 is that
| compile time has been significantly reduced, the article
| touches on basically all ways how this has been done - may I
| ask when you've last tried it?
| saiojd wrote:
| A few months ago. First time to plot is noticeably better
| than it was a few versions ago, but still extremely slow
| compared to Python. The article gives a benchmark of 9
| seconds. I mean, come on.
|
| The main problem I had was simply that what any time I needed
| to modify a struct field, or anytime my program crashed, or
| any time the buggy IDE extension crashed, I needed to
| recompile everything. I also haven't found anyone
| particularly interested in basic things like interfaces,
| despite the language supporting type hierarchies (why can't
| we enforce contracts for types? The whole language is built
| around overloading...)
|
| Overall this is very frustrating as the language is excellent
| is many regards, in particular multiple dispatch and the
| compilation model are just great. The "just ahed of time"
| compilation is one of those obvious-in-hindsight ideas IMO,
| better than full interpretation or full compilation for
| nearly all use cases, if only it could be cached between
| interpreter sessions or if you didn't need to restart all the
| time...
| Sukera wrote:
| I know that there are plans to cache even more, but other
| than that I can basically only recommend to put distinct
| projects into proper projects (with a module and
| Project.toml). That will already cache precompiled code for
| that module, even between sessions. Having things in a
| script won't have that benefit. For experimenting with
| struct layouts, I've found that NamedTuples (https://docs.j
| ulialang.org/en/v1/base/base/#Core.NamedTuple) are amazing
| for prototyping, since they can be accessed via A.b just
| like structs but don't have the limitation of being const
| global.
|
| The dynamism and flexibility combined with the compilation
| model is basically what leads down this path of
| recompilation, unfortunately. Since importing packages may
| change behaviour/invalidate some compiled method (that's
| what the SnoopCompile stuff in the article was about), it's
| nontrivial to just begin caching things left and right.
| You'd end up with an exponential explosion in the number of
| methods to cache, wasting huge amounts of disk space.
| That's not to say that there aren't more things that could
| be done, just that it's hard to do so.
| saiojd wrote:
| I've read a bit on type invalidation and I know it's a
| hard problem (in fact it's hard for me to even wrap my
| head around it, lol). Still, it's unfortunate. One thing
| I would like to know is if the difficulties with
| invalidation are a symptom of the dynamic semantics, or
| of the compilation model.
|
| Namedtuples are cool, but I'm not sure I understand the
| tradeoffs between using them and using structs. Can I
| just replace all structs in my project with named tuples,
| without having a performance hit?
| Sukera wrote:
| Note that I didn't suggest replacing structs with
| NamedTuples entirely - only during prototyping, while
| you're figuring out what you want your struct to look
| like. Structs most definitely will be faster.
| oxinabox wrote:
| > Structs most definitely will be faster.
|
| I am not 100% sure this is true.
|
| Structs will definately look cleaner in the code. Not
| sure they will be faster though.
| dTal wrote:
| > The "just ahed of time" compilation is one of those
| obvious-in-hindsight ideas IMO
|
| It's not new. One of the most widely used Lisp
| environments, SBCL, works this way. So does Chez Scheme,
| and therefore now Racket.
| saiojd wrote:
| That's interesting, I didn't know Racket worked like
| that. What I meant wasn't so much that its a new idea,
| rather that it's a good one.
| DNF2 wrote:
| No one interested in interfaces? My impression is that
| interfaces is very commonly discussed, and is one of the
| most anticipated features in the language, though it may
| not come until v2.0.
| saiojd wrote:
| OK, I was being hyperbolic. It's just that the interest
| is low compared to other features, as most people
| involved in the project are used to dynamic languages and
| don't feel the need. It's been in discussion for a long
| time, with no action so far. From what I've seen, the
| general attitude is a bit dismissive of the utility of
| static verification ("its different in our language
| because X" type attitude)
| Certhas wrote:
| My impression with this is that it's mostly coming from a
| few academics that never work on code they haven't
| written themselves, and that have no understanding of
| use-cases other than their own...
| Certhas wrote:
| I feel your pain on interfaces. As it stands, Julia simply
| doesn't encourage carefully thinking/documenting about what
| assumptions your code makes and just banging things
| together, hoping they work. Good luck dealing with any
| obscure MethodErros that result if they don't.
|
| It still, at the end of the day, is a mostly academic
| language. So mostly small projects with very few people
| working on them. No need to architect bigger
| solutions/patterns, etc...
| BadInformatics wrote:
| I'm not sure that's a fair characterization. Core team
| members have expressed serious interest in getting more
| static verification, interfaces and other type goodness
| into the language, but if you try to force the issue it
| turns into a Python 2->3 problem. Not to mention that
| there are few if any examples of how to fit type checking
| alongside multiple dispatch (e.g. C# punts with dynamic).
|
| I personally find static types indispensable when working
| with a large codebase, but to say people don't care about
| this in the Julia community is just not correct.
| Certhas wrote:
| One thing is the Core Team Members and their long term
| plans, which I'm not privy to. The other is the
| impression generated by interacting with others in the
| community online, where people disagree with the very
| premise that there is a problem.
|
| I also see your point that not too much is known in this
| design space, but I also think that's why it would be
| good for the community to step up and experiment with
| this more. Figure out what works. I had plans to do that
| last summer but life intervened so I am stuck
| commentating from the sidelines. :P
| adgjlsfhk1 wrote:
| Note that if you ask on slack, you often will get answers
| from the core team members. They're pretty open. The main
| place where plans aren't the most clear is when there are
| features that everyone knows would be good, but aren't on
| the top of any of the the main people's to-do list. The
| JuliaLang repo has over 1000 people who have contributed,
| so a lot of the time, a new feature is just the result of
| a community member making a random PR. Stack trace
| improvements for example, started as a package, got
| ported to Base, and got improved by a bunch of people
| contributing to design decisions.
| superbcarrot wrote:
| I'm trying really hard to like Julia but constantly chasing
| "ERROR: MethodError: no method matching" messages gets
| frustrating very quickly.
| DNF2 wrote:
| When do you run into this kind of problem? Is it possible
| that you are over-typing your function signatures? The
| generally recommended style is to write generic code with
| loose type restrictions, or even none at all.
| sgt101 wrote:
| >"strong type checker"????
|
| I like julia because of the super powerful and super strong
| type checking. Have I misunderstood what is meant by strong
| type checker?
| aliceryhl wrote:
| What do you even mean? It doesn't even catch that an
| argument's type doesn't match on a function call with types
| specified. It also can't catch trivial stuff like misspelling
| a struct field on a variable of known type.
|
| If you want to be called a "super strong" type checker, you
| really have to catch that kind of simple issue at compile-
| time, _not_ when I run the code.
| adgjlsfhk1 wrote:
| Technically it is catching the type error at compile time
| (but compile time is Just Ahead Of Time). If you want
| something that feels more like type checking in a
| statically compiled language, you should definitely check
| out https://github.com/aviatesk/JET.jl
| sgt101 wrote:
| Ahhhh - you want a statically typed language. That's
| definitely not Julia, you're probably best off with Java
| there.
| kitsune_ wrote:
| It takes 180s to import Statistics, PlutoUI, Images and
| OffsetArrays in Pluto on this Macbook Pro with 32GB that is a
| couple of years old, while the fans are going crazy. If I'm
| unlucky a worker process will seg fault. All in all the
| interactive / REPL part (even with stuff like Revise) is kind of
| a let down thanks to the slow precompilation / lack of caching? I
| like it, but I really don't see how it is ever going to replace
| Matlab, R or Python if these issues aren't addressed.
| Acur wrote:
| Just checked. On my slightly dated laptop the import of these
| packages in Pluto takes 5 seconds with Julia 1.6. In general
| 1.6 is a huge improvements in compilation times and usability
| for me.
| eigenspace wrote:
| I assume that's including precompilation, otherwise something
| is seriously wrong with your computer. Precompilation can take
| a while, but it's cached between sessions, so it should be much
| faster the second time.
|
| Also, in version 1.6, we have multithreaded precompilation
| which happens at installation time instead of when you first
| try to load the package. This makes it _much_ faster.
| ku-man wrote:
| Julia people say "in version 1.6... precompilation... is much
| faster". But they said the same when version 1.2 was
| released, same when 1.3, same with 1.4, and with 1.5, do you
| see a trend here?. It's clear the precompilation issue was
| never solved, otherwise it wouldn't be "fixed" in every
| single release.
| hpcjoe wrote:
| TIL of OhMyREPL. And tried it. Its a keeper.
___________________________________________________________________
(page generated 2021-02-14 23:00 UTC)