[HN Gopher] Xi-Editor Retrospective
___________________________________________________________________
Xi-Editor Retrospective
Author : raphlinus
Score : 207 points
Date : 2020-06-27 19:06 UTC (3 hours ago)
(HTM) web link (raphlinus.github.io)
(TXT) w3m dump (raphlinus.github.io)
| stephc_int13 wrote:
| Too much over-engineering here.
|
| It is always better to start with simple, dumb and reliable code,
| with minimal architecture, and then grow organically.
|
| With experience, I tend to write the less smart code possible, I
| felt in love with the power of brute-forcing everything.
| microcolonel wrote:
| The thing is, that is the story of the existing big editors,
| and we know where that story goes.
| eeereerews wrote:
| No, where does it go? Are the big editors unsuccessful?
| microcolonel wrote:
| They are unscalable.
|
| I love Emacs, and it continues to be my editor of choice,
| but it doesn't seem the design is amenable to adding good
| threading primitives. NeoVIM seems like a success story in
| this regard, though VIMScript is a much less pleasant
| extension language than Elisp, which is not perfect, but is
| at least a real programming language.
|
| There may be a way forward for Emacs, building a sandbox
| that looks like a full Emacs instance to existing Elisp,
| and slowly factoring out the whole-editor blocking issues;
| but that might be almost as complicated as starting fresh,
| with fewer benefits.
| magicalist wrote:
| > _Are the big editors unsuccessful?_
|
| No, but if that's the only criteria, we already have plenty
| of those.
| akhilcacharya wrote:
| Perhaps this is a cultural side effect?
| stephc_int13 wrote:
| You mean working at Google or in the Android team?
|
| I've rarely seen a worse example of absurdly over-engineered
| code than the Android codebase...
| akhilcacharya wrote:
| The latter includes the former, so yes.
| noobermin wrote:
| From the little of what I knew about Xi, I came in with this
| perspective. The article changed my mind. He really was trying
| to "fix all the problems" as it were, it's right there in the
| start of the article the issue with google keyboard not talking
| to text boxes, it's a problem that is real. That said, in his
| opinion, it didn't work especially with the gui being separate
| from the core.
|
| I guess my point is he was trying to "correctly solve" a hard
| problem, which does require engineering. He didn't seem to set
| out to solve a dumb problem that didn't need solving (Yet
| Another Text Editor).
| magicalist wrote:
| > _With experience, I tend to write the less smart code
| possible_
|
| I'm always amused by the hn threads patting ourselves on the
| backs for in depth technical discussions[1], when so often the
| top comments are basically non-specific thought leader tweets.
|
| For those that got this far, the article is well worth reading
| about a design space with difficult tradeoffs, and there are
| comments actually engaging with the content below this.
|
| [1] today brings us multiple instances in
| https://news.ycombinator.com/item?id=23664067
| Firehawke wrote:
| It's disappointing that things didn't work out.
|
| I'd looked at Xi about a year ago when I was finally running out
| of patience with Atom's terrible performance, but the lack of a
| workable Windows solution pushed me to VSCode in the end.
| m0zg wrote:
| This was solving problems nobody really has, so its fate was
| predictable. Here's what I think could take off like a rocket: a
| VSCode-like experience that runs completely in terminal, and
| therefore does not require a "remote" of any kind. Better yet if
| it takes most of the same plugins, to reuse the immense amount of
| work people have done there (and are not going to re-do for some
| hotshot new editor).
|
| Note how none of the above mentions "async", "ropes" or "crdt".
| saagarjha wrote:
| I think it had a lot of goals that matched I'd like from my
| text editor:
|
| * Using native technologies
|
| * Fast
|
| * Modular and extensible
|
| * Open source
| m0zg wrote:
| And I'm not disputing that. But one goal eclipses all others:
|
| * Usable
| saagarjha wrote:
| That is correct, hence why I never used Xi. But if it did
| end up usable I could imagine myself switching to it.
| akhilcacharya wrote:
| I was personally really intrigued by the ambitions and design
| decisions in Xi, but it somehow lacked the one thing I can't live
| without in a non-terminal editor - a tree view, at least when I
| first tried the OSX version.
|
| Perhaps that changed, but you'd think that would take higher
| feature precedence over something like collaborative editing
| right?
| pokstad wrote:
| Is there another modern editor like Xi that strictly separates
| the GUI front end (view) from the backend (model and controller)?
| sgrove wrote:
| I think perhaps libvim is like that?
| https://github.com/onivim/libvim
| raphlinus wrote:
| I believe Onivim is in this camp. I don't know how well that's
| working out for them, or whether there are any others.
| pokstad wrote:
| Thanks, this looks amazing. For others looking for more info:
| https://www.onivim.io/
| noobermin wrote:
| It's saying one thing that you're asking for it when the
| article explicitly says he believes this was a mistake from the
| beginning.
| pokstad wrote:
| That's fine. I don't completely agree with his analysis. Just
| because he couldn't make it work doesn't mean someone else
| can't.
| blub wrote:
| Maybe someone can make it work, but it's such a terrible
| idea that everyone else which doesn't use this oddball
| architecture will be running circles around them.
| taeric wrote:
| The assertion that gpu is required for good text rendering caught
| me off guard. I can't claim it is wrong, but it does feel like it
| should be wrong.
| saagarjha wrote:
| I think the point being made is that it is required for _fast_
| text rendering.
| cjohansson wrote:
| Front-end / interace needs to be synchronously controlled, it
| doesn't feel right otherwise. I followed xi-editor but it never
| was faster than emacs on opening really large files and editing
| them so it never had a use case for me
| saagarjha wrote:
| Aww, this is really disappointing to hear but an excellent read
| nonetheless. I always had Xi in the corner of my eye as a very
| interesting way to make a text editor that had a lot of good
| things going for it: modularity, native UI, speed...I think if it
| worked out it would have been really great. It's really sad to
| hear that it didn't. Thank you, Raph and the rest of the Xi
| contributors, for working on it for all these years; even if it
| didn't upend the text editor market I am appreciative of the
| effort and the lessons you've learned along the way.
|
| Some other side conversations I've had about this that might be
| interesting and relevant:
|
| * Native UI is not always the right choice. That might seem
| surprising coming from someone who pushes native UI extremely
| strongly in general, and I don't want to be misunderstood as
| recommending you choose something else for your next project (if
| you have to ask: don't). But in some very specific contexts, such
| as performant text editing/rendering, it can be problematic to
| use a high-level control. In fact, many of the text views I use
| in are _not_ using the standard platform controls but a custom
| view doing its own drawing: Terminal.app does its own thing (it
| doesn 't even use NSScrollView!), Xcode has the Pegasus editor
| written in Swift, iTerm draws text using Metal, and Sublime Text
| is Skia. Platform controls are meant for 99% of apps that don't
| need absolute control over performance or text rendering, and
| sometimes you can push them quite far. But in some cases it's
| appropriate to abandon them. (I will note that all of the
| examples handle actual text _editing_ very, very well. If you don
| 't know what the standard shortcuts for editing text are for my
| platform, you're not going to do it right.)
|
| * I love that LSP exists and that it makes my "dumb" editor
| "smart". But having worked with it a bit, I think it might suffer
| from some of the same problems: it uses a client-server
| architecture and JSON to communicate, which means it is not all
| that performant and needs to keep track of asynchronous state in
| a coherent way. And not just the state of one document, but the
| states of many documents, all in different contexts and
| workspaces, all with their own compiler flags and completion
| contexts and cursor locations-it is really hard to do this
| correctly and quickly, although many language servers do an
| admirably good job. Perhaps it's just "good enough" that it's
| here to stay, or maybe it has enough support that it can have
| some of its issues wrinkled out with time.
| galonk wrote:
| I want to echo this, it's disappointing, but I'm thankful to
| Raph and the team for all the work they put into it. I had
| dreams of being able to write an editor where I could
| concentrate on making a cool UI in a different
| language/framework and have Xi do all the hard stuff.
|
| I think LSP will end up being the "good enough" solution for
| niche languages, while popular languages get dedicated
| IDEs/mega-plugins, a la the various language flavours of IDEA,
| XCode, etc.
| raphlinus wrote:
| Thanks for the kind words!
|
| I agree with you that getting shortcuts (and other similar
| aspects) right is more important than which code is used to
| implement text editing. We're very much going to try to take
| that to heart as we implement more text capabilities in Druid.
|
| Regarding LSP. Yes, while the functionality is there, it is
| very difficult to implement a really polished user experience
| with it. I think a lot of that is in the design of the protocol
| itself; it's hard to avoid annotating the wrong region when
| there are concurrent edits. It's interesting to think about how
| to do better, but I fear you'd be bringing in a lot of
| complexity, especially if you started doing some form of
| Operational Transformation. I would highly recommend that if
| someone were to take this on, they study xi in detail, both for
| what it got right and what was difficult.
| bhl wrote:
| > And it [syntax highlighting] basically doesn't fit nicely in
| the CRDT model at all, as that requires the ability to resolve
| arbitrarily divergent edits between the different processes
| (imagine that one goes offline for a bit, types a bit, then the
| language server comes back online and applies indentation).
|
| What's a rationale for having syntax highlighting server-side as
| opposed to client-side? I'm working on side project that uses
| ProseMirror and CRDTs through yjs, and the idea of having a
| server for syntax highlighting for editable text never occurred
| to me.
|
| > Even so, xray was a bit of a wake-up call for me. It was
| evidence that the vision I had set out for xi was not quite
| compelling enough that people would want to join forces.
|
| I think in hindsight this was a virtue. If you look at the xray
| project, it died after the Microsoft acquisition of Github:
| https://github.com/atom-archive/xray/issues/177. By not combining
| projects, it allowed for redundancy.
| saagarjha wrote:
| Your server generally has a better idea about the language than
| you do, presumably.
| mwcampbell wrote:
| > There is no such thing as native GUI
|
| I understand why you were disillusioned with the performance of
| native GUIs. But in other areas, such as accessibility,
| nativeness absolutely does help. I'm guessing you know this; I
| just want to make sure other developers get this message.
|
| Here is a quite comprehensive list of the benefits of a native
| GUI for accessibility, particularly on macOS:
|
| https://news.ycombinator.com/item?id=18585443
| microcolonel wrote:
| The accessibility APIs can be programmed against by anyone. If
| there is a way to make something like this accessible to screen
| readers in some useful way, it would likely involve specific
| accessibility work either way.
| saagarjha wrote:
| Platform widgets do a lot of the accessibility work for you.
| In some cases you can get away with doing zero* work.
|
| *Or very close to zero
| mwcampbell wrote:
| > If there is a way to make something like this accessible to
| screen readers in some useful way
|
| If by "something like this" you mean a text editor, then
| there's no question; they can be made accessible with a
| screen reader, and some complex ones (e.g. Visual Studio
| Code, Visual Studio, and Xcode) are accessible already.
|
| > it would likely involve specific accessibility work either
| way.
|
| For the main UI of a full-featured programmer's editor or
| word processor, that's definitely true.
|
| What I want to avoid is developers using a custom GUI toolkit
| where the platform's native toolkit would do, because they
| want the best possible performance (regardless of whether the
| platform's toolkit is good enough), and forfeiting all of the
| accessibility benefits that the platform's native toolkit
| brings more or less automatically.
| raphlinus wrote:
| Very good point, and accessibility is important. We do have it
| on our longer-term roadmap for Druid, and until it's done it's
| a very good reason to consider it unfinished work-in-progress.
| I will make sure to discuss this in my follow-up blog, which is
| about building GUI. Thanks.
| mwcampbell wrote:
| I'm looking forward to that follow-up post, especially the
| part about toolkits that make heavy use of the compositor to
| work around rendering limitations. I also wonder if the
| Chromium rendering engine is doing the same thing, and if
| not, how its performance compares to, say, WinUI and Cocoa.
| chubot wrote:
| Tangent/clarification: the regex slowness issue is almost
| certainly due to using Perl-style exponential time regexes (which
| are required by Sublime syntax definitions) rather than "regular
| languages".
|
| The syntect library mentioned uses fancy-regex, which is
| explicitly different than Rust's default regex crate, in that it
| supports regex constructs that require exponential backtracking:
|
| https://github.com/fancy-regex/fancy-regex
|
| _In particular, it uses backtracking to implement "fancy"
| features such as look-around and backtracking, which are not
| supported in purely NFA-based implementations (exemplified by
| RE2, and implemented in Rust in the regex crate)._
|
| I think "regexes" have gotten bad name so I call them "regular
| languages" now. That is, RE2, rust/regex, and libc are "regular
| language" engines while Perl/Python/PCRE/Java/etc. are "regex"
| engines.
|
| In my experience, lexing with regular languages can beat hand
| written code. In the case of Oil vs. zsh, it's beating hand-
| written C code by an order of magnitude (9-10x).
|
| http://www.oilshell.org/blog/2020/01/parser-benchmarks.html
|
| https://lobste.rs/s/77nu3d/oil_s_parser_is_160x_200x_faster_...
|
| ---
|
| _First, TextMate / Sublime style syntax highlighting is not
| really all that great. It is quite slow, largely because it
| grinds through a lot of regular expressions with captures_
|
| pedantic: I think the issue is probably more lookahead than
| captures ? You can do captures quickly in a "regular language"
| engine.
|
| _It may be surprising just how much slower regex-based
| highlighting is than fast parsers. The library that xi uses,
| syntect, is probably the fastest open source implementation in
| existence (the one in Sublime is faster but not open source).
| Even so, it is approximately 2500 times slower for parsing
| Markdown than pulldown-cmark._
| an_opabinia wrote:
| The bigger problem I feel is with this:
|
| > On the plus side, there is a large and well-curated open
| source collection of syntax definitions
|
| Whenever I tried to get Java highlighting updated in GitHub for
| "var" my issue is closed minutes later by people who don't work
| at GitHub telling me to open an issue with some random unpaid
| TextMate / Sublime repo.
|
| Really the overall problem is not owning the front end. It
| means you can't do things like fix syntax highlighting, at
| least as important to programming as high performance word
| wrap.
| raphlinus wrote:
| First, syntect can use both onig (an adaptation of Ruby regex,
| same as Sublime) and fancy-regex. The latter is yet another xi-
| instigated, and was motivated by trying to do a better job at
| this.
|
| Second, the design of fancy-regex is precisely to delegate to
| the regex crate when features like lookahead aren't needed. I
| had high hopes for the performance, but it turned out to be
| lackluster, a highly tuned backtracking engine beats it in most
| cases. And much of the problem is indeed that RE2-style
| approaches are a lot faster when captures aren't required. And,
| by nature, Sublime-style syntax definitions rely extremely
| heavily on captures. A bit of discussion (including comments
| from burntsushi) here:
|
| https://www.reddit.com/r/rust/comments/5zit0e/regex_captures...
|
| Third, the question of whether "regular languages" would
| support faster parsing is somewhat academic, though perhaps you
| could build something. Though the languages being parsed often
| have some regular features, as a rule there are exceptions.
| Markdown is an extreme example, and cannot be parsed well with
| either regular languages or Perl-style regexes, though that
| hasn't stopped people from trying.
| trishume wrote:
| Author of syntect here: This isn't why
| TextMate/Sublime/VSCode/Atom style regex parsing is slow.
|
| The main reason is that the parsing model is applying a whole
| bunch of unanchored regexes to the unparsed remainder of the
| line one after another until one matches, then starting again
| for the next token. This means each token parsed can require
| dozens of regex matches over the same characters. I implemented
| a bunch of caching schemes to cut down on this number but it
| still tends to need many matches per token. It sounds like
| Oil's lexer probably does about one regex run per token, with
| probably a somewhat faster regex engine, and sure enough it's
| something like 40x faster than syntect.
|
| Oniguruma is actually pretty fast and anyhow most of the
| regexes in Sublime syntax definitions are written to not
| require backtracking, because Sublime has two regex engines and
| only uses the fast one when no backtracking is required. In
| fact fancy-regex should delegate directly to Rust's regex crate
| on these regexes but is somewhat slower than Oniguruma, for
| reasons I haven't yet looked into (edit: see Raph's comment,
| it's probably heavy use of captures, another thing Oil's lexer
| doesn't need).
|
| Also note that byte-serial table-driven lexers have a speed
| limit of one byte per l1 cache round trip (~5 cycles), whereas
| faster lexers can take advantage of multi-byte instructions
| (even SIMD) and out of order execution to go much faster, hence
| why pulldown-cmark is 2500x faster than syntect rather than
| just 40.
|
| [edit: I should also clarify that since unlike these other
| parsers, syntect takes a grammar and can parse many languages,
| so how fast it is depends a lot on the grammar, I suspect the
| markdown grammar in particular is slower than usual, given that
| pulldown-cmark runs about 250MB/s and syntect on ES6 Javascript
| (a complicated but well-implemented grammar) is about 0.5MB/s,
| so the Markdown grammar may be 5 times slower]
| saagarjha wrote:
| This is probably already implemented if it does exist, but I
| know with a bunch of fixed text strings you can create a
| NFA/trie thing using Aho-Corasick. Does such a thing exist
| for regexes (specifically: one that can match "all of them at
| once"), and is it used for the fast regex engine?
| raphlinus wrote:
| You will probably find https://github.com/BurntSushi/aho-
| corasick/blob/master/DESIG... good reading. I believe
| captures get in the way of using the fastest of these NFA-
| style techniques, though. There's a comment from burntsushi
| to this effect in https://lobste.rs/s/fq8uil/aho_corasick
| chubot wrote:
| _The main reason is that the parsing model is applying a
| whole bunch of unanchored regexes to the unparsed remainder
| of the line one after another until one matches, then
| starting again for the next token. This means each token
| parsed can require dozens of regex matches over the same
| characters._
|
| Hm but why can't you just OR them together? That is perfectly
| fine with a regular language engine. For example, I OR
| together about 50 different regexes here (including a whole
| bunch of constant strings) for the ShCommand mode:
|
| https://www.oilshell.org/release/0.8.pre6/source-
| code.wwz/_d...
|
| Despite this big mess, everything "just works", i.e. the
| lexer reads every byte of input just once. re2c picks the
| alternative with longest match, and it picks the first match
| to break ties.
|
| I suspect the reason that you can't do this is (1) Sublime
| was originally implemented with a Perl-style regex engine and
| (2) the order of | clauses matters more when using a
| backtracking engine. It doesn't have the simple rule that an
| automata-based engine has.
|
| My claim is that you avoid performance problems by using a
| "regular language" engine. So I think what you point out
| supports this, even it might be a slightly different issue
| than "more backtracking".
|
| I think you are saying that Sublime's parsing model prevents
| composition by |, which makes lexing slow, because it forces
| you to read each byte many times. (in addition to the
| captures issue, let me think about that)
| ZeroCool2u wrote:
| I've been watching/rooting for Xi from a distance for a few years
| now. This is an excellent retrospective. Well written, but also
| very honest, especially about those all too typical project highs
| and lows.
|
| I remember looking at one of Raph's early presentations and being
| both convinced by his arguments for and impressed with the design
| decisions made, the same ones he outlines here. After doing some
| of my own comparatively rudimentary research, coming to similar
| conclusions as he did. It left me very excited to watch the
| project develop over time.
|
| One thing I was wondering about is that the Xi website contains a
| list of front ends similar to the list in the GitHub repo, but
| includes a Flutter implementation for Fuchsia[1] that is now a
| dead link and missing from the GitHub list. He mentions it
| briefly in this retrospective as well. Any ideas what happened to
| it?
|
| [1]: https://xi-editor.io/frontends.html
| raphlinus wrote:
| It was maintained for a little while after I left Fuchsia
| (almost two years ago now), but the pace of development on
| Fuchsia is such that if it's not actively maintained, it
| bitrots very quickly. And there aren't really any devices
| running Fuchsia that need text editing at scale.
| blub wrote:
| Am I the only one on HN which thought this was a bad idea all
| around from the beginning? I remember that people were very
| enthusiastic back then because of the Rust hype and the author's
| pedigree, so critics were quickly silenced.
|
| For me, picking Rust and the JSON-based IPC were huge red flags
| and this post-mortem confirms that. But what I find odd is that
| many are still not willing to accept the conclusion that using
| Rust for a UI-intensive app is a bad idea. And using multiple
| languages in a project and splitting in back-end vs. front-end or
| lib + UI is also quite popular around here, unjustifiably in my
| opinion when one considers the extra complexity involved.
| GolDDranks wrote:
| I was quite enthusiastic about it, and don't see even now why
| using Rust for a UI-intensive app would be a bad idea in
| general. If doing an app with a "monolithic" design, the lack
| of good frameworks is a problem, but that's improving, so it
| wouldn't be a good idea if you want to see results right now,
| but I'm pretty comfident that's gonna change.
|
| However, I fail to see how the choice of Rust is relevant in
| this particular project, as the UI part was meant to be
| programmed in any language, making Rust an irrelevant part of
| the equation.
|
| On another note, I'm baffled why you are downvoted. I find your
| comment negative, but not unreasonable.
| okatsu wrote:
| I don't have any opinion on Rust but reading about
| microservices, JSON and IPC in the context of a high-
| performance editor did make me raise an eyebrow.
| bhl wrote:
| I wasn't there when xi-editor started, but writing a text
| editor in Rust and splitting the backend from the front-end
| still seems like a good idea. Why? You could compile Rust to
| Web-assembly -- getting a more performant text editor than if
| it was written in Javascript -- and then develop your browser
| or native frontend in HTML/CSS/JS.
| blondin wrote:
| had that feeling...
|
| as a side note, i think more and more people are starting to
| realize that the async paradigm is not a panacea. that paradigm
| or at least the way we implement it might even mean more trouble
| than a synchronous counterpart.
|
| also, other editors that i feel will be sunset or not worked on
| at some point are atom and brackets.
| sq_ wrote:
| Atom already seems to have gone somewhat towards maintenance
| mode since Microsoft bought Github. Feels like there's a lot
| less by way of features being released and a lot more just
| keeping up with ecosystem/OS changes. I could certainly be
| wrong though.
| brunoqc wrote:
| Microsoft probably wants people to use vscode.
| pjmlp wrote:
| And given how Office loves React Native, and Microsoft is
| making it work across all major desktop OSes, with the team
| continuously bashing Electron's bloat on their talks, I
| look forward to the way when RN powers VSCode.
| andrewflnr wrote:
| Atom competes pretty directly against VS Code doesn't it? It
| makes sense that Microsoft would consolidate their efforts.
| sq_ wrote:
| That's what I thought, too. Hopefully the team that was
| working on Atom has gotten/will get a chance to bring some
| of its components/features to VS Code if it doesn't already
| have them.
| bhl wrote:
| There was this one comment when Github Codespaces got
| released that that's where ex-Atom devs got moved to:
| https://news.ycombinator.com/item?id=23093150
| The_rationalist wrote:
| I believe that his async code would have been far more
| maintainable / readable / expressive if it had used a reactive
| streams implementation such as colorless kotlin coroutines +
| flow. Sadly rust doesn't yet have one.
|
| What are your thoughts on this topic @raphlinus ?
| Matthias247 wrote:
| I think this is not about coding style for async code
| (coroutines vs. callbacks), but about general asynchronity
| between various processes/threads.
|
| If you have any asynchronous background task it means the
| state of your application can change without the current
| thread having made any change. And your code needs to account
| for that possibility, which makes it a lot more complicated.
| gw wrote:
| > as a side note, i think more and more people are starting to
| realize that the async paradigm is not a panacea.
|
| This is why libvim (from the oni2 project) is based on vim
| rather than neovim. Even aside from performance, it is a huge
| simplification if you can interact with an editor engine in the
| same process, synchronously. At some point we replaced simple
| function calls with baroque APIs accessed over localhost...
| CyberDildonics wrote:
| I don't think you can blame this on an async design. Right off
| it talks about having the keyboard input as a separate process,
| not knowing what version of the buffer input should apply to,
| using timeouts and I guess, doing everything over local IP. It
| almost reads like a list of what not to do.
|
| Why not put the input into the window, put a version number
| with the buffers and keyboard input and use shared memory? Of
| course something isn't going to work if you don't confront the
| problems you have at a fundamental level.
___________________________________________________________________
(page generated 2020-06-27 23:00 UTC)