[HN Gopher] Everything you ever wanted to know about terminals
___________________________________________________________________
Everything you ever wanted to know about terminals
Author : melissalobos
Score : 108 points
Date : 2022-05-17 20:33 UTC (2 hours ago)
(HTM) web link (xn--rpa.cc)
(TXT) w3m dump (xn--rpa.cc)
| MontyCarloHall wrote:
| Off-topic, but it would be nice if HN could properly parse the
| Punycode encoding of internationalized domain names, so rather
| than the URL appearing as xn--rpa.cc, it appeared properly as
| k.cc
| lupire wrote:
| Unicode domain names are a security threat, due to homographs.
| gibolt wrote:
| Showing the real url, followed by the Punycode url in brackets
| would be even better
| DyslexicAtheist wrote:
| there are tools like dnscrypt-proxy or nfq which allow you to
| sinkhole any punicode dns domains (before DoH became a thing)
| and I'm glad that some browsers still show this as xn-- as it's
| now the only defense that stands between users clicking on
| hxxps://arr[?]e.com instead of https://apple.com
|
| punicode was a terrible idea to allow in DNS.
| MontyCarloHall wrote:
| >punicode was a terrible idea to allow in DNS
|
| This is a very western-centric viewpoint. Most of the world
| does not use the Latin alphabet as its primary writing
| system.
| DyslexicAtheist wrote:
| it is a security centric view point not a "Western" one.
| MontyCarloHall wrote:
| The solution is prohibiting characters that can easily be
| confused (across all writing systems), not banning
| Punycode altogether.
| Beltalowda wrote:
| Or just allow only using a single script. A domain in all
| Cyrillic: great! Mixing Latin & Cyrillic: nono.
|
| In practice, browsers already check for this and display
| the "raw" punycode if they detect mixed script usage, but
| I wish such domains would not be registrable at all.
| These checks are somewhat complex and difficult, and easy
| to get wrong.
| donatj wrote:
| Interestingly it's not even showing in Chrome for me. Shows
| fine in Safari.
|
| I wonder if Chrome considers the upside down "LATIN SMALL
| LETTER TURNED K" potentially deceptive.
|
| They've apparently got a whole document about what Chrome will
| and won't display
|
| https://chromium.googlesource.com/chromium/src/+/main/docs/i...
| ntoskrnl wrote:
| Firefox shows the raw punycode too. If I owned that domain
| I'd be mildly annoyed.
| knorker wrote:
| Yeah this is very very far from "everything". Among many things
| it doesn't speak of ptys.
|
| And terminfo, pty, ctty, process groups, and the rendering width
| of a utf-8 string, and, and and...
|
| > i might be one of a handful of people left on earth who
| actually has the knowledge to
|
| Uh, no. Anyone on BBSes in the 90s is very aware of ANSI, thank
| you. And we've not died off yet.
|
| And honestly it's really not that hard at all. Even on the topic
| of terminals this is not even the hardest aspect of it.
|
| > i have effectively zero pull in the tech community
|
| If you think there's all there's to terminals, yet you act as if
| you know it all, then you shouldn't have any pull.
|
| I would also not accept any pull request that looked even
| remotely like their example code.
|
| Here's a more interesting article about a less understood aspect
| of terminals:
| http://www.linusakesson.net/programming/tty/index.php
| eric4smith wrote:
| Jesus - now I'm afraid to know. But kudos what a wonderful
| resource.
|
| It seems less and less people want to use the terminal.
|
| All the new developers I notice want to stay in the IDE and
| almost never want to get their "hands dirty".
|
| Thanks for writing that great resource.
| ntoskrnl wrote:
| I was working on terminal code recently and had a hell of a time
| finding anything approaching an authoritative spec or reference
| docs. The best I found was this: https://invisible-
| island.net/xterm/ctlseqs/ctlseqs.html
|
| I'd recommend that over the wikipedia link in the article, since
| wikipedia seems to be missing some things.
| Beltalowda wrote:
| That document is very good, but it mostly describes control
| sequences _as interpreted by xterm_ , with some notes for other
| terminal emulators, too. Some of the more advances ones will
| have different sequences in other terminals (for example for
| the mouse). Most basic ones (i.e. ANSI stuff) works in pretty
| much all terminal emulators of the last few decades though.
|
| Either way, I wouldn't use it as an authoritative source
| uncritically.
| hnlmorg wrote:
| You couldn't find an authoritative spec because there isn't a
| universal standard. Lots of terminal emulators differ in
| different ways.
| metroholografix wrote:
| https://www.ecma-international.org/publications-and-
| standard...
| hnlmorg wrote:
| Not all terminal emulators follow that completely. Most
| only include a subset and some include their own
| proprietary escape codes too (like iTerm and Terminology
| have codes for rendering images. tmux has an escape code
| for changing the session title).
|
| Then there's other specs not included in that doc even
| outside of the aforementioned proprietary codes. Like
| Sixel, conventions on non-POSIX terminals, etc. Also lets
| not forget the popular-but-not-standardised conventions
| like the hyperlink escape codes (which I personally think
| shouldn't exist in the first place....but that's another
| topic entirely).
|
| Even standard ASCII characters can differ from one platform
| to another. Backspace being a classic example: ISO 646
| describe it as ^H whereas ASCII 1963 has it as ^?
|
| This mess of differing compatibilities is exactly why
| termcap is a thing.
|
| Being an author of a readline library, this is a topic
| quite close to my heart :)
| shric wrote:
| This will be obvious to any C programmer, but the macro[1] used
| throughout the article only "works" on string literals and
| arrays, not pointers.
|
| Also, it will include the null terminator. Probably won't do any
| harm but quite silly if you're redirecting to a file for example.
| I'd subtract one, or use strlen which would cover the pointer
| case above and I'd hope a modern compiler would elide the call on
| a string literal anyway.
|
| [1] #define szstr(str) str,sizeof(str)
| sophacles wrote:
| The author seems not to be aware of the recent TUI
| rennaisance[1]. There are libraries like termbox and blessings
| (python) that are a middle-ground between full ncurses and adding
| your own ansi codes. There are a lot of modern TUI frameworks
| like tui-go or tui-rs that bring common GUI conventions back to
| the TUI (heck there are TUI programming libraries that are
| designed to be similar to react) - these too tend to be a lot
| nicer than working with ncurses.
|
| Definitely worth checking out the full landscape these days if
| you're going to dive into making your console programs prettier.
|
| [1] My words, I just coined that name (although I wouldn't be
| surprised if others had said it too). I mean that in the last
| ~decade there has been a lot of TUI work in the background, with
| lots of new programs and some pretty stunning results. I blame
| unicode - once the web folks realized they could use something
| like font-awesome instead of sprite-sheets (or maybe as a pre-
| build sprite sheet?), and the "icons in a font" movement took
| hold, terminal programs got a lot prettier too. It makes sense,
| its the same font/font renderer provided by the system no matter
| if the app is a browser or terminal emulator.
| csdvrx wrote:
| > but i swear to god developers have so completely forgotten how
| terminals work that i might be one of a handful of people left on
| earth who actually has the knowledge to, so they all just layer
| their bullshit on top of ncurses (which should never have
| survived the '90s) instead and it's maddening.
|
| Actually, yes, understanding the tty in detail seems to become a
| dark art.
|
| However it's the best way to do complex things quickly: I did use
| some of these tricks like storing and then restoring the cursor
| position to have the time at which a command stop executing ABOVE
| the command itself and next to the time it started executing in
| https://github.com/csdvrx/bash-timestamping-sqlite
|
| I had to, because I was using MSYS2 and the time to execute a
| command was a limiting factor in Windows before WSL2.
| melissalobos wrote:
| It looks like you have done a lot of work with sixels based on
| your github too. Have you ever written anything about that?
| csdvrx wrote:
| > Have you ever written anything about that?
|
| No, WYSIWYG: I don't bother much more than that with
| explaining how it's done: the source code + the comments +
| the github page already give all the details away to whoever
| wants to dig deeper.
|
| I'm not much into social media or self promotion either: if
| people like what I do, they'll use it - I don't care much
| more than that, as everything was written for myself and my
| selfish needs first.
|
| The tty/sixel world is very small world anyway, we generally
| know and recognize eachother, so we know where to look for
| cool new stuff :)
| cek wrote:
| I built a cross platform app to print 'pretty formatted' source
| code [1]. I didn't want to re-invent the wheel on formatting
| source code, so looked at all the existing libraries. Originally
| I figured formatting to HTML, and then building a print-friendly
| HTML render would work. But this proved super challenging. I
| tried a dozen HTML engines (including Chromium) but none gave me
| enough control to render just a single page of the original
| source file.
|
| Then I noticed Pygments, a Python-based library for pretty
| formatting source code, has an option to output an ANSI formatted
| file. I quickly found a bunch of libraries that could render ANSI
| formatted text to a print canvas.
|
| In the end, I put the original source code file through
| 'pygmentize -16m -o tempfile.an` (`16m` is the 16M color terminal
| ANSI formatter) and pipe the `tempfile.an` through a print-
| optimized renderer to actually print the source code.
|
| ANSI escapes FTW!
|
| [1] WinPrint - https://github.com/tig/winprint [2]
| https://pygments.org/
| Tr3nton wrote:
| I'm sure this is very interesting but I couldn't make it past the
| first sentence. I'm sure there's a term for someone who writes
| like this, but all I could think was "redditspeak," or otherwise
| stale tryhard wacky. Forcibly inserting so much 'character' into
| your sentences that your syntax implodes.
| whartung wrote:
| You can't have "everything you wanted to know about terminals"
| with absolutely no mention of terminfo and (previously) termcap.
|
| You don't necessarily need ncurses to have a portable smart
| terminal experience, but you do need terminfo.
|
| ncurses magic relies on the lore stored in terminfo, which
| contains all of the obscure escape sequences and other
| information about the disparate world of terminals.
|
| They're maintained as a combined package, but you don't need
| ncurses to drive the screen. You can get the codes yourself if
| that's what you want to do. You can use tput to make colorful
| labels in shell scripts.
|
| While much of the world has moved on to the One Terminal running
| on the One OS, not everyone has.
| em500 wrote:
| TFA does mention terminfo at the very end:
|
| > being compatible only with ANSI-capable terminals is a
| feature, not a bug, go the fuck away. terminfo is a fucking
| joke. nobody needs to target obscure dumb terminals (or smart
| terminals, for that matter) from 1983 anymore.
|
| I don't have the expertise to opinion on this. But AFAICT most
| terminals that people still use[1] will self-report as xterm or
| xterm-256color so they'll only use the xterm escape sequences
| anyway.
|
| [1] kitty is a notable exception. But it's rarely supported on
| preinstalled terminfo's, and manually adding them to all the
| remote servers that you access is a PITA.
| russellendicott wrote:
| I've found that tcell works pretty well across many different
| OS terminals.
|
| https://github.com/gdamore/tcell
| kazinator wrote:
| There is no need for the complexity of terminfo or termcap,
| because we have an ECMA/ANSI/ISO standard for terminal control.
| We have had it since _before_ terminfo. ECMA-48 dates back to
| 1976. People have had 46 years to upgrade to standard-
| conforming terminals.
|
| Terminfo is not a _de jure_ standard, only _de facto_. It is
| not in POSIX.
|
| POSIX specifies a _tput_ command with exactly three operations:
| clear, init and reset.
|
| Right here:
| https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t...
|
| There is a Rationale section giving reasons for why terminfo
| wasn't standardized.
|
| Bottom line: ECMA/ANSI is the real standard that can be used to
| drive a conforming terminal from any OS, without any special
| library other than basic I/O.
| JoshTriplett wrote:
| As the article states:
|
| > being compatible only with ANSI-capable terminals is a
| feature, not a bug
|
| Driving old hardware terminals (as opposed to terminal
| emulators) is a fun stunt, not something of actual modern
| value. _Every_ modern terminal emulator supports ANSI escape
| sequences. Some features are supported by a subset of
| terminals, but you can either 1) probe for those features by
| asking the terminal, which some terminals support, or 2) try
| them and have graceful degradation if they 're not supported,
| or 3) have configuration options to use them, or 4) don't use
| them.
|
| For every one case in which terminfo allows you to support some
| obscure non-ANSI terminal, there are many _many_ more cases
| where terminfo won 't happen to have a definition of the user's
| terminal (or won't know all the capabilities of the user's
| terminal) and you'll have _less_ functionality than if you just
| used ANSI escapes. This is especially true over SSH and
| similar.
| nicoburns wrote:
| > Every modern terminal emulator supports ANSI escape
| sequences
|
| Is this true on Windows? My understanding is that quite a few
| TUI libraries don't have windows support.
| [deleted]
| ghostpepper wrote:
| off topic but I would love a version of this with a more "plain"
| colors/ punctuation/formatting. finding it very hard to read /
| navigate
|
| anyway the content looks good so will try to find time to sit
| down and read this "essay" soon
| mmanfrin wrote:
| The colors are very distracting on my monitor.
| nohackeratall wrote:
| Some actual visual examples would've been nice, too. There is
| just text that describes what it would look like but, as we all
| know, one picture is worth a thousand words.
| [deleted]
| jl6 wrote:
| The author seems to really not like ncurses. But why? Is there
| something wrong with it?
| lupire wrote:
| The article has no pictures of what it is selling, so it just
| seems like a rant, as suggested by the opening profanity. I
| don't see how I can use what they are saying to make a TUI.
| giraffe_lady wrote:
| Are you trying? I've been brutally bashing my head against
| some ancient telnet shit recently and this actually was
| concretely helpful and I could apply it immediately.
|
| The tone and presentation seem calculated to annoy HN readers
| (nice) but this article has a bunch of difficult-to-uncover
| details with just enough context to apply them if you
| actually have a use for them and a desire to.
| timw4mail wrote:
| I could take this a lot more seriously if capital letters were
| properly used.
| kompatible wrote:
| Isn't worrying about someone else's grammatical use of
| capitalization when the piece is understood anyway a little too
| pedantic?
| timw4mail wrote:
| I find the text more awkward to read without capital letters
| starting sentences, especially in paragraph form.
| vo2maxer wrote:
| I can't agree more. An author who can't be bothered to
| follow well established forms of style, grammar, and
| punctuation in a technical article somehow deflates
| whatever else they have to say even if accurate. It's not
| as if they are trying to emulate the works of E. E.
| Cummings, James Joyce, or Arno Schmidt.
| giraffe_lady wrote:
| y'all are the most boring people on the planet istg. in
| the entire legion of internet writers making in depth
| technical content there is what, _one_ who deviates from
| the conventions and you can 't handle it.
|
| This author is a much much better writer than average for
| free technical content! They have an interesting, unique
| style! The typography and punctuation are part of that
| style! It would be _tangibly worse_ if they adhered to
| the (bad, and also arbitrary!) typographical conventions
| of raw html just to please a bunch of square-ass nerds.
| Shit just makes me sad seriously.
| ntoskrnl wrote:
| > y'all are the most boring people on the planet
|
| Citation needed. Please cite your source and make sure
| you strictly follow MLA format
| glouwbug wrote:
| Best part is, every programmer has their own unique style
| as well. It doesn't stop us from running clang-format
| though
| giraffe_lady wrote:
| [deleted]
| lupire wrote:
| Why? It's clearly not meant to be taken seriously.
| billti wrote:
| I was actually thinking the opposite. I got quite a way through
| the article before I realized "Hang on - None of the paragraphs
| and sentences start with uppercase letters!"
|
| As a father trying to teach two small children to read and
| write, I've begun to think it's kind of annoying and
| unnecessary to have two different forms. It's easy enough to
| teach when they are just bigger glyphs, e.g. O -> o, C -> c, S
| -> s, etc. But having uppercase and lowercase that are entirely
| different is just a pain in the ass, e.g. A -> a, D -> d, G ->
| g, R -> r, etc.
|
| I do wonder if literacy might be improved if we got rid of
| uppercase, to little to no detriment elsewhere. (I'm sure
| someone here can probably sight some data on that!)
| broses wrote:
| A while ago I was trying to find a way to make my terminal scroll
| back up after a command executed, so that if the output was long
| I could read it from the top without having to scroll up
| manually. There are ways to get yours shell to print something
| after the command executes, so I just needed to find an ansi
| escape sequence that would scroll up. Unfortunately I didn't see
| any sequences that do this. Anyone have any ideas?
| bowmessage wrote:
| Maybe pipe output of every command into a script that invokes
| $PAGER when the output is long enough?
| rustyminnow wrote:
| less has a flag for that: -F or --quit-
| if-one-screen Causes less to automatically exit
| if the entire file can be displayed on the first
| screen.
|
| So one could just pipe everything through `less -F`. If a
| different $PAGER is preferred, well that'll be a little more
| involved.
| hnlmorg wrote:
| Once it's scrolled off the top of the screen you're basically
| at the mercy of your terminal emulator's scrollback history.
| Some might have an escape sequence available to recall it but I
| there isn't any standard way of doing it.
|
| You'd be better off piping into less / more / most. These are
| called "pagers" and are designed to do this. eg
| cat large-file | grep common-phrase | less
| jjoonathan wrote:
| I was aware of the ANSI escape sequences and even used them
| directly in scripts on occasion, but I still used ncurses "where
| it mattered" because I didn't know about compatibility. I didn't
| want to risk Windows or a random flavor of linux I'd never heard
| of or a group of anti-VT100 enthusiasts getting upset because I
| didn't use an agreed upon compatibility layer.
|
| From the tone of this piece I gather that the ANSI escape codes
| are actually standard enough to target. Cool! Thanks for the
| heads up.
| Beltalowda wrote:
| > From the tone of this piece I gather that the ANSI escape
| codes are actually standard enough to target.
|
| termfo[1] comes with a "termfo" CLI utility which - among other
| things - can group terminals by escape code; for example
| "termfo find-cap save_cursor" shows that almost all terminals
| use "\x1b7", with just a few very old ones using something
| different (full output is a bit long, but it's at [2]).
|
| It's useful to check "can I safely hard-code this escape code?"
| But like you said: for ANSI it's pretty safe to just hard-code
| most codes, especially the common ones, but never hurts to
| check.
|
| [1]: https://github.com/arp242/termfo
|
| [2]: https://pastebin.com/raw/pVHGR6aZ
___________________________________________________________________
(page generated 2022-05-17 23:00 UTC)