[HN Gopher] How does apt render its fancy progress bar?
___________________________________________________________________
How does apt render its fancy progress bar?
Author : julienpalard
Score : 460 points
Date : 2021-10-13 09:24 UTC (13 hours ago)
(HTM) web link (mdk.fr)
(TXT) w3m dump (mdk.fr)
| WayToDoor wrote:
| Nitpick about the website, but please don't use PGP short
| fingerprints like 0x46EBCD72F08E6717. They can be brute forced
| [0] very fast to get a different private key with the same
| fingerprint. You can also use them to generate keys with vanity
| fingerprints, which is pretty funny [1].
|
| [0] https://github.com/lachesis/scallion [1] I have
| 0xDE4444AAAAAAAAAA, took me 5 minutes of bruteforcing on a
| laptop.
| rhn_mk1 wrote:
| What fingerprints should be used?
| ninjin wrote:
| According to this guide (outdated though, but this still
| applies) [1], `--with-fingerprint`, as even the long ID can
| be spoofed.
|
| [1]: https://riseup.net/en/security/message-
| security/openpgp/best...
| [deleted]
| jwilk wrote:
| How come? The fastest GPU in scallion's README has hash rate
| 11646 MH/s. To brute-force a 64-bit key ID at this rate you
| would need, on average, 50 GPU-years.
| arsome wrote:
| 50 GPU-years is fairly cheap, certainly within the realm of
| days for a 3 letter agency or a botnet.
| someonewhocar3s wrote:
| Kind of pushing it if you want to use it for regular cases
| though!
|
| Funny to think in GPU-years given more's law. Payloads like
| this (stochastic) have an expected time, not an actual
| time. It's pretty rare that you can buy 'enough processing
| power', usually have to go through figuring out and perhaps
| adjusting the payload to the computational platform
| yourself. (computing market is not per ops, it's per actual
| minute of computer time - rare to see it offered with
| 'compute this before')
| p-e-w wrote:
| To brute-force a _specific_ 64-bit ID, sure. But if your only
| requirement is that the ID consist of two random characters,
| then a run of one character, then a run of another character,
| the time can be cut dramatically, since there are tens of
| thousands of 8-byte sequences that fit this pattern, and you
| can simply stop as soon as you find one of them.
| SavantIdiot wrote:
| I wouldn't call it fancy: I can't change the color so on light
| backgrounds I can't read it. And I've seen certain error messages
| slice it in two.
|
| I'd call it annoying.
| Snetry wrote:
| Neat.
|
| This is one of those things I always wondered about but never
| cared enough to actually go and look into myself
| julienpalard wrote:
| You can't imagine how many times I wondered about it before
| testing it and writing the article ;)
| someonewhocar3s wrote:
| Crazy idea for apt:
|
| Concurrency
|
| Honestly tears to the fact that it doesn't multi-source and many-
| connection. There's something to be said about "it doesn't
| improve server specs" or maybe even "I don't like the idea of
| having too many connections" - it's just known it does speed
| stuff along.
|
| Basically making a lot of people sit and wait.
| miki123211 wrote:
| This kind of fancy terminal manipulation is usually an
| accessibility nightmare.
|
| Real GUI frameworks usually provide semantic information to the
| OS, which allows screen readers to determine that a given control
| is a progress bar that is 50% filled, and then render that
| information by playing an appropriately-pitched tone. In a TUI,
| all they get is a line of fancy symbols changing quickly, with no
| context as to what those symbols mean. Same for i.e. ncurses
| based menus. When programs use colors to signify which menu
| option is selected, screen readers usually get lost.
|
| Screen readers specifically designed for text-based systems (such
| as DOS) usually had ways to work around this issue, but modern,
| GUI-focused ones don't really offer those options any more.
| darkwater wrote:
| In apt at least the fancy UI can be disabled completely, see
| 'Dpkg::Progress-Fancy "0";'
| d_watt wrote:
| Out of curiosity, when you have a large amount of text flying
| by like with package installs, what is the expected
| accessibility behavior? I imagine a screenreader trying to keep
| up with the install logs in any way would be... chaotic.
| miki123211 wrote:
| It tries to read everything, but you can flush it with
| control. If you notice it stopped speaking, you usually use
| some review commands to look at the last few lines. If you're
| looking for something specific, you'd usually use something
| like grep, even in situations where a sighted person could
| just glance at the screen.
| newswasboring wrote:
| might sound a bit insensitive. But is there a video I can
| see of this? I cant even imagine how this would look in the
| hands of an expert.
| miki123211 wrote:
| No, not to my knowledge. I could probably find a podcast
| or two, there are quite a few of those aimed at a blind
| audience, but that's about it.
| bhaak wrote:
| Not exactly what you asked for but the "Designing for the
| Visually Impaired" episode of the Roguelike Radio podcast
| has a blind player play a roguelike. The intro has a bit
| of screenreading but the gameplay is starting at around
| 1:15.
|
| While listening, you should put on headphones and
| concentrate very hard. It's quite fast and other sensory
| input might be disturbing so also close your eyes.
|
| http://www.roguelikeradio.com/2012/10/episode-48-designin
| g-f...
| lzybkr wrote:
| Saqib Shaikh, a blind developer at Microsoft, gave this
| short talk on using Visual Studio:
| https://www.youtube.com/watch?v=94swlF55tVc
|
| I realize this isn't a console, but I think it is a good
| demonstration of an expert doing typical developer
| activities.
| lelandfe wrote:
| A good point and one I hadn't considered before (shame on me).
| Does anyone have a good reference on making accessible terminal
| programs?
|
| As a web developer I'm intimately aware that there are many
| gotchas and nuances to be aware of - I have to figure the same
| is true of the terminal.
| miki123211 wrote:
| If it works correctly on an old-school TTY with a printer and
| a roll of paper, you're pretty much done. That's how you
| should view a terminal screen reader, except it talks instead
| of printing.
|
| Using it with a non-monospaced font would probably be a good
| test too. Monospace makes some implicit assumptions (i.e.
| being able to see how things are aligned) which screen
| readers don't follow. Also avoid preceding important messages
| with long strings of text, in particular containing numbers,
| i.e. overly detailed timestamps in logs. A screen reader
| reads text line by line, so those usually make using the app
| less efficient.
|
| If you want an actual user interface, not merely a command
| prompt, avoid the terminal like the plague. Exposing a simple
| web frontend might be a good idea here.
| Cthulhu_ wrote:
| All terminal applications should have a 'basic' (or to use Git
| terminology, 'porcelain') output mode by default, and an
| alternative renderer for more visually inclined people as an
| option.
|
| Most recently, Yarn 3 went way overboard with their installer,
| to the point where it's... visually overwhelming? Tree views,
| colors, dotted underlines, multiple scrolling segments, etc.
| It's a bit better if running in CI mode (no coloring), but
| still not ideal.
| 908B64B197 wrote:
| > All terminal applications should have a 'basic' (or to use
| Git terminology, 'porcelain') output mode by default, and an
| alternative renderer for more visually inclined people as an
| option.
|
| Also makes it easier to integrate them to scripts/pipelines.
| [deleted]
| TeMPOraL wrote:
| On 'porcelain', the way Git uses that term is a pet peeve of
| mine. On the one hand, the intended meaning of it is that
| "porcelain" commands are _meant for humans, not for other
| programs_ - scripts should use the "plumbing" commands
| (apparently, it's a toilet analogy). On the other hand, the
| flag that makes "porcelain" commands write machine-readable
| output is called... "--porcelain". Implying it does the
| opposite of what it does.
| xyzzy_plugh wrote:
| It makes more sense if you look at the whole option.
| ... --porcelain[=<version>]
|
| The idea is to present the _porcelain_ at a _specific
| version_. So really you should read it as "give me the
| porcelain (for v1)", e.g.: git status
| --porcelain=v1
|
| If it was an environment variable instead it would be
| GIT_PORCELAIN_VERSION
| geofft wrote:
| But that's not what it does. Try it. $
| git status On branch master No
| commits yet Untracked files: (use
| "git add <file>..." to include in what will be committed)
| foo nothing added to commit but untracked
| files present (use "git add" to track) $ git
| status --porcelain=v1 ?? foo $ git status
| --porcelain=v2 ? foo $ git status
| --porcelain=v3 fatal: unsupported porcelain
| version 'v3'
|
| No --porcelain option gives you the output in the format
| of the git status porcelain, either at the current
| version or at Git 1.x or whatever, i.e., "give me the
| porcelain" is the wrong reading of the option.
|
| The correct reading is "I am implementing a porcelain
| myself, give me a format I can parse in order to do
| that."
| avgcorrection wrote:
| It's funny how wrong this is.[1] No, `--porcelain` is
| machine output format, and the machine output format is
| (sensibly) versioned (you of course need to version such
| output--the output of `git status` writ large is not
| versioned afaik).
|
| This [2] answer by VonC even quotes a mail where someone
| said that it was "[their] fault, to some degree."
|
| [1] Because it proves the opposite point.
|
| [2] https://stackoverflow.com/a/6978402/1725151
| bayindirh wrote:
| Wasn't machine readable output is obtained via --terse
| historically? Some toolkits we use provide --terse flag to
| dump output in a pipe friendly manner.
| wsc981 wrote:
| _> ... scripts should use the "plumbing" commands
| (apparently, it's a toilet analogy) ..._
|
| Ah this term finally makes sense to me, since it's meant to
| deal with human output.
| yissp wrote:
| Gah, I'm beginning to think the git CLI is trying to be
| confusing on purpose.
| willismichael wrote:
| https://stevelosh.com/blog/2013/04/git-koans/
| blacktriangle wrote:
| Is the zen of git to stop using git? Somewhat inspired by
| OpenBSD I spun up my most recent side project with cvs.
| I'm amazed how much less stressful code management has
| become. Makes me want to go further back in time and
| learn how to properly use rcs.
| willismichael wrote:
| I think the author of that blog post prefers Mercurial.
| stormbrew wrote:
| Now try a merge.
| blacktriangle wrote:
| I've done a few, it's been fine. If you try and use cvs
| like git sure you're going to get burned. There's a
| reason Linux was using Bitkeeper and not cvs after all.
| But if your project organization works well with cvs, you
| can seriously avoid the massive wad of incidental
| complexity that is git.
| stormbrew wrote:
| I mean, when I was using cvs because cvs was the only
| thing I could use I don't think I was trying to use it
| like git and I was still pretty happy when better things
| came along. Subversion and perforce in particular
| improved merging massively long before git came onto the
| scene.
|
| CVS was fine for small projects with a couple devs at
| most. Anything more than that and you needed someone on
| your project with far more domain knowledge in cvs than
| git requires now to avoid losing all your data in a
| botched operation on the thousands of independent rcs
| files a repo consisted of, so I really can't agree about
| avoiding massive wads of incidental complexity.
|
| Git is, if anything, far _simpler_ than cvs in just about
| every way imo. I could see myself using subversion for a
| lark, but I 'm very happy if I never have to touch cvs
| again.
| peanut_worm wrote:
| > The novice thought for a few moments, then asked:
| "Surely some of these could be made more consistent, so
| as to be easier to remember in the heat of coding?"
|
| > Master Git snapped his fingers. A hobgoblin entered the
| room and ate the novice alive. In the afterlife, the
| novice was enlightened
| lmilcin wrote:
| Better yet, OS-wide flag to force most compatible output
| format. No need to rob everybody of fancy graphics.
| csmpltn wrote:
| The terminal is a visual tool. That makes it inaccessible,
| almost by definition. Terminals are text-based, but they can
| still be used to convey non-textual information (ie. progress
| bars, graphs, etc).
|
| There's no sense in expecting a visual application (such as the
| terminal) to be somehow accessible out-of-the-box.
|
| Trying to automatically translate a visual interface to an
| accessible interface is bound to be error prone and not work in
| all cases. What's accessible for a blind person, may not be
| accessible to a deaf person.
|
| Applications should share the core business logic, but have
| entirely seperate "frontends" for every situation. A graphical
| frontend, and an accessible frontend. We need an open cross-
| platform standard for this, and enforcement.
| ClumsyPilot wrote:
| "Applications should share the core business logic, but have
| entirely seperate "frontends" for every situation."
|
| I think its clear from your post that you have never dealt
| with the visually impaired or even explored accessebility
| options of iOS/MacOS/Windows. We have working accessebility
| for all applications throughfully designed with native OS
| controls.
|
| Having separate front ends will never happen, most businesse
| sproduce a single bloated JS app that crashes equally on all
| platforms.
| jhgb wrote:
| > I think its clear from your post that you have never
| dealt with the visually impaired or even explored
| accessebility options of iOS/MacOS/Windows. We have working
| accessebility for all applications throughfully designed
| with native OS controls.
|
| ...which is a simple consequence of Windows people not
| being used to polyvalent software. You're putting horse
| before the cart.
| csmpltn wrote:
| > "We have working accessebility for all applications
| throughfully designed with native OS controls."
|
| The comment I'm responding to says that progress bars in
| the terminals, and that TUIs in general are an
| accessibility nightmare.
|
| Are you disagreeing? Are you claiming that it's a solved
| problem? Would you mind elaborating on what you think can
| be done to solve this?
| nightpool wrote:
| You're misinterpreting the GP's post, TUIs are not
| "thoughtfully designed with native OS controls".
| csmpltn wrote:
| No, that's exactly my point.
|
| The bulk majority of applications people use today are
| not "thoughtfully designed with native OS controls".
| Sprinkling some ARIA labels around is better than
| nothing, but the experience itself is extremely limited
| and is not on-par with what those people deserve.
|
| I'm talking about Javascript/CSS-heavy websites in a web-
| browser, computer games, or software such as
| Blender/SketchUp/Google Earth to name a few. ARIA labels
| aren't good enough here - we would need to develop an
| entirely different interface to accomodate for each
| individual accessibility scenario. You can imagine a
| blind person and a deaf person using entirely different
| versions of Blender, each built with a different
| interface accomodating a specific disability.
| miki123211 wrote:
| This argument has been brought up in the accessibility
| community multiple times, and it's not as straightforward as
| you might believe.
|
| The problem with accessibility-focused frontends is that they
| lack in features. There's a much smaller number of users able
| to detect bugs and a much smaller number of developers
| willing to fix them and add new features whenever they appear
| in the mainstream app. This works pretty well for i.e.
| accessibility-first Twitter clients, but it probably wouldn't
| for your local utility company that serves ten or so blind
| customers, none of whom are programmers.
| csmpltn wrote:
| > "The problem with accessibility-focused frontends is that
| they lack in features."
|
| Isn't that a consequence of building different tools that
| fit people with different situations? We do exactly this
| with physical tools and constructions: we build a
| wheelchair-friendly ramp alongside a traditional stairway.
| It's going to be a trade-off either way, right?
|
| If we "choose" (I fully understand this is not an actual
| choice done by anybody) to stick to tools which try to
| auto-magically translate visual-first frontends into
| accessible frontends - we must accept that there will be
| things that mess with that translation process and cause it
| to fail. It's unrealistic to expect a translation that
| works 100% of the time, reliably. Sometimes it's a minor
| technicality that can be worked around - but other times
| it's because data presented visually in a certain way,
| can't always be translated to something else (ie. a
| heatmap, a 3D multiplayer game, etc). How do we make the
| Mona Lisa accessible to a blind person? On that same
| strength - should TUIs not be allowed to have progress bars
| or other visual elements?
| miki123211 wrote:
| The solution here is to augment our tools with the
| information a screen reader needs.
|
| To follow your example, we don't build separate bank
| branches for wheelchair users, we just extend existing
| branches with ramps, elevators and anything else they
| might need.
|
| We already do this with other kinds of user interfaces.
| For example, web pages can be annotated with ARIA
| attributes, which don't influence how a page looks, but
| tell the screen reader that something is a checkbox which
| is currently unchecked. This is only required if you
| don't use the native HTML checkbox control, of course.
| Other platforms have their own ways of doing this, see
| iAccessible and UIA on Windows, ATSPI on Linux and
| AXElement on the Mac, for example.
|
| Terminals were never meant for use cases like this, look
| at how hacky tty control codes are. If not for the fact
| that they're basically a bunch of ugly cludges that
| somehow work, there would probably be an aPI for that
| too. For now, though, we have to live with what we have.
| csmpltn wrote:
| > "The solution here is to augment our tools with the
| information a screen reader needs."
|
| So we are in agreement that there should be different
| interfaces that suit each experience, we just disagree on
| the exact implementation details. I submit it's best to
| build entirely seperate experiences, but you submit it's
| enough to take the graphical-first experience and
| annotate it enough so that a screen-reader can generate
| an equivalent experience on the fly (using different
| kinds of annotation technologies). My response to this
| is:
|
| 1. Annotations and metadata (ARIA-labels, et-al) make it
| easier for the screen-reader to display relevant
| information in an accessible manner - but they create an
| unnecessary coupling between the visual-first frontend
| and the accessible frontend, when in reality they are
| built for different kinds of users.
|
| 2. Annotations are a decent starting point, but they are
| NOT a substitute for building an "accessibility-first"
| experience because they are too limited. You can't
| annotate a graph, or a progress bar, for example. But you
| could've built an entirely seperate experience which
| conveys the same data a graph would, in an accessible
| manner (given the right tools and frameworks).
| madeofpalk wrote:
| On the other hand if we know something will break one use
| case totally, to provide marginal utility in another,
| then it's probably not a good trade off.
| jdavis703 wrote:
| > but it probably wouldn't for your local utility company
| that serves ten or so blind customers, none of whom are
| programmers.
|
| This is exactly why section 508 and ADA liability rules
| exist. If they don't know how to make their programs
| accessible, a very expensive lawsuit will show them how.
| Also, why would a small utility be writing their own
| software? Shouldn't they be using COTS?
| hinkley wrote:
| Many command line tools have been adding 'tty mode' for years
| so that if you run the command through a pipe you get simpler
| output that is easier to parse and record. If you're at an
| interactive shell you get all of the 'special effects'.
|
| You will still find many that don't do this properly,
| especially if you run the commands through a CI tool which is
| typically a dead giveaway. Browser based log viewers aren't
| going to handle VT100 escapes and so you see garbage in the
| output. In this regard using unicode emojis to pretty things
| up works more reliably.
|
| I would presume that a screen reader should do the same
| thing. Turn off tty mode so that the stream is easier to
| parse.
| taeric wrote:
| This feels very short sighted, oddly. Your visual terminal
| interface is visual, sure. But there is nothing that says
| that has to be how you interface with a terminal. It is just
| a character/symbol stream connection. With interrogation
| commands that can go back and forth, there is nothing that
| says it can't be as accessible as you want it to be.
| csmpltn wrote:
| > "there is nothing that says it can't be as accessible as
| you want it to be."
|
| And there is nothing that says that it _should_ be
| accessible. Which is exactly why a terminal, going by the
| very definition you yourself provide, is very difficult to
| work with from an accessibility perspective. There are
| simply too many degrees of freedom, allowing you to
| represent data in too many different ways, making it very
| difficult for something like a screen reader to "parse"
| the buffer (which can contain anything, not just text) and
| convert it to something meaningful.
|
| Using a terminal, one can implement a progress bar in
| hundreds of different ways - and it would be impossible for
| a screen reader to handle all such use-cases.
| bayindirh wrote:
| apt is explicitly flagged as "shall not be scripted", and older
| tools (apt-*) shall be used instead. I'm guessing that they're
| also more accessible all-over.
|
| Considering Debian installer has braille support, I don't think
| Debian could overlook something like this.
| oefrha wrote:
| Accessibility isn't the only problem, terminal logs are also
| screwed up. Try scrolling up when apt is showing its fancy
| progress bar, you'll realize logs beyond a screenful is simply
| "eaten". Overall not a fan of the unnecessary TUIzation.
| mkesper wrote:
| No problem with tmux...
| AkBKukU wrote:
| For apt the easy way around this is to just use apt-get since
| that has a traditional output.
|
| But you make an interesting point. I've lamented the current
| practice of hard coding escape sequences in the past for
| novelty reasons: https://news.ycombinator.com/item?id=26013556
| But a screen reader could theoretically implement a terminfo
| entry that could be checked by an application to see if cursor
| control is supported. If not then it could fall back to a plain
| output method which I believe should make following along
| audibly easier.
| prajwalshetty wrote:
| I implemented something like this in bash. I called this status
| bar sort of implementation as "message bar" to show the ongoing
| status of the script on a reserved lines on bottom of the screen
| - I had to implement progress bar one for each task inline (and
| not on the bottom bar) - and constantly had to switch "states" to
| save/restore the cursor positions on screen.
|
| This message bar would also become a prompt and take in user
| inputs on certain events apart from being a static status bar -
| super useful for the use case.
|
| Cursor positions, scrolling screens, events, progress bar
| updates, managing child sessions, handling screen resolution
| changes, taking user inputs...everything had to be catered for in
| this.
|
| I thoroughly enjoyed developing this and still think it was a bit
| over the board for what I was developing at the time but I was
| sent on this path of constant learning so couldn't look back -
| was worth it.
|
| Here's the project: https://github.com/PrajwalSD/runSAS
| croes wrote:
| Shouldn't the title be without question mark or "How does apt
| render it fancy progress bar?"?
| goohle wrote:
| It's "Newspaper Headlines English"
|
| https://www.thoughtco.com/understanding-newspaper-headlines-...
| bluedays wrote:
| Yeah except you dropped an s, so maybe we shouldn't be so
| critical.
| julienpalard wrote:
| Fixed, thanks! (I'm french :D)
| croes wrote:
| Wasn't meant as a critique or nitpicking, just an honest
| question. I am not a native speaker either.
| julienpalard wrote:
| Ooh so I hope you were right ;)
| christophilus wrote:
| Yes.
| skrebbel wrote:
| Not everybody is a native English speaker.
| bregma wrote:
| Including many native English speakers.
| rob74 wrote:
| I'm not a native English speaker myself, but the phrasing of
| questions in English is one of the most basic things, English
| 101 if you will, so if you consider your English good enough
| to write a blog post, you should be capable of doing that...
| julienpalard wrote:
| I'm french ;)
| [deleted]
| brainbag wrote:
| Good explanation. It is a nice progress bar, but not particularly
| unique; many command line tools use progress bars like this.
|
| I've always been curious how the Heroku CLI renders its progress
| bar, which is unique as far as I've seen. It does not seem to use
| characters, it's seemingly a smooth, pixel-based fill like you'd
| get in a non-terminal app. Downloading a pg backup is the most
| common place I see it.
| julienpalard wrote:
| In Python you can use tqdm for this: python
| -c 'import tqdm; i=[j for j in tqdm.trange(int(1e8))]'
| kqr wrote:
| There are characters that are boxes of varying width, but maybe
| you've confirmed it's not even that?
| timdorr wrote:
| They use this package under the hood:
| https://github.com/jdxcode/smooth-progress
|
| It uses glyphs of boxes that have varying widths. When joined
| together, they appear as a solid bar.
| [deleted]
| yakubin wrote:
| APT's progress bar does have issues though. When you shrink the
| terminal when the progress bar is visible, you're going to have
| gibberish in the terminal. That's because the terminal commands
| to move the cursor by lines work on visual lines, not semantic
| lines, so they aren't invariant under the operation of resizing
| the terminal window. That's a really rough area of Linux
| terminals: when you resize a window, the terminal is going to
| rewrap the lines according to semantic lines (based on line feed
| characters), so it's clear that all major terminals understand
| this distinction, but the terminal API doesn't give applications
| access to moving by semantic lines, so we're left with bugs like
| these.
|
| I think PowerShell has the concept of progress reporting
| specially recognized? That sounds like a good idea. It could be
| used to render a GUI progress bar and leave the pipeline alone,
| for the objects. That's just my vague idea, since I didn't spend
| much time with PowerShell.
| chmsky00 wrote:
| The problem you're highlighting is a simple design issue.
|
| My problem with progress bars is they are usually nonsense
| estimates based upon random programmer choices.
|
| I know former MSers who worked on progress bars for old Windows
| versions. Specifically I remember them saying the progress bar
| for copying files to USB was estimating how long until a USB
| hardware buffer was clear, without any idea how much more data
| there was intended to be jammed into the buffer once it
| cleared. Buffer would immediately refill and increase the wait
| time.
|
| Given how slowly Windows deprecates code, who knows what
| subroutines any given progress bar is relying on.
|
| They're nonsense features meant to soothe users who seek
| progress. They don't need to be interesting visually. I'd
| accept a simple countdown that made sense.
|
| I'm hoping manufacture of application specific chips takes off.
| That we embed a 3D engine into silicon and this "software
| engineering is life" mind virus can go away.
|
| We simply did not do it that way before because we lacked the
| manufacturing capabilities.
|
| If manufacturing hopes and dreams of colleagues in the chip biz
| come to fruition, software is on its way out as a routine part
| of developing new technology. But I mean they're biased;
| attention on hardware versus software makes them more valuable.
| WorldMaker wrote:
| I've been starting work on a long rant title "Counting is
| Harder Than You Think". In general, I think most people think
| counting is one of the easiest things for computers to do
| because people learn counting in elementary school and just
| forever associate it with "easy". (Someone's never asked the
| elementary school teacher's opinion of that.)
|
| "How hard can it be, it's just a Select Count() in SQL!" Uh,
| that Count() is possibly doing a ton of work in CPU/IO that
| the server could be doing for other things, and sure an Index
| might speed that up, but you can't really index an Index and
| eventually you get right back to where you can't afford the
| CPU/IO time.
|
| People just assume computers have exact numbers at all times.
| Some of that is just a problem of bad UX design ("why are we
| showing a meaningless estimate number like 1,492,631 and not
| 'about a Million things to do'?"), but so much of it just
| seems to be that people think counting is easy.
| spookthesunset wrote:
| If we are gonna bitch about progress bars, Microsoft's are
| almost always the worst. So many of them get to 99% and them
| stall out... dunno how they get their progress bars so bad.
| WorldMaker wrote:
| The problem isn't Microsoft. The problem is that progress
| bars are the worst way to indicate progress ever invented
| except for all the other terrible ways to indicate progress
| we've invented. Percentage numbers are always a lie and
| shouldn't even be shown, but some people like the soothing
| comfort of "number go big".
| tjoff wrote:
| No, you want to know if the command spewing out text is
| going to be done in about 5 seconds (I'll wait), 5
| minutes (time for coffee or whatever) or 5 hours...
|
| They also are there to indicate progress if no other
| visuals are present (no, it hasn't crashed yet).
|
| Progress bars solve that given some uncertainty in most
| cases. And that is very much appreciated. Everyone knows
| or learns that they aren't perfect, and that is fine.
| bmn__ wrote:
| > I know former MSers who worked on progress bars for old
| Windows versions.
|
| https://explainxkcd.com/612
| chmsky00 wrote:
| Yep. That was around the time I worked with those folks.
|
| I would not be surprised if the topic came up because of
| that comic.
| kccqzy wrote:
| One solution is to install a signal handler for SIGWINCH and
| redraw everything.
|
| In practice this works best for terminal apps that already make
| use of the entire screen (say vim or emacs) but not so well for
| mixed text and some visual gadgets.
| yakubin wrote:
| APT already has a signal handler for SIGWINCH, in order to
| resize the progress bar. The problem is that after APT finds
| out about the resize it clears the last line on the screen,
| but it's a visual line, which is only a part of a semantic
| line which was wrapped after shrinking the terminal window.
| APT doesn't know how many visual lines the progress bar takes
| after being rewrapped by a resize, so it doesn't know how
| many lines to clear. It could try to compensate by
| remembering the old width in characters, looking at the new
| one and calculating what rewrapping would result in, but that
| would probably not be very reliable across different
| terminals.
|
| Redrawing everything is annoying from UX point of view,
| because it breaks copying lines out of the terminal. Instead
| of getting a bunch of lines separated by line feeds
| corresponding to semantic lines, the user gets a bunch of
| lines corresponding to the visual lines in the terminal (so a
| lot of lines broken, some of them midword), including loads
| of meaningless whitespace.
|
| I wish the modern terminal was less of an old terminal
| emulator and more of a GUI for displaying pipelines. So no
| ncurses-like applications, such as vim, more first-class
| support for progress bars (instead of the hacks that we
| have), support for jumping between the lines in scrollback
| where the user typed in the commands, no possibility of
| breaking the terminal state (such as what happens when I hit
| Ctrl+C in the password prompt of openvpn) forcing the user to
| run "reset", etc. etc.
| ljm wrote:
| I hate this in homebrew and macports: they use a progress line
| that mostly works, until your terminal is too wide or too
| narrow. # ## ### ####
| ###### ####### ######### ############
| ,,,
| #####################################################
| #########################################################
| ###############################################################
| ###############################################################
| ###########################################
| agolliver wrote:
| PowerShell does have progress bars, but rendering them is
| (was?) on the critical path. I once had a script that was
| downloading a large file and took ages to finish, you could
| speed it up by well over 10x disabling the progress bar.
|
| https://github.com/PowerShell/PowerShell/issues/2138
| peakaboo wrote:
| I have a hard time believing real programmers would allow
| something like that in production code... But I guess if its
| corporate, nobody cares about the quality.
| jrockway wrote:
| Outputting text to a terminal is synchronous. Try
| connecting your favorite program's STDOUT to a program that
| doesn't read its STDIN. The program will stop running after
| the output buffer fills, because "write" blocks. Obviously
| you can buffer internally, and have a "UI thread" for
| writing stdout... but literally nobody does this.
|
| This is why people try to write the fastest terminal
| emulator -- a slow terminal emulator slows down apps. (And,
| it all makes sense. You have to apply backpressure when you
| are receiving data more quickly than you can process it.
| You can buffer to smooth over spikes, or you could drop
| data on the floor, but both of those would suck for output
| in a terminal!)
| derefr wrote:
| > have a "UI thread" for writing stdout... but literally
| nobody does this.
|
| The Erlang runtime does it for you by default :)
| Sharlin wrote:
| stdout is by default buffered in almost every language.
| Outputting lots of non-critical stuff to stderr, or
| manually flushing stdout on every write, is definitely a
| rookie mistake. Raw terminal output should not be a big
| performance sink either if you do the sane thing and only
| write on every 100th or 1000th or whatever iteration of
| your main loop. No threads needed.
| oneweekwonder wrote:
| npm was about 6 years old and im pretty sure a lot of
| corporates have adopted node by then.
|
| I don't think it has so much to do with quality but rather
| we like to re-invent shiny "new" wheels and then the new
| tooling must iron out all the corner cases.
|
| https://github.com/npm/npm/issues/11283
| blowski wrote:
| What does a programmer that's not a "real programmer" do?
| Only write code in films? Slap the keyboard on
| hackertyper.net?
| aflag wrote:
| I imagine the argument would be that the person that
| writes code at a corporation is not the same person who
| writes it for hobby. Even if they share the same body,
| one is just doing it for the cash and the other for fun.
| QuercusMax wrote:
| Are they synchronizing on the UI every time they update the
| progress bar? That's a rookie mistake.
| vlunkr wrote:
| IMO this isn't a big enough issue to worry about. Most progress
| bars break when you resize the terminal, I just expect it at
| this point.
| zorr wrote:
| It's good to know the underlying low-level terminal commands that
| make this work like this article explains. But I guess most
| people will use a terminal abstraction library that has a higher
| level API to move the cursor, clean the screen etc.
|
| Terminals are complex beasts and they can do powerful things. If
| you're interested in this, I can recommend reading the source
| code for Tmux, JLine3 and some other libraries shell formatting
| or TUI libraries.
| pantulis wrote:
| tqdm in Python was also very nice and easy to use.
| franga2000 wrote:
| +1 for tqdm - not only does it take less than one line to
| "automatically" turn any loop into a progress bar, it also
| supports native progress bars in Jupyter Notebooks and even a
| basic GUI one
| WalterGR wrote:
| > some other libraries shell formatting or TUI libraries.
|
| Any that you'd recommend?
| eliaspro wrote:
| Bubbletea [1] for Go is quite polished
|
| [1] https://github.com/charmbracelet/bubbletea
| abdusco wrote:
| Rich (for formatting) and textual (for TUI) from Will McGugan
| are great libraries:
|
| https://github.com/willmcgugan/rich
|
| https://github.com/willmcgugan/textual
| zorr wrote:
| I only have some experience with JLine (Java) and Termion
| (Rust). Heard great things about Textual (Python).
|
| There are also smaller libraries focusing on just one aspect
| like coloring. For example the Colorize Ruby gem.
| sirwitti wrote:
| I recently started using python click [1], which is awesome
| after wrapping my head around some things. Click also
| provides a useful abstraction for creating progress bars.
|
| [1] https://palletsprojects.com/p/click/
| flatiron wrote:
| just don't read gnu screen!
| formerly_proven wrote:
| Terminal abstraction libraries feel like a relict of the past
| to me, because there used to be a lot of diversity in command
| sets and capabilities. Nowadays you can pretty much just assume
| that most commands work everywhere, so there is a lot less
| reason to drag around ncurses and friends if you want to make a
| progress bar that doesn't break your output or have some
| colored text.
| eptcyka wrote:
| Oh, but you just can't. MacOS's zsh can't deal with my tmux
| shell when I ssh in. There's all kinds of great features that
| allow for basically using pixel buffers in a shell if you
| stray away from the default terminal implementations. Getting
| colors to work properly is also still a pain.
| gnachman wrote:
| There is still a ton of diversity (underline color, curly
| underlines, synchronization, and image support for example),
| but it's nearly impossible to figure out what features are
| available in a given terminal. I just finished adding vt400
| support to iTerm2 this week but I doubt any of it will be
| used because it's not widely available enough for people to
| bother.
| zorr wrote:
| Ncurses and friends could be overkill but there is still
| value in small wrapper libraries that provide a higher level
| API on top of escape codes.
|
| `setCursor(lines, 0)` is more readable than
| `write(f"\033[{lines};0f")`
| [deleted]
| jancsika wrote:
| I'd prefer a "Double Debian" distro that does the following:
|
| 1. When I run an apt command, it goes ahead and runs an automated
| version of the command on a "shadow" apt on the same machine.
|
| 2. It measures the time it took to do that operation on the
| shadow apt.
|
| 3. By the time I respond "y" to the command, it uses the duration
| from step 2 to display a _time-based_ progress bar that animates
| smoothly over said duration.
|
| 4. If the shadow apt command hasn't finished by the time I
| respond "y" to the command, it brings up Ms. Pacman for me to
| play for a bit.
|
| I bet I'd be a much more satisfied user!
| someonewhocar3s wrote:
| ...You'd prefer your terminal to terribly front-run potentially
| huge operations...
|
| .. to get a single sample estimate ..
|
| ....because you expect the loading bars to be displayed to last
| on the order of milliseconds....
| agumonkey wrote:
| terminal rendering is fun. it's often limited but the few it
| offers gets you going nicely.
| gildas wrote:
| For those who wonder how these "\033[1A..." notations are
| specified, look up "ANSI escape code" on the net, e.g.
| https://en.wikipedia.org/wiki/ANSI_escape_code
| penguin_booze wrote:
| This in-band control signalling was mind bending the first time
| I learned about it. "How come other programs print text in
| colour, but mine is only plain? Wow - I can print special
| sequences and that controls color?! This means the console must
| be interpreting what I'm sending, not doing a simple lookup and
| display!".
| katzeilla wrote:
| ... and in case you messed up your console, typing `reset`
| followed by enter will fix it.
| faho wrote:
| Or, if you managed to mess up the terminal modes and enter
| doesn't work, try pressing ctrl-j instead.
| julienpalard wrote:
| I had to do it quite a few time when implementing this one ;)
|
| I often type "C-c" or "enter" before typing reset to ensure I
| have nothing before it in the buffer.
|
| A bit like I often do "Enter Enter ~ .` instead of just `~.`
| to kill a dying SSH session (I don't know why I hit enter
| twice here... a single time is enough.) to ensure the ~ is at
| the beginning of the buffer (if it's not, it won't work).
| froh wrote:
| To make things portable look up termcap(5) and tput(1) to ask
| the termcap database for the right control sequence for your
| terminal, and use those.
| julienpalard wrote:
| Interestingly apt don't do this, they hardcode sequences as I
| did.
|
| Maybe they bet those sequences are the same on all terminals
| since 1970, espacially those marked "DEC Private" and it
| won't change soon?
|
| I honestly don't know.
| p-e-w wrote:
| And if you want to play around with escape sequences from your
| shell, you can do so using `echo -e "\e[1A..."` in most shells
| (the `-e` flag enables backslash interpretation, while `\e`
| stands for the ESC character, alias `\033`).
|
| Note also that (traditional) escape sequences cannot execute
| arbitrary code, so you can't really break anything that closing
| and reopening the terminal won't fix.
| withinboredom wrote:
| And if you really want to have fine, put them in your git
| commits and stand back as `git log` destroys your coworkers
| terminal.
| quotemstr wrote:
| You can be sloppy and hardcode escape sequences in your shell
| script, or you do it right and use the tput program to look
| up the right sequences at runtime.
| p-e-w wrote:
| ANSI escape sequences are a _standard_. Relying on that
| standard isn 't sloppy, any more than relying on any other
| standard. Terminfo is today basically a hack to work around
| the problem that some uncommon sequences differ between
| terminals, but others (like cursor movement and SGR, which
| cover 99% of use cases) have been well-established for 30
| years. I much prefer using such sequences directly over the
| complexity of shelling out to an external program. I'm not
| going to pretend that there is a chance my script might
| have to run on some obscure text terminal from the 80s some
| day.
| rbonvall wrote:
| Standardness notwithstanding, using tput is more readable
| and memorable. I can type this from memory:
| echo "I'm $(tput setaf 4)blue$(tput sgr0)."
|
| but I couldn't get this one right without looking it up:
| echo -e "I'm \033[0;34mblue\033[0m."
| [deleted]
| mmphosis wrote:
| alias red='echo -ne "\e[31m"' alias yellow='echo
| -ne "\e[33m"' alias green='echo -ne "\e[32m"'
| alias blue='echo -ne "\e[34m"' alias cyan='echo -ne
| "\e[36m"' alias violet='echo -ne "\e[35m"'
| alias grey='echo -ne "\e[90m"' alias gray='echo -ne
| "\e[90m"' alias white='echo -ne "\e[37m"'
| alias bold='echo -ne "\e[1m"' alias flash='echo -ne
| "\e[7m\e[5m"' alias normal='echo -ne "\e(B\e[m"'
| faho wrote:
| Still not _particularly_ readable or memorable. You still
| need to remember that blue is color number 4 and that
| "sgr0" is "exit_attribute_mode", what you might call
| "reset".
| tomc1985 wrote:
| Store the sequences in variables and put those instead.
| You get readability and locality!
| nick__m wrote:
| I was about to post the exact same reply so I will post
| it as an example: #!/bin/sh
| BLUE=$(tput setaf 4) RST=$(tput sgr0) echo
| "I'm ${BLUE}blue${RST}."
| junon wrote:
| Because you're using a lot more characters than you need.
| echo -e "I'm \e[34mblue\e[m."
|
| However, `tput` is definitely the more 'correct'
| approach. What actual benefit it affords you over the
| hardcoded escapes is left to edge-cases and very, _very_
| old or otherwise niche terminal emulators.
|
| By the way, the `3x` means foreground, the `4x` means
| background. `x` is a number between 0 and 7 (inclusive)
| that indicates the color. `x=8` means "extended"/non-
| standard and generally takes a few more control codes
| afterward, and `x=9` means "reset".
|
| If you know binary, you can remember the 0-7 colors. It's
| a 3-bit code corresponding to BGR (where B is the MSB).
| BGR 000 = 0 = black 001 = 1 = red
| 010 = 2 = green 011 = 3 = yellow 100 = 4
| = blue 101 = 5 = magenta 110 = 6 = cyan
| 111 = 7 = white
|
| You can also add 60 to the code to make it "bright", e.g.
| red foreground (31) can be made bright red by adding
| 31+60=91. Same can be applied to backgrounds (4x+60=10x).
|
| The bright codes are less supported though admittedly
| I've never seen a modern emulator that doesn't. It also
| gives you bright colors on old Windows cmd.exe prompts
| without needing the bold mode (1).
| junon wrote:
| > ANSI escape sequences are a standard.
|
| Hi, Chalk maintainer here. They are most certainly _not_
| standardized, despite frequently shown with the "ANSI"
| nickname - at least, not by any widely accepted standard
| I'm aware of.
|
| The only standardization I could ever find myself was
| that of a document format similar to PostScript, but it
| didn't describe the codes that XTerm originally adopted.
| You would have to ask the original implementors how/why
| they chose the codes that they did.
|
| Serial/teletype (TTY) escape codes are one of the most
| archaic, still-widely-used functions of terminal
| emulators and as such sprung up before the dawn of widely
| agreed-upon standards. There are thousands of them, not
| just those for rendering, and they were historically used
| for serial transmission control for e.g. printers and
| other UART devices.
|
| Thus, many (most, actually) escape codes have nothing to
| do with rendering and instead control the behavior of the
| two nodes at either end of a serial line.
|
| Since most of these devices used proprietary codes -
| including those for rendering - a common subset of those
| related specifically to terminal emulation/TTYs were
| derived and a database of those codes was formed. This
| database is usually installed on *nix installations via
| the `ncurses` package and is generally referred to as the
| "terminfo database" or simply "terminfo" or "termdb".
|
| Each file in the (very, very extensive) database is a
| binary file format that includes a _complete stack-based
| scripting language_ for string-based escape codes (there
| are a few datatypes supported in the file format, some of
| which are purely to indicate capabilities and whatnot).
|
| I wrote an implementation of a Terminfo parser with
| scripting language support in a C++ library if anyone is
| interested. I'd have to dig it up if someone would like
| to use it - just let me know.
|
| The entry for the given terminal session is selected via
| the `TERM` environment variable in most cases, which
| ncurses-based applications (and other applications that
| support terminfo) reads in order to select and parse the
| correct terminfo entry.
|
| Most of the common escapes we use today stem from the
| XTerm set of codes - namely, `TERM=xterm-256color` -
| since supporting Terminfo is a beast and can hurt
| performance when properly supported (since each
| 'invocation' of the escape is really an evaluation of a
| small script).
|
| Since XTerm made the codes simple to render out
| (CSI(\x1b) <mode>[;<mode>[;...]] m), and since most
| terminal emulators chose to model their escapes off of
| XTerm, many application developers started to hardcode
| the XTerm escapes directly in lieu of using Terminfo.
| This was further set in stone when the often-referenced
| Wikipedia page[0] for ANSI escapes was written, claiming
| that those were the _only_ codes - which is very far from
| the case.
|
| XTerm in particular re-uses a lot of codes from popular-
| now-archiaic terminals such as the VT100, which
| introduced bold types among other things (but not
| color!). I believe it was the VT520 or something like
| that that had the first semblance of color codes, but my
| memory is fuzzy and I'm not in a position to research it
| right now. I'll leave that as an exercise to the reader.
|
| For the most part this has worked out okay. Naive/toy
| emulators recognize these codes, some shells (not many,
| but some) actually transform the escapes, some libraries
| translate them to Windows API calls (e.g. libuv, the I/O
| and event loop behind Node.js), and most established
| terminal emulators choose to use the xterm-256color by
| default anyway.
|
| There have been a number of closed-door discussions
| between terminal emulator vendors about modernizing all
| of this but, incredibly, PowerShell is the only effort
| I've seen widely adopted or used that has made strides in
| improving this whole debacle.
|
| I'm sure there are some errors in what I wrote - feel
| free to correct me if I'm wrong. But this is my
| understanding of the messy world of "ANSI" escape
| characters.
|
| [0] https://en.wikipedia.org/wiki/ANSI_escape_code
| chasil wrote:
| Below is the original manual for a Lear-Siegler ADM-3A
| terminal, which did _not_ use any of the VT100-derrived
| escape codes (and was much less expensive at the time).
|
| The ADM-3A is notable as it was used by Bill Joy to
| develop the "vi" editor, and the key layout should be
| familiar.
|
| http://www.bitsavers.org/www.computer.museum.uq.edu.au/pd
| f/D...
| quotemstr wrote:
| OpenGL is a standard too, but an OpenGL program will
| break on a system supporting only Vulkan. Terminfo exists
| for a reason and that reason is still there today. For
| example, I can run under TERM=dumb to disable fancy
| output for various reasons.
|
| No, terminfo isn't a hack, ANSI escapes aren't
| ubiquitous, and every single time I see someone hardcode
| escape sequences in some script, my opinion of that
| person declines.
| klodolph wrote:
| Imagine living in a world where OpenGL has been around
| since the 1970s, stable for a long time, and where
| there's no sign of Vulkan on the horizon.
|
| That's the world of terminals. There's no sign that the
| way terminals work is going to change. These standard
| escape sequences are _old._
| fiddlerwoaroof wrote:
| Until your program is run in a emacs window (lint-
| staged's progress bars mess up magit's output buffer,
| making it really difficult to figure out what the problem
| is) or is piped through grep.
|
| tput will do the right thing because of terminfo, hand-
| coded escape sequences will make someone curse you.
| not2b wrote:
| Back when termcap and later terminfo were developed, most
| terminals didn't support ANSI escape sequences, at all,
| and used completely different control codes. But that was
| 30+ years ago. It was for actual dumb terminals that
| could only display text. These days, it's a safe bet that
| any terminal emulator will support the ANSI escape
| sequences.
| spudlyo wrote:
| Is it though? What if I have my TERM set to 'dumb'
| because I'm running your CLI command inside a very
| limited console embedded inside a text editor or other
| tool? At the very least you should `test -t` in your
| script to ensure STDOUT isn't being redirected to a file
| so you don't fill it with garbage escape sequences.
| MayeulC wrote:
| > you can't really break anything that closing and reopening
| the terminal won't fix
|
| Or typing `reset`, which is sometimes also necessary after
| accidentally piping binary to stdout.
| p-e-w wrote:
| `reset` won't work if you have used the escape sequence
| that disables raw mode, at which point the shell won't be
| able to process keyboard input anymore and you simply
| cannot issue commands. I'm not aware of any method for
| fixing this that doesn't involve restarting the terminal.
| remram wrote:
| You should still be able to `reset` in line-buffering
| mode.
| anothermindbomb wrote:
| `stty sane` not work?
| buserror wrote:
| I use a similar approach in libmish [0], ie I "reserve" the
| bottom 2 rows, and output the scrolling text in the top parts.
| Pretty straightforward.
|
| [0]: https://github.com/buserror/libmish
| quotemstr wrote:
| I hate it when programs hardcore control sequences like this.
| Just look them up at runtime with terminfo --- it's really not
| that hard.
| julienpalard wrote:
| How would you do the DECSTBM without hardocindg it? Honest
| question.
| [deleted]
| fladd wrote:
| Nice. I also looked into this a while ago and came up with a very
| simplistic (few lines) Python implementation that does this and
| even allows for some customization:
| https://gist.github.com/fladd/36c422f1c0e9bf02f41f9fad19609d...
| miohtama wrote:
| For building rich terminal UIs in Python, I recommend Rich the
| library
|
| https://github.com/willmcgugan/rich
| julienpalard wrote:
| There's https://tqdm.github.io/ to do nice progress bars in
| Python.
|
| But my goal was not really the progress bars, but the placing
| of it below the logs.
| stavros wrote:
| Yeah, when I was reading the article I started out thinking
| "What? It's just simply rewinding the line", and then you
| mentioned that it's always at the bottom of the window, and I
| thought "Now _that_ is a good trick, I wonder how it 's
| done".
|
| And then I didn't wonder. Good work.
___________________________________________________________________
(page generated 2021-10-13 23:00 UTC)