[HN Gopher] It's Time for Operating Systems to Rediscover Hardware
___________________________________________________________________
It's Time for Operating Systems to Rediscover Hardware
Author : matt_d
Score : 133 points
Date : 2021-08-31 22:41 UTC (11 hours ago)
(HTM) web link (www.usenix.org)
(TXT) w3m dump (www.usenix.org)
| ironman1478 wrote:
| I've worked on firmware for a few SoC's that my company has
| designed or purchased and we use embedded linux on some. I think
| his observation about how Linux doesn't abstract the hardware
| accurately is totally right. The system isn't that complicated,
| but implementing everything within the world of Linux has been
| painful imo, so for future implementations we're evaluating
| RTOS's. The main reason people use Linux on these custom SoCs
| because of driver availability of key components (ethernet phys
| and persistent storage), large library of tools, and probably $
| (who wants to pay for QNX). Those are good reasons, but if you
| have to write a large volume of code, it doesn't really feel like
| it's providing too much. It definitely felt like we were just
| using a hammer and seeing everything as a nail. Those are my 2
| cents.
| elcritch wrote:
| This issue with needing more flexibility on SoC's makes me
| wonder if its part of why the Linux Foundation has put a lot of
| work and resources into Zephyr RTOS [1]. Its gaining a lot of
| popularity and incorporates some of nice elements of linux like
| device tree syntax (DTS) files [2]. I could readily imagine
| using Linux & Zephyr on a SoC, and using shared DTS to describe
| the SoC architecture.
|
| Actually, come to think of it extending DTS syntax to describe
| the pointer swizzling the OP video mentions. Then both OO'es
| could derive correct physical address handling.
|
| 1: https://www.linuxfoundation.org/blog/the-power-of-zephyr-
| rto... 2:
| https://docs.zephyrproject.org/latest/guides/dts/index.html
| ironman1478 wrote:
| So, it's funny. It's not uncommon to run RTOSs and Linux on
| the same CPU, where the rtos coordinates tasks between the
| components that have real time requirements and the Linux
| side does stuff like stream data from a buffer that the rtos
| writes to. We had this architecture on some systems and the
| amount of cludging to get this all working was annoying to
| say the least. We specifically had issues are figuring out
| how to partition the memory regions for each OS and it wasn't
| trivial to validate.
|
| I wasn't around when the decisions at my employer when the
| decisions made around OS options, but this would've been a
| viable one if it was mature enough at that time. Thanks for
| the info!
| elcritch wrote:
| Sounds way too familiar! What's amazing (to me) is that
| there's even a VSCode plugin for DTS files that checks the
| memory offsets and gives you nice error messages like
| "flash partition 1 overlap partition 2 at 0x3e000, ...".
| It's a bit like magic after doing it manually.
|
| P.S. huh looks like Linux and Zephyr _are_ doing this via
| RPMsg subsystem as well. Wow, much nicer than the last time
| I looked into it!
| ksec wrote:
| Off Topic. Why does it need a capital R in the Rediscover? (
| Along with T in Time ) All I can see when I reading it was Redis
| something Hardware.
|
| Edit: And it turns out the title in video was actually without
| capital T and R.
| [deleted]
| Arnavion wrote:
| https://en.m.wikipedia.org/wiki/Title_case
| evilos wrote:
| Interesting talk but I'm not convinced of the value proposition
| of "rediscovering hardware". The speaker points out that there
| are lots of small processors inside a modern SoC that are often
| running their own software hidden from Linux. I agree that there
| are security implications here a la Qualcomm Hexagon
| vulnerabilities or Intel Management Engine backdoors but I'm not
| sure what the speaker's proposed alternative is. Is it that a
| "true" OS would directly execute the duties of these hidden
| processors? This is extra work that would make systems slower.
| Having dedicated hardware handle these duties is an optimization
| (hardware acceleration). Alternatively you could replace these
| small processors with fixed function hardware. This would make
| any system flaws permanent although it would be much harder to
| exploit any vulnerabilities. Does the lack of hidden processors
| make the system "better" somehow? (aside from potentially more
| secure) Are bus masters and hardware accelerators part of the OS?
|
| It would certainly be a more beautiful system that I would love
| to see, but I need help justifying the engineering cost. I hope
| to see some open hardware passion projects along these lines, but
| I doubt they will ever be mainstream.
| taneliv wrote:
| I think the "speaker's proposed alternative" is to do more
| academic research in this area.
|
| You have good questions in your comment, and I think he is
| calling out for searching answers to them and other similar
| questions. Apparently, the operating system researchers haven't
| been doing this enough, or at all. His speech is a lament for
| this state of affairs, and a call to get excited about finding
| out what benefits would be gained by doing things (very)
| differently.
|
| But that's just my take, and I'm not working on operating
| systems research.
| evilos wrote:
| I guess I'm struggling to understand what the problem to
| solve here is (besides security bugs). Maybe that is wrong,
| and the idea is just to try new things. But seems to me that
| 'try new things' isn't a particularly strong call to action
| when you are competing for research funds/time.
| SilverRed wrote:
| I'm increasingly seeing the value of Apple style products where
| almost every part of the device from the hardware, to firmware,
| to OS, to UI is all from the same company.
|
| The whole combined experience just makes things run so smoothly
| and allows security issues to be patched since Apple has the
| resources that maintain each part internally.
| alexisread wrote:
| Actually there are a few bright spots on the horizon. RISC-V is
| gaining mindshare, and I've seen separate events/papers which
| push for using modified RISC-V cores as specialised processors,
| similar to eg. The Nintendo64 which used MIPS cores. If that
| gains traction then there's a good chance that the software
| will improve.
|
| On the radio side of things, Dash7 firmware will run on many
| Lora devices, meaning you don't have to use the proprietary
| Lora software.
| elcritch wrote:
| More likely its that a "full" OS would at least be be aware of
| the heterogeneous elements. Then it could provide api's for
| negotiation between them, if not directly providing a method to
| run them.
|
| One method could be, say, extending microkernel capability
| systems to incorporate these remote processors. Or perhaps even
| the ability to provide WASM or ePBF control blocks to customize
| the power management system. Though its true that'd require
| more engineering resources.
|
| Actually USB is somewhat like this in that there's a hardware
| api and spec that the OS knows about. Linux has troubles with
| even that in my experience (often requiring a reboot to fix.
| OneTimePetes wrote:
| Oh, oh the ragnarok of systems programming, moores plateau
| stretches on, parallel is coming to a limit and now its goodbye
| abstractions & no-skill-lib-glue-code, hello 80s hardware hacks
| and efficient programming again? Will OSes be produced like
| games, were they start with a "reuseable" oo-engine and then that
| is broken down into data-orientated design.
|
| I thought we had more time..
|
| Cant we global warm postpone this?
|
| Let the next generation sort it all out?
| fithisux wrote:
| Hardware should become more clever in order to take burden from
| drivers.
| xfer wrote:
| It is a interesting debate to have but this is unlikely to happen
| for consumer hardware, since no OEM wants you to have that kind
| of access(talking of those phone SoCs), they will just give you a
| slice of ccNUMA to protect their "IP". That qualcomm bug that he
| showed is unlikely to go away for consumers.
|
| There is a message to build your own computers and that's
| probably what people will do and it could be interesting research
| for custom servers and future heterogeneous architectures. I
| suspect people in this industry are already doing it.
| new_realist wrote:
| Does modern hardware require OS-mediated isolation? If not, why
| should the OS get in the way of apps using the hardware directly?
| neltnerb wrote:
| I certainly write plenty of code where I write to registers
| directly to configure hardware modules and don't think I'm
| using an OS in any way. I assume most of the security design
| and hardware permissions are handled at a higher level than I
| usually write code.
| mikewarot wrote:
| The function of the operating system is to safely multiplex the
| hardware. If the OS doesn't control the hardware, it can't
| provide safety.
|
| If Linux is running inside a sandbox, it can't secure anything.
| pjmlp wrote:
| "an operating system is a collection of things that don't fit
| inside a language; there shouldn't be one"
|
| -- Dan Ingalls
| SilverRed wrote:
| No one wants apps to directly interface with hardware. Imagine
| loading up a notes app and it directly trys to interface with
| your hard drive using its own built in FS driver and a bug in
| that causes your whole drive to be wiped.
|
| Or that you open two programs at once and they try to access
| the same device at once causing some kind of error and likely
| data loss or system instability.
|
| I don't even want apps to directly have access to the file
| level interface of my storage but rather an extra abstraction
| which lets me pick which files it can access.
| glangdale wrote:
| A perennial complaint; Rob Pike had a trollier but similar
| version years back. I'm not nearly as optimistic that we have the
| resources to cope with the monstrous complexity of hardware now.
|
| An anecdote I like to share is that while I was TA'ing CMU's
| 15-410 operating systems course (a great course - build a
| preemptive multitasking operating system from _scratch_ ) was
| that the student projects could run on real - if old - hardware.
| There was a PC that could boot your stuff if you put it on a
| disk.
|
| This PC had to have PS/2 interface to the keyboard, though. A
| newer PC would be all USB. Apparently, the complexity of the
| codebase to talk to the USB device was around the similar level
| of complexity to the _entire preemptive multitasking operating
| system_ the students were building (and, of course, considerably
| less educational). I mean, this wasn 't a full-functioned OS, but
| it allowed multitasking and preemption of kernel calls, so
| seriously non-trivial.
|
| Multiply this story by all the devices a reasonable OS would be
| expected to talk to and it's a scary prospect for the OS
| researcher... and if you don't have those devices up and running,
| good luck supporting most workloads anyone cares about.
| 0xcde4c3db wrote:
| It seems like the done thing these days (for people who care
| about running on real hardware) is to bring up
| educational/experimental kernels on Raspberry Pi, using the
| "mini UART" for console I/O and the SD interface for storage.
| choeger wrote:
| While I totally agree with your post (especially the point that
| a course where you implement your own OS for real hardware is
| great), I would like to add that USB is a massive
| foundation/abstraction layer. Once you got USB going for HID,
| you are much closer to USB for storage, USB for networking,
| etc. So it certainly _is_ troublesome from an educational
| perspective but the complexity is where it belongs.
| User23 wrote:
| > I was TA'ing CMU's 15-410 operating systems course (a great
| course - build a preemptive multitasking operating system from
| scratch)
|
| That class bordered on legendary. Awesome that you got to TA
| it. I still have an irrational fondness for AFS.
| glangdale wrote:
| It was a fantastic experience, although TA'ing it was
| daunting. Like the apocryphal Ledru-Rollin quote: "There go
| the people. I must follow them, for I am their leader", it
| was frankly hard to keep up with all the different approaches
| students conjured up.
|
| I believe that the course number changed at some point - or
| some shallow aspect of the course was set to change - and a
| bunch of industry people called the university to express
| concern. It is/was (I can't speak for the current iteration
| of the course, although I have no indication that quality has
| slipper or anything) truly transformative.
|
| I offered to run a 'franchise' of the course at the
| University of Sydney a while back and was informed that
| anything quite that transformative wasn't really an option;
| our job was at least in part to pass engineering students who
| didn't care that much about computing.
| josephg wrote:
| > I offered to run a 'franchise' of the course at the
| University of Sydney a while back and was informed that
| anything quite that transformative wasn't really an option
|
| Thats very disappointing. You might have more luck joining
| the University of New South Wales OS team. (For context,
| UNSW is a local rival of USYD).
|
| When I was a student there, the OS course was legendary. I
| only did the "basic" OS course. Our assignments led us to
| implement a bunch of syscalls for handling file IO, and
| write a page fault handler for a toy operating system. I'm
| not sure what they do in the advanced OS course - but it
| has a reputation for a reason. And it looks like[1] its
| still run by Gernot Heiser, who's a legend. He's the brain
| behind SeL4 - which is the world's first (and only?)
| formally verified OS kernel.
|
| I'm kicking myself for not doing his advanced OS course
| while I was a student.
|
| [1] http://www.cse.unsw.edu.au/~cs9242/current/
| hoseja wrote:
| I have a modern consumer motherboard that has intermittent
| problems connecting all it's USB ports, with the latest
| "stable" BIOS supposedly fixing this but also reportedly having
| worse performance.
| throwawaylinux wrote:
| > A perennial complaint; Rob Pike had a trollier but similar
| version years back. I'm not nearly as optimistic that we have
| the resources to cope with the monstrous complexity of hardware
| now.
|
| "Hardware" (read: device drivers) is not notably complicated
| IMO.
|
| That's not to say they're simple, but driving devices from a
| software point of view is pretty similar to interacting with
| other software. You read the spec (APIs), write code to set up
| data structures or registers a certain way, parse responses,
| etc.
|
| Writing a modern full fledged USB stack is very complex, but
| would not be more complex than writing a modern full TCP/IP
| stack for example.
|
| A lot of device programming models have gotten simpler too.
| Device drivers used to be notorious deep voodoo magic e.g., in
| cases of the IDE disk driver, but that was not really
| "complexity" of the software logic so much as hundreds of
| special cases and deviations from standards or odd behavior in
| all manner of IDE controllers and devices. To the point where
| the real wizards were the ones who had access to internal data
| sheets, errata, or reverse engineered firmware from these
| devices, or otherwise spent countless hours poking at things
| and reconstructing quirks for themselves, probably bricking a
| lot of silicon and spinning rust in the process.
|
| But systems and devices are now getting to the point where
| things don't work that way anymore, silicon power is so cheap
| and firmware is on everything. The NVMe device for example sets
| up pretty simple packet-type command queues and sends requests
| and receives responses for operations - query device
| information, perform a read or write, etc). It's not quite
| _that_ simple, there are some quirks and details, but it 's
| quite a lot like writing a client for a server (HTTP, perhaps).
| I think other device interfaces will evolve toward cleaner
| simpler models like this too.
|
| One exception to this is GPUs of course. The other thing is
| with moore's law continuing to slow down there will be
| increased incentive to move more processing out to devices.
| It's long been happening with high end network devices, but
| with technologies like CXL.cache coming soon, I would expect
| that trend to keep ramping up and come to other accelerators
| (crypto, AI, even disk).
|
| So.. it's a mix. Things are definitely getting more complex,
| but AFAIKS the complexity of interacting with hardware is not
| increasing at a much greater rate than the complexity of
| interacting with other software. And, as always this complexity
| is made possible by abstractions and layers and interfaces.
| It's not clearly exceeding our ability to cope with it.
| alexgartrell wrote:
| Admittedly, I got a B in 410 and only managed to TA 213, but I
| do work in the space now.
|
| Yes, it's hard to do research if you start at "let's design a
| kernel from scratch," but you'd never need to do that. You can
| just hack Linux or a bsd, or even use something like bpf to
| extend it.
|
| The thing that annoyed me about 410 is that when I switched to
| Linux I realized that pusha/popa didn't matter at all. It was a
| good course for writing reentrant C and learning the very
| basics of hardware, but the really hard stuff is in the weird
| dynamics of memory management on NUMA systems and work
| conserving io, which you can't get anywhere near if you are
| starting from scratch.
| darksaints wrote:
| I'd argue that implementing a USB driver would probably be far
| more educational, in a practical sense, than implementing a
| kernel. The number of students who will go on to work on core
| kernels is vanishingly small, but driver work is never going to
| end.
|
| Plus, if we ever are to transition to the world of microkernels
| (something that OS researchers have been pushing for decades
| now), we're gonna need a massive amount of people writing
| drivers just to get us to the point where things are barely
| functional.
| Ericson2314 wrote:
| I suppose it would be useful if it taught students how much
| this industry is hare rather than tortoise nonsense where the
| interfaces are sheer accidents, and the implementations are
| mad-dash to keep up.
|
| http://langsec.org/
|
| But what you propose is preprofesional drival. It's bad
| enough that education is subsidized for employers already, we
| don't need to stoop to them further.
| glangdale wrote:
| No. Driver work is painful but it mostly teaches you about
| the h/w in question; you don't learn deep principles from a
| lot of it. You just grind away, spec in hand, until stuff
| works. A lot of this winds up being "one damn thing after
| another". There's a good deal of craft to it, and I don't
| mean to deprecate it, but it's not broadly applicable.
|
| Conversely, the work they did in kernels has a huge amount of
| transfer to thinking about concurrency, which is truly
| valuable - and deep. It also meant that they acquired a far
| less magical idea of where things like processes and threads
| come from, having actually built the mechanisms to make them
| happen (you build a thread library in a warm-up project).
|
| As for microkernels - they often punt a lot of the hard
| concurrency stuff up to user space servers, which will wind
| up needing exactly the kind of concurrency that the students
| learned to build.
| [deleted]
| toast0 wrote:
| It's not just a USB driver though. You've got to support PCI
| (or something) to find the USB controller, then you've got to
| fiddle with that, then you have to discover the devices, then
| you have to interface with them.
|
| Reading from a keyboard is much simpler. Probably makes more
| sense to run the computer headless with a serial terminal and
| make the students poke at a UART. PC standard UARTs are
| pretty easy, and I think it's easier to find a motherboard
| with a COM1 header than a PS/2 port (at least my latest
| hardware matches that description).
| userbinator wrote:
| _You 've got to support PCI (or something) to find the USB
| controller_
|
| PCI is nothing in comparison to the complexity of USB,
| especially since the BIOS will usually have set up the I/O
| ranges for the devices already. It's literally a few dozen
| bytes of machine code to scan the PCI(e) bus for the device
| you want.
| toast0 wrote:
| Oh sure, PCI isn't too bad, but it's way more work than
| inb(0x60) or inb(0x3f8). (although to be fair, you
| probably want to setup an interrupt, which _is_ real
| work)
| userbinator wrote:
| PS/2 interfaces are actually still quite common for
| "enthusiast" motherboards, because of the low latency (the
| simplicity compared to USB certainly has a lot to do with it.)
| Laptops also tend to have PS/2 keyboards and a "mouse" (the
| trackpad, which can be switched to a proprietary protocol later
| for more features) because the EC they use provides that "for
| free" and that industry tends to have a "it works, why change
| it and break things" attitude.[1]
|
| https://news.ycombinator.com/item?id=17692499
|
| That said, BIOSes can also emulate PS/2 with USB (no doubt via
| SMM). This is usually a setting called "legacy keyboard/mouse
| support".
|
| [1] The notable exception being Apple. Think Different.
| https://news.ycombinator.com/item?id=12924051
| DylanSp wrote:
| Unrelated to the article, but your description of the 15-410
| course piqued my interest. Is there a way to access previous
| versions of the course materials? I took a look at
| https://www.cs.cmu.edu/~410/, but it seems like some parts of
| the site are still being worked on for fall 2021.
| surajrmal wrote:
| Complexity increases over time but it's not for nought. USB,
| while complicated is a great deal simpler than dealing with a
| dozen ill documented buses. Additionally the amount of things
| you can do with it are much greater than what you could do with
| ps/2. The table stakes in general for operating systems is much
| higher which means making a new one from scratch is impractical
| for most folks but this is true for most things. Implementing a
| CPU that's competitive with the ones Intel it ARM is very
| challenging for a college student but I'm not sure I see that
| as a problem. Projects like RISC-V exist to give you a baseline
| to start from for CPU design just as Linux does got os design
| and they give you a fighting chance to compete. That's probably
| for the best, and starting from scratch, while sometimes
| useful, isn't really necessary for most.
| phendrenad2 wrote:
| Unfortunately it isn't that simple. One of the comments on the
| youtube video gets to the heart of the matter:
|
| > I don't see how this might be changed. Most of those little red
| circles are not just "tiny computers" but also different
| implementers in different companies, all trying to protect their
| little fiefdom against their customers
|
| There are of course initiatives like libreboot/freeboot/whatever,
| and efforts to reverse-engineer the firmware of microcontrollers
| in peripherals, and there exists fully documented hardware such
| as the Raspberry Pi 4 (I think?), but even with full
| documentation, it's incredibly hard to write custom firmware, and
| the benefit is very slim (better security perhaps?)
| jillesvangurp wrote:
| That actually goes to the heart of the problem, modern SOCs are
| basically proprietary components gobbled together on a chip
| with some internal networking/wiring that each come with their
| own software that does a lot of stuff before even getting
| around to booting Linux.
|
| The problem is the proprietary nature of this stuff and the
| fact that a lot of it is outside of the scope of scrutiny that
| goes into Linux. It's held together with what basically amounts
| to glorified duct tape. It's complicated. It has weird failure
| modes and occasionally this stuff has expensive issues. Like
| for example security issues.
|
| Very relevant if you want to build an open source phone
| platform or laptop. Not impossible; but it requires addressing
| a few things. Like connecting to a 5G network without relying
| on Qualcomm's proprietary software and hardware.
|
| The real solution is not merely integrating these things as is
| and trying to reverse engineer them but engineer them from the
| ground up to work together and make sense together. That
| larger, open operating system does not really exist. And many
| proprietary equivalents on the market right now have an
| emergent/accidental design rather than something that was
| designed from the ground up.
| Ericson2314 wrote:
| Conway's law at work!
|
| One one hand there is great vertical disintegration, on the other
| hand, even Apple is too lazy to _actually_ do this right, still
| treating their software stack as a black box where breaking
| changes are fine but elegance is lost on the suits.
|
| I suppose the folks at https://oxide.computer/ are smirking if
| they see this video.
| nathanaldensr wrote:
| As a regular-old application developer, I had no idea that
| operating systems were being so aggressively sequestered from the
| perspective of computer architecture. It seems to be quite a
| joke, actually, to call Linux an "operating system." Combine this
| awareness with how much closed-source code is running these chips
| without any knowledge by the operating system and thus the
| user... it seems pretty scary.
|
| I suspect much of the problem is "wall of confusion" differences:
| operating systems are primarily concerned with compatibility and
| reliability while computer architecture seems to be about rapid
| innovation and change.
| kristianov wrote:
| Linux now is basically a UI layer.
| vlovich123 wrote:
| So this has actually always been the case. Hardware firmware
| blobs that initialize the device to talk on the bus yielded to
| EEPROM chips that had SW firmware burned into it to do that
| initialization as the EEPROM chips became cheaper and let the
| bringup process be more flexible. This yielded to
| microcontrollers that had programmable flash (still some EEPROM
| for some parts of HW, but now you can glue multiple pieces
| together). Now you have full-blown ARM cores doing that
| orchestration (but HW still has EEPROM for bug fixes on top of
| the base NOR-flashed code). It's all about flexibility and
| cost.
|
| I don't know though if I fully agree with the characterization
| that Linux has been shuttled off to the corner as chip vendors
| work around it. It's more that the computer has actually always
| been a distributed system and OS/hardware developers have
| ignored that for a very long time. You wouldn't say that Linux
| isn't the OS for distributed clusters even though a lot of work
| goes into making those clusters run and that all typically runs
| in user space outside the OS boundary.
|
| It's possible that there's a better model out there that
| manages to unify things but I'm skeptical. All this code runs
| on different chips with different clock speeds. Code also
| resides in drastically different memory spaces and that's
| unavoidable - an M3 might be able to load the firmware over
| PCIE, but it's not executing out of main memory. Additionally
| these chips typically have drastically different cache
| coherency properties.
|
| Now maybe we need to revisit this all holistically, which I
| think is the actual pitch being made and one I can support.
| That needs to look like defining interfaces for how this stuff
| works (& yes, making it possible to integrate into Linux) and
| getting chip vendors to adopt. Sunk costs are real and trying
| to rework not just HW architecture but also the entire SW
| ecosystem can be a dead end. I'd certainly be keenly interested
| to hear the speaker's ideas. He's certainly far more
| knowledgeable about this space than I am.
| SLWW wrote:
| TRUTH
| 95014_refugee wrote:
| Having seen the coalface, the problem is not the OS, it's the
| (stupendously retarded) application model.
|
| There's very little (interesting) you can do when application
| developers are (for the most part) incapable and unwilling to
| embrace anything even remotely different from the DOS application
| model.
|
| The OS is, at its heart, a mapping from that application model
| onto the hardware. If a hardware feature or characteristic can't
| be expressed / surfaced within the application model, then
| there's little to no value in attempting to exploit it.
|
| Someone like at least has internal customers, and a vertically-
| integrated embedded system can sometimes do better, but general-
| purpose systems are shackled to some minor variation on the POSIX
| application model for the forseeable future.
| klyrs wrote:
| [video] please
| IncRnd wrote:
| The video is at the bottom of the page, once you turn on third
| party frames (or just youtube frames).
| klyrs wrote:
| The page consists of an abstract for the video, and not a
| transcript. Ergo, I'm asking that the story be tagged [video]
___________________________________________________________________
(page generated 2021-09-01 10:00 UTC)