[HN Gopher] Starting Over: A FOSS proposal for a new type of OS ...
___________________________________________________________________
Starting Over: A FOSS proposal for a new type of OS for a new type
of computer
Author : lproven
Score : 81 points
Date : 2021-02-08 16:38 UTC (6 hours ago)
(HTM) web link (liam-on-linux.livejournal.com)
(TXT) w3m dump (liam-on-linux.livejournal.com)
| [deleted]
| Animats wrote:
| It's not that new an idea. Keeping everything in persistent
| memory has been tried quite a few times. LISP went down that
| road, with Interlisp and Symbolics. Everything is in one
| persistent LISP image, and you save entire images if desired.
| Symbolics had some extra hardware support to tag objects in
| memory and get some level of pointer safety.
|
| OS/400, and to a lesser extent the B5500 and successor machines,
| went down that road too. That model was "everything is a
| relational database." OS/400 had a 20 year run; released in 2008,
| end of life in 2018.
|
| Other ideas that never really caught on include hardware
| key/value stores, or "Smart RAM". That may come back, since
| accessing non-volatile RAM through disk drivers is inefficient.
|
| The talk seems mostly focused on GUIs, which is a separate issue.
|
| If you want to think about a new generation of operating systems,
| think about what to do in the GPU space. That's the future of
| hardware. Single CPUs have hit the wall on speed, but there's
| plenty of room for GPUs with more functional units.
|
| GPU programs are almost stateless. Very functional. But at some
| level you need state. How should that interact with a near-
| stateless GPU? Big unsolved problem.
| hctaw wrote:
| Real applications don't hit CPU walls, they're bottlenecked by
| I/O more often, and the biggest CPU wall is memory accesses.
| There is a lot of room for innovation there if you shirk off
| POSIX compatibility.
|
| Also, state is not a problem for functional semantics. Remember
| "functional" is a semantic distinction, it doesn't imply a ton
| about optimized implementations of those semantics. For a
| stateful system the only thing different about a functional
| approach is the programmer cannot express a program where
| yielding next state invalidates current state. Most of the time
| a sound, functional program can be compiled into in place
| (mutable) operations on the state, the difference is they're
| free of logical errors.
| djrogers wrote:
| > OS/400 had a 20 year run; released in 2008, end of life in
| 2018
|
| Err, setting aside the math inconsistency there, AS/400 was
| released in 1988, and the 2 most recent versions are still
| supported..
|
| [0] https://www.ibm.com/support/pages/release-life-cycle
| Animats wrote:
| Sorry, I meant 1988. My bad. Wikipedia has old info about
| versions and end of life.
| ChuckMcM wrote:
| You are making the author's point for them :-). Now add in that
| some of the stuff that was "custom hardware" that supported the
| concept of 'always resident' OS state is now off the shelf
| (specifically Optane/Crosspoint stuff) Of course Intel killed
| of consumer Optane for SSDs so one wonders what is going on
| there.
|
| The second point, about how to think about GPUs in the context
| of "modern" computing is, for me, the more interesting one. As
| mobile processors have for years been a GPU with a processor
| accessory laptops and desktop computers seem to be headed that
| way as well. To be honest I'm surprised there isn't a GPU
| "motherboard that you can plug one or two CPU daughter cards
| into.
| juancampa wrote:
| I wasn't aware that intel killed Optane-Only SSDs for
| consumers so I looked up and here's a recent article about
| it: https://www.tomshardware.com/news/intel-kills-off-all-
| optane...
| hinkley wrote:
| I was stuck in the planning phase for a pet project up until a
| week ago, for a reason why Lisp and Smalltalk always made me
| nervous.
|
| Bugs can put your persistent state into uncharted territory,
| and there may be no clear path back. There's a reason we still
| have 'turn it off and back on again' in our bag of tools. Often
| it's the only thing that reliably works. This makes systems
| designed never to be shut down and turned back on again deeply
| unimpressive to the more practically minded members of the
| audience.
|
| One of the places persistent state crashes and burns is when
| the system of record and the source of truth combine. Once the
| source of truth breaks we can't know what's true anymore.
|
| My breakthrough was figuring out that I'm trying to get data
| from something that is already a pretty good system of record,
| I can just keep letting it do that job indefinitely. My source
| of authority needs to transform that data, not own it. As long
| as I can detect out-of-band changes to the system of record
| (which I can), I can always rebuild my models from scratch.
| That allows me some excellent test fixtures. That also gives me
| the option to do manual 'surgery' on the system of record
| rather than sinking my roadmap to implement a full feature to
| do something I might not need to do again for another year, or
| that no customer will ever see.
|
| I am just on the same continuum as everybody else. My code
| needs to make a lot of decisions. I can't afford to make them
| all from first principles every time. I need to store
| intermediate values. I also need to identify intermediate
| values and question them. If I don't get help with this from my
| architecture, myself and coworkers will blur the lines every
| time a problem seems a little beyond our abilities, and
| eventually nobody will know what's true anymore except the
| delusional ones. How do I know this? Because I have never seen
| any other outcome. The only differences are in how much
| surprise the team exhibits.
| klodolph wrote:
| > There's a reason we still have 'turn it off and back on
| again' in our bag of tools. Often it's the only thing that
| reliably works.
|
| An alternative perspective--at the very large scale, the
| opposite may be true. Turning something off and then on again
| may be one of the _worst_ ways to fix problems with certain
| large systems.
|
| At a large scale, you would document and manage the _state_
| of the system, make changes which put that system into a new
| state, and build tools that monitor and enforce system state
| (declaratively where possible, but it's not always possible
| to be declarative).
|
| Sometimes there will be duplicated effort between making sure
| a system runs smoothly and making changes while it's running,
| and making it so the system can turn off and on again. Think
| about the duplicated effort between setting up a new database
| and making schema changes to a live production database.
|
| YES, you want to have a source of truth. But the "turn it off
| and on again" source of truth is usually a sequence of
| imperative commands rather than a description of the desired
| system state, and comes with its own reliability problems. A
| description of system state as source of truth has a
| _different_ set of reliability problems but I think it's
| absolutely necessary to explore the solution space between
| these two extremes--shooting for a completely declarative
| system is futile, but an imperative system can be very hard
| to reason about.
| guidoism wrote:
| Yeah I think the whole "need to reboot to get to a clean
| state" is maybe a bug not a feature.
|
| It is certainly possible to build systems that can run for
| years or decades.
|
| I'm guessing that persistent memory will require us to
| think about changes to state in memory are more like ACID
| in databases. We should think about rollback instead of
| reboot.
| hinkley wrote:
| > is maybe a bug not a feature.
|
| Once you start thinking of safety as a feature you've
| ruined your ability to think responsibly about anything
| else. A reset is a safety mechanism. You shouldn't need
| it, but only a sociopath would try to get rid of it.
|
| Almost all of the software we write is now affecting
| people's lives. That your bug may eat someone's term
| paper has been a problem for decades, and nobody ever
| internalizes how that affects that person's life. The
| avalanche of cause and effect may never settle down. Now
| we can get people fired. Ostracized. We are trying to
| take the power but not the responsibility, and it's
| bullshit. No, hypocritical bullshit, because we spend a
| lot of time here complaining about others doing the same
| thing.
| hinkley wrote:
| Sometimes the solution is different because the scale is
| different.
|
| Sometimes, the solution is different because we think we
| understand a problem that we really don't. I'm not ready to
| allow that the current wisdom on system design is not what
| Tony Hoare was thinking of when he said, "a system so
| complex that there are no obvious problems".
|
| Arguing such things with people only seems to stick when we
| are collectively in the Trough of Disillusionment, which
| may be coming up pretty soon. So ask me again in 18 months.
| ww520 wrote:
| Wifi routers have this notorious problem with the nvram.
| Sometimes some state in the nvram is corrupted and the
| routers would stop working. Turning off and on again won't
| work. A factory-reset to wipe everything out is the only way
| to get out of the corrupted state.
| justizin wrote:
| I think this is really interesting, but I don't see any reason
| why you'd abandon the idea of a filesystem. UNIX has been
| consistently useful for all sorts of purposes largely because of
| the everything-is-a-file concept.
|
| Are we going to keep only a single copy of everything in
| memresister or whatever? someone will reinvent raid, and someone
| will reinvent the disk controller.
|
| i'm sure this tech will change all kinds of things but why
| wouldn't i want a simple hierarchial naming system to refer to
| all of the different blobs i have?
|
| that said, i'm sure i'll watch this!
| klodolph wrote:
| The filesystem, at least the Unix concept of it, is nice in
| theory but surprisingly ill-suited for anything besides, say,
| batch processing, once you start to look more closely at it.
|
| If you look at projects like SQLite or PostgreSQL, you'll see
| all the fantastic, _precise_ logic you need in order to ensure
| expected levels of consistency when you are using real
| filesystems to store data in a database. These problems come
| with serious enough pitfalls and traps that it's common to see
| blanket recommendations to avoid directly interacting with the
| filesystem altogether, and simply do everything through SQLite
| or another library when possible.
|
| At the other end, if you look at GUI applications, the
| filesystem is a bit lower-level than it should be. At least,
| the Unix concept of filesystem is too low-level. If I open a
| file in a word processor, save it, shouldn't I be able to move
| around the file and rename it in the file browser, without
| affecting the relationship of that file to the word processor?
| This is possible on a Mac, but this is done with APIs inherited
| from the pre-Unix days of macOS.
| johnvaluk wrote:
| A file is more than simply a unit or collection of data in a
| local hierarchy. It's an abstraction that supports the
| copying, exchange and even possession of data. How is one
| supposed to share part of a database with another system or
| user without ultimately resorting to the file metaphor?
| loudmax wrote:
| I applaud the drive and vision to experiment with a fundamentally
| different operating system paradigm. But even if your persistent
| storage is RAM, I don't see why you wouldn't still want a
| filesystem. Filesystems arose as a method of organizing data.
| However fast your access time, you still want that. Maybe you
| could have some kind of database backed tagging system instead of
| a directory hierarchy, but the benefits and drawbacks of such a
| system should apply to an SSD or even HDD as much as to something
| in RAM. This goes for user data (eg. photos, documents) as well
| the organization of the operating system.
|
| Anyway, I'm curious to see what what this project comes up with.
| I suspect it will be something that looks and functions a lot
| like a filesystem, but I'd be happy to be proven wrong.
| blackrock wrote:
| Even AWS S3 is a key-value storage system. Meaning your key can
| be anything.
|
| But over time, people tend to name the key using a path
| structure, separated by forward slashes, to give a logical
| separation of data into subdirectories.
| karmakaze wrote:
| Advances in performance are largely about impedance matching,
| from sequential access tapes to registers in a CPU. You want to
| use an efficient interface and abstraction for the level you're
| at. The direction of progress is toward faster, more randomly
| accessible with less seek cost: HDD -> SSD -> XPoint, etc. This
| totally makes sense--using RAM through a disk interface (e.g.
| RAM disk) is useful if you want a faster disk, to get something
| faster change the interface to take advantage of the medium.
| hderms wrote:
| I think the main benefits would be rethinking the core of
| persistence to be more about atomic operations on
| datastructures as opposed to POSIX filesystem IO.
|
| I personally believe that files are still a useful metaphor,
| and this hypothetical OS that we're discussing would likely
| still provide them, at least for interop.
|
| An example being a database. Provided we didn't have to batch
| writes, be aware of dirty pages, write things in certain order
| to maintain coherence in the event of a crash, etc... we could
| build something really simple. Ultimately it would depend a lot
| on the details like memory bandwidth+latency between DRAM and
| NVRAM
| guidoism wrote:
| I see what you are saying but I think it misses the point. We
| _need_ a filesystem right now because data disappears from RAM
| when electricity goes away.
|
| With persistent memory you don't need that extra
| serialization/deserialization step to/from the filesystem.
| Look, think it like this:
|
| 1. In memory we store our program code and data as blobs in
| various parts, in the stack, in a heap, some only for the
| current program, other stuff like shared libraries linked at
| runtime to various programs. It's a rats nest of pointers.
|
| 2. When we "save" or "restore" we takes that rats nest and pull
| out some of the important parts (e.g. we ignore the shared
| libraries in memory and the app code) and we serialize that
| state out to disk.
|
| 3. Then the file system takes that logically contiguous "file"
| and breaks it up into pieces and stores those across the disk
| in various blocks and writes metadata which describe how to put
| the pieces back together again.
|
| With persistent memory we can ignore steps 2 and 3 most of the
| time, but they still might be useful for sending data to other
| computers, especially step 2.
|
| Disclaimer: I read the slides but didn't watch the talk
| agalunar wrote:
| I've never used Multics myself, but from what I understand it was
| also based on the idea of mapping files in memory. When I first
| heard this, I thought the idea was pretty neat, but then the
| person I was speaking to remarked that it wasn't all that great
| in practice. I'm not really sure why, but I'm curious.
| tenebrisalietum wrote:
| `mmap()` sucks if you don't have a lot of addresses. 64-bit
| computers now have a lot of addresses.
| agalunar wrote:
| I'm skeptical this was the reason, though (at least in and of
| itself). Going off of
| https://en.wikipedia.org/wiki/Multics#Novel_ideas, it seems
| that the address space (36 address bits = 36 bits/word x 236
| / 8 bits/byte = 288 GiB) was much larger than memory of the
| machine Multics was designed for (about 2 MB), and the size
| of a segment was limited to about 1 MB (36 bits/word x 218 /
| 8 bits/byte), nearly half of all memory - so I doubt this was
| troublesome at the time.
| moonbug wrote:
| because now you've turned failed IO operations from an errno
| into a segfault.
| egypturnash wrote:
| I feel like I have so many problems with this. Like, look, I have
| some huge projects in Adobe Illustrator and they are _quite_
| deliberately spread out among a ton of files because anything
| over a certain size is just _asking_ for working on it to become
| super-slow, and for some obscure edge case to completely trash
| the file.
|
| I'm skimming the PDF of his slides and I'm just seeing a long
| list of quirky environments that have continued to be dead-ends.
| Maybe he talks about how this would actually work with huge power
| tools if I watch the talk, maybe he grapples with issues like
| "programs crash" and "iterative backups are good". Maybe he even
| deals with things like "files are a metaphor based in the
| physical world and perhaps the reason we keep on coming back to
| them is because they provide a good way to say that _this_
| particular piece of information is over _here_ ". I dunno, I'm
| not seeing suggestions of anything that makes me want to invest
| the time in listening to the talk.
|
| I wish him the best of luck, maybe he will be the one to finally
| find a way out of the world of files to something so clearly
| better that it's worth dealing with the raw edges of it only
| being a few years old vs a modern GUI atop a filesystem, but I
| sure ain't gonna hold my breath.
| austincheney wrote:
| The author is conflating hardware and logic. A hard drive is
| hardware but a file system is a logical tree. The author provided
| simplistic reasoning for abandoning a type of hardware but no
| reasoning for abandoning file systems.
|
| If not a file system then what? Don't say RAM, because that is
| still hardware.
| chubot wrote:
| Doesn't iOS have some kind of persistent memory abstraction? It
| seems more like a single level to me.
|
| Safari on iOS saves tons of tab state going back months, but it's
| not using all your memory, etc. I have never really done iOS
| programming so I don't know the details.
| astrange wrote:
| That's called restorable state and it's better than assuming
| everything can be always persistent, like Lisp/Smalltalk images
| did.
|
| For one thing, restarting all the time makes it less fragile.
| You don't have to untangle your state that's been persistent
| for months if there's a bug in it.
| jarym wrote:
| For me at least this link breaks the browser back button and
| prevents me from coming back to HN (iOS Safari)
| astrange wrote:
| The problem with starting over is that you don't understand how
| the current state came to be, so you'll probably just reinvent it
| instead of getting anywhere new. See Chesterton's fence.
| GekkePrutser wrote:
| Intel actually discontinued Optane for consumers so this won't
| really fly... At least not yet.
|
| I do think a completely out-of-the-box rethink of computing could
| lead to very interesting results though!
| trasz wrote:
| Persistent memory sounds whole lot like OS/400.
| s314159265358 wrote:
| Doesn't this almost say no to the con Neumann architecture? Is
| this a good idea? Isn't the point of having multiple files to
| simplify managment and reduce risk of corruption?
| qayxc wrote:
| The concepts are orthogonal. Von Neumann machines can be used
| without file systems and vice versa.
|
| Persistent state has its pros and cons - for me the ability to
| reset the machine's state is an important feature. For the
| author this doesn't seem to be the case.
|
| As for abandoning the concept of a file system, that's
| something that has been worked on for decades. Microsoft even
| wanted to include this as the main feature for Windows Vista,
| back in 2003 (dubbed "WinFS"). They basically wanted to replace
| traditional files with a single relational database backed by
| schemas for describing objects.
|
| This would've been a truly revolutionary concept, as it
| would've allowed for features like searching, sorting,
| grouping, and versioning of objects without any complex
| application-level code.
|
| Another advantage of such system would be the ability to
| exchange data between apps by just passing an id. All required
| meta-information would be available through the schema and the
| app could choose a view that suits its needs: providing the id
| of an audio file could result in the title and artist for a
| word processing program, the audio data for a music player, or
| the associated cover art in case of an image editing program.
|
| There are many options for replacing the concept of files and
| directories that are just as robust and provide extra features.
| The general architecture of the CPU is not affected by or
| related to this.
| zokier wrote:
| I went through the slides, saw bunch of pictures of old
| computers, but where is the proposal?
| isaacimagine wrote:
| The Google Doc has the proposal at the bottom. Basically, take
| the bottom half of Oberon and slap on the top half of Squeak*,
| _bara-bim bara-boom_ - you 've a next-gen OS.
|
| *Or SBCL + Dylan
|
| The whole schpiel is to make a high-level interactive OS /
| computing environment that straddles the line between
| interpreter and compiler. The whole is is run in-memory, and
| can be snapshotted and restored quickly. No filesystem, just
| RAM.
|
| Interesting idea, I guess :)
| zepatrik wrote:
| I also don't quite get why you would ditch files/directories
| which are just a way of organizing data. You still have to
| organize them if everything is in main memory. Also, sharing data
| between programs has to be possible somehow. You will need some
| kind of reference/pointer to the data you talk about. Why are
| directories, files, and paths such a bad idea for that?
| scroot wrote:
| To take the Smalltalk angle on this, instead of files you
| simply have everything in the system described as live objects.
| These are different from files because they are not just data.
| A system comprised of such objects has no "applications" or
| "programs" in the conventional sense -- you just have certain
| arrangements of objects interacting with each other. This is
| much more flexible, dynamic, and explorable than just having
| files for data and stovepiped programs that read those files.
| astrange wrote:
| Apps would still appear in a system like this as soon as you
| have third-party developers, as a natural consequence of
| Conway's law. And once you have that, there's also security
| and principles like the rule of least power that motivate the
| current design which you haven't gotten rid of.
| boris wrote:
| To me things that are worth starting over would be the CPUs that
| we no longer understand and the OSes that are debugged into
| existence. Not a computer that has no SSD because memory is
| persistent.
| qayxc wrote:
| Both OSes and complex CPUs are a direct result of their
| capabilities.
|
| I'd wish physics would end at Newtonian mechanics and Maxwell
| equations, but alas special- and general relativity cannot be
| avoided in some circumstances and neither can quantum mechanics
| in others.
|
| The same is true for CPUs and operating systems - if you need
| high performance and the ability to do real-time high-
| resolution media playback, digital content creation, high-
| bandwidth networking, high-colour and high-resolution graphic
| displays with font smoothing, auto-scaling, multi-monitor
| support, pluggable peripherals, etc. etc. you'll get
| complexity.
|
| There's only so much that can be achieved with simple TTL and
| core memory...
|
| Microkernels and provably correct OSes are great and available,
| too. But things start to get messy quick once you add support
| for various protocols, devices, and capabilities. Not just
| because it gets harder and harder to do, but also because
| there's diminishing returns: the vast majority of users aren't
| OS developers and just don't care.
|
| The same way the vast majority of people don't care about the
| layout, build quality, and logical soundness of the plumbing in
| their homes (as long as it works sufficiently well) or the
| technical details of their refrigerator.
|
| That's why there's very little incentive to build small,
| "clean", and provably correct OSes for the general public. You
| will find them, though, and they are in use.
| dshpala wrote:
| Surprised no one mentioned
| https://en.wikipedia.org/wiki/Phantom_OS
| fl0wenol wrote:
| This is basically how the Palm Pilot worked. The RAM in the
| system was a developer-invisible cache that sat on top of a
| record-oriented access layer on top of the nonvolatile flash and
| system ROM.
|
| I'm simplifying slightly but it's the same idea. It worked okay
| but had its quirks.
| mongol wrote:
| Did not also Newton have something of that kind?
| andrewshadura wrote:
| In fact, Palm Pilot didn't have proper non-volatile memory,
| instead, it had static RAM, which would preserve data for some
| time even if you swapped the batteries.
| martinskou wrote:
| I have had this thought about much web software these days.
| Requesting a lot of data from DB instead of just keeping a
| persistent object system in the memory.
| tzs wrote:
| This part:
|
| > A possible next evolutionary step for computers is persistent
| memory: large capacity non-volatile main memory. With a few
| terabytes of nonvolatile RAM, who needs an SSD any more? I will
| sketch out a proposal for how to build an versatile, general-
| purpose OS for a computer that doesn't need or use filesystems or
| files, and how such a thing could be built from existing FOSS
| code and techniques, using lessons from systems that existed
| decades ago and which inspired the computers we use today.
|
| reminds me a bit of something I've long wanted to experiment with
| (or see someone else experiment with). Here's how I described it
| in a comment here on HN a while back [1]:
|
| > I've toyed with the idea of replacing files with processes. If
| you have some data that you want to keep, you have a process that
| holds it in its process memory, and can give it other process via
| an IPC mechanism (if the other process is local) or over the
| network (if remote, although you could of course also use the
| network locally).
|
| > I never got around to trying it out. I think I may have tried
| to start some discussion on usenet along these lines maybe 10-15
| years ago, but no one seemed interested.
|
| > A "directory" would simply by a process that provides some kind
| of lookup service to let other processes find the data storage
| processes that contain the data they are looking for.
|
| > You'd still have disks on your computer, but they would be
| mostly used as swap space.
|
| > The system would include some standard simple data holding and
| directory processes that implement a Unix-like namespace and
| permission system, but it would be easy to override this for data
| that needs special treatment. Just write a new data holding
| program that implements the special treatment you want and knows
| how to register with the standard directory processes.
|
| [1] https://news.ycombinator.com/item?id=8311532
| gcblkjaidfj wrote:
| decades ago i was following a kid's graduation project of a OS
| with full permanence design. that was before Android and other
| always on "computers".
|
| it was called something like unununium (yeah, same name as the
| element, for extra hard-to-search points). I noticed the project
| when he suggested unununium-time as an alternative to linux time
| in a list was at, to fix some time skew problems... But what
| caught my interest was his vision that in the near future
| (remember, before android/IOS) computers would not care about
| offline data storage and a OS should be optimized for always-on
| and RAM only.
|
| I can still find some of the assembly versions, but the fun stuff
| and interesting ideas showed up on a rewrite in python(!) and
| that i can't find anything any more.
|
| edit: here's the best i could find
| https://web.archive.org/web/20060208191407/http://en.wikiped...
|
| seems the vision was to threat the persistent storage as the only
| system memory.
| moonbug wrote:
| parahrqph five before mentioning lisp and Psion. pretty
| restrained for Liam!
___________________________________________________________________
(page generated 2021-02-08 23:00 UTC)