[HN Gopher] Unix-like OS in Rust inspired by xv6-riscv
___________________________________________________________________
Unix-like OS in Rust inspired by xv6-riscv
Author : o8vm
Score : 91 points
Date : 2023-07-25 17:10 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| bakul wrote:
| How does this compare with https://github.com/dancrossnyc/rxv64 ?
| From its README.md:
|
| rxv64 is a pedagogical operating system written in Rust that
| targets multiprocessor x86_64 machines. It is a reimplementation
| of the xv6 operating system from MIT.
| o8vm wrote:
| octox is also different in that the kernel, userland, mkfs, and
| build system are all implemented in Rust, and I think it is
| also different at the type level, for example, octox using mpmc
| for Pipe, and using OnceLock and LazyLock for static variable
| initialization.
| bakul wrote:
| Thanks. rxv64 kernel (the much harder part), syslib and ulib
| are in rust. Replacing userland C with rust would not be
| hard. Ideally a general purpose OS should support programs
| written in any of the common programming languages so it
| would make sense to have some programs in another language to
| keep the OS API "honest"!
| snvzz wrote:
| I immediately notice that o8vm targets RISC-V, whereas rxv64
| targets a legacy ISA.
| xolve wrote:
| OP, would be nice if you add a license to the repo.
| o8vm wrote:
| Thanks! Is this notation in the README not enough:
| https://github.com/o8vm/octox#license ?
| mysterydip wrote:
| What does "written in safe Rust as much as possible" mean? Are
| there functions with no equivalent in rust?
| OtomotO wrote:
| Interfacing with hardware means you have to drop to "unsafe"
| rust at some (few) points.
|
| "Unsafe" rust isn't named good, because it's still safer than
| e.g. C.
|
| Some invariants you cannot break in rust, no matter if "safe"
| or "unsafe"
| dathinab wrote:
| > Some invariants you cannot break in rust, no matter if
| "safe" or "unsafe"
|
| a better description IMHO is that unsafe rust enables
| additional functions which are normally not usable as they
| can create unsoundness if used incorrectly
|
| because this are just additional functions it means also the
| checks and type safety of all other code is still always
| there and normally sound, as long as you don't misuse the
| additional functions to brake invariants
|
| I do count pointer dereferencing as a function converting a
| pointer to a reference in this context, it's technically not
| quite right, but conceptually not really wrong either.
|
| So e.g. unsafe rust don't allow you to write to a &T
| (immutable reference), but using unsafe you could technically
| bit-wise transmute the &T into a &mut T (100% guaranteed
| unsound!!!) and write to that. Still at any point all
| constraints when handling &T did still apply and so do
| constraints of handling &mut T, you just (unsoundly)
| converted one into another using an additional function
| unlocked by using unsafe.
| xeonmc wrote:
| Would it be more aptly named "risky rust"?
| kzrdude wrote:
| Yes! Another apt name would be "trustme". (The normal case
| in Rust is trust the compiler - and all the people who
| wrote "trustme" code that you depend on!)
| tialaramex wrote:
| > it's still safer than e.g. C.
|
| This is arguable and I think overall it's actually harder to
| correctly write unsafe Rust, even if sometimes maybe in some
| sense safer than C when you screw up.
|
| In Rust everything has to obey Rust's semantic constraints.
| For _safe_ Rust that 's fine because the language itself
| promises you're obeying. You can't introduce anything which
| would be a problem, so you needn't even care what those
| problems are.
|
| But in _unsafe_ Rust you are responsible for the same
| guarantees that safe Rust gave everybody. And the rules you
| 're responsible for obeying are _truly difficult_ so that you
| may not properly understand them. If you screw up, that 's
| instantly Undefined Behaviour.
|
| Let's take a fairly old but brutal real example from Rust's
| standard library. core::mem::uninitialized<T>(). This
| function is labelled deprecated (as well as unsafe) in your
| Rust, but once upon a time it was the usual way to make some
| uninitialized buffer in which to construct something.
|
| But it was actually UB almost always+. Because what it says
| is, OK, I know I didn't initialize a T, but trust me, I'll
| sort that out later, lets say this is a T anyway. And for a
| time people persuaded themselves that this is OK for at least
| some types. After all, if T was u8 (a byte) then who cares
| what its value is, any value is valid, isn't it? Well, yes,
| but "uninitialized" isn't a value, it's a 257th possible
| state, the compiler _knows_ we didn 't initialize this, and
| therefore all optimisations are valid even if they wouldn't
| be valid for _any_ possible initialized state of the memory -
| we didn 't initialize it so we're not entitled to assume it
| had any of those values.
|
| In C you will get away with this but in Rust you've created
| Undefined Behaviour, which is not OK. Today you would use the
| MaybeUninit<T> type so that you can explicitly initialize it
| (once you have something to initialize it with) and _then_
| MaybeUninit::assume_init() to get your T instead now that it
| 's initialized, and (if you did it correctly) _that_ is safe.
|
| + If T is a Zero Size Type then this function isn't
| dangerous, because it makes nothing and then says this
| nothing is actually a T, and the compiler says well, thanks
| for telling me, I don't really care but whatever. No UB.
| Likely this only happens in generic code, but it's safe.
| dathinab wrote:
| part of the problem is that rust has not yet a standardized
| memory model (there are candidates, wip)
|
| this means there are limits to soundness analysis tools and
| guardrails you can provide ins table rust
|
| through there had been pretty convincing examples about how
| under some of the (more promising) memory model candidates
| you can provide additional/different functions which are
| much harder to accidentally misuse
|
| and soundness analysis tools do already exist, too
|
| I believe that rust has the _potential_ to make it easier
| to write a lot of unsafe code correctly in rust then in C
| -- in the future.
|
| Through the issue with people using a "it's only bits" mind
| set when doing unsafe code stays around, and is wrong, not
| just in rust but in C, too. No matter how much some people
| try to pretend C is a high level assembly.
| [deleted]
| Santosh83 wrote:
| Why does it seem like 80 to 90% of hobby OS projects that are
| started are "Unix-like?" Don't we already have a huge variety of
| Unix-like OSes out there? Why not explore new models?
| nextaccountic wrote:
| What is an OS good for if not to run programs? You either port
| existing programs to your API (lots of work) or design the OS
| with an existing API in mind. Or forget about existing programs
| and write entirely new programs
|
| "Unix-like" describes the second approach, you provide a POSIX
| API but can have different internals
|
| Now, maybe you wanted to have two APIs: one native API for the
| apps you write, and a translator from another, more popular API
| to your native API. Sounds like a lot of work when almost all
| software that will ever run on your OS will the translator
| vilunov wrote:
| Often the user-space API limits the design of internals and
| available features. I for one would like to see the classical
| hierarchical filesystem gone, but POSIX (and UNIX) at its
| core is about the filesystem.
|
| If you need to run the existing programs, feel free to
| develop a translation layer, at the same time allowing a new
| type of OS paradigm to emerge.
| yjftsjthsd-h wrote:
| Nobody's stopping you from slapping on a compat layer, and
| ex. Haiku does that, but it's more work than just being
| unixy to start with.
| jijji wrote:
| its better than being OS400-like, isnt it?
| kjs3 wrote:
| OS/400 is actually really interesting and very well thought
| out, particularly as an example of "Not Unix". I wouldn't
| want to make my living there, but lots of people do.
| sneed_chucker wrote:
| Because we effectively live in a Unix monoculture world, in
| terms of operating systems that you can actually use and study
| the inner workings of.
|
| It was a legal anomaly that resulted in early Unix (aka
| Research Unix) being distributed with its source code for free
| to universities, which was enough to get the ball rolling on
| stuff like BSD and Lion's annotated V6 source code, that by the
| time AT&T decided that closed source commercial Unix was the
| game it wanted to play, the cat was already out of the bag.
|
| By the time the free software and open source movements had
| gained some ground, enough people had studied or worked on some
| kind of Unix kernel and Userland source code that projects like
| Linux, Minix, and Free/Net/Open BSD were feasible. The fact
| that Linux running on x86 subsequently ate the world was
| probably something few people saw coming.
|
| The other lineages of operating systems, e.g. Windows NT,
| OpenVMS, IBM's various offerings, either never have their
| source released or only have their source released long after
| they're obsolete.
| bsder wrote:
| Because the only large, available, free ecosystem of software
| (both applications and hardware drivers) is completely built
| around the Unix abstractions.
|
| If you don't want to do Unix, you have to reduplicate _ALL_ of
| it. And that 's like trying to boil the ocean.
| chaxor wrote:
| I think the main selling point for Unix-like OS, but in Rust,
| is focused on Rust. It's to ensure that the memory related
| errors are less likely, and hopefully with enough work, the
| system can be essentially what we have today, but with less
| CVEs.
|
| It's honestly a decent goal and I'm in support of it.
|
| I know that there will inevitably be many now that come to
| state the obvious "well it doesn't _guarantee_ safety " and
| "there are other reasons for CVE", etc. Nonetheless, it's not a
| bad idea.
| packetlost wrote:
| We need more Plan 9-likes!
|
| But more seriously, the Unix/POSIX-like OS is a pretty good
| model, but IMO we could do better, especially in the
| microkernel front.
| o8vm wrote:
| I would like to implement a plan9-like OS someday!
| bakul wrote:
| I'd love to see an OS with a Unix API but plan9 like kernel!
| I will never get around to writing one myself though!
| boricj wrote:
| There are a lot of things wrong with plain standard POSIX:
|
| - it's hopelessly out of date and incomplete w.r.t. modern
| expectations
|
| - fork() is very problematic for multiple reasons, especially
| in a modern environment (https://www.microsoft.com/en-
| us/research/uploads/prod/2019/0...)
|
| - process management functions take PIDs and not
| capabilities, which can cause race conditions
|
| - POSIX hasn't standardized anything better than poll(), yes
| it works fine in a hobby context but it's not 1987 anymore
| (and don't get me started on select():
| https://github.com/SerenityOS/serenity/pull/11229)
|
| - signals are a mess
|
| - Unix processes have a huge amount of ambient authority,
| which is problematic when trying to isolate them (chroot(),
| https://fuchsia.dev/fuchsia-src/concepts/filesystems/dotdot,
| ...)
|
| - the C POSIX library has a lot of cruft while also missing
| stuff what programmers actually need (for example, POSIX took
| nearly 6 years to standardize strlcat()/strlcpy(), the
| process itself starting 17 years after OpenBSD introduced
| those functions:
| https://www.austingroupbugs.net/view.php?id=986)
|
| - ...
|
| Granted, modern production-grade Unix-like operating systems
| have extensions to deal with most of these issues
| (posix_spawn, kqueue, pidfd_open...), but they are often non-
| standard and can be quite janky at times (dnotify, SIGIO...).
| It also doesn't fix the huge amount of code out there using
| the legacy facilities like it's still the 1980s.
|
| There are other models out there (Plan 9, Windows,
| Fuchsia...), but what we really need is to stop putting
| Unix/POSIX on a pedestal like some holy scripture that shall
| not be questioned. It's the pinnacle of 1970s operating
| system designs and it has fossilized so much it's actively
| turning into oil.
|
| Or at the very least, please stop teaching the next
| generation that fork() is the greatest thing ever. It's a 50
| year old hack kept alive through gratuitous amounts of copy-
| on-write that should've been scrapped the day Unix was ported
| to computers with virtual memory and paging.
| cdcarter wrote:
| > Or at the very least, please stop teaching the next
| generation that fork() is the greatest thing ever.
|
| Who is saying this? fork() is widely maligned.
| howinteresting wrote:
| My OS class definitely didn't characterize fork() as
| maligned, just as a powerful building block (which it
| is). It took me professional experience to realize it
| sucks.
| boricj wrote:
| That's not my experience as a French student in the early
| 2010s. At the very least, I remember that the only
| process creation model I saw in class was fork()+exec()
| and there were no disclaimers about it.
| yjftsjthsd-h wrote:
| This feels contradictory; unix isn't good enough, so nobody
| implements standard unix, but people need to stop putting
| unix on a pedestal?
| zvmaz wrote:
| There's XINU (Xinu's Not Unix). There's a book that walks you
| through the complete implementation of the OS in C [1].
|
| [1] https://xinu.cs.purdue.edu/
| smasher164 wrote:
| It's hard enough to learn osdev. If you have a concrete design
| to fall back on, you can focus on implementation.
|
| Coming up with an original design for OSes is like a 3rd
| project thing.
| crickey wrote:
| What is even Unix-like ? I usually associate it with POSIX but
| maybe thats naive
| linguae wrote:
| Through various market forces, Unix (and its
| descendants/clones) and Windows killed most of the rest of the
| OS ecosystem over 20 years ago. There are generations of
| software engineers and computer scientists who've never studied
| operating systems that weren't Unix- or Windows-based. Most
| leading OS textbooks (Tanenbaum's books, the dinosaur book,
| Three Easy Pieces) have a Unix slant. Even the systems software
| research community is heavily Unix-centric; I say that as
| someone who used to be immersed in the research storage systems
| community. The only non-Unix or Windows operating systems many
| practitioners and even researchers may have used in their lives
| are MS-DOS and the classic Mac OS, and there's a growing number
| of people who weren't even born yet by the time these systems
| fell out of common use.
|
| However, the history of computing contains examples of other
| operating systems that didn't survive the marketplace but have
| interesting lessons that can apply to improving today's
| operating systems. _The Unix Hater's Handbook_ is a nice
| example of alternative worlds of computing that were still
| alive in the 1980s and early 1990s. VMS, IBM mainframe systems,
| Smalltalk, Symbolics Genera, Xerox Mesa and Cedar, Xerox
| Interlisp-D, and the Apple Newton were all real-world systems
| that demonstrate alternatives to the Unix and Windows ways of
| thinking. Project Oberon is an entire system developed by Wirth
| (of Pascal fame) whose design goal is to build a complete OS
| and development environment that is small enough to be
| understood for pedagogical purposes, similar to MINIX but
| without any Unix compatibility. Reading the history of failed
| Apple projects such as the planned Lisp OS for the Newton and
| the ill-fated Pink /Taligent project are also instructive.
| Microsoft Research did a lot of interesting research in the
| 2000s on operating systems implemented in memory-safe
| languages, notably Singularity and Midori.
|
| From learning about these past projects, we can then imagine
| future directions for OS design.
| lproven wrote:
| Beautifully put. I couldn't have said it better myself.
| mananaysiempre wrote:
| How would one go about exploring "other" real-world systems?
| I've read some old OS textbooks, poked at Symbian and OS/2
| books some, and have texts on Oberon and Symbolics in my
| queue, but the docs for RSX-11 and VMS seem to bury me in
| operational minutiae without really explaining the design
| choices, and the Multics docs look like a huge pile of
| research notes, which is going a bit too far in the other
| direction. The current bytecode on IBM i is apparentily
| outright NDA'd, and the RPG docs are eager to presume I know
| how to operate the original punch-card tabulators. Any
| pointers?
| linguae wrote:
| Depending on the system this could range from an easy
| project that one can do in a few hours to a very long
| quest. For some research systems (particularly those
| implemented in the pre-FOSS era or those from companies) it
| may be impossible to try out the systems without attempting
| to implement them yourself. However, there are other
| systems that are available to try out. The ideal situation
| is a FOSS release, like the MIT CADR Lisp machine code and
| Interlisp-D. Barring that, the next best thing is some type
| of freeware or non-commercial license; I think this is the
| case for OpenVMS, though I could be mistaken. Some systems
| cannot be obtained easily without traveling the high seas,
| if you catch my drift, matey (cough Open Genera cough).
|
| I think a lot of value can be gained from not only using
| software, but by reading papers and manuals, especially
| when the software is unavailable or unattainable. There's a
| site called Bitsavers that is a treasure trove of old
| documents.
|
| Come to think of it, a significant reason for Unix's
| dominance in research and education is its availability,
| going all the way back to the 1970s when Bell Labs sold
| licenses to universities at very low prices. Even when
| licensing became more restrictive in the 1980s, this
| spurred the development of MINIX, GNU, later BSDs, and
| finally the Linux kernel, in order to recreate the
| environment students enjoyed with 1970s Unix. This openness
| is a far cry from the work of Xerox and Lisp machine
| vendors, where one needed to have been privileged enough to
| work for one of these vendors or their customers to use
| these environments, which were far more expensive than Unix
| workstations and especially personal computers. Thankfully
| there's a wealth of documentation about these non-Unix
| systems, as well as testimony from their users. In
| addition, some systems were open sourced. But we must
| remember the times that these systems emerged, and we must
| remember why Unix became so dominant in the first place;
| its openness and availability set it apart from its
| competitors.
| jjjfdjunnmko wrote:
| Your best bet is probably the old MVS that was open sourced
| back in the 70s or 80s and lives on via an emulator called
| Hercules. Its sufficiently different to be worth a look,
| it's got a living direct descendent (although that has
| become increasingly turned into a Unix over the years), and
| there is enough of a community around it due to it having a
| very expensive direct descendent that the docs and other
| tools necessary for working with it are relatively high
| quality and up to date in comparison to most long dead
| operating systems.
|
| The issue with it is that there is no way around having to
| do some pretty serious assembly language work for an
| instruction set that no longer exists and whose modern
| descendents probably microcode to hell in order to run some
| of the old instructions.
| flyinghamster wrote:
| Another key issue: the concept of open source barely existed.
| Sure, you got a copy of the source code used to SYSGEN your
| RT-11 or RSTS/E system, but it had comments removed and was
| not for redistribution. In the end, the closest in feel to
| the old DEC world today would be the traditional Windows
| command line (not PowerShell).
|
| The rise of GNU and the general open source movement was a
| reaction to the rug-pull when access to Unix source was
| restricted, and that gave us Linux (MINIX wasn't initially
| open source, but could be put into a state that made Linux
| possible).
|
| The Unix paradigm stuck with us, IMO, because Unix (and later
| Linux and BSDs) have been ported to so many vastly different
| architectures. So many other operating systems have never
| moved beyond (and often died with) their initial platforms.
| amedvednikov wrote:
| Project Oberon is amazing. Shame it isn't more popular.
| kjs3 wrote:
| Why does it seem like 80 to 90% of hobby OS projects that are
| announced here as "Unix-like" invariably get _someone_ who
| replies with "that thing you're doing as a hobby, for
| fun...you're doing it wrong and I don't approve". Zero content,
| zero insight posts about someones else's toy aren't useful. You
| want a 'new model', start coding.
| guerrilla wrote:
| If it ain't broke, don't fix it.
|
| All kinds of new OS ideas can be implemented on UNIX like
| Mach[1], FLASK[2] and 9P[3] while internally a UNIX-like system
| doesn't need to be anything like a UNIX[4]... So who cares?
| What are you worried about losing? What can't be implemented on
| a UNIX-like system?
|
| 1. see MacOS
|
| 2. see SELinux
|
| 3. see v9fs
|
| 4. see Windows and BeOS which both have POSIX APIs
| josephg wrote:
| The problem is that anything you build on top of Unix will
| always be a second class citizen in the Unix world.
|
| For example, suppose you want a database-like filesystem.
| Either you implement it in the kernel, and now your special
| apps barely work on anyone's computers. Or you implement it
| in userspace - preferably as a library. And now your apps can
| run anywhere without special kernel features but the
| terminal, and all the other applications on the computer
| can't / won't understand your new abstraction. And you'll be
| fighting an uphill battle to get anyone to care about your
| new thing, let alone integrate it.
|
| It's like saying - why rust? Why not just add a borrow
| checker to C? Why didn't C# just add a garbage collector to
| C++? Sometimes starting fresh and establishing a clear, clean
| environment with different norms is the most effective way to
| make something new. You don't have to fight as many battles.
| You can remove obsolete things. You don't have to fight with
| the platform conventions, or fight the old guard who like
| things as they are.
|
| It's a shame with operating systems that modern device
| drivers are so complicated. Hobbyist operating systems seem
| inordinately difficult to make these days as a result, and
| that's a pity. There's all sorts of good ideas out there that
| I'd love to see explored.
| guerrilla wrote:
| Nobody's stopping you from doing that but apparently none
| of OP's ideas required that, nor has it been worth it for
| anyone else's yet either. If it ain't broke, don't fix it.
| Let me know when it's actually broke in this actual
| reality. Then we can start over*.
|
| Also all those languages are Cs in the same way BSD and
| Linux are UNIXs. Same family. You should have mentioned
| Haskell or APL instead.
|
| * Note that many experiments did start over, e.g. Plan9,
| but were then integrated into a UNIX.
| LoganDark wrote:
| Didn't Windows drop the POSIX APIs and then eventually
| introduce WSL instead?
| lmm wrote:
| Unix is a lot more amenable than most older OSes to being
| implemented by a disparate group of people with limited
| communication, hence why GNU was originally set up as a
| reimplementation of unix.
| im_down_w_otp wrote:
| We created a funky little OS on top of seL4 & Rust that's most
| certainly not Unix-like, and is more akin to an RTOS-like
| approach to building & bundling software. More for purpose-
| built appliances than a general purpose OS.
| cnuts wrote:
| That's really cool, great job!
| prydt wrote:
| Awesome! What are some good resources on making a simple UNIX
| like operating system? I know osdev wiki exists, anything else?
| o8vm wrote:
| Thanks!For me https://pdos.csail.mit.edu/6.S081/2020/xv6/book-
| riscv-rev1.p... was quite helpful! but Someday I will write a
| book on how to implement this OS step by step in Japanese.
___________________________________________________________________
(page generated 2023-07-25 23:00 UTC)