[HN Gopher] How is the Linux kernel tested?
___________________________________________________________________
How is the Linux kernel tested?
Author : sprado
Score : 88 points
Date : 2020-03-27 11:46 UTC (1 days ago)
(HTM) web link (embeddedbits.org)
(TXT) w3m dump (embeddedbits.org)
| ndesaulniers wrote:
| We run our own ci for building Linux kernels with clang.
| (ClangBuiltLinux.github.io). We take Debian's nightly package of
| ToT, make a docker image with the minimum tools we need, and use
| buildroot ramdisks that have a custom init script that powers
| down the machine once it reaches init. Our CI fetches various
| kernel trees and branches, builds them, then boots them in Qemu.
| The machine has 2 minutes to power up and down (usually takes
| less than 10s), otherwise we consider the machine hung and fail
| the run. We use travisci for the reporting, but are looking to
| offload the building.
|
| Also, Linaro's Tool chain Working Group runs a ton of CI on the
| kernel as well. There's an effort from RedHat called KCI to
| aggregate all of these reports.
| Twirrim wrote:
| There was a bug in a released kernel I ended up having to hunt
| down back last summer. It required a very specific set of
| circumstances to actually trigger the bug, but luckily for us
| we found some obvious side effects from the bug such that it
| was possible to get the relevant information via a very simple
| statically compiled C program (there would be a certain value
| under /sys).
|
| I used that C program as the initramfs for qemu, with it
| spinning up an instance based on the test kernel, and could
| grep for what I wanted in the output to get a relevant exit
| code.
|
| Combine that git bisect's automatic bisecting process, and I
| was able to automatically git bisect through a few thousand
| commits in the kernel to find the cause. It only took it about
| 50 minutes to find it in the end, which was helped a bunch by
| ccache (and the sheer build parallelism granted by building on
| a 52 core system).
|
| That was a fun, somewhat out of the blue, task.
|
| The process I took was based on https://ldpreload.com/blog/git-
| bisect-run, with various adaptations to suit the needs and the
| relevant build environment.
| cat199 wrote:
| so you have a very well tested boot sequence -
|
| is this not a slightly better kernel equivalent of 'it builds,
| ship it' ?
| yjftsjthsd-h wrote:
| > a slightly better kernel equivalent of 'it builds, ship it'
| ?
|
| Surely, "it _runs_ , ship it"? That seems quite a bit better.
| umvi wrote:
| So what happens when Linus Torvalds dies? Is there someone else
| who will become BDFL?
| LukeShu wrote:
| Greg Kroah-Hartman has filled in before when Linus has had to
| take time off.
| thedance wrote:
| Long way of saying "it's not".
| GhettoMaestro wrote:
| Fuck you.
|
| I hate comments like this on HN.
|
| The article clearly elaborates which methods/tools are used in
| their test process.
| DSingularity wrote:
| No, it is.
|
| https://github.com/linux-test-project/ltp
| bonzini wrote:
| LTP is only a small part of testing Linux.
| 0xFFC wrote:
| This has not been mainlined as far as I know.
| recov wrote:
| The commit history looks pretty healthy
| yjftsjthsd-h wrote:
| "Mainlined", not "maintained".
| alkonaut wrote:
| Why is there such little emphasis on "traditional" testing, that
| is, regular unit tests? At least some portion of the code base is
| surely suitable for normal unit tests. For example data
| structures, scheduling algorithms, file systems, ...
| ndesaulniers wrote:
| How do I test a driver for which I do not possess the hardware?
| vbezhenar wrote:
| Write hardware emulation for VM with any behaviour you want
| to test.
| Bnshsysjab wrote:
| Then I end up with a driver that conforms to emulated
| hardware and not it's real counterpart
| thedance wrote:
| You write the software in such a way that instead of just
| reading and writing registers or memory you exercise some set
| of functions. In normal operation you pass the driver a set
| of real functions that read and write real registers. In
| testing you pass functions that do other things. This makes
| it quite easy to exercise the features of the hardware that
| are rarely seen in the wild. For example most IO adapters and
| NICs have some kind of signal that they are overheating. Most
| Linux drivers simply ignore or malfunction when these
| conditions are raised, because the author of the driver never
| got a chance to manually exercise that feature.
|
| This is basic design for unit testing but it's impossible in
| Linux because Linux lacks a zero-cost abstraction that would
| let you mock out a device. C only has costly abstractions
| such as tables of function pointers.
| vbezhenar wrote:
| You can compile object file in isolation and provide mocked
| implementations for all imported functions.
| 555dreams wrote:
| I think many kernel developers view that as a problem as well.
| KUnit hopefully will change this when it becomes more widely
| used. I recommend more kernel developers check it out, it's
| quite nice even in its current form!
| varjag wrote:
| Linux kernel project predates what you call traditional unit
| testing practices+. Its success in the first decade and a half
| coupled with pragmatics of hardware testing make the flow what
| it is now.
|
| + Regression testing was certainly known then, but it was not a
| dogmatic movement yet.
| alkonaut wrote:
| I understand there are hurdles due to legacy, language, low
| level etc. But if 1 or 2% of a huge code base is easily
| testable, shouldn't it be? At least if/when regressions are
| found in functionality that _can_ be easily testable (pure
| functions etc) it would seem prudent to add regression tests
| to prevent the thing from happening again. Even in a 30 year
| old C code base.
| EdSchouten wrote:
| I think the main reason is that the Linux kernel (and similarly
| the *BSD kernels) are written in a programming language (C)
| that doesn't make it easy to do that. Code is often directly
| built on top of other kernel subsystems without any dependency
| injection whatsoever. This means that it's still possible to do
| unit testing of parts of the kernel, but it takes a crazy
| amount of effort, such as overriding symbols, overriding
| include paths and provide stub headers, etc..
|
| I am well aware that it's also possible to have dependency
| injection in C by using structs with function pointers, but I
| think we can all agree that it's a lot less pleasant to use
| than C++ abstract base classes, Go interfaces or Rust traits.
| This is why the Linux kernel only tends to use this sparingly
| (e.g., inode operations).
| eschaton wrote:
| You don't need a dependency injection framework to write unit
| tests, you just need cleanly separable units with well
| defined interfaces.
| [deleted]
| joveian wrote:
| This is a major advantage of NetBSD's Rump kernels that are
| used for automated testing. Some people have tried doing the
| same for Linux but I'm not sure if any such efforts are still
| in progress.
| rustybolt wrote:
| This is probably the reason. I work at a place where most
| software is written in C, and I see the same thing here:
| literally all tests are either manual or integration tests.
| Unfortunately, this also means that it takes about about half
| a day to 'run the tests'.
| cwzwarich wrote:
| You could test data structures (which tend to be implemented
| using macros), but any nontrivial subsystem of a monolithic
| kernel lives in a web of dependencies with other subsystems.
| This is especially true of Linux.
|
| To solve this, you would need to either use a hierarchical
| decomposition of subsystems or do some crazy mocking to run
| subsystems outside of the full kernel.
|
| There is a recent project (KUnit) to add a unit testing
| framework to the Linux kernel, but it remains to be seen how
| much adoption it will get.
| yjftsjthsd-h wrote:
| NetBSD has their rump kernel tech, which specifically exists
| to run chunks of kernel code independently. It can do a lot
| of neat things (I liked the "run netbsd drivers on other OSs"
| trick, personally), but one of the big uses they've mentioned
| is that it helps testing and development.
___________________________________________________________________
(page generated 2020-03-28 23:00 UTC)