[HN Gopher] The AMD Radeon Graphics Driver Makes Up Roughly 10.5...
___________________________________________________________________
The AMD Radeon Graphics Driver Makes Up Roughly 10.5% of the Linux
Kernel
Author : akvadrako
Score : 226 points
Date : 2020-10-11 19:45 UTC (3 hours ago)
(HTM) web link (www.phoronix.com)
(TXT) w3m dump (www.phoronix.com)
| notaplumber wrote:
| AMDGPU It was significantly larger than the entire OpenBSD kernel
| sources. The dev/pci/drm directory ballooned to around ~130M. As
| of 6.8/-current, it's now 191M. It was under ~20M before May of
| 2019, prior to it first being committed.
|
| This driver is I believe part of the common codebase that AMD
| upstreamed to the Linux kernel, it's shared with its proprietary
| driver on other platforms.
| miki123211 wrote:
| According to the stats in the article, the kernel contains 225085
| lines of assembly.
|
| Just for comparison, the whole DOS 2.0 operating system, which
| was written in assembly entirely, has just 37955.
| barumi wrote:
| > Just for comparison, the whole DOS 2.0 operating system,
| which was written in assembly entirely, has just 37955.
|
| Apples to oranges.
|
| I mean, how many drivers were shipped with DOS 2.0?
| nybble41 wrote:
| Right, and how many distinct instruction set architectures
| did DOS 2.0 support? The fact that there are _only_ 225k
| lines of assembly in the Linux kernel, across the 30 or so
| major architecture families Linux supports plus any number of
| variations within each family, is rather impressive. If you
| ported DOS 2.0 to one variant of each of those 30
| architectures, assuming roughly equal code density, that
| would be over a million lines of assembly code altogether,
| for a far, far less capable system. That 's considering only
| the OS core, not device drivers, which is reasonable since
| DOS 2.0 didn't really have any drivers worth speaking of by
| modern standards. Most of the real low-level I/O was handled
| by the BIOS.
| zamadatix wrote:
| I think you're reading the comment backwards from intent. To
| me this comment is just pointing that while the vast majority
| of the Linux kernel C it supports so much stuff it still has
| an order of magnitude more assembly than old operating
| systems that were 100% written in assembly.
|
| Yes, apples to oranges... but interestingly so not as a knock
| that DOS was coded better or something.
| cardiffspaceman wrote:
| What percentage of loaded and running binary is it?
| edoceo wrote:
| 0% on mine (cause I'm on Intel and Nvidia HW) and have a custom
| kernel config.
| yjftsjthsd-h wrote:
| While the down votes are understandable, this is actually a
| somewhat legitimate take; the kernel's pretty good about not
| loading stuff that it's not actually going to use on your
| actual hardware, so the impact of this is limited to people
| who are actually using that hardware, which by extension
| means they kinda do need the driver in some form.
| armatav wrote:
| And they still don't work.
| henriquez wrote:
| The real shame is that Nvidia is still doing binary blob drivers
| 15 years after I started caring about Linux. Are they really that
| afraid of someone taking their Lucky Charms?
| severino wrote:
| Why is it a shame? They had been providing quality Linux
| drivers for years, when nobody else cared about high-end
| graphics for Linux. Remember fglrx?
|
| Now AMD is opensource? Great! However, it's still very far from
| perfect. You only have to take a look at the list of AMDGPU
| issues at freedesktop[1]... because being opensource is easy,
| but working fine in a stable manner is another.
|
| [1]https://gitlab.freedesktop.org/drm/amd/-/issues
| Aeolun wrote:
| Providing support for a marginal platform is also much harder
| when you're <0.1 times the size.
| banachtarski wrote:
| Well, in their defense, I generally haven't run into any issues
| with their driver and it's also pretty easy to package/install
| as an enduser.
| mcguire wrote:
| I have. Like keeping a copy of the last page you viewed in
| Chrome and overlaying it on the screen after exiting the
| browser.
|
| That one is fun.
| colechristensen wrote:
| I encountered a real fun bug where a game on linux crashed
| and a ghost of the game remained on the monitor even after
| it was connected to a different computer and power cycled,
| it remained for days. Some really interesting cascading
| bugs there.
| xyzzy_plugh wrote:
| I get this on Macbook Pros pretty often when connecting
| external displays. I think the nvidia drivers are
| universally bad.
| Nullabillity wrote:
| I still haven't been able to get my laptop's 1660Ti to
| display anything else than glitch art, proprietary or
| nouveau.
| asddubs wrote:
| wayland doesn't work at all so if you have a 4k monitor and a
| non-4k monitor and an nvidia card, you're basically just
| fucked, because you can't selectively scale things
| PurpleFoxy wrote:
| I have. I needed a prerelease kernel for a new driver but
| nvidia had not released a binary for the new kernel yet so I
| was unable to use anything but the open source nvidia driver.
| sushisource wrote:
| I agree, if it weren't for the fact that they give zero shits
| about Wayland support. I'd be totally fine with them staying
| closed source as long as they kept up with the standards.
| izacus wrote:
| What are you talking about? Wayland is supported on DEs
| that wanted to support nVidia chips.
|
| Meanwhile projects like Sway have a direct "Go to hell if
| you use nVidia, we won't let you run this code." It's
| bizarre that you blame nVidia for this.
| PurpleFoxy wrote:
| It's """supported""". It's apparently very buggy and very
| difficult to debug. Sway lets you run it after you set a
| flag making it very clear that if something is broken you
| may not report a bug since the developers are unable to
| reasonably fix it.
|
| Most Linux distro will also prevent you submitting a bug
| report for a kernel issue if you have a tainted kernel.
| nybble41 wrote:
| It's possible to create a Wayland compositor that works
| with the proprietary nVidia drivers, but it requires
| using nVidia-specific interfaces because nVidia refuses
| to support the same interfaces for non-GLX, non-X11
| hardware acceleration provided by every other Linux
| graphics driver.
|
| It's hardly surprising that a lot of Wayland compositor
| developers would rather not put in a ton of extra effort
| to add a special case for one particular set of
| proprietary drivers, which they would then need to
| maintain and support separately from the common code
| path.
| tomc1985 wrote:
| Is it? This is the same driver that makes me shut down X and
| fucks up xorg.conf every time I need to update?
| CamperBob2 wrote:
| _Are they really that afraid of someone taking their Lucky
| Charms?_
|
| They're afraid of patent trolls.
| [deleted]
| etaioinshrdlu wrote:
| My new theory is the Nvidia driver can't be GPL and in the
| Linux kernel, because then they couldn't ban datacenter usage
| of their GeForce cards by not licensing the driver for
| datacenter use. The upcharge on the Tesla series of cards is
| huge compared to GeForce for mostly the same chips. (For those
| not aware, see if you can find and GTX 2080 or 3080 from a
| cloud provider. It's not a thing. This is actually a huge deal
| for the machine learning industry, massively increasing costs.
| I doubt Google would have made the TPU if not for this.)
|
| Also, their driver is very complex, and they are constantly
| improving their hardware. They don't want to be dependent on
| getting new features and performance improvements upstreamed.
| aaomidi wrote:
| Meanwhile people say capitalism drives innovation.
|
| How much is nvidia single handedly holding back innovation
| and new discoveries?
| reader_mode wrote:
| I mean Nvidia is just cashing in their innovation
| advantage, AMD stack was worse forever and OSS is their
| white flag/hope someone else picks up the ball and creates
| an ecosystem to leverage their HW.
| AnthonyMouse wrote:
| > Meanwhile people say capitalism drives innovation.
|
| The saying is really that free market _competition_ drives
| innovation.
|
| Obviously patents and copyrights are government-issued
| monopolies, and monopolies are by definition lacking in
| competition.
|
| The theory is that by granting the monopolies we get more
| innovation. Often the theory is wrong.
|
| Especially when we allow the company to leverage the
| monopoly on the thing they actually invented into a
| monopoly on ancillary things that are only used in
| combination with that class of product.
| paledot wrote:
| That viewpoint is adorably naive. The Computer History
| Museum in Mountain View pretty clearly falls into three
| categories: government projects, genuinely innovative ideas
| from the private sector that failed, and the people who
| ripped off those ideas and made a killing. There is very
| little overlap between the last two categories.
| gruez wrote:
| Your second sentence doesn't contradict the first sentence.
| Capitalism (or more precisely, IP law) can simultaneously
| drive innovation and hold back innovation. The more
| worthwhile question is whether capitalism drives more
| innovation _overall_ , but that's hard to prove either way
| with snarky 1 liner.
| amelius wrote:
| > not licensing the driver for datacenter use.
|
| Why are these kind of licenses even allowed. If I buy a
| product, surely I can do with it as I please?
|
| Also, why doesn't TSMC slap a license on every IC that leaves
| their fab, taking (say) a 30% profit from every application
| in which their ICs are being used?
| closeparen wrote:
| This is called a royalty and is a pretty common business
| arrangement when licensing e.g. a patent for your product
| or some stock footage for your movie.
| tylerhou wrote:
| > Also, why doesn't TSMC slap a license on every IC that
| leaves their fab, taking (say) a 30% profit from every
| application in which their ICs are being used?
|
| Because companies would stop using TSMC chips...?
|
| Not to mention the logistical problems to attribute
| "profit" to any chip in particular.
| lostmsu wrote:
| I don't think not using TSMC is a viable option anymore.
| There's (slightly worse?) Samsung alternative, but they
| can't satisfy the demand.
| amelius wrote:
| Also, if TSMC introduces the license, then Samsung may do
| the same.
|
| But perhaps I'm too much thinking from the perspective of
| what large US based businesses would do.
| srtjstjsj wrote:
| US trusts are nothing compared to Asian trusts.
|
| en.wikipedia.org/wiki/Keiretsu
|
| en.m.wikipedia.org/wiki/Chaebol
| tylerhou wrote:
| For GPUs, I agree that you currently have to use TSMC.
| But if TSMC were to charge 30% of profits, you would
| almost certainly see a migration to other fabs which
| would harm TSMC's long-term profitability.
| formerly_proven wrote:
| > For GPUs, I agree that you currently have to use TSMC.
|
| RTX 30 is manufactured by Samsung.
| tylerhou wrote:
| Not the Quadros used in datacenters. GA100 is on TSMC 7nm
| according Nvidia themselves:
| https://developer.nvidia.com/blog/nvidia-ampere-
| architecture...
| bleepblorp wrote:
| Have enough details about TSMC contracts ever been
| released/leaked to know they _don 't_ do this?
|
| I don't follow the semiconductor industry closely enough to
| know anything about TSMC's business practices, but these
| kind of contracts are far from unheard of in other sectors.
| zamadatix wrote:
| It's cheaper than designing, manufacturing, and stocking
| more chip models. It's also cheaper than designing and
| manufacturing 1 model and physically disabling the pieces
| after.
|
| You could try to regulate that what is manufactured is not
| gimped on it's way to the consumer for ideological reasons
| but in the end you'd just end up paying more for a separate
| physical model because the profit margins on these advanced
| use cases are simply what drives GPU design.
|
| As for the royalty licensing TSMC is ahead in abilities and
| has captured an enormous portion of the market but it's not
| so far ahead that it can eat however far into customer
| income streams as it wants. Other manufacturers still exist
| and get deals, Nvidia is using Samsung 8nm for the latest
| round of GPUs for example. If it continues to increase its
| lead then we may see that type of agreement grow though.
| matheusmoreira wrote:
| > If I buy a product, surely I can do with it as I please?
|
| We no longer buy products these days. We license them.
| Another form of rent that allows the true owner to maintain
| control. Somehow this became the norm.
| srtjstjsj wrote:
| There's nearly no market for people willing to pay to
| own.
| nine_k wrote:
| Sure you can do anything with the hardware which you
| actually ha e bought!
|
| The problem is in the software (the driver) which you never
| can buy, only license under a long list of conditions which
| prohibit specific uses.
|
| If e.g. Noveau could implement interfaces needed for CUDA,
| you could probably try to use a 3050 in a datacenter. I bet
| NVidia has provisions against this turn of events, too.
| toyg wrote:
| If the hardware is effectively useless without the
| software driver, it could be argued the whole thing is a
| bit of a fraud / misrepresentation. But I guess nobody
| wants to sue somebody with pockets as deep as nVidia to
| change the status quo.
| srtjstjsj wrote:
| It's not fraud; you can return the purchase if you don't
| like the software license.
| newdude wrote:
| "The problem is in the software (the driver) which you
| never can buy, only license under a long list of
| conditions which prohibit specific uses."
|
| Well, buying and licensing are not so different in Europe
| (first sale doctrine). The company can not forbid you to
| resell a license (Exhaustion of intellectual property
| rights) in Europe.
| amelius wrote:
| > The problem is in the software (the driver) which you
| never can buy, only license under a long list of
| conditions which prohibit specific uses.
|
| Ok, so who gave software a special status over hardware?
| Is this desirable? Can we reverse it?
| nexuist wrote:
| It's not a special status, anyone has the right to deny
| you a hardware product as well. I don't have to sell cars
| to anyone if I don't want to. If I do want to sell
| someone a car, I can specify a contract or license that
| they must follow if they buy my car. Ferrari famously
| only sells exclusive models to customers who have been
| pre-approved, i.e. they have a certain amount of income
| and own 5+ Ferraris already. I also cannot walk into a
| Lockheed Martin dealership and tell them to sell me a
| F-22, even if I can afford it, even if my country has
| permissive laws regarding the ownership of fighter
| aircraft.
|
| As for software, well, EA has the right to ban me from
| their servers if I hack their games, even if I did pay
| for the product, and this makes sense because it ruins
| everyone else's experience. I don't pay for HN but if I
| did they still would have a right to ban my account if I
| start posting slurs or other abusive content.
|
| Is it desirable? Of course it's desirable; imagine having
| no control over your own creations and having to deal
| with the consequences of other people abusing it.
| amelius wrote:
| > Ferrari famously only sells exclusive models to
| customers who have been pre-approved, i.e. they have a
| certain amount of income and own 5+ Ferraris already.
|
| Sounds like discrimination to me, and not desirable.
| rowanG077 wrote:
| The examples of software you list aren't close to
| equivalent. They are all services and you can get banned
| from a service if you misbehave. But a piece of software
| such as a driver is not a service.
| ImprobableTruth wrote:
| >If I do want to sell someone a car, I can specify a
| contract or license that they must follow if they buy my
| car.
|
| Only in a limited form e.g. exhaustion doctrine prevents
| you from restricting resale. If someone wants to resell
| their exclusive Ferrari, there's nothing Ferrari can
| legally do (though this'll probably get you blacklisted
| from ever receiving an exclusive vehicle).
|
| In general, terms can't go against existing laws and have
| to be 'conscionable' to be enforceable (i.e. they can't
| be obviously 'unfair').
| laumars wrote:
| None of those examples are equivalent. The hardware
| examples are where companies refuse to sell a product (so
| you never own the product to begin with) where as the EA
| example is where you've been kicked off online services
| (you still have the capability to play the game offline,
| you just can't access their servers, but you don't buy
| their servers when you buy the game) and the HN example
| is a termination of subscription. Neither of those
| examples demonstrate legal limitations to software usage
| with a product you own (though the EA one at least comes
| close from a superficial perspective).
| ImprobableTruth wrote:
| There's nothing stopping nvidia from theoretically
| offering a gpu that is only rented out rather than sold.
| It's just not really considered acceptable for hardware
| (at the moment).
| srtjstjsj wrote:
| That's only because the hardware is a useless dongle
| without the software.
|
| Sure in theory you could run an open source driver, and
| in practice sometimes the river won't crash, but there's
| no point because you could get an equally good open
| source driver video card for the same price, since you
| can't get the fancy card's peak performance from the open
| source driver
| jonas21 wrote:
| Keep in mind it's the same special status that allows the
| GPL to have the condition that you must release your
| source code if you distribute something that includes GPL
| code. So, "reversing" it would also reverse the GPL.
| mr_toad wrote:
| > Ok, so who gave software a special status over
| hardware?
|
| Software is rarely sold (outside of bespoke development).
| All the off the shelf software is essentially rented.
|
| Software itself has no legal value - the copyright is
| what is considered to be property. That property can be
| leased or sold. This is why copyright infringement is
| called infringement and not theft.
|
| When you "buy" software, you are actually entering into a
| lease contract to use the software (sometimes perpetual,
| but increasingly only temporary) which can have various
| terms and conditions (that you really should read, but
| never do). But that lease doesn't grant you the
| copyright.
| amelius wrote:
| I think that's misleading then, because when I buy a GPU,
| they make me believe I own it, when, apparently, in
| reality I don't.
|
| I don't think this way of selling (or as you say renting)
| stuff should be considered legal.
| matheusmoreira wrote:
| > Ok, so who gave software a special status over
| hardware?
|
| Some american politician extended copyright protection
| towards software. The rest of the world eventually did
| the same.
|
| > Is this desirable?
|
| No.
|
| > Can we reverse it?
|
| Sure. We just need to have billions of dollars just like
| the copyright industry. That money can buy a lot of
| influence.
| hiram112 wrote:
| > Some american politician extended copyright protection
| towards software. The rest of the world eventually did
| the same.
|
| >> Is this desirable?
|
| > No.
|
| So I'm sure you'd be happy if I just took the software
| for whatever great startup idea you'd been slaving away
| on for the last two years, slapped better marketing on
| it, and undercut you by 50% since I didn't have to employ
| all those pesky overpaid engineers.
| stale2002 wrote:
| If someone purchased a product from me, and then used it
| however they want, that's fine.
|
| It is their product at that point, because they purchased
| it.
|
| You should not have the rights to control someone else's
| product, such a a graphics card, or whatever, after they
| have purchased it. It's theirs now.
| matheusmoreira wrote:
| How can you take software that was never published in the
| first place? There's a reason everything is a service
| these days. So what's the point of these protections?
| jiggawatts wrote:
| Yes, maybe, but step #1 is to sue Oracle.
|
| See if you can convince them to "let you" publish a
| benchmark of their database management system.
|
| Start there.
| Retric wrote:
| Amusingly, Oracle is known as the slowest major DB
| despite their heavy handed tactics. So, actual benchmarks
| might actually help their sales rather than people simply
| assuming it's unacceptably slow.
| formerly_proven wrote:
| The EULA / driver license may prevent you from reverse
| engineering the driver to enable these features, but that
| is only legal protection. nVidia sells these cards saying
| they don't provide feature X; nVidia also sells some cards,
| which do provide feature X (at a different price point).
| There is imho nothing per se wrong with this practice. The
| silicon being the same in both products is an
| implementation detail.
| stale2002 wrote:
| > nothing per se wrong with this practice.
|
| There is nothing wrong with someone doing whatever they
| want with a product that they now own.
|
| If someone wants to modify their own hardware, that is
| their right.
| labawi wrote:
| If you sold pork with different price on the condition of
| eating it in a wood vs. a stone house, some people would
| consider it market segmentation or maximizing profits.
| Others might call it illegal price discrimination.
| srtjstjsj wrote:
| Who calls it illegal?
| sandworm101 wrote:
| lol. When was the last time anyone actually bought a copy
| of software? You own nothing. You are a party to a contract
| written by nvidia and signed by you when you installed
| their driver. You can do only what they allow and they can
| yank thier permission whenever they see fit.
|
| Not happy? That is what makes FOSS so appealing.
| vbezhenar wrote:
| > When was the last time anyone actually bought a copy of
| software?
|
| Satya Nadella bought Skyrim recently. All of it.
| jackvalentine wrote:
| Skyrim includes licensed code such as Bink Video.
| kanox wrote:
| This sounds dubious, isn't it easier/more reliable to just
| block parts of the hardware using fuses?
| chrisseaton wrote:
| Maybe they _can 't_ open-source it because they don't own all
| the IP? That's very likely the case for Windows as well, for
| example, Microsoft just didn't licence all the code they used
| for releasing the source, and now you can't go back to 1000
| different IP owners and negotiate anything reasonable.
| bonzini wrote:
| Didn't they have to manually prepare a binary patch for a
| security issue in the Word Equation Editor, because they
| either lost or could not compile the source code anymore?
| skissane wrote:
| The original Equation Editor was licensed from a third
| party (Design Science), and it is possible that Microsoft
| never had the source code. Maybe the third party vendor
| lost the source code, but I think it is more likely that
| getting the third party vendor to fix the bug would have
| required negotiation with that vendor, and maybe Microsoft
| and that vendor were having trouble agreeing. (This is
| speculation on my part, I have no inside info.)
| MayeulC wrote:
| Well, at the very least, they could allow loading unsigned
| firmware, or allow their firmware to be redistributed, then.
|
| This is the number one usability issue with nouveau: no
| firmware means no re-clocking, which means bad perf.
| vbezhenar wrote:
| I think that Microsoft can open-source most of Windows
| sources. Nobody would care too much about few binary blobs
| and I don't believe that they don't own license for a
| significant portion of OS.
| yjftsjthsd-h wrote:
| This is exactly what happened with Solaris, and it turned
| out to be a rather massive problem because it meant that
| the community couldn't actually functionally produce a
| derivative distribution because the original released
| source code didn't actually represent the entire
| distribution. And a project that the community can't build
| will always be critically undermined by that flaw.
| paulmd wrote:
| second-hand information but apparently the reason they can't is
| that the driver contains code licensed from other companies and
| they can't open-source that.
|
| https://www.reddit.com/r/hardware/comments/j217oo/gamers_nex...
|
| while obviously not an official source, that isn't particularly
| surprising either.
|
| as an additional relatively-well-known-but-possibly-incorrect
| bit of internet lore, right now their Linux driver is basically
| a wrapper around their Windows driver, so that explanation
| makes a lot of sense. They would have to go through and
| disentangle what parts they own and what needs to be stripped
| out / replaced for the linux version at an absolute minimum.
| amelius wrote:
| In source code, or in compiled bytes?
| tomcam wrote:
| According to the first line of the second paragraph, it is
| source code
| capableweb wrote:
| Or just the huge output from cloc that is showing the count
| of lines of code.
| amelius wrote:
| It turns out that it's generated source code. So the actual
| source code may be substantially smaller.
| _chris_ wrote:
| Are they also open-sourcing the tool that generates the
| committed code? If not, then there's little difference
| between "generated" and "actual" source code.
| oblio wrote:
| I find it funny that developers (especially less experienced
| ones?) are so fascinated with the "big size" of the Linux kernel.
| I've seen this in various discussions, both online and offline.
|
| By commercial standards, the Linux kernel is really small. The
| average enterprise application which has been in development for
| that long (almost 30 years) usually has at least several million
| lines of code, if not tens of millions or more.
|
| User facing functionality is HUGE in size compared to tech
| functionality, usually.
| nybble41 wrote:
| The size is impressive relative to other kernels. A line of
| kernel code is much more expensive to develop than a line of
| application code. Of course, size is not be best metric by
| which to judge a project; it's always better to have more
| capability with less code if you can manage it. For its size
| Linux supports quite a wide variety of architectures and add-on
| hardware. Also, a great deal of effort goes into keeping the
| kernel codebase maintainable, which is more likely to manifest
| as lines of code _removed_ rather than added.
| atq2119 wrote:
| Most of it is dumb automatic header generation as the article
| points out, but looking at the source[0], there seems to be a
| _lot_ of code duplication, e.g. 5 different vcn_*.c files that
| seem of significant size and largely identical at a glance.
|
| That frankly seems like horrible software design.
|
| [0]
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
| deaddodo wrote:
| How do you figure? VCN 1.0[0] and VCN 2.0[1] differ within the
| first few lines and only do so further, the more you compare
| them. The files seem to correlate to each version of AMD's
| Video Core Next hardware[2]; and not anything to do with
| autogeneration.
|
| [0]
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
|
| [1]
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
|
| [2] https://en.wikipedia.org/wiki/Video_Core_Next
| afwe wrote:
| AMD drivers, at least on Windows, have a terrible reputation.
| fizixer wrote:
| Yes. Despite being an avid code minimalism and refactoring
| advocate, I have zero problems with auto generated code size,
| as long as the size of "the source that generated the code" is
| reasonable.
|
| Here's how I count the size of the code:
|
| So if a codebase is 1 million lines, but 900k was autogenerated
| using, let's say, a 10k line 'metaprogram' (which is otherwise
| not included in the source), the real size of the code is 100k
| + 10k = 110k.
|
| Now, with the stupid vcn_* .c duplication, that I have serious
| problems with. But what you could do is take a diff of the two
| vcn_* .c files, keep those diffs as part of the source code,
| plus only one version of vcn_* .c file, but now add a small
| script that generates the remaining vcn_* .c files during the
| build process by using the diffs, e.g., using the patch
| command. Now the size of the diffs and the size of the patch
| script, plus the size of one vcn_* .c files, is a better
| measure of the size of the code.
|
| Of course this is a short-term workaround. There's still a need
| to refactor the code properly.
| IncRnd wrote:
| That can be a useful way to calculate lines of code, however
| it is only nominally correct.
|
| In many cases it is more useful to calculate the lines of
| code according to what was checked into the build system,
| since those files can drift from the primary source. This is
| true, even if a million lines of checked-in source originated
| from 1k lines of meta-source.
| fizixer wrote:
| I'm not saying a line-of-code calculating program like cloc
| should change its behavior.
|
| I'm talking about the right way (IMO) of assessing the
| software complexity of the codebase.
|
| And I'm also talking about the fact that it is trivial to
| make cloc match my definition of source code size "if" ...
| the developers are willing to do a small amount of work by
| moving code-gen into the build process and not committing
| pre-generated code into the source, but the metaprogram
| scripts instead.
| IncRnd wrote:
| > I'm talking about the right way (IMO) of assessing the
| software complexity of the codebase.
|
| Yes, I get that, and that is what I meant, too. The
| reason is that complexity scales with the code that is
| checked into the build system, not just with the meta-
| code.
|
| I've written a lot of code generators for various
| purposes, and despite the efforts to only get meta-code
| checked into the build system, what has happened in
| almost every case is the generated code has been checked
| into the vcs. Of course, this depends on the particular
| teams (or consumers) of the generator.
|
| Eventually, the checked-in code gets changed. Maybe not
| right away, but it will most likely happen someday. That
| is why the complexity scales with the checked-in code and
| not with the original meta-code.
|
| There are also other force-multipliers, so to speak. For
| example, if there is a vulnerability in the generated
| code that was checked-in, the actual attack surface of
| the company can be multiplied by the number of instances
| of generated code _not_ by the meta code. Fixing one
| instance doesn 't fix any other instance. Complexity and
| risk are inseparably entwined and should not be looked at
| separately.
| fizixer wrote:
| Why would you check-in generated code and not the meta
| code?
|
| > Eventually, the checked-in code gets changed.
|
| Doesn't have to be. Especially with giant headers like in
| AMDGPU, it should be much easier to change the meta-code
| if there is a need to make modification in the generated
| code. Essentially you look at what needs to be changed,
| and work backwards to figure out what change in the meta-
| code will result in the same change in generated code.
|
| I believe you might be referring to stuff like
| boilerplate code, the whole purpose of which is to be
| generated for further development. In which case I agree
| with you, but then boilerplate codes don't balloon the
| way AMDGPU header files did.
| IncRnd wrote:
| > Why would you check-in generated code and not the meta
| code?
|
| I wouldn't. However, many teams do that. Perhaps they
| view it as an efficiency. In many cases people check-in
| generated code in order to perform their own risk
| reduction, removing a dependency and the possiblity of
| the generated code changing outside of their control.
|
| > I believe you might be referring to stuff like
| boilerplate code, the whole purpose of which is to be
| generated for further development. In which case I agree
| with you, but then boilerplate codes don't balloon the
| way AMDGPU header files did.
|
| No. I'm definitely not referring to boilerplate code.
| est31 wrote:
| IDK it seems pretty common in the embedded world to add support
| for a new piece of hardware by copying the entire driver and
| then changing the parts that the new hardware supports instead
| of trying to come up with a driver that supports all
| generations of the hardware. Pretty sad if you ask me and very
| unmaintainable, but, at least if you ask the companies, people
| should just buy newer generations of their hardware if they
| want those bugfixes :).
| rictic wrote:
| Wild speculation from someone who doesn't do hardware, but
| I'd guess that it's not so much writing the software as
| testing it that motivates that. It'd be hard to keep the
| testing matrix of apps X hardware X driver version from
| exploding out of control.
|
| Your customers will be hella pissed at you if their favorite
| game suddenly doesn't work on their system after a driver
| update. So you need to test a representative sample of
| previously-working software across all hardware that the
| updated driver applies to.
|
| That sounds like a nightmare, and a capital-intensive one
| because you need to keep machines up with all of these cards,
| both in continuous integration and for interactive debugging.
|
| I can understand the impulse to keep the list of hardware
| that new driver versions support small.
| [deleted]
| colonwqbang wrote:
| This is the real reason in my experience, combined with the
| bringup/support cycle that comes with hardware development.
| If you fork the driver for bringup, you buy yourself a lot
| of freedom to change things. During support phase, when the
| hardware is already in the hands of the customer, changing
| things is more risky and so you have to be conservative.
|
| For example, maybe you had a chip bug in the previous
| hardware generation which caused the system to hang after
| several days of stress testing. You found a software
| workaround for the bug, but every time you touch the code
| you need to re-verify the workaround, which takes days.
|
| Of course, the downsides of forking the code are also very
| apparent...
| gsnedders wrote:
| Another example of this in the kernel is filesystems: in
| theory, the ext4 driver can read/write ext2/ext3
| filesystems fine. And yet the kernel for years had ext2,
| ext3, and ext4 implementations, each later version
| derived from the previous. The ext3 driver was eventually
| removed in 2015, but ext2 and ext4 survive.
| oblio wrote:
| The thing is, DRY has some downsides. If you've pretty much
| nailed down the functionality of the old thing, especially if
| it involves some kind of manual testing/open beta test with
| users, the last thing you want to do is touch the old code.
|
| I'm not defending them, but sometimes something is good
| enough and you have to move on.
| reader_mode wrote:
| This doesn't sound like an issue with DRY as much as an
| issue of monolithic design.
|
| If driver version X worked but X+1 doesn't on your driver -
| don't update the driver. But when the driver is in the
| kernel trunk then that option isn't that simple.
| barumi wrote:
| > IDK it seems pretty common in the embedded world to add
| support for a new piece of hardware by copying the entire
| driver and then changing the parts that the new hardware
| supports instead of trying to come up with a driver that
| supports all generations of the hardware.
|
| The rule of three is not exclusive to the embedded world.
| It's far better to have duplicate code around then to have to
| deal with poor generalizations that affect multiple
| independent modules, which in this case mean hardware
| support.
|
| To put it in perspective, how many pieces of hardware would
| you require to re-test and validate if you decided to
| refactor bits of drivers that might share a common code path?
| gavinray wrote:
| This might not exist in C (I've never written it), but
| something like this seems like a fairly clean pattern you
| could use: abstract class BaseDriver {
| abstract commonThing() } abstract class
| BaseDriverV1 extends BaseDriver { commonThing() {
| console.log('This is the common method implementation for
| all V1 drivers') } abstract
| doThingOnlyV1Does() } class DriverV1_1
| extends BaseDriverV1 { doThingOnlyV1Does() {
| console.log('This is how Driver V1.1 does the V1 thing')
| } }
|
| This way you can use either an interface or an "abstract"
| definition that declares the base driver methods and the
| versioned driver methods, then provide version-dependent
| implementations where needed, or else share the common
| implementations by inheritance/extension.
|
| Maybe this turns into spaghetti at scale and it actually is
| easier to just copy-paste all of it, who knows.
| barumi wrote:
| > This might not exist in C (I've never written it), but
| something like this seems like a fairly clean pattern you
| could use:
|
| How sure are you?
|
| I mean, if the code path of a driver is touched, that
| triggers all verification and validation steps on all
| pieces of hardware that are required to ensure that the
| pieces of hardware affected by it will continue to work
| as expected.
|
| Furthermore, how many bugs are introduced on a daily
| basis because people like you and me push "fairly clean
| patterns" that end up triggering all sorts of unexpected
| consequences? I know I did, and still do, my share.
| AlotOfReading wrote:
| C doesn't provide any language support for this,
| including any notion of abstracts or classes. You can
| still do it manually and that's basically how drivers are
| implemented in Linux, but it doesn't address the
| combinatorial test matrix explosion.
| qppo wrote:
| I don't know, to me this is a bit like taking two c programs
| that print "hello world" and "goodbye world", looking at the
| assembly that is compiled then remarking "pretty sad and very
| unmaintainable."
|
| Point being, maintain the code generator and not the
| generated code.
| ptx wrote:
| Right. But here the generated code is checked into the
| source tree and the code generator is kept proprietary,
| which is why it's sad and unmaintainable.
| pantalaimon wrote:
| > That frankly seems like horrible software design.
|
| Why? If it's auto-generated and the generated code is never
| touched by humans, that's not too bad.
|
| This information is sourced by some CAD/synthesis tool and just
| describes the hardware register layout.
|
| Manually maintaining that would be much worse and error prone.
| tylerhou wrote:
| Compile times increase superlinearly with program size. How
| many cycles and developer hours are wasted if this
| duplication is not necessary?
| mcguire wrote:
| " _Compile times increase superlinearly with program size._
| "
|
| I've seen this said a couple of times lately. What is the
| source?
| saagarjha wrote:
| Internal compiler datastructures and the need to keep
| more things in memory?
| jchw wrote:
| This particular thought thread is about more translation
| units, not larger translation units. I can't see why
| those would not mostly scale linearly.
| tylerhou wrote:
| Tail latency. See my other comment
| (https://news.ycombinator.com/item?id=24749000).
| rational_indian wrote:
| Are you saying C compilers are O(n)?
| FartyMcFarter wrote:
| They may well be, provided one stays away from bad
| patterns such as humongous complex functions?
| formerly_proven wrote:
| I very much _expect_ O(n) compilation of stuff like array
| declarations, enums, CPP macro definitions and so forth.
| What else would it be?
| barumi wrote:
| > Are you saying C compilers are O(n)?
|
| OP asked for sources that substantiated a technical
| claim. No assertion was given.
|
| Unless you have a source that either supports or refutes
| the claim, trying to deflect the question does nothing to
| address the problem.
| tylerhou wrote:
| The other time that you saw it was also probably me. It's
| from this talk, which is about how a large amount of
| generated protocol buffer code at Google led to a
| quadratic increase in compile times:
| https://youtu.be/rHIkrotSwcc?t=720.
|
| TL;DW: The reasoning is because if you use a distributed
| build system, then your compile time is gated by the file
| with the longest compile time (tail latency). The more
| files you have, the greater chance that one of them takes
| a long time. When you generate source files, you tend to
| produce more files than if you didn't.
|
| Most users don't use a distributed build system to
| compile the kernel, so on further thought, in that case
| compile times probably scale closer to linear with the
| number of translation units. But wasted cycles are still
| wasted cycles, and regardless of how exactly compile
| times scale, you should still consider the cost of longer
| compile times when you duplicate code.
|
| With regards to link time optimization: sophisticated
| analyses take superlinear complexity:
| https://cs.stackexchange.com/questions/22435/time-
| complexity....
|
| Disclaimer: I work at Google.
| [deleted]
| jcranmer wrote:
| Not knowing the context of the quote, I can guess a few
| causes:
|
| * Compiler optimizations mostly work on a per-function
| basis, so that the 'n' in O(n) or O(n2) is the size of a
| function and not the size of the entire codebase. Not all
| optimization algorithms are linear, and while good
| superlinear optimizations should have code threshold
| cutoffs, in practice these may be omitted. Function sizes
| follow something like a power law distribution, which
| means larger code bases contain larger functions that are
| more expensive to optimize.
|
| * For C/C++, you compile code by pasting all the header
| definitions into your translation unit, and feeding that
| to the actual lexer/parser/codegen. Larger projects are
| probably going to have correspondingly larger headers,
| and files will probably include more of them, so that the
| actual translation unit sizes are going to be larger even
| than the on-disk source files would indicate.
|
| * This is more speculative, but larger projects are also
| probably likelier to blow out the various caches on your
| computer, so that the operations you figure are O(1)
| (such as hashtable lookup!) are actually more like O(lg
| N) in practice. And as you get larger projects that are
| less likely to fit any cache, the effect of the non-
| constant time becomes more apparent.
| barumi wrote:
| > Compile times increase superlinearly with program size.
| How many cycles and developer hours are wasted if this
| duplication is not necessary?
|
| Care to weight in the impact on developer hours and cycles
| of having to debug problems caused in how the autogenerated
| headers are created?
|
| If the code is not expected to change and was already
| subjected to verification and validation, ensuring that it
| stays the same is a good way to avoid wasting time tracking
| bugs on code paths that assuredly were already verified.
| alxlaz wrote:
| Looking at things like e.g. https://git.kernel.org/pub/scm/
| linux/kernel/git/torvalds/lin... , this is very obviously
| specific to the underlying hardware's control model in all
| sorts of ways. If the hardware has a gazillion control
| registers, then the software will have to handle a
| gazillion registers and will therefore need several
| gazillion definitions in order to do it. The hardware won't
| get any simpler just because it takes a long time to
| compile the driver.
| chrisseaton wrote:
| 83k lines of YAML. Wow.
| Impossible wrote:
| People complain about Nvidia's proprietary driver, but imagine if
| they opensourced all of their software stack and had it in the
| kernel. If AMD is 10.5%, Nvidia might be like 35% :)
| tinco wrote:
| What percentage of the Linux kernel is actually still there after
| you run it through the preprocessor on a modern system?
| jws wrote:
| A rough rule of thumb is 20 bytes of object for 1 line of C.[1]
| So if you have a 5MB kernel you might have 250k source lines of
| code.
|
| That said, the Linux kernel is not typical code. Great swaths
| of it are architectures and devices you will not need, so they
| are rightly ignored.
|
| [1] https://www.leshatton.org/Documents/LOC2005.pdf
| dodobirdlord wrote:
| I think that's the core of the question. How much of the
| kernel is left if you strip out all of the sections that
| would be stripped by the preprocessor when building for a
| current-generation 64-bit x86 target.
|
| AWS had a blog post about this topic a few years back. They
| created a cut-down Linux variant for running Lambda functions
| that only supported exactly their server architecture, only
| contained exactly the drivers they actually use, and notably
| only had 1 keyboard driver for a keyboard with only 1 key,
| which was mapped to Halt.
| znpy wrote:
| IMHO a fairer comparison would be the amd drivers vs the nvidia
| drivers, but we don't have the nvidia sources (of course).
| nemetroid wrote:
| > Though as reported previously, much of the AMDGPU driver code
| base is so large because of auto-generated header files for GPU
| registers, etc. In fact, 1.79 million lines as of Linux 5.9 for
| AMDGPU is simply header files that are predominantly auto-
| generated. It's 366k lines of the 2.71 million lines of code that
| is actual C code.
| spullara wrote:
| Why not generate it during the build? Is there a good reason
| not to do that?
| stefan_ wrote:
| The Linux kernel doesn't play the Firefox game of requiring
| Ruby, Python3, Python2, NodeJS and Rust to even be able to
| build the thing.
| nathell wrote:
| Not for the build itself, but there are Perl and Python
| scripts in the kernel source, which are referenced by the
| kernel's main Makefile.
| Ar-Curunir wrote:
| If you don't want to use languages that enable code reuse,
| then you can't complain about repeated code
| barumi wrote:
| > Why not generate it during the build?
|
| If those headers aren't expected to change then, with regards
| to accountability, it's far better to have the code checked
| into the version control and processed as is.
|
| More importantly, if the code is already generated then
| there's no need to make the build system more brittle by
| adding a non-standard build target that depends on
| custom/third-party tools.
| saxonww wrote:
| If it never changes, generating the headers during the build
| is just wasted time for whoever/whatever is running that
| build.
| ploxiln wrote:
| It was generated by the hardware division. These are the
| registers that are authorized for disclosure in the open-
| source driver by the AMD employed open-source driver
| developers.
|
| So, it includes many times more register definitions than are
| ever used (consider there are 8x more register definition
| lines than actual code lines that could use them) and it
| includes many sets of 16 or 64 definitions that a software
| developer would have made one parameterized definition (all
| the same except for _00, _01, _02, _03, etc). But this is
| exactly what the hardware guys generated for public release,
| and it is to be used as-is.
|
| IMHO it's kinda annoying and sad. The rest of the kernel is
| held to a higher standard, that's why all the other non-
| trivial multi-arch multi-family multi-generation code in the
| linux kernel is much more concise / less sloppy. It takes a
| lot of effort to make it that way, and commercial companies
| pretty much never bother, except when required by the Linux
| maintainers.
|
| But, modern graphics drivers are way too complex and way too
| much work, and most people do want some proper modern GPU
| support in the kernel, so compromises have to be made. It's
| not too bad, just a bunch of inert header lines, git and the
| compiler handle them just fine I guess.
| gridlockd wrote:
| > It was generated by the hardware division. These are the
| registers that are authorized for disclosure in the open-
| source driver by the AMD employed open-source driver
| developers.
|
| ...which is arguably not compatible with the GPL:
|
| _" The source code for a work means the preferred form of
| the work for making modifications to it."_
| tom_ wrote:
| What modifications would you make that might be useful?
| The (proprietary) hardware isn't going to change.
| ptx wrote:
| If the generated code is a representation of certain
| unchangeable data about the hardware, you might still
| want to
|
| 1) represent it more compactly;
|
| 2) represent it in a form that can more easily be read
| and transformed to handle future use-cases for the data;
|
| 3) after some future restructuring of the driver,
| represent the data in a form that better fits with that
| structure.
|
| If you have to regenerate the code using the proprietary
| tool in order to restructure the driver, the generated
| code is not "the preferred form of the work for making
| modifications".
| dmurray wrote:
| Remove some bugs, or improve its performance. Hardware
| drivers get updated all the time even when the hardware
| remains the same.
|
| I'm not an open-source absolutist: I think the pragmatic
| solution Linux went with is good here. But it's silly to
| suggest that the driver couldn't be improved if it were
| more open.
| akvadrako wrote:
| The topic is not the driver - it's the definition of the
| lowest level hardware interface.
|
| It's lists of registers and stuff like that; not things
| that can really be fixed by external devs.
| rleigh wrote:
| It's not applicable, in practice.
|
| This is the published hardware interface for the driver,
| the formal public contract. You can't change it without
| changing the hardware itself.
|
| If you really want to run the generator... well, the
| preferred form for modification is open to interpretation
| and if it's some proprietary tool then just getting the
| output is preferable to a dependency. Sometimes the
| rabbit hole is too deep, and we have to draw a line.
| fizixer wrote:
| What you're describing sounds like a binary blob
| masquerading as C header files.
| Taniwha wrote:
| >Why not generate it during the build? Is there a good reason
| not to do that?
|
| In many hardware shops the C definitions for the visible
| registers are generated automatically from the hardware's
| source code
| loop0 wrote:
| Honest question: why these gpu drivers need some many lines of
| code? Anybody with more background can explain?
| bleepblorp wrote:
| GPU drivers are essentially operating systems for the GPU.
| [deleted]
| extropy wrote:
| And also ~real time compilers for any user code (shaders,
| etc) that will be running on the GPU.
| monocasa wrote:
| The compiler isn't in the kernel though. That's in the user
| space half.
| detaro wrote:
| GPUs are complex computers in themselves, and the driver needs
| to provide compilers etc that target them (e.g. when it comes
| to shaders, applications hand the driver source code that is
| then compiled for the specific GPU architecture). Combine that
| with many different product generations and configurations
| being supported, code apparently being autogenerated and thus
| probably not optimized for size and you get to this.
|
| EDIT: monocasa points out below that the compiler is not a
| correct example - true, that does live in e.g. mesa userspace
| code.
| [deleted]
| radium3d wrote:
| And it still has a bug with residual cursor after wake from sleep
| on my desktop running Ubuntu 20.04...
|
| https://gitlab.gnome.org/GNOME/mutter/-/issues/1108
|
| Hopefully not for long though with this
|
| https://cgit.freedesktop.org/~agd5f/linux/commit/?h=amd-stag...
| pantalaimon wrote:
| What I find worse is the lack of analog output support with DC.
| One of my monitors only has a VGA input and I have no
| intentions of replacing it.
|
| However, the new Display Core does not support analog output
| and it will stay dark.
|
| Fortunately amdgpu.dc=0 is still an option, but I dread the day
| when this code path is removed (or bit rots away).
| p1necone wrote:
| Is it a _really_ good VGA monitor or something?
|
| You can buy relatively inexpensive active DVI/HDMI to VGA
| converters if that helps.
| pantalaimon wrote:
| > Is it a really good VGA monitor or something?
|
| Not at all. It's an old 19" with a resolution of 1280x1024,
| you can get those pretty much for free these days.
|
| But hey, n + 1 screens > n screens :D
|
| I usually just have my browser on it in full screen. And
| because of the resolution I don't have to worry about
| overly long lines on websites that aren't restricted in
| with (like hacker news).
|
| At work I picked up an old 19" TFT from the trash that
| connects neatly to the unused VGA port of the docking
| station. It has abysmal ghosting issues, so it's no use for
| the browser, but still good enough for a terminal.
|
| With modern graphics cards supporting many outputs and
| monitors that you can literally get for free, you can have
| the luxury of dedicating a display to a single task or
| application, so that's pretty cool.
| jtl999 wrote:
| I have an early Dell IPS from circa 2005 which is
| 1280x1024 19" and it supports DVI input, fortunately.
| blunte wrote:
| So does my latest macos with intel video and second monitor.
| ysleepy wrote:
| Yeah, the intel driver in Mojave on my 2015 13" pro has a
| bunch of issues.
|
| I see random buffer garble when waking the external screen,
| sometimes the machine freezes with hw video decoding and the
| resolution switches to low-def on monitor sleep only to
| switch back on monitor wake. It's a shitshow.
| nexuist wrote:
| 109 lines of a "Windows Module Definition"? What's that about?
| k4rli wrote:
| I found [this](https://old.reddit.com/r/linux/comments/2e255r/t
| his_is_how_b...)
|
| > False positives because def and config are pretty common
| suffixes:
| blunte wrote:
| It is another highly complex computer after all.
| tkuraku wrote:
| I really like that the AMD drivers are open source and up
| streamed into the kernel. However, that doesn't solve all the use
| cases. If I want to use newer hardware on an older distribution
| (RHEL, DEBIAN, Ubuntu LTS, etc) the upstreamed drivers will be
| too old. Distributing the driver like Nvidia or Radeon Software
| for Linux will still be necessary.
| foxrider wrote:
| You can always install a newer kernel - thankfully Linux is
| notorious for never breaking the userspace as a general rule.
| tkuraku wrote:
| It is a reasonable thing to want to keep the kernel stable
| for custom kernel drivers. So stable kernel space is also
| important to me.
| unsignedchar wrote:
| The title is misleading, there is a nontrivial distinction
| between the Kernel and its source code.
| mey wrote:
| Can someone with more understanding of the Linux Kernel
| arch/design describe why these drivers are in the kernel tree?
| Are these not part of a HAL or are they more "integrated" then
| that?
| detaro wrote:
| Linux does not use a stable API to talk to drivers, so drivers
| are intended to be part of the kernel tree so they can change
| with the rest of the kernel evolving.
| deaddodo wrote:
| Eh, that's not exactly true. Most of the API is pretty
| standard and stable. They don't have a fixed _ABI_ , which is
| how closed source OSes handle driver compatibility.
| mey wrote:
| There is a significant difference between having a
| supported defined interface boundary, and a semi-stable
| interface that doesn't necessarily guarantee backwards
| comparability to prior interfaces.
| deaddodo wrote:
| > There is a significant difference between having a
| supported defined interface boundary
|
| Which is literally the delineating mark between an API
| and and ABI.
|
| A stable ABI means your binary interface will remain the
| same and your compiled drivers should work, despite any
| changes on the kernel level. A stable API means you'll
| probably have to recompile your drivers when the kernel
| changes, but that they'll require little (if any)
| modifications.
| dTal wrote:
| A stable API means you will _definitely_ not have to make
| _any_ modifications, and Linux provides no such
| guarantees. "Pretty" standard and stable is not the same
| at all.
| detaro wrote:
| It's a matter of degree I guess. My understanding is that
| the kernel pretty much reserves the right to change these
| APIs as they feel fit, without much regard for the
| consequences for out-of-tree drivers, even if in practice
| the interface has settled and isn't in need of changing
| much currently.
| deaddodo wrote:
| Refer to my other response:
| https://news.ycombinator.com/item?id=24748970
| l3s2d wrote:
| Here's some more HN discussion around this:
| https://news.ycombinator.com/item?id=14533398
|
| And the relevant doc in the kernel source: https://github.com
| /torvalds/linux/blob/master/Documentation/...
| ISL wrote:
| What is the one sed file?
___________________________________________________________________
(page generated 2020-10-11 23:00 UTC)