[HN Gopher] Two sparsities are better than one: Performance of s... ___________________________________________________________________ Two sparsities are better than one: Performance of sparse-sparse networks Author : Anon84 Score : 42 points Date : 2022-02-11 15:25 UTC (7 hours ago) (HTM) web link (arxiv.org) (TXT) w3m dump (arxiv.org) | PaulHoule wrote: | Sparsity could really be a good thing but so many times I've | tried to use it and walked away disappointed in terms of | accuracy. | Voloskaya wrote: | It is already a good thing, but it currently requires a lot of | engineering effort to actually get it to work with acceptable | quality. It's not something that works out of the box like half | precision, or for some models, int8. And to your point, for | many production scenarios the ratio of engineering work vs | performance gains is maybe not worth it. But for models that | are going to handle massive load in inference it is worth it in | my experience. | | I expect that this will be made much easier in future with | better hardware support and smarter sparsification libraries. | synthos wrote: | It's the power savings that are the real goal. If you can | create and (re)train a network for incredible sparsity you | can fit inference in some pretty low power envelopes. I think | to that end the work required to get to adequate perform is | justified | md2020 wrote: | Always cool to see research from Numenta. They don't get as much | love as DeepMind, Google Brain, and OpenAI because their results | aren't as flashy, but I do feel like they've got a principled | approach to engineering intelligent systems distinct from that of | the big players. | Chinjut wrote: | An abundance of sparsities. ___________________________________________________________________ (page generated 2022-02-11 23:01 UTC)