[HN Gopher] Why I'm Lukewarm on Graph Neural Networks
___________________________________________________________________
Why I'm Lukewarm on Graph Neural Networks
Author : VHRanger
Score : 87 points
Date : 2021-01-04 15:34 UTC (7 hours ago)
(HTM) web link (www.singlelunch.com)
(TXT) w3m dump (www.singlelunch.com)
| [deleted]
| osipov wrote:
| You can use
| https://web.archive.org/web/20210104154304/https://www.singl...
| since the original website is experiencing HN bear hug.
| beaupletga wrote:
| I need to handle (for work) a graph with 40 million nodes and
| more than 130 million edges. As expected, networkx couldn't
| handle more than a million nodes so I had to search for python
| libs which might handle that much data.
|
| This is why I've been using your lib
| (https://github.com/VHRanger/nodevectors) for at least 2 weeks
| now as well as these 2 other libs:
| https://github.com/louisabraham/fastnode2vec and
| https://github.com/sknetwork-team/scikit-network. What do they
| have in common? They handle sparse graphs (using CSR
| representations).
|
| Having a graph with several million nodes isn't just some edge
| case, social graph for instance grow way faster than anyone could
| expect. So I do totally agree that there is a gap between popular
| graph libs which handle very small graphs and real life libs
| which need to handle way bigger graphs.
|
| Btw, thanks for the work you've done with 'nodevectors'.
|
| PS: I'm not criticizing either networkx which is very handy and
| quite good when prototyping a solution.
| lostmsu wrote:
| By no means I am an expert in deep learning, but lately I've been
| considering graph NNs snake oil of neural networks. There are no
| impressive results on any common tasks and that picture is
| supported by a recent paper claiming transformers to "contain a
| graph network inside".
| vladTheInhaler wrote:
| A bunch of fully connected layers also "contains a CNN inside",
| but empirically CNNs lead to better performance on image
| classification tasks. When you can cut away an enormous, and
| mostly irrelevant part of the parameter space, that has a lot
| of value.
| astrophysician wrote:
| In my opinion there aren't many deep learning applications that
| are fundamentally snake-oil, it's more that people who don't
| quite understand the limitations and advantages of particular
| methods that end up being (witting or unwitting) snake-oil
| salespeople.
|
| Deep graph NN's are very useful for lots of data that are
| inherently graph-structured (another commenter above mentioned
| chemistry applications, and there are lots of other examples).
| Whether or not for the time-being they give SOTA results on
| common datasets, being able to work directly with graph-
| structured data is quite appealing in many cases.
| mrfox321 wrote:
| Google maps improved their worse-case residuals of traffic
| estimation using GNNs (messaging passing Nets, not graph
| embeddings).
| timr wrote:
| Graph NNs are widely used in the chemical domain -- molecules
| are easily modeled as graphs, so it's a natural fit.
| VHRanger wrote:
| So are graph embedding methods, which the post discusses
| dumb1224 wrote:
| The post can't be accessed with a Internal Server Error. Or
| is it just my ISP? I often confuse myself with other
| representation learning methods such as graph kernels
| https://en.wikipedia.org/wiki/Graph_kernel
|
| Are there any relations between them at all?
| VHRanger wrote:
| No, my blog went down from hitting the front page of HN
| enriquto wrote:
| And for large molecules (e.g., proteins) graph methods really
| start to shine.
| leecarraher wrote:
| I suspect we will not see a revolution in deep learning from GNN
| without a corresponding hardware(FPGA?) or optimization
| method(HSIC? HTMs?) advance.
|
| GNNs trained by backprop are making many of the same mistakes
| that LSTMs did: solve one important problem(exploding/vanishing
| gradients), but introduce a bunch of hyperparameters that bring
| along their own set of problems. GRUs are successful in my
| opinion, because they remove some of those tunable parameters.
|
| Of course as the post suggested, not being able to tune
| something, often loses out against some more tuned and curated
| solution. GNNs being the newest version, having tons of
| parameters to tweek. In the end do we get a better, ultimately
| more generalizable solution? Or do we just get more
| hyperparameters to tune, and spend more time and money for a
| modest, unrepeatable gain.
| orange3xchicken wrote:
| A lot of valid criticism of this post on the reddit ml subreddit:
|
| https://www.reddit.com/r/MachineLearning/comments/kqazpd/d_w...
| sischoel wrote:
| I don't know much about graph neural networks although it is a
| topic that I want to study in the next few months.
|
| But what bothered me in your article is what you wrote about
| graph data structures.
|
| NetworkX is indeed very slow, this is due to two facts: -
| NetworkX is a pure Python implementation and does not relay on
| some methods written in a faster language like C. - They use
| dictionaries to represent the graphs, which may have some
| advantages when mutating graphs, but of course have much worse
| locality than an adjacency list or a some sparse matrix format.
|
| But even for python there are much faster libraries such as
| igraph. The data structure in igraph is an edge list.
|
| A lot of single core graph libraries use an adjacency list
| internally, and while it is true that the index list for each
| vertex can be somewhere arbitrary in memory, they usually do not
| behave like a linked list, unless you graph is really sparse. One
| of the most used operations in graph algorithms is to iterate
| over the neighbors of a vertex, and for this, adjacency lists are
| very good.
|
| They also have a small advantage over CSR matrices for adding or
| removing edges, and they might use slightly less memory, as their
| index type only needs to be able to index all vertices and not
| all edges, so they need half of the space, which is better for
| caches.
| enriquto wrote:
| Indeed, the best support for large graphs in Python is found
| inside scipy.sparse
| VHRanger wrote:
| Right, the data structure depends on the tradeoff being made.
|
| Do you add or remove nodes as often as you traverse the graph?
| Then a dictionary/network-of-pointers make sense.
|
| But generally, if you're doing that a lot, a database also
| makes sense which is why I don't like the tradeoff NetworkX
| made at all.
|
| If you don't add nodes often at all, but traverse from edge-to-
| edge a lot, a CSR representation makes sense.
|
| Adjacency array representations have other tradeoffs vis-a-vis
| CSR representaitons and are kind of an in-between solution.
| tehjoker wrote:
| There's graph-tool but it is annoying to compile sometimes:
| https://graph-tool.skewed.de/
| GradientAssent wrote:
| Anyone have a mirror?
| VHRanger wrote:
| Author here.
|
| Seems like another post of mine (my take on BTC) was randomly
| posted and became popular at the same time I posted this one.
|
| I'm trying to get the wordpress host's cache to handle the
| load.
| whymauri wrote:
| Also posted on Reddit:
|
| https://www.reddit.com/r/MachineLearning/comments/kqazpd/d_w...
| mrtranscendence wrote:
| NetworkX isn't a bad library, nor is it only suitable for
| "babies". Yeesh. Actual people work on this stuff, you know, and
| they may have different goals, requirements, etc than you. This
| whole post reeks of someone who's so in love with being a
| maverick speaking uncomfortable truths that they've lost sight of
| this human element.
|
| If you're offended by (say) research papers that fail to break
| enough new ground to satisfy you, sorry about your luck, but
| again, not everyone's optimizing the same function as you. You'll
| probably do better bringing people around when you're not
| implying (or outright saying) their work is shit.
| VHRanger wrote:
| I never said NX is bad, but it is for "baby" graphs. NX can't
| scale past a few hundred thousand nodes.
|
| I appreciate NX's place in the ecosystem, but its
| implementation leaves a large gap to be filled for an
| intermediate library that is a Pandas analogue for graphs.
| pred_ wrote:
| From the post:
|
| > NetworkX is a bad library.
| dallathee wrote:
| There is igraph, graph-tool, snap and even a CUDA enabled one
| (cuGraph, they try to follow networkx API).
| dimatura wrote:
| I have to say I agree. NX is totally fine for the applications
| it was built for. I appreciate the simple API and the
| flexibility of using pure Python. There's a reason it's so
| popular.
|
| I'm not an expert in graph neural networks, so can't really say
| much about the novelty of Node2Vec. But I do think it's often
| misguided to judge scientific work as trivial or incremental in
| retrospect. Specially in a relatively young field like deep
| learning, where four years (Node2Vec is from 2016) is a _long_
| time.
___________________________________________________________________
(page generated 2021-01-04 23:01 UTC)