[HN Gopher] Machine learning is still too hard for software engi...
___________________________________________________________________
Machine learning is still too hard for software engineers
Author : gk1
Score : 98 points
Date : 2022-02-22 20:07 UTC (2 hours ago)
(HTM) web link (www.nyckel.com)
(TXT) w3m dump (www.nyckel.com)
| deepsun wrote:
| I've dived into ML (and DL) with 17 years of software development
| experience. I'd say it's much easier than software. Yes, there's
| A TON to learn and experiment with, but still much less than with
| software. I was able to feel confident enough after just 1.5
| years learning and kaggling, and passed easily ML interviews to
| SF Bay Area companies (hint -- all data science people are
| extremely glad to see software experience, much more than data
| science).
|
| "Good pipeline and bad model is much better than bad pipeline and
| good model" (c) someone
| mrfusion wrote:
| Can you share how you switched into ML? How and what did you
| study? Any tips?
| abc_lisper wrote:
| Yeah, would appreciate if parent answers this.
| monkeybutton wrote:
| Bad pipeline means bad data, and bad data means bad model. Its
| as simple as that.
| jacobr1 wrote:
| Bad pipeline is too strong. But good enough pipeline + good
| enough models > bad pipeline + great model
| exdsq wrote:
| My wife is a researcher at Stanford doing some ML stuff and
| the constant painpoint isn't the researching of novel
| models or maths or whatever, it's the IT hell of managing a
| pipeline and data acquisition. I've helped her introduce
| things like Docker to the lab which seems to have helped
| but still - software engineering is a totally different
| skill that really hinders ML work.
| vorpalhex wrote:
| I'm ok with this. ML should remain expensive and hard to use
| while we grasp how to regulate it and understand it. We need less
| ML, not more of it.
| ramesh31 wrote:
| Can confirm. Data science team just schooled the entire
| engineering department in our Deep Racer tournament.
| fxtentacle wrote:
| In the gocoder Bomberland competition, my DL AI trained on 1.5
| billion timesteps just lost against someone handcrafting
| algorithms in Python in one day.
|
| You're absolutely right, sometimes, simple and predictable
| solutions are much better than AI magic ^^
| exdsq wrote:
| Makes sense though, right? Data team did data stuff better. If
| you had a different task with an engineering focus you'd see
| the opposite.
| rememberlenny wrote:
| This is timely.
|
| I just compiled a few resources I found useful on groking
| computer vision and deep learning.
|
| Short list I've enjoyed recently:
|
| - PyImageSearch blog: https://pyimagesearch.com/blog/
|
| - Fastai library and course: https://www.fast.ai/
|
| - Yan LeCun's 2021 Spring NYU class:
| https://m.youtube.com/playlist?list=PLLHTzKZzVU9e6xUfG10TkTW...
| Irishsteve wrote:
| I'm doing ML a long time, practice makes perfect etc. If anything
| is too hard, it is front end.
| tiborsaas wrote:
| As a senior frontend engineer, practice makes perfect, frontend
| is easy-peasy. ML/AI seems to have this thick wall of math
| around it, if I really want to understand what makes the models
| going.
| marcinzm wrote:
| Yeah, it felt like a sales person being confused why can't
| become an expert fullstack software engineer on up to date
| modern stacks without a lot of effort and time.
| modernpink wrote:
| Most CS grads coming onto the engineer market now will have ML
| exposure through their chosen college courses. As this wave of
| knowledge makes its way through the industry, the value of ML
| specialist knowledge will decline, especially as off-the-shelf
| pre-trained models improve. Very few companies then will be able
| to justify the luxury of a dedicated in-house data science/ML
| engineering team. In other words, most software engineers will be
| ML engineers, in the same way most software engineers today are
| Docker/cloud proficient engineers.
| clircle wrote:
| No, this article is wrong on the first sentence. The hardest part
| of machine learning _is_ in fact curating quality data.
| disgruntledphd2 wrote:
| That's the essential complexity. The OP is correct that there's
| far more accidental complexity involved than seems strictly
| necessary. In an ideal world, I wouldn't have had to spend the
| last few years learning software engineering in order to be a
| better data scientist.
| PLenz wrote:
| Physically training the models isn't the hard part. It knowing
| why you using x method, how to interpret the result, and what
| dials to turn that's the hard part. And there's no automating
| that.
| bob1029 wrote:
| We made an experimental foray into the ML space in attempts to
| accelerate authoring of SQL queries.
|
| After about a week of reading through literature and playing with
| open ai, it became pretty obvious we were still super far away
| from being able to build something the business could actually
| find value in.
|
| My problem scales horribly with the ML training angle we have
| today, because it's the super complex one-off queries we need the
| most help with, not the simple ones we can anticipate and train
| against.
|
| What we need is _actual_ intelligence for many of our problems.
| Things like subjective criteria are important to us. Realizing
| maybe a recursive query is a fair compromise to reduce a 400 line
| monster to 30 lines. Assuming the 400 "looks nasty", that is. I
| guess you could train that bit too, but then your solution space
| gets even more impossible to target.
| spaetzleesser wrote:
| I still remember when SQL was a dark science that could only be
| managed by administrators that were initiated in secret rooms.
| Nowadays a lot of people use SQL without deep understanding. They
| do useful stuff but if you are very skilled you can do way more.
| And SQL experts are often frustrated with all the amateurs that
| use their database in such a sub-optimal way.
|
| I expect the same for ML. The tooling will improve until regular
| developers can use it without understanding the fundamentals.
| They will do useful things but some skilled people will be able
| to do way more advanced stuff.
|
| That's pretty much the path of all technologies. They get
| simplified to a level where they useful for a lot of people but
| some experts will be able to get way more out it. And usually the
| experts look down on the amateurs.
| rchaves wrote:
| I really like the premise, and I really agree with it, but I
| don't think a SaaS is a solution, the solution is trying to find
| better abstractions that makes things simple and easier for the
| developers, using code, and without limiting their flexibility,
| but that's the hardest thing to do!
|
| As an experienced software developer who used to learn a new
| framework every week, I thought ML was going to be piece of cake.
| In reality, I went down this rabbit hole 4 years ago, and I'm
| still in there, I was so innocent back then
| lbriner wrote:
| The issue with abstractions is that ML isn't just a few
| variables or dissimilar systems to choose from, the whole
| problem domain that could be helped with ML has so many
| dimensions. Natural language, speech processing, pattern
| recognition etc. are all completely different. The only thing
| they have in common is that you might use computers for them.
|
| I think that is why people find ML so hard. It isn't a single
| sausage machine to create insight from, it is a set of entire
| stacks from philosophy all the way down to the electronics.
| oblak wrote:
| Good points. But truth is SaaS can be a great solution if you
| pick Nyckel!
|
| Sorry, it had to be done.
| saintarian wrote:
| Author of the blog post here - it's very cool to see this on HN!
|
| I wrote this as someone who considers himself a half-decent
| software engineer trying to use ML for a side project and feeling
| frustrated by all the effort and "accidental complexity"
| involved. Why focus on software engineers and ML in this
| post/rant/company? Because "software is eating the world" and
| having ML be more accessible to software engineers will broaden
| the range of problems they can solve.
|
| Thanks for all the comments - I acknowledge all/most of the
| criticisms as valid. A SaaS/AutoML solution won't work for
| everyone and definitely not for every problem, and it won't be
| the only answer to making ML more approachable.
| jakupovic wrote:
| The hardest thing about ML is getting right data in right format.
| This is the same of pretty much any other program, but ML has a
| special place.
| version_five wrote:
| I'd argue software engineering is still too hard for ML
| engineers. Most of ML (not research but commercial applications
| of what's already been demonstrated) is now well within the realm
| of engineering, but there are few standard practices, bodies of
| knowledge, or agreed on processes for doing anything. These are
| problems that engineering formalism solve, not another auto-ml
| tool. Maybe I'm saying the same thing as the article, just framed
| differently?
| yldedly wrote:
| In my experience, both are true. I'm more on the ML side, and I
| can tell I don't have the kind of routine and habits that good
| software engineers have, though I'm learning. But on the other
| hand, and I've seen this from software engineers who've made
| the transition to ML, and clearly have a good handle on the
| concepts (in one case even published papers in ML journals),
| they don't seem to have the intuition that allows them to
| select the right tool for the job, or understand when a given
| technique is appropriate; you could call that "modeling
| maturity" - a combination of mathematical maturity and the
| skill of relating domain knowledge to modeling choices.
| 7h3kk1d wrote:
| I think the above commenter is more critiquing the notion of
| relying on intuition all together.
| yldedly wrote:
| Right, I see. That's not really possible imo. For things
| like mlops, sure. But model development, selection,
| evaluation? From what I've seen, it's exactly when
| engineers reach for standard tools without giving thought
| to how it applies to the given problem that they do a bad
| job.
| jacobr1 wrote:
| Even in non-ML software engineering you still have
| architectural tradeoffs, that while you can in part make
| reasoned arguments about, you still are relying on the
| intuition of your technical leadership.
| otras wrote:
| To add to your point, the _Hidden Technical Debt in Machine
| Learning Systems_ paper:
| https://proceedings.neurips.cc/paper/2015/file/86df7dcfd896f...
| zcw100 wrote:
| I'd argue software engineering is still too hard for software
| engineers.
| garyfirestorm wrote:
| lol I did an ML course in 2017. I am a mechanical engineer with
| some computer skills and managed to do lot of tensor flow models
| quite successfully. It's funny to read that this could be hard
| for software guys.
| monksy wrote:
| Machine learning isn't for SWEs. It's for data scienctists and in
| a limited sense of Data Engineers.
| [deleted]
| surrTurr wrote:
| Nice read. However, it seems a bit fishy that the website
| publishing this article is a company offering ML as a service.
| They as a company directly profit from people not wanting to
| learn ML themselves. I'm not saying they wrote this article to
| increase sales, but it's a thing to keep in mind.
| etangent wrote:
| One might as well write an article claiming that "UI design is
| still too hard for software engineers" or "controlling a nuclear
| power plant is still too hard for software engineers" -- which
| are true (and it is equally true that software engineering is
| hard for UI designers and nuclear power plant operators).
|
| Who came up with this silly idea that something that is a valid
| knowledge domain of its own is suddenly going to become "easy"?
| sbeckeriv wrote:
| Can we not apply Machine Learning to teaching Machine Learning to
| software engineers?
| stevenalowe wrote:
| So is software engineering
| uoaei wrote:
| Wow, it's almost like machine learning and software engineering
| are different disciplines entirely!
|
| Just because both involve coding doesn't mean software engineers
| should be expected to have the math chops (stats/prob, linalg,
| calc, etc.) to make machine learning work for them...
|
| Vice versa is a little more complicated, because ML/DS can be
| done very inefficiently without the proper coding practices, but
| understanding the math is independent of that so that point still
| holds for this comparison.
| jstx1 wrote:
| > machine learning and software engineering are different
| disciplines entirely
|
| I don't think so. Or more precisely, they might look different
| academically but you need to have both as a skill to build
| something useful.
| uoaei wrote:
| In-built is the assumption that one person must have both
| competencies. This may be true for cash-strapped startups but
| this hardly plays well as general advice.
| jstx1 wrote:
| It's not an assumption, that's my main point - you need
| both skills in the same person. And it's true everywhere
| including in the biggest companies.
| uoaei wrote:
| Any good tech lead will remove that need, so no, still
| not good general advice.
| disgruntledphd2 wrote:
| Regardless of how low you get your communication
| overhead, it still exists. It's rare to find people who
| can both run and test all the infrastructure and model
| code, and notice that the transformation you apply on
| line 34876 of file foobar_now_with_added_ml.py is
| statistically inappropriate for your problem.
|
| That's not even to mention the really hard part,
| selecting a good outcome variable and appropriate ways to
| measure the performance of your system once it hits prod.
|
| You can definitely split this stuff between people, but
| it gets super-linearly harder as you add more people, so
| it's really incredible to find people who can do both
| (and honestly, there aren't that many of them (I'd like
| to say us, but I'm probably not there yet)).
| torbTurret wrote:
| Amazon has some cool interactive articles that make it easy to
| understand some concepts, eg:
|
| https://mlu-explain.github.io/decision-tree/
| zwieback wrote:
| Missing in these advertisements-disguised-as-blogs is an estimate
| of effort or time. Let's say ML is critical to a new product or
| internal tool, how many man-years is it reasonable to invest? If
| your expectation is using ML like a drag-and-drop app builder
| then you're probably best off using a canned tool but then you
| won't really have any competitive advantage.
| lysecret wrote:
| So, I went directly into data science after an econ degree,
| worked there for 2 years and then transitioned to SWE (at
| startups). First, I am 100% certain ML will become a part of the
| standard SWE toolkit (just like apis, docker, sql, etc..).
| However, to the relative "hardness" I would say ML currently is
| much less things but they can be really hard to get your head
| around (like starting to think in embeddings and vectors), and
| many APIs are wonky, because they are new. And the whole data
| science workflow space it in an early spot (we are still missing
| a "create-data-app", simmilar to the create-react-app.
| antisthenes wrote:
| Your career path seems really interesting. Do you have a blog
| or some kind of work showcase where we can follow your journey
| and work?
| howmayiannoyyou wrote:
| The best tutorials for indoctrinating software engineers in ML
| are written accessibly & use real-world business-case data, or
| easily comprehended data sets. One great example of this, from
| graph theory, was this fantastic 2013 article by Kieran Healy:
| https://kieranhealy.org/blog/archives/2013/06/09/using-metad...
|
| There's no getting around the complexities of fit, bias and
| customized models for many ML problems, so my observation above
| is obviously limited in its applicability.
| binarymax wrote:
| This line _" Set up low-latency, elastic, highly-available, and
| cost-effective inference close to your data."_ is the problem
| I've found that annoys everyone - Software Eng don't understand
| MLOps, Data Scientists don't understand systems programming, and
| everything ends up costing way too much money and taking way too
| much time.
|
| Even with a magic API the latency still isnt good enough, so the
| choice is often entire ML solutions like the OPs product, months
| of development time, or a really expensive container.
|
| Shameless plug since my Show HN went completely ignored earlier
| today ;) ...but that's why i built this:
| https://news.ycombinator.com/item?id=30428664
| tehsauce wrote:
| Another perspective, someone with no programming experience
| whatsoever today can in 1 minute create a new ML application for
| natural language tasks with performance that would blow most SOTA
| systems from 5 years ago out of the water, using things like the
| openai API. And this trend will almost certainly continue, where
| many tasks can be programmed by simply asking/describing the
| problem to a massive model and letting it work magic.
| lbriner wrote:
| That might be true but my experience is that people are much
| better at building tooling than they are at using it properly.
|
| So many organisations can't even get the basics of efficiency
| and custom service right but they still invest heavily into
| cutting edge tech. I can imagine plenty of companies joining
| the ML bandwagon and still not even really knowing what they do
| as a business.
| dTal wrote:
| This seems a trifle hyperbolic. No programming experience
| whatsoever? Remind me what the 'P' in 'API' stands for?
| dhairya wrote:
| The challenge for the engineers at our AI startup is that
| deterministic testing paradigms don't adapt well to probabilistic
| models that are continually being retrained. As a scientist is
| hard to convey the acceptable range of variance and often the
| random change of individual predictions at the decision boundary.
| It's also hard debug behavioral issues that actually are
| systematic model failure versus those that are traditional
| infrastructure bugs. Often times the band aid is that build
| lookup tables to ensure certain behavior which in turn also
| underlying issues from being discovered.
|
| Testing paradigms are either too high level or too specific.
| Recent work on evolving behavioral tests addresses this but it
| requires more manual effort and interpretation which kinda
| defeats to point of automated tests.
| Xcelerate wrote:
| Coming from a background in computational quantum chemistry, it's
| interesting to see all of the people who say ML is "easy" after
| taking a few online courses and reading some books on data
| science. If it's so easy, invent AGI then, since that is the holy
| grail of machine learning.
|
| Most of these people claiming expertise do not have a deep grasp
| of the mathematical fundamentals required for state of the art
| research in the field. Can you develop a neural network with
| features that are invariant to permutational and rotational
| symmetries? If so, how do you efficiently generate the
| irreproducible representations of the product of the symmetric
| and special orthogonal groups for use in a fast Fourier
| transform? What is minimum description length and why is it so
| fundamental? How do you solve trust-region problems on Riemannian
| manifolds? Throwing an off-the-shelf PyTorch library at a problem
| does not make one an expert in machine learning.
| uejfiweun wrote:
| Not to disagree with your point, but your comment really
| reminded me of the following scene from Breaking Bad:
| https://www.youtube.com/watch?v=W_dxteeedgs
| surrTurr wrote:
| I'd argue that integrating ML into a project isn't the hard
| problem, as it was described in this article. Most people will,
| after some research, be able to build some basic ML functionality
| using popular libraries and a bit of python. The hard thing about
| ML is, when you try to actually understand it. I've met many ,,ML
| engineers" who do their job but do not actually understand what
| they're doing. Understanding ML doesn't have to do a lot with
| coding. It's math and statistics.
| Thaxll wrote:
| I view ML the same as crypto, a fad that will go away in the
| future and remains in certain nich realms.
| binarymax wrote:
| ML has already added actual value to so many industries. While
| the AGI fantasy may never happen and we may even see another AI
| winter, the existing products and solutions that you use every
| day are powered by ML.
| swframe2 wrote:
| In general, getting a phd is the best way to go but it is not the
| only way. "The AI Epiphany" channel by Aleksa Gordic is worth
| watching. Check out his origin story:
| https://www.youtube.com/watch?v=SgaN-4po_cA He works at DeepMind.
| He is self taught; without a phd.
| alexcnwy wrote:
| Aleksa is great.
|
| Also check out Jeremy Howard from fast.ai - also no PhD but
| amazing teacher and contributes actively to research.
|
| Chris Olah (Google Brain, OpenAI, etc) go to university at all.
|
| PhD definitely not required.
| xiphias2 wrote:
| I tried to build on top of fast.ai, and it was very easy to
| start, but all the hooks and magic in fast.ai 2 just made it
| extremely hard for me to understand and extend the code. I
| believe it went in a bad direction.
| master_yoda_1 wrote:
| OP first define which software engineers you are talking about.
| Otherwise stop bullshitting just to market your company?
| xavxav wrote:
| Sort of hijacking, but I've always wondered: Where are our
| 'neural binutils'?
|
| I want to be able to compose these tools like I would random unix
| ones: Something like 'Identify album covers in this image |
| extract the text in said covers | spotify api'.
|
| It seems like there are so many breakthrough models but both due
| to technical (size/compute) and industrial ($$$) concerns they
| remain out of reach for random devs, let alone packageable into a
| `grep` style composable tool.
| jldugger wrote:
| > Where are our 'neural binutils'?
|
| Assuming you mean coreutils. binutils is for
| managing/inspecting binary executables.
|
| But to your point: there were two key innovations and criteria
| of UNIX pipelines: a common and understandable data format, and
| writing programs to send and receive anonymous data. Crucially,
| the input and output formats were the same: plain text,
| separated by newlines.
|
| In contrast neural networks are applied to a variety of data
| formats. Images, video, audio, text, social networks etc. each
| with their own encoding into something an NN can work with,
| with varying dimensions, features, metadata etc. So it doesn't
| make sense to bundle them as 'neural utils' but rather utils
| along whatever pipeline already exists, like GraphicsMagick.
| Which does leave a huge blind spot for the domain transforms
| like text recognition.
|
| If you stay within the AI ecosystem, you _can_ set up reusable
| layers for tensorFlow, but typically you cant swap out
| something in the middle without retraining all layers below it.
| Which you might treat as a violation of the anonymity
| criterion, since the behavior / performance of a layer relies
| on the specific behavior of those above it.
| fxtentacle wrote:
| "Machine Learning is too difficult" ... says company selling "ML
| platform [which] can be used by anyone and it only takes minutes
| to train your first model."
|
| No, actually, you're just being dishonest. Even if you hide
| TensorFlow and the keras models behind a nice GUI, people still
| need that mathematics knowledge to succeed. And yes, pre-training
| is great. But you need a shitload of stochastic analysis to make
| sure that the pre-trained embedding won't distort your results.
|
| "For a software engineer, the hardest thing about developing
| Machine Learning functionality should be finding clean and
| representative ground-truth data, but it often isn't."
|
| That is (in my opinion) an entirely bogus request. Machine
| learning is a mathematical / statistical tool for modeling large
| unknown functions. I feel like this sentence is akin in
| usefulness to:
|
| "For a nuclear power plant, the hardest thing about building one
| should be to draw how the finished building will look like in the
| press release".
|
| Someone "doing" machine learning without the requisite math
| knowledge is effectively driving blind. And worse than that, they
| don't even know what they don't see, because they lack the skills
| to identify their blind spots. That's how you end up with a "tank
| detection AI" that in reality just classifies the weather into
| bright vs. dark. [1]
|
| Companies like this who promise advanced mathematical algorithms
| with no prior skill or knowledge are how we unleash a plague of
| buggy unverified automatons upon the world.
|
| [1] https://www.lesswrong.com/posts/5o3CxyvZ2XKawRB5w/machine-
| le...
| l33t2328 wrote:
| > That's how you end up with a "tank detection AI" that in
| reality just classifies the weather into bright vs. dark.
|
| It seems to me the lack of knowledge was not knowing to use a
| diverse sample set, not some lack of mathematics knowledge.
| vladoh wrote:
| While driving, do you understand how the engine ECU of your car
| computes how much fuel to inject into the cylinder, how the
| ECUs distribute the power, and the braking force on individual
| wheels, or how your rear wheel steering calculates the turning
| angle of the rear wheels based on your seed and steering input?
|
| You don't need to know most of the details how your car works
| in order to drive. You need much more knowledge to build one,
| yes, but not to drive.
|
| There are different levels of abstractions and depending on
| your problem you need to understand them only up to a certain
| level. And different people have different problems to solve.
|
| In most real-world problems today, the difficult part is indeed
| the data, not the underlying math of the activation function,
| loss function, or optimizer. Just Google "data-centric AI
| Andrew Ng" to read more on the topic from one of the most well-
| known people in ML.
| glouwbug wrote:
| Resume driven development is real. Who wants a crud app on
| their resume when they can have a crud + ML app on their
| resume? I remember back in like 2016 recruiters devoured anyone
| with the slightest bit of ML experience on their resumes: it
| fed back into the ego of developers, and suddenly everyone was
| an ML expert who could do no better than load a JSON of data
| and import keras. What a strange trip that time was
| fxtentacle wrote:
| I feel like we're seeing the next round of that now with all
| those "build your own advanced Deep Learning AI in 3 simple
| steps" websites.
|
| Clarifai, Amazon Rekognition, Google AutoML Vision, Nykel
| (the article here), Amazon Comprehend, Google AutoML Natural
| Language, MonkeyLearn, Lateral, BigML, Azure ML, Lobe,
| DataRobot, Rapidminer, Dataiku ...
|
| Did I forget anyone?
|
| EDIT: H2O's Driverless AI, Floyd, AWS SageMaker, Databricks
|
| EDIT2: Pega Platform, MLFlow, Comet.ml
|
| EDIT3: $SNOW SnowFlake, Spell.ml, Cloudera ML, Alibaba Cloud
| disgruntledphd2 wrote:
| Snowflake is not like the others in that list, as they just
| provide a pretty good SQL dialect over cloud storage, and a
| tolerable UI and API access, for a pretty indeterminate
| price (who knows how much a credit is worth this quarter?).
|
| Personally, if there's ever a downturn, I plan to play
| Snowflake sales people off each other and get enough
| credits to last me a lifetime ;)
| tluyben2 wrote:
| Unfortunately there are a shitload of ML 'experts' out there
| who do not know what they are doing but still get results that
| are good enough to not get fired and receive copious amounts of
| money every year. These tools help doing that; companies
| generally don't see the difference anyway and they don't know
| how to set or evaluate KPIs on these ventures; they don't even
| know what or why they are asking for; they just know they need
| to show progress with AI to not become obsolete.
| habitue wrote:
| This goes a little too far. For traditional ML, sure, you need
| lots of deep statistical knowledge. But the fact is that deep
| learning is different: it's mostly a black box. _no one_
| understands exactly what they 're doing, how they're biased,
| and how exactly these models understand things differently than
| humans, whether you have a PhD in statistics or not.
|
| Because of that, doing deep learning consists of a bunch of
| cobbled together heuristics for getting good results and
| probing the model to give a human an intuition for whether it's
| learning correctly. The tricks and tips for steering that black
| box have mostly been developed in the last decade: it is not a
| super deep well.
|
| These tricks and heuristics are like the knowledge needed to be
| a technician in a nuclear facility, not the knowledge needed to
| build the nuclear facility in the first place. It's not
| nothing, to be sure, but unless you're a researcher developing
| new novel architectures, a very shallow understanding of the
| statistics will go a very long way.
| zozbot234 wrote:
| It's no different, statistical knowledge is still needed to
| draw the best possible inferences out of the combination of
| limited data points and prior general information with
| varying strengths/confidence levels attached. Not to mention
| that loosely "black box" methods have a long history of their
| own in non-parametric and semi-parametric stats, so it's not
| like deep learning is doing anything radically different.
| etangent wrote:
| Even so, there are procedures, protocols, and best practices
| for working with (and validating) black boxes, acquiring
| which may require time, skill, and patience.
| habitue wrote:
| Sure, but it's finite, reasonably circumscribed, and
| honestly not that mathy.
|
| I mean, even the example given by the OP about the tanks is
| super well known (apocryphal[0]) and doesn't require math
| knowledge to avoid. You just have to have heard of this
| kind of failure mode
|
| [0] https://www.gwern.net/Tanks
| FridgeSeal wrote:
| > You just have to have heard of this kind of failure
| mode
|
| Yes exactly. You have to be aware of it, you have to know
| what it entails and what can cause it and how to diagnose
| and fix it.
|
| That's the other half of the domain knowledge, and just
| "autoML-ing it" or following some set of prescribed steps
| won't necessarily get you that solution.
| igorkraw wrote:
| The "deep learning is a black box" meme is about 5 years past
| it's due date. It's not as tight as for convex models but we
| _do_ understand what 's going on inside, just not perfectly
| yet.
| habitue wrote:
| I think we're talking about different levels of
| understanding. For things like convex optimization, we have
| optimality results. For deep learning we have "try to stop
| training early because it tends to get overfit if you run
| it too long", "increase the number of parameters in the
| transformer to magically get uncanny impressions of written
| text out". These are not the same kind of understanding.
|
| If I hand you a 175B parameter language model, are you
| really contending we know what's going on in there? At a
| mechanical level, sure, it's tensor products and activation
| functions, but that's like saying we know how human brains
| work because the standard model is very predictive.
| hooande wrote:
| Not sure why you're making a distinction between ML and deep
| learning here. both of them can be black boxes. Calculating
| the area of a square can be a black box if all you know is
| how to plug numbers into the formula.
|
| A big part of machine learning is looking at weights and
| outputs to make sure the results are sane and that you have
| an understanding of what's going on. This is true no matter
| what algorithm you use to make predictions.
| rjsw wrote:
| If the data is garbage then it doesn't really matter how good
| your maths knowledge is, I challenge you to get a working "tank
| detection AI" when you are just training it on pictures of
| different cats.
|
| The Nuclear Power industry is starting to think about stopping
| doing all designs on paper, maybe in a few decades they will
| have achieved this, sending a message that good data is the
| thing they should work on first isn't a bad idea.
| CoastalCoder wrote:
| > I challenge you to get a working "tank detection AI" when
| you are just training it on pictures of different cats.
|
| Unless you're focusing on German tanks :)
| efitz wrote:
| Who cares if you overfitted? See, the model has 100% success
| rate vs the training set!
|
| Who cares if it denies bail to minorities or hits a few
| pedestrians from time to time?
|
| The problem isn't that ML is too hard, it's that it's too easy.
| Crazy people keep connecting ML to systems that matter- that
| have real, irreversible impact to humans- and they don't
| understand it.
|
| I wish ML were 1000x harder/more expensive to integrate so the
| economics would drive away frivolity.
| counters wrote:
| This is exactly right.
|
| I've seen it time and time again: Team has a black box ML/AI
| solution to a "problem." Team wants to eke out better P/R or
| deal with some complex edge cases. But team's problem is
| fundamentally ill-posed and no amount of hacking or kludges
| will actually produce the success criterion that they need.
|
| The problem is the accessibility to these tools, which in
| many times has led folks to neglect the subject matter
| expertise required to effectively _apply them_ in the first
| place. At least as these tools catch on in popularity in
| myriad problem domains, there will be a new generation of
| subject matter / domain experts who are familiar with them,
| and we'll probably jump over this hurdle.
| hinkley wrote:
| "For a software engineer, the hardest thing about developing
| Machine Learning functionality should be finding clean and
| representative ground-truth data, but it often isn't."
|
| Which is so much bullshit. The hardest thing is validating your
| hypotheses, which machine learning turns into a black box. When
| we have coworkers who insist on operating on wishful thinking
| we try to maneuver them out of a job. Except every 10-15 years
| when the built up pressure of fads overwhelms reason and we all
| get stupid for a generation (which in software is about five
| years).
|
| The things that started as AI that we don't call AI anymore,
| and don't lump in with AI when discussing successes or
| failures? It's because they can be explained in plain English
| and implemented without much or even any special jargon that
| marks it as anything more than exceptionally clever Logic.
| picardo wrote:
| I agree with this. The emphasis of any product that wants to
| democratize ML should be on making it easy for lay people to
| train models, and to collaborate with ML experts.
| horsawlarway wrote:
| I agree entirely with this.
|
| Software developers _HAVE_ to have an understanding of the
| subject they 're developing for. Computers are not brains, and
| they are not able to understand the objective or context in
| which they run.
|
| I could spit out their crappy tagline - "the hardest thing
| about developing __X__ should be __Y__, but it often isn't." -
| for almost any topic.
|
| "The hardest thing about developing an inertial navigation
| system should be getting clean sensor readings, but it often
| isn't"
|
| "The hardest thing about developing MITM proxies should be
| getting certs configured, but it often isn't"
|
| "The hardest thing about developing web extensions should be
| setting up your manifest file, but it often isn't"
|
| "The hardest thing about web development should be handling
| https requests, but it often isn't"
|
| "The hardest thing about having a baby should be labor, but it
| often isn't"
|
| "The hardest thing about making a car should be getting high
| quality steel, but it often isn't"
|
| ON AND ON.
___________________________________________________________________
(page generated 2022-02-22 23:00 UTC)