[HN Gopher] Explaining machine learning pitfalls to managers (2019) ___________________________________________________________________ Explaining machine learning pitfalls to managers (2019) Author : yamrzou Score : 35 points Date : 2022-10-28 22:26 UTC (3 days ago) (HTM) web link (gudok.xyz) (TXT) w3m dump (gudok.xyz) | mistrial9 wrote: | the communication skills here are good IMHO -- essay-style | writing with digestable parts, a decent graphic, basic page | layout. I don't disagree with the content -- exactly.. from | practical experience I believe that this is an | oversimplification, but a useful one. The practical tips to Watch | Out For are worth the time to read this. | angarg12 wrote: | I have been interviewing ML Engineers for a while now and I've | seen companies fall into the same pitfall over and over. I | informally call this the "low hanging fruit" or the "illusion of | competence" pitfall. | | It goes something like this: someone (a PM, manager, or sometimes | an engineer) has a brilliant idea to use ML to enhance some part | of the business, say automate a manual process. A random folk is | asked to do a PoC, and they slap together a model in a couple of | days. | | This model often shows impressive performance, say 80% accuracy | in a problem where 90% is considered acceptable. Leadership gets | all excited and they sign off the project. And they get | themselves into a world of pain. | | The pain takes many forms, but the most common ones are: | | a) That extra 10% accuracy is extremely hard to achieve. | | b) Running ML in production is really difficult and the company | doesn't have the skills/maturity/expertise to run this new, | possibly mission critical components. | | The pitfall that lead to this situation are: | | 1) Assuming ML is easy from a PoC. | | If we built a PoC that achieves 80% accuracy in 2 days, how hard | can it be to achieve 90%? it turns out it can be really | difficult. Performance improvement of ML models is not linear and | it can be really difficult to get even a few % points better. | | The second part is running ML in production. It might be obvious | to an engineer that there is a big difference between slapping | together a prototype in a hurry and running a mission critical | service in production, but people unfamiliar with ML tend to | assume that the process of building the model is all there is to | it. | | 2) Assuming ML is not all or nothing (for your particular | problem). | | One might tend to think that a model with 80% accuracy is just a | little bit worse than one with 90% accuracy. However, depending | on the domain, this isn't true. For some problems models need to | perform better than a certain threshold to be of any use. In that | case, a model with 80% accuracy is as good as one with 0%. | | This is an oversimplified explanation but I've seen it happen | often enough that I consider it an (anti) pattern. ___________________________________________________________________ (page generated 2022-10-31 23:00 UTC)