[HN Gopher] The Neural Net Tank Urban Legend ___________________________________________________________________ The Neural Net Tank Urban Legend Author : belter Score : 20 points Date : 2023-06-21 10:51 UTC (12 hours ago) (HTM) web link (gwern.net) (TXT) w3m dump (gwern.net) | morkalork wrote: | I vividly remember hearing that story from a friend when I was in | highschool around the mid 2000s. | [deleted] | riskneutral wrote: | Oh wow, I read this story as evidence that neural nets are not | the future back when I was an undergrad studying machine | learning. Maybe I should have followed my instincts back then | because even then neural nets seemed intuitively very interesting | even if the statistics and math professors hated them because | they weren't derived from any first principals. | teraflop wrote: | A fictional variant of this story also shows up in Peter Watts' | novel _Starfish_. | | In that version, the neural net is trained to manage the | ventilation system of an underground train station, based on data | about passenger movements. Unfortunately, it ends up simply | paying attention to the display of a clock that happens to be | visible through one of its cameras. And when the clock breaks, a | bunch of people asphyxiate. A bit implausible, but memorable. | jumploops wrote: | My friend once asked me to help him with an autonomous RC car | race, where we'd go to a warehouse and train the car around the | track. | | After a few weekends of iterating and improving the | model/training set, we were convinced we'd win the next race... | only to lose almost immediately by crossing the lines on the | track. | | We did some inverse model explanation work which quickly showed | that our car was paying attention to the overhead skylights more | than the actual tracks. Unlike the other weekends, it was a foggy | day! | | A quick hack to cut out the top 50% of each training image | brought our car back to its prior reliability. | api wrote: | While this may be an urban legend, I have definitely played with | ML and seen things overfit in comical ways. | morkalork wrote: | It reminds me of a similar story about using deep learning on | medical imagery for diagnosis. In one study, it supposedly | learned the to tell which device was used to take the images. | Apparently that was somewhat correlated with the results as | more serious / heavily suspected cases were sent to a specific | hospital in the region. ___________________________________________________________________ (page generated 2023-06-21 23:00 UTC)