We're planning a long-distance move, and had one moving company
insist on a virtual walk-through for the estimate. The idea was that
just before the scheduled time, they text a web link, which opens
a video call. During the call, you walk around your house holding
up your phone, and the person on the other end takes screengrabs.
In my phone conversation with the salesperson beforehand, he made a
point of saying that they use an AI to estimate weights of household
items, and that the AI had proved accurate to within 5%.

Now, I've moved before, and have always had the movers actually
visit and walk through the house. There is lots of back-and-forth
about what is moving and what is not, or what is in that box or
closet or container or shed, especially if they can't see all of it
easily. So the claim of an AI making estimates to within 5%, solely
based on video screengrabs seemed like bullshit.

And in fact, we had another estimate done the "old fashioned"
way - with a real walk-through, and in the end the AI-powered
estimate was almost double the other. That's huge when you're
already spending thousands of dollars to move the contents of your
house. Luckily, both estimates listed the rooms and objects in them,
and the estimated weight of each. It was pretty clear that the AI
had greatly over-estimated the weight of even some obvious items.

I don't know if the AI estimate was allowed to just stand on its
own, but it sure seemed that way. Any halfway-sentient houseplant
would see that 700lbs listed for an outdoor patio table and four
chairs as maybe too much. It seemed like the salesperson was lazy,
and was just firing off these AI-generated estimates with no
checking whatsoever. Needless to say, we went with the cheaper (and
more accurate) estimate.

In this case, this one moving company had changed their entire
business model, most likely thinking that not having to visit
potential customers in-person would save money. Short-term, I'm sure
it did. But I'm guessing we'll start to see the knock-on effects of
bad AI as companies rely more and more on decisions made by them.
Nothing high-profile, but maybe a slow loss of business and revenue,
which may or may not even be attributed to the AI.

Of course, if everyone starts relying on bad AI, we're in trouble.