[HN Gopher] Having metrics for something attracts your attention...
___________________________________________________________________
Having metrics for something attracts your attention to it
Author : zdw
Score : 41 points
Date : 2023-05-18 03:30 UTC (1 days ago)
(HTM) web link (utcc.utoronto.ca)
(TXT) w3m dump (utcc.utoronto.ca)
| RajT88 wrote:
| Re: Dangerous Metrics
|
| https://utcc.utoronto.ca/~cks/space/blog/tech/DangerousMetri...
|
| Another use case I have found is providing too many metrics can
| lead you to focus on metrics which are not important. Panicky ops
| people will chase spikes on a graph, which don't represent a real
| problem. It can be tough to talk them down off the ledge.
| neonate wrote:
| I suppose this is synergistic with
| https://en.wikipedia.org/wiki/Goodhart%27s_law, "When a measure
| becomes a target, it ceases to be a good measure".
|
| If having a metric attracts attention, and (suppose that)
| attention eventually makes it a target, and that makes it a bad
| metric, then you can't have any good metrics for very long.
| hinkley wrote:
| My thesis is that 'for long' does not need to mean 'forever'.
|
| Take some metrics, focus on one or two for a while, then focus
| on different metrics, come back to the old ones in a year.
|
| There is plenty of precedent for this in athletics, and
| competitions of intellect. We just haven't connected the dots.
|
| If you focus on bench reps the whole year it's going to fuck up
| your performance. If you focus on opening moves all year, it's
| going to fuck up your chess. That doesn't mean _never_ focus on
| them. But we are stuck in the fallacy of the excluded middle,
| so we think we have to do something all the time or not at all,
| which is just the dumbest fucking thing this entire profession
| has ever come up with. Worse than PHP.
| jerf wrote:
| This is related to the Streetlight effect:
| https://en.wikipedia.org/wiki/Streetlight_effect except you do it
| to yourself by putting up a streetlight in a place you hope is
| useful.
|
| In general, whenever you look somewhere new, you will find
| something you didn't expect. Science is just replete with this.
| In the funniest cases you get elderly scientists confidently
| declaring that some science is Done and All The Things Have Been
| Discovered... then someone puts up a new streetlight/metric and
| it turns out there were more things under heaven and earth than
| was dreamt of in their philosophy.
|
| I haven't found a good word for this in general but I've been
| internally using the phrase "cognitively available" for this sort
| of thing. There are well-worn paths through the space of all
| thoughts we can think, and they are generally co-evolved with the
| world we live in and structured the way they are for reasons, but
| they are not _always_ aligned with the details of where we are
| right now. There 's a lot of variants on this theme, such as
| adding a new metric and freaking out as the article says,
| bikeshedding where a group would rather argue about something
| cognitively available and easy to argue about than the hard thing
| they really ought to be arguing about, the whole System 1 versus
| System 2 thing (google it if you don't know what that is), all
| sorts of manifestations of cognitive laziness resulting in a sort
| of psuedo-force attracting you to the cognitively available easy
| concepts instead of the correct ones you need to engage with.
|
| Interestingly and topically, my experimenting with ChatGPT shows
| that it has a very similar phenomenon, and others have seen it
| too. For ChatGPT what is "cognitively available" is discussions
| that have been done to death on the internet. Ask it something
| super popular and it will definitely give you the right answer.
| Ask it a very closely related question but one that is different
| in some critical detail, and the "attraction" of the common
| question will overwhelm the model and it'll still give you the
| answer for what it finds cognitively available. (This suggests
| that mentally modelling ChatGPT as an _extremely well trained_
| System 1, but still ultimately a System 1, may be a productive
| line of thinking.)
| hinkley wrote:
| I've come to a realization, largely care of Test Driven
| Development:
|
| Some of the things we do are good exercises, some are good
| practices. Confusing the two hurts, and avoiding exercises that
| make dubious practices also hurts, but takes a lot longer to
| show up.
|
| If I go a year without doing TDD, I start to feel it negatively
| in my code quality. But if I do TDD every day for more than a
| few months, I also start to feel it negatively in my code
| quality. So when I get a story I decide whether I am going to
| do it or not, and I try to alternate between easy stories and
| difficult ones.
| Mountain_Skies wrote:
| This happens sometimes with fitness, especially products like
| Fitbit that are gamified. People start trying to max the metrics
| instead of what they're meant to represent.
| JohnFen wrote:
| That's the main danger of metrics: they have a nasty tendency
| to turn into goals, rather than data points that can be
| combined with other knowledge in order to provide a more
| complete picture.
|
| This problem seems quite common in software these days, and is
| why I've soured a lot on the use of metrics to aid development.
| hinkley wrote:
| I was an endurance athlete as a teenager and lately was trying
| to find my way back via hiking and long distance walking. I had
| plans to walk a half marathon (just below the time cutoff for
| the course) this spring but kept running into a wall around 10
| miles.
|
| Some days I focused on my pace. Other days I focused on trying
| to estimate my pace without looking at my watch. Some days I
| looked at my heart rate. Others I just concentrated on my gate.
| What are my knees doing? What are my hips doing? What is my
| core doing?
|
| But after a couple months, I spent about half my time just
| zoning out and doing walking meditation or enjoying the
| scenery.
|
| The trick is not to fixate on any one thing. The opposite of
| fixation is not apathy, it's moderation.
| ilyt wrote:
| I just go by "If I feel I am getting on the limit that would
| give me muscle pain next day I ease out". I am not sportsman
| and I don't want to beat some goals, I exercise to feel
| healthy and be able to lift some things I need for other
| hobbies
| ilyt wrote:
| And Goodhart's law strikes again!
| CrampusDestrus wrote:
| Yeah, sometimes ignorance _is_ bliss.
|
| Just strap a device onto your body during the day and then it
| just tells you "you're doing great!" or "you're not doing
| well!" without delving into specifics. Of course the raw data
| can be analyzed by a medical professional who will make his own
| considerations without you meddling with the metrics
| leroy-is-here wrote:
| Observation changes the thing being observed. Having a
| measurement of it allows you to find the measure lacking.
| discarded1023 wrote:
| A cognate of this is what Gernot Heiser of seL4 fame calls the
| "McNamara Effect" on his homepage: https://gernot-heiser.org/
|
| > The first step is to measure whatever can be easily measured.
| This is ok as far as it goes.
|
| > The second step is to disregard that which can't be easily
| measured or to give it an arbitrary quantitative value. This is
| artificial and misleading.
|
| > The third step is to presume that what can't be measured easily
| really isn't important. This is blindness.
|
| > The fourth step is to say that what can't be measured really
| doesn't exist. This is suicide.
| ilyt wrote:
| I just go by "if it moves, measure it".
|
| Cut only if you lack space/capacity and you have reasonable
| suspicion it absolutely won't be useful.
___________________________________________________________________
(page generated 2023-05-19 23:00 UTC)