[HN Gopher] Looks vs. Results: My ugly ad got 150% more clicks t...
___________________________________________________________________
Looks vs. Results: My ugly ad got 150% more clicks than a
professional design
Author : gk1
Score : 70 points
Date : 2021-01-02 12:15 UTC (1 days ago)
(HTM) web link (www.gkogan.co)
(TXT) w3m dump (www.gkogan.co)
| H8crilA wrote:
| There were 12 (24) clicks total, you don't even have to use
| p-value or any other calculation to know that this result means
| pretty much nothing ...
| boas wrote:
| p value by Fisher's exact test is 0.007. It's a significant
| difference.
| joegahona wrote:
| The author argues in the comments that it's statistically
| significant: https://www.gkogan.co/blog/looks-vs-
| results/?r=2#comment-520...
| andreilys wrote:
| Their argument consists of linking an optimizely screenshot.
|
| 12 vs 24 clicks is not significant, it could've gone either
| way. Also given this minuscule sample, it's easy to conduct
| p-hacking to get the desired outcome
| jonex wrote:
| Could you explain the calculations that lead to the claim
| that the result is not significant? From what I can tell,
| if we assume that clicking the ad is a weighted binary
| variable, eg. what is modelled as a "proportional
| distribution" there's a statistically significant
| difference between the two results. It's even pretty strong
| at P=0.006 (per https://epitools.ausvet.com.au/ztesttwo),
| but I might be missing something?
|
| In other words, if I'm doing the math right, for him to
| p-hack this by rerunning the experiment in the case of no
| difference, he'd have to run it more than 100 times to get
| a 50% chance of getting as good or better significance.
|
| There's of course plenty of other things that could be
| wrong outside of the simple statistical test, he could be
| making the numbers up, the groups might not be properly
| randomized etc.
| throwaway2245 wrote:
| It _is absolutely_ 'significant' for the usual statistical
| meaning, p<0.05.
| xmichael0 wrote:
| Thank you for commenting, I am rather annoyed I spent the time
| to read this... The author waisted everyone and his time.
| whimsicalism wrote:
| Ah yes, the brilliant "reject a priori" inference strategy in
| the wild.
|
| It's a statistically significant difference.
| NationalPark wrote:
| I'm not sure the author really understands the math (from his
| comments) but he's right that it's significant, where
| significant means "I am at least 95% confident that the
| observed improvement is caused by this change". The test is
| fine (assuming he didn't just check the results every day until
| he found one that would make for a fun blog post).
|
| The reason it feels intuitively broken is because the
| conversion rate is so low. But there were about 10k impressions
| in his test.
| canjobear wrote:
| That's not what significance means.
| NationalPark wrote:
| I mean... yes it is? 95% confidence in an effect is what
| most people mean when they say "statistically significant".
| throwaway2245 wrote:
| It depends.
|
| The fact that it has a significant p-value is interesting, but
| the lack of information about how the author decided when to
| stop is suggestive of p-hacking (i.e. we don't know how many
| screenshots were taken, but we understand that the author
| posted only the most favourable one)
| bpodgursky wrote:
| In the real world, we often have to work on a "preponderance of
| evidence" standard to actually get things done.
|
| Especially if the second option is cheaper and faster, there's
| IMO no bayesian prior that the professional ad being better
| (the null hypothesis) is true.
|
| So... seems like useful data to me.
| Tinyyy wrote:
| I think there can be prior that a professional looking ad can
| generate more clicks. Your argument shows a lack of
| statistical understanding - conditional on this data, the
| Bayesian approach would be to update the prior (whether A is
| better, or they're equally as good) with the data collected.
| With such a small dataset, you might end up with a belief
| that there's a 60% probability that B is better than A, but
| that's not significant enough to conclude that B is in fact
| better than A, as you still have a lot of uncertainty.
|
| With a prior that A is superior, you may still end up
| believing that A > B after updating, because there's just so
| little data.
| mlyle wrote:
| Yes, but the significance is high here; it's a pretty
| (un)lucky outcome to get if A and B are equivalent, let
| alone if A is better than B.
| mlyle wrote:
| When I calculate a p value, it looks like p<0.01.
|
| That seems like a highly significant result to me...
| contravariant wrote:
| Well, with low probability events you can go a bit further with
| your back of the envelop calculations, because that means
| they're more or less Poisson distributed. The average is
| somewhere around 16 so that gives a standard deviation of 4.
|
| So there's about 3 standard deviations between the two, this
| sounds like quite a bit but really means they're both 1.5
| standard deviations from the supposed mean. Which is, not
| great, though it might pass some of the weaker statistical
| tests.
|
| Now you should actually weight the values by the total number
| of impressions in which case you might get a slightly higher
| significance since the one with 12 clicks was seen by more
| people.
|
| So on the balance you should be wondering what you're paying
| the graphic designer for, but perhaps not start a new career
| designing low-budget adds.
| whimsicalism wrote:
| Fisher's exact test p-value of 0.007 is pretty decent, not
| "not great."
| [deleted]
| cabaalis wrote:
| The first thing I read on the hand drawn version was "free
| guide". In the "professional" version, my mind glossed over those
| words completely. I don't know why, other than maybe the pretty
| pictures at the bottom pulled my eye away.
|
| I'll click on free.
| anonytrary wrote:
| Option A doesn't stand out. It just looks like "another ad" or
| another component on a website. Stand out.
| insickness wrote:
| It's old news that in copywriting, copy with some misspellings
| typically performs better. While you may lose some who see the ad
| as less trustful, people respond to ads that make them feel like
| they've discovered something secret, new or underground.
| bichiliad wrote:
| Ignoring the other points about bounce rate, brand identity, and
| statistical significance for a second, I think an aspect that's
| missed a lot is that good design accomplishes a goal. The hand-
| drawn ad, by this definition, stands out more than the "well-
| designed" one just by virtue of being visually distinct. Asking a
| firm to help you design an ad capaign should involve iteration,
| testing, and refinement as different ideas are tested on your
| audience. If you ask a firm to take some words and make them look
| good you're missing out on most of the design process.
| solarkraft wrote:
| It does look very _different_ and thus inspires curiosity. But it
| wouldn 't exactly inspire _trust_ in me.
| Puts wrote:
| Making people click is not the problem. People follow garbage
| click-bait links all the time. Design is not just about "making
| things look nice". It's just as much about creating identity.
| What does it matter if you bring users to your site if it's the
| wrong kind that does not convert to real customers? Seems like
| there's a particular type of business leaders that thinks
| marketing is just a numbers game.
| joegahona wrote:
| Yeah exactly, another thing he could've done is put a girl in a
| bikini or a gross pic of toenails, a la Outbrain. That'll get
| people to click more than a "normal" ad but only the hackiest
| of hacks thinks that's the ultimate goal.
| ppeetteerr wrote:
| It worked for godaddy...
| MattGaiser wrote:
| The hand drawn one seems more authentic.
| jonnycomputer wrote:
| Hand drawn one was easier to read, and had fewer distractions. It
| also looked like what a manager might scribble down in their
| notes during a meeting.
| [deleted]
| 0_gravitas wrote:
| We see this very often at where I work as well
| chiefalchemist wrote:
| > An ad that took me 15 seconds to create had a 2.5x better
| clickthrough rate than one done by a paid designer. If this were
| an actual campaign it would mean 2.5x more sales leads or user
| signups at a lower acquisition cost, on top of shipping faster
| with less overhead.
|
| That is a false assumption.
|
| Not all clickers are equal.
|
| Not all leads are equal.
|
| More is not necessarily better.
|
| Better is better. Full stop.
| cblconfederate wrote:
| Digital ads used to be noticeable (because flash and gifs).
| Google and fb ads are bland and don't stick out, and you don't
| even have an option. I wonder how advertisers missed this bit,
| novelty used to be the desired goal in old media advertising.
| aaronbrethorst wrote:
| I find the paid-for design to be uninteresting and unattractive.
| If you were to ask me what I'd find when I clicked through, I
| would say that it's likely I'd be asked to sign up for a low-
| quality mailing list to receive a bland white paper.
|
| The scribble version is, at least, novel. Still probably wouldn't
| click through, but at least it would grab my attention for a
| moment.
| amelius wrote:
| Yeah but the scribble version wears off faster, when everybody
| else starts copying the trick.
| aaronbrethorst wrote:
| Would I rather:
|
| Use a design that costs me money to produce and doesn't work,
| or...
|
| Use a design that does not cost me to produce and _does_
| work, and then find something new to do when the novelty
| wears off?
|
| Not a tough call for me.
| amelius wrote:
| Sure, but that doesn't make the article's outcome
| universally true.
| throwaway2245 wrote:
| The ugly scribble design is simply better design - using better
| design principles.
|
| It communicates the useful part (in particular, the "call to
| action") clearly and instantly, without distraction.
|
| There's more going into this than just the concept of writing
| on paper.
| xupybd wrote:
| Ugly can be good advertising. Just have a look at
| https://lingscars.com
|
| Very ugly site, but impossible to forget that branding. It's my
| understanding they do very well.
| andai wrote:
| Looks just like the stuff I find in my (physical) mailbox.
|
| A lot of Indian websites look like this too (like paper
| advertising leaflets).
| Ayraa wrote:
| What was the bounce rate, avg visit time and conversion rate
| (email signup) between people who came from the non-designed vs.
| designed ad though?
|
| A higher clickthrough rate isn't necessarily better if more of
| the people who came from the non-designed ad didn't do much or
| anything on the page and mostly clicked out of curiosity.
| daniellarusso wrote:
| My best performing ads have been black backgrounds with white
| text, no grayscale.
| amelius wrote:
| If you target hackers, try black backgrounds and green text.
| bdcravens wrote:
| The professional "design" is too busy and noisy, distracting me
| from title. The call to action is tiny. The "ugly" design
| increases the lettering to a larger % of the total graphic. A
| design with these changes, and graphics that are a better
| abstraction for the message, would likely perform much better.
| (tl;dr pretty design != good design)
| ben174 wrote:
| This might simply be performing better due to shock factor. I'd
| be interested in seeing how these ads would perform if they
| weren't a novelty.
| anonytrary wrote:
| That's the entire point.
| Ayesh wrote:
| Ad blindness is real. The "professional" ad is a text book
| corporate ad that people often ignore.
| Finnucane wrote:
| It looks like the party invite email that you get from HR to
| tell you there's leftover pizza in the break room.
| notahacker wrote:
| Probably worse than most textbook corporate ads: the 'free
| guide' and 'read more' are too small to catch the eye, and the
| title is boring and generic even so even the sort of person
| that reads ads and is in the target audience might pass.
| Unsurprising the novelty ad performs better, but a better
| conventional one might too.
| christiansakai wrote:
| Out of novelty.
| masonlee wrote:
| This reminds me of "the most popular reddit ad ever" the "Magic
| Internet Money" MS Paint wizard.
|
| https://medium.com/@paulbars/magic-internet-money-how-a-redd...
| fossuser wrote:
| I'd wonder about click through vs. conversion and if there's a
| difference there.
|
| Might just be because of novelty factor, but also may be tricking
| less sophisticated users because it looks different and those
| users may be less likely to convert.
|
| It'd be interesting to see conversion results too.
| glangdale wrote:
| Yes, had the same thought. The hand-drawn one is unusual. If
| this strategy proliferated then it might well stop working,
| losing the novelty effect.
| corysama wrote:
| I've seen variations of this article before. It's the
| novelty. "That's different... What would someone pay real
| money to advertise with something that looks so cheap and
| crappy? Click."
| ppeetteerr wrote:
| I've read somewhere that porn ads and political ads perform
| better when they don't look professionally made. If I were a
| publisher, I would be very selective about the quality of the ads
| being displayed. I would also be in the minority since even CNN
| has ugly ads at the bottom of every article ("You won't believe
| what she looks like today!")
| biolurker1 wrote:
| I mean of you put a foto of an ass people would click to see wtf
| is going on but would never subscribe or buy software etc.
| Meaningless post really
| ClumsyPilot wrote:
| Are you sure they wouldn't? I am not.
|
| Some donkeys look cute
___________________________________________________________________
(page generated 2021-01-03 23:00 UTC)