[HN Gopher] Cloudflare silently deleted my DNS records
___________________________________________________________________
Cloudflare silently deleted my DNS records
Author : iudqnolq
Score : 501 points
Date : 2020-02-24 17:43 UTC (5 hours ago)
(HTM) web link (txti.es)
(TXT) w3m dump (txti.es)
| hobofan wrote:
| Had the same thing happen to me some years ago. Had a (not so
| important) domain with Gandi, which pointed to the Cloudflare
| nameservers, and after some time, the domain was gone from the CF
| dashboard together with all DNS entries. The NS records were
| still pointing to CF and there also weren't any anomalies with
| renewal of the domain.
|
| I didn't give much thought to it, as I wasn't using CF for
| anything in production at the time, but sad to see that it also
| seems to happen to other people.
| 1337n008 wrote:
| after they began to turn on their own customers i moved all my
| domains and closed my account. looks like i have not missed out
| much.
|
| imagine if one day your bank decided to close your entire bank
| account without telling you...lol.
| paulfurley wrote:
| FWIW I recently evaluated a few DNS companies after Namecheap
| ballsed up our MX records in a similar way.
|
| I actively looked for someone we could pay money to, so we are
| their customer (as opposed to being a free tier user, effectively
| a cost)
|
| The winner was DNSimple[1], who do exactly 1 thing, and they do
| it extremely well. And they are small enough to not take
| themselves too seriously[2], which I really appreciate.
|
| Oh and their normal support channel is email, and everyone in the
| company takes a turn. I tested out their support before signing
| up and quickly heard back from a competent engineer, so they
| passed that test too.
|
| [1] https://dnsimple.com [2] https://dnsimple.com/dnsound <--
| bonkers
| znpy wrote:
| Have you considered Route53 ?
| stevekemp wrote:
| Wrapped with git; https://dns-api.com/
| iudqnolq wrote:
| Thank you. Looks like I'll just have to pay more. Any
| recommendations for a registrar?
| hedwall wrote:
| Dnsimple is a registrar as well. I have my personal domains
| there.
| PopeDotNinja wrote:
| Hurricane Electric has a pretty solid free DNS offering. I've
| been using it for like 10 years.
|
| https://dns.he.net/
|
| I haven't needed to talk to them much, but one time I tried
| to add a .ninja domain, and there backend wouldn't handle it.
| I emailed them to report the problem at 4:49 p.m. I got an
| email at 7:09 p.m. the same day (2 hours 20 minutes later
| later) asking me to try adding it again. [1] When a free
| service fixes your problem in a few hours, they get +1 gold
| star from me.
|
| [1] I just checked my email to look up the actual times. This
| was on Mar 15, 2017.
| BenjiWiebe wrote:
| +1 for Hurricane Electric. Right now I'm giving
| CloudFlare's DNS a try, but HE gave me solid service
| (including dynamic DNS) for years.
| jlgaddis wrote:
| Also, HE DNS will "secondary" from your own server.
|
| For example, you can run your own DNS server on a VPS or
| something, and HE will AXFR the zones from your VPS and
| serve them authoritatively.
|
| This allows you to run a hidden master, for example, which
| I can imagine some HN folks being interested in.
| PopeDotNinja wrote:
| I don't know what you just said, but it sounds awesome. I
| bookmarked this comment to review the next time I mess
| with DNS :)
| e_proxus wrote:
| I've been using iwantmyname for many, many years and have
| been super satisfied with them.
| iruoy wrote:
| NS1 could be another one to look at. I have never used their
| services (directly), but I've noticed Netlify uses them for
| their DNS services.
| samcrawford wrote:
| I've used ns1 for a few years, they've been great!
| jrockway wrote:
| We used ns1 at my last job, they were indeed great to us.
| We moved from self-hosting DNS because the DNS servers
| would randomly become unresponsive and would start
| returning fake records. After switching to ns1 and getting
| our first bill, we realized that a lot of our network
| equipment apparently did a DNS lookup for every log line.
| This resulted in an exceedingly large bill, which ns1
| happily reversed (we did fix our stuff ;).
| SnowingXIV wrote:
| I did the same after getting tired of NC's DNS interface. I
| host a few client sites with Netlify[1] anyways and moving over
| to their DNS (NS1) has been a breath of fresh air. It is _free_
| but they do have some paid options and the is UI dead simple
| which should be a requirement. Feel fairly confident I can rely
| on them to not muck up DNS records as this is critical to mail
| systems, websites, etc.
|
| Two years ago there was a moment where I was close to working
| for them too so I always try to use their products where I see
| fit. :)
|
| [1] https://docs.netlify.com/domains-https/netlify-dns/
| pmlnr wrote:
| Digitalocean has free dns service with an api; it's good and
| reliable.
|
| Running my own dns looks more and more reasonable though.
| johnklos wrote:
| Perhaps, but Digital Ocean also host spammers / scammers and
| doesn't do anything about them when they're reported.
| cmcd wrote:
| I am sure AWS, OCI, GCP, etc. all host scam websites with
| varying degrees of removal efficiency. What cases are you
| referring to specifically? Did they state they were not
| going to take these sites down or what was the context that
| you object to?
| wolco wrote:
| Are you saying if you use their dns you will get
| spammed/scammed?
| jrockway wrote:
| I think they're mad that DigitalOcean's IP range shows up
| in their ssh logs with failed authentication. A lot of
| people think that it's the ISP's job to regulate all
| traffic on their network, judging from the comments here,
| DigitalOcean at one time or another has failed to do
| that.
|
| I host all my personal stuff there, including something
| that updates their DNS via an API. They've been great to
| me.
| yjftsjthsd-h wrote:
| In this context, that sounds like an endorsement, honestly.
| If we're discussing providers that are willing to kill your
| services too easily, then saying that a provider is
| unwilling to cut service even to problem customers sounds
| like an amazing reason to use them.
| pmlnr wrote:
| I don't see how this is relevant.
| zymhan wrote:
| Then you probably shouldn't be recommending DNS
| providers.
| magicalhippo wrote:
| I use no-ip as dyndns for my home ip, so I can log in at
| home from outside. Recently at work my putty failed to
| connect, so I figured my internet line was down, it
| happens.
|
| Came home, internet works fine. Everything looked just
| fine.
|
| Back at work next day still can't connect. So I tried
| pinging, and I immediately see that the ip my home
| hostname resolves to is not what my ISP has. So I go to
| nslookup and try a DNS server I know (another local ISP),
| and it resolves to what I expected.
|
| A bit of checking later I find that at work they've
| started using OpenDNS, and OpenDNS has blocked all of no-
| ip due to malware and spam.
|
| So yeah, could be relevant.
| dspillett wrote:
| You listed their good points, the other poster listed
| some counterpoints. The one post is no less relevant than
| the other in a discussion about possible DNS hosting
| options IMO.
|
| Though I think the post would benefit from some citations
| to improve its relevance/usefulness otherwise it is
| little better than personal opinion/conjecture.
|
| Unless you are specifically questioning the relevance of
| hosting spammers, on which case: If that is true (again,
| some examples would be helpful here) and you intend to
| host your own mail servers via their services not just
| the MX records pointing to other mail services, you could
| find yourself blocked by association at some point. False
| positives are a big problem in this area and can be much
| admin to clear up.
| jermaustin1 wrote:
| same thing happened to me on GoDaddy for multiple domains when I
| got a call from a client that their emails stopped working. All
| the zones were factory reset, and no backup of the zones
| apparently existed at GoDaddy. I was on the call with them for
| hours refusing to hang up until it was resolved or they would
| lose the remainder of my business. After 2.5 hours of no valid
| reason that multiple domains when back to default DNS values and
| no log of access to my account for moths, I let them go.
|
| That's when I moved the couple of handfuls of domains I had left
| at GoDaddy over to Hover. It's more expensive, but the Hover
| interface is better, and I trust Hover (Tucows) more (well, I
| trust GoDaddy less).
| dergachev wrote:
| Out of this very fear, when Evolving Web started using CloudFlare
| for DNS, we wrote this backup script that runs on cron and pushes
| our settings to a git repo.
| https://github.com/evolvingweb/cloudflare-dns-backup-tool
| therealmarv wrote:
| Also don't forget: Cloudflare breaks many second and third world
| countries' Internet with their DNS captchas because they think
| the good guys live only in first world countries (maybe look up
| the word discrimination in your dictionary cloudflare) and force
| them to install extensions like PrivacyPass because they think
| "we are so big and know what is right for the world".
| input_sh wrote:
| That's CDN captcha, not DNS. If you use Cloudflare solely as a
| DNS provider, your users don't see the captcha. If you route
| your traffic through their servers, then they do.
| therealmarv wrote:
| you're right, it's their CDN not their DNS. Nevertheless many
| site owners choose Cloudflare (paid or not paid) and use
| Cloudflare's default settings and maybe they also never check
| their sites from second or third world countries. Result is
| that the Internet is utterly broken on many Cloudflare hosted
| sites (and that's a lot of sites) outside of first world
| countries.
| iudqnolq wrote:
| OP here. You're right, and even in the US I still get
| endless CAPTCHAs because I browse on Firefox on Linux with
| tracking prevention.
|
| My website was down for yak shaving when this happened, but
| before then I had DDOS protection turned off.
| behringer wrote:
| This is why you need name servers from 2 different companies and
| dns monitoring. It doesn't matter who your provider is. Errors
| happen and waiting half a week to fix it is insane.
| potency wrote:
| Cloudflare lost my support when they started de-platforming
| people for holding opinions they didn't agree with. Censorship
| outside of strictly legal bounds should not be tolerated from a
| company as powerful as Cloudflare.
| J5892 wrote:
| What sites have they de-platformed outside of legal bounds?
| rabite wrote:
| Daily Stormer, archive.ph, 8chan
| dependenttypes wrote:
| > archive.ph
|
| When/why did they remove that one? Have you got any source?
| RL_Quine wrote:
| Why do you think you have a right to host with them? You don't,
| you have a privilege that's extended by them. You're welcome to
| host your own thing somewhere else.
| HBKXNCUO wrote:
| >You don't
|
| You don't now.
|
| Why do successful technology companies have a right to have a
| proportionally large influence on the public political
| debate? Is it good for society to allow successful technology
| companies to have such a large degree of control over
| something so incredibly vital, merely because they were
| effective at running a particular type of business?
| mavhc wrote:
| Is it censorship if they refuse their money for a service?
| Pretty sure that's just business. Are they stopping you having
| a website?
| wyoh wrote:
| Would you be OK with your phone provider or imternet provider
| to stop doing business with because you said some unsavory
| thing to a friend or have blog supporting the wrong
| candidate?
| rabite wrote:
| Yes, it is censorship. The entire history of First Amendment
| jurisprudence was set around the idea that powerful people
| were not allowed to stop political and religious speech.
| Marsh v. Alabama is a great example: a company town owned
| sidewalks that they didn't want religious prosyletizers on.
| The courts ruled that the fact that they owned the sidewalks
| and roads is irrelevant. For the entire history of my country
| powerful people were not allowed to buy up the public square
| and prevent the little guy from speaking. Everyone had a
| right to enumerate their grievances in a free and open
| marketplace of ideas. This has of course changed in the age
| of the Internet, where a bunch of scheming Stanford grads
| have bought up the courts, wrested control of the key
| Internet infrastructure away from the public who funded its
| creation, and sit there and grin as they take the role of
| arbiter over all speech on the Internet. The wealth and power
| disparity between the rich and poor is at its height, and it
| is clear that there will be no legal or democratic solution
| to the concentration of power in the hands of a handful of
| Silicon valley billionaires.
| sjburt wrote:
| At least in some cases, those people were claiming that because
| they hadn't been removed, Cloudflare supported them. I don't
| see what other option Cloudflare had at that point.
| rabite wrote:
| Except that Matthew Prince, Cloudflare's CEO, made that up
| out of thin air. There's no point where Daily Stormer said
| that Cloudflare supported their ideology. There's no record
| of this on the Internet. Can't find it, because nobody at the
| site ever made such claims. Stormer was kicked off of dozens
| of domain registrars and registries (GoDaddy, Google,
| Namecheap, Dreamhost, several national cctlds) in the same
| period -- none of them had to come up with a fake excuse like
| "people will think we support their ideology". Cloudflare
| does do infrastructure plenty of pedophile and Islamic
| terrorist sites, so now we can assume that they actually do
| support those as they aren't removing them from the service.
|
| Cloudflare also didn't even bother telling that lie anymore
| when the dozens of sites they censored afterwards including
| 8chan were systematically barred from basic commerce.
| [deleted]
| Mojah wrote:
| Occasions are rare where I get to say "hey, I built a thing that
| might help here!" - so forgive me as I take this opportunity with
| both hands.
|
| Whether this was a bug or a rare protective mechanisme, there
| will be times when your DNS provider makes a mistake and removes
| records. You mentioned in your post your DNS isn't hard to
| reproduce, but how certain are you that _all_ records are
| restored? How long do you have to fight DNS issues before it's
| OK?
|
| I built DNS Spy [1] for this exact occasion. It monitors your DNS
| for any changes made, keeps a version of all DNS records (current
| & former) and allows you to restore/download a BIND9 zone file
| for your zone. You can easily import this into any commercial DNS
| provider or in your own BIND9/PowerDNS setup.
|
| I would love to hear feedback on how DNS Spy could be improved
| when DNS disasters like these occur!
|
| [1] https://dnsspy.io/
| im3w1l wrote:
| The issue I see with this is that
|
| 1) You can't use it after the fact.
|
| 2) It's very specialized.
|
| People are not going to set up dozens and dozens of services to
| monitor for really rare things. It should be part of general
| purpose monitoring suite.
| hashhar wrote:
| Looks really useful and fulfills a very important purpose. What
| good is all your backups if you can't get your services back up
| due to missing DNS configuration.
| iudqnolq wrote:
| (OP here). That looks really useful. If I was running a real
| service I would definitely look into it. Because this is just
| the personal website and email of a college student I don't
| think I could justify the expense when using something like
| Uptime Robot to monitor if a single record points to a web
| server would probably give me close to the same reliability.
| Mojah wrote:
| Oh I absolutely agree!
|
| If you're a business, whether it's a SaaS or "just" a
| marketing website for your brick & mortar store, I think it's
| crucial to have back-ups. Most people think of backups as
| files, database dumps, previous versions, etc of their
| website. But the configuration data (in the form of DNS)
| isn't often considered.
|
| You're tech-savvy and can restore your DNS records because
| you know yoru servers' IP address and your MX records, but
| who else could do the same?
| [deleted]
| jgrahamc wrote:
| This is being looked into internally and I am involved. Likely
| won't post an update here as it pertains to a customer account
| (unless customer agrees).
|
| BTW If you, dear reader, ever find yourself so frustrated with
| Cloudflare that you feel like your only recourse is a blog
| post... my email is jgc@cloudflare.com and I'm happy to hear from
| people.
| p1necone wrote:
| The problem is that big companies don't care about giving
| quality support for their products, and for the most part they
| get away with it. From their perspective there's no problem to
| solve.
|
| Your solution basically boils down to "companies are failing at
| escalated support issues well, so they should escalate support
| issues well."
| martin1975 wrote:
| You guys are the worst censors even on your own blog. Any
| criticism toward your CEO or the way things have been done,
| completely out of integrity with your own policies in the past
| (such as cutting out providers because your CEO woke up self-
| righteous on the wrong foot that day) gets moderated away or
| not even admitted to the CF blog.
|
| You've screwed up so many times, I am surprised by now more
| people aren't onto your tired antics. Thankfully, you cannot
| delete this post - perhaps many fanboys will downgrade it, but
| at least I can tell you how I feel.
| scrollaway wrote:
| Not a fanboy, still downvoted your post because it's
| incredibly whiny and does not contribute anything. All you're
| doing is accuse of censorship without evidence, you say they
| screwed up "so many times" yet fail to show one example, and
| then you preemptively accuse people who would dare downvote
| you to be fanboys.
|
| Sorry, but that comment is noise. Mine is too, but hopefully
| it helps you see things more clearly rather than let you pat
| yourself on the back thinking "HN is full of fanboys anyway".
| paulddraper wrote:
| Please do update if possible.
|
| It's likely a good learning for all.
| jgrahamc wrote:
| Of course, I just don't want to promise something when it
| might be revealing information about an individual account.
| andrewstuart wrote:
| I've put this idea forward a number of times here on HN in
| regards to other big tech companies.
|
| Technology companies need an "ombudsman" - a contact that
| customers can go to when the normal tech support processes have
| failed.
|
| The Ombudsman must _not_ be part of the technology companies
| ordinary support processes, it must be entirely separate, and
| have highest level authority to demand action within the
| company.
|
| To avoid the Ombudsman being overused, you could give it a
| price of say $20, which is always refunded when the case is
| resolved.
|
| HN constantly has front page posts from people for whom big
| tech companies have support processes have failed but there is
| simply no other recourse unless you have "a friend in the
| business".
|
| It just doesn't work to have some random Cloudflare person
| offer their email address as some post disaster issue
| resolution process on social media. Formalise it with an
| official Ombudsman and maybe then companies like Cloudflare
| might avoid HN front page bad publicity.
|
| I had an issue at "one of the biggest tech companies" that went
| on for days and days in which tech support kept telling me I
| had set up something wrong, until eventually I emailed one of
| the top managers who I happen to "know" at that company - it
| was fixed within hours. That "contact a friend in the business
| who can actually get things done" is a necessary part of a
| large support organisation and it simply does not exist yet in
| any tech company that I know of.
| gist wrote:
| > Technology companies need an "ombudsman" - a contact that
| customers can go to when the normal tech support processes
| have failed.
|
| From what I read this is nothing to indicate the process
| failed (so far) just that the user decided to skip to the
| head of the line by writing a blog post internet style to get
| something resolved and attention. Failed is not 'I didn't get
| a reply or find what I needed as quickly as I think it should
| happen so now let me complain publicly so I get a reply'.
|
| > To avoid the Ombudsman being overused, you could give it a
| price of say $20, which is always refunded when the case is
| resolved.
|
| In theory nice but first it would be a 'deposit' and also
| opens up a host of new issues as far as the money being paid
| back and how that would be done and so on.
| iudqnolq wrote:
| The core of what you say is correct. I posted about this
| publicly with two goals in mind: getting help from someone
| at cloudflare and getting advice on how to avoid this sort
| of issue from the HN community.
|
| From my perspective Cloudflare's process did fail. Assuming
| I didn't do something insanely dumb and what I think
| happened did happen, I would consider that a failure on its
| own even if the support was perfect afterwards.
|
| Copying from elsewhere:
|
| I address this in TFA.
|
| Essentially I felt that this was alright because when I
| filed a ticket I was informed that I should expect a long
| wait and that they recommend that their non-business
| customers post publicly on their support forum for
| crowdsourced support because that leads to faster replies.
| I was unable to log into that forum, and I suspect that may
| be because the way they set up SSO between the forum login
| and their main login may have failed in Firefox (with all
| tracking prevention and ad blockers disabled).
|
| I felt that if a company invites me to ask for support
| publicly on their forum to save on customer support costs
| it's reasonable to talk about the issue in another public
| place.
| basch wrote:
| Ive run into Cloudflare admin pages that fail with ad
| blocking before. Test your theory in a private session,
| it works for me when their site "has issues."
|
| (I think in my case it was adding google metrics from the
| apps page.)
| wolco wrote:
| This is a fair point. If users are directed to post
| publically for attention for support this user did the
| right thing.
| bonestamp2 wrote:
| I like this idea a lot. Some people below are suggesting that
| the ombudsman should be the last stop in a support queue if
| your problem isn't resolved, and that makes sense sometimes,
| but other times you can't wait that long!
|
| So, all support systems should have a triage type system with
| a "nurse" having a constant eye on every new case that comes
| into the support system. When there's an emergency, such as
| the one associated with this post, then it should be
| forwarded to the ombudsman or some other emergency team
| immediately.
| briandear wrote:
| > Technology companies need an "ombudsman" - a contact that
| customers can go to when the normal tech support processes
| have failed.
|
| All companies should have that!
| sbarre wrote:
| This sounds nice in theory, but you know that tons of
| people would just go straight to the ombudsman, thinking
| they can jump the support queue or bypass established
| process.
|
| The "shit filtering" workload would be tremendous..
| wolco wrote:
| The way this works everywhere is you go through the
| normal support process until you reach top level support
| if the situation isn't resolve you go to the ombudmen. If
| you go before you tried support they will direct you to
| support first.
| ignoramous wrote:
| In Amazon's case, Bezos' email is just that. For AWS, Andy
| Jassy's (CEO) or Charlie Bell's (SVP) might do just as
| fine.
|
| Source: Someone who emails them from time to time to get
| impasses resolved.
| bbarnett wrote:
| Just had to chime in here, but fifteen+ years ago, Bezo's
| email was ombudsman like, with him or a top level person
| reading it.
|
| Now, it's just another level of very poor, very scripted
| support.
|
| I'm fairly sure Amazon has taken the (perhaps wise, in a
| business sense) approach of not caring if a small
| percentage of users leave, due to support issues.
|
| The cost of keeping customers with certain support
| issues, greatly outweighing supporting them.
|
| This is why you have to hunt madly around Amazon's
| webpage to find contact info, why all forms of help point
| away from contacting a person, including their chat being
| bots now, until you move outside of their scripts.
| ignoramous wrote:
| Just one data point but, sending Bezos a nice hello and
| airing my grievances has worked for me every single time.
|
| Obviously, Bezos may not read those emails but his aides
| and assistants who do have access to his inbox and act on
| the emails on his behalf do inherit his complete
| authority.
|
| Some refs:
|
| https://news.ycombinator.com/item?id=16341154
|
| https://news.ycombinator.com/item?id=17193363
|
| https://news.ycombinator.com/item?id=22286350
|
| https://news.ycombinator.com/item?id=9356182
|
| https://news.ycombinator.com/item?id=20782392
|
| https://news.ycombinator.com/item?id=13512106
| gregd wrote:
| A properly setup helpdesk negates the need for an Ombudsman.
| If a tech company cannot get a helpdesk escalation path
| correct, what makes you think they'll get an Ombudsman
| scenario correct?
| p0sixlang wrote:
| Some companies, this doesn't work at all. EG: Postmates.
| Emailed pretty much every one of their executive/management
| team about a literally brand damaging issue, and received
| zero bounces, but also zero replies. Some companies have a
| policy to ignore unsolicited emails, no matter how serious
| the issue, as to not fuel the idea that doing so will get
| results in the future.
| awill wrote:
| This is a really great idea, but I don't think it's possible
| for this to not get overused for every little issue. Once
| it's overused, it becomes useless.
| [deleted]
| nexuist wrote:
| Isn't this the point of the monetary hold? You can just
| raise it until the amount of entries become manageable. I'm
| sure higher level orgs would easily put down $10k to talk
| to a developer at Microsoft. For indie devs, numbers like
| $100 or $1000 could be manageable, as long as they can
| trigger the refund and close the case whenever they want.
|
| Arguably this does block out poorer people from receiving
| "special" customer service, but there are not really other
| things people are willing to lose (or put up as collateral)
| for this type of service. I can't really ship Cloudflare my
| toaster or car until they resolve my case.
| benologist wrote:
| Tech companies being inundated with complaints would
| actually be expected. Making it easy to complain to
| ombudspersons will cause the complaints to dwindle
| naturally as tech companies stop the unethical, even
| criminal, behavior they engage in.
|
| Very few people will complain about their lifetime Google
| ban _after_ Google employs appropriate personnel to
| evaluate such cases.
|
| Today very few people complain about Steam's refund policy,
| after Valve rewrote their refund policy to actually include
| refunds, after a judge ended their decade-long crime spree
| that saw an estimated 20,000 Australians robbed and an
| unknown quantity globally.
| CamelCaseName wrote:
| I like the setup my bank's Ombudsman has -- you must first
| take your issue to first level support, then escalate it
| with them if not resolved. If the second level of support
| denies you, then and only then can you reach out to the
| Ombudsman.
|
| Any requests that haven't gone through the proper process
| get auto-rejected.
| lubujackson wrote:
| That is the logical way for things to work, but it
| requires first level tech support letting you escalate,
| which is not always the case with non-bank industries.
|
| The current go-to move is to tweet a complaint at the
| company's Twitter account. This is surprisingly effective
| across multiple industries and actually was something my
| wife did that helped resolve a time-sensitive AirBnB
| issue.
| cmroanirgo wrote:
| Great idea.
|
| Maybe it could be something that is given to someone when
| their ticket is closed (or maybe after the first tech
| response... it depends on the company/ corporate structure).
|
| That way the ombudsman has something to work with, and would
| slow down the barrage that would occur by having a such a
| public contact point.
|
| I'm never a fan of 'pay then get refunded' for something
| that's not your fault, and is entirely out of your control.
| nickjj wrote:
| > To avoid the Ombudsman being overused, you could give it a
| price of say $20, which is always refunded when the case is
| resolved.
|
| What if the cost was put onto the business instead of the
| consumer and the business just hired support people who are
| all Ombudsmans by default?
|
| Instead of focusing on copy / pasting boilerplate scripts and
| answering as many tickets as possible, they should focus on
| the problem the customer is having by default and do
| everything possible to reduce the number of incoming
| questions by fixing bugs, making a better product, improving
| their docs, etc..
|
| I personally do around the clock email support for 35,000+
| people who sign up to my programming courses and support
| isn't bogging me down. Relative to the number of minutes I'm
| awake, support is one of the least business related time
| consuming things I do per day, but I send individually
| personalized in-depth answers to everyone who asks me
| questions -- usually within an hour or less.
| foota wrote:
| Because requiring the customer to pay a cost fixes the
| incentives problem. A customer may have an issue that is a
| minor inconvenience to them, that can either stay as not
| resolved, is due to error, or is simply not worth resolving
| to them. By having them pay for fixing the issue it could
| greatly reduce the number of inbound cases, allowing them
| to go straight to people that can act on them instead of
| sitting in triage and getting canned responses for
| O(days|forever).
|
| I agree that in theory you can accomplish the same thing by
| making the product foolproof, but I don't think you can
| accomplish that for consumer facing products, and that
| doing say may not be a worth while trade-off. Additionally
| focusing on issues that greatly impact people rather than
| small things that cause friction with the product may (or
| may not! if it causes lack of retention) be worth more.
| TylerE wrote:
| Because that isn't a useful allocation of resources.
|
| Support that powerful are going to basically be devs. With
| dev salary expectations.
| dorfsmay wrote:
| At 20 USD a pop, Google would make a small fortune!
|
| I'd be the first to pay to get them explain to me some of
| their misterious weirdnesses.
| [deleted]
| sofaofthedamned wrote:
| That's actually a really good idea. Bit like a Credit Card
| Chargeback, except it's charged to the customer and is
| refundable.
| RcouF1uZ4gsC wrote:
| > To avoid the Ombudsman being overused, you could give it a
| price of say $20, which is always refunded when the case is
| resolved.
|
| HN actually acts somewhat like a crowd-sourced ombudsman.
| People who have an issue write a description and post it to
| HN. If enough people find it compelling, it makes the front
| page. Once it makes the front page, someone in authority at
| the involved tech company will see it, and try their best to
| resolve it.
| subhro wrote:
| This is a really nice idea. This already exists for some
| companies in the form of Twitter accounts. I doubt the people
| doing the typee-typee actually has any authority over
| business processes, or demand a change, but I think they at
| least have the business owners on their speed dial, vs normal
| support tickets. But having an email is far better.
| acangiano wrote:
| Amusingly, Twitter itself could use one. My account
| (@acangiano) has all of its images censored under
| "sensitive content" even though they are 100% benign
| images. No amount of tweets to @TwitterSupport has done
| anything at all to change it. There is basically no
| recourse. My account is like 12 years old and has 4.5K
| followers, so it's not like it's a random spam account,
| either. -\\_(tsu)_/-
| andrewstuart wrote:
| Yes it's a good point that companies currently have some
| sort of Twitter presence trying to address bad publicity
| posts, often they seem to be able to get things done.
|
| The Ombudsman role is there to get things fixed when all
| else has failed, and _before_ the angry customer posts to
| social media.
| p0sixlang wrote:
| I find this whole infrastructure, despite my taking
| advantage of it, to be very flawed. There're many customers
| who might have alarming issues, who never get attention
| because they're fearful to be perceived as a "Karen" for
| bitching on Twitter.
| iotku wrote:
| >they're fearful to be perceived as a "Karen" for
| bitching on Twitter.
|
| On the other side of the equation sometimes I do want to
| complain for the sake of complaining without being
| harassed by some support account.
|
| Especially hate getting obviously automated responses for
| daring to mention company names even if it would actually
| escalate to a human.
| tolstoshev wrote:
| That often ends up being the defacto job of a online
| community manager, if they have one.
| jfkebwjsbx wrote:
| $20 won't stop anyone with using it.
|
| If it is refundable, make it $300. That is high enough that
| only people and businesses with a showstopper situation will
| use it.
| jlawer wrote:
| $300 becomes a pretty high barrier for a lot of people.
| There a places with decent connectivity where that is a
| good weeks wages
| basch wrote:
| But you get it back, the point is to make it a penalty if
| you are wasting their time.
| herbstein wrote:
| > But you get it back
|
| You're missing the part where some people, that actually
| would need this support, literally wouldn't be able to
| find that money because of the difference in purchasing
| power of the local currency.
| basch wrote:
| I was thinking price for business not personal accounts.
| Maybe youre thinking those are the same thing in
| freelance.
| bradknowles wrote:
| A single dollar is a high barrier for some people.
|
| You need an adjustable amount that is based on the annual
| income/revenue of the person/entity making the request.
|
| Make it high enough to be non-trivial, but not so high
| that it blocks all effective usage of the safety valve.
|
| Now, if you can solve that problem, I've got some bridges
| for you in Arizona.
| jlawer wrote:
| I think the only sane thing you can do is price it at the
| cost it takes to review it. It will still be out of reach
| of some people, but at least its not arbitory.
| bsder wrote:
| Make it $20 but not refundable. Or $300 and not refundable.
|
| This is a good way to filter the full level of importance.
| Most places that need a problem _solved, now, dammit_ will
| be willing to pay.
|
| And, if you want to be nice, you _can_ refund it. Or, if
| the person was a jerk, you can keep it.
| chopin wrote:
| If I have a rightful complaint and it costs me $300 to
| resolve I am done with that business.
| turbostyler wrote:
| I think you're incorrectly assuming most companies want to
| provide good customer support.
| mpitt wrote:
| jgc is not a "random" Cloudflare person ;)
| andrewstuart wrote:
| My point is precisely that to me, "jgc" IS a random person.
| How the heck do I know who this person is.
|
| It shouldn't matter, and it should not be required, that
| someone "known and important" within an organisation
| decides to start doing hands on tech support in social
| media following a PR disaster.
|
| If "jgc" is actually someone important within this company
| then maybe after fixing this issue, they can then go fiox
| their tech support by setting up and ombudsman and get
| their PR disasters off the front page of HN.
| Swizec wrote:
| It's the CTO according to their HN profile ;)
| gjs278 wrote:
| stop winking you fucking weirdo
| inetknght wrote:
| You're missing the point. It shouldn't be necessary for
| someone to know who's-who in order to get things
| resolved.
| cabaalis wrote:
| You may be right, but knowing "who's who" is very largely
| how general business gets done. Buying services over the
| internet from an anonymous black box with no support is a
| recent disruption.
| kortilla wrote:
| No, normally you didn't have to know someone in the C
| suite to "get business done". That's totally unscalable.
|
| What's a recent development is the complete lack of
| support when shit goes south. Back when you were
| interacting with real reps you had people that could see
| when stuff was obviously wrong and escalate
| appropriately.
| cabaalis wrote:
| Did I say anything about C-suite? Your point is nearly
| word for word the same as mine, I'm not sure why you're
| replying as a refutation.
| davchana wrote:
| For a regular generic Cloudflare customer like me, for
| personal use, jgc is one of the random Cloudflare person. I
| have started a spreadsheet with his name, email, comment
| link, and my copy of screenshot of comment; just in case if
| I need to email him anything in future.
| rationalfaith wrote:
| You better add redundancies here on untracked transactions on
| your DNS record ledger.
| iudqnolq wrote:
| OP here.
|
| You can post updates with any relevant information. Probably
| goes without saying, but if the issue has to do with my billing
| or address please don't post specific details without asking me
| first.
|
| I will link to this comment from TFA for verification. (Edit:
| added to the bottom. If you need more verification you have my
| email.)
|
| Edit2: I see that the domain is back in my account and listed
| as "Pending Nameserver Update". I don't think that's because of
| something I did.
| graiz wrote:
| Maybe contact support first, figure out what happened and why
| before writing the blog post?
|
| Maybe it's a cloudflare issue, maybe it's an honest mistake,
| maybe they are bad at customer service... either way, it's
| not great to shame a company before you even open a support
| ticket or talk to someone to find out.
| iudqnolq wrote:
| I address this in TFA.
|
| Essentially I felt that this was alright because when I
| filed a ticket I was informed that I should expect a long
| wait and that they recommend that their non-business
| customers post publicly on their support forum for
| crowdsourced support because that leads to faster replies.
| I was unable to log into that forum, and I suspect that may
| be because the way they set up SSO between the forum login
| and their main login may have failed in Firefox (with all
| tracking prevention and ad blockers disabled).
|
| I felt that if a company invites me to ask for support
| publicly on their forum to save on customer support costs
| it's reasonable to talk about the issue in another public
| place.
| bithaze wrote:
| Oh, the horror of recommending a public forum. (/s, of
| course.)
|
| A fair number of questions aren't unique - product
| questions, how to use an API, etc. Someone may have asked
| a similar question, in which case you'll find an answer,
| find it faster than it'll take to hear back from the
| support team, and it deflects an unnecessary (already
| answered) ticket. That should be a win all around.
|
| Now if you do have a novel question or something account-
| specific, by all means, open a ticket. There you'll get
| replies from people who can look up your account and give
| you specific answers.
|
| The ombudsman tip in this post doesn't make a whole lot
| of sense when the normal support process wasn't really
| given a chance before making the blog post.
| tracker1 wrote:
| When your entire site and suite of applications are
| offline, especially during business hours, waiting who
| knows how long for a support queue isn't really an
| option. Outages like this can kill a business or cost
| millions.
| iudqnolq wrote:
| (op here). To be fair to Cloudflare a business with
| millions on the line should be paying a lot more money to
| Cloudflare than I, a college student with significantly
| less on the line, would. If I had the option to pay
| $1,000 right now and get everything back up instantly I
| wouldn't take it because it isn't worth that much to me.
| JungleGymSam wrote:
| And maybe not too.
| subhro wrote:
| I think it is a bit harsh to say that the OP was trying to
| shame anyone. He did post a support ticket. When your
| domain is "off-line" it is literally a shit hit the fan
| moment, and no one likes to be given the run-around.
|
| To combat, hey my issue is always a Sev1 ticket, one can
| probably institute something like, here is a red button and
| if you click it, we will charge 100$. If it is indeed an
| issue that caused you to lose 90%(say) traffic and it was
| our fault, we will return the money.
| packetslave wrote:
| Microsoft used to have this policy (maybe still does) for
| some support options. It cost $100 to open a case, and if
| the case was the result of a bug in the product, you got
| your money back.
| dkersten wrote:
| Imagine how you would feel though if you were in OP's
| shoes, where your DNS entries disappeared and client
| emails lost, through no fault of your own, but support
| won't even listen to you until you pay money.
| iudqnolq wrote:
| I think I'd be fine with that?
|
| I would trust Cloudflare to pay me back, and I'm already
| putting something on the line. If it really was my fault
| this is going to be very embarrassing.
|
| Edit: In case it isn't clear from context I'm the OP so
| I'm pretty sure I know what I feel.
| goatsi wrote:
| In the blog post he mentions that Cloudflare recommends
| posting your problem to their open community forum to try
| and get things resolved. He couldn't access their forum, so
| he posted it on a different one.
| jgrahamc wrote:
| Thanks. Appreciate it. I'll let the team look into it and
| communicate with you first.
| AviationAtom wrote:
| John, you guys seem pretty awesome about posting root cause
| analysis and being transparent, can we expect to see at
| least a comment (if not a blog post) summarizing what took
| place here?
| iudqnolq wrote:
| I appreciate you're looking in to this.
| jgrahamc wrote:
| I've worked for Cloudflare since it was 24 people. I care
| a hell of a lot about our customers. I know eastdakota
| does also.
| iudqnolq wrote:
| I can tell. Cloudflare has been a delight to use up to
| now and I'm grateful I'm getting this kind of support
| when I've only used your loss-leader services.
| melq wrote:
| This is probably obvious but you're getting this kind of
| support because they're getting terrible PR by virtue of
| your post being at the top of HN. Not because he's such a
| great guy that cares so much about the customers. Not
| saying he isn't a great guy, or that anyone at cloudflare
| actually wanted this to be your experience.
| craftinator wrote:
| With many other companies I would agree with you, but in
| this case it is actually just because he's running a
| tight ship. I've had friends who've reached out directly
| via email and received the same level of support.
| jgrahamc wrote:
| If he'd emailed me directly I would have done the same
| thing I did. I emailed the head of engineering and
| support and asked for an explanation. I then jumped into
| the relevant chat room.
|
| I do this sort of thing all the time. Sure, it's
| unfortunate this is #1 on HN, but shrug. Fixing the
| problem and figured out what happened is important.
| dkersten wrote:
| The more important question is what will be done to
| prevent this from happening again to somebody else
| (obviously this doesn't need a response here, I assume an
| internal investigation is underway and this will be
| publicly communicated after its complete).
| melq wrote:
| How was he supposed to know to email you?
| adar wrote:
| I don't think he's saying that OP should've known to
| email him, just that he would've done the same thing over
| private emails where nobody was watching, as he is doing
| here publicly.
| nickjj wrote:
| To be fair this seems to happen with a bunch of tech
| companies, such as Stripe too. I wish more companies took
| customer support seriously.
|
| I remember about 4-5 months ago I spent like 2 weeks
| going back and forth with Stripe's regular email support
| trying to understand their docs for SCA.
|
| I kept getting a new rep who repeated the same things the
| previous reps were saying, which also had no bearing on
| what I was asking. It was basically a copy / paste from a
| script loop.
|
| Then something negative about Stripe was on the HN front
| page and I happened to comment about a bad experience
| with the new SCA docs.
|
| Within a few hours I was put in contact with a lead
| developer from Stripe who went as far as creating custom
| flow charts for my use case that wasn't covered in the
| docs and it was a pleasant experience, where "pleasant"
| felt like the person receiving the email was reading the
| words I wrote instead of just skimming them and pasting a
| boilerplate response.
|
| But it only happened because of the HN comment. If that
| thread never appeared on HN, I'm not even sure I would
| still be using Stripe.
| lallysingh wrote:
| > was a pleasant experience, where "pleasant" felt like
| the person receiving the email was reading the words I
| wrote instead of just skimming them and pasting a
| boilerplate response.
|
| Crazy how far down the bar has gone.
| AviationAtom wrote:
| You mention loss-leader services, but the model is no
| doubt setup to funnel you towards profitable services.
|
| I switched a personal domain to them, but it won't be
| long before I get my employer to move their resources
| over to them, assuming I see this all play out well.
| nodesocket wrote:
| I do wonder; when a domain is suspended, why don't you send
| out a courtesy notification e-mail?
| wu_187 wrote:
| what if their email address is @ the domain
| belorn wrote:
| If the only contact information the registrar has is the
| @ of the domain then that likely mean that the registrar
| is failing the contractual obligations that exist between
| the registry and registrar. While I have not read the
| exact contract that exist for .com registrars, I am
| confident enough to say that you can not do that.
| iudqnolq wrote:
| I think it's pretty common advice that anything domain-
| related needs to be a different email. I've heard that,
| and I've only ever admin'd my personal domain. For
| exactly that reason I registered for Cloudflare with a
| gmail account.
| clowd wrote:
| Well, that's poor planning on the user's part. In this
| case the OP stated he registered using a gmail address.
| nodesocket wrote:
| Also display a prominent notification in the web
| interface then.
| jiggawatts wrote:
| Please explain something to me.
|
| For me and my of my customers, having your "entire cloud
| deleted" is like... the #1 nightmare scenario.
|
| So why does this capability/function even _exist_ for active
| accounts at CloudFlare? It sounds like the OP fell victim to
| what is essentially a regular process.
|
| Or to put it another way: No amount of explanation or assurance
| is ever going to make me feel comfortable with my doctor having
| a handgun as one of his medical instruments.
| diegoperini wrote:
| First, you are awesome, really :)
|
| Second, a bunch of honest questions:
|
| Did you consult to your supervisor (or anyone with authority)
| to be able to bypass the support process (if there is any) like
| this? If so what was the response? If the response was
| negative, how did you convince people? After things resolve,
| can you kindly post how many spam or unrelated emails you
| receive so that it will be an example to the industry?
|
| I'd like to put my skepticism on hold and blindly believe that
| your post is a reflection of pure concern and not just a PR
| stunt for damage control.
| wila wrote:
| The CTO at Cloudflare has to ask his supervisor before trying
| to help someone over at hacker news? :)
| diegoperini wrote:
| I didn't know which probably slightly proves some point
| other people made.
| stedaniels wrote:
| jgrahamc is the CTO of Cloudflare. His only supervisor is
| eastdakota/ Matthew Prince, the CEO of Cloudflare.
| diegoperini wrote:
| Then I'd like to know the CEO's opinion then.
| rattray wrote:
| Context for the lazy, jgrahamc is the longtime CTO of
| Cloudflare.
| ajonit wrote:
| Now that the OP has given a go ahead to go public, We will
| eagerly wait for your update jgc
| gist wrote:
| > BTW If you, dear reader, ever find yourself so frustrated
| with Cloudflare that you feel like your only recourse is a blog
| post... my email is jgc@cloudflare.com and I'm happy to hear
| from people.
|
| I know that people will think it's great that you are doing
| this and I also know that you think it's good (for you) to have
| a feel for the issues that frustrate every day users. But I
| think it's not a great use of a company execs time and I am not
| even sure it's a good way to deploy resources at Cloudflare.
|
| The reason is people will tend to (as a rule) do as little as
| they can themselves but then use as a hammer the court of
| public opinion to get something resolved.
|
| You say 'ever find yourself so frustrated with Cloudflare' but
| you know that in itself is different for different people. What
| will happen is you will get people using you as a help desk and
| then after you don't help them as quickly as they think you
| should they will then follow up with a post, comment or story
| about how you did nothing.
|
| Separately if someone is posting publicly about an issue (as
| this person is) and if you can verify that it's actually coming
| from the customer (I mean who says it is actually?) I don't
| think you need them to say it's ok to resolve online. In fact
| to me it's the opposite. You take the time to reach out
| publicly and you take what follows good or bad even calling you
| out (the customer yes you can do that by the way) if you think
| they didn't put the appropriate effort into finding an answer.
| jgrahamc wrote:
| I disagree with you. It matters enormously that people like
| me are available. Sure there are time wasters who'll send me
| email. But I don't care. Dealing with a small number of real
| customers doesn't take a lot of my time and matters.
|
| Everyone optimizes for the worst case. They think "if I give
| out my email I'll get tons of useless email". I can assure
| you I get 10x the crap sent to me on LinkedIn than via direct
| emails from customers or others.
| dkersten wrote:
| Thank you, this is a good attitude to have and I wish more
| company exec's thought like this. We might have fewer
| post's like OP's story.
| gerdesj wrote:
| My little firm is nothing like the scale of CF but I do
| the same and have dropped my email address around a fair
| bit. I do get some pretty interesting missives that make
| it past the filters but signal to noise is very high in
| my inbox.
|
| I deleted my Linkedin account well over a year ago and I
| still get emails from them saying my profile is being
| viewed. Tossers.
| jschmitz28 wrote:
| > I know that people will think it's great that you are doing
| this and I also know that you think it's good (for you) to
| have a feel for the issues that frustrate every day users.
| But I think it's not a great use of a company execs time and
| I am not even sure it's a good way to deploy resources at
| Cloudflare.
|
| As a counter example, Jeff Bezos (whose time may be worth
| more than anybody else's) famously audits his email for
| customer complaints and occasionally derails an organization
| for a day or two in order to figure out what happened. He
| stands behind this practice and has said that he often picks
| out cases where the anecdotal complaint is counter to data
| that he's been presented, and that more often than not the
| anecdotes are correct and find a shortcoming in the data. IMO
| it also demonstrates a culture of caring and following up
| about anecdotes to others whose time is worth less than his
| own.
| inetknght wrote:
| > _As a counter example, Jeff Bezos (whose time may be
| worth more than anybody else 's) famously audits his email
| for customer complaints and occasionally derails an
| organization for a day or two in order to figure out what
| happened._
|
| Is that why Amazon is growing more and more notorious for
| selling fraudulent items over the years?
| tempestn wrote:
| Maybe someone needs to email Jeff about that.
| [deleted]
| SanchoPanda wrote:
| Both of those things could be true without one being
| causal or even related.
| parliament32 wrote:
| Cloudflare is pretty trash regardless, but putting all your eggs
| in one basket (no matter which provider) is just a terrible idea.
| johnklos wrote:
| Is it really all that surprising when a big company that claims
| to be good but hosts phishing content in the name of free speech
| does whatever they want, including breaking things and not
| explaining why?
|
| I don't trust Cloudflare one bit, and I think everyone should
| question whether their attempt to re-centralize everything is
| beneficial to the planet.
|
| There are two major problems here: one, the problem itself, which
| is the deletion of DNS for apparently no good reason, and two,
| which is the bigger problem, is that it's incredibly difficult to
| talk to a human about what happened, so there's no assurance it
| won't happen again.
|
| If people want things to be reliable, we've got to stop using
| companies with which we cannot communicate.
| djsumdog wrote:
| Does OP have a free account with just DDOS protection. Does a
| paid account still have the notice to ask in the community
| forums first?
| iudqnolq wrote:
| OP here. I pay them for domain registration, which they offer
| at cost. I use their free DNS. I disable DDOS protection.
| ocdtrekkie wrote:
| IMHO (and I know the parent post includes significant
| difficulties getting back out of Cloudflare), services like
| Cloudflare may be crucial to decentralization. I _can 't_ deal
| with something like my blog post being frontpaged on HN if my
| website is hosted in my house, unless I have a good CDN.
|
| As a self-hosting enthusiast, something like Cloudflare is one
| of the best chances of having a plan that competes with "just
| hosting it in the cloud".
| vorpalhex wrote:
| There's alternatives to Cloudflare that offer affordable-but-
| not-free CDNs which has always felt less risky to me. I'd
| rather know I'm the customer instead of the product.
| johnklos wrote:
| I hear you, but their DDoS services are painful to the rest
| of the world and to people who want or need to use Tor, and
| others.
|
| I'm talking about their rather political move to re-
| centralize DNS by shoehorning themselves in to Firefox via
| DoH, for instance. Their unwillingness to be transparent
| makes this all the more frightening. Add to that their
| blatant desire to make money at the cost of doing the right
| thing (and I'm talking about unambiguous things - is someone
| going to argue that freedom of speech allows people run a
| phishing site of your bank?), and you've got a scenario where
| once they reach critical mass, they will be exercising their
| position to the detriment of everyone who isn't paying them,
| similarly to how Gmail, through doing and not communicating,
| say "screw you" to many small email services.
|
| When people who don't use large providers have email issues
| with Gmail, lots of people have knee-jerk reactions saying
| that everything should move to the big providers, that people
| and small businesses should not host their own email, and so
| on. This is NOT the way the Internet should work, and we
| should never allow Gmail to just arbitrarily do whatever they
| want, then accept it as the new normal.
|
| If you have more than a dozen megabits of outgoing bandwidth,
| you can easily host a blog from your home network which can
| handle a front paging here. Just don't expect to dynamically
| generate a new copy of the site for every visitor, and if
| your bandwidth is tight, then host your images on a static
| server off of your network. Cloudflare is not necessary -
| perhaps it's easier, but it isn't necessarily best to blindly
| trust a company that wants to become a monopoly.
| zozbot234 wrote:
| > I can't deal with something like my blog post being
| frontpaged on HN if my website is hosted in my house, unless
| I have a good CDN.
|
| IPFS can work as a CDN, at least for static content that
| users are willing to seed. This is especially relevant to the
| "blog post hits frontpage on HN" case. Of course, dynamic
| content is not quite as easy.
| luckylion wrote:
| I don't know. Sure, maybe not on your barely-broadband DSL
| connection, but I'm pretty sure you can run most things on a
| average shared webhosting, even if you're using WP. You just
| need to make sure that you have a working caching system in
| place. It's a gigantic difference even on an apache to just
| read & send a plain file and invoke PHP & run all the costly
| code. I don't believe HN should bring down any site that can
| essentially be cached as static HTML.
|
| Cloudflare will only help once your server has gone down with
| their "Always On" thingy, if you have that enabled. They
| don't cache HTML by default.
| djsumdog wrote:
| I really had a strong dislike for Cloudflare after they banned
| certain customers for political reasons[1]. The CEO mentioned how
| maybe it wasn't the right thing to do .. and then they did it
| again.
|
| There aren't really any self-hosted solutions for DDoS protection
| like Cloudflare since it requires things happening in the network
| layer. Implementing a solution would require access to monitor
| and reshape the local network, but I'm glad to see companies like
| Linode and DO offering DDos package.
|
| I want to start running my own DNS-over-HTTPS server as well, so
| I can pump firefox DNS requests to a self-hosted solution and not
| to Google or Cloudflare. I really don't trust them and am having
| trouble understanding why so many other people do.
|
| [1]: https://battlepenguin.com/politics/the-new-era-of-
| corporate-...
| craftinator wrote:
| If you run a restaurant, you can refuse to do business with
| anyone you choose. If that was not the case, you would
| effectively be a slave; unable to choose actions for yourself
| and your business. Cloudflare refused to do business with
| people and content; that is their prerogative.
| djsumdog wrote:
| Did you read the article I wrote/cited?
|
| > A store cannot have blacks only and whites only bathrooms
| or water fountains. Bars and restaurants in some
| jurisdictions can allow smoking within their establishments,
| while in other municipalities, smoking indoors is banned for
| all businesses. Companies who chose to be equal opportunity
| employers have several criteria for which they cannot
| discriminate against. Laws such as the Americans with
| Disabilities Act mandates certain accessibility requirements
| in order to maintain a storefront ... Speech does not yet
| fall into any of these existing regularity frameworks.
|
| So no, you're wrong. You cannot refuse to do business with
| anyone you choose. The Colorado cake case is a really special
| one, because it had to do with art. As an artist, you can
| refuse a commission to build a creative work if it goes
| against your values. The guy who ran that shop just stopped
| accepting custom orders, and then later got in trouble again
| when he refused to sell plain non-custom cupcakes to a gay
| couple.
| HBKXNCUO wrote:
| >If you run a restaurant, you can refuse to do business with
| anyone you choose.
|
| You will suffer legal consequences if it's determined that
| you refused to do business with them on the basis of certain
| characteristics protected by law. Property rights are not
| absolute. Exceptions to them can be made if people think
| there is good reason to do so.
|
| The situation we are in now, where technology companies have
| found themselves with wide power to control the public
| political debate occurring among regular people merely as a
| consequence of successfully running some particular types of
| business, is not one we've really seen before. There are some
| very persuasive arguments for limiting their property rights,
| similar to how they were limited e.g. ~50 years ago by the
| civil rights act.
| lexicality wrote:
| While there are a lot of reasons not to trust cloudflare, the
| fact that they stopped hosting nazis and pedophiles doesn't
| seem like a good one to open with imo
| [deleted]
| dependenttypes wrote:
| > and pedophiles
|
| Is that a reference to 8chan? Because if so I am pretty sure
| that they removed controversial boards of this form a few
| years back - long before cloudflare banned them.
| djsumdog wrote:
| I think it is a problem when you talk about fear of ideas.
| You can label anyone as a Nazi today, and when you ban
| people, you kinda give them power.
|
| It shows you're afraid of their ideas, and the persecution
| can embolden them or give them a sense of legitimacy. It can
| being the Streisand Effect to their cause, taking a no-name
| site no one knew or cared about and blowing it up into
| something everyone is deeply aware of.
| claudiawerner wrote:
| >and when you ban people, you kinda give them power.
|
| The key word being "kinda". You give the perception of
| power, but there are plenty of things in society we deal
| with by a form of "banning" (such as incarceration) in
| which we judge the ban to be good regardless of the power
| it gives people, or the ideas they represent or practice.
| Locking up child abusers, for instance, may give the
| spectre of child abuse power, and highly-publicized
| instances may fuel the moral panic around "strangers out to
| get your children", but that doesn't mean they should not
| be locked up.
|
| In civil society, we are justifiably afraid of many ideas -
| I don't know anyone who wouldn't be afraid of a Nazi-style
| dictatorship, or its prospect. Fear can be a legitimate way
| of preventing bad things from happening. I fear dying in a
| house fire, thus I take certain precautions when cooking.
| In the same way, a demonstration of power _over_ a person
| or idea may outweigh the power supposedly given to that
| idea by banning it.
|
| The key is a balance; it may well be that you get an
| instance of the Streisand Effect, but it has to be shown
| that the consequences of that outweigh the very material
| consequences of such ideas coming to fruition in real life.
| For example, many people don't know who Barbara Streisand
| even is, and if they do, they likely don't know about the
| pictures of her house. The very canonical example of the
| Streisand Effect shows a short-lived controversy about the
| actual matter and ensuing attention for a few months after
| the incident. Then people forgot, or simply stopped caring.
| "The Streisand Effect" is more of a Streisand Effect than
| the actual incident that created it.
| HBKXNCUO wrote:
| >I don't know anyone who wouldn't be afraid of a Nazi-
| style dictatorship
|
| Do you believe that if people with ideas that you
| consider to be Nazi-like are allowed participate in the
| public debate as freely and meaningfully as everyone else
| (e.g. by using electronic services regularly available to
| everyone else), a Nazi-style dictatorship is likely to
| come about?
| claudiawerner wrote:
| No, I don't, but I do think it increases its chances of
| happening - and there is historical precedent for it. For
| that reason, I sit somewhere between Popper's paradox of
| tolerance on these matters (and I go further than him),
| Sartre's notes on anti-semitism, and Marcuse's criticism
| of simple plurality as a substitute for educated and
| rational thought.
|
| To be clear, I don't advocate for censorship of ideas
| that I simply "don't like". That's not a sufficiently
| rigorous standard. Ideas which advocate for targeting
| marginalized groups, or entire groups of people for ideas
| they have no control over, are fair game, in my opinion.
| I don't pretend to have no bias in my answer to that
| question. I am biased, and others have their own biases.
| I draw the line where I want to draw it, with no concern
| for pretending to derive it from first principles.
| HBKXNCUO wrote:
| To get this straight, you are stating that you believe it
| is in your interest for certain other people who live in
| the same society as you to be prohibited from attempting
| to further their own interests by freely and meaningfully
| participating in the public political debate to the same
| degree that you and others in your society are able.
|
| Do you think that's going to end well? How do you expect
| those people to feel about you? There is a very good
| reason why societies have protected the right to
| political speech, and it is to prevent the inevitable
| conflict that arises when some people in society feel
| that the rest of society is preventing them from
| attempting to further their own interests in the same
| capacity that other groups are able.
|
| In the case of political speech that you consider Nazi-
| style, your rationale is that you believe it will make an
| event you consider to be unlikely become even less
| likely. You believe it will have a sufficiently large
| influence on the likelihood to make it worth bearing the
| consequences of telling people in your society that they
| cannot participate in the public political debate as
| freely and meaningfully as others. Why do you believe it
| would make that event less likely? And why do you believe
| it would reduce it enough to justify the risk?
| [deleted]
| HBKXNCUO wrote:
| Thank you for the candid reply.
|
| It sounds to me like you have no desire to share a
| society with people that hold the views you describe, and
| would rather kill them or expel them over their ideas
| than share your society with them and grant them the same
| ability and freedom as others to participate in the
| public political debate. Would you agree with that?
|
| >when those certain other people hold views fundamentally
| incompatible with what we as a society have agreed
| (whether tacitly or otherwise) are the foundational
| values of our society
|
| Which values are you referring to, out of curiosity?
| thenewnewguy wrote:
| 1. Cloudflare has only banned large websites, not any "no-
| name sites".
|
| 2. Admittedly anecdotal, but while bans like these do
| increase _knowledge of_ said websites, I see no evidence
| they significantly increase their popularity or userbase.
| dana321 wrote:
| Cloudflare. A great solution if you want nobody to be able to
| easily access your website.
| wackget wrote:
| What are some alternatives which offer DDOS/flood/spam
| protection?
| superkuh wrote:
| It's very easy to manage incoming bandwidth when you're
| hosting a tor onion service. The entire Tor ecosystem kind of
| helps to since there's a limit on the instantaneous amount of
| data in any circuit. Overall tor is great because I own my
| domain name (rather than leasing it on the whim of some corp)
| and it has nice DoS and bandwidth tools built in.
|
| And if you were using cloudflare before you should be okay
| with some people not being able to access your site since
| that's the norm there.
| ryanlol wrote:
| > It's very easy to manage incoming bandwidth when you're
| hosting a tor onion service.
|
| Yeaah, I don't think anyone who's operated a larger onion
| service would agree with you.
| iudqnolq wrote:
| OP here. My website wasn't up when this happened because of
| some yak shaving, but when it is I disable DDOS protection. I
| was only using Cloudflare for domain registration and DNS.
|
| I don't think I have ethical issues with DDOS protection in
| general, but as someone who browses using Firefox on Linux with
| tracking blocking I know how annoying it can get. If I don't
| need it why bother? Plus I generally like to minimize opaque
| layers in my "stack".
| tus88 wrote:
| The ultimate website blocker.
| dana321 wrote:
| That and recaptcha!
| Legogris wrote:
| I had this exact thing happen to me as well, but wrote it off to
| having been compromised (fortunately I was only using Cloudflare
| as secondary DNS servers on a non-production account and am not
| using them as a registrar, so I only noticed months after the
| fact). I think a major reason going with someone like Cloudflare
| for DNS in the first place is reliability and availability and
| this does not speak to that.
|
| Zero communication in my case as well.
| n0bel wrote:
| We've just been dealing with this for my company as well.
| Cloudflare has repeatedly deleted our DNS and cannot provide a
| reason why it happened. Last time thousands of dollars of PPC Ads
| were running uselessly.
| sgnls wrote:
| Last week, I have had an issue where a number of domains were
| purged from the 2nd tier registrar (Claranet) with exactly the
| same symptoms (domains suspended, zone-files blown away)... and
| Network Solutions are to blame.
|
| An assumption of false-payment led to them suspending "300-500"
| accounts (mostly UK based). I am still of the opinion something
| far more sinister is at play... and this doesn't comfort me.
| pvtmert wrote:
| i am using api to download/backup zone every week (by cron) to
| gdrive (fuse drive / cheap solution)
|
| i do this for all the domains i use/manage
|
| this post has been a good reminder to check them :)
|
| imho about audit log: since they "delete" everything, nothing is
| left in the zone/domain.
|
| thus, initial log (127.0.0.1/creation) comes up. kind of feature
| of the bug/logic error.
| RcouF1uZ4gsC wrote:
| Be wary of being part of something that is a cost center for the
| company instead of a profit center.
|
| CloudFlare is selling domains at cost. That means they are not
| making any money from being a domain registrar, which means they
| will do everything to keep the cost of doing it as low possible
| to themselves. This means lack of customer service and use of ML
| dragnets for "anomalous" behavior.
| owenmarshall wrote:
| .com has a price floor of $7.85. Most registrars seem to target
| anywhere from the $9.99 - $14.99 range for registration
| because, as far as I can tell, there is no real differentiation
| outside of price.
|
| Sure, I could spend $lots to get a dedicated account rep from
| MarkMonitor or CSC but that's not really feasible for my
| personal site.
|
| Are there really any registrars that hit a reasonable price
| point for individuals and offer service beyond bargain
| basement? Because if so I'm doing some transfers this weekend.
| ocdtrekkie wrote:
| One of the reasons I've stuck with GoDaddy is 24/7, American
| phone support. Their .com pricing is closer to $20 for
| renewal at this point, but I've called them at 2 AM before
| and gotten help.
|
| From previous research, at least, most domain registrars have
| ticket support at best. I did move all my "less important"
| domains to Cloudflare for cost savings recently, but they
| have my most important domains.
| sbarre wrote:
| What do you consider reasonable price point?
|
| I'll speak for myself and say that all my domains have been
| with Hover for well over a decade now, and the times I've had
| to deal with their customer service, they've been excellent.
|
| In fact, I even had to call them once, and I got a human
| almost immediately, and that human was able to resolve my
| issue while I was on the phone.. I don't recall the exact
| issue and I'm sure it wasn't anything major, but it was still
| nice.
|
| So yeah, Hover. They're nice. And I think their prices are
| decent?
| randomdude402 wrote:
| Namesilo has been a great, cheap registrar for me for many
| years and has always had privacy included for free.
|
| I tried several of the lower price registrar's back in the
| day, and they all sucked in their own way, despite me not
| needed anything except the thing to just stay registered.
|
| One or two would change the price of their domain privacy,
| most renew the privacy for like 3 dollars and then send you
| the renewal email that your domain needs to be renewed, one
| of them used to charge me separately like 80 cents from some
| weird Canadian shell company...
|
| I actually have a domain still with probably the biggest
| "cheap" provider, and they now have a thing where you are
| supposed to keep a deposit in your account to cover automatic
| renewals. Just charge my damn credit card guys, please.
|
| So I'm saying namesilo all the way. Only one that hasn't ever
| pulled any shenanigans on me.
| judge2020 wrote:
| Can't think of a reason this domain was touched (I don't work for
| CF) but I'd recommend reading the threads related to this search:
|
| https://community.cloudflare.com/search?q=127.0.0.1%20audit
|
| Every related incident seems to be due to either nameservers
| temporarily/incidentally chanced away from CF (and CF's service
| not re-checking it perhaps) or the registration billing failing
| (which doesn't look to be the case since registration expires
| 2021[0]). The latest change to the domain was about a week
| ago[0], so if that was when it was transferred to CF, it might be
| the first scenario.
|
| > Because Cloudflare deleted my domain registration I can't
| change the status from clientTransferProhibited through their
| dashboard so I don't think I can even leave.
|
| Unless something else happened, deleting the zone from your
| account doesn't affect the registration. Re-adding the domain
| will instantly allow you to view the registration info and likely
| transfer away; this would only not work if the zone is banned for
| some reason.
|
| 0: https://who.is/whois/danielzfranklin.org
| crooked-v wrote:
| > Re-adding the domain will instantly allow you to view the
| registration info
|
| "Your domain registration configuration depends on your DNS
| zone configuration" is a very strange way to do things.
| iudqnolq wrote:
| OP here.
|
| > Every related incident seems to be due to either nameservers
| temporarily/incidentally chanced away from CF (and CF's service
| not re-checking it perhaps) or the registration billing failing
| (which doesn't look to be the case since registration expires
| 2021[0]).
|
| The changes a week ago involves adding and deleting TXT and A
| records only. Cloudflare manages the nameservers I use as my
| registrar and I never changed them from the default. I just
| confirmed all of that in the Cloudflare audit log.
|
| > Unless something else happened, deleting the zone from your
| account doesn't affect the registration. Re-adding the domain
| will instantly allow you to view the registration info and
| likely transfer away; this would only not work if the zone is
| banned for some reason.
|
| Thank you so much! Trying that now.
| mercora wrote:
| i think the parent poster meant changes done by or for a
| registrar like shown in whois. Zone changes wont show up
| there.
| iudqnolq wrote:
| Cloudflare is my register and manages my DNS. I would
| expect anything them to log any significant changes to
| either in their audit log.
| mercora wrote:
| i think that it would be rather unusual to update
| timestamps in whois (which i guessed the parent poster
| was referring to) based on updates to in-zone data. A
| transfer would be an example of something that would
| change information in whois and thus update the timestamp
| noted there for the latest update. it is sometime
| possible to infer the date of the latest change of in-
| zone data because the serial of the zone is often
| constructed by using a date and a counter. But that is
| actually just convention and not reliable. Its also
| unlikely the parent poster was referring to this.
| iudqnolq wrote:
| Huh. I definitely didn't make any such changes within the
| last two weeks. Maybe the whois date got changed because
| of something opaque and internal to Cloudflare?
| mercora wrote:
| yes that is most likely what happened. A change of the
| nameservers with authority for your zone for example or
| updates of DNSEC keys would trigger that too, i think.
| But most commonly it probably happens when the domain
| gets a renewed registration period or the contact details
| for some person changed.
| Paul-ish wrote:
| > However, I'm unable to log in to their community forum. When I
| click the login button I'm redirected to my dashboard, and when I
| then click Support on the dashboard I'm redirected back to the
| forum without being logged in. I suppose it's possibly an issue
| with Firefox blocking cookies (although I disabled tracking
| prevention) so it's possible this part is partly a problem on my
| end.
|
| I'm into issues like this more and more, where you run into some
| strange behavior on a website and you wonder "How did this ever
| make it into production?", then you open the website in Chrome
| and the flows work fine. I worry that Firefox is becoming less
| and less viable.
| _def wrote:
| If a service doesn't function properly without tracking I
| wouldn't blame it on a privacy respecting browser.
| mark_and_sweep wrote:
| This is not Firefox becoming less and less viable. This is
| developers caring less and less about supporting older
| browsers, less capable hardware and, I guess, long-term
| maintenance in general.
|
| Just had a similar case today: My Mom tried to order something
| online on her old Android tablet - and it didn't work. She
| blamed the tablet for it, saying "It's just too old, it doesn't
| work correctly anymore! I used to be able to order stuff on
| this website". I had to explain to her that her tablet is still
| working fine, it's just the website that is broken because it's
| not supporting her device (or browser) anymore. Shockingly, she
| listed quite a few websites, which she has used for years,
| which have stopped working for her in the past few months and
| years; all of these she mentioned as evidence that the problem
| must be her tablet - not the websites. When I opened two of the
| sites she mentioned, I wasn't too surprised to find very shiny,
| very modern single-page applications (with service workers
| registered and even WebAssembly used on one of them)..
|
| So when you are creating a modern web app, please don't just
| test in Chrome on your new MacBook Pro. Think about your Mom.
| Ask yourself: "Is this still gonna work on her crappy old
| device?"
| SahAssar wrote:
| Well, it's also a problem of device manufacturers dropping
| support for devices too quickly. There are still android 4.1
| devices sold on amazon, and you really can't expect web
| developers to support that.
|
| The manufacturer should be required to support it for the
| full lifetime of the device. Especially since your mom uses
| it to order stuff, which usually includes some pretty
| security sensitive information. I think you are putting the
| burden on the wrong party.
| mark_and_sweep wrote:
| Well, I haven't analysed the exact technical reason for why
| submitting the order failed. But I'm pretty certain that
| submitting a HTML form is a solved problem in web
| development.. Or at least it should be. I haven't tried
| submitting a form with an async fetch from a web worker
| that communicates with a redux store implemented in
| WebAssembly yet (or whatever that web app is doing..).
| SahAssar wrote:
| If the order site is just submitting an HTML form in the
| old way with credentials stored in a cookie (also the old
| way) that would probably be open to trivial CSRF attacks.
|
| If it is somehow checking for support for SameSite,
| Secure, CSP or any of the other mechanisms that have been
| implemented in the last years then it might fail. Or they
| might be using mechanisms that work around the problem
| that those three are supposed to help since they are not
| available in older clients, but just don't have the
| resources to test the random android 4.12 version that
| you use. I think it should have a proper error message if
| that is the case.
|
| But I feel like you are pointing the finger in the wrong
| direction. I try to build my apps without extraneous
| fads, but keeping a webapp secure (in other words keeping
| up to date with the latest protections) does not mean
| "submitting a form", and it does not mean letting any old
| client lacking the required protections through.
|
| It also does not mean doing "WASM compiled redux reducers
| in ES6 module workers authenticating over JWT to send
| gRPC commands to a kafka broker talking with
| ingressrouting over anycast and a internal service mesh
| with m2m-TLS auth with TLS3.9 curve9999.9 using token
| binding and Wireguard to secure internal communications
| over a VPC-less multi-cloud k8s cluster that uses Multi-
| Raft, Single-Paxos to have a single, distributed,
| disputably non-consistent CRDT-consensus algo over
| blockchain RS-323".
|
| So, yeah, I'm not for fads over usability in tech. But
| I'm also not for supporting insecure clients just because
| the manufacturer of those clients doesn't give a shit.
| cnst wrote:
| > Well, it's also a problem of device manufacturers
| dropping support for devices too quickly. There are still
| android 4.1 devices sold on amazon, and you really can't
| expect web developers to support that.
|
| Are you kidding me? If you're looking for shiny stuff to
| add to your resume, yeah, you can't possibly support those!
| If you're an HTML5 game developer, yeah, gotta use the
| latest and greatest. But if you're in the business of
| selling shoes, why do you need anything newer than Android
| 4.1 in order to process the transactions?!
| partiallypro wrote:
| I've Cloudflare delete an entire zone before, and I could never
| get an answer as to what happened. They said it was deleted
| because the NS were changed on the domain...but they never were.
| dariusj18 wrote:
| Cloudflare once deleted one of my domains because the NS records
| were set in the wrong order.
| jlgaddis wrote:
| Wrong order? Since when do NS RRs have to be in any certain
| order?
| LinuxBender wrote:
| What do you mean by wrong order? Do you mean the NS records in
| the zone file were after a delegation / referral? What RFC was
| your zone breaking?
| matthewmorgan wrote:
| Cloudflare can suck a fuck
| Jerry2 wrote:
| This is Google-tier lack of support and general 'customer'
| gaslighting.
| whatthesmack wrote:
| This is frightening. I just started the process of moving all ~60
| of my domains from Amazon Registrar + Google Cloud DNS to
| Cloudflare, and will definitely wait until somebody from
| Cloudflare chimes in here to clarify what's going on.
| Jerry2 wrote:
| > _moving all ~60 of my domains from Amazon Registrar + Google
| Cloud DNS to Cloudflare_
|
| You're very brave considering that Cloudflare doesn't even have
| U2F yet Google and Amazon do.
| whatthesmack wrote:
| Great point! And let's just say that the migration project is
| now on-hold :)
| ocdtrekkie wrote:
| Are you using physical U2F keys for your Google or Amazon
| accounts?
|
| Cloudflare does support standard TOTP-based 2FA like most
| people use for Amazon and Google. So whether or not the
| lack of U2F support should matter depends on whether you
| actually use it elsewhere anyways.
| nhoven wrote:
| U2F is under active development. My team is actually working
| on it as we speak
| somehnguy wrote:
| They have TOTP 2 factor however.
| flurdy wrote:
| Don't put all your eggs in one basket, ie. don't just use one
| provider.
|
| Also for your core domains, do not let the registrar and dns
| provider be the same entity.
|
| Also, don't decide on not migrating just because of one bad
| experience. None of them are perfect, though vigilance is wise.
|
| (I know am probably preaching to the choir :) )
| iudqnolq wrote:
| OP here. I'm considering moving to Amazon Registrar. Why are
| you leaving?
| whatthesmack wrote:
| I was only moving to Cloudflare because they do registrar
| services at-cost, which would be cheaper than Amazon
| Registrar.
| ocdtrekkie wrote:
| How many of your sixty domains are business-critical?
|
| Cloudflare's Domains service is new, and some of it's
| management tools are lacking, but I also moved _most_ of my
| domains to it over the last year for cost savings. I 'm
| thrilled with it, but I'm still keeping a few of my most
| critical domains with GoDaddy. (Hate them all you want, but
| GoDaddy hasn't screwed up my domains in well over a
| decade.)
|
| You may be able to save a lot of money without risking your
| primary domain that you route email through.
| freedomben wrote:
| I've been planning too soon, am also now going to wait to see
| where this goes. DNS is obviously a critical system and I don't
| know if I can trust Cloudflare now. I'm not a big fish that can
| make noise. I'm an easy victim.
| ocdtrekkie wrote:
| There's a number of Cloudflare folks who are HN regulars, so
| hopefully you'll get some answers. Hopefully it's something they
| can reverse.
|
| But as a general reminder to everyone (I think this is an
| unfortunately common problem from a number of companies): If this
| is how your company handles account issues, you're probably
| wrong. Whether it's automated or manual, a user should be able to
| access all of their own information even when you decide to no
| longer provide them service. And you should test and retest the
| ability for people who you now deny service to transfer out.
| dvno42 wrote:
| Funny that this is coming up. I just transferred over from
| Namecheap to Cloudflare a few days ago and had a similar issue.
| One of my A records (out of about 20) were missing after the
| transfer.
| iudqnolq wrote:
| I noticed that if you don't unfocus the input field by focusing
| somewhere else on the page it may not save. That may be what
| happened to you.
| oefrha wrote:
| Unrelated issue but sometimes Cloudflare docs/communications are
| not in sync with their actual system which is immensely
| frustrating. I was bitten a few times.
|
| For instance, a while back I forgot to renew one of my side
| project domains so it briefly expired for maybe a day or two. Got
| this email from Cloudflare saying
|
| > Your DNS records will be completely removed from our system in
| 7 days.
|
| > ...
|
| > Once you have completed this change, click the "Recheck
| Nameservers" button in your Cloudflare dashboard to ensure your
| domain stays active on Cloudflare.
|
| I promptly renewed, except there's no "Recheck Nameservers"
| button anywhere, and the dashboard still read "Moved" for maybe a
| day. Eventually the problem was just gone, but the communication
| worried me that entire time.
|
| (I do appreciate Cloudflare's service, though.)
| outworlder wrote:
| > Your DNS records will be completely removed from our system
| in 7 days.
|
| This sounds like a plot of a japanese horror movie.
| fernandotakai wrote:
| as much as i like cloudflare (and i like them a lot), it's kind
| of absurd that this kind of thing can happen. a lot of red flags
| that, if true, would mean that their infrastructure require a lot
| more care (127.0.0.1 as the source of an audit event? no email
| when DNS records are deleted? no 1-to-1 message due to this
| happening?).
| ocdtrekkie wrote:
| At the very least, this sort of lack of good process is
| definitely what happens when Google decides to cut you off (and
| another person just commented a similar experience with
| Amazon), but I suspect it's likely the case for a much larger
| number of companies and services than people realize. It's
| fundamental internet architecture, and often little more
| thought goes into account termination than what you'd do to ban
| someone from your mid-2000s phpBB forum.
|
| So much business focus goes into the onboarding experience, and
| since you assume all of the people your service terminates are
| "probably bad people anyways", not a lot of thought goes into
| offboarding, or ideally, appeals.
| use-net wrote:
| just wait until MS revokes certain certs and all Win machines
| with TPM stop booting LOL!
| thedanbob wrote:
| I had an issue with them recently where a SRV record pointing
| to "." (meaning "service unavailable") was being rewritten to
| the string "false". It didn't take them too long to fix it, but
| it made me wonder how they managed to push a bug like that to
| production without some sort of automated test catching it.
| ocdtrekkie wrote:
| IIRC, if you're on a free plan you get exposed to code
| changes a little faster than their paying customers.
| Operyl wrote:
| Correct, if I recall correctly they outlined this in their
| SEC filings.
| thedanbob wrote:
| Which is fair, I'd rather be a guinea pig than look at ads
| in exchange for a free service. I was just surprised that
| the thing they broke was as well defined and testable as
| DNS validation.
| johnklos wrote:
| Simple. They don't give a damn about doing what we've all
| been doing properly for a quarter of a century. Apparently
| these large companies are above owning O'Reilly books.
| daenz wrote:
| This happened to me with AWS somewhat recently[0], and I never
| found out exactly what happened. I just chalk it up to some dev
| made a mistake and didn't tell anyone. It's pretty alarming when
| things like this happen though.
|
| 0. https://news.ycombinator.com/item?id=21326014
| jcrites wrote:
| I've been involved in using Route 53 to manage thousands of DNS
| zones, and haven't come across something like that. I'd
| recommend putting in a support request via the account that was
| affected to ensure that it gets looked at.
|
| If you haven't already, you might consider checking the
| CloudTrail logs for the account in question to see if there
| were any API commands related to the zone.
| PetahNZ wrote:
| Although not DNS related, I have had weird things happen on
| AWS, such as spikes of 5xx errors reported from CloudFront
| which was backed by ELB/EB, but the ELB is showing no errors.
| Even after contacting AWS support they couldn't resolve it,
| said they required application logs, but there is no logs
| because the requests never reached the application servers.
| use-net wrote:
| cloudns.net does it a bit more customer-friendly way:
|
| they e-mailed me saying they deleted some domains not because
| some entries were broken or had problematic entries, but just
| because it was "underused", i.e. too few DNS resolve calls. So
| the tiny data packets in their nameserver caused them
| unnecessary consumption of electricity or whatever. Very
| compelling! This is how they do business these days.
|
| They bombarded people with all sorts of useless info, but not
| about this policy of theirs. Makes you feel very much like the
| proverbial "valued customer".
|
| Everything is going downhill in this century, that's a fact.
| britmob wrote:
| That is... quite scary. Why would you EVER have a way for auto
| deletion of domains?
| gist wrote:
| > Does anyone know what might have caused Cloudflare to delete my
| domain? Any ideas for how I could transfer my domain away from
| Cloudflare sooner?
|
| I don't get the point of 'shoot first ask questions later' type
| approach. Obviously it would pay to get some kind of affirmative
| reply from Cloudflare prior to a post which everyone here with
| incomplete information speculates and wastes time on (like I am
| doing).
|
| Also Cloudflare did not 'delete my (the) domain. It deleted the
| dns records. There is a difference and no I am not being pedantic
| either. How would 'the internet' know why this was done there
| could be any number of good or bad reasons.
|
| Lastly the domain is not expired and as such the registrar is
| required (per ICANN) to supply an auth code so someone can
| transfer out. Or to allow the customer to change the primary and
| secondary dns to another dns provider. There is zero
| (legitimately) that allows cloudflare as either a dns provider or
| a registrar to lock the domain up pretty much (other than for a
| legal court order) just for some reason they might decide to do
| that.
| johnklos wrote:
| > I don't get the point of 'shoot first ask questions later'
| type approach.
|
| At first I thought you were talking about Cloudflare shooting
| first, but apparently not.
| iudqnolq wrote:
| OP here.
|
| > Also Cloudflare did not 'delete my (the) domain. It deleted
| the dns records. There is a difference and no I am not being
| pedantic either.
|
| Thanks. You're absolutely right. I meant delete their record of
| the domain as it shows up in the UI of their dashboard.
|
| > How would 'the internet' know why this was done there could
| be any number of good or bad reasons.
|
| For many reasons luckily HN isn't 'the internet'. I've already
| gotten some good suggestions.
|
| > Lastly the domain is not expired and as such the registrar is
| required (per ICANN) to supply an auth code so someone can
| transfer out. Or to allow the customer to change the primary
| and secondary dns to another dns provider. There is zero
| (legitimately) that allows cloudflare as either a dns provider
| or a registrar to lock the domain up pretty much (other than
| for a legal court order) just for some reason they might decide
| to do that.
|
| I know. Again, I guess I was insufficiently specific.
| Cloudflare has warned me to expect long wait times before I can
| talk to a customer support rep. My question was if there's a
| way to transfer out without needing to wait on a slow support
| loop.
| isclever wrote:
| My takeaway:
|
| 1. Setup up monitoring on your critical domains. UptimeRobot and
| Hetrixtools are good starters with generous free tier. You should
| know when your website/email/dns isn't working.
|
| 2. Don't tie your domain registration with your DNS provider. You
| lose everything if something goes wrong with your account.
|
| 3. Be able to jump ship easily, have backups of your zone,
| already know where you will transfer to.
| djsumdog wrote:
| > UptimeRobot and Hetrixtools are good starters with generous
| free tier
|
| Are there any open source status pages/monitor programs that
| have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary
| port checks, etc? I'd rather just setup a status page/alert app
| on a $5 minimal DO/Vultr node and self-host/support/contribute
| to a FOSS program than use a commercial provider.
| falcolas wrote:
| <opinion class="unpopular">
|
| Nagios. Or its descendant with a better configuration
| language, Icinga2. They're fairly easy to do a minimal
| install and configure in a container or on a VM.
|
| </opinion>
| [deleted]
| stevekemp wrote:
| I wrote a scalable system for this:
|
| https://github.com/skx/overseer/
|
| Handles SSL-checks, DNS-checks, SMTP-checks, & etc. Runs a
| thousand-checks every two minutes for me, give or take.
| Pluggable output via a redis-queue.
| vorpalhex wrote:
| You need to host across several nodes in different geographic
| locations and data centers to resist network splits. Then you
| need some way to slowly roll out upgrades to your monitoring
| platform over time.
| djsumdog wrote:
| I'm just talking about my personal infrastructure. If I
| host my crap in Vultr or Linode, I should be able to buy
| one cheap node on another provider just to run a simple
| status app: something with celary or sidekiq jobs to check
| my other stuff and intervals and generate a page with some
| red/yellow/green dots.
| vorpalhex wrote:
| How do you know if the monitoring node goes down at the
| same time as the other servers?
|
| Remember that Linode/Vultr/etc don't run their own
| datacenters, they share datacenters and sometimes
| downtime events can exist outside of datacenters.
| isclever wrote:
| Here is a good list: https://github.com/n1trux/awesome-
| sysadmin#monitoring
|
| Maybe one fits what you are looking for.
| iudqnolq wrote:
| If you want email or text message alerts I would assume
| that's a complicated enough system you would want uptime
| alerts on it, and so on recursively ad infinitum.
| unilynx wrote:
| If you can set up nagios (which one would probably consider
| an interesting evening challenge if you were already
| willing to go for your own monitoring droplet) setting up
| pushover or amazon sns (for sms) should be easy enough.
| falcolas wrote:
| FWIW, a lot of cellular providers have an email gateway
| for delivering SMS messages. There's also paid SMS
| gateways, and options for providing arbitrary push
| notifications to smartphones.
| iudqnolq wrote:
| I'm pretty sure the free email gateways have no posted
| SLA. Plus, that requires a reliable email server, which
| would also need its own monitoring.
| falcolas wrote:
| This was a few (3) years past, but they accepted
| root@localhost sendmail messages just fine in most cases,
| and delivered alerts within a minute or two of sending.
| We didn't rely on this long term, but it was a "good
| enough" first pass.
|
| I'd probably recommend using one of the gateways (or a
| more fully-featured service like Pagerduty) for more
| serious businesses, but for personal use (or where an
| outage detected the next day isn't crippling), it's
| remarkably useful.
| iudqnolq wrote:
| I would try to set up a completely open source monitoring
| setup just for fun, but once I'm paying for SNS I
| personally would rather just pay epsilon more and
| buy/rent the whole system. I get that may just be
| personal taste. I absolutely don't trust myself to run my
| own highly-reliable mailserver to send status alerts.
| petre wrote:
| Just send them locally and pull them with IMAP onto your
| phone.
| iudqnolq wrote:
| I didn't even know that was possible. Thanks for teaching
| me something new. It's always nice to learn I need less
| SAAS magic than I thought.
| hedsht wrote:
| check out https://github.com/hunterlong/statping - thats what
| i'm using.
| iudqnolq wrote:
| > Setup up monitoring on your critical domains. UptimeRobot and
| Hetrixtools are good starters with generous free tier. You
| should know when your website/email/dns isn't working.
|
| Lesson learned :)
|
| > Don't tie your domain registration with your DNS provider.
| You lose everything if something goes wrong with your account.
|
| I don't see how that helps. How do I recover from my registrar
| deleting/disabling my account even if DNS is somewhere else? I
| think there's still only one failure point and the lesson is
| that I need to pay that failure point more money.
|
| > Be able to jump ship easily, have backups of your zone,
|
| Luckily I have that
|
| > already know where you will transfer to.
|
| Any suggestions? Ironically I recently moved from Google
| Domains to Cloudflare because I was worried about issues with
| opaque support. I've learned my lesson picking based on cost
| alone, but I'm a college student who can't afford something too
| heavy-duty.
| woofcat wrote:
| >I don't see how that helps. How do I recover from my
| registrar deleting/disabling my account even if DNS is
| somewhere else? I think there's still only one failure point
| and the lesson is that I need to pay that failure point more
| money.
|
| Your outage was a DNS outage, not a registrar outage. If you
| still had control of the domain you could update your name
| servers to another provider, import your backed up records
| and get the site back online without talking to CloudFlare.
| iudqnolq wrote:
| > Your outage was a DNS outage, not a registrar outage. If
| you still had control of the domain you could update your
| name servers to another provider, import your backed up
| records and get the site back online without talking to
| CloudFlare.
|
| I believe it was both.
|
| If I have a registrar outage I'm hosed. If I don't have a
| registrar outage and do have a DNS outage I can recover
| with a little work. But in the only case I can recover my
| registrar was reliable, so why didn't I just have them do
| DNS as well?
| isclever wrote:
| A domain registered at a provider (but not DNS) can be
| down with no impact to your domain, so long as the domain
| is still in the TLD root servers, everything will keep
| going.
| iudqnolq wrote:
| Thank you for teaching me something new. I didn't know
| that got cached.
| petre wrote:
| > But in the only case I can recover my registrar was
| reliable, so why didn't I just have them do DNS as well?
|
| Because they have just proved being uncapable of doing
| it? Because redundancy? Because you shouldn't keep all
| your eggs in the same basket.
|
| I've been self hosting for at least 15 years and did not
| have any huge problems like the domain becoming non
| resolvable. I would _never_ host my DNS on my registrar
| 's infrastructure. It's being sloppy and lazy and it gets
| you embarassed.
| throwawaydns101 wrote:
| DNS has become frighteningly unreliable. Here are previous
| stories that show how it is possible to lose access to your
| domain for no fault of yours:
|
| (1) https://news.ycombinator.com/item?id=21700139 - Sinkholed
|
| (2) https://news.ycombinator.com/item?id=19322966 - I lost my
| domain and everything that goes with it
|
| No different than this story where the author's DNS records were
| deleted because of so called "anomaly".
|
| Here are so many more stories:
| https://news.ycombinator.com/item?id=21710939
|
| DNS was a good idea but now there are organizations that have the
| power to arbitrarily take control and even remove your domain
| names and records. We really need to come up with a peer-to-peer
| solution and take back control of the naming system from these
| authorities.
| Defenestresque wrote:
| >DNS has become frighteningly unreliable. Here are previous
| stories that show how it is possible to lose access to your
| domain for no fault of yours:
|
| The second story you posted is about a user who forgot to renew
| their domain and did not wish to pay the overly-inflated fee to
| re-register it while it was in the grace period.
|
| I hold no love for any registrar that jacks up rates for
| getting back an expired domain and agree that they should have
| sent a reminder email, but describing this as someone "losing
| their domain through no fault of their own" is, frankly,
| incredibly misleading.
|
| The user:
|
| 1) forgot to renew their domain 2) had full right to recover
| their domain but objected to the price 3) had full right to
| transfer the domain out to another registrar for the original
| 15EUR price and 4) eventually got back full control of the
| domain
| nathancahill wrote:
| Odd comment to make a throwaway for, not very controversial
| (unless you work for Cloudflare?)
| throwawaydns101 wrote:
| I don't work for Cloudflare but I work for another large
| company that also manages domain names and DNS records. I
| don't want to risk the possibility that my comment could be
| interpreted by my employer as conflict the interest.
| Operyl wrote:
| Probably should have made a mention to that in the first
| comment.
| Legogris wrote:
| I looked into self-hosting DNS and it doesn't seem like that
| big of a deal as long as you can ensure uptime to be honest. If
| you set up the two first on different hosts and possibly have
| #3/4 being cloud providers I think you're pretty good.
|
| Does anyone here have experience with running their own DNS
| servers for their domains?
| petre wrote:
| I've been self hosting for years. Currently using online.net
| secondary DNS service as my 3rd or 4th backup NS. They've
| lost my 10EUR/month box once (shitty cheap intel avoton
| hardware with everything soldered on I suspect) but the
| domain still resolved fine. I had backups and restored it in
| a day. You can also use a VPS image to self host DNS. Some
| providers offer automatic or manual snapshots. Hetzner comes
| to mind. They've annoyingly asked for a copy of my id card
| (welcome to Germany), but their services are fine.
| cnst wrote:
| You don't even need multiple servers (especially if both your
| website and mail run on the same server), it's a
| misconception debunked by the author of djbdns:
|
| http://cr.yp.to/djbdns/third-party.html
| icedchai wrote:
| I've been self hosting DNS for 20+ years. It's easy as pie. I
| have a couple name servers on my home network (business
| cable) and another on a VPS.
| throwawaydns101 wrote:
| That would solve the problem of losing DNS records. What do
| you do when you lose access to the domain name in the first
| place?
| teddyh wrote:
| The main problem which people seem to have is that their
| domain name registrar decides to pull their domain.
| Luckily, there is ample competition in this space, my place
| of employment included, which should make it reasonable to
| pick a place which 1. doesn't do that and 2. has reasonable
| real-live-person support.
|
| Of course, if the _registry_ (i.e. the TLD) wants your
| domain gone, you are out of luck whatever you do. If this
| is a concern then you should pick a TLD with what you
| consider reasonable management. There are a lot of ccTLDs
| and gTLDs to choose from.
|
| Therefore, what you absolutely _shouldn't_ do is to pick
| whatever domain registrar is either cheapest or largest,
| and pick whatever domain name which happens to look cool
| and be available. Both are recipies for potential disaster.
| Legogris wrote:
| Indeed. I am curious to see what comes out of attempts at
| decentralizing this such as Handshake[0] and ENS[1]. I
| think I saw something similar with prominent backers come
| up here on HN the other week but can't recall it now.
| Namecoin[2] was very early on this.
|
| [0]: https://www.namebase.io/
|
| [1]: https://ens.domains/
|
| [2]: https://bit.namecoin.org/
| Karupan wrote:
| Stories like these scare the hell out of me. What do you do if
| one of the big internet corporation deletes some resource or
| account that is critical to your business? What happens when
| support isn't responsive and you don't have contacts in the
| company or your HN post doesn't get visibility?
|
| I get it - these are free services. You should factor that into
| every decision. But the risk is real even if you pay for an
| account. I've been slowly moving away from Gmail to a custom
| domain, but something like loosing DNS records and not being able
| to restore them quickly is even worse.
|
| Back up everything that can be backed up, don't rely on a single
| provider and always have a continuity plan!
___________________________________________________________________
(page generated 2020-02-24 23:00 UTC)