[HN Gopher] 250M Microsoft customer service and support records ...
___________________________________________________________________
250M Microsoft customer service and support records exposed on the
web
Author : el_duderino
Score : 265 points
Date : 2020-01-22 17:47 UTC (5 hours ago)
(HTM) web link (www.comparitech.com)
(TXT) w3m dump (www.comparitech.com)
| jsgo wrote:
| I got an email from Microsoft Azure in relation to this (didn't
| read the article, but people are quoting parts of the email I
| received here).
|
| I appreciate that they sent something, but sometimes it'd be nice
| for them to allow someone to access the data related to them that
| was exposed as they say "our analysis of the support information
| indicates that specific personal or organizational identifiable
| information related to your support case was potentially
| visible." Okay, what specific personal or organizational
| identifiable information of mine was visible?
|
| I assume the representative or I may've listed said info in our
| communications back and forth so let me see what was exposed so I
| can make a judgement of what, if anything, I should do here.
| GordonS wrote:
| I got the same email, and I agree with what you said - I'd
| really like to know if this is even personally relevant, and if
| it is, I'd really like to know precisely what information is
| relevant. I'm in the EU, so I guess I could ask under the GDPR,
| but I wouldn't even know _who_ to ask, and with such a large
| organisation, I can only imagine there would be a lot of run-
| arpund, requiring a lot of follow-ups from me : /
| sorokod wrote:
| "In total, the data was exposed for about two days before we
| alerted Microsoft and the records were secured.
| December 28, 2019 - The databases were indexed by search engine
| BinaryEdge... "
|
| ... at least two days then.
| reaperducer wrote:
| _250M Microsoft customer service and support records exposed on
| the web_
|
| Someone should grep this to find out how many times people were
| told to turn it off and turn it on again.
| blakes wrote:
| I want to know how many answers are sfc /scannow
| ehsankia wrote:
| Is there any signs that this data is actually out in the wild?
| From the article, it was found, reporter and fixed within 24
| hours, and they claim there's no sign of other unauthorized
| access.
| reaperducer wrote:
| _Is there any signs that this data is actually out in the
| wild?_
|
| Check the dark web.
|
| _From the article, it was found, reporter and fixed within
| 24 hours_
|
| Being fixed within 24 hours of being reported does not mean
| it was only available for 24 hours. It could have been 24
| days or 24 months.
|
| _they claim there 's no sign of other unauthorized access._
|
| Anyone smart enough to access this would also be smart enough
| to cover their tracks. When I was black hat in the 80's, this
| was Infiltration 101.
| xixixao wrote:
| Covering up is not always technically possible. It's easy
| to expose data through some unprotected end point, but that
| end point might still be logged, and turning off the
| logging/deleting the logs might be a completely different
| challenge.
| thisisnico wrote:
| Even more challenging if the log destination is external,
| and if the logging system is an entirely independent
| system, even potentially provided by a third party. Makes
| this hard to do.
| ehsankia wrote:
| I know full well it could've been accessed, I never
| rejected such a possibility. I'm just saying that _so far_
| , there is no sign that the data has been dumped anywhere.
| It could exist, but right now we can't "grep through it"
| because there isn't a dump of it in the wild yet.
| trhway wrote:
| or run sentiment analysis
|
| >The records contained logs of conversations between Microsoft
| support agents and customers
| gexla wrote:
| I imagine most of these are support issues handled by contractors
| they have had over the years. Windows 95 through XP had Keane and
| Convergy's in Tucson running their Windows support (which then
| forked into Canada and India.) Not sure who they have doing it
| now.
|
| The Windows parts of these records might be a good resource as
| it's probably part of the documentation which builds up to become
| the MSKB articles. Each support case was documented and linked to
| either a KB article, an internal "not yet KB article" or you had
| to submit it as a unique issue. After the "not yet KB articles"
| were referenced X times, then it would go to consideration as a
| KB article. Collectively, all this formed their internal KB.
|
| Worked there. Pay was terrible once Convergy's took over. Then
| they moved everything to India and the support got terrible also.
| Too bad. They had quite the brain drain from that process. There
| were a lot of Windows gurus in that building. I learned far more
| than I needed to know about Windows and went way more in depth
| than I ever have tinkering with Linux.
| coliveira wrote:
| My opinion is that ALL information that has ever being put online
| will, sooner or later, be made public. Despite the advances in
| crypto, there are so many ways to exploit security flaws and
| vulnerability in all kinds of software. And now with machine
| learning, which can also be used to help in hacking exploits,
| there not much that can be done.
| twodave wrote:
| I think Microsoft's response time to this exposure (during a
| holiday even) is more noteworthy than the fact that it happened.
| We can sit in our ivory towers all day and shake our heads at
| what an inept organization Microsoft is for allowing human beings
| to make mistakes, or we can applaud the fact that once the
| mistake was identified they chose to act immediately,
| appropriately and transparently. What are we really expecting
| here? Perfection?
| throwawayjava wrote:
| _> What are we really expecting here? Perfection?_
|
| No, I don't expect perfection. However, I do expect very
| careful implementation of access management for very large
| databases containing lots of PII and other sensitive customer
| information. Things like huge databases being accessible
| without credentials shouldn't require perfection on the part of
| some human. That sort of stuff should be continuously audited
| in an automated fashion.
|
| But the software industry is quite bad, as a whole, so even the
| relatively competent actors make surprising, high-impact
| mistakes.
|
| Maybe it's because the stakes are relatively low (c.f., bridge
| collapsing vs. PII leak) and the competition relatively fierce?
| Maybe software engineering is still very young and moving
| quickly?
|
| In any case, I think it's totally reasonable to hold the
| opinion that MSFT is doing things pretty well relative to the
| rest of the industry _and also_ that the industry as a whole is
| doing a pretty poor job.
|
| IDK, for me the story has to be one of the following:
|
| 1. MSFT made a huge and inexcusable mistake, so maybe there's
| something systemically wrong with MSFT; or,
|
| 2. MSFT is very competent, and even very competent people are
| making very big mistakes, so maybe there's something
| systemically wrong with the entire industry.
| Tallasatree wrote:
| Architect here: from the outside looking in, you hit the nail
| on the head. In addition to The industry being so young the
| _relatively_ low-impact when bad things happen make things
| like this 'not a big deal'. When your mistakes result in a
| public outcry for a day, then fades into obscurity into the
| night, why change? why invest money into figuring out a
| better way?
|
| When your mistake makes a building fall over...well, there's
| a reason why that almost never happens.
| keithnz wrote:
| I don't think this is quite right. Most buildings don't get
| all their design parameters tested in reality. But say when
| there is an earthquake, and the building collapses and you
| find that various checks and balances in the design process
| went wrong. I know here in NZ where we have had a number of
| significant earthquakes all kinds of known and unknown
| things have been discovered about buildings, either ones
| that have ended up killing people or ones which now are
| condemned because things played out differently than the
| designers thought they would
| [deleted]
| huzaif wrote:
| "I am calling from Microsoft" calls were bad enough. Now they
| will know some details to a past case and sound slightly more
| legit.
| iudqnolq wrote:
| Funnily enough, I learned that if you submit a support ticket
| on a $12/month single-user Microsoft business account you get a
| call back from someone who saus they're with Microsoft Support.
|
| The rep was very helpful, but a bit puzzled that I wanted him
| to read me my ticket title. He seemed to think him knowing my
| name should be sufficient verification.
|
| Note: I can never understand Microsoft's names for different
| levels of the same product. It might not be called a business
| account, maybe professional or pro or small business or
| something.
| texasbigdata wrote:
| Yeah the no notice.
|
| By the way Microsoft has absolutely terrible azure support. If
| you have a legitimate issue and you dont have a dedicated
| support consultant good luck to you.
| Analemma_ wrote:
| All the cloud providers are like that though. If you're on
| the cheapo tiers of AWS or GCE, you get the cheapo support.
| AWS might be slightly better just because more people have
| used it and so there are more hacky workarounds posted on
| StackOverflow, but that's small comfort at best.
| keithnz wrote:
| I've had good experience with Rackspace and DigitalOcean
| support (other than having to repeat my problem multiple
| times until I get to the right person, but at least they
| are keen to help).... Azure support was a disaster with too
| many support staff that know almost nothing about the
| platform except by reading the same websites I can read
| until you spam every possible support mechanisim you can
| find and finally get to a "real" support person. This will
| take around 2-4 weeks.
| emerongi wrote:
| AWS paid support is pretty good in my experience.
| mikece wrote:
| An idea for someone looking for a fun "Show HN" project: build a
| scoreboard that searches all of the known data breaches for this
| year and tells me where I rank for how many breaches I've been in
| (eg: I'm 89/132 on breaches of 50,000 records or more).
|
| Over 8.5BB customer records were exposed last year; the estimate
| for this year is in excess of 10BB.
| Barrin92 wrote:
| https://monitor.firefox.com/
|
| this might be something you're looking for.
| emerongi wrote:
| How about a leaderboard? You get points for each breach that
| you were in and how much of your data was exposed. Each data
| point could score different points: your name is 5 points,
| social security number 10 points etc.
|
| Then the you can see that you're 880654th out of 1.1B people on
| the leaderboard and maybe feel slightly better.. or worse.
| dijit wrote:
| Isn't that what snusbase and dehashed are doing?
|
| https://snusbase.com/
|
| https://www.dehashed.com/
| geddy wrote:
| These sites really bother me sometimes. I just registered on
| Dehashed and it requires me to pay for a subscription... to
| see my own stolen data. I reject that on principle alone.
| tcd wrote:
| I find it really intriguing hearing about all these data
| breaches - never before in human history have we been able to
| store so much information about ourselves and our world and how
| readily accessible that information is, just sitting on hard
| disks around the world.
|
| Which makes me wonder, is there information that's leaked so
| much it's no longer "private"? Names, addresses, phone numbers,
| contacts lists, photos, emails, cloud documents, IP address
| logs, search history...It's all there, waiting to be leaked...
|
| And why the insistence on storing information for an unlimited
| period of time - it should be illegal to store data above 5+
| years without explicit consent from the user (after reviewing
| the data and clicking "I am okay with this data continuing to
| be stored").
| HenryKissinger wrote:
| https://haveibeenpwned.com/
| dgrin91 wrote:
| You can pretty much do this with haveibeenpwned.
| salex89 wrote:
| Pretty sure at least 100M are from me...
| Spooky23 wrote:
| What is it about elastisearch that dopey people stuff them with
| information and leave them on the open internet, all of the time?
| farisjarrah wrote:
| Elasticsearch started life as a free product and security was a
| paid addon to that product via the X-Pack, now Elastic Co has
| made the security stuff free but people still don't implement
| it. Elasticsearch is insecure out of the box and it takes extra
| steps to get it secured, and most people don't do those steps
| even though its pretty well documented right here:
|
| https://www.elastic.co/guide/en/elasticsearch/reference/curr...
| blinkingled wrote:
| Security features in Elastic still require paid subscription.
| The link you pasted even says that. You can use the xpack
| features for free on a trial basis but for production use
| you're required to buy a license.
| farisjarrah wrote:
| Thank you for the correction. I thought I heard rumblings
| about their X Pack being free, but I had probably just
| heard about the trial.
| blinkingled wrote:
| You can use the Amazon OpenDistro provided plugins on top
| of oss ES to enable security features.
| jturpin wrote:
| This is not true, you can use some xpack security features
| such as basic auth, client TLS and node-to-node TLS for
| free. We use basic auth (with Vault integration) at my
| company using just the basic license.
| https://www.elastic.co/subscriptions has details on the
| subscription levels.
| tristor wrote:
| Honestly, some of the most preventable and dumbest outages and
| failures in my career have involved ElasticSearch. Most of the
| time it's deployed and managed by a dev team with no
| operational oversight, and therefore nobody to think about or
| catch these types of issues. It is compounded by the fact that
| all the security features in ES were paywalled for a very long
| time and most technologists don't understand basic networking
| anymore.
|
| As many other answers to your query have stated, this is caused
| by a broken understanding of the devops methodology among
| organizational management forcing developers who are not
| competent in systems administration to be responsible for these
| systems.
| monksy wrote:
| We don't have sysadmins anymore "because of the cloud"
| mbreedlove wrote:
| "Sysadmins? No, we just have devops now!"
| dvfjsdhgfv wrote:
| Many managers use "devops" as an excuse to put a lot of
| burden on a small team, then this team is doing their best
| to automate managing a large number of machines but it's
| physically impossible to delve deeper into details and
| polish things, hence mishaps are bound to happen. And don't
| get me started on on what is happening inside containers.
| tylfin wrote:
| Sorry but I'd like to get you started on what is
| happening inside containers ;P
|
| Specifically can you go into more details about what
| worries you with containers. Is it insecure images with
| out of date software, or risky applications inside the
| containers? Something else?
| mschuster91 wrote:
| Let's imagine your JIRA is insecure, someone owns it and
| obtains RCE, then does a privilege escalation on the
| host, whoops suddenly all services are accessible whereas
| that would have required more steps and owning in the old
| one-vm/bare metal-server-per-service model.
| arpa wrote:
| Escaping properly namespaced/pivot_root'ed environment
| and owning a host is non-trivial too.
| [deleted]
| bcrosby95 wrote:
| Equally important, we don't have network admins. It would be
| physically impossible to expose our search database like this
| to the open internet. Extra layers of protection are great.
|
| It reminds me of companies I've worked with before that
| accidentally had a production site pointed to a dev database.
| Why the hell is that even physically possible with your
| network setup?
| mschuster91 wrote:
| Welcome to the wonderful world of Kubernetes (or, for that
| matter, any Docker orchestration solution, such as DC/OS).
|
| Anything can reach anything, provided you know the naming
| schema... and there's no easy way to fix it on anything
| that is not AWS/Azure/GCP, not without losing all the
| benefits of a self hosted k8s cluster in the first place.
|
| Openstack at least provides ways to isolate machines, but
| that's VM-level only and _truly_ an ultimate PITA to set
| up.
| arpa wrote:
| Not entirely true, there are network security policies
| (on select few CNI providers) and other means of
| segregation using good old iptables (although probably
| need to update alternatives for iptables to point to
| iptables-legacy for them to work).
| mschuster91 wrote:
| Ew. iptables (or any other way of messing around with the
| black magic that Docker and the orchestrators do to
| provide intra-container networking) is one thing only and
| that is a nice way to shoot yourself in the head while
| aiming at your legs.
| arpa wrote:
| To be entirely honest, if you know what you're doing and
| how lets say kube-proxy works in essence, things get
| pretty easy and simple. If you start every configuration
| of firewall with iptables -F, you're gonna have an
| interesting time. However, if you spend some time around
| these beasts, they are pretty well and logically built
| and it is trivial to coexist and modify your chains
| without touching those managed by docker/k8s. There is no
| black magic/and or wrong with the way they manage the
| rules. I'm much more angry at proper iptables being moved
| to iptables-legacy and systemd messing around with my
| resolv.conf :)
| mdavidn wrote:
| Amazon does provide a mechanism to define private networks
| disconnected from the internet.
|
| https://aws.amazon.com/vpc/
| Filligree wrote:
| It's the default configuration. Unless you go out of your way
| to ensure security, you'll get this result.
| thekyle wrote:
| I believe Elasticsearch does not have authentication built into
| the open source version.
| btown wrote:
| This is no longer true, but ONLY as of May 2019 in response
| to pressure from the containerization world.
|
| https://www.elastic.co/blog/security-for-elasticsearch-is-
| no...
|
| It's not default insecure like Mongo was - this was far far
| worse. You couldn't even prototype in a secure way even if
| you wanted to, without a massive contract. One of the most
| frustrating things in software - IMO they deserved to have
| AWS commoditize their stack.
| shawnz wrote:
| The module is now free, but it's not open source, it is
| licensed under the proprietary Elastic license. The source
| is available but it is not licensed to be used with
| anything except the Elastic licensed version of
| Elasticsearch (not even the Apache licensed version of
| Elasticsearch)
|
| However, Amazon has thankfully released a free and open
| source security module for Elasticsearch as part of their
| Open Distro project. It is based on another project called
| Search Guard. See: https://opendistro.github.io/
| xvector wrote:
| Perhaps it's a controversial opinion, but I feel like it's
| just flat out unethical to relegate basic security to the
| paid/enterprise version of your product.
|
| Of course it's unethical to use said product to store real
| user data too, but the road goes both ways.
| bouke wrote:
| Just deploy it on your local network. No need to expose
| it to the internet. Sure, authentication is a nice bonus,
| but a simple firewall goes a long way.
| CydeWeys wrote:
| Defense in depth is important. Lots of data breaches have
| been caused because things that should have just been
| viewable from a local network, weren't, or the network
| was compromised. Unless you think every single employee
| is invulnerable to spear-phishing (which is impossible),
| you should never be leaving anything sensitive wide open
| on your local network.
| jturpin wrote:
| I don't get why it ever needs to be on the internet even
| when it does have authentication. Surely the
| public/private subnet split is a common practice.
| kirstenbirgit wrote:
| The way I see it, password auth for db servers is the
| last-resort protection mechanism.
|
| If you get to the point where it's what's protecting your
| data, you're already fundamentally screwed.
| CydeWeys wrote:
| You should definitely have it too, though.
| CydeWeys wrote:
| Unless the private subnet is airgapped from the Internet,
| it's not a good enough separation.
|
| Hell, even if it is airgapped, it can still be
| compromised by viruses on USB sticks and such.
|
| You should never be leaving sensitive systems wide open,
| period, regardless of how secure you might _think_ that
| network is. Thousands of data breaches have been caused
| because networks didn 't end up being as secure or as
| separated as hoped for.
| fulafel wrote:
| ES is very poorly engineered but unexplainably popular. The
| interesting question is why it's popular.
| freeone3000 wrote:
| Fast free full-text search over arbitrary documents. It
| solves a problem lots of people need.
| jacquesm wrote:
| In a word: devops.
|
| Developers are not operators and operators are not developers.
| The whole idea that we can do away with this specialization and
| and relegate operations to the people that create software
| because it is now possible to script infrastructure and to
| install complex packages with a few mouseclicks does not make
| it true. Operations and the complexity that goes with it is a
| job in its own right, no competent operator would have left
| this situation as it came out of the box.
| AnIdiotOnTheNet wrote:
| A combination of businesses' desire to spend less on labor
| and your average developer's inherent sense of superiority
| mean this trend is unlikely to go away any time soon though.
| arpa wrote:
| I believe you can be both competent operator and a reasonable
| developer at the same time. The skills complement each other
| nicely. It is a lot of work to be these things though.
| Silhouette wrote:
| When people ask why we're so concerned about the privacy
| implications and specifically the telemetry functionality of
| modern software... This. This is why.
|
| Even if that functionality is implemented with good intentions
| and the data is only intended to be used for responsible
| purposes, the biggest and most technically capable organisations
| in the world can still make mistakes and suffer data leaks, which
| are potentially a gift to criminals, commercial competitors, and
| so on.
|
| If there's anything sensitive in there -- personal data,
| commercial information that was provided under NDA -- we're
| probably still on the hook for it legally, too.
| jonplackett wrote:
| > All of the data was left accessible to anyone with a web
| browser, with no password or other authentication needed.
|
| Really quite incompetent. But we don't know for sure anyone else
| actually accessed it.
| netsharc wrote:
| I like the legally correct phrasing the MS blog (https://msrc-
| blog.microsoft.com/2020/01/22/access-misconfigu...) said:
| "While the investigation found no malicious use".
|
| If the DB server was configured so access was not logged, could
| you claim "We investigated, and we didn't see any evidence of
| access"?
| ryanlol wrote:
| Surely they'd still see any exfiltration in their bandwidth
| graphs. And anyways, ES spits out a lot of logs by default.
| resfirestar wrote:
| That assumes they have bandwidth graphs. And sure, ES
| generates a lot of logs, but have you ever tried using them
| to investigate an exposure like this? Unless the
| "xpack.security" module is on (off by default), it's
| nothing useful.
| ryanlol wrote:
| Linux itself gives you decent data from procfs (see
| /sbin/ifconfig, shows you data transfer in/out per
| adapter), you can just compare data transfer from the
| server to any of the boxes that are supposed to connected
| to.
|
| I can't imagine that even MS would be running ES on
| windows, although then you'd probably have even more data
| available.
| jonplackett wrote:
| That is definitely not 'the whole truth'!
| ryanlol wrote:
| How do you know that? this data was exposed for a day or
| so, did you dump it yourself?
| Teever wrote:
| Which sucks because everyone affected now has to operate under
| the assumption that some one else did access it.
| el_duderino wrote:
| Microsoft released further details in its own blog post:
| https://msrc-blog.microsoft.com/2020/01/22/access-misconfigu...
| IanDrake wrote:
| "Misconfigurations are unfortunately a common error across the
| industry. We have solutions to help prevent this kind of
| mistake, but unfortunately, they were not enabled for this
| database."
|
| They need a solution to watch their solution that watches their
| configs.
| shaabanban wrote:
| Notably, elastic's Kubernetes operator which just went 1.0
| defaults to requiring a username and password (and generates one
| if it isn't provided). It also doesn't seem to allow you to opt
| out of using TLS.
|
| https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-ove...
| whatever1 wrote:
| Databases that involve more than X users need to be regulated.No
| big database should be deployed in public before being vetted on
| whether it is secured properly. I am tired of reading every week
| for breaches of personal data and passwords saved in plain text.
| If no company can secure our data voluntarily then we should use
| the law to force them to at least meet a bare minimum of
| standards.
| meristem wrote:
| When so many elasticsearch bad condiga get published, MS ought to
| reevaluate their UI and default config.
| harikb wrote:
| Wake me up when we expose more than 3 billion.
| ifthenelseend wrote:
| How much money did you get from Microsoft for disclosing that
| vulnerability?
| jacquesm wrote:
| Nothing like working for free for giant companies that fail
| utterly at their responsibilities.
| owlninja wrote:
| I mean if they didn't have an open bounty or posting, you
| should assume you are 'working' for free.
| withinrafael wrote:
| I reported similar issues in the past and there's no bounty,
| but of course Microsoft reserves the right to deviate. (And I
| hope they did in this case!) Minimally, you get placement on
| the Microsoft Online Services Acknowledgments page.
| https://portal.msrc.microsoft.com/en-us/security-guidance/re...
| cobookman wrote:
| Good reason why defense in depth should be used.
|
| Simply stating that your network configuration prevents access
| isn't the best answer.
| wang_li wrote:
| >Simply stating that your network configuration prevents access
| isn't the best answer.
|
| Right. The network should actually be configured to prevent
| access.
| CydeWeys wrote:
| And, in the event that this configuration fails to do what
| you expect it to you, or your network is breached via other
| means, you should be utilizing defense and depth and all of
| your DBs and other sensitive systems should require
| authentication.
| bluedino wrote:
| "misconfigured security roles" means the dinks that set it up
| never 'configured" a thing, right?
| TomVDB wrote:
| That's not how I read Microsoft's statement about it: the
| permissions were incorrectly changed on December 5th and were
| corrected on December 31st.
___________________________________________________________________
(page generated 2020-01-22 23:00 UTC)