[HN Gopher] Have lots of AWS accounts
___________________________________________________________________
Have lots of AWS accounts
Author : rcrowley
Score : 151 points
Date : 2022-10-03 16:14 UTC (6 hours ago)
(HTM) web link (src-bin.com)
(TXT) w3m dump (src-bin.com)
| coding123 wrote:
| This is one of those cases where the wrong thing gets optimized
| because of something stupid.
|
| ...Which is probably fine because that's pretty much how it
| always goes.
| traspler wrote:
| At the company I work they just built their ,,Landing Zone" as
| AWS calls it with all the on-prem connectivity, shared VPCs,
| Product Catalogs, IAM restrictions to the moon and so on and
| every team gets their own account linked to that one. It's an
| enormous amount of work to get all of that to work nicely
| together but when it works it's very nice and allows for a great
| way to partition responsibilities. For smaller deployments I
| think you will be in a mess pretty fast when you start doing
| cross-account things without enough planning ahead. But per-
| project accounts? Sure!
| rcrowley wrote:
| Substrate [1] is meant to help folks not make a mess of lots of
| AWS accounts. Would love to know if it feels less enormous.
|
| [1] <https://src-bin.com/substrate/>
| simonw wrote:
| This is fascinating.
|
| One thing that worries me: billing. If I have six different AWS
| accounts will I have to update six different places any time my
| credit card expires?
| forgomonika wrote:
| AWS Organizations (https://aws.amazon.com/organizations/) is
| supposed to solve this for you. If all your accounts are in the
| same Organization then you get one bill instead of six.
| jackson1442 wrote:
| You can also use AWS SSO (IAM Identity Center now) in
| conjunction with Organizations to federate into your AWS
| accounts using any SAML IdP.
| rcrowley wrote:
| As the other commenter notes, no, you still have just one bill.
|
| Better, though, that one bill is broken down by account so you
| can see where the money's going.
| [deleted]
| twblalock wrote:
| I've seen this tried. It required a considerable investment in
| tooling and people to run it, because the dev teams just want to
| run their apps and don't want to manage accounts.
|
| And that's just the management complexity -- cross-account
| network adds a ton of complexity to other stuff, including VPC
| management, transit gateways, and sometimes DNS.
|
| Compared to having fewer accounts, the difference in complexity
| is enormous.
| rcrowley wrote:
| Substrate [1] is meant to lessen the investment required to use
| lots of AWS accounts. Would love to know how it looks to you.
|
| [1] <https://src-bin.com/substrate/>
| tazjin wrote:
| You should have no AWS accounts, it's a giant timesink.
| jackconsidine wrote:
| I have indeed found that it's difficult to silo access to
| specific resources with AWS IAM.
|
| I use these custom policies a lot, which give write access to a
| specific S3 Bucket [0], and give sending capabilities for a
| specific SES Identity [1] respectively:
|
| [0]https://koptional.notion.site/IAM-Policy-for-
| select-S3-Acces...
|
| [1] https://koptional.notion.site/IAM-Policy-for-email-
| sending-o...
| simonw wrote:
| I ended up writing a whole custom tool for generating
| credentials and policies for specific S3 buckets:
|
| - https://s3-credentials.readthedocs.io/
|
| - https://simonwillison.net/2021/Nov/3/s3-credentials/
|
| - https://simonwillison.net/2022/Jan/18/weeknotes/
| jackconsidine wrote:
| Whoa this is awesome! Wish I knew about this. It's really
| polished
| issa wrote:
| I am fairly certain this article could either be read as totally
| serious or complete satire. It works both ways.
| [deleted]
| 015a wrote:
| I would argue that the optimal number of AWS Accounts is Zero.
| jiggawatts wrote:
| A related problem is multi-cloud orgs trying to copy this advice
| into other clouds. For example, at $dayjob, the cloud guy is
| trying to replicate the AWS account structure into Azure, where
| it's largely unnecessary. In Azure the equivalent of an account
| is a subscription, but then there's a second hierarchy level in
| the form of resource groups. Combined with tagging, this makes it
| very easy to do internal charge-backs or RBAC.
|
| They've even added a feature to redistribute the costs of shared
| resources: https://azure.microsoft.com/blog/simplify-financial-
| reportin...
|
| Splitting things across accounts or subscriptions is complex and
| results in all sorts of strange limitations. For example, some
| resources can't be connected to each other across subscription
| boundaries, so you have to duplicate them.
|
| Another example is Kubernetes (EKS or AKS). The typical thing to
| do is to create a single cluster with multiple node pools, all in
| the same account or subscription. Then this is split using
| namespaces. All of this will be in one account, and can't be
| "associated" with other accounts or subscriptions. The second a
| shared platform like this is introduced, the fine-grained account
| model breaks down.
| DerekBickerton wrote:
| I thought this was an article about having separate compartmented
| Amazon accounts. So one for shopping, another for Amazon
| Associates, another for AWS, another for Prime Video and Alexa,
| etc
|
| Does anyone here do this?
| ManuelKiessling wrote:
| Here's a very detailed step-by-step tutorial on how to realize
| resource-separation without giving up single sign-on with AWS:
|
| https://manuel.kiessling.net/2020/12/29/single-sign-on-and-r...
|
| Only one of several ways to achieve that, but one that works
| really well for me.
| mattpallissard wrote:
| > AWS accounts are the most complete form of isolation on offer.
|
| Because it's a pretty brutal namespace. While I usually wind up
| with separate accounts for multiple reasons, I strongly prefer to
| get rbac/iam properly implemented in a single account wherever
| possible.
| forgomonika wrote:
| > I strongly prefer to get rbac/iam properly implemented in a
| single account wherever possible.
|
| This can really help manage complexity but it the more this
| account grows the riskier it becomes if a bad actor breaks into
| the account (ranging from fraudsters/hackers to disgruntled
| employees).
| sameg14 wrote:
| Multiple AWS accounts makes life a living nightmare. We have 38
| AWS accounts and it is incredibly difficult to maintain each one
| of them. IAM and even worse cross account IAM is horrible to
| author and maintain! Keeping track of resource limits and billing
| sucks. When using SSO, which we do, you cannot have more than one
| account open in the same browser at the same time. Use GCP
| instead, segregate your infra by project, use cloud identity/IAM
| and keep the head of hair you had in your 20s! You've been
| warned...
| avereveard wrote:
| You don't need to sso in each and every account, you can just
| have a user in the main org (or at any point in the org tree
| that is most appropriate) and assume the role within the
| account you want to manage.
| sameg14 wrote:
| We use Okta and put ppl in groups so I'm not sure if that
| would work.
| zeroimpl wrote:
| The browser will only remember the last 5 roles you've
| assumed, so it's still a pain.
| rcrowley wrote:
| That UX is atrocious.
|
| Substrate [1] instead presents you with a list of all your
| accounts with a link to assume your role in that account in
| the AWS Console (and parallel tools for assuming that role
| in a terminal, too).
|
| [1] <https://src-bin.com/substrate/>
| scubbo wrote:
| > When using SSO, which we do, you cannot have more than one
| account open in the same browser at the same time.
|
| http://willthames.github.io/2018/02/28/managing-multiple-aws...
|
| You're welcome :)
| rcrowley wrote:
| Curious / product research: Are your 38 accounts all in the
| same organization? Do you have any human IAM users left or is
| it all IdP, all the time? Do you use Terraform or anything like
| it?
|
| Also, yes, a pox on the single-player AWS Console. I've at
| least found a way to logout from one account and login to
| another in the same motion but it's still a poor experience.
| sameg14 wrote:
| Yeah all accounts are in the same OU. We do have human IAM
| users but those are "legacy". Nowadays Okta has been the
| preferred method of accessing AWS console and CLI. We do use
| terraform but that is also fragmented since each team has the
| freedom to innovate in their own way. People use CDK, SAM,
| CloudFormation, Terraform etc. This fracturing of IaC
| techniques has been a natural consequence of having too many
| silos aka. accounts and has made it hard to enforce
| consistency. I think having 2 or 3 accounts is probably ok
| for a small to medium size org. We are 96 humans so far.
| woojae wrote:
| I've seen really stupid things done whenever there are multiple
| accounts. Very few developers know how to assume a role properly.
| This leads to assume-role permissions that are too permissive to
| make things work.
| felipelalli wrote:
| There is a practical problem: AWS has a very restrictive policy
| for deleting sub-accounts.
|
| From:
| https://docs.aws.amazon.com/organizations/latest/APIReferenc...
|
| "You can only close 10% of active member accounts within a
| rolling 30 day period. This quota is not bound by a calendar
| month, but starts when you close an account. Within 30 days of
| that initial account closure, you can't exceed the 10% account
| closure limit."
|
| I have create so many accounts that when I had to erase them I
| was stuck in this stupid limit.
| nijave wrote:
| Somewhere I worked used to just run awsnuke and move them to a
| "deleteme" OU. I guess that's an option if you're okay with a
| bunch of empty accounts (they have a few hundred in there)
| rcrowley wrote:
| This is a bummer, yes. I haven't looked but I wonder if that
| 10% is a soft limit.
|
| At any rate, this is a good reason to use accounts for
| architectural divisions, not teams, and certainly not
| individual engineers.
| Sevii wrote:
| "Imagine you're trying to create the kind of isolation necessary
| to deliver the security, reliability, and compliance that
| business customers demand in one AWS account."
|
| Got to be my favorite way to describe two nines ever!
| rcrowley wrote:
| actual lol
| gw99 wrote:
| I disagree with this perspective. You should have multiple
| accounts but only if your organisation requires it for isolation
| or data protection reasons and only enough to perform the task.
|
| Every other reason here is because you fucked up. You have poor
| architecture, poor tagging, poor VPC design, poor IAM policy and
| role modelling or don't know what you are doing to start with.
| And some of the stuff doesn't even make sense, particularly the
| point about EKS upgrades (I run three different versions of EKS
| in the same account).
|
| There are many negative effects of multiple accounts including
| billing aggregation is very difficult, having to dig through
| several accounts worth of Cloudwatch logs, unexpected egress
| traffic costs and the overall complexity is much higher. If you
| have to switch to an org account and set up SSO you then have an
| administrative cost which is immense.
|
| My favourite thing doing is spending 2 days opening support
| tickets in 10 different accounts to get a limit raised and then
| tracking the state of all the tickets and limit changes...
| cactus2093 wrote:
| > You have poor architecture, poor tagging, poor VPC design,
| poor IAM policy and role modelling or don't know what you are
| doing to start with.
|
| I would flip it around and say that historically AWS has had
| extremely inconsistent architecture and IAM policy design that
| can make it very hard, sometimes impossible, to do it the
| "right way".
|
| The nice thing about using separate accounts is you don't have
| to get into as many of the hairy weeds and the permissions you
| end up might end up being much simpler to create and then also
| to maintain down the road, since everything is isolated by
| default and then you allowlist only the things you need.
|
| I don't see why you would frame this as "you fucked up" in your
| design.
| malfist wrote:
| Even if it only protects you from fuck ups, isn't that part
| of good design? People WILL make mistakes, limiting the blast
| radius of a fuck up is important.
| gw99 wrote:
| Historically yes. Your job is to evolve this configuration as
| security controls improve. It's not a fire and forget
| process, it's continuous improvement.
| cameronh90 wrote:
| Or you can just use multiple accounts, which makes things a
| whole bunch easier.
|
| Frankly, AWS is just missing a level of abstraction here.
| Azure has resource groups, Gcloud has projects. An AWS
| account now is just used instead of those concepts, despite
| it being heavyweight and awkward to do so.
| nijave wrote:
| There's plenty of tools to automate the creation and
| management of new accounts. The biggest hurdle afaik is
| there's no automated way to delete an account
|
| Azure also has higher-level subscriptions
| rcrowley wrote:
| AWS recently added the organizations:CloseAccount API
| (albeit with some caveats discussed elsewhere in this
| comment tree).
| rcrowley wrote:
| You're absolutely right, I've fucked up plenty. That's why I
| believe so strongly in making the right thing the easy thing. I
| think I'm doomed if it's critical for tags to be perfect
| because they always drift, doomed if IAM policies must be
| least-privilege because sometimes that's impossible, and doomed
| if I can't adapt what I built yesterday to what my business
| needs today.
|
| PS good job partitioning your EKS clusters, even within one
| account. That'll save you some sleep one day, I'm sure.
| stingraycharles wrote:
| We keep dev/staging/prod in different accounts, and we host
| some special environments for customers which is in separate
| accounts as well.
|
| The only reason you want to do these things is for security and
| liability purposes. They are not an advantage for organizing
| things, but rather they protect against accidental resource
| deletion and unauthorized / unintended access.
|
| It's cumbersome to manage many accounts, but that's a feature,
| not a bug: it's _intended_ to be difficult to switch between
| accounts, so that you can't accidentally fat finger the wrong
| button.
| gw99 wrote:
| They don't really though because in the several systems I've
| seen configured like that, the same staff have access to all
| accounts. Accidental deletion and unauthorized access should
| not be a possibility via policy rather than isolation,
| because if the former isn't done then the latter will not
| help. If the client demands it, sure, but charge them for the
| inconvenience.
|
| The main risk is hitting resource limits which is why I would
| and do give development teams their own account to cause
| mayhem in.
| nijave wrote:
| The places I've worked just control that with SSO and
| groups in their auth system. Much easier to make a group in
| your auth software than try to create multiple permission
| boundaries in the same AWS account. It also makes cost
| allocation much easier (you don't need complicated tagging
| policies everywhere)
| Hikikomori wrote:
| Disagree. Shared accounts may work for a smaller org, but not
| for larger ones.
|
| If you need multiple accounts for isolation or data protection
| you can accomplish the same thing with scoped IAM policies, so
| you can only touch things with the correct tag or name in path,
| this is hard to maintain and confusing for developers.
|
| When you are large enough there's no avoiding multiple accounts
| as you run into API limits and hard quotas. The added benefit
| of having an account per system is that you can have simple IAM
| policies, essentially _:_ compared to scoped described above.
|
| Networking is simple with shared VPCs, even with hundreds or
| thousands of accounts. Unclear how this affects egress at all.
| Don't know about billing but we have an Org with enterprise
| support, guess that helps. The awslogs cli tool makes it easy
| to extract logs via cli, just login to each account in its own
| terminal window or use profiles.
| nijave wrote:
| I worked with one of those tag based accounts a few years
| ago. Incredibly frustrating because it was locked down to the
| point you couldn't see the IAM policies and figure out what
| tags or metadata you were missing.
| TallGuyShort wrote:
| There's definitely a place for these counterarguments, but I
| have had Amazon's own consultants tell me that quota issues are
| exactly the reason they think you should architect to span many
| Amazon accounts if you're big. There are some quotas they
| simply can't raise as high as you want them on a per-account
| basis.
| arinlen wrote:
| > _I disagree with this perspective. You should have multiple
| accounts but only if your organisation requires it for
| isolation or data protection reasons and only enough to perform
| the task._
|
| I'm not sure where you got your take from. The blog post you're
| commenting explicitly makes the case for multiple AWS accounts
| with isolation in mind. In fact, the blog post barely mentions
| anything beyond isolation. How can you claim you agree and
| disagree at the same time?
| [deleted]
| forgomonika wrote:
| Is billing aggregation a problem and do you need to open up 10
| different support tickets if all the accounts are part of an
| AWS Organization?
|
| As for keeping the number of accounts down, I've seen that blow
| up significantly if you have someone break into an account or
| have a disgruntled employee that can get into a single account
| and wreak massive damage. Or if your software runs in multiple
| regions and you need to meet compliance requirements like GDPR,
| etc.
| gw99 wrote:
| None of these concerns require multiple accounts. An account
| is a container for resources and resources do have tangible
| isolation between them if configured properly and properly
| delegated credentials should be issued to staff which are
| specific and limited in capability. Same with assumed roles.
| Same with VPC configurations.
|
| If you didn't do that, you fucked up. Adding more accounts
| doesn't guarantee that you didn't fuck up.
|
| And yes you need to open 10 tickets, even if you have enough
| spend to have a direct line to AWS internal staff with
| multiple enterprise accounts...
| scarface74 wrote:
| - get all of the account numbers in your org
|
| - for each account number - aws service-
| quotas request-service-quota-increase
| rcrowley wrote:
| Actually upgrading your Support plan is one of the very, very
| few things you still need to break into the root of each
| account to do. However, if you're big enough you just sign an
| EDP contract that forces all your accounts onto Enterprise
| Support anyway.
| rcrowley wrote:
| Actually, as of Friday, that's no longer true. \o/
|
| <https://aws.amazon.com/about-aws/whats-new/2022/09/aws-
| updat...>
| Merrack wrote:
| This papercut has, as of last week, finally been fixed:
| https://aws.amazon.com/about-aws/whats-new/2022/09/aws-
| updat...
| ragona wrote:
| > poor IAM policy and role modeling
|
| This is a bad take. Making good IAM Roles and Policies is
| incredibly complicated if you have a complicated account. You
| WILL get it wrong. This becomes much more tractable if you have
| reasonable account boundaries between workloads.
|
| If you insist on a single massive account you're fighting
| against the way that AWS designs the system, and you're gonna
| have a bad time.
| thedougd wrote:
| Totally agree. Defense in depth, security in layers. You're
| not protecting against just the most elite hackers, you're
| protecting against mistakes. Mistakes and change are
| inevitable, they should be in the design.
| scarface74 wrote:
| Having things "in a VPC" doesn't help unless you are just using
| AWS as a glorified Colo. Once you start using AWS native
| services - many of which aren't in a VPC, VPC isolation doesn't
| help.
|
| Each AWS account also has its own service limits and quotas.
| One out of control Lambda in the dev "environment" can impact
| production.
|
| And you don't have to open support tickets manually. There are
| APIs to request service limits and you can monitor the progress
| programmatically.
| icedchai wrote:
| I run into "IT professionals" all the time who don't even
| realize you can run services without a VPC.
| scarface74 wrote:
| Besides, every time you try to create a "least privileged
| role" to run infrastructure as code, that role has so many
| privileges it's easy for mistakes to cause mistakes in your
| production "environment" that you meant to only affect your
| dev "environment".
|
| And I know I'm probably preaching to the choir. But one of
| the misconceptions I have to constantly fight is "we run
| our Lambdas in a VPC for security reasons". (Lambdas are
| never "run in" a customer VPC)
| epberry wrote:
| Multiple AWS accounts is definitely a best practice in larger
| engineering orgs. We implemented a CloudFormation StackSet to
| ingest them into our billing tool and lay them out appropriately.
| This proved to be a very slick solution from AWS so it made me
| believe that they want their larger customers to use multiple
| accounts too.
|
| If you are smaller I would not recommend it. Many things become a
| little more difficult, as others have pointed out. Oftentimes a
| devops or platform engineering org will paper over these things.
| nijave wrote:
| AWS recommends a multi account strategy to help achieve their
| Well Architected framework.
| https://docs.aws.amazon.com/whitepapers/latest/organizing-yo...
| moralestapia wrote:
| Every serious project I engage on has its own:
|
| * domain (obviously)
|
| * emails
|
| also for every provider I have a different email like
| (google@domain, twilio@domain, etc...)
|
| * credit cards (my bank makes it super easy to just create new
| ones)
|
| * phone no. (I just buy a burner phone)
|
| It's a bit of a PITA but the benefits outweigh the cons. I have a
| clear understanding of how much each of them costs me, for
| instance. Plus, the ban hammer will not be able to destroy all my
| income in one blow.
| simonw wrote:
| Which bank is that?
| moralestapia wrote:
| BBVA (operates in LATAM and Spain)
|
| They give you an app where you can create virtual credit
| cards. Quite neat, I literally waited for something like that
| for _decades_.
|
| Coincidentally, I just set up a Mercury account last week and
| I think they have a similar feature but haven't explored it
| yet.
| twelve40 wrote:
| in the US, at least Citi (that i know of) lets you create
| virtual card numbers.
| tomrod wrote:
| For phone, what are your thoughts on something like Google
| Voice instead?
|
| For emails, how do you handle email? Outlook/Google Workspace?
| Something else?
| moralestapia wrote:
| I've never used Google Voice so I cannot say about that.
| Where I'm at you can buy a cheap phone for like $20 and it
| comes with a prepaid SIM that never expires. I have about 10
| of those on a drawer on my desk LOL, it goes well with the
| hacker vibe B). Funny thing is I don't have them labeled, so
| when I need it I have to turn on all of them to see which one
| gets an SMS.
|
| For email, my domain provider handles email for me as well, I
| have a Google Workspace account and first thing I do when I
| set up an email is to link it there so I have a single Inbox
| for everything. (Although my real Inbox is the Spam folder,
| where a lot of genuine emails land now).
| nijave wrote:
| A lot of services block VoIP numbers (including Google
| Voice), especially if two factor or some sort of verification
| is involved.
| jesuspiece wrote:
| Only if you're gonna use AWS Orgs/AWS Control Tower to help
| manage everything otherwise it gets wild.
|
| Internally, Amazonians also use AWS accounts per "app" basically.
| redditor98654 wrote:
| In fact per app per region that the AWS service exists in. My
| AWS service exists in 24 regions so that is 24 "prod" accounts,
| 8 "gamma" and 6 "beta" and 1 "alpha" account. Some newer teams
| go even further. If the app is made out of different micro
| services, then each micro service gets its own account per
| region. Typically a medium sized AWS service will have like 30
| micro services behind it (not really micro; these tend to hold
| lots of code and APIs) so that would be 24 regions * 30 micro-
| service accounts. Tooling is important without which you cannot
| manage so many accounts.
| libria wrote:
| https://www.lastweekinaws.com/blog/the-aws-service-i-hate-
| th...
|
| > The fact that it's the internal service used to provision
| AWS accounts means that AWS engineers building AWS are
| insulated from the way that the rest of the world manages AWS
| accounts-or should I say, ways. They don't have to deal in
| the same way with AWS Organizations or Landing Zones or
| Control Tower or AWS SSO (an absolute hidden gem of a
| service, by the way). And that's the crux of my beef with the
| service.
|
| Apparently, /u/quinnypig dislikes this account-manager-thing
| specifically because AWS has been hoarding it to themselves.
|
| His point is valid: When is AWS going to advertise account
| per region per service as an official default policy and
| bring AWS Orgs up to par with it? They think it's the right
| way, why not for the customer?
| malcolp wrote:
| I love this approach, although I'm yet to work anywhere that does
| this.
|
| I guess the million (thousand?) dollar questios now become where
| do you draw the boundary across accounts? Presumably there are
| many bad ways to slice accounts up. And what happens when
| accounts do need to communicate?
|
| I can imagine three major scenarios for cross account
| permissions:
|
| * Cross account iam policies (painful in my experience) * Adding
| trust policies to enable cross account assuming roles (better but
| limited) * Limiting cross account policies to specific easy to
| configure services. E.g. S3 (best option, I've seen but maybe too
| limited)
|
| Would love to hear the author's view.
| iLoveOncall wrote:
| > Would love to hear the author's view.
|
| Instead you can hear AWS's view, which is to have one account
| per stage per region per service.
|
| I can't find a source but I work for Amazon and this is what
| was recommended to us by ProServe (the contracting branch of
| AWS) when we talked with them.
|
| I think it's idiotic though (because regions are 100% separated
| within an account, and it would easily triple the number of
| accounts to manage), and so did my team, so we stuck with one
| account per stage per service.
|
| That said, cross account permissions is really not an issue,
| it's very easy and straightforward to setup. You also should
| not need it in 90% of the cases if your application is properly
| split with the right ownership for each microservice.
|
| For my current team we manage probably more than a thousand AWS
| accounts, and permissions are never an issue. Neither is
| anything else actually. We aggregate metrics in a single
| account for the stuff that needs to be aggregated, we have
| small CLI scripts that automate tedious steps like requesting
| limit increases, etc.
| brodouevencode wrote:
| > Instead you can hear AWS's view, which is to have one
| account per stage per region per service.
|
| This is exactly correct.
|
| > I think it's idiotic though (because regions are 100%
| separated within an account
|
| But still bound to the same service limits, no?
| iLoveOncall wrote:
| > But still bound to the same service limits, no?
|
| I'm not sure what you are referring to (pricing or service
| quotas?, I will assume the latter), but I think it depends
| on the service. For some setting the service quota will be
| per region and for others it will be global.
|
| One I know that is for sure regional is the limit reserved
| concurrent executions of Lambda functions.
| rcrowley wrote:
| Service limits for regional services like EC2 are regional.
|
| Service limits for global services like Organizations
| appear to only be manageable from us-east-1.
| redditor98654 wrote:
| There are global quotas per account that can sneak up on you
| if you have too many services running in the same account
| albeit in different regions. DynamoDB total read and write
| capacity comes to mind.
| zmgsabst wrote:
| One example of why you might want one account per region:
|
| You regionalize data (eg, US data in the US, EU data in the
| EU) and you want to be able to show (and enforce) separation
| for compliance or security reasons. You might even take that
| further, and have multiple accounts per region to create
| "cells" that correspond to segments of that region.
|
| Disclosure: I worked on Amazons tax pipelines.
| rcrowley wrote:
| Author here: I have settled on account per service per
| environment with a couple of exceptions.
|
| Sometimes I run multiple services in one account if they're so
| tightly coupled as to be useless as a group if any one is down.
| (This has practically come up when two services are codesigned
| to multiplex TCP connections to support tens of millions of
| clients.)
|
| Sometimes I run a single stateless production service in two
| accounts and route 10% of traffic to the canary account and 90%
| to the other one.
| nijave wrote:
| I'd say application boundaries at a high-level. Your various
| environments should be completely isolated. For instance, you
| can split web services at the load balancer level.
|
| Application boundaries tend to mirror org structure so that
| allows you to scope down team access.
|
| To get a better idea of your architecture, you can create a
| dependency diagram and look for clusters of things.
|
| As for connectivity, you could go over the internet, use VPC
| peering, use Transit Gateways, PrivateLinks, or follow a
| hub/spoke network architecture with a "network account"
|
| If you're using managed services, you can also use things like
| SNS and SQS to create shareable communication channels for
| other accounts to use.
| Havoc wrote:
| Isn't that against terms since it activates a free tier credit?
| traspler wrote:
| Multi Account is a normal pattern for AWS. They even have tools
| to handle that better like AWS Control Tower. Someone from AWS
| we talked to even mentioned it as a perk (to get the free tier
| on every account).
| bilalq wrote:
| That person was misinformed or maybe the context was outside
| the scope of business needs. If you use link accounts within
| an AWS Organization, one of them becomes the payer account
| with consolidated billing. There's a shared free tier across
| all accounts. If you actually spin them up as completely
| separate accounts and don't link within an org, you do get
| extra free tiers, but most corporations would rather not have
| the headache that comes with that.
| rcrowley wrote:
| I'm not honestly sure how AWS Organizations interacts with the
| AWS free tier.
|
| Rest assured, though, having lots of AWS accounts (and using
| AWS Organizations, their service designed to _help_ you use
| lots of AWS accounts) is _not_ against the terms of service.
| brodouevencode wrote:
| Each account within the organization gets the limits of the
| free tier, just as a single account would. I think the
| reasoning is that if you're going to go through the hassle of
| setting up orgs then you're probably an enterprise user
| slated to take the long haul anyway.
| hgh wrote:
| AWS gets many things more right, but I think GCP wins on how they
| handle Projects and Managed Instance Groups for auto-scaling.
| robohoe wrote:
| GCP organization structure is such a breath of fresh air after
| dealing with AWS. AWS Orgs and all the complexity with VPC and
| DNS management at a scale of hundreds of accounts is just a
| complete pain in the arse. GCP makes it far easier with Shared
| VPC and org/folder/project structure.
| remram wrote:
| Easily separating billing accounts is definitely a must.
| yonixw wrote:
| My anecdote on how we do it:
|
| - We have AWS Org
|
| - Each account has no root IAM and cost/pricing goes through root
| AWS Org Account
|
| - You move between accounts with AWS SSO (now IAM Federation) -
| No more password per account
|
| - AWS SSO standardizes boundaries across account with IAM
| policies, like eu-centeral-1 only for dev IAM etc.
|
| - Inside Account more granular access with IAM Assume Roles
|
| - Each account Cloudtrail to a central S3 for audit (Same region
| pricing works for diff accounts!)
|
| - Each account is `project.env` like micro_service1.dev or
| company1_k8s+s3.prod
|
| - That way, we have pricing per project built in where tag
| support ends (and it do have limits)
|
| We came to this structure realization after we saw Azure Resource
| Groups and Google Projects. We also saw the 5 VPC soft limit AWS
| has per region and think it's kind of a clue from amazon: "pss..
| this account soft/hard limits is sized for 1 project/deployment"
|
| The only problems come from automating stuff, like Route53
| records that can point automatically to load balancers in that
| account or as mentioned in the article, VPC2VPC, although we use
| serverless stuff like lambda and s3 and experience less of that.
|
| But as time go on we realized, like the VPC example in the
| article, that those problems forced us to structure our stuff in
| a more SOLID way, which became a feature for us. Just like moving
| docker-compose to k8s is not creating app-mesh and ops problem,
| but REVEALING them. So the solution for the Route53 example is to
| have a separate subdomain zone in each account for automation and
| ADDING another account `main_route53.prod` with a root zone
| pointing to them.
|
| Hope this helps for the curious.
| zimbatm wrote:
| AWS Control Tower is great to set all of this up. It's
| basically a layer on top of AWS SSO, AWS Org, Cloudtrail, AWS
| Config. With some sane default security policies.
| rcrowley wrote:
| Control Tower is cool if the problem is "I need lots of AWS
| accounts." Substrate [1] is cool if the problem is "I need to
| accomplish something and I'm cool with using lots of AWS
| accounts to do it."
|
| [1] <https://src-bin.com/substrate/>
| pan69 wrote:
| > You move between accounts with AWS SSO (now IAM Federation) -
| No more password per account
|
| The only thing I really hate about this is that it is
| tied/bound to your browser. If you switch browsers or use an
| incognito window you have to through the whole dance of setting
| up your account switching set up. Imagine you're in multiple
| orgs that are set up this way...
| dijit wrote:
| One of the things I love most about google cloud is that
| "projects" are easy to create and easy to link to other projects.
|
| Roles and service accounts can even reference across projects,
| though I'm not sure I'd recommend doing that.
|
| No more faffing about with special accounts, passwords and
| difficult to configure shared VPCs, it all becomes so easy.
|
| Even managing the different accounts is difficult without browser
| extensions such as this one:
| https://chrome.google.com/webstore/detail/aws-extend-switch-...
| rcrowley wrote:
| Author of the article here: I agree. GCP projects are a better
| abstraction. I still think AWS is, on balance, a better cloud.
| dijit wrote:
| I'd love to hear more about why you think that's the case!
|
| Maybe the next blog post?
| simonw wrote:
| The killer feature of AWS is that they've broken almost
| nothing since first launching over 15 years ago. Their
| commitment to keeping old stuff working is truly amazing.
|
| No other cloud hosting service comes close, because no
| other cloud service has had 15 years to prove themselves in
| the same way!
| jimt1234 wrote:
| SimpleDB is still running!
| atonse wrote:
| AWS SSO has made it incredibly easy for us to secure and manage
| access to (and switch between) all our AWS accounts in the org.
|
| I'm a huge fan and recommend it.
| tpmx wrote:
| They recently rebranded it to the super-catchy "AWS IAM
| Identity Center (successor to AWS SSO)". Gotta love Seattle-
| based tech marketing people, presumably trained at Microsoft.
| :)
|
| I guess it will soon be the default user management system,
| and 'proper' IAM will be the low-level one. I see this as a
| reaction towards GCP's IMHO superior UX/system design in this
| aspect. I don't think they can entirely catch up because of
| early and bad architectural decisions regarding
| projects/accounts.
|
| Also a fan, so far.
| 0xbadcafebee wrote:
| I really dislike the Google Cloud way of doing things, for
| example Projects, Folders and Orgs. There's a dozen different
| weird paradigms that you have to consider to securely and
| reliably manage a large number of different projects, accounts,
| tenants, business units, etc in Google Cloud; if you don't do
| things "The Google Cloud Way" you are screwed.
|
| AWS is much simpler and more straightforward. You don't have to
| think about _anything_ to segregate infrastructure, networks,
| applications, users, data, etc. Just put it in a different
| account. If someone needs access, you need to explicitly add
| extra connections /grant that access. You can't just
| accidentally create one user that implicitly has access to
| hundreds of accounts.
|
| Honestly the entire security model of Google Cloud is
| frightening. It's like they wanted to make it easy to expose
| everything.
| lbhdc wrote:
| You can still work like that. You just don't have to bother
| changing your project id, and switch accounts to jump to
| other projects. I do this for some things I work on where
| there are account level controls and the accounts cant be
| shared across projects.
| [deleted]
| psanford wrote:
| I don't know. I've found it to be pretty difficult to answer
| the question "which of these 1000 gcp projects are running a
| production workload and which are random one offs created by a
| dev messing around or by a google sheet script?"
| dijit wrote:
| I solve that by using folders. But when you have many of
| anything it will become hard to reason with.
|
| It's not worse than your filesystem, in theory, but speaking
| for myself: my filesystem is a mess, so...
| GauntletWizard wrote:
| One reason to have your roles and service accounts reference
| cross-project is to give your CI builder access to your
| artifacts project; Have your prod and dev environments use
| containers from a third account that serves only as an archive
| of your build artifacts.
| pid-1 wrote:
| > My favorite way to create a network between all my services
| hosted in different AWS accounts is to share a VPC from a network
| account into all my service accounts and use security groups to
| authorize service-to-service communication. There's no per-byte
| tax, zonal architectures are easy to reason about, and security
| groups work just like you expect.
|
| That's gold advice. I wish AWS RAM supported more services (like
| AWS EKS).
|
| A small complain: working with AWS SSO is a bit tedious. My
| current solution is to share my ~/aws/config with everyone so we
| all have the same profile names and scripts can work for
| everyone.
| rcrowley wrote:
| I have thought a great deal about whether I also want EKS
| clusters to officially support nodes in multiple AWS accounts.
| On the one hand, having the option to create that additional
| low-level isolation would be lovely, even and maybe even
| especially if I didn't always take it. On the other hand,
| isolating two things from each other but then tying them to the
| same Kubernetes cluster upgrade schedule feels wrong.
|
| In the end, I decided that if I care about isolating two things
| enough to put them in separate AWS accounts, I'm willing to
| spend the $75 per month that it takes to have separate EKS
| clusters, too. (This opinion perhaps obviously doesn't fit will
| with hobby/side project budgets.)
| rcrowley wrote:
| Synchronizing ~/aws/config gets harder and harder as your team
| grows because there are both more people who need to receive
| changes and more people making changes.
|
| I think the human-readable names for AWS accounts need to be
| part of the account, not part of the laptop. Substrate [1] does
| this so that you can type commands like `substrate assume-role
| -domain example -environment production -quality beta` [2] to
| get where you're going.
|
| [1] <https://src-bin.com/substrate/> [2] <https://src-
| bin.com/substrate/manual/moving-between-aws-acco...>
| nijave wrote:
| One place I worked just committed it to the git repo
| (obviously just SSO and account numbers, no passwords or
| tokens)
| thedougd wrote:
| Why can't the CLI generate the config if I can see all the
| accounts and roles in the SSO start page? That's a desperately
| needed feature.
|
| I would love to see a browser extension for SSO account tabs if
| AWS can't solve it natively.
| pid-1 wrote:
| `aws configure sso` does that, but:
|
| 1 - You need to do that with each account / role pair
|
| 2 - It gives profiles very long names by default
| (<role>-<account_number>)
|
| 3 - It does not set $AWS_PROFILE, so you need to pass `<...>
| --profile <...>` manually
|
| So the code is actually there already, they just need to make
| the experience better.
| noahmasur wrote:
| > My current solution is to share my ~/aws/config with everyone
| so we all have the same profile names and scripts can work for
| everyone.
|
| If you're on Mac/Linux you could have everyone use direnv. Add
| a .envrc file in each git repo (or your script's subdirectory)
| with `export AWS_PROFILE=profilename`. Now everyone is working
| with the same profile names without having to pass around
| config files.
| tobyjsullivan wrote:
| Define "lots." Because the default limit in AWS is 10 accounts
| per org.
|
| Quota increases can be requested but the default limit tells us
| what AWS thinks normal usage should be for most use cases.
|
| https://docs.aws.amazon.com/organizations/latest/userguide/o...
|
| Perhaps the author meant "more than one"?
| psanford wrote:
| > Quota increases can be requested but the default limit tells
| us what AWS thinks normal usage should be for most use cases.
|
| This is certainly not true for a lot of AWS quotas. If you hit
| an ec2 quota for an instance type, that isn't AWS telling you
| that you are using too much compute. Its there as a speedbump
| to make sure you have some idea that you know what you are
| doing and to make sure AWS can actually service your requests.
|
| AWS will happily let you have hundreds of child accounts. In
| fact, if you are talking to them about your architecture they
| will even encourage it (assuming that it is actually
| appropriate for the scale of your organization).
___________________________________________________________________
(page generated 2022-10-03 23:01 UTC)