[HN Gopher] Querying AWS at scale across APIs, regions, and acco...
___________________________________________________________________
Querying AWS at scale across APIs, regions, and accounts
Author : nathanwallace
Score : 48 points
Date : 2021-11-11 18:44 UTC (4 hours ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| oatmeal_croc wrote:
| Wonder how it handles the API rate limits
| nathanwallace wrote:
| Hi - I'm a lead on the Steampipe (https://steampipe.io)
| project. Rate limits are definitely an interesting challenge,
| but actually less of an issue for AWS than many of our other
| plugins.
|
| Because rate limits are separate across accounts, regions and
| services, the parallel queries work really well in those cases.
|
| We do sub-API queries for tables, but only when the column is
| requested. So, "select name from aws_s3_bucket" just does a
| list call while "select * from aws_s3_bucket" does multiple API
| calls per row. These sub-API calls are the main potential
| source of rate limits since they hit the same API [1]. BTW,
| Cloud Control from AWS is actually much more subsceptible to
| this problem! [2].
|
| We also use a custom backoff algorithm that is fast then slow
| to give good speed in the usual cases and ensure results if
| throttling ramps up [3].
|
| Finally, we automatically cache results in memory between
| requests and can always save results into materialized views or
| similar to avoid repeated calls in larger cases.
|
| TBH, live querying of APIs has proven so much more effective
| than we even hoped when we started!
|
| 1 - https://steampipe.io/blog/selective-select 2 -
| https://steampipe.io/blog/aws-cloud-control 3 -
| https://github.com/turbot/steampipe-plugin-aws/blob/4cbd8813...
| posnet wrote:
| A few years ago I built something similar to this, in that it
| used the JSON API definitions to automatically scrape the
| entire AWS API surface area.
|
| But I was doing it every 5 minutes and storing the result in
| postgres instead of it being on demand. I very quickly
| received a call from our account rep asking for it to not
| query particular services as frequently.
|
| Apparently some of the newer services at the time I was
| scraping were not as optimized in terms of response caching
| or metadata queries as they should have been.
|
| (Note: apparently it wasn't the overall frequency that was
| the problem, it was a few services that returned paginated
| results and fetching every page every 5 minutes was causing
| issues.)
| cddotdotslash wrote:
| Interesting! Feels similar to CloudQuery[1] which is also open
| source.
|
| [1] https://www.cloudquery.io/
| jiggawatts wrote:
| Seems to be catching up to Azure's Resource Graph...
| jcims wrote:
| Boggles my mind that AWS still doesn't support something like
| this.
| jiggawatts wrote:
| AWS and Azure have different philosophies:
|
| Azure uses a unified resource manager that has everything in
| "one place".
|
| AWS uses a "federation of systems" with almost nothing in
| common, except for authentication.
|
| Azure's approach is more consistent and makes things like
| resource graph relatively easy to implement.
|
| AWS's approach is more robust against failure, because the
| various service offerings are more independent and have fewer
| common failure points.
|
| This isn't just theory, it matches my experience pretty well.
| I hate how in AWS I have to switch portals to go from
| service-to-service, whereas in Azure the "Resource Groups"
| contain all related resources in one place, even if it's a
| random mix of IaaS and PaaS. Conversely, I've been bitten by
| several wide-spread Azure outages. Repeatedly.
|
| It all depends on your business requirements. For me, the
| consistency is more important. For other people, availability
| is far more critical.
___________________________________________________________________
(page generated 2021-11-11 23:01 UTC)