[HN Gopher] Localstack - Local AWS Emulator ___________________________________________________________________ Localstack - Local AWS Emulator Author : emersonrsantos Score : 209 points Date : 2021-10-10 17:07 UTC (5 hours ago) (HTM) web link (github.com) (TXT) w3m dump (github.com) | brycelarkin wrote: | How does Localstack compare to SAM and Serverless local? Is it | easier to use? | | Have a couple lambda services backed by API GW that I'd like to | have a better dev env for. | whummer wrote: | One of the main differentiators we see is that frameworks like | SAM or Serverless are pretty opinionated about how your app | should be defined/structured/implemented. These frameworks work | absolutely great, as long as you 100% buy in to their way of | doing things. This creates a certain lock-in effect, and may | make it harder to switch to a different framework/approach | later on. | | LocalStack takes a different approach by working on the API | emulation layer - hence making it easier to switch and | integrate with various different application development | frameworks. | | Btw - there's also a Serverless plugin, as well as a "samlocal" | command line - in case you'd like to give it a try: | | https://github.com/localstack/serverless-localstack | | https://github.com/localstack/aws-sam-cli-local | brycelarkin wrote: | Awesome! I'll give it a try | whummer wrote: | Hey all - LocalStack co-founder here. Great to see this article | trending. :) | | Happy for any additional suggestions and questions you may have. | Looking forward to getting your feedback! | DandyDev wrote: | Localstack Pro looks quite nice and comprehensive, but the CI | pricing seems to really discourage proper continuous | integration best practices. At 10 cents per run you're | encouraging developers to run their CI pipelines as little as | possible, while ideally you'd run CI on each commit and you'd | commit often, probably 10s is times a day. In that case | Localstack Pro for CI would get costly very quickly. | | Why not have CI pricing per project/repo. Regardless of how | often it runs. | whummer wrote: | Thanks for the feedback, great points. We're going to put a | strong focus on our CI offering (both from a feature and | pricing point of view) over the next couple of weeks/months. | | For example, we're working on a new feature that will make it | easy to snapshot the state of the LocalStack instance during | and after each CI run, making the state "browsable" in the | UI, and providing advanced insights and analytics into how | the application stack and tests evolved over time. | | We're also looking into alternative options for pricing - | your point is well taken and will be considered for our | roadmap. We definitely want to encourage testing frequently | and on every commit - but I'd like to emphasize that our | current CI offering is just a starting point, with lots more | exciting features to come soon.. Thanks! | hw8kw13 wrote: | How would you differentiate yourself from a tool like Moto, in | terms of testing? | | https://github.com/spulec/moto | whummer wrote: | Great question - we're working very closely with moto, in | fact our team very actively contributes to the moto core on a | regular basis. | | The main differentiator is that moto is a Python library that | mocks out / implements the AWS APIs, whereas we see | LocalStack as a platform that aims to support the end-to-end | development lifecycle for your cloud apps. | | A lot of the work we do in LocalStack is to create a seamless | dev experience in your local environment - providing local | DNS integrations, persistence features, Lambda code mounting, | CI integrations, transparent injection of "localhost" | endpoints into AWS SDKs (e.g., for your Lambda functions), | and much more. Also, today we already provide some fairly | sophisticated integrations - e.g., our Athena API which | allows you to run your SQL-like big data queries natively | over the local S3 filesystem. This is out of scope for moto, | but a big focus area for us. | | Our aspiration is that you can take any (AWS) cloud app and | deploy it natively to LocalStack - which is already the case | in many scenarios, and improving on a daily basis. | | Btw, we're currently working on revamping/polishing our docs | with lots of more content and details - we'll push out an | update to https://localstack.cloud/docs in the next couple of | days! | rickstanley wrote: | Is there any equivalent for Azure? | whummer wrote: | Hi, we do have Azure on our roadmap - recently launched a beta | program with a few selected customers. We'd love to learn more | about your use case - if you're interested, please shoot us an | email to info@localstack.cloud . Thanks! | EADGBE wrote: | Can I use it in conjunction with something like CDK? I'm trying | to learn the AWS CDK, but I would feel much better if I could run | all these orchestrations on a local docker instead of a real AWS | account. | | ...And just trust that I destroy them before they incur cost. | whummer wrote: | Absolutely - CDK under the covers generates CloudFormation | templates, which deploy natively on LocalStack. Certainly a | great use case for local dev&test - especially if you want | quick feedback cycles, ability to destroy the stack | immediately, etc. | | There's also a "cdklocal" command line which should make it | fairly easy to get started with local CDK development: | https://github.com/localstack/aws-cdk-local | ismailegilmez wrote: | The LocalStack community is so supportive and the team is lovely! | k8sToGo wrote: | Hmm this will come in handy when writing my Terraform scripts. | whummer wrote: | Yep, local Terraform testing is a great use case - we're seeing | a lot of usage in this space. Here are some some pointers to | get started with the provider configuration: | https://registry.terraform.io/providers/hashicorp/aws/latest... | | We'll soon publish some more details on the TF integration in | the LocalStack docs... | | Side note - similar to the *.tfstate file maintained by | Terraform (storing metadata about resource deployment states), | we're increasingly making use of persistence files (called | "local cloud pods") that allow you to easily store the state of | your LocalStack instance, share it with team members, keep | track of changes over time, etc. Really nifty feature... | TylerJaacks wrote: | I wish some of the services weren't locked behind a paywall. | richardfey wrote: | You can also create resources using a YAML file, I found that | feature extremely useful! | whummer wrote: | Yes, great point - you can define and deploy your resources to | LocalStack using CloudFormation (either as YAML or JSON files). | This makes it very easy to maintain and exchange stacks in a | platform-neutral way. | | Btw - we're also offering integrations for Terraform (https://r | egistry.terraform.io/providers/hashicorp/aws/latest...) and | Pulumi (https://github.com/localstack/pulumi-local), among | others. | chrsig wrote: | i wish aws...and to be honest, most cloud providers put more | effort into the development experience. | | they have local dynamo, but i believe that's the only official | appliance they provide. | | I think it's especially necessary for the proprietary products -- | dynamo being a great example. | | I'll take a moment to call out snowflake for this as well -- it's | really frustrating to not be able to quickly (within | milliseconds) and easily (i shouldn't need to get credentials | from elsewhere in my org) setup & teardown databases while under | test. | emersonrsantos wrote: | I used Serverless framework to develop locally many times, it | works very well for my case. | thrauat wrote: | i think cloud providers have little incentive to help you | develop and test everything locally, since they benefit more | from onboarding you quickly into using their infrastructure, | which they can much better monetize on. it seems to me that a | completely local cloud stack would be threatening to cloud | providers also in that you can then start running non-critical | applications on-prem at no cost. | chrsig wrote: | i don't think that's a real threat -- i don't think | development environments are likely a huge income source for | them, either. | | i think they'd be better served by letting me develop more | quickly, so I can help my company grow, and presumably need | (and have the budget for!) more resources in our production | environment. | | if i'm mistaken, i think it's reasonable to have some sort of | built in time limit to how long the appliance will run for, | or some other way of preventing it being used for non- | development purposes. | GusEggert wrote: | There is also Step Functions Local | (https://docs.aws.amazon.com/step-functions/latest/dg/sfn- | loc..., it is also integrated with LocalStack). I built it as a | side project when I was at AWS, motivated by the same | frustrations. The best way to get these prioritized is to | request it through your account manager/SA/etc. AWS does listen | closely to customer feature requests. | mcbain wrote: | It isn't what you are asking for, but I was a bit | surprised/happy to discover that the Ruby AWS SDK has a good | stub client implementation: | https://aws.amazon.com/blogs/developer/advanced-client-stubb... | | I found it useful when updating some existing old ruby scripts | we had. (Of course they had little to no testing before that.) | jacurtis wrote: | In addition to local dynamo, they also have local AWS Lambda | execution via AWS SAM (Serverless Application Model). | | You invoke it via the command line | | `sam local invoke [OPTIONS] [FUNCTION_LOGICAL_ID]` | | Behind the scenes it is running an attached docker container | and feeding you the output. I found that it works actually very | well and mimics the cloud invocation environment perfectly. | | The problem is that it stubs out any other services you run. So | if your Lambda adds something to SQS or SNS or SES or S3 or | anything else, it simply stubs those out. So advanced functions | do get difficult to test locally. But This Localstack emulator | may solve those problems (but I haven't used it yet) | | Details: https://docs.aws.amazon.com/serverless-application- | model/lat... | mcintyre1994 wrote: | I've used localstack to mock things like Dynamo, S3 and Step | Functions for dev/integration testing of lambdas built with | SAM with great results. I actually didn't know that SAM | stubbed those out by default and haven't used it for a while, | but I found that if you point your AWS library at | localstack:4566 then it'll run it all against localstack | correctly. | Hikikomori wrote: | They also support local stepfunctions. | sverhagen wrote: | It frustrates me how they always go for the biggest common | denominator there, the lowest level shared by everyone, but | what's hardly used by professional users: don't let me download | a jar file, please give me Maven coordinates. Don't tell me how | to click in the console to get an EC2 instance, please give me | an example Terraform snippet. Don't just show me how AWS CLI | returns a blob of JSON when I assume role, please tell me how | to use aws-mfa properly. I understand and respect their hands | off approach to these more opinionated tools. But ergo, I tend | to skip over the Amazon properties when finding what I need in | the search results! | | PS: Localstack is awesome! Also in combination with | Testcontainers. | whummer wrote: | +100 | | Another example to add to your list - the new AWS CLI v2: | | > don't force me to download a 30MB binary with embedded | Python interpreter baked in, please just give us a pip | package instead! | | ;) | Sanguinaire wrote: | Or better yet, rewrite it in Go to avoid both the fat blob | and python dependency management at the same time! | whummer wrote: | Touche! | ed_elliott_asc wrote: | I've been on both sides of this and some consumers are fine | but some have no idea how to consume or initial setup is so | broken, this is much easier | whummer wrote: | Right, no doubt that some customers required fully | packaged solutions. | | I guess it's generally hard to find the right balance, if | you're catering to customers of different level of | technical expertise/maturity. | | (Btw, the previous comment was in reference to: | https://github.com/aws/aws-cli/issues/4947) | zokier wrote: | I general I do agree, but otoh I'm not sure if local appliances | are the solution. I believe AWS' vision is that development | happens against real cloud resources, which seems neat on paper | but to that to really work provisioning resources needs to be | super smooth to easily create developer sandboxes. | CloudFormation definitely is not it. | gyre007 wrote: | This tool is very useful for integration testing, but | unfortunately it can be rather slow. | whummer wrote: | Hey - LocalStack founder here. Thanks for the feedback. | | We're currently investing some time and resources on | refactoring the code base to improve the performance. We have | some improvements in the pipeline which we should be able to | roll out within the next 1-2 weeks. | | Performance (as well as general UX) is definitely high up on | our priority list, and we'll continue to invest heavily in this | area. Stay tuned! | | Happy for any additional suggestions and questions you may | have. | mdoms wrote: | Yes I use localstack a lot and it's very nice but VERY slow and | I can't really understand why. | thrauat wrote: | Is it the startup time or the overall performance during | runtime? | thdxr wrote: | It's impressive what Localstack has been able to do to keep up | with AWS. | | However for my own projects, we consider local development like | this an anti-pattern. IaC tools like Terraform/CDK/SST make it | easy to spin up environments for each developer in the cloud. | This may sound crazy at first but would recommend giving it a | shot. | | If you're fully taking advantage of managed AWS services the best | development environment is running everything in AWS. There's | definitely some friction but you reduce the "was working on my | machine but not in production" situations. | sozal wrote: | LocalStack is super useful tool for developers during local | development and e2e testing. We have been using it for one year | and it really helped us to debug and find issues without | deploying to real AWS environment. | shiftpgdn wrote: | I've been saying for a couple of years whoever builds a tool to | one click deploy an AWS alike to run on local bare metal has | built the next billion dollar a year company. | | I know of a lot of big shops that are desperate to get out of the | cloud due to 7-8 figure AWS bills but their software and | engineering is too tightly wound into AWS tooling. | asim wrote: | Agreed. Although it's a big ask. I have something that's slowly | going in that direction but it'll be a while before it gets the | interest I'm looking for. | | https://github.com/micro | rkeene2 wrote: | I built this a few times. The most current implementation is | called CloudSeed, though I am trying to fundraise for an open | source version (with a RedHat-like business model of support). | | CloudSeed has been a key part of several >$1bn contracts for | KnightPoint/Perspecta/Peraton, most recently DHS DCCO ($2.7bn). | kongin wrote: | It's kind of stunning that starting with fully open | technologies we build another closed mainframe on top of it. | | All that's old is new again. | whummer wrote: | Agreed, that's really stunning - seen the analogy of cloud as | the "modern mainframe" come up more and more recently. | | I guess the (business) reason is that the centralized | computing model enables enormous economies of scale. (Makes | perfect sense from an operations point of view.) | | What it seems to miss in the current incarnation/iteration, | though, is the "power of distribution" - leveraging the fact | that local compute capabilities (especially during | development/integration) can help reduce the cognitive load, | increase the efficiency, and thereby reduce overall costs. | | There seems to be a tendency to focus mostly on the | operational aspects, rather than the overall end-to-end | developer journey. What remains to be seen is whether future | iterations of "the cloud" will do a better job of embracing | the power of distributed and/or hybrid. | ragona wrote: | I think this misunderstands the hard part of running AWS. | You're paying for the software, sure. What you're really paying | for is the hundreds of oncall engineers supporting you at all | times, and the culture of ensuring that those services keep | working. | kevinsundar wrote: | As an Amazon engineer who frequently goes oncall, this is | Amazon's true competitive advantage. | | The company works because every night there's 1000s of | engineers sleeping with a pager by their head. | devit wrote: | Why do that for general customer support instead of hiring | in different time zones? | IceWreck wrote: | But AWS is huge. They employ thousands of engineers. Granted | most of the complexity is due to its huge huge scale, but still | it has too many features to do what youre asking. And then | there are AWS exclusives like DynamoDB. | | Yes there are open source public/private cloud alternatives | like OpenStack but they won't solve the problem of lock in and | you'd still have to change all your tooling. | kortilla wrote: | > they won't solve the problem of lock in | | Conflating lock in to an open source tool with lock in to a | vendor who you literally have to keep paying money to survive | is a shitty tactic used by companies like Amazon. | | Using open stack absolutely solves the problem of lock in | from a business perspective. You can't be cancelled, the | price can't be changed, etc. | NathanKP wrote: | I legitimately wonder how many people feel free from lock | in after they finish paying $75k to Canonical for a 12 node | OpenStack "private cloud build" | (https://assets.ubuntu.com/v1/d4766546-Private-Cloud- | Build_Da...), not to mention ongoing support bills and | billing for addons "starting at $25k". | | Just because something is open source does not mean your | org will actually have the expertise to be successful at | running it yourself. That's how most "open source" projects | like Kubernetes and OpenStack become so successful: the | supporting orgs behind the project make lots of money | charging companies for support and maintenance and issues | that pop up with the setup. | | There are many forms of lock in. Buying a big upfront | investment that has to pay itself off for the next X years | is a form of lock in. Paying for expensive professional | consulting support to build out the system and keep the | system up and running is a form of lock in. Or even worse, | hiring or training expensive dedicated employees to learn | the open source system and become the internal professional | consultants for the rest of the org, that is also lock in. | kortilla wrote: | > I legitimately wonder how many people feel free from | lock in after they finish paying $75k to Canonical for a | 12 node OpenStack "private cloud build" | | Here's the part where it's not lock-in. We stopped paying | for support from Redhat when we got big enough that our | own SRE expertise to run our own open stack cloud. | | You know what the impact was on our applications running | on it? Absolutely nothing. | | How much do you pay Canonical to use grep every month? | Inexperienced developers have just been conditioned by | cloud providers to think that an IaaS platform must cost | _something_ in payments per month to some tech company. | It does not. | | This is no different than Windows vs Linux on servers. I | can't wait until 20 years from now when all of the | proprietary shit looks ridiculous in hindsight. | zimbatm wrote: | I'm sure you can find other OpenStack professionals that | are willing to manage your Ubuntu OpenStack deployment. | | Of course every solution costs something. Lock-in means | that there is no other choice, or that the cost to switch | is so steep that it becomes prohibitive to do so. | temp_praneshp wrote: | > 12 node OpenStack | | I share your disdain(?) for openstack, but the 12 node | count there is minimum I think (to deploy the control | plane). You can likely add a reasonable number of | hypervisors at no cost. | | (I think I'm correct based on the table in the last page | of your doc, but happy to be corrected). | nostrebored wrote: | It's the same thing. You're paying for engineers to work | with a service or you're paying for a service. | | See how your comment ages in five years with kubernetes. | | I've absolutely seen disasters with open source that bleed | companies dry. Openstack is a great example. | sekh60 wrote: | OpenStack isn't a redhat project, it is contributed to by a | large number of players. Redhat, SUSE and Canonical all | provide their own OpenStack service subscriptions, but one | can easily deploy OpenStack with a project like OpenStack | Kolla. I run it at home and it is great. | Sanguinaire wrote: | OpenStack is an over-complicated mess. Most enterprise IT | orgs already struggle to hire the people with skills to | operate a public cloud account, nevermind having to build | the underlying infrastructure as well. | Spivak wrote: | I mean the value is really implementing AWS's API surface | area in a single control-plane overtop of the usual on-prem | tooling. K8s is kinda getting there but is too narrow in | scope I think for what people actually want. | thayne wrote: | For DynamoDB specifically, ScyllaDB had a Dynamodb compatible | mode. There are several options (open source and otherwise) | with s3 compatible APIs. I think it is more likely we'll end | up with replacements for individual services and maybe at | some point it will be possible to glue them together as a | replacement for a significant subset of AWS | belter wrote: | You can install DynamoDB locally although you will still miss | the rest of the stack. | | "Setting Up DynamoDB Local (Downloadable Version)" https://do | cs.aws.amazon.com/amazondynamodb/latest/developerg... | jdreaver wrote: | https://docs.aws.amazon.com/amazondynamodb/latest/developer | g... | | > The downloadable version of DynamoDB is intended for | development and testing purposes only. By comparison, the | DynamoDB web service is a managed service with scalability, | availability, and durability features that make it ideal | for production use. | belter wrote: | Correct. I meant to use it to save some money while | developing your apps, not as a solution for local | deployment. | jdreaver wrote: | Ah understood. I thought this comment thread was talking | about replacing _production_ AWS use cases with a bare | metal system that has a compatible API. | belter wrote: | LocalStack says its about testing: | | "...Whether you are testing complex CDK applications or | Terraform configurations, or just beginning to learn | about AWS services, LocalStack helps speed up and | simplify your testing and development workflow..." | unethical_ban wrote: | I think it would not be impossible for a talented, well- | funded group to pick the top 20% of AWS's services and then | implement them with the top 80% of features, with an API- | compatible layer. | | Even "just" to emulate a single region with 3 AZs would be | useful for testing and for prod for many people. I know of | several large organizations that to this day haven't scaled | their CI beyond one region. | 1vuio0pswjnm7 wrote: | Need more details. Matching AWS service for service would be | overkill I imagine, so what services does this need to have. Is | hardware virtualisation is must-have. | | The way I made systems/applications before AWS existed was to | create bootable images. I could already use a hypervisor, Xen, | because the OS fully supported both guest and host, all | virtualisation modes, before Linux did, and before AWS existed. | But because I am not a hosting provider I saw little need for | virtualisation. The OS I used also had "unikernel" capability, | before Docker, etc. existed. As it happened, this inexpensive, | self-determination was not to be the future, instead we got | "the cloud". Sharing servers with other "tenants". Less | expensive for the hosting provider, but more expensive and more | limited for the subscriber. No argument, the "limited" part has | improved since then, but it is still expensive (for continuous | use that is, spot pricing was a neat benefit of "the cloud"). | | Anyway, I "deploy" images to "local bare metal" which is a | laptop, RPi, or some other smaller form factor. I can use | unikernel for some kernel drivers in userspace. Basic | filesystem is embedded in the kernel, and larger filesystem is | on the USB stick filesystem in compressed format. Updates are | easy enough. I put two kernels on the USB stick, one is the | running kernel and the other is the update kernel. Same for the | larger filesystem which may contain servers and configuration | files. Can update or go back to last working | kernel/configuration by selecting one or the other in the boot | menu/renaming the larger compressed filesystem. | | Here is someone running a search engine out of his living room. | AKAIK, his setup survived a sustained HN thundering herd, hug | of death without a hiccup. | | https://news.ycombinator.com/item?id=28552805 | | The expense of AWS is an obvious point of discussion but | another one not mentioned here is _control_. When I create | images for bare metal I do not need to jump through any hoops | as I would in order to create an image that will run on AWS. | Nor do I need to fiddle with all the AWS knobs. There are no | silly marketing names for every program I run. I know the | system I am creating as well as I know the OS and the software | I choose. That is much better than how well I know every aspect | of AWS which just gets more and more complex every year. AWS | documentation is as cringeworthy as it it is voluminous. The | ever-increasing complexity of AWS, including the "tooling", | is, IMO, How To Create Lock-In 101 | pdimitar wrote: | I would absolutely _love_ reading you blogging in details | about this. It 's a lost art, and one I've ever been good at. | I want to lately, but people like you are always undercover | and don't document their knowledge. :( | simonh wrote: | I worked on a system called Quartz at Bank of America. It | wasn't an AWS clone, but was targeting the same flexibility for | the development and deployment of applications on | 'programmable' infrastructure. The team that built it had done | similar projects before at J. P. Morgan and Goldmans. It was a | set of infrastructure and services all accessed through Python | APIs, with YAML config, all stored in a single distributed | version control system. It was a lot of fun to work on. | whummer wrote: | Wow that sounds super interesting! If you can shoot us a | message to info@localstack.cloud we'd love to chat and learn | more about your journey! | [deleted] | bob229 wrote: | Which will never happen you child | [deleted] | salex89 wrote: | Ever looked into Nutanix (https://www.nutanix.com/)? | rezafuru wrote: | It's not just an amazing project, but also has an amazing | community around it! | whummer wrote: | Absolutely - couldn't agree more. This project would have never | been possible without such an amazing community. Kudos to you | all! | arpna wrote: | i got a chance to learn this tool and it's super useful tool for | developers during local development and e2e testing | arpna wrote: | i got the chance to learn this tool and it is super useful tool | for developers during local development and e2e testing | surfer7837 wrote: | I prefer mocking my AWS calls, so much more lightweight and means | exert developer doesn't have to install local stack | whummer wrote: | Interesting! Staying lightweight and zero-dependency is | certainly a good argument. Out of curiosity - which programming | language(s) are you / your team developing in? Do you have a | notion of how much (%) of your time, approximately, goes into | maintaining the mocks - does it scale / pay off over time? | qaq wrote: | Well if they use https://github.com/spulec/moto they don't | have to maintain the mocks | itake wrote: | Do you mock for local dev too? | jpgvm wrote: | I do. But I also primarily use JVM which means either high | quality mocks are already available or it's easy to mock them | out using Mockito or creating a fake implementation if I | want. | dafa wrote: | Super useful tool, we use it as part of our CI pipeline to test | our Lambdas, works great | gigatexal wrote: | Does a similar such thing exist for GCP? | reilly3000 wrote: | There is some coverage within the gcloud sdk: | | https://cloud.google.com/sdk/gcloud/reference/beta/emulators | | The firebase stack can run 100% local as well. | dkuebric wrote: | Here's a rundown of the local stack components we've found, as | well as some we've developed, for GCP at FullStory: | https://github.com/fullstorydev/emulators ___________________________________________________________________ (page generated 2021-10-10 23:00 UTC)