[HN Gopher] Ask HN: Does anyone else find the AWS Lambda develop...
___________________________________________________________________
Ask HN: Does anyone else find the AWS Lambda developer experience
frustrating?
Hey HN, I've been using AWS lambda a bit recently, mostly as a way
to glue together various bits and pieces. Maybe I'm doing this
wrong but does anyone else find the experience to be really
frustrating? I can unit test bits of the code just fine, but at
some point I always end up stuck in a slow feedback loop where I
deploy the code, do some manual invoking, go and dig through the
logs in CloudWatch, add another print statement in my lambda... and
so on. What I want is to run the lambdas locally, ideally more
than one, and then exercise them with streams of test events
(perhaps captured from a real environment). It would be quite cool
if I could define BDD style tests around them too. Anyone have any
suggestions or share my frustrations? I have heard localstack is
quite good although I haven't given it a go yet. Would that work
for me? I did try SAM but I was a bit underwhelmed and I don't want
to use a separate IaC tool for these. Alternatively, do other FaaS
providers solve this problem? Thanks for any help.
Author : tdfirth
Score : 114 points
Date : 2021-04-18 18:35 UTC (4 hours ago)
| pier25 wrote:
| You should check out Vercel.
|
| It uses AWS Lambda underneath but it makes the dev experience
| just awesome.
| tdfirth wrote:
| I thought Vercel was for building static sites/backends, or
| would it also work for generic AWS glue code? (e.g. run this
| lambda when the bucket changes).
| pier25 wrote:
| It's a combination of static + cloud functions (AWS Lambda) +
| CDN.
|
| AFAIK even though it's running AWS Lambda, the cloud
| functions can't be integrated directly with other AWS
| services.
| tonfreed wrote:
| Lambda works fantastically when you're working in AWS' little
| box.
|
| Processing kinesis streams, monitoring and munging S3 objects,
| auditing dynamo tables, etc. Web lambdas are not worth it imo,
| very complex and difficult to get right.
| popotamonga wrote:
| c# with lambas you just click 'play' locally and you can invoke
| debug step by step etc, could not be any easier. Which lang?
| tdfirth wrote:
| That sounds really neat, I must admit I've never written a
| single line of c# although I keep meaning to give it a go.
| Maybe this is my moment...
|
| I'm using python mostly atm. I've played around with go/node
| too (both of which I'm familiar with in and of themselves) and
| I was thinking about switching over to one of those already.
|
| The dependency management is easier for both of them than with
| python I found.
| klohto wrote:
| I personally like SAM, but there is
| https://github.com/dherault/serverless-offline as well
| iends wrote:
| "Think of the history of data access strategies to come out of
| Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET - All New! Are
| these technological imperatives? The result of an incompetent
| design group that needs to reinvent data access every goddamn
| year? (That's probably it, actually.) But the end result is just
| cover fire. The competition has no choice but to spend all their
| time porting and keeping up, time that they can't spend writing
| new features. Look closely at the software landscape. The
| companies that do well are the ones who rely least on big
| companies and don't have to spend all their cycles catching up
| and reimplementing and fixing bugs that crop up only on Windows
| XP." - Fire And Motion, Joel on Software
|
| In my own work with serverless I think about this a lot. In my
| day job, we built a new service and we pay AWS a few dollars per
| month to run. It would have cost us around $100 a month in AWS
| costs. However, the operational complexity is extremely high and
| we are also coupled to another service via Kinesis. For a billion
| dollar business, the trade off doesn't seem worth it.
|
| I wonder how much of serverless is just AWS firing at Google (and
| Heroku, DO, etc) and other competitors. It certainly hasn't made
| my life as a developer easier. It's certainly much cheaper for
| some use cases but the complexity of the system goes up very
| quickly, and you're end up having to manage a lot of that
| complexity using an inferior tool like Cloud Formation (or
| Terraform).
| rualca wrote:
| > It's certainly much cheaper for some use cases but the
| complexity of the system goes up very quickly, and you're end
| up having to manage a lot of that complexity using an inferior
| tool like Cloud Formation (or Terraform).
|
| The complexity bogeyman is a red herring. What is software
| development if not a constant fight between increasing
| complexity and adding features, including making systems more
| robust and resilient and reliable?
|
| Sure, you are free to claim that function-as-a-service paradigm
| is very complex because the code runs somewhere else, and you
| have to use a separate workflow than the one you're used to,
| and that you actually have to write unit tests to have some
| assurances that stuff does work as expected.
|
| But what would be the alternative approach?
|
| Would it be simpler to write a full-blown service that polls
| events and listens to message queues and launches background
| tasks? Obviously, no.
|
| AWS Lambda is a fancy way to let developers develop event-
| driven systems by just writing tiny event handlers and plug in
| events. It's no rocket science. In terms of complexity, they
| pale in comparison with even the tiniest hello world windows
| GUI app, or even Qt. I'm terms of web services, they are a
| service where you just have to write the request controller bit
| after that runs after all auth things passed and request
| messages were parsed and validated. Considering this, isn't it
| unreasonable to talk about increasing complexity, when
| everything was made.far simpler than what it would otherwise
| be?
| mattmanser wrote:
| It's like the SPA craze, microservices craze, NoSQL craze or
| similar things.
|
| Appropriate for a small amount of apps, but used
| inappropriately by a lot more for a while because it's the hot
| new thing.
| AlphaSite wrote:
| IMO SPA is the natural consequence of API 1st design, of you
| can do everything via the API why reimplement it all again
| for the UI, as much as it sucks in some ways, it can yield
| really good results.
| orf wrote:
| > I wonder how much of serverless is just AWS firing at Google
| (and Heroku, DO, etc) and other competitors.
|
| It depends on what you mean by "serverless". Serverless
| webapps? Sure, maybe a fair bit. But the real killer feature of
| the Lambda service is that it's a universal extension mechanism
| for other AWS services. Want to run some shoddy virus scanning
| whenever an object is added to a bucket for compliance reasons?
| That's a lambda. Want to manipulate Firehose records before
| they are sent to the sink? That's a lambda. Want to execute
| some crazy Python function from an Athena SQL query? You
| guessed it. That's a lambda.
|
| So when you say "It certainly hasn't made my life as a
| developer easier" it means you haven't needed to do these
| bespoke gluing together of stuff, because "being able to do
| this somehow" is infinitely easier than "not being able to do
| it at all because AWS hasn't exposed an API for this use case".
| salawat wrote:
| ...You realize you can do 90% of those usecases on your own
| hardware, with your own tools, within your own infratructure,
| with complete visibility into every bit without Amazon right?
|
| I think the true geniuses are the ones who spent so much time
| trying to sell people on "Server Admin is hard, lets hire it
| out!" as well as building the billing and metering systems.
|
| In a gold rush, screw panning, sell shovels (and pans).
| rualca wrote:
| > ...You realize you can do 90% of those usecases on your
| own hardware, with your own tools, within your own
| infratructure, with complete visibility into every bit
| without Amazon right?
|
| This statement is a red herring. People don't flock to AWS
| because they want to use lambdas. Lambdas are adopted by
| AWS customers who already have their data in S3 or DynamoDB
| and/or already use one of AWS message brokers, and happen
| to stumble on a requirement for a background that runs
| asynchronously in response to an event, such as someone
| uploading a file to a bucket.
|
| When that usecases pops up, the decision process consists
| of either adding a brand new service to be ran on ECS, add
| a background task on a service already being developed, or
| simply write the event handler and deploy it somewhere for
| AWS to run without us bothering about scaling and
| maintaining stuff. Writing and deploying the handler is an
| order of magnitude simpler to deploy and maintain, which
| happens to win over some technical discussions.
| chrisutz wrote:
| I've found lambda helpful for adding bits of code in other
| languages to your stack. Want to use a NodeJS library from
| your Python app? Jam the NodeJS in a lambda and don't worry
| about maintaining/scaling it.
| crummy wrote:
| So is the solution to avoid Lambda unless you have to? Don't
| use it to build regular software, just use it for AWS glue
| when necessary?
| orf wrote:
| That's the use case where it shines, but it can be an
| effective part of "regular software" as well.
|
| It's a hammer, and if it's a good fit for your particular
| nail is very dependent on it's shape.
| heavyset_go wrote:
| Yes. I only use Lambda when forced to by a client. 99% of the
| time it's a poor fit for their use case, anyway, but they've
| still got to have it to make someone happy.
|
| I feel as though there are similar amounts of friction across AWS
| offerings that make them unpleasant to work with in general, too.
| I'm numb to it now, but imagine my surprise when I actually went
| to use it for the first time after being hyped up about AWS.
|
| For these reasons, and because you pay a significant premium for
| network traffic on AWS, I never use AWS for personal projects and
| I'm very happy with that choice.
| dragonwriter wrote:
| > What I want is to run the lambdas locally, ideally more than
| one, and then exercise them with streams of test events (perhaps
| captured from a real environment).
|
| lambci has dockerized near perfect replicas of the lambda runtime
| environments, so as long as you aren't relying on VPC endpoints
| tied to specific lambdas, or things like that, you should be able
| to, even without SAM.
|
| But you can also: (1) deploy separate test copies of lambdas and
| fire test events at them the same way you would locally, which
| has full fidelity and no third-party deps. (2) throw test events
| at your lambda entry points locally and capture and validate what
| they return.
|
| I do both regularly with python lambdas (I find that the latter
| is adequate for testing lambdas and their downstream
| interactions, but the former is useful for validating
| interactions upstream, particularly--for most of the lambdas I
| work with--with APIGateway and some auth stuff we have hooked in
| there.)
| sonthonax wrote:
| Don't bother with Localstack, it's a bunch of Docker images
| combined with Python's Moto library.
|
| AWS has Step Functions, which is a vaguely functional and
| declarative language for linking your lambdas together. It's a
| bit verbose and annoying to write, but at least a it makes your
| lambada setup repeatable.
|
| The state machine which actually runs your step functions is
| publicly available.
|
| https://docs.aws.amazon.com/step-functions/latest/dg/sfn-loc...
| coredog64 wrote:
| Seconded that Localstack is pointless. So much of what I want
| to do isn't supported and/or throws weird errors that are
| nothing like the AWS service.
| gorbypark wrote:
| I am working on a side project (more of a way to learn go than
| anything) that makes "serverless" functions as simple to deploy
| as possible. Basically, a developer posts some JavaScript to an
| API endpoint, and they are returned a function id and a
| management token. The function can be run at URL.com/functionid
| and to delete or modify the function the management token must be
| used. It all runs in a V8 isolate (ala Cloudflare Workers). It's
| a self contained go binary, so it should be super easy to run
| locally, too.
| fifthofeight wrote:
| Have you tried using serverless/serverless-offline?
| https://www.serverless.com/
| cosmotic wrote:
| Our team has; it's been very difficult. Our jenkins build agent
| (running as a container) runs integration tests that start a
| node+serverless container, that starts a localstack container,
| that starts a lambda contianer.
| anfrank wrote:
| It's quite limiting on: 1. It only mocks API Gateway, and not
| accurately (ie. IAM permissions). You also need to use a bunch
| of other similar plugins to mock DynamoDB, SNS, SQS, etc. 2.
| You need to change your code (wrapping your aws-sdk to connect
| to localhost) to use these mocking plugins.
|
| I posted elsewhere in this thread but checkout SST
| (https://github.com/serverless-stack/serverless-stack), it
| lets' you work on your Lambda functions live without having to
| mock AWS services locally. Here's a short demo
| (https://youtu.be/hnTSTm5n11g)
| tdfirth wrote:
| I haven't tried it yet, no. It always struck me as being
| focused on making deployment quick/easy rather than on creating
| a good local dev workflow (i.e. the same workflow I currently
| have but a quicker cycle time). This never really appealed to
| me.
|
| Cosmotic's comment kind of confirms my suspicions to be honest.
| jppope wrote:
| Couple of notes I'd like to throw out there... - The
| Serverless Framework does help ease quite a few of your
| concerns, local development is made much easier (they have a
| node server for js development). The company working on it is
| kinda funny, but I still think it's a good tool. I currently
| use the raw lambda environment at work and can say that the
| serverless framework is a valuable abstraction. - I think
| Cosmotic is making their deployment setup more complicated
| than it needs to be. I obviously don't know their use case
| but there are easier ways to accomplish a CICD w/ FaaS... - I
| would recommend trying it. Its pretty painless, and you'll
| know within ~2-4 hours whether or not it will work for you
| notsureaboutpg wrote:
| Was going to post this.
|
| It's not a first-class experience, but you can get into a
| groove with it and for 90% of projects (which need sparing use
| of Lambda functions for specific purposes).
|
| As an example, I use Hasura for some of my backends, and it
| requires you to set up authentication somewhere else (e.g.
| Auth0). I usually set up a JWT based authentication module on
| AWS lambda that works specifically for the given project, and I
| use serverless to get that done pretty quickly.
| nindalf wrote:
| > do other FaaS providers solve this problem?
|
| Cloudflare's Workers have a good dev experience. It takes a
| second to build and locally preview a function. It live-reloads
| when you make a change. Later, it takes a second to build and
| deploy. On the flip side, Workers can't run arbitrary Docker
| containers. Your code has to be JS/TS or WebAssembly.
|
| I'm not affiliated with Cloudflare, just a satisfied customer.
| ignoramous wrote:
| > _On the flip side, Workers can 't run arbitrary Docker
| containers._
|
| Containers are coming to the Cloudflare edge:
| https://blog.cloudflare.com/containers-on-the-edge/ (inevitable
| really)
|
| Nit: Workers should really be compared to Lambda@Edge instead.
| tdfirth wrote:
| That's interesting, thanks for the recommendation. I have heard
| good things about cloudflare workers too and the local dev
| setup you've described sounds quite good.
|
| Am I right in saying that they serve a somewhat narrower use
| case than lambdas though? My understanding is that they are
| mostly used for things like modifying requests/responses rather
| than for building a whole backend.
| pier25 wrote:
| > Cloudflare's Workers have a good dev experience
|
| My experience has been the opposite.
|
| Wrangler doesn't give you any meaningful errors (since it
| converts JS errors to Rust) and it crashes way too often.
|
| Recently I worked on a Workers project and I had an undefined
| missing variable. The only error Wrangler would output is
| "error".
|
| IMO Workers are only viable for really small and simple
| projects.
| astashov wrote:
| I recently migrated from Cloudflare Workers to AWS Lambda +
| Dynamo for my relatively large pet project.
|
| It was surprisingly hard - the local development of lambdas is
| still very raw, documentation is scarce, and there are various
| small issues appearing here and there. I should probably write a
| blogpost about how to setup decent developer environment for AWS
| CDK and Lambdas, because there's not much on the Internet about
| it.
|
| I set up the whole AWS infrastructure via AWS CDK. I have one
| TypeScript file, that creates a stack with lambdas, dynamodb
| tables, API gateways, S3 buckets, wires up the secrets manager,
| etc - all of that in 2 environments - dev and prod.
|
| AWS CLI also can generate a Cloudformation YAML file from the CDK
| file (via `cdk synth`), which could be fed into SAM. So, I
| generate a `template.yaml` Cloudformation file this way, then run
| SAM like `sam local start-api`, and it runs exactly the same
| lambda locally, as in AWS, using that Cloudformation YAML file.
| SAM also supports live reload, so if you change any source files,
| it will automatically get those changes.
|
| So, okay-ish developer experience for lambdas is possible.
| There're caveats though:
|
| * I couldn't figure out how to use "nameless" Dynamodb tables
| (i.e. when CDK assigns the name to them automatically), because
| if I omit a table name in CDK template, then the local lambda and
| lambda in AWS assume different names for some reason.
|
| * Locally, the binary outputs don't work. It ignores
| `isBase64Encoded` by some reason, and lambda just returns Base64
| output, instead of binary.
|
| * And the main problem - local lambdas are SLOW. It seems like it
| restarts some container or something under the hood on each API
| call, which adds 2-3 seconds to each API call. So, the calls that
| should be like 30ms, are actually 2 seconds now. This is super
| frustrating.
| mkl wrote:
| What made you switch? How do Cloudflare workers and their
| development compare?
| astashov wrote:
| Pros of Cloudflare Workers - they're way simpler. Their docs
| are nice, it's straightforward to set up a dev environment
| for them, it's easy to set up a key-value store for them
| (Cloudflare KV), secrets management is simple. Pricing is
| nice - it's flat $5 a month, and you get access to almost all
| of it. It works fast and distributed across the whole world,
| admin console is nicely integrated with the Cloudflare CDN
| dashboard.
|
| But they're too limiting. Cloudflare Worker should be a pure
| JS script (that looks like a service worker script in a
| browser). You cannot use any Node libraries. In my app I need
| to generate images on the fly, so I couldn't figure out how
| to do that in a Cloudflare worker, so I ended up creating
| lambdas for those, and proxying the calls from Cloudflare
| worker to those lambdas.
|
| KV (their key-value store) is way too simple. No secondary
| indexes, no sorting, no querying (you can only sorta query
| list of keys, but then you have to fetch a record by each key
| individually). My app hit the limits of KV very quickly.
|
| Also, there's no monitoring or any good way to do backups for
| KV. So, it's almost unusable in a real-world production
| system, except as maybe a cache layer.
|
| Monitoring was a big pain for Cloudflare Workers. Like, even
| accessing production logs is weird - you have to "tail" them
| from your CLI. Even Cloudwatch is 100x better here.
|
| Development environment is okay, I guess, but the local
| server crashes constantly. Also, it kinda forces you to use
| Webpack and bundle the sources. It also doesn't have anything
| like layers in AWS lambdas, and they have a limit of 1MB for
| a bundle, which I almost hit (I had like a 900kb JS file), so
| that part got me worried too.
|
| Basically all of that made me worry I will hit a wall soon
| with Cloudflare Workers, so I moved to AWS. It's not piece of
| cake too, but at least I get a way to run proper Node
| scripts, a decent document database, continuous backups and
| pretty good monitoring out of the box.
| tqkxzugoaupvwqr wrote:
| I tried Cloudflare Workers and found these pain points:
|
| - local dev server ("wrangler dev") crashes often
|
| - local dev server executes code always on Cloudflare ->
| seeing changes takes a few seconds, you need an Internet
| connection
|
| - forces Webpack on you
|
| - managing KV store is difficult (How to back up? How to
| update schema of millions of entries?)
|
| - KV store is slow (400 ms in Europe) unless keys are being
| hit every couple of seconds to keep the edge cache warm
|
| Currently I'm trying an alternative - Deno Deploy - and
| love it so far. Local dev server executes your code
| locally, quick compile times, native TypeScript support due
| to Deno runtime.
|
| https://deno.com/deploy
| viraptor wrote:
| > Like, even accessing production logs is weird - you have
| to "tail" them from your CLI.
|
| Which seems to fail for access-protected sites since `tail`
| uses the same domain as the real endpoint. Also no error
| reporting - the exception may be in the logs, but it's
| missing the stack trace unless you log it explicitly.
|
| The monitoring side of workers is extremely poor currently.
|
| Also unit testing workers is hard to get right - there are
| no good mock libraries for request and kv which behave
| exactly like the production versions, so you end up with
| surprises after deployment.
| vinnymac wrote:
| I could have written your message word for word. I recently had
| a very similar experience moving to AWS Lambda using AWS CDK
| and SAM. It is pretty shocking how unhelpful the AWS
| documentation can be at times.
|
| Each use case is so unique, that it is likely difficult for
| them to account for every single situation where Lambdas are
| being used. Instead they opt for a generalized and non-specific
| documentation site, which results in docs I find practically
| useless.
|
| I frequently find solutions to problems for AWS on someone's
| obscure blog, who after many days of suffering stumbled onto
| the answer, rather than finding anything of use directly from
| those who wrote the software.
|
| Does a site exist where programmers can submit useful findings?
| Similar to stackoverflow, but rather than asking for questions,
| you just submit answers. I would think being able to search
| across such a crowdsourced utility would save programmers a ton
| of time.
| crummy wrote:
| > Does a site exist where programmers can submit useful
| findings? Similar to stackoverflow, but rather than asking
| for questions, you just submit answers. I would think being
| able to search across such a crowdsourced utility would save
| programmers a ton of time.
|
| SO kind of lets you do this. You can write a question and
| immediately answer it yourself - there's an option in the
| question UI for it.
| pensatoio wrote:
| Here are a few libraries that make testing lambdas (and
| simulating adjacent infrastructure like queues) very easy in
| tests.
|
| localstack
|
| mintel/pytest-localstack
|
| charlieparkes/boto3-fixtures
| tracer4201 wrote:
| I developed on Lambda with SAM and testing was pretty dang quick.
| IMO, Lambda works great for light weight event processing. I
| don't know your use case. I wrote some Java based Lambdas, and
| despite not being a huge Java fan, the Lambda experience was
| fairly easy for me. I was essentially processing events coming in
| from an SQS queue, performing some enrichment, and dropping
| things in other queues for downstream services to consume. Are
| you sure Lambda is the right solution for your problem?
| endgame wrote:
| Your comment basically says "I did it and it's great" but I
| don't see actionable advice for OP.
| tracer4201 wrote:
| Op asked:
|
| > Anyone have any suggestions or share my frustrations?
|
| I shared my experience and asked them a question. I at least
| contributed to the discussion.
|
| Your comment basically adds nothing to the discussion and is
| essentially cyber bullying.
| winstonchecksin wrote:
| Most of the top comments are saying how Lambda is half
| baked. It might very well be, but it's funny to see the one
| comment so far that is positive about Lambda get downvoted
| and the poster singled out for not providing a clear
| solution to ops problem. The HN hive mind disapproves of
| your experience!
| richardanaya wrote:
| Have you considered just using lambda to http server adapter?
| That way you can just write a normal http server and test against
| that locally as normal, then when you publish to a lambda, the
| adapter turns AWS lambda json to http requests?
| tdfirth wrote:
| I have considered it, yes, and there are a couple of projects
| that make the HTTP aspect of lambda easier, it's true. However
| running HTTP requests through lambdas is just one use case for
| them! For other kinds of events there doesn't really seem to be
| an equivalent.
|
| It would be great if there was a consistent way to do all of
| them.
| goodSyntax wrote:
| I don't understand the benefit of that. Could you explain
| please? It sounds to me like doing what you describe would
| prevent all the wins of serverless - scaling and only paying
| for the runtime you use. In what you describe, you get zero
| auto-scaling unless you implement it yourself, and you pay more
| than just lambda or just a http server. It sounds like adding
| another layer of complexity on top for no benefit
| frompdx wrote:
| I think the idea is to run a regular http server locally that
| forwards requests to your lambda handler function for the
| purpose of testing. When you actually deploy your lambda, you
| just hook up the lambda handler function and leave the http
| server out of the picture.
| i_v wrote:
| The best solution I've found for local testing is the Lambda
| Runtime Interface Emulator (RIE)[1]. It's basically a little Go
| wrapper[2] that runs your handler and makes it callable through
| an HTTP API. I considered writing a test harness around this for
| convenient integration testing but in my recent evaluation, I
| ended up avoiding Lambda (it's been really nice for small one-off
| tasks but was too expensive for the most recent and somewhat
| fringe use-case I considered using it for).
|
| [1]: https://docs.aws.amazon.com/lambda/latest/dg/images-
| test.htm... [2]: https://github.com/aws/aws-lambda-runtime-
| interface-emulator
| tdfirth wrote:
| Ah that's a good idea - I have actually played around with
| that. When they released support for running lambdas with
| container images I used it to have a go at making a custom
| lambda image.
|
| I didn't join the dots up for this use case though, thanks!
| jmb12686 wrote:
| That can be true if you are using tightly coupled event sources
| (like s3, SNS, etc) where you need to inspect the incoming
| object. If you are doing a HTTP / REST API, try to decouple the
| code as much as possible from the API Gateway invocation by using
| ExpressJS / aws-serverless-express to aid in local testing and
| debugging. Then testing and debugging locally becomes much
| easier.
| pc86 wrote:
| Is AWS a requirement? I've just started a project that is
| roughly... 75% Azure Functions and so far it's been a dream.
| Between the triggers and the outputs, and being able to run
| functions locally in VSCode I feel like it's the perfect way to
| "glue together various bits and pieces" as you said. Obviously a
| lot of the triggers are Azure-specific but between HTTP and
| Service Bus Queue/Service Bus Topic you should be able to do just
| about anything you need.
|
| On the plus side of having a lot of the pieces in Azure, if
| you're lucky enough to have an input and output match (e.g. you
| want to persist SB Topic messages into a SQL API CosmosDB
| instance) it's a half dozen lines of code and you're done.
| meursault wrote:
| Have you tried the recently added container support? There's a
| component provided called the Lambda Runtime Interface Emulator
| that exposes an API for the locally running container. In my
| experience this is miles better than the original approach.
| jollybean wrote:
| Yes, the AWS serverless is essentially half-baked for this reason
| it would be considerably more powerful - to the point of changing
| how we make apps - if there was a local environment provided by
| them that could really make testing and deployment seamless.
| Being able to see a debug of 'live' lambdas in your own cli etc.
| would be ideal.
| anfrank wrote:
| Wow, that is exactly what we are making. I posted elsewhere in
| this thread but SST (https://github.com/serverless-
| stack/serverless-stack) lets' you work on your Lambda functions
| live without having to mock AWS services locally.
|
| Here's a short demo (https://youtu.be/hnTSTm5n11g)
|
| I'd love for you to give it a try.
| jollybean wrote:
| That looks cool I will check it out.
| frompdx wrote:
| My tinfoil hat theory is that making local development
| difficult is the goal rather than a side effect. Why not just
| run the stack yourself if it's that easy? Better for Amazon if
| the development process for using their services involves as
| many other AWS services as possible.
| jollybean wrote:
| I wouldn't put it past them to do something like that if that
| were indeed going to work out, but the more reasonable thing
| is that they're just bad at it. Lambda was weak for years.
| Also, a 'local stack' for testing probably wouldn't cover
| nearly enough basis for people to 'just to it themselves'.
|
| Literally today I had a billing problem with AWS and it was
| one of the worst, broken UI experiences I've ever seen.
| Involving money to boot (usually those things have higher
| standards).
|
| They are really bad at quite a lot of things, it is what it
| is - there might be some rhyme or reason to it, for example,
| they might not care about documentation, or may be much more
| focused on bread and butter ec2, bit it's totally plausible
| that there are a lot of frayed edges.
|
| You know that difficult-to-attribute quote: 'Don't attribute
| to malice that which can just as easily be attributed to
| incompetence'.
| meritt wrote:
| There's a recent YC launch that attempts to solve this:
| https://news.ycombinator.com/item?id=26315585
|
| https://serverless-stack.com/
| redhale wrote:
| I completely agree, especially having just come from working with
| Azure Functions. This likely also applies to other runtimes, but
| definitely for dotnet, you can run all Functions in a solution
| locally via the default F5 debugging feature in Visual Studio.
| Stellar developer experience.
| flyinglizard wrote:
| It's astonishing how entrenched is the HN type in AWS that this
| no one said it before.
|
| P.S you can even attach the VS debugger to the live, deployed
| remote function and get full debugging (although quite slow).
| Don't forget to deploy with the Debug (rather than Release)
| profile if you wanna do that. Breakpoints and all!
| aerovistae wrote:
| I find all of AWS frustrating. Their interfaces feel like they
| were designed by an 80s IBM analyst.
| ufmace wrote:
| I use them heavily for work. IMO, they're pretty well designed
| for managing large numbers of servers with sophisticated
| coordination and management systems. If you just wanna run a
| single server, it's kind of ridiculously over-complex. IMO,
| Digital Ocean, Linode, etc have much better workflows for just
| spin up a single Linux server attached to this DNS and let me
| set everything up on it.
| StratusBen wrote:
| Time to check out https://vantage.sh/ :)
| majewsky wrote:
| It is considered impolite to plug your own stuff without
| disclosing your relation to it within the same post.
| StratusBen wrote:
| ...didn't know that - thanks for the heads up.
| dragonwriter wrote:
| > Their interfaces feel like they were designed by an 80s IBM
| analyst.
|
| The purpose of the AWS Console, AFAICT, is:
|
| (1) To be used by semi-technical ops and management types, and
| people doing tutorials, and
|
| (2) To drive everyone else as quickly as possible to the
| APIs/CLIs/SDKs.
| cpach wrote:
| Haha! Funny, and there is probably a kernel of truth in there.
|
| I've been thinking a lot about AWS lately and it definitely
| seems to me like a modern take on what IBM did in the 80's.
| bamboozled wrote:
| Ultra slow too, logging into the console is one of my least
| favourite things to do.
| tdfirth wrote:
| This made me laugh, thanks.
|
| If I'm honest, I do find AWS in general fairly high friction at
| times but it's way better than renting servers directly I
| think.
|
| It's the only cloud provider I've really worked with to be
| honest though (other than a few experiments), maybe some others
| are much better in this regard?
| mattmanser wrote:
| For me the Azure management portal is frustratingly slow, AWS
| is at least moderately responsive in comparison. I've not
| tried the Google stuff for years, but in my experience
| everything they do is ridiculously overcomplicated.
|
| I don't really get where you're coming from on renting
| servers directly, using AWS is far more expensive if you
| actually need a server for a while.
|
| Though I do admit being able to fire up a server for a couple
| of hours and then delete it is pretty awesome.
| tdfirth wrote:
| I wasn't referring to cost as, yes, it's definitely cheaper
| to just rent some servers.
|
| The operational overhead of provisioning them, configuring
| them (and so on) is really difficult for a lot of teams
| though.
| endgame wrote:
| Suggestions:
|
| 1. If you are building APIs and using Lambda functions as targets
| from an API Gateway API, look into libraries like serverless-wsgi
| (Python) or wai-handler-hal (Haskell) that translate between API
| Gateway request/response payloads and some kind of ecosystem-
| native representation. Then as long as you're writing code where
| all state gets persisted outside of the request/response cycle,
| you can develop locally as if you were writing for a more normal
| deploy environment.
|
| 2. Look into the lambda runtime interface emulator (
| https://github.com/aws/aws-lambda-runtime-interface-emulator...
| ). This lets you send invoke requests to a fake listener and
| locally test the lambda more easily. While the emulator is
| provided in the AWS container base images, you don't need to run
| it inside a container if you're deploying with zip files. (AWS-
| provided container images automatically enable the emulator if
| not running in a lambda runtime environment, and using docker for
| port remapping, which is nice but not at all required.)
|
| 3. Get really good at capturing all requests to external
| services, and mocking them out for local testing. Whether this is
| done with free monads, effect systems, or by routing everything
| through gateway classes will depend on your language and library
| choices.
| tdfirth wrote:
| Thanks for these suggestions, they seem pretty solid. I'm
| particularly keen on the idea of capturing requests/events for
| use in mocking.
|
| It would be quite a powerful workflow for running integration
| tests. I shall do some research!
| k__ wrote:
| There are many monitoring/observability services that help.
|
| - Dashbird
|
| - Thundra
|
| - Lumigo
|
| Cloud9 has direct Lambda integration, which helps too.
| dmlittle wrote:
| If all you need is the ability to run a lambda function's code
| locally you might interested in docker-lambda[1]. I haven't
| really used localstack or SAM but a couple of years ago when we
| needed to run some lambda functions locally for development
| docker-lambda worked well enough.
|
| [1] https://github.com/lambci/docker-lambda
| tdfirth wrote:
| Oh that does look quite interesting, I hadn't heard of that.
| Thanks for the tip.
|
| As you say, it only does the 'host lambdas locally' bit though.
| If you wanted to do stuff like test the interactions between
| lambdas, or test them over HTTP (for lambdas that act as
| components in REST APIs), you'd need to build something
| yourself.
| sonthonax wrote:
| If I were you, I'd write a little framework that abstracts
| these interactions. An HTTP call is an RPC call, and putting
| something onto SQS is just a queue.
|
| You're not testing that HTTP works, you're testing that the
| two functions work together. If you can abstract a calling
| interface between them with different backends you'll have a
| better time developing lambdas, while also not being tightly
| coupled to them.
|
| Create an RPC container object that has method stubs for the
| other lambdas you want to call. When you deploy it on AWS
| configure it to make an HTTP call; when running it locally,
| configure it to just call the code as normal.
|
| Just think of the HTTP call and JSON blob as the calling
| convention. The API is really simple.
|
| I think one of the biggest mistakes people make when
| developing Lambadas is designing them differently to normal
| software.
| tdfirth wrote:
| > I think one of the biggest mistakes people make when
| developing Lambadas is designing them differently to normal
| software.
|
| That's probably very true (and I'm probably quite guilty of
| it!).
|
| Thanks for your suggestions, that does sound like a good
| approach.
| joshuanapoli wrote:
| My company uses separate stages (through serverless framework)
| for each branch of feature development. CI uses these to test,
| running within AWS infrastructure. It works well for isolating
| test environments. We lean more on unit testing to debug the
| service with external dependencies mocked. We try to avoid
| situations where we are adding log statements one-by-one;
| CloudFormation deployments take several minutes, so iterating
| small changes is indeed frustrating. We thought about localstack,
| but haven't tried it. I'm worried that it will increase our IaC
| complexity; everything will have to deploy to both AWS and also
| localstack.
| anfrank wrote:
| Yeah I wouldn't go down the local stack path. IMHO, you should
| develop/test against real AWS services, and it falls on the
| tooling to make that experience fast/smooth.
|
| I posted elsewhere in this thread but checkout SST
| (https://github.com/serverless-stack/serverless-stack), it
| lets' you work on your Lambda functions live.
|
| Think of it like your Lambda functions are Live Reloaded as you
| make changes to your code. You don't need to mock AWS services
| locally, and you don't have to wait for CloudFormation to
| deploy your changes.
|
| Here's a short demo (https://youtu.be/hnTSTm5n11g)
| swyx wrote:
| > Alternatively, do other FaaS providers solve this problem?
|
| I worked on Netlify Dev, which does offer a simulated lambda
| environment for testing locally:
| https://news.ycombinator.com/item?id=19615546
|
| I was also quite disappointed to see that AWS Amplify did not
| offer a running dev server for lambdas, best you could do was a
| local onetime manual invoke and even that often failed
| (mismatched to the live lambda environment where it succeeds)
| tlarkworthy wrote:
| Part of my frustrations with FaaS lead me to develop serverless
| cells where there is no deploy step.
|
| https://observablehq.com/@endpointservices/serverless-cells
|
| Source code is loaded dynamically each invocation.
|
| Still, some work todo, it's still annoying to program. Next hope
| to make them easy to use is pipe the runtime to Chrome devtools
| TruthWillHurt wrote:
| We're using it "the right way" with SAM, but still very
| frustrating.
|
| We had to detect if running locally or deployed in our code
| because Cognito auth was giving us trouble when running locally,
| and Step Functions can only be tested deployed.
|
| Cloud Watch logs take several minutes to appear, funcs take some
| time to package and deploy (even with caching) - making feedback
| loop very slow.
|
| I personally think the best way is to develop serverless funcs
| _on platform_ - using the built in editor. You get logs
| instantly. But can 't get devs to give up their IDEs...
| jppope wrote:
| +2x - The logs are funny. they'll duplicate, show up late, etc
| sharms wrote:
| I share these frustrations and don't have a great answer. When
| integrating N cloud services, at some point it gets incredibly
| hard to build out fast feedback / local environments. The best
| way I know how to deal with it is use less services and ensure
| the ones you stick with are simple.
| tdfirth wrote:
| I am not alone then at least! The particularly annoying thing
| is that I feel like N isn't even that high. Even 2 or 3 things
| interacting is a pain (unless it's stuff like RDS or just a
| plain old container of course).
| kaishiro wrote:
| It sounds like you're committed to Lambda, but we've spent
| extensive time across both Lambdas and Google Cloud Functions,
| and my off the cuff, somewhat biased opinion is that GCF gives a
| significantly better development experience the Lambdas. I note
| biased because a majority of the work we do lives in
| RTDB/Firestore.
|
| However, in response to your specific pain point - after
| initializing a new functions project you can immediately run an
| 'npm run serve' and have local execution firing for any
| scaffolded endpoints. These - by default - will connect to a
| project associated RTDB/Firestore instance without the need for
| additional service account/IAM configuration.
|
| I've always enjoyed that this was a part of the mainline dev flow
| - and it removes a huge barrier of entry for me (mostly mental)
| to spin up little toy projects or test endpoints whenever the
| thought crosses my mind.
| wussboy wrote:
| This response surprises me. I find Google Cloud Functions to be
| orders of magnitude worse than Lambdas. Everything that you
| deal with in AWS with the added bonus that instead of taking 5
| seconds to deploy it takes 5 minutes. I would love to know
| better options. How do you deal with that horrible deployment
| time?
| atraac wrote:
| We run Azure Functions with C#, it's not perfect but it's very
| easy to test, runs with a single click locally, deploys fairly
| easily to Azure as well(though it has some quirks) can use 99% of
| our codebase as a dependency without any issues.
| andreineculau wrote:
| I've been in the AWS world for 4+ years now and my immediate
| feedback is don't run any local emulators. None! Write unit tests
| for the internals and test your system end-to-end. I say that
| both because AWS services can have unpredictable behavior that
| you need to account for but also because local emulators are at
| best functional, but in reality far from emulating the AWS world
| on a 1:1 scale (especially the unpredictable behaviors i
| mentioned). So instead optimize for many local unit tests and
| many live end-to-end tests (implies many deployments and parallel
| environments prod, staging, dev, etc)
|
| When it comes to Lambdas, given the reasons above, there's only
| one thing that can improve the experience: PROXY. Before i went
| on parental leave i had the idea is creating a proxy lambda which
| can be configured with an IP and port number. That IP and port is
| for your local dev environment. This way, when developing you can
| instruct a live system to short-circuit and to proxy calls to a
| local lambda available om the respective port. Trigger end-to-end
| tests by invoking AWS services that will eventually call the
| proxy lambda, and then your local lambda with the same
| environment/context/input, reply with output which will reach the
| proxy lambda, which will output the same content forward.
| AlphaSite wrote:
| Sound a little like Tilt, Skaffold or build packs?
| joshribakoff wrote:
| More like telepresence than skaffold
| andreineculau wrote:
| I don't know them in depth but yes, I'd also go for
| telepresence (which is smth i discovered after i had my
| evrika moment and so i started googling for similar ideas
| or maybe even implementations)
| tdfirth wrote:
| To check my understanding, you basically deploy a place holder
| lambda which proxies events to your dev machine?
|
| That's pretty neat if so, I might have to try that out. Are
| there any downsides to this approach do you think?
|
| Thanks for your thoughts on emulators/tests too.
| andreineculau wrote:
| Correct. I'm very pedantic and usually I can't help but find
| flaws in all proposals :)) but I really can't think of
| anything here. I can see limitations though: if you use
| lambda layers or native code specifically for the AWS
| architecture etc, then you need to account for that when
| proxying but... It's not impossible. You also need to account
| for having a public IP or use a service that exposes a public
| ip/port and redirects to your local dev machine. The idea is
| not super simple in practice but it is very simple compared
| to the emulators and it gets you very close to 1:1 parity for
| all those corner cases.
| tdfirth wrote:
| Ha, I'm glad my simplification did not fall too short!
|
| I did wonder with regards to the IP address. If you had a
| VPN set up for access to a VPC and ran the lambda in that
| VPC then that would be another option I think. You would
| just need to configure a static IP for your dev machine
| within the private network.
| andreineculau wrote:
| Right! I haven't explored that thought. My team kept
| everything serverless so we touched zero VPC/etc. From a
| generic perspective, having a setup to proxy into a LAN
| is more valuable I think - i can imagine for instance how
| my team had access to certain AWS services, but not all,
| erc etc. But these are details - the idea would be worth
| exploring
| pavel_lishin wrote:
| > _and test your system end-to-end._
|
| How do you automate this?
| andreineculau wrote:
| Living in my team's bubble i thought everyone runs or tries
| to run parallel environments: prod, staging, dev, but also an
| individual (person) or feature env. Why? Because there's no
| emulator or documentation that will teach you real behavior.
| Like others have said, AWS seems out of this world. Just like
| GCP and Azure i might add. Some things you don't expect and
| they mesmerize you how smart they are. Some you expect and
| you can't fathom how come you're the "only" one screaming.
| Random thought: this is how i ended up logging all I bumped
| into into "Fl-aws" https://github.com/andreineculau/fl-aws
|
| Back to the point: reality is that many build their AWS
| environment (prod) manually, maybe they duplicate once (dev)
| also manually, maybe they use some automation for their
| "code" (lambda) but that's it. This implies it's practically
| impossible to run end-to-end tests. You can't do that in prod
| for obvious reasons and you can't do it in dev either - you
| have many devs queueing, maybe dev is not in sync with prod
| etc.
|
| My team ran cloudformation end-to-end. We actually
| orchestrated and wrapped cloudformation (this is yet another
| topic for not using terraform etc) so that if smth couldn't
| be done in CFN, it would still be automated and reproducible.
| Long story short, in 30 minutes (it was this long because we
| had to wait for cloudfront etc) we had a new environment,
| ready to play with. A total sandbox. Every dev had their own
| and it was easy to deploy from a release artifact or a git
| branch to this environment. Similarly you could create a
| separate env for more elaborate changes to the architecture.
| And test in a live environment.
|
| Finally to your question: how do you test end-to-end?
|
| If we talk about lambdas because that's where the business
| logic lies in a "serverless" architecture, then the answer is
| by calling the system which will eventually call your
| lambda/s along the way. If your lambda ia sitting behind AWS
| gateway, then fire an http request. Is it triggered when
| objects land on S3? Then push some object to S3. How do you
| assert? Just the same - http response, S3 changes etc. Not to
| mention you can also check cloudwatch for specific log
| entries (though they are not instant).
|
| With this type of a setup, which sounds complex, but it is
| not since it is 100% reproducible (also from project to
| project - I had several), adding this proxy-to-my-dev-machine
| lambda would mean I can make local changes and then fire unit
| AND end-to-end tests without any changes pushed to AWS, which
| is the main time/energy consumer imo.
|
| PS: sorry for the wall of text. Like i said i recently
| realized that the development realities have huge
| discrepancies, so i tried to summarize my reality :)
| anfrank wrote:
| SST is an Open-Source framework worth checking out if you are
| looking for a lighting fast local dev environment for serverless.
| https://github.com/serverless-stack/serverless-stack
|
| I often see ppl adopt one of two development patterns:
|
| 1. Locally mock all the services that your Lambda function uses.
| Like API Gateway, SNS, SQS, etc. This is hard to do. If you are
| using a tool that mocks a specific service (like API Gateway),
| you won't be able to test a Lambda that's invoked by a different
| service (like SNS). On the other hand a service like LocalStack,
| that tries to mock a whole suite of services, is slow and the
| mocked services can be out of date.
|
| 2. Or, you'll need to deploy your changes to test them. Each
| deployment can take at least a minute. And repeatedly deploying
| to test a change really slows down the feedback loop.
|
| SST lets you develop your Lambda functions locally while
| connecting to others AWS services without needing to mock them
| locally. Behind the scene, the lambda requests are streamed to ur
| local, runs ur local function, response streamed back to Lambda.
| So you can iterate on ur function code without redeploying.
|
| Think of your Lambda functions are Live Reloaded. You can see it
| in action here -- https://youtu.be/hnTSTm5n11g
|
| I'm the core maintainer of the project. And folks in our
| community are saying using SST feels like "working in the future"
| ;)
|
| I'd love for you to give it a try. We are also a part of the YC
| W21.
| tdfirth wrote:
| Hey, thanks for the link. It does look quite good - there was
| another comment where someone had a home-brewed set up that did
| something fairly similar.
|
| I'll check it out!
| anfrank wrote:
| Awesome! We've got a super passionate community at
| https://launchpass.com/serverless-stack
|
| You can also direct any questions/feedback to frank@anoma.ly
| :)
| davmar wrote:
| I use this companies services (seed.run as well). Frank and Jay
| are doing something truly special in this space.
|
| Also, this guide they put together is definitive:
| https://serverless-stack.com/#guide
|
| Had I found that years ago, I would have saved SO MANY
| headaches. I've implemented local debugging (hell, and doesn't
| actually replicate AWS). I've followed the CloudWatch logs,
| just like you, painful.
|
| The SST local debugging with lambda is the best way forward.
| Deploying micro services on seed.run is also the way forward.
| kerryritter wrote:
| I typically write my Lambda functions as Nest apps, develop
| locally, and then deploy with the aws-serverless-express (or
| fastify) NPM package, and I enable The ability to trigger a
| kinesis/s3/etc events locally as well. The fact that my app gets
| deployed to Lambda doesn't really have any impact on my developer
| experience. What stops you from working locally?
| timdev2 wrote:
| This has worked very well for me, too. I suppose it might start
| to break down if you're using some of the more complex AWS-
| specific offerings.
|
| But if your function more or less resembles a little web app
| that maybe talks to S3 and DynamoDB, this works really well.
| infinityplus1 wrote:
| Firebase Functions has a local emulator and a unit testing
| module. The emulator is quite good for testing your code locally
| quickly.
___________________________________________________________________
(page generated 2021-04-18 23:01 UTC)