[HN Gopher] Pulumi 3.0
___________________________________________________________________
Pulumi 3.0
Author : bovermyer
Score : 176 points
Date : 2021-04-20 20:14 UTC (2 hours ago)
(HTM) web link (www.pulumi.com)
(TXT) w3m dump (www.pulumi.com)
| kjjjjjjjjjjjjjj wrote:
| I just don't get why someone would use Pulumi
| bloopernova wrote:
| Can someone who has used both Pulumi and AWS CDK describe the
| differences between the two?
|
| I am using Terraform 100% now, but sometimes wish I had more than
| the HCL (hashicorp configuration language) syntax available to
| use in my code.
| aequitas wrote:
| You could try Terraform CDK[0] which supports TypeScript,
| Python, Java, and C#.
|
| [0] https://github.com/hashicorp/terraform-cdk
| k__ wrote:
| The CDK synthesizes CloudFormation templates (for AWS) and
| Pulumi seems to use the AWS APIs to provision and doesn't touch
| CloudFormation.
| bloopernova wrote:
| Right - I was wondering about the developer "experience"
| between the two.
| carlosf wrote:
| I migrated from Terraform to AWS CDK and I'm very happy.
|
| The good:
|
| - Tooling for any major language (Typescript, Python)... is
| lightyears ahead of anything you will find for Terraform / HCL.
|
| - You can write your code as declaratively as possible (like
| you would do using TF), but you always have the escape hatch of
| using all libs available to your language of choice.
|
| - AWS CDK uses CloudFormation under the hood, so you get cool
| stuff like automatic rollbacks in case of failure.
|
| - You have access to much more mature testing frameworks,
| compared to what is available to TF. Because AWS CDK synthetize
| CF templates, you can also snapshot those for regression
| testing. Applying good software engineering practices is
| overall much easier. Most languages are much easier to extend
| than HCL.
|
| - AWS is constantly releasing higher level libraries so you
| don't need to fiddle with low level API details. When using
| Terraform, you generally must understand your cloud provider
| API at a very fine level to implement IaC.
|
| The bad:
|
| - AWS CDK is written in Typescript and automatically translated
| to other languages. Even though you can use Python, C#, etc...
| Most examples and tutorials exist in Typescript. Also, you
| always must install npm + cdk + the libraries in the language
| you are using, so using any language other than Typescript
| means supporting two toolchains, which is a pain in the ass. I
| started with AWS CDK in Python and now I'm migrating to
| Typescript.
|
| - Some modules only exist for Typescript.
|
| - Since CF is used under the hood, it only supports resources
| supported by CF. Sometimes CF support takes a while. Since TF
| is just a wrapper for a Cloud Provider API, it generally
| implements new resources much quicker.
|
| - It has its own vocabulary and and ways to do stuff (L1/L2
| Construct? Stack? Retention Policy?)... even if you've used a
| lot of TF and know AWS very well, there's a learning curve.
|
| Some cool videos:
|
| - An AWS dev shows how to create your own construct and test it
| (that's the less "toy example" tutorial I have found):
| https://www.youtube.com/watch?v=cTsSXYOYQPw
|
| - Same guy shows how to contribute to AWS CDK. I've found that
| looking at their source code is a great way to learn about good
| practices and patterns:
| https://www.youtube.com/watch?v=OXQSSibrt-A
| satya71 wrote:
| CDK maintains the state with AWS itself, Pulumi operates their
| own state service (with option of self management). I have
| never used CDK, but I've heard that it's hard to recover from
| state corruption in CDK.
| jen20 wrote:
| While this is true, the reason is not called out: CDK simply
| generates a CloudFormation Template, and thus uses
| CloudFormation for state management, rather than using the
| cloud APIs directly.
| nhoughto wrote:
| CDK is just a different way of writing cloudformation
| essentially, its code that is transformed into cloudformation
| templates, so inherits all its problems.
|
| Pulumi is more like Terraform but without HCL. So imagine
| instead of having to find-the-HCL-way-to-do-things and work
| within those constraints to make terraform work, you can use
| one of a few languages to build up resources (similar to you
| would in HCL) and have pulumi provision/act on them for you
| including managing their state.
|
| Quite different to CDK/CloudFormation and much more like
| Terraform without the handcuffs of HCL. (sometimes handcuffs
| are useful tho..)
| mdeeks wrote:
| In addition to what others have said, it is important to point
| out that the CDK is AWS specific. Your "infrastructure" is
| rarely just "AWS" even if you're not multi-cloud. Terraform and
| Pulumi can operate on things outside of AWS like Github, Okta,
| Cloudflare, Datadog, Hashicorp Vault, etc.
|
| It is nice to have one system or even repo for managing all of
| that.
|
| https://www.pulumi.com/docs/intro/vs/cloud_template_transpil...
| cmclaughlin wrote:
| > In addition to what others have said, it is important to
| point out that the CDK is AWS specific.
|
| As mentioned in other comments here, CDK for Terraform is
| another option and it's not AWS specific - it should work
| with any Terraform provider and there are heaps available.
|
| https://github.com/hashicorp/terraform-cdk
| mdeeks wrote:
| Oh interesting (and kind of weird). I didn't know this was
| an option. Though it does say this at the top:
|
| > This experimental repository contains software which is
| still being developed and in the alpha testing stage. It is
| not ready for production use.
| simtel20 wrote:
| It also seems to make using modules awkward or
| impossible, making non trivial use of the terraform cdk
| impractical.
| rossmohax wrote:
| I used both in depth. CDK as it is now is a wrapper around
| CloudFormation and therefore suffers from same limitations: AWS
| only, 'static' templates.
|
| Pulimi on the other hand is closer to terraform, except you can
| write in a language which actually doesn't stand in your way
| and allows you to programmatically access values available
| during execution only. With CDK you can only reference them in
| CF template (lets say instance ID which you just created), but
| these are never "lifted" to your code. Also CDK misses native
| 'datasources' and offers limited mechanism for lookups.
|
| Pulumi also has a killer Automatioin featre, where you can code
| infrastructure migrations, not unlike you'd do it for SQL.
| Neither TF nor CDK allows you to do that, and you'd need to
| code infrastructure state transitions in error prone bash
| wrappers.
|
| As a developer, I quite liked Pulumi resource model,
| documentation and team responsiveness on Github and slack
| channel. I am working with CDK now, because of costs and I'd
| prefer Pulumi if there was a choice.
| tasssko wrote:
| When i looked the SDK was only available in certain languages
| which was a issue for me at the time. I just checked and they
| have python and csharp. Pulumi has potential to reduce the
| verbosity of infrastructure configuration and enable
| engineers to tailor it more to the application on demand.
| Adding a new queue can be done as part of a app release and
| not handled in a separate pipeline. It isn't fun to wrangle
| infrastructure pipelines and when necessary orchestrate
| changes.
| ncmncm wrote:
| Imagine, a release announcement containing a sentence that
| bothers to say what the thing being announced is.
|
| They will surely lose points for reduced brevity.
| methyl wrote:
| Native Provider means it no longer uses Terraform under the hood
| and it excites me a lot. We were having hard time with some more
| complex setups and it seems it can be remedy for issues that we
| faced. Stoked to check it out!
| leg100 wrote:
| According to [1], "Native providers are built directly and
| automatically from the cloud provider's API...", and are not
| hand-coded, unlike Terraform's providers.
|
| I'm surprised this is possible. If it was, why didn't Terraform
| follow this approach a long time ago?
|
| The cloud providers' APIs provide the endpoints for creating,
| updating, reading and deleting resources. Terraform and
| Pulumi's value is to provide an _idempotent_ abstraction on top
| of that. And that abstraction is not straightforward to write,
| because it has to handle numerous nuances and anachronisms in
| the underlying APIs. For example, in the event of updating an
| existing storage bucket, the abstraction has to determine
| whether the bucket can be simply updated, or if it needs to be
| re-created (say if you were changing the name or location). And
| the underlying API will not necessarily reveal this kind of
| information.
|
| Hence one would think completely avoiding handwritten code is
| incredibly difficult if not an insurmountable problem.
|
| [1]: https://www.pulumi.com/blog/pulumiup-native-providers/
| k__ wrote:
| Wouldn't this mean that Pulumi has better coverage than
| CloudFormation?
|
| Some things in the AWS API aren't (yet) available in CFN.
| leg100 wrote:
| It would mean Pulumi has better coverage than CFN _and_
| terraform. Indeed as their blurb states, Pulumi would have
| 100% coverage.
|
| But I'm skeptical.
| antoncohen wrote:
| The Terraform provider for Google Cloud uses partial
| autogeneration, here is the repo that does the autogeneration
| for multiple automation tools:
|
| https://github.com/GoogleCloudPlatform/magic-modules
| catern wrote:
| I hope Pulumi eventually develops a "bare-metal" mode. I'm very
| much in favor of using a real programming language to set up
| systems[0], but I don't like to use cloud services beyond basic
| hosting. It would be nicer if Pulumi provided, as a library, code
| to launch and maintain self-hosted databases, proxies, etc,
| rather than just providing clients for cloud APIs.
|
| [0] http://catern.com/progsys.html
| octopoc wrote:
| IIUC it already does this. Many of the libraries Pulumi exposes
| are not specific to any cloud platform. Examples:
|
| - Docker: https://www.pulumi.com/docs/reference/pkg/docker/
|
| - Postgresql: https://www.pulumi.com/docs/intro/cloud-
| providers/postgresql...
|
| - Keycloak: https://www.pulumi.com/docs/reference/pkg/keycloak/
| wpietri wrote:
| Do folks have opinions on how Pulumi compares to Terraform CDK?
|
| I'm the only ops-y person in my group. I've done things so far
| with Terraform HCL, but a) it's not a great language, and b) I
| can't really ask other folks to learn it. But they all know
| Python, so my theory is that if I can wrap our ops stuff up in
| familiar-looking code, they'll be able to work with it
| effectively.
|
| Is one of these toolkits better than the other? I'm inclined to
| go with Terraform CDK just because they company's further along,
| but I'd love other people's takes.
| cmclaughlin wrote:
| > Do folks have opinions on how Pulumi compares to Terraform
| CDK?
|
| I last tried Pulumi around the 2.0 release and it flat-out
| didn't work for me. After working through a few bugs with their
| tech sales I gave up on it. I'm guessing their TypeScript
| support is better than their Python - for me it just wasn't
| ready for production.
|
| On the other hand I've been using Terraform CDK in production
| since the early alpha releases. Had to work around a couple of
| bugs here and there, but no blockers at all. It's been a game
| changer for my team and we're really enjoying it!
| wpietri wrote:
| Thanks! Very helpful.
| andrewSC wrote:
| "Self-hosting is only available with the Pulumi Enterprise
| Edition. Contact us if you would like to evaluate the Self-Hosted
| Enterprise Edition." [0]
|
| [0] https://www.pulumi.com/docs/guides/self-hosted/
| spooneybarger wrote:
| You don't need to use the service, although it solves a lot of
| problems. You can use only the Pulumi SDK and then solve the
| various state management issues that their service provides in
| another fashion.
|
| I think its a reasonable business model. The SDK is completely
| open and free. The service is how they make money .
| daralthus wrote:
| Just to elaborate:
|
| State is a snapshot of what is deployed at a given time, it
| is used at the next run to compare if there were any drifts
| since, eg.: "Did anyone delete a server manually?" etc.
|
| By default it is stored with Pulumi, but if you are on a
| budget you can just use S3 with a few tradeoffs (concurrent
| deployments would need a lock that you implement yourself):
|
| https://www.pulumi.com/docs/intro/concepts/state/
| delnaught wrote:
| In my quick read, I didn't see a way not to use the service.
|
| "If this is your first time running pulumi new or most other
| pulumi commands, you will be prompted to log in to the Pulumi
| service. The Pulumi CLI works in tandem with the Pulumi
| service in order to deliver a reliable experience."[0]
|
| [0] https://www.pulumi.com/docs/get-
| started/kubernetes/create-pr...
| spooneybarger wrote:
| https://www.pulumi.com/docs/intro/cloud-
| providers/kubernetes...
| thearrow wrote:
| I think Pulimi is similar to terraform in that you can use
| multiple backends for state management, of which their
| "service" is one[0]. Meaning you could use the CLI backed
| by an S3 bucket for free.
|
| [0]:
| https://www.pulumi.com/docs/intro/concepts/state/#backends
| marcinzm wrote:
| Looks like Pulimi, unlike terraform, doesn't provide a
| means of locking for concurrent access if you use the
| self-managed state.
| delnaught wrote:
| On its way? [0] Not sure how to tell if / when this PR
| makes it into 3.0
|
| [0] https://github.com/pulumi/pulumi/pull/2697
| thearrow wrote:
| Looks like this feature may have made it into v2.23.0,
| behind an env var: https://github.com/pulumi/pulumi/blob/
| v2.23.0/CHANGELOG.md#i...
|
| Nice!
| antoncohen wrote:
| > Open Source SDK
|
| > The SDK is a CLI and collection of libraries for defining and
| deploying cloud apps and infrastructure in code.
|
| https://www.pulumi.com/pricing/
|
| With Terraform the open source CLI and libraries for defining
| and deploying infrastructure are fully usable, even for large
| systems.
|
| What I would like to know is, how usable Pulumi is without a
| paid subscription?
|
| I thought when I first looked at Pulumi, the only non-local
| state backend was the paid Pulumi Service. But looking at it
| now, they seem to support the normal object store backends that
| Terraform does
| (https://www.pulumi.com/docs/intro/concepts/state/). Though the
| talk at lack of concurrency control seems to imply they don't
| support locking like Terraform does.
| Jgrubb wrote:
| Just curious, why'd you post that bit?
| all2 wrote:
| A big appeal of "open source" is control.
|
| Not allowing self-hosting for non-paid tiers is cutting
| control off from the user and moving it to the producer.
| xyzzy123 wrote:
| This is just the state storage backend. You can use Pulumi
| cloud or keep your state in a bucket, or write your own
| backend.
|
| Terraform cloud is a similar deal.
| nine_k wrote:
| Because it says: <<the next major version of the Pulumi open
| source project>>. Apparently, either not very complete, or
| not very open-source, if self-hosting requires a license.
| jacques_chester wrote:
| I suspect the key words are "Enterprise Edition". There
| will be features that matter to larger customers that
| matter less to others.
| fock wrote:
| just because I don't get it from their marketing: their API is a
| cloud-service wrapping other cloudservices. Or is it a classical
| library with wrappers?
| spooneybarger wrote:
| The Pulumi cloud offering provides state management.
|
| With it, you can bring up some infrastructure on machine A and
| tear it down on another machine.
|
| Without the service, you can only tear down on the same machine
| you brought it up. Ditto for adding to existing infrastructure.
|
| You don't need their service but you have to solve problems
| like knowing what is running, replicating the needed state etc.
|
| The service is very nice and makes it easy to know what is
| running, how long it has been running, who created it etc-
| across clouds. But, it is not strictly needed. As an individual
| user, you could get away without using. For a team, it fills a
| very vital gap.
| aChrisSmith wrote:
| It's a bit more involved than that...
|
| Suppose you wrote a program to go and create your cloud
| infrastructure. Using AWS' APIs you write an app to spin up a
| VM, setup a load balancer, and provision a database. You run
| your program, it works like a charm. Now you have all the
| infrastructure setup just the way you want it!
|
| The problem is what happens when you want to _update_ that
| infrastructure? The program you wrote, using the
| aws.CreateVM(..) and aws.CreateLoadBalancer(..) API calls is no
| longer applicable. Instead, you probably want to use a
| different set of APIs, like aws.UpdateVM(...), to update your
| existing cloud resources. For example, change some port
| settings on your load balancer. So your app needs to be smart
| enough to check if the resources already exist, and if so,
| update them. Otherwise, create them fresh.
|
| And it gets even worse. What if you want to create some new
| resources, such as attach an SSL certificate to your load
| balancer... but still keep all of your existing infrastructure
| as-is. Or what if you want to update an IAM usage policy that
| is already in-use by several other resources... Somehow your
| app needs to know the impact of that change, and how it will
| ripple out across other cloud resources.
|
| Does that start to make sense? You don't really want a
| "wrapper" for cloud APIs. You really want something that allows
| you to effectively describe your cloud infrastructure, and
| "make it happen". And leave the specifics of "how" as an
| implementation detail... accomplished by a cloud provider's
| APIs.
|
| That is what Pulumi does -- and other Infrastructure as Code
| tools, like Terraform. It provides you a way to describe your
| cloud infrastructure in a programming language, so that every
| time you run your app it will make the cloud reflect that
| target state. It will:
|
| - Create resources if they don't exist. - Update existing
| resources if they do. - Delete any resources that you no longer
| need.
|
| I work at Pulumi and am happy to go into details about the joys
| of not dealing with cloud APIs directly, and just using Pulumi
| :)
| pgroves wrote:
| Thanks for saying this ^^, it's the best explanation of what
| this thing actually is (sorry but I didn't really get it from
| the homepage). Define a state of infrastructure using a real
| sdk (not yaml files) and it can figure out and apply the
| migrations from the current state to the new state. (right?)
| aChrisSmith wrote:
| Yup!
|
| And in addition to making it easier to manage cloud
| resources by defining that state in a programming language,
| Pulumi can do other interesting things with your resource
| graph too. For example, analyze resources and check that
| they are compliant with security best practices and what
| not. https://www.pulumi.com/docs/get-started/crossguard/
| octopoc wrote:
| The Pulumi Automation API lets you have an API to update
| infrastructure from your own process instead of via CLI commands
| that call your executable. I hope this makes it easy to implement
| multi-tenancy by spinning up separate infrastructure for
| different customers.
|
| This would make it easier for software for dynamic schema built
| on top of a relational database. Dynamic schema is one area where
| most libraries don't help you out much, but it's an increasingly
| important feature, especially for businesses.
| daralthus wrote:
| <3 Pulumi. It's really nice defining what you want to exist at a
| cloud provider or as a k8s service, and it will make sure that it
| is there and nothing else. All in typescript.
| chrischen wrote:
| Any thoughts on Pulumi vs Terraform?
| cmclaughlin wrote:
| > Any thoughts on Pulumi vs Terraform?
|
| If you're just getting started with Terraform be sure to check
| out CDK for Terraform. It allows you to drive Terraform from a
| variety of high-level programming languages instead of it's
| DSL.
|
| https://github.com/hashicorp/terraform-cdk/
| StavrosK wrote:
| > Getting Started with Python. Prerequisites: Terraform >=
| v0.12, Node.js >= v12.16
|
| What? :(
| flurie wrote:
| I believe both the aws cdk and this use code generation to
| create the libraries for languages other than typescript,
| and you're reliant on the typescript package for updates to
| those libraries.
| StavrosK wrote:
| Hmm, how do you mean? Doesn't the Python code generate
| JSON that Terraform then reads?
| lucidone wrote:
| It's like Gradle versus Maven in Javaland.
| 1_player wrote:
| FWIW I found Terraform buggy, and really disliked its language.
| The fact that Pulumi runs on a classic, Turing-complete
| language is a huge plus to me.
|
| Unless your infrastructure is so large and managed by so many
| departments you need the least powerful declarative language to
| stay sane, my suggestion is to go with Pulumi.
| whydoineedthis wrote:
| imho, if you think typecasting the AWS api in your local code
| repository is useful, than Pulumi is the way to go. Maybe one
| day someone will find an api that AWS didn't properly typecast
| before releasing and then the hundreds of hours of typecasting
| in your local repo will be worth it.
|
| Other than that, it's really annoying way to keep
| infrastructure. I prefer the Terraform DSL myself, but I'm
| certain there is an audience that disagrees with me.
| dith3r wrote:
| Using Pulumi for about 1 year to manage gcp and AWS environments
| (5+) - each deployment is around 500 resources. At the begining
| it was buggy but when they started to add typing i see a lot of
| improvment.
| mssundaram wrote:
| I am having trouble understand _what_ Pulumi does, since I rarely
| work in web architecture. Could someone explain it?
| tomca32 wrote:
| It's a library around Terraform that lets you write infra stuff
| in Python, JavaScript, Go, and maybe some other langs.
|
| For me the biggest benefit is the fact that I can write my
| infra with Types in TypeScript.
| spooneybarger wrote:
| It's no longer built around terraform. It uses the cloud
| provider APIs directly now.
|
| See native provider.
| qbasic_forever wrote:
| It's infrastructure-as-code to the logical conclusion that you
| write real programming language code (Python, Javascript, Go,
| etc.) to build your infrastructure. Something like Terraform
| lets you declare infrastructure with HCL/JSON but doesn't give
| you a lot of flexibility to reason about or dynamically change
| things. If you want to spin up a frontend server for every
| backend for example, you have to write out the configs and
| tweak them to match the desired state. Pulumi and similar tools
| go the next step and let you write code to build your
| infrastructure--if you want your frontends and backends to be
| in sync, write some Python to make it happen instead of
| spelunking in a pile of configs.
|
| It's super useful and compelling at scale, but there are some
| potential downsides. For one, it's a real program and IMHO you
| should treat it as such with the same rigor of documentation,
| testing, etc. as the production code you're deploying. This
| might actually make your processes a bit slower or more risky.
| Because if you don't do that then you just risk building
| unmaintainable, untrustworthy deployment code that nobody wants
| to use.
|
| The second downside is you're taking a big bet on Pulumi to
| stick around. At this point, it's probably a safe bet. But the
| more you use their SDK and system the more tightly coupled your
| production deployment is to the continued existence of Pulumi.
| If they go bust and can't maintain the SDK you might have a
| production system that can't be deployed anymore.
| aetherspawn wrote:
| Thanks, I wanted to know the main difference between
| Terraform and this (ie why would I pay $50/mo), but this sums
| it up nicely.
|
| Still, Terraform has my needs at this stage and looks a lot
| more mature. But if I was a small web host or something this
| could be a really interesting integration.
| ludwigvan wrote:
| I think they don't advertise it that prominently for
| marketing reasons, but note that you don't have to pay
| unless you use Pulumi SaaS (the web ui).
|
| In that case, you can save your state in an S3 bucket, just
| like you save your Terraform state somewhere.
| jacobmischka wrote:
| Thank you for this concise summary. While not a dedicated
| cloud services administrator, I keep up with new technologies
| and follow tech news fairly religiously, and this press
| release made my head hurt with all of the cloud-specific
| buzzwords and proper nouns.
|
| Sounds like a neat and useful project for complex setups.
| qbasic_forever wrote:
| Yeah there's a lot of churn going on, but the
| infrastructure space is innovating in really interesting
| ways right now. On one side you have things like Pulumi
| diving into the full programming language for configs, on
| the other side you have an entirely different ethos of git-
| ops where you only have 'dumb' JSON/YAML configs and store
| all their state in a git repo with hooks to deploy and
| manage everything--(ArgoCD, flux, etc. are tools here). And
| in the middle you have some ideas for non-turing complete
| languages like Starlark, Jsonnet, Cue, Dhall, etc. to let
| you do some config generation but limit the risk of
| building broken configs.
|
| There's really no definitive 'winner' or best option yet.
| Each has strengths and weaknesses relative to others. It
| might just come down to how a team prefers to manage and
| work with infrastructure.
| satya71 wrote:
| It lets you provision and configure cloud infrastracture using
| code. Think about all the things you could do via cloud
| provider consoles, but now via code. So you can automate
| deployment, testing, migration. It's all well documented in one
| place, shareable, repeatable.
| adflux wrote:
| Define your cloud infrastructure in code (python, typescript,
| other languages) and deploy it with ease. I think it beats
| other formats.
___________________________________________________________________
(page generated 2021-04-20 23:00 UTC)