[HN Gopher] Fetch API has landed into Node.js
___________________________________________________________________
Fetch API has landed into Node.js
Author : yamafaktory
Score : 577 points
Date : 2022-02-01 12:31 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| tekstar wrote:
| Can you set more than one cookie in the request and read more
| than one cookie in the response? Was shocked when I learnt that's
| not supported by the spec..
| styfle wrote:
| There's a relevant spec issue you can follow here
| https://github.com/whatwg/fetch/issues/973
| stereocodes wrote:
| Wow I can't believe how light years ahead Deno is than Node. You
| want imports you want fetch you want TS, use Deno.
| ecf wrote:
| Seriously. I maintain an open-source Node.js CLI utility and
| was shocked when import/export syntax is only achievable if you
| use a transpiler like Babel. Such overkill for something that's
| been in the JS spec for nearly a decade.
| ash_gti wrote:
| Node does support ESM now by default (and has supported it
| for a while behind a flag), https://nodejs.org/api/esm.html
| andyfleming wrote:
| Why can't you use TypeScript with node? It's not built in, but
| it's easy enough to use by building or running it with ts-node.
| Also, once you are using TypeScript, you have import syntax
| support. Deno has chosen a different path, but there are
| definite trade-offs.
| Scarbutt wrote:
| Yes, and the go light years back in ecosystem ;)
| inglor wrote:
| Hey, Node core person here (and the person who triggered the
| land) - we're super excited for this and would love help and
| feedback.
|
| This is still experimental and we'd love to hear from the
| community what you'd like to see.
| eurasiantiger wrote:
| FormData and Blob/File support?
| staticelf wrote:
| Quick question, judging from the commit alone this seems like a
| rather small change. I assume there's more to it though. I just
| wonder how come features like this that kind of seems obvious
| to include in the ecosystem takes quite some time to land? I
| understand the reality is more complex perhaps, so I am
| genuinely curious.
|
| I hope you realize no disrespect, this will greatly improve the
| daily work for me since I use node at work every day.
| oefrha wrote:
| Adding 8k lines is hardly a small change.
| inglor wrote:
| The commit adds undici (another Node.js project at
| https://github.com/nodejs/undici ) as a dependency and
| exposes its `fetch` - the code changes you see are probably
| just adding a flag :)
|
| > I just wonder how come features like this that kind of
| seems obvious to include in the ecosystem takes quite some
| time to land?
|
| I answered that below (check it out) but note how expensive
| adding a bad API is vs. asking people for one more `npm
| install` :) There is more discussion in
| https://github.com/nodejs/node/issues/19393 and in https://do
| cs.google.com/document/d/1tn_-0S_FG_sla81wFohi8Sc8... a
| discussion from 2018 we had on it
| ilkkao wrote:
| I had a quick look, the implementation is interesting
|
| Node uses undici JS library. Undici includes a C HTTP
| parser that is used via webassembly. The parser is
| generated from rules that are implemented in JS/TS using
| llparse framework.
|
| A native C binary there in the middle is quite surprising.
| staticelf wrote:
| Ok cool and thanks for the clarification, I imagined it was
| more to the topic than I realized.
|
| Last question I have is will this be included in the next
| version of the LTS version or how do experimental features
| usually progress for someone who are unfamiliar?
|
| Can't wait until I can use fetch in my codebase without any
| additional dependencies!
| Narretz wrote:
| To add, unidici is an http client. Node already has http /
| https, but unidici is much faster and probably has other
| improvements, too.
|
| They could have built a fetch interface on top of http, but
| they probably saw an opportunity to make more drastic
| improvements.
| pilif wrote:
| my guess is that this adds stuff to the global object which
| has backwards compatibility concerns.
|
| And with every public API: Once you add it, you can't really
| hope to ever change it and even bug-fixes could be breaking
| some user's code (because they relied on the bug), so you
| have to be very careful to ship your public API as bug-free
| as possible.
| nikanj wrote:
| Meh, the js/ts ecosystem breaks things all the time. At
| least 2 weeks ago you couldn't run typescript with ts-node,
| unless you explicitly enabled highly experimental loader
| flags.
|
| And this is no obscure corner case, ts-node is huge
| staticelf wrote:
| Yes I could imagine and I have experience of developing
| apis with "bugs" that you kind of need to keep since people
| come to depend on it working a certain way.
|
| But I can't help to feel like Node is progressing kind of
| slow compared to other languages / tools. This is just a
| personal impression from a bystander who is not really into
| the actual progression though. I have a feeling that node
| used to push new features all the time but in last couple
| of years have stagnated somewhat. Stuff like imports etc is
| still not really here.
|
| I understand that developing an api that has such huge
| userbase as node is a massive undertaking that probably has
| issues that I can't imagine but it doesn't really help me
| in my day-to-day experience with it.
| notriddle wrote:
| I think some of it is because Node doesn't have very many
| ways to let people know about potentially breaking
| changes. A compiler (like Elixir) has an opportunity to
| communicate potential breaking changes to the user before
| the program actually runs, and a static type system (like
| Rust) makes it easy to detect such broken code and reject
| it outright instead of silently changing behavior. It's
| harder for Node: where would the diagnostic even go? The
| console? Some apps use Curses, so this will break them,
| and many of them send their own logs through something
| other than the console, so nobody's actually watching it.
|
| Node also isn't actually a language, so there's a lot of
| stuff that Node developers get in new version, but it's
| actually being done in the V8 project.
| asdfman123 wrote:
| Thank you for making "fetch" happen
| kichimi wrote:
| I do have a question that's only tangibly related to fetch, I
| was wondering what Nodes stance is on the seemingly increasing
| gap between the NodeJS and browser implementations of the
| JavaScript engine.
|
| It's still very common to see Node modules that use require(),
| for example, but would otherwise work flawlessly in a browser
| where there is no require() support (without using shims and
| other libraries).
| inglor wrote:
| I don't speak for the project a whole :)
|
| Node has supported ESModules (`import`) for a while and you
| can use it (either by naming your files `.mjs` or setting
| `type: module` in your package.json.
|
| There is a solid interop story between require/import and you
| can mix if you want.
|
| `require` is going to be supported "forever" so code doesn't
| break and both module systems work.
|
| As for the general question: Node is committed to supporting
| modern JavaScript and to not allow such a gap to be created.
| If you see a place Node doesn't do a great job at that please
| tell us!
| arcatek wrote:
| However note that ESM in Node comes with drawbacks that
| prevent end-users from relying them in various situations.
| Those will be mostly solved once loaders become stable, but
| until then it's still advised to ship packages as both CJS
| and ESM.
| rattray wrote:
| How do you do both?
| inglor wrote:
| It's very easy to re-export ESM (import) as CJS (require)
| and vice-versa. The main issue is that ESM by default
|
| For example to use `require` inside ESModules you would
| do:
|
| ```mjs import { createRequire } from 'module'; const
| require = createRequire(import.meta.url);
| require('./whatever-in-cjs'); ```
|
| There is a reason this isn't "by default" though since
| ESM doesn't "silently" interop with CJS to not make
| writing universal code harder.
| arcatek wrote:
| Usually you setup a transpiler to target both. For
| example, my packages are bundled with rollup towards both
| cjs & esm:
|
| https://github.com/arcanis/clipanion/blob/master/rollup.c
| onf...
| rattray wrote:
| Helpful, thank you!
| jeswin wrote:
| > I was wondering what Nodes stance is on the seemingly
| increasing gap between the NodeJS and browser implementations
| of the JavaScript engine. It's still very common to see Node
| modules that use require()...
|
| "Seemingly increasing gap" - how did you come to that
| conclusion? If anything the gap has reduced since the
| availability of ES modules. I see more and more libraries
| preferring ES modules, but ultimately that's up to the
| library developers.
| rglover wrote:
| This is awesome, thank you. Excited to take it for a spin.
| zebracanevra wrote:
| Is there any chance to break spec and allow manual redirect
| handling? the fetch API makes a lot of sense in a browser, but
| imo this is a pretty crucial feature that undici's
| implementation lacks. [0]
|
| Deno decided to break spec [1][2] so the following code works
| fine: fetch('https://httpbin.org/status/302',
| {redirect: 'manual'}) .then(res =>
| console.log(res.status, res.headers))
|
| In undici this will succeed but with res.status 0 and no
| headers, as per spec. You aren't allowed to see the content of
| a redirect, like in a browser.
|
| [0] https://github.com/nodejs/undici/issues/1072 [1]
| https://github.com/denoland/deno/pull/8353 [2]
| https://github.com/denoland/deno/issues/4389
| tshaddox wrote:
| Is this not what redirect: "manual" is in the MDN fetch docs?
|
| https://developer.mozilla.org/en-US/docs/Web/API/fetch
| astrosi wrote:
| In that in the browser you are able to see that you have
| gotten a 301/302 response but can't see where the redirect
| would have sent you.
| tshaddox wrote:
| Oh right, because they don't want cross-site scripts to
| be able to see redirected URLs since they could contain
| secrets. I wish we could completely do away with cross-
| site scripts and just have nice things!
| asiachick wrote:
| Yea, so instead it would just encourage more 3rd party
| libraries doing random things on your site. This is what
| happens in native. Instead of embedding an ad in an
| iframe and isolating its damage you embed your ad
| service's library in your code and it spies on way more
| activity than it ever could otherwise.
| tshaddox wrote:
| I would also be okay ( _ish_ ) with explicitly isolated
| third-party code execution, like your example of an
| iframe to a different domain. I'm pretty sure that should
| already be the case with iframes, in fact (you obviously
| shouldn't be able to embed an iframe to facebook.com on
| your website and then use your website's JavaScript to
| inspect the DOM on that facebook.com iframe).
| [deleted]
| engineeringwoke wrote:
| It's pretty funny that such an RPC framework as the
| browser exists that gives the end user a genuinely decent
| sandbox, yet all it receives is criticism for its flaws.
| People will then happily install a screen dimmer or
| "productivity" tool with superuser privileges from a
| completely untrusted source.
| inglor wrote:
| Yes, this is a place where Node.js will diverge from the spec
| see discussion on the fetch PR https://github.com/nodejs/node
| /pull/41749#issuecomment-10254...
| nailer wrote:
| A better idea would be to replace / improve the fetch spec.
| Handle redirects, set JSON as the default request content
| type, encode URI components for users, decode response bodies
| with JSON headers as JS objects, etc.
|
| Right now most developers either write their own library to
| do these things on top of fetch or another HTTP client or use
| a third party library from npm.
|
| A new standard would allow us to make http requests out of
| the box comparable to high level HTTP clients like
| superagent, axios etc. that have been in use for the last
| decade, since before fetch was conceived, with whatever
| benefits fetch provides (I think it's cancellable now?).
| inglor wrote:
| A new standard would need buy-in from browsers to actually
| be a standard. Browsers likely don't have a lot of
| incentive to make spec changes only Node is interested in
| (like making the whole spec more complicated from their
| point of view because of Node's (or Deno's) different
| security model).
|
| The level of collaboration and good-faith we've been
| getting from spec bodies like WHATWG is very high as it is
| and we really don't want to push or abuse it.
| nailer wrote:
| > Browsers likely don't have a lot of incentive to make
| spec changes only Node is interested in
|
| All the existing HTTP clients discussed in the previous
| post also run in the browser.
| true_religion wrote:
| It would be useful for trusted browser extensions too, as
| well as probably Electon apps as well.
|
| I don't know if chrome apps are still in existence, but
| if they are the "server side" fetch spec could apply to
| them too.
| andrewl-hn wrote:
| Does it have to be a buy-in, though? You could agree on
| server-specific extensions to fetch and codify them in
| the same spec (so it doesn't get lost otherwise). Browser
| vendors wouldn't need to implement it but they will still
| keep an eye on it when working on future browser apis.
| tehbeard wrote:
| Codifying the redirect behaviour for server/"trusted"
| envs makes some sense with Deno/node doing the same
| thing. I'm not versed enough in the standards
| orginazation politics to say if that's viable though.
|
| The rest of your "improvements" aren't that. They're
| opinions, opinions laser focused on a JSON centric api.
|
| I'm sorry to say this, but not all of the web is one big
| JSON blob.
|
| Some of us have to talk to SOAP (xml) services.
|
| Sometimes it's weird rpc stuff with protobuf.
|
| Or even just pulling down raw binary data and feeding
| that through an API/library that wasn't built for web
| streams.
|
| Honestly, the amount of code you need to add over the top
| of a primitive like the Fetch API to get what you wanted
| (bar the redirect change) is trivial and minimal.
| Scarbutt wrote:
| Can you share your setup for talking to SOAP services?
| the_duke wrote:
| None of the things you mentioned are reasonable default
| behaviours, and are easy one liners. You don't need a
| library for any of that.
| nailer wrote:
| Thanks for your opinion, however as mentioned, or as
| you'll be aware if you've used JavaScript, these are well
| worn cow paths that have been default in the most popular
| HTTP clients for a decade now.
| vanviegen wrote:
| Why is this being down voted? This comment seems to be
| making a reasonable case.
|
| In case you disagree, you may want to _reply why_ instead
| of down voting.
| simlevesque wrote:
| Because fetch _just_ landed in Node.js, making it a
| standard present in every context after like 5 years and
| he want to change the standard now.
|
| Edit: also, you are being downvoted because the etiquette
| on HN are that you don't ask "why was X downvoted".
| nailer wrote:
| As mentioned existing HTTP clients had this behaviour
| back in 2012, the designers of fetch chose to ignore
| these cowpaths in favour of a more minimal implementation
| requiring the use of third party HTTP clients in both
| browsers and node, simply to have reasonable defaults and
| not repeat oneself, for the foreseeable future.
| andreigheorghe wrote:
| the value the js community gets from `fetch` being a
| unified standard is far, far greater than the benefit you
| as a developer would get from not having to add 15 extra
| LOC around `fetch`, that you'll probably hide behind a
| `fetchJSON` function anyway.
| [deleted]
| nailer wrote:
| Point is everyone had a slightly different 150 LOC or a
| library to achieve essentially the same thing.
|
| We could have stuck with 'if index is not equal to minus
| one' but we have array includes now for the same reason.
| franciscop wrote:
| Really happy to see fetch() merged into core! Looking at the
| PR, what are those WASM blobs in Undici and how is it that
| Node.js accepts a random very large compiled blob instead of
| asking for the source file + compilation step?
| https://github.com/nodejs/node/commit/6ec225392675c92b102d3c...
| inglor wrote:
| Those wasm blobs are Node's own llhttp
| https://github.com/nodejs/llhttp in wasm to speed up HTTP
| parsing. The project itself added as a dependency is also by
| Node https://github.com/nodejs/undici .
|
| The question is totally legitimate but please assume core
| doesn't make "load random binary" level kind of goofs :)
| krono wrote:
| Except that's exactly what Corepack does, just silently and
| opaquely on the user's system. At least, the moment it
| switches from opt-in to opt-out.
|
| Its unverified binaries are just as good as random
| binaries, and silently replacing any pre-existing
| yarn/pnpm/whathaveyou without checking if it was perhaps a
| custom build is not exactly predictable either.
|
| Other than Corepack, though, I think you people are doing
| an amazing job!
| franciscop wrote:
| Sorry def didn't mean in a bad way! I just wanted to
| ask/note that in the PR, and with the very little context I
| have, it seems like an arbitrary code blob so I was
| surprised, I'm 100% it'll be well documented in e.g. undici
| itself, and prob in other places.
| nefitty wrote:
| Hey this one change is going to positively impact my day to day
| life. Just wanted to let you know. Thanks for sharing your hard
| work and time.
| inglor wrote:
| I'm happy to hear that but I had a very small part in it all!
|
| The person who took it through the finish line (and deserves
| the most props here IMO) is Michael https://github.com/targos
|
| The people who worked most on Undici are Matteo
| https://github.com/mcollina and Robert
| https://github.com/ronag
|
| The person to work most on fetch in Undici is Ethan
| https://github.com/Ethan-Arrowood
|
| Also worth calling our James whose work on web streams in
| Node as well as events/cancellation helped drive a bunch of
| this.
|
| I can name maybe 50 people who worked on this effort overall
| at some capacity (I am one of them - spending maybe 40 years
| on APIs to help enable this).
|
| Note the job isn't entirely done and help is very much
| appreciated!
| mccorrinall wrote:
| What does undici mean?
| Eriks wrote:
| eleven
| paulfitz wrote:
| https://undici.nodejs.org/ "Undici means eleven in
| Italian. 1.1 -> 11 -> Eleven -> Undici. It is also a
| Stranger Things reference."
| _ZeD_ wrote:
| "undici" means "eleven" in italian. But I don't know the
| rationale behind the name
| epolanski wrote:
| http 1.1
| mitchellgoffpc wrote:
| Huge thanks to the node team for adding this, I've been wanting
| fetch in node for years now! Installing node-fetch for every
| project was getting kind of old haha
| adityapatadia wrote:
| Thanks a ton for this. Fetch API is elegant and it working on
| node natively will so many edge cases we need to handle
| currently. Thanks a ton, please know that this will positively
| change our life and tens of thousands of other developers
| worldwide.
| joshxyz wrote:
| Does it mean we have a built-in FormData in Node.js core for
| multi-part requests?
| ivanb wrote:
| Is there support for timeouts? It's the main reason I use
| https://github.com/sindresorhus/got
| inglor wrote:
| You can use timeouts through `AbortController/AbortSignal` -
| eventually it'll even be built in with `AbortSignal.timeout`
| which is currently under-works in the spec level.
| gk1256 wrote:
| I wish if Node had a way to enforce any case in its requests.
| neves wrote:
| Where can I find an explanation about what is Fetch and why it
| should be in Node.js?
| tehbeard wrote:
| fetch is a "modern" (ES6 era, so not brand new) API for
| making HTTP requests, to replace XMLHttpRequest (the older
| method that IE pioneered).
|
| It's promise based (thus easier to integrate into code than
| older callback styles), and designed to give some better
| options around CORS and handling responses.
|
| > Why it should be in node?
|
| There is a push to adopt some certain "web apis" (APIs that
| emerged in web browsers) to increase compatibility,
| reusability and reduce cognitive load when working with both
| backend and frontend javascript.
|
| Having a common low level call like fetch() means that
| libraries that are built atop it (SDKs to talk to services,
| or apply common middleware like JSON:API, oAuth etc) can be
| shared between node/deno and browsers.
| golergka wrote:
| Fetch is a popular browser api to download and request stuff
| from the internet. Having the same api both in browser and in
| node allows easier re-use of the same application code or
| libraries on frontend and backend.
| inglor wrote:
| This is fetch: https://github.com/whatwg/fetch
| https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
|
| Here is discussion about it in Node with some (pretty old)
| discussion https://docs.google.com/document/d/1tn_-0S_FG_sla8
| 1wFohi8Sc8...
| neves wrote:
| It looks like it is a new standard:
| https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
|
| I thought it was a Node popular library.
| fiddlerwoaroof wrote:
| It's "new" as of 6 years ago or so.
| ignoramous wrote:
| Hi there,
|
| > _...we 'd love to hear from the community what you'd like to
| see._
|
| In our tests, _fetch_ on Deno was wayy faster than _undici
| fetch_ that now has been merged into nodejs.
|
| I couldn't figure out why that was case, but forwarding
| requests over node's http2 client (instead of _undici fetch_ )
| then had comparable (but not as fast) performance as Deno
| (presumably because our hand rolled impl lacked connection
| pooling).
| inglor wrote:
| This is very new and the implementation _just_ landed today
| so it makes sense Deno would be faster with an API they have
| been working on for years.
|
| I am confident that Node's implementation will _eventually_
| have comparable performance.
|
| That said: please do open an issue in the undici repo at
| https://nodejs.org/node/undici so this gets tracked.
| mstade wrote:
| That URL gives me a 404 - should it be
| https://github.com/nodejs/undici ?
| thysultan wrote:
| Deno's is probably written in rust via the tokio rust crate.
| This version of fetch in NodeJS is written in JavaScript. It
| would probably get upgraded to C++ in due time once it's
| mature enough.
| tbarbugli wrote:
| yes, yes, yes.
|
| This was a huge pain for us. We develop an SDK that needs to
| work on Node, browsers and React Native.
|
| I am super happy to hear about this :)
| andrew_ wrote:
| pretty great. I've been using undici for a while now and very
| happy with the speed and experience.
| can16358p wrote:
| Great work!
|
| Curious to check but quick question: does it support upload
| progress? If not, is it considered?
| inglor wrote:
| You can upload a stream and monitor that for upload progress
| - so sure.
|
| If you upload a string for example you would have to take an
| extra step to get upload progress (make it into a stream and
| monitor that).
|
| There is additional (unrelated) talk in the fetch standard
| itself to provide this sort of functionality as part of
| `fetch` which would save the middle step.
| mnutt wrote:
| Do you have any sense of the expected performance of web
| streams as compared to the existing node.js streams?
| stevage wrote:
| Out of curiosity, what took so long? It's a small API that has
| been stable for a long time? I'm probably missing something.
| pjc50 wrote:
| Brief explanation?
| forty wrote:
| Fetch is a web JS API to do HTTP requests. NodeJS has its own
| API to do HTTP requests. This add supports for the fetch API.
|
| I think the only benefit it that it makes it easier to share
| code between front end and back end.
| Allstar wrote:
| For more on Fetch, check out MDN:
| https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
| giffarage wrote:
| It makes it easier to learn and develop both frontend and
| backend.
| 9dev wrote:
| This is an understatement. With fetch available in node, we
| have a single, standardised, promise-based HTTP client
| Interface available on all (more or less common) JavaScript
| platforms.
|
| This is great news for an ecosystem traditionally extremely
| fragmented and may lead the way to more consolidation in the
| JS library space.
| forty wrote:
| NodeJS API are traditionally not promised based (although
| nowadays some do offer a promisified option), so I'm not
| especially excited to have a promise-based API :)
|
| It's a great news for people doing both front end and
| backend I guess, but for people doing NodeJS developpement
| it's just one more option to do http request (which I guess
| is going to be very similar to the module node-fetch that
| have existed for a while, without special consolidation
| effect)
| svachalek wrote:
| > NodeJS API are traditionally not promised based
|
| ...and that's part of why this is great news. Node APIs
| are the avocado colored appliances of the JS ecosystem.
| zemo wrote:
| node js is a web software platform over a decade old that does
| not include a stable API for making a web request so there are
| a bunch of community libraries for making a web request, and
| which one is "best practice" has changed like four or five
| times through the years. this is a new api for making a web
| request. yayy progress.
| saos wrote:
| Noicee
| joelbondurant1 wrote:
| sealthedeal wrote:
| Ive used Fetch in the past, great lib. I'm curious why Node core
| has decided to make this an official part of Node? From one
| perspective there could be a hard argument that this should never
| be a part of Node as it is an external library and not a building
| block of the language? Almost feels like feature creep for the
| core language? I know that's not the intent, but wanting to
| understand the logic behind this decision :)
| pitaj wrote:
| Fetch is a standard browser API.
| wruza wrote:
| And so are navigator, localStorage and classes under window,
| like EventTarget.
| themikesanto wrote:
| Yeah. What's your point?
| wruza wrote:
| Comments like "it's standard" do not answer the original
| question:
|
| "I'm curious why Node core has decided to make this an
| official part of Node?"
|
| The standard describes many functions and classes. What
| makes fetch more land-able than e.g. the event system, of
| which node has incompatible implementation? Or feature
| detection via navigator. Why it? What were the main
| point(s) of adding fetch specifically?
| ravenstine wrote:
| Sure it does.
|
| If something is standard, and it makes sense in the
| context of a backend, then it's convenient to just use
| the same API on both ends. Navigator makes almost no
| sense on the backend unless maybe it worked as a dummy
| polyfill object.
| ramblerman wrote:
| fetch is part of the client side standard:
| https://fetch.spec.whatwg.org/, and had been for a while.
|
| I don't think it is so strange for node to incorporate it as
| part of their standard lib as well.
|
| 1. https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
| prollings wrote:
| They're talking about JavaScript's fetch API. It's not an
| external library.
| wruza wrote:
| I second this, since fetch is a wrapper around xhr (though
| doesn't have to be because node is a non-restricted runtime),
| and it diverged already for redirect semantics. It's news, but
| is it big?
| samtho wrote:
| > It's news, but is it big?
|
| As someone who has been working almost exclusively on Node.js
| applications for years now, this is big.
|
| Front-end developers have, for years, enjoyed this wrapper
| around the clunky XHR API. It is simple, intuitive, and it
| comes _free_ in the browser environment (and was easily
| shimmable before it was standard). Additionally, the
| depreciation of the popular request[0] a while ago, there has
| been a gap in The Right Way to do http requests without going
| into the low-level API that 'http(s)' provides. Fetch is
| promised-based (enabling seamless async/await) and, most
| importantly, standardized.
|
| Basically, fetch provides a meaningful abstraction atop each
| platforms' implementation of http-request-doing further
| unifying server-side and client-side patterns and idioms.
|
| [0]: https://www.npmjs.com/package/request
| wruza wrote:
| npm i request node-fetch make-fetch-happen axios. Just out
| of top of my head.
| sealthedeal wrote:
| FWIW, this was the answer I was hoping to get, grabbed from a
| separate comment :)
|
| "With fetch available in node, we have a single, standardised,
| promise-based HTTP client Interface available on all (more or
| less common) JavaScript platforms.
|
| This is great news for an ecosystem traditionally extremely
| fragmented and may lead the way to more consolidation in the JS
| library space. "
| Meic wrote:
| Undici seems to be an HTTP 1.1 client. Is there any work or
| timescale to provide HTTP 2 support?
| inglor wrote:
| Not yet but it's planned to eventually happen.
| bricss wrote:
| check https://www.npmjs.com/package/rekwest
| bricss wrote:
| Try adv fetch-alike with HTTP2 support
|
| https://www.npmjs.com/package/rekwest
| itsbits wrote:
| Which NodeJS versions will this be landed to?
| inglor wrote:
| It's behind a flag so that it can land in the next v16 semver
| minor - so soon'ish but keep in mind it _is_ experimental.
| krossitalk wrote:
| As someone browser focused this surprised me. I've been using
| fetch() for years now.
| DrFell wrote:
| Intercommunicating backend services isn't new, but having an
| HTTP API call another one from the backend is a case that used
| to be more normally accomplished by calling both APIs from the
| frontend. The slow ooze of browser-oriented thinking onto the
| backend took time. Existing backend engineers, who thought it a
| harebrained scheme, had to eventually wear down, and give up.
| motogpjimbo wrote:
| So when you write an application that integrates with an
| external service via an HTTP API, you think the normal way to
| architect such an application is to make all API requests
| from the browser? And you think that making such calls from
| the backend is a new and "harebrained" idea that backend devs
| have been forced to adopt against their better judgement?
|
| I'm curious to know what position you occupy in our industry?
| DrFell wrote:
| Yes, I think the normal way to integrate an app with an
| HTTP API is via the browser.
|
| Calling an external API from a backend that integrates
| directly with an client app has been rare for me. Payment
| processing is all that comes to mind, and that was usually
| through an imported API package. I've made data scraping
| tools, but they were unique projects, not part of an app
| backend. I am a senior full-stack webdev.
| motogpjimbo wrote:
| Has it ever occurred to you that your users can (a) see
| all requests you make to third party services, including
| your credentials and client secrets, (b) modify and
| replay those requests at will and (c) spoof responses
| from those services?
| DrFell wrote:
| I think the misunderstanding is, beyond special cases
| like a payment processing, why would you be calling third
| party services from your service? When I think of calling
| a 3rd party HTTP API, I am thinking of public APIs, like
| say a widget that shows the weather in a users area.
| motogpjimbo wrote:
| I'm going to take a wild stab in the dark here and guess
| that despite having the title "senior", you have little
| or no experience in commercial software development?
| Because what you dismiss as "special cases" covers a vast
| amount of the plumbing of the modern web. In modern apps
| even something as simple as storing and retrieving files
| requires HTTP calls at some point in the stack, and such
| calls should never under any circumstances be initiated
| from someone else's computer.
|
| Which brings us back to the original question you
| responded to above. Contrary to your odd views of how
| software is architected, issuing HTTP requests from the
| backend is both normal and common enough to be banal.
| Node adopting the fetch API is generally a good thing as
| it allows JS developers to code against the same
| interface whether they are working from the frontend or
| the backend.
| emteycz wrote:
| Sure - but that weather/map/whatever widget might be
| paid, and unless you want to pay for others' API usage,
| you should make the relevant API calls from your backend,
| not frontend. Also consider caching 3rd party service
| replies, etc.
| LudwigNagasena wrote:
| As for someone who just dabbles in JavaScript from time to
| time, there are so many things that surprise me...
| zamadatix wrote:
| Not every browser API is an instant candidates to be added to
| Node's core. If there isn't a big performance hit from doing it
| as a user library or lack of functionality to do it as such in
| the first place then it's rare it'll be brought into core right
| away, if it will at all.
|
| It's nice to see the really popular ones make it in though,
| makes for fewer dependencies when doing browser compatible
| projects and also allows one to be lazy and not learn the Node
| way of doing it. Maybe we'll see websocket go into core some
| day too.
| herpderperator wrote:
| This is what frustrates me so much about javascript.
| Everything is all over the place.
| zamadatix wrote:
| Eh, I mean I get it JS lacks a real standard library but at
| the same time you don't see HN fill up with people saying C
| is all over the place because when they went to implement
| an HTTP call glib doesn't mirror libuv or C++ is all over
| the place because POCO doesn't mirror Boost and so on.
|
| The "all over the place"-ness tends to come more from
| people who act like changing to a newer library is the only
| way to code something new not from the lack of JS forcing a
| single API be used across every app regardless of type.
| [deleted]
| outside1234 wrote:
| Thank god - no more node-fetch shims for node.js!
| esamatti wrote:
| Does this contain `request.formData()` and `FormData`?
|
| https://developer.mozilla.org/en-US/docs/Web/API/Request/for...
| https://developer.mozilla.org/en-US/docs/Web/API/FormData
| inglor wrote:
| > `request.formData()`
|
| Yes.
|
| > FormData
|
| Yes, but not in that PR since `FormData` needs to behave
| differently as part of the platform but there is intent to
| support it before moving from experimental.
| mcraiha wrote:
| Why it took so long? e.g. Deno had fetch support for ages
| inglor wrote:
| Basically because fetch isn't a great API for servers it took a
| while to get consensus on actually landing it for the
| interoprability/simple API value.
|
| Then it took a while to get consensus it's fine to do even if
| we can't implement the standard fully and diverge from it on
| stuff like CORS (like Deno does).
|
| Then there were a bunch of work adding APIs like `EventTarget`
| and `AbortSignal` to Node.js which are quirky'ish web APIs.
|
| As a side note just to show positive collaboration: that work
| made me make maybe 10 PRs fixing things in EventTarget in Deno
| - and Deno helped Node a bunch now when landing fetch.
|
| Then web streams and blobs and a few other things as well as
| the undici work in the background.
|
| Then undici-fetch by Ethan, merge into Undici and then the PR
| by Michael - and it's still not done :)
|
| Finally
| msoad wrote:
| Interesting! What CORS even means in server? Never thought
| about this. What's a server's origin to begin with?!
| dstroot wrote:
| First, this is awesome and congratulations! Second this
| sounds like a lot of work! Can you comment why was it so hard
| to add fetch to node, yet packages like "node-fetch" have
| existed for some time and seem to implement fetch rather
| easily? Only asking from a curiosity perspective.
| inglor wrote:
| node-fetch (which is great btw!) implements fetch
| "reasonably" as in stuff like `fetch('./foo').then(x =>
| x.json())` works but it's very far from a "real" fetch.
|
| Some people argued users would still like it and there was
| an attempt[1] by Myles but at the end of the day - the
| changes were pretty big.
|
| For example: a response body in fetch is a web stream (a
| whole new stream type) and to cancel you use an
| AbortController (web type) which is an EventTarget (another
| web type). `node-fetch` uses Node streams instead which is
| very reasonable but not something a platform can "get away"
| with while still calling it fetch.
|
| Here is a comment I wrote about it in the tracker during
| the PR review process https://github.com/nodejs/node/pull/4
| 1749#issuecomment-10257...
|
| [1](https://github.com/nodejs/node/pull/27979)
| dstroot wrote:
| THANKS!
| cphoover wrote:
| Fetch evolved after promises had been standarized as the way of
| doing async, whereas node is older than that. Node originally
| used callback style for its core API and had another method for
| making http requests. Since fetch, developed in the browser, OS
| contributors have filled the void of a promise supporting http
| requesr library, with packages like axios, and isomorphic-
| fetch. Only now that the standards have finalized and we have
| reached consistent browser support has core team has made it a
| priority to include it in core.
| Mikeb85 wrote:
| Because Node is legacy and Ryan Dahl moved on to Deno... Node
| is just an outdated 'standard' at this point.
| engineerthrwawy wrote:
| I don't think that's an accurate characterization of the
| fork.
| Mikeb85 wrote:
| Calling it a fork certainly isn't accurate...
| songzme wrote:
| sorry if this is a stupid question, but how do I try this out? I
| ran node with the experimental flag on a file that calls fetch:
|
| node server.js --experimental-fetch
|
| Got 'fetch is not defined' reference error.
| inglor wrote:
| First: not a stupid question - the release cadence, multiple
| release lines and other minutia of a project like Node are not
| something I'd expect users to be intimately familiar with!
|
| This _just landed today_. You can get it by building from
| master:
|
| ```
|
| # there is more details in building.md
|
| git clone https://github.com/nodejs/node
|
| cd node
|
| # may need to pass --openssl-no-asm
|
| ./configure
|
| make -j12
|
| ./out/Release/node --experimental-fetch
|
| ```
|
| Otherwise - wait a bit for the next v17 release to land per the
| normal release cycle :)
| songzme wrote:
| oh I see so when node v17.4.1 is released, I would be able to
| run
|
| node server.js --experimental-fetch
| inglor wrote:
| v17.5 but yeah
| huksley wrote:
| Just slightly related to this - I wish I could ask GitHub to
| send me an email when specific release are available for the
| repo!
| fs111 wrote:
| You can subscribe to only releases on github.
| mattdeboard wrote:
| Sounds like maybe an "if this, then that" kind of thing
| https://ifttt.com/
| thyrox wrote:
| My biggest issue with node when I was working on it briefly was I
| couldn't do the 'import' statements like wepback. Are they
| supported too now?
|
| I'm not a web developer so I had a very hard time understanding
| why there are so many different type of imports in JavaScript
| like require, import, umd, amd, etc and which one works in
| browser and which one works in node?
|
| Also why do so many libraries have this strange 5 line header
| code for this umd, amd, business. Is that to make their packages
| work with nodejs?
|
| Does anyone who knows enough JavaScript point me in the right
| direction about it. I find all this very confusing.
| fn1 wrote:
| I know this comment is not very helpful, but the lack of a
| module-system and the lack of threading are unfortunately the
| biggest issues that JavaScript had from the start.
| dpweb wrote:
| I been naming the files with .mjs file extension which allows
| import keyword and top level await. No special flags needed in
| the latest version. I use JS not TS.
| Cthulhu_ wrote:
| > I'm not a web developer so I had a very hard time
| understanding why there are so many different type of imports
| in JavaScript like require, import, umd, amd, etc and which one
| works in browser and which one works in node?
|
| If I remember my history correctly, it's because when Node
| first came out, there was no import system in JS, let alone a
| standardized one; there was no sense of scoping (everything
| global), nothing about dynamic or lazy loading of dependencies,
| no tree shaking / removing unused code, and even going to the
| definition of something in an editor was difficult.
|
| NodeJS adopted CommonJS (I don't recall if they invented it),
| which is a module and dependency system based on require() and
| exports. It was only a few years later when the JS standards
| body settled on import; by then, the JS (dependency / module
| management) world was already very divided.
| mcaruso wrote:
| Node supports ES Modules (the import statements you mention)
| natively now: https://nodejs.org/api/esm.html
| inglor wrote:
| Those have been supported for several years now :)
|
| You need to tell Node to use the format by either naming your
| file `.mjs` or setting `"type": "module"` in your
| `package.json` file.
|
| > Also why do so many libraries have this strange 5 line header
| code for this umd, amd, business. Is that to make their
| packages work with nodejs?
|
| That's just old for "adapt the module system" from the old days
| and libraries just didn't bother updating.
|
| I still use amd in my job at Microsoft though for some things
| so I guess it's not useless :)
| mschuetz wrote:
| > either naming your file `.mjs`
|
| But why? This makes it much harder to ensure that identical
| code works in browser and nodejs, to the point where I
| sometimes just can't use nodejs. And I don't use
| package.json, so having to name files ".mjs" is a really
| painfull restriction.
| inglor wrote:
| Browsers will happily accept `.mjs` files they don't
| actually care about file extensions only about `Content-
| Type` and type=module on the script tag :]
| mschuetz wrote:
| Plenty of (third-party) imports specifically reference
| "*.js" files so no, this doesn't work.
| distrill wrote:
| why are you writing node applications without using
| package.json?
|
| it sounds like you're looking to have a problem with
| javascript tbh. don't use it, whatever
| Natfan wrote:
| If you're hosting the browser-based version over a server
| like Apache2 (to host the files), you could add a
| `RewriteRule` to redirect queries to `.js` resources to
| their `.mjs` counter-part files.
|
| Example: RewriteRule "(.*).js$" "$1.mjs" [R=301 NC]
|
| Reference documentation:
| https://httpd.apache.org/docs/2.4/rewrite/intro.html
| IshKebab wrote:
| Several years is a bit of an exaggeration. They were
| "experimental" until 20 months ago.
| tomphoolery wrote:
| "supported" is somewhat of a misnomer IMHO. It's only
| supported if the other tools you're using support ESM.
| Otherwise, it's not. And it's really annoying when libraries
| choose to go full ESM because it breaks things like tests and
| deployments in weird ways.
|
| It's a pain in the ass.
| rattlesnakedave wrote:
| It's really annoying when libraries don't support a 7 year
| old standard, as well.
| icholy wrote:
| Node supports ESM, full stop.
| [deleted]
| balls187 wrote:
| > Does anyone who knows enough JavaScript point me in the right
| direction about it. I find all this very confusing.
|
| There isn't a straight forward solution, but the closest for me
| is the combination of using a transpiler (Babel), and a bundler
| (Webpack).
|
| A common criticism on node/javascript projects is the boiler
| plate setup required. As far as I know, there isn't an IDE that
| takes care of doing this part for you, where your experience
| developing outside of the web might (I like to think it akin to
| starting a project from scratch with C++, gcc, and make).
|
| Some larger projects, do have scripts that do a lot of boiler
| plate for you, such as Create-React-App.
| https://github.com/facebook/create-react-app but that is for a
| specific use case.
| kodemager wrote:
| Import is slowly becoming the standard, but working with
| typescript professionally modules has easily been the most
| annoying part of the Node experience.
|
| It works pretty well, and then you need to use something like
| Azure Functions and then suddenly it doesn't. For various
| reasons.
|
| My most recent example was using lodash, which works perfectly
| fine with import with typescript targeting esnext in node16,
| but then needs to be setup with require when you target an
| azure function and commonjs. I mean, maaaybe you could avoid it
| by using mjs, which is currently sort of needed to move into
| the node16 functionality in azure functions, even though they
| sort of run node16 just fine in part of them without it, and
| you sort of don't want to use mjs files and so on.
|
| I'm sure it'll get there in a few years, but it is no doubt
| annoying to have to fight the toolset ever so often. Over
| something that feels like it should just be working.
|
| That last part isn't really exclusive to node these days
| though, is it?
| huksley wrote:
| I had the same problem so for serverless functions I am using
| standalone webpack config which transpires functions into the
| supported by the cloud format
| wruza wrote:
| In short, node has imports, browsers have imports, but node-
| related toolkits do not support it, and people still webpack
| bundles because development and deployment processes are a
| thing, and it doesn't matter much which module system a
| "binary" bundle uses in the end.
|
| Some people tried to force ESM adoption by making popular
| modules ESM-only for no technical reason, but tools are not
| there yet, and it only annoyed (predictably) half of the
| internet. Because even if you are pro-ESM or indifferent to it,
| you can't do much to migrate your projects' build pipelines. If
| you see typescript and webpack, you wouldn't see "ESM" in
| there. It either doesn't work or is too fragile for production.
|
| People who claim it's done say so because they are using
| particularly unaffected stacks (including no stack). Idk which
| way to promote ESM would be the most correct, but this one is
| too deceptive.
| rattlesnakedave wrote:
| > Some people tried to force ESM adoption by making popular
| modules ESM-only for no technical reason
|
| Being the standard for the lsat 7 years strikes me as a good
| reason.
| bachmeier wrote:
| Good that some people are happy about this, but it really is
| pointless to post a link to "lib: add fetch" followed by code
| diffs. HN should have a requirement that there be an actual post,
| rather than a discussion starter for those in a particular
| community. All you'd have to do is type in a brief text post with
| an explanation and then link to the code diffs for those
| interested.
| inglor wrote:
| There will be a blog post on this in the Node.js blog and
| official media communication. This _just_ landed today and I
| suspect the HN crowd are more in the "get news early" camp and
| not "wait for the official press release" camp.
| chx wrote:
| Good for NodeJS
|
| However, I feel the fetch API is completely botched because it
| lacks timeout support. I have cobbled some together for
| https://github.com/Backblaze/gists/pull/8 but gosh. I really hope
| all that is actually not necessary :/
| nikeee wrote:
| You can use an AbortController for that.
|
| Also, keep in mind that the linked solution does not abort the
| web request. It only aborts waiting for the response in the JS
| code. The browser does not close the connection.
| mariogintili wrote:
| I love that the most popular language's core lib now brings a
| trivial bit of code available in every other platform and its a
| celebrated super achievement
|
| software, why do you suck now?
| jmull wrote:
| I don't think you understand this.
|
| e.g., here's your "trivial" API's spec. Then there are the
| intentional deviations for CORS, e.g., which there is no spec
| for.
|
| https://fetch.spec.whatwg.org
|
| Not to mention the Fetch API isn't available on every other
| platform... maybe you're thinking of the general ability to
| make HTTP requests, but nodejs had that from the start.
| feupan wrote:
| It's not trivial and it's not groundbreaking. Node has had an
| HTTP client since its inception, this is just a different API.
| It's "great" just because it's a decent abstraction that's also
| available in browsers.
|
| The issue in the JavaScript ecosystem for this kind of stuff is
| that it runs on two very different environments and they need
| time to find solutions that work for all the parties involved.
| nfriedly wrote:
| I love fetch, except for the way that it combines headers. If
| more than one of the same header is set (such as multiple set-
| cookie headers), it combines them all into a single comma-
| separated string. I know this is allowed in the HTTP spec, and
| it's probably even a sensible default. But the fetch spec doesn't
| allow any access to the raw headers, so there's no
| straightforward way to get the original uncombined headers back.
| Set-Cookie headers, in particular, often contain commas in their
| expiration dates, so just splitting around `, ` will lead to
| problems.
|
| I took a look through the source of this new fetch API, and it
| seems to have inherited that wart:
| https://github.com/nodejs/undici/blob/2dd3437e20c5a3cc466226...
|
| I've argued with the authors of the fetch spec about this before,
| and ultimately settled on using
| https://www.npmjs.com/package/set-cookie-parser#user-content...
| to work around this flaw. (For clarity, I published the package,
| but chrusart wrote that method - https://github.com/nfriedly/set-
| cookie-parser/pull/19)
| styfle wrote:
| There's a relevant spec issue here
| https://github.com/whatwg/fetch/issues/973
| nfriedly wrote:
| Oh, good. The first comment there references my closed issue
| from 2017, but it's nice to see this being given some more
| consideration.
| easton wrote:
| For those of us unfamiliar, how long until stuff comes out of
| experimental?
| inglor wrote:
| In general "when we feel it's ready". For some features I've
| worked recently (like `flatMap` and friends for streams) I
| think it'll take about a year.
|
| For `fetch` it will take us around a year or two probably.
|
| I wrote more details on the PR itself but I (and others)
| basically feel strongly that it should be as close to compliant
| as possible (Deno have a nice list which they happily shared
| with us of how they diverge and that seems reasonable).
|
| Basically there is a bunch of stuff that should happen first:
|
| - We need to run (and pass) the web platform tests.
|
| - We want to go over the spec (again) and check compliance and
| add more tests and fix bugs.
|
| - We want community feedback on the implementation and to
| improve the DX to be better (while not sacrificing spec
| compliance).
|
| - We want better support for stuff like `File` in core.
|
| - Web streams need to go out of experimental first.
|
| This was a _long_ road that had many stepping stones (like
| EventTarget, AbortController, Blob, ReadableStream etc). It's
| important to get it right.
|
| Worth mentioning the fact it's experimental doesn't mean it's
| not safe to use in simple code in this case - it just means if
| you make your code behave in a way that diverges from the
| specification we may change our implementation to match the
| specification and break your code.
|
| Anything like `await fetch('./someUrl').then(x => x.text())`
| should be fine.
| [deleted]
| quickthrower2 wrote:
| Funny how my brain works, I thought it was there already, but I
| must have got so used to having it as a dependency I forgot that
| it was a dependency not built in. To be fair I don't "Node" all
| the time! It's on and off.
| jitl wrote:
| There's an NPM package containing this implementation called
| 'undici' that you can install in previous versions of Node; the
| package.json claims support back to 12.x.
|
| https://www.npmjs.com/package/undici
|
| However the fetch implementation itself is documented as unstable
| and only supports Node 16.x. I believe this is because it is not
| quite spec compliant yet, so there is latitude for breaking
| changes that make it more spec compliant.
| ezekg wrote:
| One thing I hate about fetch() is that you can't manually follow
| redirects. Using { redirect: 'manual' } doesn't expose the
| Location header of the redirect, so it's essentially useless. I
| know that node-fetch fixed this issue, so I hope the official
| Node fetch() does not have the same problem.
| inglor wrote:
| We are not, this is a place the server-side implementation is
| going to diverge and this was discussed in the PR:
| https://github.com/nodejs/node/pull/41749#issuecomment-10254...
| ezekg wrote:
| This is really, really great to hear. Thanks for all the hard
| work!
___________________________________________________________________
(page generated 2022-02-01 23:00 UTC)