[HN Gopher] GraphCast: AI model for weather forecasting
___________________________________________________________________
GraphCast: AI model for weather forecasting
Author : bretthoerner
Score : 363 points
Date : 2023-11-14 15:42 UTC (7 hours ago)
(HTM) web link (deepmind.google)
(TXT) w3m dump (deepmind.google)
| xnx wrote:
| I continue to be a little confused by the distinction between
| Google, Google Research and DeepMind. Google Research, had made
| this announcement about 24-hour forecasting just 2 weeks ago:
| https://blog.research.google/2023/11/metnet-3-state-of-art-n...
| (which is also mentioned in the GraphCast announcement from
| today)
| mukara wrote:
| DeepMind recently merged with the Brain team from Google
| Research to form `Google DeepMind`. It seems this was done to
| have Google DeepMind focused primarily (only?) on AI research,
| leaving Google Research to work on other things in more than 20
| research areas. Still, some AI research involves both orgs,
| including MetNet in weather forecasting.
|
| In any case, GraphCast is a 10-day global model, whereas MetNet
| is a 24-hour regional model, among other differences.
| xnx wrote:
| Good explanation. Now that both the 24-hour regional and
| 10-day global models have been announced in
| technical/research detail, I supposed there might still be a
| general blog post about how improved forecasting is when you
| search for "weather" or check the forecast on Android.
| mnky9800n wrote:
| That would require your local weather service to use these
| models
| kridsdale3 wrote:
| IIRC the MetNet announcement a few weeks ago said that
| their model is now used when you literally Google your
| local weather. I don't think it's available yet to any API
| that third party weather apps pull from, so you'll have to
| keep searching "weather in Seattle" to see it.
| daemonologist wrote:
| It's also used, at least for the high resolution
| precipitation forecast, in the default Android weather
| app (which is really part of the "Google" app situation).
| danielmarkbruce wrote:
| Is there a colab example (and/or have they released the
| models) for MetNet like they have here for GraphCast?
| mukara wrote:
| MetNet-3 is not open-source, and the announcement said it's
| already integrated into Google products/services needing
| weather info. So, I'd doubt there's anything like a colab
| example.
| robertlagrant wrote:
| This is fascinating:
|
| > For inputs, GraphCast requires just two sets of data: the state
| of the weather 6 hours ago, and the current state of the weather.
| The model then predicts the weather 6 hours in the future. This
| process can then be rolled forward in 6-hour increments to
| provide state-of-the-art forecasts up to 10 days in advance.
| broast wrote:
| Weather is markovian
| Imanari wrote:
| Interesting indeed, only one lagged feature for time series
| forecasting? I'd imagine that including more lagged inputs
| would increase performance. Rolling the forecasts forward to
| get n-step-ahead forecasts is a common approach. I'd be
| interested in how they mitigated the problem of the errors
| accumulating/compounding.
| Al-Khwarizmi wrote:
| I don't know much about weather prediction, but if a model can
| improve the state of the art only with that data as input, my
| conclusion is that previous models were crap... or am I missing
| something?
| postalrat wrote:
| Read the other comments.
| counters wrote:
| It's worth pointing out that "state of the weather" is a little
| bit hand-wavy. The GraphCast model requires a fully-assimilated
| 3D atmospheric state - which means you still need to run a
| full-complexity numerical weather prediction system with a
| massive amount of inputs to actually get to the starting line
| for using this forecast tool.
|
| Initializing directly from, say, geostationary and LEO
| satellite data with complementary surface station observations
| - skipping the assimilation step entirely - is clearly where
| this revolution is headed, but it's very important to
| explicitly note that we're not there yet (even in a research
| capacity).
| baq wrote:
| Yeah current models aren't quite ready to ingest real time
| noisy data like the actual weather... I hear they go off the
| rails if preprocessing is skipped (outliers, etc)
| lispisok wrote:
| I've been following these global ML weather models. The fact they
| make good forecasts at all was very impressive. What is blowing
| my mind is how fast they run. It takes hours on giant super
| computers for numerical weather prediction models to forecast the
| entire globe. These ML models are taking minutes or seconds. This
| is potentially huge for operational forecasting.
|
| Weather forecasting has been moving focus towards ensembles to
| account for uncertainty in forecasts. I see a future of large
| ensembles of ML models being ran hourly incorporating the latest
| measurements
| wenc wrote:
| Not to take away from the excitement but ML weather prediction
| builds upon the years of data produced by numerical models on
| supercomputers. It cannot do anything without that computation
| and its forecasts are dependent on the quality of that
| computation. Ensemble models are already used to quantify
| uncertainty (it's referenced in their paper).
|
| But it is exciting that they are able to recognize patterns in
| multi year and produce medium term forecasts.
|
| Some comments here suggest this replaces supercomputers models.
| This would a wrong conclusion.It does not (the paper explicitly
| states this). It uses their output as input data.
| boxed wrote:
| I don't get this. Surely past and real weather should be the
| input training data, not the output of numerical modeling?
| counters wrote:
| Well, what is "real weather data?"
|
| We have dozens of complementary and contradictory sources
| of weather information. Different types of satellites
| measuring EM radiation in different bands, weather
| stations, terrestrial weather radars, buoys, weather
| balloons... it's a massive hodge-podge of different systems
| measuring different things in an uncoordinated fashion.
|
| Today, it's not really practical to assemble that data and
| directly feed it into an AI system. So the state-of-the-art
| in AI weather forecasting involves using an intermediate
| representation - "reanalysis" datasets which apply a
| sophisticated physics based weather model to assimilate all
| of these data sets into a single, self-consistent 3D and
| time-varying record of the state of the atmosphere. This
| data is the unsung hero of the weather revolution - just as
| the WMO's coordinated synoptic time observations for
| weather balloons catalyzed effective early numerical
| weather prediction in the 50's and 60's, accessible re-
| analysis data - and the computational tools and platforms
| to actually work with these peta-scale datasets - has
| catalyzed the advent of "pure AI" weather forecasting
| systems.
| goosinmouse wrote:
| Great comment, thank you for sharing your insights. I
| don't think many people truly understand just how massive
| these weather models are and the sheer volume of data
| assimilation work that's been done for decades to get us
| to this point today.
|
| I always have a lot of ideas about using AI to solve very
| small scale weather forecasting issues, but there's just
| so much to it. It's always a learning experience for
| sure.
| mnky9800n wrote:
| It uses era5 data which is reanalysis. These models will always
| need the numerical training data. What's impressive is how well
| the emulate the physics in those models so cheaply. But since
| the climate changes there will eventually be different weather
| in different places.
|
| https://www.ecmwf.int/en/forecasts/documentation-and-support
| counters wrote:
| Absolutely - but large ensembles are just the tip of the
| iceberg. Why bother producing an ensemble when you could just
| output the posterior distribution of many forecast predictands
| on a dense grid? One could generate the entire ensemble-derived
| probabilities from a single forward model run.
|
| Another very cool application could incorporate generative
| modeling. Inject a bit of uncertainty in a some observations
| and study how the manifold of forecast outputs changes...
| ultimately, you could tackle things like studying the
| sensitivity of forecast uncertainty for, say, a tropical
| cyclone or nor'easter relative to targeted observations.
| Imagine a tool where you could optimize where a Global Hawk
| should drop rawindsondes over the Pacific Ocean to maximally
| decrease forecast uncertainty for a big winter storm impacting
| New England...
|
| We may not be able to engineer the weather anytime soon, but in
| the next few years we may have a new type of crystal ball for
| anticipating its nuances with far more fidelity than ever
| before.
| kridsdale3 wrote:
| This is basically equivalent to NVIDIA's DLSS machine learning
| running on Tensor Cores to "up-res" or "frame-interpolate" the
| extremely computationally intensive job the traditional GPU
| rasterizer does to simulate a world.
|
| You could numerically render a 4k scene at 120FPS at extreme
| cost, or you could render a 2k scene at 60FPS, then feed that
| to DLSS to get a close-enough approximation of the former at
| enormous energy and hardware savings.
| Gys wrote:
| I live in an area which regularly has a climate differently then
| forecasted: often less rain and more sunny. Would be great if I
| can connect my local weather station (and/or its history) to some
| model and have more accurate forecasts.
| speps wrote:
| Because weather data is interpolated between multiple stations,
| you wouldn't even need the local station position, your own
| position would be more accurate as it'd take a lot more
| parameters into account.
| tash9 wrote:
| One piece of context to note here is that models like ECMWF are
| used by forecasters as a tool to make predictions - they aren't
| taken as gospel, just another input.
|
| The global models tend to consistently miss in places that have
| local weather "quirks" - which is why local forecasters tend to
| do better than, say, accuweather, where it just posts what the
| models say.
|
| Local forecasters might have learned over time that, in early
| Autumn, the models tend to overpredict rain, and so when they
| give their forecasts, they'll tweak the predictions based on
| the model tendencies.
| dist-epoch wrote:
| There are models which take as input both global forecasts and
| local ones, and which then can transpose a global forecast into
| a local one.
|
| National weather institutions sometimes do this, since they
| don't have the resources to run a massive supercomputer model.
| Gys wrote:
| Interesting. So what I am looking for is probably an even
| more scaled down version? Or something that runs in the cloud
| with an api to upload my local measurements.
| supdudesupdude wrote:
| Hate to break it but one weather station wont improve a
| forecast? What are they supposed to do? Ignore the output
| of our state of the art forecast models and add an if
| statement for your specific weather station??
| freedomben wrote:
| weather prediction seems to me like a terrific use of machine
| learning aka statistics. The challenge I suppose is in the data.
| To get perfect predictions you'd need to have a mapping of what
| conditions were like 6 hours, 12 hours, etc before, and what the
| various outcomes were, which butterflies flapped their wings and
| where (this last one is a joke about how hard this data would
| be). Hard but not impossible. Maybe impossible. I know very
| little about weather data though. Is there already such a format?
| tash9 wrote:
| It's been a while since I was a grad student but I think the
| raw station/radiosonde data is interpolated into a grid format
| before it's put into the standard models.
| kridsdale3 wrote:
| This was also in the article. It splits the sphere surface in
| to 1M grids (not actually grids in the cartesian sense of a
| plane, these are radial units). Then there's 37 altitude
| layers.
|
| So there's radial-coordinate voxels that represent a low
| resolution of the physical state of the entire atmosphere.
| serjester wrote:
| To call this impressive is an understatement. Using a single GPU,
| outperforms models that run on the world's largest super
| computers. Completely open sourced - not just model weights. And
| fairly simple training / input data.
|
| > ... with the current version being the largest we can
| practically fit under current engineering constraints, but which
| have potential to scale much further in the future with greater
| compute resources and higher resolution data.
|
| I can't wait to see how far other people take this.
| thatguysaguy wrote:
| They said single TPU machine to be fair, which means like 8
| TPUs (still impressive)
| wenc wrote:
| It builds on top of supercomputer model output and does better
| at the specific task of medium term forecasts.
|
| It is a kind of iterative refinement on the data that
| supercomputers produce -- it doesn't supplant supercomputers.
| In fact the paper calls out that it has a hard dependency on
| the output produced by supercomputers.
| carbocation wrote:
| I don't understand why this is downvoted. This is a classic
| thing to do with deep learning: take something that has a
| solution that is expensive to compute, and then train a deep
| learning model from that. And along the way, your model might
| yield improvements, too, and you can layer in additional
| features, interpolate at finer-grained resolution, etc. If
| nothing else, the forward pass in a deep learning model is
| almost certainly way faster than simulating the next step in
| a numerical simulation, but there is room for improvement as
| they show here. Doesn't invalidate the input data!
| danielmarkbruce wrote:
| Because "iterative refinement" is sort of wrong. It's not a
| refinement and it's not iterative. It's an entirely
| different model to physical simulation which works entirely
| differently and the speed up is order of magnitude.
|
| Building a statistical model to approximate a physical
| process isn't a new idea for sure.. there are literally
| dozens of them for weather.. the idea itself isn't really
| even iterative, it's the same idea... but it's all in the
| execution. If you built a model to predict stock prices
| tomorrow and it generated 1000% pa, it wouldn't be
| reasonable for me to call it iterative.
| kridsdale3 wrote:
| It is iterative when you look at the scope of "humans
| trying to solve things over time".
| danielmarkbruce wrote:
| lol, touche.
| andbberger wrote:
| "amortized inference" is a better name for it
| borg16 wrote:
| > the forward pass in a deep learning model is almost
| certainly way faster than simulating the next step in a
| numerical simulation
|
| Is this the case in most of such refinements (architecture
| wise)?
| danielmarkbruce wrote:
| Practically speaking yes. You'd not likely build a
| statistical model when you could build a good simulation
| of the underlying process if the simulation was already
| really fast and accurate.
| silveraxe93 wrote:
| Could you point me to the part where it says it depends on
| supercomputer output?
|
| I didn't read the paper but the linked post seems to say
| otherwise? It mentions it used the supercomputer output to
| impute data during training. But for prediction it just
| needs:
|
| > For inputs, GraphCast requires just two sets of data: the
| state of the weather 6 hours ago, and the current state of
| the weather. The model then predicts the weather 6 hours in
| the future. This process can then be rolled forward in 6-hour
| increments to provide state-of-the-art forecasts up to 10
| days in advance.
| serjester wrote:
| You can read about it more in their paper. Specifically
| page 36. Their dataset, ERA5, is created using a process
| called reanalysis. It combines historical weather
| observations with modern weather models to create a
| consistent record of past weather conditions.
|
| https://storage.googleapis.com/deepmind-
| media/DeepMind.com/B...
| silveraxe93 wrote:
| Ah nice. Thanks!
| dekhn wrote:
| I can't find the details, but if the supercomputer job
| only had to run once, or a few times, while this model
| can make accurate predictions repeatedly on unique
| situations, then it doesn't matter as much that a
| supercomputer was required. The goal is to use the
| supercomputer once, to create a high value simulated
| dataset, then repeatedly make predictions from the lower-
| cost models.
| whatever1 wrote:
| So best case scenario we can avoid some computation for
| inference, assuming that historical system dynamics are still
| valid. This model needs to be constantly monitored by full
| scale simulations and rectified over time.
| westurner wrote:
| "BLD,ENH: Dask-scheduler (SLURM,)," https://github.com/NOAA-
| EMC/global-workflow/issues/796
|
| Dask-jobqueue https://jobqueue.dask.org/ :
|
| > _provides cluster managers for PBS, SLURM, LSF, SGE and
| other [HPC supercomputer] resource managers_
|
| Helpful tools for this work: Dask-labextension, DaskML, CuPY,
| SymPy's lambdify(), Parquet, Arrow
|
| GFS: Global Forecast System:
| https://en.wikipedia.org/wiki/Global_Forecast_System
|
| TIL about Raspberry-NOAA and pywws in researching and
| summarizing for a comment on "Nrsc5: Receive NRSC-5 digital
| radio stations using an RTL-SDR dongle" (2023)
| https://news.ycombinator.com/item?id=38158091
| pkulak wrote:
| Why can't they just train on historical data?
| xapata wrote:
| We don't have enough data. There's only one universe, and
| it's helpful to train on counter-factual events.
| meteo-jeff wrote:
| In case someone is looking for historical weather data for ML
| training and prediction, I created an open-source weather API
| which continuously archives weather data.
|
| Using past and forecast data from multiple numerical weather
| models can be combined using ML to achieve better forecast skill
| than any individual model. Because each model is physically
| bound, the resulting ML model should be stable.
|
| See: https://open-meteo.com
| boxed wrote:
| Open-Meteo has a great API too. I used it to build my iOS
| weather app Frej (open source and free:
| https://github.com/boxed/frej)
|
| It was super easy and the responses are very fast.
| mdbmdb wrote:
| Is it able to provide data on extreme events. Say, the current
| and potential path of a hurricane? similar to .kml that NOAA
| provides
| meteo-jeff wrote:
| Extreme weather is predicted by numerical weather models.
| Correctly representing hurricanes has driven development on
| the NOAA GFS model for centuries.
|
| Open-Meteo focuses on providing access to weather data for
| single locations or small areas. If you look at data for
| coastal areas, forecast and past weather data will show
| severe winds. Storm tracks or maps are not available, but
| might be implemented in the future.
| mdbmdb wrote:
| Appreciate the response. Do you know of any services that
| provide what I described in the previous comments? I'm
| specifically interested in extreme weather conditions and
| their visual representation (hurricanes, tornados, hails
| etc.) with API capabilities
| swells34 wrote:
| Go to: nhc.noaa.gov/gis There's a list of data and
| products with kmls and kmzs and geojsons and all sorts of
| stuff. I haven't actually used the API for retrieving
| these, but NOAA has a pretty solid track record with data
| dissemination.
| dmd wrote:
| I would love to hear about this centuries-old NOAA GFS
| model. The one I know about definitely doesn't have that
| kind of history behind it.
| K2h wrote:
| Some of the oldest data may come from ships logs back to
| 1836
|
| https://www.reuters.com/graphics/CLIMATE-CHANGE-ICE-
| SHIPLOGS...
| meteo-jeff wrote:
| Sorry, decades.
|
| KML files for storm tracks are still the best way to go.
| You could calculate storm tracks yourself for other weather
| models like DWD ICON, ECMWF IFS or MeteoFrance ARPEGE, but
| storm tracks based on GFS ensembles are easy to use with
| sufficient accuracy
| comment_ran wrote:
| How about https://pirateweather.net/en/latest/ ?
|
| Does anyone have a compare this API with the latest API we have
| here?
| meteo-jeff wrote:
| Both APIs use weather models from NOAA GFS and HRRR,
| providing accurate forecasts in North America. HRRR updates
| every hour, capturing recent showers and storms in the
| upcoming hours. PirateWeather gained popularity last year as
| a replacement for the Dark Sky API when Dark Sky servers were
| shut down.
|
| With Open-Meteo, I'm working to integrate more weather
| models, offering access not only to current forecasts but
| also past data. For Europe and South-East Asia, high-
| resolution models from 7 different weather services improve
| forecast accuracy compared to global models. The data covers
| not only common weather variables like temperature, wind, and
| precipitation but also includes information on wind at higher
| altitudes, solar radiation forecasts, and soil properties.
|
| Using custom compression methods, large historical weather
| datasets like ERA5 are compressed from 20 TB to 4 TB, making
| them accessible through a time-series API. All data is stored
| in local files; no database set-up required. If you're
| interested in creating your own weather API, Docker images
| are provided, and you can download open data from NOAA GFS or
| other weather models.
| Fatnino wrote:
| Is there somewhere to see historical forecasts?
|
| So not "the weather on 25 December 2022 was such and such" but
| rather "on 20 December 2022 the forecast for 25 December 2022
| was such and such"
| meteo-jeff wrote:
| Not yet, but I am working towards it:
| https://github.com/open-meteo/open-meteo/issues/206
| Vagantem wrote:
| That's awesome! I've hooked something similar up to my service
| - https://dropory.com which predicts which day it will rain the
| least for any location
|
| Based on historical data!
| willsmith72 wrote:
| this is really cool, I've been looking for good snow-related
| weather APIs for my business. I tried looking on the site, but
| how does it work, being coordinates-based?
|
| I'm used to working with different weather stations, e.g.
| seeing different snowfall prediction at the bottom of a
| mountain, halfway up, and at the top, where the coordinates are
| quite similar.
| amluto wrote:
| I've never studied weather forecasting, but I can't say I'm
| surprised. All of these models, AFAICT, are based on the "state"
| of the weather, but "state" deserves massive scare quotes: it's a
| bunch of 2D fields (wind speed, pressure, etc) -- note the _2D_.
| Actual weather dynamics happen in three dimensions, and three
| dimensional land features, buildings, etc as well as gnarly 2D
| surface phenomena (ocean surface temperature, ground surface
| temperature, etc) surely have strong effects.
|
| On top of this, surely the actual observations that feed into the
| model are terrible -- they come from weather stations, sounding
| rockets, balloons, radar, etc, none of which seem likely to be
| especially accurate in all locations. Except that, where a
| weather station exists, the output of that station _is_ the
| observation that people care about -- unless you 're in an
| airplane, you don't personally care about the geopotential, but
| you do care about how windy it is, what the temperature and
| humidity are, and how much precipitation there is.
|
| ISTM these dynamics ought to be better captured by learning them
| from actual observations than from trying to map physics both
| ways onto the rather limited datasets that are available. And a
| trained model could also learn about the idiosyncrasies of the
| observation and the extra bits of forcing (buildings, etc) that
| simply are not captured by the inputs.
|
| (Heck, my personal in-my-head neural network can learn a mapping
| from NWS forecasts to NWS observations later in the same day that
| seems better than what the NWS itself produces. Surely someone
| could train a very simple model that takes NWS forecasts as
| inputs and produces its estimates of NWS observations during the
| forecast period as outputs, thus handling things like "the NWS
| consistently underestimates the daily high temperature at such-
| and-such location during a summer heat wave.")
| Difwif wrote:
| I'm not sure why you're emphasizing that weather forecasting is
| just 2D fields. Even in the article they mention GraphCast
| predicts multiple data points at each global location across a
| variety of altitudes. All existing global computational
| forecast models work the same way. They're all 3d spherical
| coordinate systems.
| WhitneyLand wrote:
| How does it make sense to say this is something you've "never
| studied", followed by how they "ought to be" doing it better?
|
| It also seems like some of your facts differ from theirs, may I
| ask how far you read into the paper?
| kridsdale3 wrote:
| No need, they're a software engineer (presumably). That just
| means they're better than everyone.
| cryptoz wrote:
| Again haha! Still no mention of using barometers in phones. Maybe
| some day.
| user_7832 wrote:
| (If someone with knowledge or experience can chime in, please
| feel free.)
|
| To the best of my knowledge, poor weather (especially wind
| shear/microbursts) are one of the most dangerous things possible
| in aviation. Is there any chance, or plans, to implement this in
| the current weather radars in planes?
| tash9 wrote:
| If you're talking about small scale phenomena (less than 1km),
| then this wouldn't help other than to be able to signal when
| the conditions are such that these phenomena are more likely to
| happen.
| jauntywundrkind wrote:
| From what I can tell from reading & based off
| https://colab.research.google.com/github/deepmind/graphcast/... ,
| one needs access to ECMWF Era5 or HRES data-sets or something
| similar to be able to run and use this model.
|
| Unknown what licensing options ECMWF offers for Era5, but to use
| this model in any live fashion, I think one is probably going to
| need a small fortune. Maybe some other dataset can be adapted
| (likely at great pain)...
| sunshinesnacks wrote:
| ERA5 is free. The API is a bit slow.
|
| I think that only some variables from the HRES are free, but
| not 100% sure.
| sagarpatil wrote:
| How does one go about hosting this and using this as an API?
| syntaxing wrote:
| Maybe I missed it but does anyone know what it will take to run
| this model? Seems something fun to try out but not sure if 24GB
| of VRAM is suffice.
| kridsdale3 wrote:
| It says in the article that it runs on Google's tensor units.
| So, go down to your nearest Google data center, dodge security,
| and grab one. Then escape the cops.
| azeirah wrote:
| You could also just buy a very large amount of their coral
| consumer TPUs :D
| comment_ran wrote:
| So for a daily user, to make it a practical usage, let's say if I
| have a local measurement of X, I can predict, let's say, 10 days
| later, or even just tomorrow, or the day after tomorrow, let's
| say the wind direction, is it possible to do that?
|
| If it is possible, then I will try using the sensor to measure my
| velocity at some place where I live, and I can run the model and
| see how the results look like. I don't know if it's going to
| accurately predict the future or within a 10% error bar range.
| dist-epoch wrote:
| No, this model uses as input the current state of the weather
| across the whole planet.
| carabiner wrote:
| > GraphCast makes forecasts at the high resolution of 0.25
| degrees longitude/latitude (28km x 28km at the equator).
|
| Any way to run this at even higher resolution, like 1 km? Could
| this resolve terrain forced effects like lenticular clouds on
| mountain tops?
| dist-epoch wrote:
| One big problem is input weather data. It's resolution is poor.
| carabiner wrote:
| Yeah, not to mention trying to validate results. Unless we
| grid install weather stations every 200 m on a mountain
| top...
| max_ wrote:
| I have far more respect for the AI team at DeepMind even thou
| they may be less popular than say OpenAI or "Grok".
|
| Why? Other AI studios seem to work on gimmicks while DeepMind
| seems to work on genuinely useful AI applications [0].
|
| Thanks for the good work!
|
| [0] Not to say that Chat GPT & Midjourney are not useful, I just
| find DeepMinds quality of research more interesting.
| max_ wrote:
| Has anyone here heard of "Numerical Forecasting" models for
| weather? I heard they "work so well".
|
| Does GraphCast come close to them?
| max_ wrote:
| What's the difference between a "Graph Neural Network" and a deep
| neural network?
| dil8 wrote:
| Graph neural networks are deep learning models that trained on
| graph data.
| RandomWorker wrote:
| Do you have any resources where I could learn more about
| these networks?
| haolez wrote:
| Are there any experts around that can chime in on the possible
| impacts of this technology if widely adopted?
| supdudesupdude wrote:
| It doesnt predict rainfall so i doubt most of us will actually
| care about it until then. Still it depends on input data (the
| current state of weather etc). How are we supposed to
| accurately model the weather at every point in the world?
| Especially when tech bro Joe living in San Fran expects things
| to be accurate to a meter within his doorstep
| miserableuse wrote:
| Does anybody know if its possible to initialize the model using
| GFS initial conditions used for the GFS HRES model? If so, where
| can I find this file and how can I use it? Any help would be
| greatly appreciated!
| counters wrote:
| You can try, but other models in this class have struggled when
| initialized using model states pulled from other analysis
| systems.
|
| ECMWF publishes a tool that can help bootstrap simple inference
| runs with different AI models [1] (they have plugins for
| several). You could write a tool that re-maps a GDAS analysis
| to "look like" ERA-5 or IFS analysis, and then try feeding it
| into GraphCast. But YMMV if the integration is stable or not -
| models like PanguWx do not work off-the-shelf with this
| approach.
|
| [1]: https://github.com/ecmwf-lab/ai-models
| miserableuse wrote:
| Thank you for your response. Are these ML models initialized
| by gridded initial conditions measurements (such as the GDAS
| pointed out) or by NWP model forecast results (such as hour-
| zero forecast from the GFS)? Or are those one and the same?
| counters wrote:
| They're more-or-less the same thing.
| pyb wrote:
| Curious. How can AI/ML perform on a problem that is, as far as I
| understand, inherently chaotic / unpredictable ? It sounds like a
| fundamental contradiction to me.
| vosper wrote:
| Weather isn't fundamentally unpredictable. We predict weather
| with a fairly high degree of accuracy (for most practical
| uses), and the accuracy getting better all the time.
|
| https://scijinks.gov/forecast-reliability
| sosodev wrote:
| I'm kinda surprised that this government science website
| doesn't seem to link sources. I'd like to read the research
| to understand how they're measuring the accuracy.
| keule wrote:
| IMO a chaotic system will not allow for long-term forecast, but
| if there is any type of pattern to recognize (and I would
| assume there are plenty), an AI/ML model should be able to
| create short-term prediction with high accuracy.
| pyb wrote:
| Not an expert, but "Up to 10 days in advance" sounds like
| long-term to me ?
| joaogui1 wrote:
| I think 10 days is basically the normal term for weather,
| in that we can get decent predictions for that span using
| "classical"/non-ML methods.
| pyb wrote:
| IDK, I wouldn't plan a hike in the mountains based on
| 10-day predictions.
| keule wrote:
| To be clear: With short-term I meant the mentioned 6 hours
| of the article. They use those 6 hours to create forecasts
| for up to 10 days. I would think that the initial
| predictors for a phenomenon (like a hurricane) are well
| inside that timespan. With long-term, I meant way beyond a
| 14-day window.
| kouru225 wrote:
| But AI/ML models require good data and the issue with chaotic
| systems like weather is that we don't have good enough data.
| joaogui1 wrote:
| The issue with chaotic systems is not data, is that the
| error grows superlinearly with time, and since you always
| start with some kind of error (normally due to measurement
| limitations) this means that after a certain time horizon
| the error becomes to significant to trust the prediction.
| That hasn't a lot to do with data quality for ML models
| kouru225 wrote:
| That's an issue with data: If your initial conditions are
| wrong (Aka your data collection has any error or isn't
| thorough enough) then you get a completely different
| result.
| kouru225 wrote:
| Yes. Very accurate as long as you don't need to predict the
| unpredictable. So it's useless.
|
| Edit: I do see a benefit to the idea if you compare it to the
| Chaos Theorists "gaining intuition" about systems.
| pyb wrote:
| IDK if it's useless, but it's counter-intuitive to me.
| simonebrunozzi wrote:
| Amazing. Is there an easy way to run this on a local laptop?
| dnlkwk wrote:
| Curious how this factors in long-range shifts or patterns eg el
| nino. Most accurate is a bold claim
| stabbles wrote:
| If you live in a country where local, short-term rain / shower
| forecast is essential (like [1] [2]), it's funny to see how
| incredibly bad radar forecast is.
|
| There are really convenient apps that show an animated map with
| radar data of rain, historical data + prediction (typically).
|
| The prediction is always completely bonkers.
|
| You can eyeball it better.
|
| No wonder "AI" can improve that. Even linear extrapolation is
| better.
|
| Yes, local rain prediction is a different thing from global
| forecasting.
|
| [1] https://www.buienradar.nl [2]
| https://www.meteoschweiz.admin.ch/service-und-publikationen/...
| bberenberg wrote:
| Interesting that you say this. I spent in month in AMS 7-8
| years ago and buienradar was accurate down to the minute when I
| used it. Has something changed?
| supdudesupdude wrote:
| Funny to mention. None of the AI forecasts can actually predict
| precip. None of them mention this and i assume everyone thinks
| this means the rain forecasts are better. Nope just temperature
| and humidity and wind. Important but come on, it's a bunch of
| shite
| brap wrote:
| Beyond the difficulty of running calculations (or even accurately
| measuring the current state), is there a reason to believe
| weather is unpredictable?
|
| I would imagine we probably have a solid mathematical model of
| how weather behaves, so given enough resources to measure and
| calculate, could you, in theory, predict the daily weather going
| 10 years into the future? Or is there something inherently
| "random" there?
| danbrooks wrote:
| Small changes in initial state can lead to huge changes down
| the line. See: the butterfly effect or chaos theory.
|
| https://en.wikipedia.org/wiki/Chaos_theory
| ethanbond wrote:
| AFAIK there's nothing _random_ anywhere except near atomic
| /subatomic scale. Everything else is just highly chaotic/hard-
| to-forecast deterministic causal chains.
| counters wrote:
| What you're describing is effectively how climate models work;
| we run a physical model which solves the equations that govern
| how the atmosphere works out forward in time for very long time
| integrations. You get "daily weather" out as far as you choose
| to run the model.
|
| But this isn't a "weather forecast." Weather forecasting is an
| initial value problem - you care a great deal about how the
| weather will evolve from the current atmospheric conditions.
| Precisely because weather is a result of what happens in this
| complex, 3D fluid atmosphere surrounding the Earth, it happens
| that small changes in those initial conditions can have a very
| big impact on the forecast on relatively short time-periods -
| as little as 6-12 hours. Small perturbations grow into larger
| ones and feedback across spatial scales. Ultimately, by day
| ~3-7, you wind up with a very different atmospheric state than
| what you'd have if you undid those small changes in the initial
| conditions.
|
| This is the essence of what "chaos" means in the context of
| weather prediction; we can't perfectly know the initial
| conditions we feed into the model, so over some relatively
| short time, the "model world" will start to look very different
| than the "real world." Even if we had perfect models - capable
| of representing all the physics in the atmosphere - we'd still
| have this issue as long as we had to imperfectly sample the
| atmosphere for our initial conditions.
|
| So weather isn't inherently "unpredictable." And in fact, by
| running lots of weather models simultaneously with slightly
| perturbed initial conditions, we can suss out this uncertainty
| and improve our estimate of the forecast weather. In fact, this
| is what's so exciting to meteorologists about the new AI models
| - they're so much cheaper to run that we can much more
| effectively explore this uncertainty in initial conditions,
| which will indirectly lead to improved forecasts.
| willsmith72 wrote:
| is it possible to self-correct, looking at initial value
| errors in the past? Is it too hard to prescribe the error in
| the initial value?
| supdudesupdude wrote:
| I'll be impressed when it can predict rainfall better than GFS /
| HRRR / EURO etc
| Vagantem wrote:
| Related to this, I built a service that shows what day it has
| rained the least on in the last 10 years - for any location and
| month! Perfect to find your perfect wedding date. Feel free to
| check out :)
|
| https://dropory.com
| knicholes wrote:
| What are the similarities between weather forecasting and
| financial market forecasting?
| KRAKRISMOTT wrote:
| Both are complex systems traditionally modeled with
| differential equations and statistics.
| sonya-ai wrote:
| Well it's a start, but weather forecasting is far more
| predictable imo
| csours wrote:
| Makes me wonder how much it would take to do this for a city at
| something like 100 meter resolution.
| layoric wrote:
| I can't see any citation to accuracy comparisons, or maybe I just
| missed them? Given the amount of data, and complexity of the
| domain, it would be good to see a much more detailed breakdown of
| their performance vs other models.
|
| My experience in this space is that I was first employee at
| Solcast building a live 'nowcast' system for 4+ years (left
| ~2021) targeting solar radiation and cloud opacity initially, but
| expanding into all aspects of weather, focusing on the use of the
| newer generation of satellites, but also heavily using NWP models
| like ECMWF. Last I knew,nowcasts were made in minutes on a decent
| size cluster of systems, and has been shown in various studies
| and comparisons to produce extremely accurate data (This article
| claims 'the best' without links which is weird..), be interesting
| on how many TPUsv4 were used to produce these forecasts and how
| quickly? Solcast used ML as a part of their systems, but when it
| comes down to it, there is a lot more operationally to producing
| accurate and reliable forecasts, eg it would be arrogant to say
| the least to switch from something like ECMWF to this black box
| anytime soon.
|
| Something I said as just before I left Solcast was that their
| biggest competition would come from Amazon/Google/Microsoft and
| not other incumbent weather companies. They have some really
| smart modelers, but its hard to compete with big tech resources.
| I believe Amazon has been acquiring power usage IoT related
| companies over the past few years, I can see AI heavily moving
| into that space as well.. for better or worse.
___________________________________________________________________
(page generated 2023-11-14 23:00 UTC)