[HN Gopher] Show HN: Weather API for non-commercial use
___________________________________________________________________
Show HN: Weather API for non-commercial use
Author : meteo-jeff
Score : 132 points
Date : 2021-09-12 10:18 UTC (1 days ago)
(HTM) web link (open-meteo.com)
(TXT) w3m dump (open-meteo.com)
| [deleted]
| syedkarim wrote:
| How do your models compare to those from NOAA?
|
| https://nomads.ncep.noaa.gov/
| ricksunny wrote:
| Looks great! I really like the UI to make it clear what can be
| had from the API - it really facilitates the on-boarding process.
| Have got a couple of questions if you don't mind? :
|
| * What is the source of the soil moisture data?
|
| * Also can you describe which ET model is being applied and which
| of its parameter values have to be assumed?
| meteo-jeff wrote:
| * Soil moisture data is based on DWD ICON models with up to 2
| km resolution in central Europe.
|
| * ET is based on latent heat flux. This is not the potential
| evapotranspiration or ET0 reference evapotranspiration. I did
| not dive deep into the actual radiative transfer model for
| latent heat flux, but you can find it in the DWD ICON
| description.
| flippyhead wrote:
| This is really great. I've been looking for exactly this kind of
| tool as I look at possible projects related to aviation. Thank
| you for this.
| salamander014 wrote:
| I've used the Open Weather Map API in the past
| (https://openweathermap.org/api).
|
| It's free for a reasonable number of requests and has a solid API
| and a good variety of functionality.
|
| Yours looks well designed! I will give it a shot if I ever need
| to revisit my local weather script.
|
| Did you chose swift for any reason besides existing knowledge?
| Python can be compiled down to a binary pretty easily, along with
| many other languages. Just curious. Cheers.
| meteo-jeff wrote:
| My personal preference for Swift was for sure a major aspect. I
| also like to profile my code and see how much I can out of it.
| It is more an academical challenge to compare performance mmap
| vs file reads calls, but I enjoy it :)
|
| With Python I always struggled to get performance to a very
| high level. I am sure it is possible, but Swift with some C
| code was more natural for me.
| vorpalhex wrote:
| Looks great. What is the funding model for the project? What are
| the current costs?
| meteo-jeff wrote:
| There is no funding. Costs are negligible. Currently it is
| running on a cheap rented VM. Per single VM, it should scale to
| 3+ million API calls per days easily.
|
| To keep it simple, I also do not want to have any income with
| it. In case larger API consumers are on the horizon, I will
| contact them and ask to sponsor a couple more VMs.
| counters wrote:
| Can you share who your vendor/host is? In another comment you
| mentioned that you're mmap'ing files on SSDs... on vendors
| like GCP that sort of hardware (especially the storage volume
| you'd need to have access to a handful of the local/global
| models, as you reference on your website) isn't particularly
| cheap if you want to have 100% uptime.
|
| A cheaper/scalable approach instead would be to re-process
| your data into an appropriately chunked, cloud-optimized
| storage format like Zarr and save it in object storage. Then
| your scaling bottleneck would just be the VMs or compute you
| use to query from object storage, as a function of
| traffic/load.
| meteo-jeff wrote:
| The host is netcup in Germany. I use regular KVMs.
|
| Yes, the amount of storage can be an issue, but I want to
| stay below 500GB of hot-data. One bottleneck is network
| traffic to copy updated files after each weather model
| update.
|
| My binary files are just plain Float16 files without an
| meta data. Logically they similar to 3D Zarr files. I know
| exactly which bytes I have to read and the kernel cache
| helps a look to keep it fast.
|
| In theory this data does not have to be a file on SSD. I
| could also use a block storage directly or request data
| from S3 via range requests.
|
| One hopelessly over-engineered approach would be to use an
| `AWS EBS io2 Block Express Volume` use `Multi-Attach` and
| spawn up to 16 API instances to serve data from it.
| 0des wrote:
| What a lad, thanks!
| dylan604 wrote:
| I like the UI generating the URL to send to the API vs just a PDF
| of all of the possible key/value pairs to generate manually. A
| little above&beyond for most APIs I've used personally. Its
| effort not lost on me.
| fifthofhisname wrote:
| > All data is offered for non-commercial use. With speedy APIs,
| all data can be served by just a couple of virtual machines for
| less than a coffee a day.
|
| Very impressive! What does your backend stack look like? Are you
| using any caching or does every API call hit the binary file?
|
| > The project went live 2 weeks ago and is slowly being used.
|
| What kind of traffic are you getting?
|
| Looking forward to future updates!
| meteo-jeff wrote:
| The API backend is written in Swift with some code in C
| (conversion Float16, interpolations, some solar radiation
| libraries). Data are read directly via mmap from local NVMe SSD
| disks. To scale, I can add new VMs and copy my binary files.
|
| There is no frontend cache and I want to keep it this way. Next
| steps include point-of-presence API virtual machines in
| different counties. Via GeoDNS this is even faster than a CDN.
|
| Peak API usage was around 50.000 API calls per hour, but on
| average it is quite low.
| ricksunny wrote:
| What would it take for it to also be offered on a commercial-
| use basis?
| BHSPitMonkey wrote:
| Looks nice! I love seeing more free options in this space. I'll
| be keeping an eye out for support in Home Assistant[1] :)
|
| [1] https://www.home-assistant.io/integrations/#weather
| sails wrote:
| Thats cool, I'll be having a closer look. I'm working on an xbar
| plugin app [1][2] that alerts me when there is a warmer day in
| the coming week.
|
| I used VisualCrossing [3] which seems quite good, and crucially
| allows 2 weeks historical data, which I think is quite rare. It
| looks like you offer 2? Darksky did not offer this feature I
| don't think.
|
| For my use the accuracy is not critical.
|
| [1] https://github.com/mattarderne/clearmycal
|
| [2] xbarapp.com/
|
| [3] https://www.visualcrossing.com/weather-api
| meteo-jeff wrote:
| For now I only offer 2 days of past weather data.
|
| Most likely I can store some basic weather variables like
| temperature, humidity, wind, precipitation and weather code for
| a couple of weeks. I will keep you request in my backlog:
| https://github.com/open-meteo/open-meteo/issues/27 Thanks!
| sails wrote:
| Great! I'll take a closer look at the API soon Thanks
| cetra3 wrote:
| This is great, I'd love to have dive visibility predictions for
| my webapp based on location: https://divedb.net/
|
| The challenge I've had is the same, the data format that a lot of
| these sources provide is in "academic" formats like GRIB, NetCDF
| etc.. and I've spent a not-insignificant amount of time trying to
| work through all these formats and what they represent.
| meteo-jeff wrote:
| Do you know an open-data source for ocean visibility data?
|
| I want to integrate regular wave forecasts in the next month,
| but I am unaware of something that can improve dive visibility.
| cetra3 wrote:
| Dive visibility is actually quite hard to forecast (at least
| in the areas I dive) and I don't know if there have been
| empirical studies on it.
|
| Anecdotally I find it's best when the wind is offshore and
| there is low tidal movement. But there are other factors at
| play, such as rainfall, etc..
|
| This is one source I found (for wave height):
| https://www.aviso.altimetry.fr/en/data/products.html, however
| it was not clear on the website whether it was paid or not.
|
| Another one I had a poke around with:
| https://www.ecmwf.int/en/forecasts/datasets/wmo-essential
| (trying to read Grib2)
| meteo-jeff wrote:
| Creator from Open-Meteo here, I build a small, but very fast and
| precise weather forecast API for non-commercial use. I am a
| private individual working on it in my spare time.
|
| Open-Meteo started as an exercise to process weather model data
| from the German weather service with up to 2 km resolution. Their
| forecasts are great, but hard to use for non-data-scientists who
| regularly work with NetCDF and GRIB-files. Using this data in
| simple apps, websites, your home-automation software, or robot
| lawn mower is complex.
|
| The Open-Meteo API makes using this data easier. APIs accept
| standard WGS84 coordinates and return a weather forecast for 7
| days in hourly resolution.
|
| The forecast quality is surprisingly good. Open-Meteo includes
| global and regional weather models. Global models use 11 km
| resolution with up to 180 hours of forecast. Local models vary
| between 2 and 7 km resolution and 48 to 72 hours. Updates every 3
| hours. The best model is automatically selected and combined to
| produce a single 7-day hourly forecast. Currently the best
| forecast model coverage is in Europe. Models for North America
| will be integrated next.
|
| Under the hood, all data is stored in binary files using Float16
| and updated in-place after a new weather model arrives. The API
| is very efficient. Returning weather forecasts takes usually less
| than 5 milliseconds. Internet latency is usually much higher.
|
| All data is offered for non-commercial use. With speedy APIs, all
| data can be served by just a couple of virtual machines for less
| than a coffee a day.
|
| What's next? Some important features are still missing like daily
| aggregations, additional weather models, ocean, and air quality
| forecasts. Additionally, I would like to deploy some servers in
| North America and Asia to improve latency.
|
| The project went live 2 weeks ago and is slowly being used. I
| would be grateful for feedback, suggestions, ideas, and
| questions.
|
| All documentation can be found at https://open-meteo.com/en/docs
| filleokus wrote:
| Looks really awesome! Very quick responses.
|
| I would be interested in seeing the implementation of the
| service, interesting choice going with Swift. I'm guessing your
| using something like Vapor for hosting the API?
|
| How are the files designed? I'm guessing you have some cheap way
| of mapping (lat, long) into a file location? Maybe just by
| truncating the coordinates to your precision and mapping them
| onto the file? Using some fancy-pants Uber Hexagons[0]. How is
| the forecast delivered?
|
| Hmm! Many questions :-). I've been thinking lately of similar
| (but not for weather) "geographical" API's, and how to
| store/query data cheaply, so this interested me :-)
|
| [0]: https://github.com/uber/h3
| meteo-jeff wrote:
| Yes, the API part is using Vapor. But there is not much code
| that actually relies on Vapor. It is mostly doing request
| parameter parsing and routing.
|
| I use simple 2D lat lon grids for different regions and
| resolutions. E.g. the global grid uses 2879x1441 pixel grid.
| The third dimension is time. All data are stored directly as
| floating points on disk. Because I know the dimensions of the
| grid, I can read data exactly at the correct position on disk.
| I use Float16 as well, which saves me 50% disk space compared
| to Float32.
|
| Fancy hexagons like H3 are not necessary. They could reduce
| storage by ~30%, but require much more effort and I have to
| "modify" the forecast. I keep forecast data from national
| weather services as untouched as possible.
| ubergesundheit wrote:
| Very nice! Thank you for posting this.
|
| I run a cronjob which sends me the forecast for the next day
| based on german weather service MOSMIX forecasts. I am using
| https://brightsky.dev for this, which makes the clunky xml files
| of MOSMIX available as a http api.
|
| I'll definitely check out open-meteo
| meteo-jeff wrote:
| I also tried brightsky.dev, but it is limited to weather
| stations in the MOSMIX system. IRRC only Germany. In terms of
| forecast quality, MOSMIX should be great!
|
| One goal of mine, would be to provide historical data of the
| past months to enable users to run their own statistical
| forecasts combined with weather models. It is actually quite
| easy to use measurement data and correct forecasts using simple
| machine learning models like random forrest. If you compare
| wind speed measures and forecasts, there usually is a simple
| statistical error that can be reduced easily. Maybe in the next
| month I will to some tests and example code
| xcambar wrote:
| how do you compare with Kachelmann and kachelmannwetter.com in
| terms of data?
|
| For visualization, the UI there is great for a trained eye only,
| yours seems more accessible. Congrats on that point!
| meteo-jeff wrote:
| Kachelmann seems to be more media oriented. I am not aware that
| there is a free API offer.
|
| It looks like Kachelmann is also integrating weather forecasts
| from different national weather services and the paid version
| from ECMWF. I am sure the "HD" forecasts mainly use DWD ICON
| models as well. In this case, data quality would be the same.
|
| I also spend some time to carefully integrate local and global
| weather models. For solar radiation forecasts, a clear sky
| radiation model is used to correctly interpolate data to 1h
| resolution. In the end it is a tradeoff between simple API with
| low operation cost and data quality/amount.
| Leftium wrote:
| I've investigated and tried several (free) weather API's for my
| web app: https://uw.leftium.com.
|
| Open-Meteo looks pretty good! The only thing that seems to be
| missing for me is precipitation probability. The weathercode is
| sort of a proxy for this... I'm also interested in sunrise/sunset
| times; direct_radiation is kind of a proxy for this.
|
| Kudos for providing optional historical data with the same API
| call for the forecast. Many weather API's don't provide
| historical data, and even if they do, it requires extra calls. My
| weather app charts the previous two days of weather with the
| forecast for comparison. I feel this gives a more intuitive sense
| of the weather vs. raw numbers because weather is very relative.
| ("Warm" vs "cool" depends on your location and season.)
|
| In addition, I am in the process of adding AQI forecasts, which
| requires even more network calls. It seems like this is on the
| roadmap for open-meteo. I was surprised to find there are so many
| different standards for AQI. Curious to know which one you plan
| to use.
|
| One possible suggestion for optimizing the output format: sending
| seconds since the Unix epoch would save a few bytes per
| timestamp. I'm not sure if this would make any noticeable
| difference with gzip compression. The current datetime format is
| much more human-readable and may save a conversion before
| displaying.
|
| These were the best (free) weather API's I could find. It's
| interesting how the three different weather forecasts can
| disagree so much:
|
| - https://openweathermap.org/api
|
| - https://www.visualcrossing.com/weather-api
|
| - https://darksky.net/dev (deprecated)
|
| When I find the time, I will add open-meteo as to my app! I'll
| probably have more feedback then.
| meteo-jeff wrote:
| Thanks for your suggestions!
|
| I will add precipitation probability as soon as I include
| 14-day ensemble weather models. Deterministic whether models
| usually do not offer this directly.
|
| I also really like the 2 days historical access :) At some
| point I would like to add more, but the storage requirements
| skyrocket quickly. Not only in space, but also disk IO to
| perform in-place updates.
|
| For air quality data I want to add first all variables
| individually. AQI can be calculated afterwards on-the-fly. Some
| AQI indices require data from the previous hours, but it should
| work well with 2 days past data. For sure I will directly
| integrate the European and US AQI.
|
| I considered unix-timestamp instead of ISO9601 string
| timestamps. Working with unix-timestamps is a bit trickier if
| you also want to use timezones. For the start, i settled with
| ISO8601 timestamps. I might consider adding unix timestamps as
| an `&timeformat=unix` parameter to the API. Regarding saving
| bytes in the output: I would like to implement protobuf as a
| format and also use it to build some SDKs. Especially with
| large floating point arrays, this saves a lot of parsing time
| and preserves precision.
|
| All your suggestions are now kept in the Open-Meteo backlog
| https://github.com/open-meteo/open-meteo/issues
| Leftium wrote:
| Cool! I think unix timestamps should always be in UTC (no
| timezone). But the timezone may affect the exact time range
| of the results.
|
| What does "starts at 0:00 today" mean? Ideally, it is 0:00 of
| the local timezone. (Some other weather API's mess this up!)
| meteo-jeff wrote:
| At whatever time of day you call the API, data always
| starts at 0:00. Either UTC or the time zone you selected.
|
| Many weather APIs follow weather model runs. Therefore in
| the evening, data suddenly starts at 12:00 UTC regardless
| of the timezone. It is quite a pain, if you want to build
| an app displaying todays data, but in the evening you do
| not get data from the morning anymore.
|
| With Open-Meteo multiple weather model runs are merged
| together and partly overwritten. At around 4:00 UTC, the
| first weather model with data starting at 0:00 UTC arrived
| (usually called 0z run). 3 hours later, the 3z run arrives
| with data starting at 3:00 UTC. After a couple or model
| runs arrive, it is quite a mix out of different weather
| model runs. If done right, you notice nothing of this
| behaviour ;) < 0z run > < 3z
| run> < 6z run> < 9z run>
|
| Data is constantly merged into a time-series. The API then
| selects the appropriate start time, which is set to 0:00
| UTC (or if a timezone is set, local-time at 0:00, which
| could be 4:00 UTC depending on the utc offset).
|
| Yes, unix-timestamps must be in UTC+0. If a timezone is
| set, data still starts at "2021-09-13T00:00", but this is
| now local-time. With a 4 hour UTC offset, I would have to
| set the unix-timestamp to "2021-09-12T20:00" and the
| developer has to correctly reapply the UTC offset again to
| get local time. This can be done, but is prone to errors.
| shoto_io wrote:
| Nice app! Could you please include Celsius as option?
| meteo-jeff wrote:
| Celsius is already the default temperature unit. You can
| optionally switch to Fahrenheit.
|
| For wind speeds, all typical units are available as well. Let
| me know if I missed one.
___________________________________________________________________
(page generated 2021-09-13 23:00 UTC)