(DIR) <- Back # hurl: HTTP, HTTPS and Gopher file grabber Last modification on 2020-07-20 hurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber. ## Why? Sometimes (or most of the time?) you just want to fetch a file via the HTTP, HTTPS or Gopher protocol. The focus of this tool is only this. ## Features * Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to stdout). * Impose time-out and maximum size limits. * Use well-defined exitcodes for reliable scripting (curl sucks at this). * Send as little information as possible (no User-Agent etc by default). ## Anti-features * No HTTP byte range support. * No HTTP User-Agent. * No HTTP If-Modified-Since/If-* support. * No HTTP auth support. * No HTTP/2+ support. * No HTTP keep-alive. * No HTTP chunked-encoding support. * No HTTP redirect support. * No (GZIP) compression support. * No cookie-jar or cookie parsing support. * No Gopher text handling (".\r\n"). * ... etc... ## Dependencies * C compiler (C99). * libc + some BSD functions like err() and strlcat(). * LibreSSL(-portable) * libtls (part of LibreSSL). ## Optional dependencies * POSIX make(1) (for Makefile). (HTM) * mandoc for documentation: »https://mdocml.bsd.lv/« ## Clone git clone git://git.codemadness.org/hurl ## Browse You can browse the source-code at: (HTM) * https://git.codemadness.org/hurl/ (DIR) * gopher://codemadness.org/1/git/hurl ## Download releases Releases are available at: (HTM) * https://codemadness.org/releases/hurl/ (DIR) * gopher://codemadness.org/1/releases/hurl ## Build and install $ make # make install ## Examples Fetch the Atom feed from this site using a maximum filesize limit of 1MB and a time-out limit of 15 seconds: hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml" There is an -H option to add custom headers. This way some of the anti-features listed above are supported. For example some CDNs like Cloudflare are known to block empty or certain User-Agents. User-Agent: hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml' HTTP Basic Auth (base64-encoded username:password): hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \ 'https://codemadness.org/atom.xml' GZIP (this assumes the served response Content-Type is gzip): hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d