hurl - www.codemadness.org - www.codemadness.org saait content files (HTM) git clone git://git.codemadness.org/www.codemadness.org (DIR) Log (DIR) Files (DIR) Refs (DIR) README (DIR) LICENSE --- hurl (4888B) --- 1 1<- Back / codemadness.org 70 2 i codemadness.org 70 3 i codemadness.org 70 4 i# hurl: HTTP, HTTPS and Gopher file grabber codemadness.org 70 5 i codemadness.org 70 6 iLast modification on 2020-07-20 codemadness.org 70 7 i codemadness.org 70 8 ihurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber. codemadness.org 70 9 i codemadness.org 70 10 i codemadness.org 70 11 i## Why? codemadness.org 70 12 i codemadness.org 70 13 iSometimes (or most of the time?) you just want to fetch a file via the HTTP, codemadness.org 70 14 iHTTPS or Gopher protocol. codemadness.org 70 15 i codemadness.org 70 16 iThe focus of this tool is only this. codemadness.org 70 17 i codemadness.org 70 18 i codemadness.org 70 19 i## Features codemadness.org 70 20 i codemadness.org 70 21 i* Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to codemadness.org 70 22 i stdout). codemadness.org 70 23 i* Impose time-out and maximum size limits. codemadness.org 70 24 i* Use well-defined exitcodes for reliable scripting (curl sucks at this). codemadness.org 70 25 i* Send as little information as possible (no User-Agent etc by default). codemadness.org 70 26 i codemadness.org 70 27 i codemadness.org 70 28 i## Anti-features codemadness.org 70 29 i codemadness.org 70 30 i* No HTTP byte range support. codemadness.org 70 31 i* No HTTP User-Agent. codemadness.org 70 32 i* No HTTP If-Modified-Since/If-* support. codemadness.org 70 33 i* No HTTP auth support. codemadness.org 70 34 i* No HTTP/2+ support. codemadness.org 70 35 i* No HTTP keep-alive. codemadness.org 70 36 i* No HTTP chunked-encoding support. codemadness.org 70 37 i* No HTTP redirect support. codemadness.org 70 38 i* No (GZIP) compression support. codemadness.org 70 39 i* No cookie-jar or cookie parsing support. codemadness.org 70 40 i* No Gopher text handling (".\r\n"). codemadness.org 70 41 i* ... etc... codemadness.org 70 42 i codemadness.org 70 43 i codemadness.org 70 44 i## Dependencies codemadness.org 70 45 i codemadness.org 70 46 i* C compiler (C99). codemadness.org 70 47 i* libc + some BSD functions like err() and strlcat(). codemadness.org 70 48 i* LibreSSL(-portable) codemadness.org 70 49 i* libtls (part of LibreSSL). codemadness.org 70 50 i codemadness.org 70 51 i codemadness.org 70 52 i## Optional dependencies codemadness.org 70 53 i codemadness.org 70 54 i* POSIX make(1) (for Makefile). codemadness.org 70 55 h* mandoc for documentation: »https://mdocml.bsd.lv/« URL:https://mdocml.bsd.lv/ codemadness.org 70 56 i codemadness.org 70 57 i codemadness.org 70 58 i## Clone codemadness.org 70 59 i codemadness.org 70 60 i git clone git://git.codemadness.org/hurl codemadness.org 70 61 i codemadness.org 70 62 i codemadness.org 70 63 i## Browse codemadness.org 70 64 i codemadness.org 70 65 iYou can browse the source-code at: codemadness.org 70 66 i codemadness.org 70 67 h* https://git.codemadness.org/hurl/ URL:https://git.codemadness.org/hurl/ codemadness.org 70 68 1* gopher://codemadness.org/1/git/hurl /git/hurl codemadness.org 70 69 i codemadness.org 70 70 i codemadness.org 70 71 i## Download releases codemadness.org 70 72 i codemadness.org 70 73 iReleases are available at: codemadness.org 70 74 i codemadness.org 70 75 h* https://codemadness.org/releases/hurl/ URL:https://codemadness.org/releases/hurl/ codemadness.org 70 76 1* gopher://codemadness.org/1/releases/hurl /releases/hurl codemadness.org 70 77 i codemadness.org 70 78 i codemadness.org 70 79 i## Build and install codemadness.org 70 80 i codemadness.org 70 81 i $ make codemadness.org 70 82 i # make install codemadness.org 70 83 i codemadness.org 70 84 i codemadness.org 70 85 i## Examples codemadness.org 70 86 i codemadness.org 70 87 iFetch the Atom feed from this site using a maximum filesize limit of 1MB and codemadness.org 70 88 ia time-out limit of 15 seconds: codemadness.org 70 89 i codemadness.org 70 90 i hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml" codemadness.org 70 91 i codemadness.org 70 92 iThere is an -H option to add custom headers. This way some of the anti-features codemadness.org 70 93 ilisted above are supported. For example some CDNs like Cloudflare are known to codemadness.org 70 94 iblock empty or certain User-Agents. codemadness.org 70 95 i codemadness.org 70 96 iUser-Agent: codemadness.org 70 97 i codemadness.org 70 98 i hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml' codemadness.org 70 99 i codemadness.org 70 100 iHTTP Basic Auth (base64-encoded username:password): codemadness.org 70 101 i codemadness.org 70 102 i hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \ codemadness.org 70 103 i 'https://codemadness.org/atom.xml' codemadness.org 70 104 i codemadness.org 70 105 iGZIP (this assumes the served response Content-Type is gzip): codemadness.org 70 106 i codemadness.org 70 107 i hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d codemadness.org 70 108 .