hurl - www.codemadness.org - www.codemadness.org saait content files
(HTM) git clone git://git.codemadness.org/www.codemadness.org
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
hurl (4888B)
---
1 1<- Back / codemadness.org 70
2 i codemadness.org 70
3 i codemadness.org 70
4 i# hurl: HTTP, HTTPS and Gopher file grabber codemadness.org 70
5 i codemadness.org 70
6 iLast modification on 2020-07-20 codemadness.org 70
7 i codemadness.org 70
8 ihurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber. codemadness.org 70
9 i codemadness.org 70
10 i codemadness.org 70
11 i## Why? codemadness.org 70
12 i codemadness.org 70
13 iSometimes (or most of the time?) you just want to fetch a file via the HTTP, codemadness.org 70
14 iHTTPS or Gopher protocol. codemadness.org 70
15 i codemadness.org 70
16 iThe focus of this tool is only this. codemadness.org 70
17 i codemadness.org 70
18 i codemadness.org 70
19 i## Features codemadness.org 70
20 i codemadness.org 70
21 i* Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to codemadness.org 70
22 i stdout). codemadness.org 70
23 i* Impose time-out and maximum size limits. codemadness.org 70
24 i* Use well-defined exitcodes for reliable scripting (curl sucks at this). codemadness.org 70
25 i* Send as little information as possible (no User-Agent etc by default). codemadness.org 70
26 i codemadness.org 70
27 i codemadness.org 70
28 i## Anti-features codemadness.org 70
29 i codemadness.org 70
30 i* No HTTP byte range support. codemadness.org 70
31 i* No HTTP User-Agent. codemadness.org 70
32 i* No HTTP If-Modified-Since/If-* support. codemadness.org 70
33 i* No HTTP auth support. codemadness.org 70
34 i* No HTTP/2+ support. codemadness.org 70
35 i* No HTTP keep-alive. codemadness.org 70
36 i* No HTTP chunked-encoding support. codemadness.org 70
37 i* No HTTP redirect support. codemadness.org 70
38 i* No (GZIP) compression support. codemadness.org 70
39 i* No cookie-jar or cookie parsing support. codemadness.org 70
40 i* No Gopher text handling (".\r\n"). codemadness.org 70
41 i* ... etc... codemadness.org 70
42 i codemadness.org 70
43 i codemadness.org 70
44 i## Dependencies codemadness.org 70
45 i codemadness.org 70
46 i* C compiler (C99). codemadness.org 70
47 i* libc + some BSD functions like err() and strlcat(). codemadness.org 70
48 i* LibreSSL(-portable) codemadness.org 70
49 i* libtls (part of LibreSSL). codemadness.org 70
50 i codemadness.org 70
51 i codemadness.org 70
52 i## Optional dependencies codemadness.org 70
53 i codemadness.org 70
54 i* POSIX make(1) (for Makefile). codemadness.org 70
55 h* mandoc for documentation: »https://mdocml.bsd.lv/« URL:https://mdocml.bsd.lv/ codemadness.org 70
56 i codemadness.org 70
57 i codemadness.org 70
58 i## Clone codemadness.org 70
59 i codemadness.org 70
60 i git clone git://git.codemadness.org/hurl codemadness.org 70
61 i codemadness.org 70
62 i codemadness.org 70
63 i## Browse codemadness.org 70
64 i codemadness.org 70
65 iYou can browse the source-code at: codemadness.org 70
66 i codemadness.org 70
67 h* https://git.codemadness.org/hurl/ URL:https://git.codemadness.org/hurl/ codemadness.org 70
68 1* gopher://codemadness.org/1/git/hurl /git/hurl codemadness.org 70
69 i codemadness.org 70
70 i codemadness.org 70
71 i## Download releases codemadness.org 70
72 i codemadness.org 70
73 iReleases are available at: codemadness.org 70
74 i codemadness.org 70
75 h* https://codemadness.org/releases/hurl/ URL:https://codemadness.org/releases/hurl/ codemadness.org 70
76 1* gopher://codemadness.org/1/releases/hurl /releases/hurl codemadness.org 70
77 i codemadness.org 70
78 i codemadness.org 70
79 i## Build and install codemadness.org 70
80 i codemadness.org 70
81 i $ make codemadness.org 70
82 i # make install codemadness.org 70
83 i codemadness.org 70
84 i codemadness.org 70
85 i## Examples codemadness.org 70
86 i codemadness.org 70
87 iFetch the Atom feed from this site using a maximum filesize limit of 1MB and codemadness.org 70
88 ia time-out limit of 15 seconds: codemadness.org 70
89 i codemadness.org 70
90 i hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml" codemadness.org 70
91 i codemadness.org 70
92 iThere is an -H option to add custom headers. This way some of the anti-features codemadness.org 70
93 ilisted above are supported. For example some CDNs like Cloudflare are known to codemadness.org 70
94 iblock empty or certain User-Agents. codemadness.org 70
95 i codemadness.org 70
96 iUser-Agent: codemadness.org 70
97 i codemadness.org 70
98 i hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml' codemadness.org 70
99 i codemadness.org 70
100 iHTTP Basic Auth (base64-encoded username:password): codemadness.org 70
101 i codemadness.org 70
102 i hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \ codemadness.org 70
103 i 'https://codemadness.org/atom.xml' codemadness.org 70
104 i codemadness.org 70
105 iGZIP (this assumes the served response Content-Type is gzip): codemadness.org 70
106 i codemadness.org 70
107 i hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d codemadness.org 70
108 .