hurl.html - www.codemadness.org - www.codemadness.org saait content files
(HTM) git clone git://git.codemadness.org/www.codemadness.org
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
hurl.html (4870B)
---
1 <!DOCTYPE html>
2 <html dir="ltr" lang="en">
3 <head>
4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
5 <meta http-equiv="Content-Language" content="en" />
6 <meta name="viewport" content="width=device-width" />
7 <meta name="keywords" content="Gopher, HTTP, HTTPS" />
8 <meta name="description" content="hurl: HTTP, HTTPS and Gopher file grabber" />
9 <meta name="author" content="Hiltjo" />
10 <meta name="generator" content="Static content generated using saait: https://codemadness.org/saait.html" />
11 <title>hurl: HTTP, HTTPS and Gopher file grabber - Codemadness</title>
12 <link rel="stylesheet" href="style.css" type="text/css" media="screen" />
13 <link rel="stylesheet" href="print.css" type="text/css" media="print" />
14 <link rel="alternate" href="atom.xml" type="application/atom+xml" title="Codemadness Atom Feed" />
15 <link rel="alternate" href="atom_content.xml" type="application/atom+xml" title="Codemadness Atom Feed with content" />
16 <link rel="icon" href="/favicon.png" type="image/png" />
17 </head>
18 <body>
19 <nav id="menuwrap">
20 <table id="menu" width="100%" border="0">
21 <tr>
22 <td id="links" align="left">
23 <a href="index.html">Blog</a> |
24 <a href="/git/" title="Git repository with some of my projects">Git</a> |
25 <a href="/releases/">Releases</a> |
26 <a href="gopher://codemadness.org">Gopherhole</a>
27 </td>
28 <td id="links-contact" align="right">
29 <span class="hidden"> | </span>
30 <a href="/donate/">Donate</a> |
31 <a href="feeds.html">Feeds</a> |
32 <a href="pgp.asc">PGP</a> |
33 <a href="mailto:hiltjo@AT@codemadness.DOT.org">Mail</a>
34 </td>
35 </tr>
36 </table>
37 </nav>
38 <hr class="hidden" />
39 <main id="mainwrap">
40 <div id="main">
41 <article>
42 <header>
43 <h1>hurl: HTTP, HTTPS and Gopher file grabber</h1>
44 <p>
45 <strong>Last modification on </strong> <time>2020-07-20</time>
46 </p>
47 </header>
48
49 <p>hurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber.</p>
50 <h2>Why?</h2>
51 <p>Sometimes (or most of the time?) you just want to fetch a file via the HTTP,
52 HTTPS or Gopher protocol.</p>
53 <p>The focus of this tool is only this.</p>
54 <h2>Features</h2>
55 <ul>
56 <li>Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to
57 stdout).</li>
58 <li>Impose time-out and maximum size limits.</li>
59 <li>Use well-defined exitcodes for reliable scripting (curl sucks at this).</li>
60 <li>Send as little information as possible (no User-Agent etc by default).</li>
61 </ul>
62 <h2>Anti-features</h2>
63 <ul>
64 <li>No HTTP byte range support.</li>
65 <li>No HTTP User-Agent.</li>
66 <li>No HTTP If-Modified-Since/If-* support.</li>
67 <li>No HTTP auth support.</li>
68 <li>No HTTP/2+ support.</li>
69 <li>No HTTP keep-alive.</li>
70 <li>No HTTP chunked-encoding support.</li>
71 <li>No HTTP redirect support.</li>
72 <li>No (GZIP) compression support.</li>
73 <li>No cookie-jar or cookie parsing support.</li>
74 <li>No Gopher text handling (".\r\n").</li>
75 <li>... etc...</li>
76 </ul>
77 <h2>Dependencies</h2>
78 <ul>
79 <li>C compiler (C99).</li>
80 <li>libc + some BSD functions like err() and strlcat().</li>
81 <li>LibreSSL(-portable)</li>
82 <li>libtls (part of LibreSSL).</li>
83 </ul>
84 <h2>Optional dependencies</h2>
85 <ul>
86 <li>POSIX make(1) (for Makefile).</li>
87 <li>mandoc for documentation: <a href="https://mdocml.bsd.lv/">https://mdocml.bsd.lv/</a></li>
88 </ul>
89 <h2>Clone</h2>
90 <pre><code>git clone git://git.codemadness.org/hurl
91 </code></pre>
92 <h2>Browse</h2>
93 <p>You can browse the source-code at:</p>
94 <ul>
95 <li><a href="https://git.codemadness.org/hurl/">https://git.codemadness.org/hurl/</a></li>
96 <li><a href="gopher://codemadness.org/1/git/hurl">gopher://codemadness.org/1/git/hurl</a></li>
97 </ul>
98 <h2>Download releases</h2>
99 <p>Releases are available at:</p>
100 <ul>
101 <li><a href="https://codemadness.org/releases/hurl/">https://codemadness.org/releases/hurl/</a></li>
102 <li><a href="gopher://codemadness.org/1/releases/hurl">gopher://codemadness.org/1/releases/hurl</a></li>
103 </ul>
104 <h2>Build and install</h2>
105 <pre><code>$ make
106 # make install
107 </code></pre>
108 <h2>Examples</h2>
109 <p>Fetch the Atom feed from this site using a maximum filesize limit of 1MB and
110 a time-out limit of 15 seconds:</p>
111 <pre><code>hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml"
112 </code></pre>
113 <p>There is an -H option to add custom headers. This way some of the anti-features
114 listed above are supported. For example some CDNs like Cloudflare are known to
115 block empty or certain User-Agents.</p>
116 <p>User-Agent:</p>
117 <pre><code>hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml'
118 </code></pre>
119 <p>HTTP Basic Auth (base64-encoded username:password):</p>
120 <pre><code>hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
121 'https://codemadness.org/atom.xml'
122 </code></pre>
123 <p>GZIP (this assumes the served response Content-Type is gzip):</p>
124 <pre><code>hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d
125 </code></pre>
126
127 </article>
128 </div>
129 </main>
130 </body>
131 </html>