hurl.html - www.codemadness.org - www.codemadness.org saait content files (HTM) git clone git://git.codemadness.org/www.codemadness.org (DIR) Log (DIR) Files (DIR) Refs (DIR) README (DIR) LICENSE --- hurl.html (4870B) --- 1 <!DOCTYPE html> 2 <html dir="ltr" lang="en"> 3 <head> 4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 5 <meta http-equiv="Content-Language" content="en" /> 6 <meta name="viewport" content="width=device-width" /> 7 <meta name="keywords" content="Gopher, HTTP, HTTPS" /> 8 <meta name="description" content="hurl: HTTP, HTTPS and Gopher file grabber" /> 9 <meta name="author" content="Hiltjo" /> 10 <meta name="generator" content="Static content generated using saait: https://codemadness.org/saait.html" /> 11 <title>hurl: HTTP, HTTPS and Gopher file grabber - Codemadness</title> 12 <link rel="stylesheet" href="style.css" type="text/css" media="screen" /> 13 <link rel="stylesheet" href="print.css" type="text/css" media="print" /> 14 <link rel="alternate" href="atom.xml" type="application/atom+xml" title="Codemadness Atom Feed" /> 15 <link rel="alternate" href="atom_content.xml" type="application/atom+xml" title="Codemadness Atom Feed with content" /> 16 <link rel="icon" href="/favicon.png" type="image/png" /> 17 </head> 18 <body> 19 <nav id="menuwrap"> 20 <table id="menu" width="100%" border="0"> 21 <tr> 22 <td id="links" align="left"> 23 <a href="index.html">Blog</a> | 24 <a href="/git/" title="Git repository with some of my projects">Git</a> | 25 <a href="/releases/">Releases</a> | 26 <a href="gopher://codemadness.org">Gopherhole</a> 27 </td> 28 <td id="links-contact" align="right"> 29 <span class="hidden"> | </span> 30 <a href="/donate/">Donate</a> | 31 <a href="feeds.html">Feeds</a> | 32 <a href="pgp.asc">PGP</a> | 33 <a href="mailto:hiltjo@AT@codemadness.DOT.org">Mail</a> 34 </td> 35 </tr> 36 </table> 37 </nav> 38 <hr class="hidden" /> 39 <main id="mainwrap"> 40 <div id="main"> 41 <article> 42 <header> 43 <h1>hurl: HTTP, HTTPS and Gopher file grabber</h1> 44 <p> 45 <strong>Last modification on </strong> <time>2020-07-20</time> 46 </p> 47 </header> 48 49 <p>hurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber.</p> 50 <h2>Why?</h2> 51 <p>Sometimes (or most of the time?) you just want to fetch a file via the HTTP, 52 HTTPS or Gopher protocol.</p> 53 <p>The focus of this tool is only this.</p> 54 <h2>Features</h2> 55 <ul> 56 <li>Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to 57 stdout).</li> 58 <li>Impose time-out and maximum size limits.</li> 59 <li>Use well-defined exitcodes for reliable scripting (curl sucks at this).</li> 60 <li>Send as little information as possible (no User-Agent etc by default).</li> 61 </ul> 62 <h2>Anti-features</h2> 63 <ul> 64 <li>No HTTP byte range support.</li> 65 <li>No HTTP User-Agent.</li> 66 <li>No HTTP If-Modified-Since/If-* support.</li> 67 <li>No HTTP auth support.</li> 68 <li>No HTTP/2+ support.</li> 69 <li>No HTTP keep-alive.</li> 70 <li>No HTTP chunked-encoding support.</li> 71 <li>No HTTP redirect support.</li> 72 <li>No (GZIP) compression support.</li> 73 <li>No cookie-jar or cookie parsing support.</li> 74 <li>No Gopher text handling (".\r\n").</li> 75 <li>... etc...</li> 76 </ul> 77 <h2>Dependencies</h2> 78 <ul> 79 <li>C compiler (C99).</li> 80 <li>libc + some BSD functions like err() and strlcat().</li> 81 <li>LibreSSL(-portable)</li> 82 <li>libtls (part of LibreSSL).</li> 83 </ul> 84 <h2>Optional dependencies</h2> 85 <ul> 86 <li>POSIX make(1) (for Makefile).</li> 87 <li>mandoc for documentation: <a href="https://mdocml.bsd.lv/">https://mdocml.bsd.lv/</a></li> 88 </ul> 89 <h2>Clone</h2> 90 <pre><code>git clone git://git.codemadness.org/hurl 91 </code></pre> 92 <h2>Browse</h2> 93 <p>You can browse the source-code at:</p> 94 <ul> 95 <li><a href="https://git.codemadness.org/hurl/">https://git.codemadness.org/hurl/</a></li> 96 <li><a href="gopher://codemadness.org/1/git/hurl">gopher://codemadness.org/1/git/hurl</a></li> 97 </ul> 98 <h2>Download releases</h2> 99 <p>Releases are available at:</p> 100 <ul> 101 <li><a href="https://codemadness.org/releases/hurl/">https://codemadness.org/releases/hurl/</a></li> 102 <li><a href="gopher://codemadness.org/1/releases/hurl">gopher://codemadness.org/1/releases/hurl</a></li> 103 </ul> 104 <h2>Build and install</h2> 105 <pre><code>$ make 106 # make install 107 </code></pre> 108 <h2>Examples</h2> 109 <p>Fetch the Atom feed from this site using a maximum filesize limit of 1MB and 110 a time-out limit of 15 seconds:</p> 111 <pre><code>hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml" 112 </code></pre> 113 <p>There is an -H option to add custom headers. This way some of the anti-features 114 listed above are supported. For example some CDNs like Cloudflare are known to 115 block empty or certain User-Agents.</p> 116 <p>User-Agent:</p> 117 <pre><code>hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml' 118 </code></pre> 119 <p>HTTP Basic Auth (base64-encoded username:password):</p> 120 <pre><code>hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \ 121 'https://codemadness.org/atom.xml' 122 </code></pre> 123 <p>GZIP (this assumes the served response Content-Type is gzip):</p> 124 <pre><code>hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d 125 </code></pre> 126 127 </article> 128 </div> 129 </main> 130 </body> 131 </html>