[HN Gopher] HTTP Toolkit
___________________________________________________________________
HTTP Toolkit
Author : hliyan
Score : 401 points
Date : 2021-12-20 12:51 UTC (10 hours ago)
(HTM) web link (httptoolkit.tech)
(TXT) w3m dump (httptoolkit.tech)
| Sytten wrote:
| We are in a similar space, our tool (https://caido.io) is geared
| toward bug bounty hunters and pentesters. HTTP Toolkit looks
| great congrats to the dev! It seems to compete more with
| requestly (https://requestly.io/) than burp suite
| (https://portswigger.net/).
| jeffalo wrote:
| TomAnthony wrote:
| Excited about Caido! Been waiting on the beta list for a while!
| Sytten wrote:
| I dont want to hijack the thread but I will just say that we
| are going full time on it starting January so it will speed
| things up!
| IceWreck wrote:
| a) This has less features than mitmproxy, burp suite or owasp
| zap.
|
| b) Export request is a very essential feature that's available
| for free in either of these alternatives and its pay-walled here.
|
| c) Intercept request in a terminal very badly broken. It broke
| (go get) and probably breaks others. I expect it adds an
| environment variable (which can be ignored by an application) or
| uses LD_PRELOAD (which doesn't work in statically linked
| applications).
|
| Other than that, it functions like you would expect it to. Worked
| out of the box for Firefox and curl.
| pimterry wrote:
| I'm the author. I'm not a Go developer though, can you give me
| an example I can reproduce for `go get` that doesn't work for
| you?
|
| For the terminal, there's a few mechanisms, but environment
| variables are the catch-all there, yes (full list:
| https://github.com/httptoolkit/httptoolkit-
| server/blob/maste...). Those do work for most cases, but it is
| absolutely not a hard guarantee for applications that actively
| ignore standard proxy configuration (handling that is very
| hard, and definitely out of scope here).
|
| Go does generally observes `http_proxy` correctly by default in
| other cases I've tested, so this vert simple code from the test
| suite is automatically intercepted for example:
| https://github.com/httptoolkit/httptoolkit-
| server/blob/maste.... Very happy to look into any failing cases
| you can share.
| IceWreck wrote:
| Yes, go based programs observe the proxy variable unless
| specifically compiled not to. But that doesn't break
| applications, they just ignore the env var. Go's default
| package manager doesn't seem to work at all with HTTP
| Toolkit.
|
| Here's what I did.
|
| * Intercept Tab > Fresh terminal.
|
| * In the terminal, do your usual stuff, it intercepts curl,
| etc.
|
| * If you try to use go's package manager, example: `go get
| golang.org/x/oauth2` It errors out with
|
| ``` go get: module golang.org/x/oauth2: reading
| http://127.0.0.1:8000/golang.org/x/oauth2/@v/list: 500 Server
| error ```
|
| Ideally it shouldn't break the application, just ignore if it
| can't intercept.
|
| > absolutely not a hard guarantee for applications that
| actively ignore standard proxy configuration (handling that
| is very hard, and definitely out of scope here).
|
| I encountered a usecase where this was needed and LD_PRELOAD
| trick (used by proxychains) etc failed because the
| application was statically compiled. I ended up using
| https://github.com/hmgle/graftcp which somehow manages to
| force tcp traffic through a socks5 proxy.
| pimterry wrote:
| Thanks, that's super useful.
|
| > If you try to use go's package manager, example: `go get
| golang.org/x/oauth2`
|
| I just tested, and `go get golang.org/x/oauth2` seems to
| work fine for me, I can see all the requests being happily
| intercepted immediately: https://imgur.com/a/Cb1y9Q2
|
| Can you see the 500 in HTTP Toolkit, and any more info
| there (in the body or as an error at the top) related to
| that? Or can you see a "certificate rejected" message? If
| nothing turns up there at all then yes, something must be
| overriding the proxy configuration.
|
| Maybe you have some other Go package manager configuration
| that conflicts with this? I'd be very interested to know
| about that if so, I'm sure there's others with the same
| thing. It's always very hard to know if my configuration is
| representative of normal devs for any given language/tool.
|
| Probably best to debug this outside of a HN thread though
| :-). You can file a proper issue about this at
| https://github.com/httptoolkit/httptoolkit/issues/new, I'd
| love to know what's going on there and get this fixed.
|
| > I ended up using https://github.com/hmgle/graftcp which
| somehow manages to force tcp traffic through a socks5
| proxy.
|
| Really interesting, thanks! I'll look into that.
| IceWreck wrote:
| This is the error I see in the response body in HTTP
| Toolkit.
|
| ``` Error: Passthrough loop detected. This probably means
| you're sending a request directly to a passthrough
| endpoint, which is forwarding it to the target URL, which
| is a passthrough endpoint, which is forwarding it to the
| target URL, which is a passthrough endpoint...
|
| You should either explicitly mock a response for this URL
| (http://127.0.0.1:8000/golang.org/x/oauth2/@v/list), or
| use the server as a proxy, instead of making requests to
| it directly. ```
| based2 wrote:
| alt: https://mockoon.com/
|
| https://github.com/mockoon/mockoon
| mathfailure wrote:
| And it's open-source!
| can16358p wrote:
| This would be great but it doesn't have the only features I truly
| need: intercepting HTTPS connections from iOS apps.
|
| I understand that intercepting HTTPS might be a bit complicated
| for iOS and still think this is a great project though!
| pimterry wrote:
| I'm the author. Yes, I'm afraid automatic iOS interception
| isn't available yet, but it's definitely planned for the
| future! You can subscribe to the GitHub issue here for updates:
| https://github.com/httptoolkit/httptoolkit/issues/11
|
| In the meantime it's still totally possible to intercept iOS
| devices, but you just have to do the initial setup manually
| unfortunately.
| can16358p wrote:
| Lovely! Looking forward for a automated-iOS setup (at least
| as much as the walled garden" permits). Keep up the good
| work!
| s_dev wrote:
| Maybe you need something like Charles Proxy or Proxyman?
| can16358p wrote:
| Yeah, but a streamlined "just works" setup would be much more
| useful (which the project really aims on).
|
| Otherwise, yeah it's possible, but again, it has always been
| possible with more complicated (compared to this) tools
| anyway.
| kevinbowman wrote:
| I think Charles proxy can do that, by installing itself as a
| VPN on iOS devices and also as a device configuration profile
| (so it can use a custom CA to MITM the traffic).
| orliesaurus wrote:
| I am glad to see HTTP Toolkit back on HN: The author is an old
| friend who dedicated a lot of time and resources to make this
| tool and as someone who fiddled with it, it's a great alternative
| to Postman - which I used to love but now it's a little bit too
| heavy on pushing their marketing agenda. Also how on earth is
| Postman a $5.6 billion dollar company - I am super curious to see
| how much money Postman actually makes from enterprise sales - it
| must be a lot from the looks of that valuation.
| goalieca wrote:
| I've been looking for an alternative to postman that isn't so
| insanely geared to monetization. I feel postman compromises it's
| entire UX to force a simple offline tool to be cloud based.
| pletnes wrote:
| Monetization dark patterns is one thing. But how on earth do
| they lure people away from the good old plaintext-in-git
| workflow? I found everything about postman to be anti-git in
| nature, and then I lose versioning, CI/CD becomes weird, etc. I
| just don't see the advantage for teams.
|
| If you're a lone wolf developer who prefers a GUI - fine. But
| then, why pay up for the <<pro>> service?
| vladvasiliu wrote:
| I've seen this used by people who are not quite devs but work
| with devs, as in not-really-technical PMs. They're happy to
| not have to deal with git, but they still need to see that
| some API replies with the expected result. They absolutely
| love postman.
| pletnes wrote:
| Sure thing, but why not keep the source in git, then
| distribute to end users, as we do with other software
| artifacts?
| squeaky-clean wrote:
| In my experience, people aren't buying the pro version of
| Postman as solo/indie developers. Companies purchase it for
| their QA team, or for technical but non-developer roles that
| need to make requests, share them among teammates, and keep a
| cloud-sync'd collection of them.
| thecopy wrote:
| I can recommend https://paw.cloud/ for MacOS
| mariusseufzer wrote:
| I second this. It's not open source, yes, but it's a fair
| one-time payment. No subscription or anything!
| BillinghamJ wrote:
| They did shift to a subscription model for team use
| unfortunately: https://paw.cloud/pricing
|
| And were recently acquired, so likely to continue down that
| path: https://blog.paw.cloud/paw-joins-forces-with-
| rapidapi/
|
| My previous company was an early & very heavy user of Paw -
| with thousands of endpoints in our project. Unfortunately
| at that scale, it has some major problems, particularly
| with syncing.
|
| A friend of mine has been working on https://getbeak.app to
| try and address those problems, but it's quite early stage
| still
| haswell wrote:
| I realize that certain UI patterns are going to
| inevitably overlap in this space, but this looks like a
| literal clone of Paw. Seems questionable.
| jen20 wrote:
| It also does not solve the problem of subscription
| pricing - it's $25/year with no obvious one-time payment
| option.
| dylanowen wrote:
| 100% agree. After postman asked me to make an account I
| uninstalled it and have been building my own. It's not all
| wired together yet but I'm using this fork of
| https://github.com/dylanowen/dot-http/tree/wasm-library
|
| I wanted something where I could store and share my requests
| over git instead of creating some random account. I also wanted
| the api description to be text not a UI+JSON blob.
|
| I also made this https://github.com/dylanowen/sublime-dot-http
| ananthakumaran wrote:
| After having used postman for sometime, I realized it favors
| mouse click based workflow and is very resource heavy. I have
| switched to emacs verb[1] package and combined with some custom
| function for env (staging/local) management and org-roam for
| file management, it's much more comfortable to use. I don't
| have to learn a new set of shortcuts and can use all the other
| emacs features.
|
| [1]: https://github.com/federicotdn/verb
| meibo wrote:
| I used to recommend https://insomnia.rest/ as it was a free,
| sleek alternative... Alas it's been bought by Kong and
| enterprise'd up the same way. Not as bad yet, but can't be
| long.
| pull_my_finger wrote:
| Why would you think Kong would spoil it? Their main product
| is an API gateway, it makes sense to have a nice HTTP client
| under their umbrella. Kong is also open source and almost all
| of their plugins are freely available. I actually tried (and
| still use) Insomnia _because_ it was affiliated with Kong.
| YMMV but I've had no complaints about Insomnia at all.
| squeaky-clean wrote:
| Am I having a Mandela Effect moment or did Insomnia used to
| be Mac only? I remember seeing it a long time ago and being
| disappointed I couldn't use it. Now I'm doubting my memory,
| lol.
|
| Either way thanks for the recommendation/reminder for
| Insomnia, it looks so much better than the current version of
| postman.
| easton wrote:
| You aren't thinking of Paw, are you? https://paw.cloud/
| phunehehe0 wrote:
| You are not alone! I just checked the Git history and it
| seems we were indeed mistaken, at least since 2017. It
| might have been just hard to see from the website.
|
| "Insomnia is available for Mac, Windows, and Linux and can
| be downloaded from http://insomnia.rest/download/"
|
| https://github.com/Kong/insomnia/commit/23cca8c42b80aa6d9de
| 2...
| udfalkso wrote:
| Insomnia is great, I use it daily. Highly reccommend.
| vladvasiliu wrote:
| For the time being it's still OK. It asked me once to try the
| "premium" version or whatever it's called, I said no, and it
| never bothered me again.
|
| I don't know what licence Postman uses. A quick search on
| GitHub didn't turn up the source of the app on the Postman
| Labs page. Insomnia is MIT [0], so it could still be forked
| if Kong got decided to stop supporting the free version.
|
| [0] https://github.com/Kong/insomnia/blob/develop/LICENSE
| chrisweekly wrote:
| I still like Insomnia too.
| meibo wrote:
| I actually wasn't aware of it being open source at all,
| thanks. It's not mentioned anywhere on their page, gives me
| a bit more confidence :)
| adolph wrote:
| I don't think this is necessarily an alternative to Postman. It
| is more like an alternative to Fiddler or other mitm network
| traffic analysis tools oriented around 80/443 HTTP.
|
| The best alternative to Postman is curl coupled with jq and
| xmllint.
| izyda wrote:
| How about HTTPie: https://httpie.io/
|
| Well known for the CLI, they now have a web/desktop client as
| well.
| the_arun wrote:
| HTTPie is simple to use from command line. I use it for
| testing our apis.
| aedocw wrote:
| I'm really surprised no one else has mentioned this so far, but
| Hoppscotch (formerly postwoman) is an excellent open
| alternative: https://github.com/hoppscotch/hoppscotch
| gadrev wrote:
| For simple requests where it's just convenient to have them in
| a list, with a description/easy to find for replaying, a VSCode
| extension like Rest Client [0] can be useful instead of the
| desktop apps. Well, VSCode is a desktop app too, but if you
| already use it... :P
|
| I like it because you can just write a text file with the
| request and any comments you need around them, and... being
| just text, it's so easy to manipulate.
|
| When I need scripts for special auth I fall back to postman
| though, haven't digged enough to see if I can make it work with
| that addon (or any other one).
|
| Also there's Thunder Client [1] which I haven't tried but
| apparenly has more features.
|
| EDIT: references
|
| [0] https://github.com/Huachao/vscode-restclient [1]
| https://www.thunderclient.io/
| nine_k wrote:
| Same thing is available for Emacs.
|
| I suspect that Vim, Sublime, or Notepad++ should likely
| already have a good equivalent of it, too.
| dpipemazo wrote:
| Sublime has requester which works quite well if you're
| familiar with the Python requests syntax. You write the
| query in the equivalent Python line and then use some
| hotkeys to execute/send it. Variables can be used/stored in
| the file for common, verbose variables such as auth tokens.
| The response comes back in a new tab and it's fairly easy
| to search through.
| shaan7 wrote:
| Yep. VSCode's Rest Client extension is amazing, I can just
| have plain text files with everything I need to my cloud
| drive (and maybe even git repo to share with team).
| vbezhenar wrote:
| If you're using Intellij Idea, it supports similar http
| requests as well. Very convenient as those are just text
| files.
| vogre wrote:
| I use jupiter notebook. It allows writing and saving requests,
| in much more convenient(for me) way than postman does
| [deleted]
| cute_boi wrote:
| I have used charles/proxyman/burp/figma and the best software I
| have used is httptoolkit. The opensource model of httptoolkit is
| praise worthy. And the maintainer is down to earth :).
|
| I am very delighted to use software like httptoolkit.
|
| The only issue with httptoolkit is electron but it isn't problem
| for me because I can always run it in browser <3.
| dneri wrote:
| This is really impressive. I've been using Charles for the last
| decade which performs the same actions but takes more work to set
| up, especially around the root certificate installation process.
| Will definitely be adding this to my toolkit!
| [deleted]
| FreeHugs wrote:
| It would be cool if something like this would be possible with
| standard Linux tools on the command line.
|
| How hard would it be to implement a network rule "if
| http://abc.com/def is requested reply with this data: ..."?
|
| Or is it possible to inject something like this on the fly into
| Apache?
|
| Would be very nice to mock end-2-end tests.
| cedricvanrompay wrote:
| Note that HTTPS, unlike says VPNs (IPSec, Wireguard...),
| provides authenticity protection up to the application itself.
| If the kernel sends a modified packet to the application, it
| will be rejected by the application.
|
| As a result, to implement HTTPS interception / rewrite /
| injection you need some degree of modification of the
| application itself. The "minimal" way is to add a new TLS
| certificate to the certificate trust store the application uses
| that is marked as "allowed for every domain" (that's what Burp
| suite does). It seems that HTTP toolkit does it differently for
| the browsers it supports, probably a plugin/extension added to
| the browser that alters the traffic _after_ the TLS block
| (HTTPS is HTTP over TLS)
| pimterry wrote:
| I'm the author of HTTP Toolkit, I actually built the internals
| much earlier as an open-source library (Mockttp:
| https://github.com/httptoolkit/mockttp) designed for exactly
| the end-to-end testing mocking use case you're talking about.
|
| It's MIT-licensed, and you can build an automated HTTP/HTTPS
| rewriting proxy using that in a handful of lines of JS, and
| script any kind of transformations or inject any responses you
| like.
|
| There's a general guide to getting started here:
| https://httptoolkit.tech/blog/javascript-mitm-proxy-mockttp/.
|
| For the more general interactive testing/debugging case, you
| can also use HTTP Toolkit itself (it has a rules builder for
| this kind of thing) but if you're building automation you
| should just use the internals directly, they have exactly the
| same capabilities. HTTP Toolkit just provides a UI and
| convenient interception setup tools over the top.
| intpx wrote:
| Is there an internet law for when you spend weeks looking for
| a framework or library to solve a problem and it only reveals
| itself after your organization makes a really bad choice to
| do something else? I have been looking for exactly this kind
| of local proxy to dynamically hack headers and dynamically
| spoof responses for an internal app. This would have been the
| perfect starting place...
| inglor wrote:
| This is great! I've been looking for this sort of thing
| today. It's very helpful and also nice to see some projects I
| maintain or contribute to in the dependencies list :)
|
| You might want to consider migrating from node-abort-
| controller to native AbortController by the way.
| marcosdumay wrote:
| I'm not sure what you are expecting that isn't a main feature
| of Apache.
|
| I imagine you expect something like a CGI script with mod-
| rewrite, but your comment only actually requires plain Apache.
| A network rule of "if URL is requested reply with this data" is
| implemented by putting a file at the expected place.
| dotancohen wrote:
| > only actually requires plain Apache
|
| ... and a hosts file entry. Still, trivial for any machine
| that the operator administers.
| owlbynight wrote:
| Price point on the personal account is too high over time. Would
| expect this to be a one time payment rather than a monthly fee
| for this type of software.
| spacephysics wrote:
| One time payment? Surely being in this community you understand
| what maintenance and ongoing feature additions entail? The
| whole reason the software world turned to subscriptions is
| because one time payments don't fit the model of continuous
| work.
|
| Now saying it should be just cheaper in general, or perhaps
| more tiers, sure.
| johnchristopher wrote:
| It used to be that it was expected/normal to pay for a new
| major version though (eg: Photoshop 4, 5, 5.5, 6).
| qw wrote:
| That has several drawbacks for a business:
|
| 1. Due to demands from marketing/sales, the supplier tends
| to increase versions for what is actually a minor feature,
| just to justify a new payment.
|
| 2. A subscription process is the most honest way of selling
| software. Jetbrains is a good example of this, where you
| get to keep your "fallback" version when you stop the
| subscription.
|
| 3. You often have to wait a long time for the next release,
| if you don't want to register in an "early access" program
| with versions that break constantly (basically providing
| free test resources).
|
| 4. You need to justify the new version to your boss, so
| that you can get it covered and start the approval process
| higher up in the hierarchy.
|
| 5. Approval processes in large Enterprises are often
| complex and time consuming. This not only applies to
| approving the payment, but often need to involve central
| IT. With a subscription model, this is only done once.
| owlbynight wrote:
| 1. I don't know, maybe just don't be a piece of shit
| company.
|
| 2. JetBrains charges me $12 a month ($149 annually) for
| its entire suite of software. You think this tool by
| itself is worth $120 a year? For a personal license?
|
| 3. Developer's problem, not mine.
|
| 4. This is not an issue everywhere. You work with penny-
| pinching mongoloids.
|
| 5. This is not an issue everywhere. You work with penny-
| pinching mongoloids.
| arbitrage wrote:
| even before that, it was expected/normal to pay for a piece
| of software once, regardless of major revisions or ongoing
| maintenance.
|
| trying to trick people into giving you money forever is a
| paradigm shift that really needs to be rethunk.
| owlbynight wrote:
| I do understand what maintenance and ongoing feature
| additions entail. That's a problem that FOSS tackles.
|
| Asking me, the end-user, to commit to $14 a month in
| perpetuity for this type of software is a big stretch in my
| opinion. I understand it for the Team tier, but for the
| personal tier, it doesn't make sense. $5 a month? Maybe.
|
| Maybe it's not for a user like me, who would probably use it
| twice a month, if that. But I was interested in checking it
| out but got immediately priced out. For a startup, it seems
| like an ill conceived practice.
|
| But what do I know? I've only been using and paying for
| software for 30 years or so.
| lux wrote:
| Curious if there's a tool in this category that does websockets
| well?
| the_arun wrote:
| I am looking for java library which logs metadata about any
| egress call (HTTP or others) from JVM. Do we have any?
| santigr wrote:
| I code a small replacement for the postman mock servers and I see
| that only the pro version of http toolkit come with this feature.
| If you are interested in a simple and small replacement:
| https://github.com/sgrodriguez/yams
| sn0wtrooper wrote:
| I liked this a lot. The easy ADB integration makes everything
| much faster to me.
| jmkni wrote:
| This is really really nice, I've actually been thinking about
| building something similar myself recently, thanks!!
| rhtgrg wrote:
| It's fascinating how many of these tools exist. Some users have
| already posted similar tools in this thread, here's a YC backed
| one [0].
|
| [0] https://www.ycombinator.com/companies/requestly
| zerkten wrote:
| It's unfortunate that they often fail to give you anything that
| wasn't possible with Fiddler over 10 years ago. There is much
| potential for automating the things that folks are doing in
| these tools repeatedly, especially things that devs are doing
| for other stakeholders like support teams. Obviously security
| folks have an interest in these tools, but they generally
| understand the potential and exploit it more effectively than
| the generalists.
| cute_boi wrote:
| httptoolkit is opensource and you can see contribute, look
| source code etc which is automatically superior to me.
| ocdtrekkie wrote:
| I'm not sure if it's only because I'm on mobile, but the "enter
| your email to download" thing gives me weird marketing stuff
| vibes. I find it a weird touch on open source software I
| presumably can get without giving them my email.
| pimterry wrote:
| I'm the creator of HTTP Toolkit - you don't need to share your
| email to use it.
|
| It is a desktop app though, so you can't download it usefully
| on mobile. If you visit the landing page on mobile, it just
| offers to take your email address and send download links to
| your computer to get you started (and as an easy "bookmark this
| for later" option).
|
| That sends one follow up "Did you try it? Let me know what you
| think" email a week later, then it deletes your email, that's
| it. It's never shared elsewhere, it doesn't sign you up to the
| mailing list, there are no "great new offers", nada.
|
| Meanwhile, if you're on desktop it just shows a download button
| directly. After that starts it asks if you want to join the
| mailing list for updates, but you don't need to - the download
| starts immediately.
| cyberge99 wrote:
| This is exactly what should be noted. You'll probably get
| more signups if people know how it's going to be used.
| ocdtrekkie wrote:
| I both understand the goal there, since indeed I can't
| meaningfully download it on mobile, but I would also point
| out many others will get the same vibes I noted above from
| it.
|
| Which is probably just to say after being burned too many
| times by bad actors, folks will start to group good actors in
| the same lot for similar patterns, even if the intent and
| design is better.
| IceWreck wrote:
| I tried it, it started the download and gave the enter email
| box at the same time. So its optional.
| herpderperator wrote:
| Looks like it runs emulated via Rosetta 2. Would love to see an
| Apple Silicon version!
| gregoriol wrote:
| Is there really a difference for such a simple app?
| herpderperator wrote:
| Absolutely. It's emulating Electron which is anything but
| simple. That being said, it should be fairly simple to
| recompile an Electron app into arm64.
| gatewaynode wrote:
| So like BURP Suite(https://portswigger.net/) but with fewer
| features?
| sixothree wrote:
| Maybe some user interface improvements?
| [deleted]
| thealistra wrote:
| I recommend https://paw.cloud for just the Postman clone - it has
| native UI, not an Electron shell.
|
| For doing proxies, there is https://proxyman.io, which I think is
| also native (haven't used it a lot, not sure)
| The_Colonel wrote:
| > it has native UI, not an Electron shell.
|
| ... and as a result works only on Mac, so not usable for close
| to 90% of people. Every coin has 2 sides.
| yolo3000 wrote:
| How is this decrypting https out of the box, while with Charles
| proxy I needed to install a root certificate?
| kevinbowman wrote:
| From a quick look at the code, it seems to be doing something
| similar, except it does it for you. Here's a link to the
| Firefox bit, where it makes a custom Firefox profile and then
| injects a certificate into it:
| https://github.com/httptoolkit/httptoolkit-server/blob/maste...
|
| I haven't seen the source for the .deb package, but in theory
| it could add a system cert at installation time. I don't know
| if it does, though.
| pimterry wrote:
| I'm the author, that's exactly it! The contents of that
| interceptors folder should give you an idea how it all works:
| https://github.com/httptoolkit/httptoolkit-
| server/tree/maste...
|
| It actually doesn't install system certificates at all
| though. It doesn't change any system configuration
| whatsoever, and it doesn't need any admin/root privileges.
| The deb package doesn't do anything different to any others.
|
| That's because the key differentiator of HTTP Toolkit vs
| Fiddler/Charles/mitmproxy etc, is that it provides targeted
| interception, rather than intercepting your entire system at
| once.
|
| That works by injecting cert & proxy config into a single
| browser window, intercepting specific Android apps, targeting
| individual Docker containers etc. That way you get much less
| noisy intercepted traffic for your debugging, and you can
| freely add rules to rewrite/break traffic without interfering
| with anything else.
|
| You can even open two HTTP Toolkit windows on one machine,
| and intercept things separately into each one.
|
| If you want, you can still do the normal steps to do full
| system interception manually if you'd prefer that, but by
| default it uses entirely transient and permissionless
| targeted interception instead, and that's almost always the
| better approach.
| timdorr wrote:
| Does this work with cert pinning on Android? I was only
| ever able to bypass that by rooting my device and
| installing an Xposed module to bypass the pin check.
| pimterry wrote:
| There's a detailed Android guide here:
| https://httptoolkit.tech/docs/guides/android/
|
| In short, most of the time you need to either:
|
| - Connect an Android emulator or a rooted device to ADB,
| in which case HTTP Toolkit can do totally automated setup
| for you.
|
| - Use a non-rooted device, and make some minor config
| changes to the target application (trivial if it's your
| own application, slightly more difficult if it's not).
|
| That handles 99% of Android apps, which usually don't
| actually pin certificates - they generally rely on
| Android's built-in non-modifiable system certificate
| store instead.
|
| Lots more detail on how this all works here:
| https://httptoolkit.tech/blog/intercepting-android-https/
|
| For apps that really do manually pin certificates, I've
| also written a general purpose Frida script that covers
| most cases out of the box. There's a full guide with more
| detail here: https://httptoolkit.tech/blog/frida-
| certificate-pinning/. And if even that doesn't work, I've
| also written a "reverse engineering an Android app from
| scratch so you can write you own Frida script" guide
| here: https://httptoolkit.tech/blog/android-reverse-
| engineering/
| cute_boi wrote:
| httptoolkit is best software in market that works so
| easily and can bypass certs pinning.
|
| Here are steps: Download frida script from httptoolkit
| server and binary from frida github repo and download
| httptoolkit app in andriod. Here are my notes.
|
| ``` # Copy the server to the device adb push ./frida-
| server-$version-android-$arch /data/local/tmp/frida-
| server # ^Change this to match the name of the binary you
| just extracted
|
| # Enable root access to the device adb root
|
| # Make the server binary executable adb shell "chmod 755
| /data/local/tmp/frida-server"
|
| # Start the server on your device adb shell
| "/data/local/tmp/frida-server &"
|
| pip3 install frida-tools frida-ps -U frida --no-pause -U
| -l ./frida.js -f com.appname
|
| # derived from https://httptoolkit.tech/blog/frida-
| certificate-pinning/ ```
| e12e wrote:
| Frida? That's https://frida.re/ ?
| cute_boi wrote:
| yea :)
|
| You can download server binaries from here
| https://github.com/frida/frida/releases
| kevinbowman wrote:
| Good to know, thanks for the confirmation!
| [deleted]
| e12e wrote:
| Looks very slick. As the author is active in this thread - why
| would I prefer this to mitmproxy? It seems to be missing from:
| https://httptoolkit.tech/alternatives/
|
| I'd be interested both in why I'd prefer the open source
| httptoolkit and pro?
| pimterry wrote:
| > why would I prefer this to mitmproxy?
|
| Compared to mitmproxy, HTTP Toolkit:
|
| - Has fully automated setup for most browsers, docker
| containers, Android, all Node.js/Ruby/Python/PHP/Go
| applications run from intercepted terminal windows, all JVM
| processes, any Electron apps etc etc. Some of these automated
| setup steps are very difficult to do manually (e.g.
| intercepting Android devices, where you can't normally install
| your own certificates nowadays, or intercepting Node.js, which
| completely ignores system proxy settings) so this can make a
| huge difference in non-trivial case.
|
| - Supports _targeted_ interception (intercept just one app
| /container/browser window) whilst all mitmproxy's manual setup
| steps are generally focused on helping you intercept your whole
| machine at once. Intercepting the whole machine means very
| noisy interception and means that rewriting traffic interferes
| with all other usage of your machine. Targeted interception
| means you can do neat things like run two HTTP Toolkit
| instances independently at the same time, and means you don't
| need root privileges or permanent configuration settings.
|
| - Has a VPN app for Android, which allows it to capture traffic
| even if it tries to ignore proxy configuration, means you don't
| have to manually edit and delete Android proxy settings, and
| which can automatically tunnel traffic over ADB connections, so
| you can intercept a device connected via ADB even if its not
| connectable over the wifi from your computer.
|
| - Has generally friendlier UI & UX (imo). For example,
| mitmproxy uses a unique custom syntax
| (https://docs.mitmproxy.org/stable/concepts-filters/) of
| special characters to define matching & rewriting rules, or
| requires you to write a full python script. HTTP Toolkit lets
| you click 'new rule' -> 'GET requests' -> 'match regex <blah>'
| -> 'then reply with <blah>', and then immediately start
| injecting automated fake responses. From HTTP Toolkit you can
| then build named groups or these rules, and import & export
| them (as JSON) to build libraries you can share with your
| colleagues.
|
| - Provides lots more background information automatically: e.g.
| built-in documentation for all standard HTTP headers, body
| autoformatting for lots more formats, syntax highlighting, code
| folding, regex searching etc of request & response bodies, plus
| 'this is how and why this response could be cached' caching
| explanations, OpenAPI-powered docs for recognized endpoints on
| 1400+ APIs, etc.
|
| - Includes advanced features to do things like exporting
| requests as ready-to-use code for various languages & tools, or
| automatically testing the performance of different compression
| algorithms on a given response body.
|
| - Is more easily scriptable for automation & end-to-end
| testing, because all the HTTP-handling internals are usable as
| a standalone open-source JS library:
| https://github.com/httptoolkit/mockttp
|
| That said, mitmproxy has been around longer, it's definitely
| more mature, and it was a big inspiration in many places. It's
| a great project! It does have some advantages of its own:
|
| - If you strongly prefer a CLI interface, mitmproxy is very
| focused on that, and HTTP Toolkit is not. HTTP Toolkit could
| support that too in theory (the backend & frontend are
| independent) but it definitely doesn't right now, and it's not
| high on my todo list (contributions welcome though!)
|
| - Mitmproxy is primarily scriptable in Python. You can build
| automation around HTTP Toolkit's internals using mockttp, but
| that's JS, and it's mostly usable standalone right now, rather
| than integrated into normal workflows within the app. If you
| want very complex scripted rules, mitmproxy has a few more
| options right now, and lets you do things in python instead of
| JS, which some people will prefer.
|
| - WebSocket debugging - this is coming for HTTP Toolkit soon,
| but it's not available today. WebSockets get passed through
| fine, but they don't appear in the UI, and you can't set up
| mock rules for them.
|
| > I'd be interested both in why I'd prefer the open source
| httptoolkit and pro?
|
| There's a list of Pro features at
| https://httptoolkit.tech/pricing/. Note that it's all open
| source, even the Pro code, everything.
|
| The general idea is that everything you need to intercept,
| inspect and manually fiddle with traffic is totally free.
| Anything optional that most users don't need, but which is
| helpful for advanced usage or enterprise use cases, requires
| Pro.
| e12e wrote:
| Thank you!
| stavros wrote:
| At some point I wanted to see what an Android API looked
| like, and HTTP toolkit was, by a huge margin, the easiest way
| to do that. I had previously tried mitmproxy, Charles, and a
| few others, but only HTTP toolkit worked reliably, and with
| only a few clicks. I was instantly a convert.
| svnpenn wrote:
| MITM proxy doesnt include any builtin way to install a system
| certificate. So if whatever you are trying to monitor doesnt
| accept user certificate (which is most stuff), then MITM proxy
| wont even work:
|
| https://github.com/mitmproxy/mitmproxy/issues/4838
| e12e wrote:
| > So if whatever you are trying to monitor doesnt accept user
| certificate (which is most stuff),
|
| There's certainly examples that does not use openssl/gnutls
| (and compatible friends) - but it's a bit of a stretch to say
| _most_ stuff doesn 't support it?
|
| Most (all) Linux distros also have an easy way to add a
| system level cert, without messing with system _managed_
| certs. And AFAIK it 's straightforward to install custom
| certs in the windows cert store as well.
|
| > MITM proxy doesnt include any builtin way to install a
| system certificate.
|
| Absolutely fair point of comparison. Most tls stacks will
| allow you to do this - via environment vars - so you can set
| a cert path for openssl when launching a ruby (or nodejs?)
| process, and things will just work.
|
| But you then need to know mitmproxy and your tls stack.
___________________________________________________________________
(page generated 2021-12-20 23:00 UTC)