[HN Gopher] Build Your Own Mobile Proxy for Web Scraping
___________________________________________________________________
Build Your Own Mobile Proxy for Web Scraping
Author : mateuszbuda
Score : 117 points
Date : 2022-08-17 14:30 UTC (8 hours ago)
(HTM) web link (scrapingfish.com)
(TXT) w3m dump (scrapingfish.com)
| almog wrote:
| Have you tried probing for 4G provider(s?) IP renew limits and IP
| allocation space size?
|
| Things I can think of that would be interesting to know:
| 1. How uniformly distributed are the IP addresses you're getting.
| 2. The size of the IP addresses allocation space, assuming a
| fixed geographic location (I imagine this can vary a lot between
| providers and regions) 3. How often can an IP be requested?
| 4. Is there some sort of rate limiting? If so, how does it
| manifest itself (Sticky IP? slow reconnect? Something else?)?
| mateuszbuda wrote:
| The distribution of IP addresses depends to a large extent on
| 4G provider. Some of them reuse IPs and you can get the same IP
| after you change network mode 4G > 3G > 4G. The same applies to
| IP addresses allocation space. You can request IP change as
| often as you want. Resetting network mode takes 5-10 seconds.
| There is not rate limiting. From time to time reconnection
| fails but it has nothing to do with rate limiting. Sticky IP is
| an issue for some 4G providers but, again, it has nothing to do
| with rate limiting
| Anunayj wrote:
| I was wondering, is it possible to use these 4G USB modem to make
| voice calls over VoLTE? Can somebody direct me to resources that
| make it possible?
| Nextgrid wrote:
| Kind of.
|
| The modem can attach to the IMS APNs, however you'll need to do
| SIM-specific authentication which requires being able to send
| the right APDUs to the SIM. Some modems expose the SIM card in
| one way or another (either an AT command to send APDUs and get
| responses or as an actual USB smartcard reader) and it would be
| possible with those to do the authentication flow and then
| register onto the IMS server (which speaks SIP).
|
| Alternatively, if you're happy with circuit-switched calls,
| some modems will expose the raw audio (as PCM data) as a
| separate serial port from which you read/write, and control the
| call flow using AT commands on the main serial port. Some
| modems need this to be first enabled via a poorly-documented AT
| command, and some might need authentication (security by
| obscurity) to do so which software such as DC Unlocker can
| break.
| g_p wrote:
| In addition to the EAP-AKA' authentication, the strange and
| highly opaque ways through which VoWiFi support is signalled
| and enabled might be an issue here too.
|
| I've seen brand new Pixel devices (which fully support
| VoWifi) act like they have never heard of the feature, until
| a special cell broadcast SMS is received from the carrier, to
| turn the feature on.
|
| It would be interesting to see if you could manually tick all
| the boxes and open up a tunnel to the ePDG etc without that
| happening, but given the number of moving parts, and the
| differences between each network's implementation, I'd be
| sceptical it would work reliably.
|
| (I'm pretty sure I had a situation where individual batches
| of IMEIs were being whitelisted for WiFi calling on another
| network, where the switch appeared, but VoWiFi didn't work...
| Yet on the Nexus device in question, one bought from the
| carrier directly did work. Even when you manually flashed
| both devices with identical firmwares from Google directly,
| this remained the case - must have been an IMEI whitelist or
| similar).
| Anunayj wrote:
| Ah I see, so I did find someone that tried to do this here
| [1], but it seems like they were not even able to connect to
| the SIP server. Bummer, I wonder if someone else has had
| success.
|
| 1. https://worthdoingbadly.com/vowifi/
| upupandup wrote:
| why dont you let us know what you really intend to do here
| Anunayj wrote:
| Just making voice calls from my laptop lol, or better over
| internet from a RPI connected to this modem.
| upupandup wrote:
| you can already to this with 4g/lte dongles, a proxy seems
| overkill
| Anunayj wrote:
| I wasn't aware, and most of them do not advertise such a
| feature. It would be great if you could point me to one
| that allows this out of the box.
| gsich wrote:
| Examples?
|
| https://laforge.gnumonks.org/blog/20170902-cellular_modem
| s-v...
|
| comes to mind. Regardless of age.
| Nextgrid wrote:
| Keep in mind that this doesn't need to be a Raspberry Pi and can
| be any computer. With some networking magic (and potentially
| network namespaces) you can also run multiple dongles & proxies
| on the same machine. A bit of hardware engineering will allow you
| to decouple the SIM card from the modems themselves, so SIMs can
| be kept at a central location and assigned to modems at will,
| potentially automatically depending on remaining credit/carrier
| blocks/rate-limiting/etc.
| getcrunk wrote:
| How would you decouple the sims from the modems? Would it just
| be about reading some info off the sim and then passing it to
| the modem?
| monai wrote:
| One of the Osmocom projects[0] does precisely that.
|
| [0] https://osmocom.org/projects/osmo-remsim/wiki
| pawelkobojek wrote:
| Yes, RPi is totally optional. This article just provides an
| example setup.
| cooldrcool3 wrote:
| What would the total cost be for everything needed?
| pawelkobojek wrote:
| That's not easy to estimate, depends on your location and if
| you want to use RPi.
|
| A (very) rough estatimate is ~$40 per dongle, RPi starts from
| $35, totalling ~$115. You'd also need to buy two 4G plans.
| szoniacz wrote:
| Good insight!
| moonit wrote:
___________________________________________________________________
(page generated 2022-08-17 23:01 UTC)