Set up and Verify Applications and Terminal Commands to Work Reliably with Proxy on macOS and Linux

Wasin Thonkaew - Mar 14, 2017

If you live behind firewall i.e. GFW and interact with terminal commands a lot in day-to-day basis and of course surfing Internet, I believe you usually utilize proxy to carry on your request in Internet world.

Without a doubt, verifying that those applications espcially browser and commands behave as you set and expect it to be is crucial. Wasting hours in waiting for return from npm install command, or even git clone <project-url> for project with huge amount of files but not necessary in bigger size without knowing whether it's going to succeed or not is not a good way to ruin your productive day. Leave alone seeing result from non-working proxy setup on browser, athough it's immediate and a lot faster to troubleshoot but it is also applied.

Thus this time, I think it's a good idea to sit down and make sure those applications and commands works reliably with proxy setup (by any mean).

Proxy Setup

Your proxy setup can be anything. It just needs to be a middle man between you and destination, and acts as a gatekeeper to forward your request to destination in which destination will not know that such request is coming from you but proxy itself.

We're interested in HTTP/HTTPs, and SOCKS5 proxy.

There're bunch of options to setup proxy. It can be Shadowsocks (which operates on HTTP, and SOCKS5 proxy), or just normal HTTP proxy (several out there such as Charles Proxy, TinyProxy, etc).

Solution presented in this writing aims for macOS, and Linux.

For my case, I'm on macOS 10.12.4, with Shadowsock proxy set up. This mean my local machine provides HTTP, and SOCKS5 proxy. So I have options to either set proxy to use HTTP, or SOCKS5.

Set up to Use Proxy for UI-based applications

Although not limited to browser, it is a prime example of this kind of application.
Let's proceed with Chrome as it's widely used, and is a top choice to debug and work with by developers. It should be very similar steps for other variant of browsers as well.

Secure Web Proxy (HTTPS) - 127.0.0.1 with port 1087
My proxy doesn't provide HTTPS thus I set it to just normally use what it provides in HTTP. So this means it will forward HTTPS request to HTTP one.

Personally, I don't think we really need HTTPS for local proxy stuff as it waste time and resource on doing handshake, encryption and decryption in which we don't really need it if we know we are on the secure side. Of course proxy will handle secure request after that (if we browse or request to secure site).

We will see how we can verify whether it can operate as expect in Verifying section later. Now if you can browse without 404 error, that means most likely it works as expect.

In additional, from my use case I didn't have other UI-based applications that needs to be configured its network to be through proxy. So in case you have some other applications that need to work with proxy. Occasionally, that application should provide such configuration similar to what we've done above. Just look inside its preferences, or setting somewhere.

Set up to Use Proxy for Terminal Commands

This part might need slightly more effort. Some commands have their own way of configuring proxy setting. They might need to be set via either environment variables, or specific configuration file. Some commands might not work with SOCKS proxy but only HTTP proxy.

After that execute source ~/.bash_profile to source the setting and make sure those environment variables are taken into effect. I recommend to configure it with your ~/.bash_profile script as it will take effect even after reboot.

Beware that if you set it to SOCKS5 proxy, commands that doesn't support SOCKS5 will shout out error. A great example is ngrok which doesn't support SOCKS5 but it will let you know.

So in general case, HTTP/HTTPS proxy might serve you better unless you're willing to re-configure it back and forth.

Verifying

Now it's time to verify our proxy configuration no matter which way you had done to configure it.

You have an option to use UI-based application i.e. WireShark. But I prefer commandline-based solution. Anyway, the same concept can be applied to use with WireShark too.

Preparation

We're going to use ngrep to capture packets for our local network interface filtering for port we're interested in knowing either 1087 for HTTP/HTTPS, or 1086 for SOCKS5.

From result, we know that our local network interface's name is lo0 which you can see it has IP address of 127.0.0.1.

Verify npm

We had ngrep and local network interface name already. Next we're going to verify npm install bfet command whether it actually went through our proxy as set to be on HTTP/HTTPS (port 1087) setting or not.

Execute sudo ngrep -q -d lo0 -W byline port 1087
There're bunch of options used in this command. You can see this man page for explanation of each parameter used in above command. Basically, it will start to capture packets on lo0 network interface filtering for packets involving port 1087 with other options affecting how much of information should printed out on console.

You can also remove port 1087 out thus more packets although might not be the one you're interested will show on console but for the ease in digesting information here let's go with port filtering.

It uses HTTP CONNECT method to initiate HTTP tunneling by which the command (running on port 63066) asks local proxy server (port 1087) to forward such HTTPS request. The tunnel (thus CONNECT method) is especially created for requesting HTTPS website which is registry.npmjs.org:443 as seen in this case.

T 127.0.0.1:1087 -> 127.0.0.1:63066 [AP]

Proxy sent acknowledge packet back indicating that it's successfully created a HTTP tunnel. It's ready to exchange data through proxy from now on.

T 127.0.0.1:63066 -> 127.0.0.1:1087 [AP]

One of many packets sending back and forth during HTTP tunneling session. In this case, npm command usually sends some request for a package at registry.npmjs.org. It is encrypted due to SSL as it's end-to-end communication. The dot (.) we see is default character to show when such character is non-printable.

So you can see that the connection is well made with proxy. Basically verified that it works as expect.

Verify curl

We had ngrep and local network interface name as previously tested. Next we're going to verify curl -i http://blog.wasin.io command whether it actually went through our proxy as set to be on SOCKS5 (port 1086) or not.

Reason to use http://blog.wasin.io is that it's not HTTPS thus we're going to see actual text result from request to such target website.

You see that such command on port 52924 created a request message in SOCKS5 message and sent it to local proxy server listening on port 1086. In return, proxy gets the actual response back including website's source code in HTML.

So it works as expect.

Conclusion

We have learned how to properly configure proxy setting for UI-based applications and terminal commands. Also we knew a basic way to verify that our tools are working as expect via ngrep. Combine these together to carry out Internet operation through proxy in network environment that needs it, and be able to troubleshoot the problem when we faced with a long waiting time in getting a result back from using application or commandlines.

I blog about projects I'm working on. Mostly about mobile games, tech, web service, little tiny things I built for myself or for public, and tech/game industry-wide. Currently I'm living in Shenzhen, China. I'm running a small creative company with my girlfriend here.