tuntuntun – Combine Multiple Internet Connections Into One

GitHub repo: https://github.com/erjiang/tuntuntun (proof of concept status)

I was trying to play Minecraft by tethering over a Sprint data connection but was having awful random latency and dropped packets. The Sprint hotspot seems to only allow a limited number of connections to utilize the bandwidth at a time – a download in Chrome would sometimes stall all other connections. This was a huge problem in Minecraft, as loading the world chunks would stall my movements, meaning that I could teleport somewhere and die to an enemy by the time the map finished loading.

I’ve been seeing the idea of channel bonding here and there for a while, and it always seems like a cool idea without any popular and free implementations. Most of the approaches, though, were restricted to assigning different connections to different network interfaces. Essentially, a connection to YouTube might go over one link, while a software download might go out another. This works OK if you’re trying to watch YouTube while downloading updates, and it works great for many-connection uses like BitTorrent but in this case, I wanted to create a single load-balanced connection. So, I created tuntuntun.

Somewhat like a VPN

This requires the help of an outside server to act as the proxy for all of the connections, because most current Internet protocols require connections to originate from one address. The idea is to establish a connection to the proxy using a custom protocol that allows data to be split between two links. Tuntuntun works by encapsulating IP traffic in UDP packets, and does this in userspace using tun interfaces. A tun interface is a virtual network interface that has a userspace program as the other end. This means that any packets sent to the tun are read by the userspace program, and anything that the userspace program writes to it becomes “real” packets in the kernel.

Tuntuntun establishes a UDP socket on each interface that you bond. In the hypothetical scenario where eth0 and eth1 are combined, then tuntuntun will open two sockets and use SO_BINDTODEVICE to force these two sockets to only use eth0 and eth1. Then, it creates another interface, tun0, and modifies the routing table so that the operating system routes all Internet traffic through tun0. This way, tuntuntun can intercept all Internet traffic on the machine.

On the server side, the server process listens to a certain port for UDP packets and also opens a tun interface. It does the same basic function of passes packets back and forth, like a middleman. This is effectively what software like OpenVPN does, except the purpose is quite different.

Load-balancing packets

The client, currently, just uses a round-robin load-balancer that will send all even packets out eth0 and all odd packets out eth1. Effectively, all normal packets that go into tun0 on the client are wrapped in a UDP packet and then sent to the server. The server just needs to receive these packets, extract the original packet, and push it out to the Internet. Going the other way, any packets that the server receives from the Internet that are meant for the client are wrapped in a UDP packet and sent back to the client. This introduces a small data overhead – the MTU for the load-balanced connection is a few bytes smaller than the smallest of the individual links.

There are some things that aren’t implemented yet. Smarter scheduling strategies could be used for certain use cases. For example, if you are using a metered connection and an unmetered connection, you could choose to have all packets sent out the unmetered connection until it’s maxed out, and any extra packets would go out the metered connection. Or, in the case of two asymmetric links, send 70% of packets out the higher-bandwidth link, and send the other 30% out the lower-bandwidth link.

The other problem is dealing with out-of-order packets. If one connection has a higher latency than the other (and this is going to be the case if you’re using two carriers for redundancy), then packets will arrive out of order. Normally, TCP deals with this, but since we have a custom protocol between the client and server, then we can write a sequence number into every packet and then do our own reordering of any received packets.

Why Go?

Most of the program is written in Go. Why Go? Go’s concurrency made it easier to listen to multiple sockets at once. For sending/receiving data split across multiple connections, it was natural to model receiving and sending as one channel each. We don’t care about which interface a packet came in through, so we stick them all in the same pipe.

http://notes.ericjiang.com/posts/676

Plus, I had to integrate some C code, and the types in stdint.h and Go match up quite well. Using cgo to call some C functions from Go was a breeze. It was unfortunate that dealing with the OS required me to scrap a bunch of Go code after I realized that I hit the limits of Go’s net library, but the amount of C is relatively small.

Go’s type system is OK. Just having a static type system saved me a lot of development time, and Go’s interface type helped me replace the underlying sockets code by mimicking the original interface, so I didn’t have to modify the calling code. Is Go’s type system the best? No, but I’d rather have it than a dynamically typed language.

Go’s approach to error handling worked quite well. The likely alternative language, C, makes it easy to not handle errors, and that can be a deadly footgun when dealing with a lot of library calls (do I check errno? or the return code? or just ignore it and pray?). With Go, ignoring errors is explicit and intentional.

Future work…

Currently, tuntuntun is very rough around the edges.

Absolutely no security is implemented yet. Some mechanism for authentication between the server and client would be nice, and as of now, I recommend that you don’t run this on the public Internet. Server-client encryption is probably outside of the scope of this project – you can instead run a real VPN through a tuntuntun connection to get encrypted transport

There is more code that could be written to deal with interfaces that go away or come back. For example, if wlan0 is a USB radio that accidentally gets unplugged, tuntuntun continues to try to use it. But ideally, it would catch the error when trying to use wlan0, drop that interface from the list, and then periodically try recreating a socket on wlan0.

Partially implemented but not used yet is a circular buffer that can be used for packet reordering, and a simple ping test that can be used to measure latency or detect broken connections. A ping test would enable automatic connection failover.

For monitoring the status of tuntuntun (e.g. monitoring link status, bandwidth, ping, etc.), the current plan is to record statistics in the iface struct and then expose a JSON API that can be combined with an Angular UI. Since writing web servers in Go is so easy, this seems like a natural way of doing it.

GitHub repo: https://github.com/erjiang/tuntuntun (proof of concept status)