Title: Fair Internet bandwidth management on a network using OpenBSD
Author: Solène
Date: 30 August 2021
Tags: openbsd bandwidth
Description:
# Introduction
I have a simple DSL line with a 15 Mb/s download and 900 kb/s upload
rates and there are many devices using the Internet and two people in
remote work. Some poorly designed software (mostly on windows) will
auto update without allowing to reduce the bandwidth or some huge
bloated website will require lot of download and will impact workers
using the network.
The point of this article is to explain how to use OpenBSD as a router
on your network to allow the Internet access to be used fairly by
devices on the network to guarantee everyone they will have at least a
bit of Internet to continue working flawlessly.
I will use the queuing features from the OpenBSD firewall PF (Packet
Filter) which relies on the CoDel network scheduler algorithm, which
seems to bring all the features we need to do what we want.
(HTM) pf.conf manual page: QUEUEING section
(HTM) Wikipedia page about the CoDel network scheduler algorithm
# Important
I'm writing this in a separate section of the article because it is
important to understand.
It is not possible to limit the download bandwidth, because once the
data are already in the router, this mean they came from the modem and
it's too late to try to do anything. But there is still hope, if the
router receives data from the Internet it's that some devices on the
network asked to receive it, you can act on the uploaded data to
throttle what we receive. This is not obvious at first but it makes
totally sense once you get the idea.
The biggest point to understand is that you can throttle download speed
through the ACK packets. Think of two people on a phone, let's say
Alice and Bob, Alice is your network and calls Bob who is very happy to
tell his life to Alice. Bob speaking is data you download. In a
normal conversation, Bob will talk and will hear some sounds from Alice
who acknowledge what Bob is saying. If Alice stops or shut her
microphone, Bob may ask if Alice is still listening and will wait for
an answer. When Alice is making a sound (like "hmmhm or yes"), this is
an acknowledgement for Bob to continue. Literally, Bob is sending a
voice stream to Alice who is sending ACK (acknowledgement short name)
packets to Bob so he can continue.
This is exactly where you can control bandwidth, if we reduce the
bandwidth used by ACK packets for a download, you can reduce the given
download. If you can allow multiple systems to fairly send their share
of ACK, they should have a fair share of the downloaded data.
What's even more important is that you absolutely don't use all the
upload bandwidth with ACK packets to reach your maximum download
bandwidth. We will have to separate ACK from uploaded data so we don't
limit file upload or similar flows.
# Setup
For the setup I used a laptop with two network cards, one was connected
to the ISP box and the other was on the LAN side. I've enabled a DHCP
server on the OpenBSD router to automatically give IP addresses and
gateway and name servers addresses to devices on the network.
Basically, you can just plug an equivalent router on your current LAN,
disable DHCP on your ISP router and enable DHCP on your OpenBSD system
using a different subnet, both subnets will be available on the network
but for tests it requires little changes, when you want to switch from
a router to another by default, toggle the DHCP service on both and
renew DHCP leases on your devices. This is extremely easy.
```ASCII network diagram
+---------+
| ISP |
| router |
+---------+
|
|
| re0
+---------+
| OpenBSD |
| router |
+---------+
| em0
|
|
+---------+
| network |
| switch |
+---------+
```
# Configuration explained
## Line by line
I'll explain first all the config lines from my /etc/pf.conf file, and
later in this article you will find a block with the complete rules
set.
The following lines are default and can be kept as-is except if you
want to filter what's going in or out, but it's another topic as we
only want to apply queues. Filtering would be as usual.
```pf.conf configuration line
set skip on lo
block return # block stateless traffic
pass # establish keep-state
```
This is where it get interesting. The upstream router is accessed
through the interface re0, so we create a queue of the speed of the
link of that interface, which is 1 Gb/s. pf.conf syntax requires to
use bits per second (b/s or bps) and not bytes per second (Bps or B/s)
which can be misleading.
```pf.conf configuration line
queue std on re0 bandwidth 1G
```
Then, we create a queue that inherits from the parent created before,
this represent the whole upload bandwidth to reach the Internet. We
will make all the traffic reaching the Internet to go through this
queue.
I've set a bandwidth of 900K with a max of 900K, this mean, that this
queue can't let pass more than 900 kilo bits per second (which
represent 900/8 = 112.5 kB/s or kilo Bytes per second). This is the
extreme maximum my Internet access allows me.
```pf.conf configuration line
queue internet parent std bandwidth 900K max 900K
```
The following lines are all sub queues to divide the upload usage, we
want to have a separate queue for DNS request which must not be delayed
to keep responsiveness, but also voip or VPN queues to guarantee a
minimum available for the users.
The web queue is the one which is likely to pass the most data, if you
upload a file through a website, it will pass through the web queue.
The unknown queue is the outgoing traffic that is not known, it's up to
you to put a maximum or not.
Finally, the ackp queue that is split into two other queues, it's the
most important part of the setup.
The "bandwidth xxxK" values should sum up to something around the 900K
defined as a maximum in the parent, this only mean we target to keep
this amount for this queue, this doesn't enforce a minimum or a maximum
which can be defined with min and max keywords.
As explained earlier, you can control the downloading speed by
regulating the sent ACK packets, all ACK will go through the queues
ack_web and ack.
ack_web is a queue dedicated for http/https downloads and the other ack
queue is used for other protocol, I preferred to divide it in two so
other protocol will have a bit more room for themselves to
counterbalance a huge http download (Steam game platform like to make
things hard on this topic by making downloads to simultaneous server
for maximum bandwidth usage).
The two ack queues accumulated can't get over the parent queue set as
406K here. Finding the correct value is empirical, I'll explain later.
All these queues created will allow each queue to guarantee a minimum
from the router point of view, roughly said per protocol here.
Unfortunately, this won't guarantee computers on the network will have
a fair share of the queues! This is a crucial understanding I lacked
at first when trying to do this a few years ago. The solution is to
use the "flow" scheduler by using the flow keyword in the queue, this
will give some slot to every session on the network, guarantying (at
least theoretically) every session have the same time passed to send
data.
I used "flows" only for ACK, it proved to work perfectly fine for me as
it's the most critical part but in fact, it could be applied to every
leaf queues.
```pf.conf configuration line
queue web parent internet bandwidth 220K qlimit 100
queue dns parent internet bandwidth 5K
queue unknown parent internet bandwidth 150K min 100K qlimit 150 default
queue vpn parent internet bandwidth 150K min 200K qlimit 100
queue voip parent internet bandwidth 150K min 150K
queue ping parent internet bandwidth 10K min 10K
queue ackp parent internet bandwidth 200K max 406K
queue ack_web parent ackp bandwidth 200K flows 256
queue ack parent ackp bandwidth 200K flows 256
```
Because packets aren't magically assigned to queues, we need some match
rules for the job. You may notice the notation with parenthesis, this
mean the second member of the parenthesis is the queue dedicated for
ACK packets.
The VOIP queuing is done a bit wide, it seems Microsoft Teams and
Discord VOIP goes through these port ranges, it worked fine from my
experience but may depend of protocols.
```pf.conf configuration line
match proto tcp from em0:network to any queue (unknown,ack)
match proto tcp from em0:network to any port { 80 443 8008 8080 } queue (web,ack_web)
match proto tcp from em0:network to any port { 53 } queue (dns,ack)
match proto udp from em0:network to any port { 53 } queue dns
# VPN (wireguard, ssh, openvpn)
match proto udp from em0:network to any port { 4443 1194 } queue vpn
match proto tcp from em0:network to any port { 1194 22 } queue (vpn,ack)
# voip (teams)
match proto tcp from em0:network to any port { 3479 50000:50060 } queue voip
match proto udp from em0:network to any port { 3479 50000:50060 } queue voip
# keep some bandwidth for ping packets
match proto icmp from em0:network to any queue ping
```
Simple rule to enable NAT so devices from the LAN network can reach the
Internet.
```pf.conf configuration line
# NAT to the outside
pass out on egress from !(egress:network) nat-to (egress)
```
Default OpenBSD rules that can be kept here.
```pf.conf configuration line
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
# Port build user does not need network
block return out log proto {tcp udp} user _pbuild
```
## How to choose values
In the previous section I used absolute values, like 900K or even 406K.
A simple way to define them is to upload a big file to the Internet
and check the upload rate, I use bwm-ng but vnstat or even netstat
(with the correct combination of flags) could work, see your average
bandwidth over 10 or 20 seconds while transferring, and use that value
as a maximum in BITS as a maximum for the internet queue.
As for the ACK queue, it's a bit more tricky and you may tweak it a
lot, this is a balance between full download mode or conservative
download speed. I've lost a bit of download rate for the benefit of
keeping room for more overall responsiveness. Like previously, monitor
your upload rate when you download a big file (or even multiples files
to be sure to fill your download link) and you will see how much will
be used for ACK. It will certainly be a few try and guesses before you
get the perfect value, too low and the maximum download rate will be
reduced, and too high and your link will be filled entirely when
downloading.
## Full configuration
```pf.conf configuration file
set skip on lo
block return # block stateless traffic
pass # establish keep-state
queue std on re0 bandwidth 1G
queue internet parent std bandwidth 900K min 900K max 900K
queue web parent internet bandwidth 220K qlimit 100
queue dns parent internet bandwidth 5K
queue unknown parent internet bandwidth 150K min 100K qlimit 120 default
queue vpn parent internet bandwidth 150K min 200K qlimit 100
queue voip parent internet bandwidth 150K min 150K
queue ping parent internet bandwidth 10K min 10K
queue ackp parent internet bandwidth 200K max 406K
queue ack_web parent ackp bandwidth 200K flows 256
queue ack parent ackp bandwidth 200K flows 256
match proto tcp from em0:network to any queue (unknown,ack)
match proto tcp from em0:network to any port { 80 443 8008 8080 } queue (web,ack_web)
match proto tcp from em0:network to any port { 53 } queue (dns,ack)
match proto udp from em0:network to any port { 53 } queue dns
# VPN (ssh, wireguard, openvpn)
match proto udp from em0:network to any port { 4443 1194 } queue vpn
match proto tcp from em0:network to any port { 1194 22 } queue (vpn,ack)
# voip (teams)
match proto tcp from em0:network to any port { 3479 50000:50060 } queue voip
match proto udp from em0:network to any port { 3479 50000:50060 } queue voip
# ICMP
match proto icmp from em0:network to any queue ping
# NAT
pass out on egress from !(egress:network) nat-to (egress)
# default OpenBSD rules
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
# Port build user does not need network
block return out log proto {tcp udp} user _pbuild
```
# How to monitor
There is an excellent tool to monitor the queues in OpenBSD which is
systat in its queue view. Simply call it with "systat queue", you can
define the refresh rate by pressing "s" and a number. If you see
packets being dropped in a queue, you can try to increase the qlimit of
the queue which is the amount of packets kept in the queue and delayed
(it's a FIFO) before dropping them. The default qlimit is 50 and may
be too low.
(HTM) systat man page anchored to the queues parameter
# Conclusion
I've spent a week scrutinizing pf.conf manual and doing many tests with
many hardware until I understand that ACK were the key and that the
flow queuing mode was what I was looking for. As a result, my network
is much more responsive and still usable even when someone/some device
is using the network without any kind of limit.
The setup can appear a bit complicated but in the end it's only a few
pf.conf lines and using the correct values for your internet access. I
chose to make a lot of queues, but simply separating ack from the
default queue may be enough.