Category Archives: FreeBSD

Configuring OpenBGPD to announce VM’s virtual networks

We use BGP quite heavily at work, and even though I'm not interacting with that directly, it feels like it's something very useful to learn at least on some basic level. The most effective and fun way of learning technology is finding some practical application, so I decided to see if it could help to improve networking management for my Virtual Machines.

My setup is fairly simple: I have a host that runs bhyve VMs and I have a desktop system from where I ssh to VMs, both hosts run FreeBSD. All VMs are connected to each other through a bridge and have a common network 10.0.1/24. The point of this exercise is to be able to ssh to these VMs from desktop without adding static routes and without adding vmhost's external interfaces to the VMs bridge.

I've installed openbgpd on both hosts and configured it like this:

vmhost: /usr/local/etc/bgpd.conf

AS 65002
router-id 192.168.87.48
fib-update no

network 10.0.1.1/24

neighbor 192.168.87.41 {
descr "desktop"
remote-as 65001
}

Here, router-id is set vmhost's IP address in my home network (192.168.87/24), fib-update no is set to forbid routing table update, which I initially set for testing, but keeping it as vmhost is not supposed to learn new routes from desktop anyway. network announces my VMs network and neighbor describes my desktop box.

Now the desktop box:

desktop: /usr/local/etc/bgpd.conf

AS 65001
router-id 192.168.87.41
fib-update yes

neighbor 192.168.87.48 {
descr "vmhost"
remote-as 65002
}

It's pretty similar to vmhost's bgpd.conf, but no networks are announced here, and fib-update is set to yes because the whole point is to get VM routes added.

Both hosts have to have the openbgpd service enabled:

/etc/rc.conf.local

openbgpd_enable="YES"

Now start the service (or wait until next reboot) using service openbgpd start and check if neighbors are there:

vmhost: bgpctl show summary

$ bgpctl show summary                                                                                                                                                                    
Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd
desktop 65001 1089 1090 0 09:03:17 0
$

desktop: bgpctl show summary

$ bgpctl show summary
Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd
vmhost 65002 1507 1502 0 09:04:58 1
$

Get some detailed information about the neighbor:

desktop: bgpctl sh nei vmhost

$ bgpctl sh nei vmhost                                                                                                                                                                    
BGP neighbor is 192.168.87.48, remote AS 65002
Description: vmhost
BGP version 4, remote router-id 192.168.87.48
BGP state = Established, up for 09:06:25
Last read 00:00:21, holdtime 90s, keepalive interval 30s
Neighbor capabilities:
Multiprotocol extensions: IPv4 unicast
Route Refresh
Graceful Restart: Timeout: 90, restarted, IPv4 unicast
4-byte AS numbers

Message statistics:
Sent Received
Opens 3 3
Notifications 0 2
Updates 3 6
Keepalives 1499 1499
Route Refresh 0 0
Total 1505 1510

Update statistics:
Sent Received
Updates 0 1
Withdraws 0 0
End-of-Rib 1 1

Local host: 192.168.87.41, Local port: 179
Remote host: 192.168.87.48, Remote port: 13528

$

By the way, as you can see, bgpctl supports shortened commands, e.g. sh nei instead of show neighbor.

Now look for that VMs route:

desktop: bgpctl show rib

$ sudo bgpctl show rib
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination gateway lpref med aspath origin
*> 10.0.1.0/24 192.168.87.48 100 0 65002 i
$

So that VMs network, 10.0.1/24, it's there! Now check if the system routing table was updated and has this route:

desktop

$ route -n get 10.0.1.45   
route to: 10.0.1.45
destination: 10.0.1.0
mask: 255.255.255.0
gateway: 192.168.87.48
fib: 0
interface: re0
flags:
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
$ ping -c 1 10.0.1.45
PING 10.0.1.45 (10.0.1.45): 56 data bytes
64 bytes from 10.0.1.45: icmp_seq=0 ttl=63 time=0.192 ms

--- 10.0.1.45 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.192/0.192/0.192/0.000 ms
$

Whoa, things work as expected!

Conclusion

As mentioned already, similar result could be achieved without using BGP by using either static routes or bridging interfaces differently, but the purpose of this exercise is to get some basic hands-on experience with BGP. Right now I'm looking into extending my setup in order to try more complex BGP schema. I'm thinking about adding some software switches in front of my VMs or maybe adding a second VM host (if budget allows). You're welcome to comment if you have some ideas how to extend this setup for educational purposes in the context of BGP and networking.

As a side note, I really like openbgpd so far. Its configuration file format is clean and simple, documentation is good, error and information messages are clear, and CLI has intuitive syntax.

Configuring OpenBGPD to announce VM’s virtual networks

We use BGP quite heavily at work, and even though I'm not interacting with that directly, it feels like it's something very useful to learn at least on some basic level. The most effective and fun way of learning technology is finding some practical application, so I decided to see if it could help to improve networking management for my Virtual Machines.

My setup is fairly simple: I have a host that runs bhyve VMs and I have a desktop system from where I ssh to VMs, both hosts run FreeBSD. All VMs are connected to each other through a bridge and have a common network 10.0.1/24. The point of this exercise is to be able to ssh to these VMs from desktop without adding static routes and without adding vmhost's external interfaces to the VMs bridge.

I've installed openbgpd on both hosts and configured it like this:

vmhost: /usr/local/etc/bgpd.conf

AS 65002
router-id 192.168.87.48
fib-update no

network 10.0.1.1/24

neighbor 192.168.87.41 {
descr "desktop"
remote-as 65001
}

Here, router-id is set vmhost's IP address in my home network (192.168.87/24), fib-update no is set to forbid routing table update, which I initially set for testing, but keeping it as vmhost is not supposed to learn new routes from desktop anyway. network announces my VMs network and neighbor describes my desktop box.

Now the desktop box:

desktop: /usr/local/etc/bgpd.conf

AS 65001
router-id 192.168.87.41
fib-update yes

neighbor 192.168.87.48 {
descr "vmhost"
remote-as 65002
}

It's pretty similar to vmhost's bgpd.conf, but no networks are announced here, and fib-update is set to yes because the whole point is to get VM routes added.

Both hosts have to have the openbgpd service enabled:

/etc/rc.conf.local

openbgpd_enable="YES"

Now start the service (or wait until next reboot) using service openbgpd start and check if neighbors are there:

vmhost: bgpctl show summary

$ bgpctl show summary                                                                                                                                                                    
Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd
desktop 65001 1089 1090 0 09:03:17 0
$

desktop: bgpctl show summary

$ bgpctl show summary
Neighbor AS MsgRcvd MsgSent OutQ Up/Down State/PrfRcvd
vmhost 65002 1507 1502 0 09:04:58 1
$

Get some detailed information about the neighbor:

desktop: bgpctl sh nei vmhost

$ bgpctl sh nei vmhost                                                                                                                                                                    
BGP neighbor is 192.168.87.48, remote AS 65002
Description: vmhost
BGP version 4, remote router-id 192.168.87.48
BGP state = Established, up for 09:06:25
Last read 00:00:21, holdtime 90s, keepalive interval 30s
Neighbor capabilities:
Multiprotocol extensions: IPv4 unicast
Route Refresh
Graceful Restart: Timeout: 90, restarted, IPv4 unicast
4-byte AS numbers

Message statistics:
Sent Received
Opens 3 3
Notifications 0 2
Updates 3 6
Keepalives 1499 1499
Route Refresh 0 0
Total 1505 1510

Update statistics:
Sent Received
Updates 0 1
Withdraws 0 0
End-of-Rib 1 1

Local host: 192.168.87.41, Local port: 179
Remote host: 192.168.87.48, Remote port: 13528

$

By the way, as you can see, bgpctl supports shortened commands, e.g. sh nei instead of show neighbor.

Now look for that VMs route:

desktop: bgpctl show rib

$ sudo bgpctl show rib
flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination gateway lpref med aspath origin
*> 10.0.1.0/24 192.168.87.48 100 0 65002 i
$

So that VMs network, 10.0.1/24, it's there! Now check if the system routing table was updated and has this route:

desktop

$ route -n get 10.0.1.45   
route to: 10.0.1.45
destination: 10.0.1.0
mask: 255.255.255.0
gateway: 192.168.87.48
fib: 0
interface: re0
flags:
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
$ ping -c 1 10.0.1.45
PING 10.0.1.45 (10.0.1.45): 56 data bytes
64 bytes from 10.0.1.45: icmp_seq=0 ttl=63 time=0.192 ms

--- 10.0.1.45 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.192/0.192/0.192/0.000 ms
$

Whoa, things work as expected!

Conclusion

As mentioned already, similar result could be achieved without using BGP by using either static routes or bridging interfaces differently, but the purpose of this exercise is to get some basic hands-on experience with BGP. Right now I'm looking into extending my setup in order to try more complex BGP schema. I'm thinking about adding some software switches in front of my VMs or maybe adding a second VM host (if budget allows). You're welcome to comment if you have some ideas how to extend this setup for educational purposes in the context of BGP and networking.

As a side note, I really like openbgpd so far. Its configuration file format is clean and simple, documentation is good, error and information messages are clear, and CLI has intuitive syntax.

Twenty years

Yesterday was the twentieth anniversary of my FreeBSD commit bit, and tomorrow will be the twentieth anniversary of my first commit. I figured I’d split the difference and write a few words about it today.

My level of engagement with the FreeBSD project has varied greatly over the twenty years I’ve been a committer. There have been times when I worked on it full-time, and times when I did not touch it for months. The last few years, health issues and life events have consumed my time and sapped my energy, and my contributions have come in bursts. Commit statistics do not tell the whole story, though: even when not working on FreeBSD directly, I have worked on side projects which, like OpenPAM, may one day find their way into FreeBSD.

My contributions have not been limited to code. I was the project’s first Bugmeister; I’ve served on the Security Team for a long time, and have been both Security Officer and Deputy Security Officer; I managed the last four Core Team elections and am doing so again this year.

In return, the project has taught me much about programming and software engineering. It taught me code hygiene and the importance of clarity over cleverness; it taught me the ins and outs of revision control; it taught me the importance of good documentation, and how to write it; and it taught me good release engineering practices.

Last but not least, it has provided me with the opportunity to work with some of the best people in the field. I have the privilege today to count several of them among my friends.

For better or worse, the FreeBSD project has shaped my career and my life. It set me on the path to information security in general and IAA in particular, and opened many a door for me. I would not be where I am now without it.

I won’t pretend to be able to tell the future. I don’t know how long I will remain active in the FreeBSD project and community. It could be another twenty years; or it could be ten, or five, or less. All I know is that FreeBSD and I still have things to teach each other, and I don’t intend to call it quits any time soon.

Not merging stuff from FreeBSD-HEAD into production branches, or "hey FreeBSD-HEAD should just be production"

I get asked all the time why I don't backport my patches into stable FreeBSD release branches. It's a good question, so let me explain it here.

I don't get paid to do it.

Ok, so now you ask "but wait, surely the users matter?" Yes, of course they do! But, I also have other things going on in my life, and the stuff I do for fun is .. well, it's the stuff I do for fun. I'm not paid to do FreeBSD work, let alone open source wireless stuff in general.

So then I see posts like this:

https://www.anserinae.net/adventures-in-wifi-freebsd-edition.html

I understand his point of view, I really do. I'm also that user when it comes to a variety of other open source software and I ask why features aren't implemented that seem easy, or why they're not in a stable release. But then I remember that I'm also doing this for fun and it's totally up to me to spend my time however I want.

Now, why am I like this?

Well, the short-hand version is - I used to bend over backwards to try and get stuff in to stable releases of the open source software I once worked on. And that was taken advantage of by a lot of people and companies who turned around to incorporate that work into successful commercial software releases without any useful financial contribution to either myself or the project as a whole. After enough time of this, you realise that hey, maybe my spare time should just be my spare time.

My hope is that if people wish to backport my FreeBSD work to a stable release then they'll either pay me to do it, pay someone else to do it, or see if a company will sponsor that work for their own benefit. I don't want to get into the game of trying to backport things to one and potentially two stable releases and deal with all the ABI changes and support fallout that happens when you are porting things into a mostly ABI stable release. And yes, my spare time is my own.


Not merging stuff from FreeBSD-HEAD into production branches, or "hey FreeBSD-HEAD should just be production"

I get asked all the time why I don't backport my patches into stable FreeBSD release branches. It's a good question, so let me explain it here.

I don't get paid to do it.

Ok, so now you ask "but wait, surely the users matter?" Yes, of course they do! But, I also have other things going on in my life, and the stuff I do for fun is .. well, it's the stuff I do for fun. I'm not paid to do FreeBSD work, let alone open source wireless stuff in general.

So then I see posts like this:

https://www.anserinae.net/adventures-in-wifi-freebsd-edition.html

I understand his point of view, I really do. I'm also that user when it comes to a variety of other open source software and I ask why features aren't implemented that seem easy, or why they're not in a stable release. But then I remember that I'm also doing this for fun and it's totally up to me to spend my time however I want.

Now, why am I like this?

Well, the short-hand version is - I used to bend over backwards to try and get stuff in to stable releases of the open source software I once worked on. And that was taken advantage of by a lot of people and companies who turned around to incorporate that work into successful commercial software releases without any useful financial contribution to either myself or the project as a whole. After enough time of this, you realise that hey, maybe my spare time should just be my spare time.

My hope is that if people wish to backport my FreeBSD work to a stable release then they'll either pay me to do it, pay someone else to do it, or see if a company will sponsor that work for their own benefit. I don't want to get into the game of trying to backport things to one and potentially two stable releases and deal with all the ABI changes and support fallout that happens when you are porting things into a mostly ABI stable release. And yes, my spare time is my own.


Reponse zones in BIND (RPZ/Blocking unwanted traffic).

A while ago, my dear colleague Mattijs came with an interesting option in BIND. Response zones. One can create custom "zones" and enforce a policy on that.

I never worked with it before, so I had no clue at all what to expect from it. Mattijs told me how to configure it (see below for an example) and offered to slave his RPZ policy-domains.

All of a sudden I was no longer getting a lot of ADS/SPAM and other things. It was filtered. Wow!

His RPZ zones were custom made and based on PiHole, where PiHole adds hosts to the local "hosts" file and sends it to 127.0.0.1 (your local machine), which prevents it to reach the actual server at all, RPZ policies are much stronger and more dynamic.

RPZ policies offer the use of "redirecting" queries. What do I mean with that? well you can force a ADVERTISEMENT (AD for short) site / domain to the RPZ policy and return a NXDOMAIN. It no longer exists for the end-user. But you can also CNAME it to a domain/host you own and then add a webserver to that host and tell the user query'ing the page: "The site you are trying to reach had been pro-actively blocked by the DNS software. This is an automated action and an automated response. If you feel that this is not appropriate, please let us know on <mail link>", or something like that.

Once I noticed that and saw the value, I immediately saw the benefit for companies and most likely schools and home people. Mattijs had a busy time at work and I was recovering from health issues, so I had "plenty" of time to investigate and read on this. The RPZ policies where not updated a lot and caused some problems for my ereaders for example (msftcncsi.com was used by them, see another post on this website for being grumpy about that). And I wanted to learn more about it. So what did I do?

Yes, I wrote my own parser. In perl. I wrote a "rpz-generator" (its actually called like that). I added the sources Mattijs used and generated my own files. They are rather huge, since I blocked ads, malware, fraud, exploits, windows stuff and various other things (gambling, fakenews, and stuff like that).

I also included some whitelists, because msfctinc was added to the lists and it made my ereaders go beserk, and we play a few games here and there which uses some advertisement sites, so we wanted to exempt them as well. It's better to know which ones they are and selectively allow them, then having traffic to every data collector out there.

This works rather well. I do not get a lot of complaints that things are not working. I do see a lot of queries going to "banned" sites everyday. So it is doing something .The most obvious one is that search results on google, not always are clickable. The ones that have those [ADV] sites, are blocked because they are advertising google sponsored sites, and they are on the list.. and google-analytics etc. It doesn't cause much harm to our internet surfing or use experience, with the exception of the ADV sites I just mentioned. My wife sometimes wants to click on those because she searches for something that happends to be on that list, but apart from that we are doing just fine.

One thing though, I wrote my setup and this article with my setup using "NXDOMAIN" which just gives back "site does not exist" messages. I want to make my script more smart by making it a selectable, so that some categories are CNAMED to a filtering domain and webpage, and some are NXDOMAIN'ed. If someone has experience with that, please show me some idea's and how that looks like and whether your end-users can do something with it or not. I think schools will be happy to present a block-page instead of NXdomain'ing some sites 🙂

Acknowledgements: Mattijs for teaching and showing me RPZ, ISC for placing RPZ in NAMED, and zytrax.com for having such excellent documentation to RPZ. The perl developers for having such a great tool around, and the various sites I use to get the blocklists from. Thank you all!

If you want to know more about the tool, please contact me and we can share whatever information is available 🙂

More TS-440S hijinx, or "ok, what if you wanna homebrew a digital hookup?"

I've been homebrewing digital hookups between my amateur radios (HF, VHF, UHF) and a FreeBSD PC. It all ... mostly works. There's one or two FreeBSD hiccups though, which are summarised thusly:

The default package selection for audio paths is .. suboptimal. Some can be configured to use OSS and that is nice. Some provide ALSA but FreeBSD's "ALSA" implementation doesn't provide full ALSA device emulation so we don't get a list of ALSA devices by default. You have to put your devices into asound.conf with names for things. However ..

.. FreeBSD doesn't currently make it easy to hard-code say, USB device paths to serial port names or sound devices to something predictable. So every time I reboot or mess with the setup it goes pear shaped.

Then there's a bug where I can run three USB audio devices, but I can't do mic input on the last one. Output works fine. That's going to be amusing to diagnose.

So I do have my TS-440S, TS-711A and TS-811E all doing digital modes. They're just all .. subtly different.

The TS-711A and TS-811E have an accessory jack (ACC2) that has input and output. There's a PTT control line and a mic mute line. The line levels of those signals is a couple hundred millivolts, so it's good enough to build a little resistor divider with a potentiometer to get the computer output down to the right level. I'm using mic input on the USB audio devices, so that also works fine at a couple hundred millivolts.

The TS-440S also has a similar accessory jack, however the audio input in that jack seems to be quite a big higher than a couple hundred millivolts. It looks like it needs to be around 4-5v peak-to-peak for it to be at the right level internally on the IF board. The ACC2 path has a couple of resistor attenuators so it looks like this was intentional - after asking around it looks like it's expecting professional line audio output levels (~4v peak-to-peak) instead of consumer grade levels (~1.5v peak-to-peak.) I'll go dig into it some more. This path bypasses the microphone pre-amplifier entirely and goes straight into the Mic Gain control pot.

The TS-440S also has AFSK input/output RCA jacks on the back. The audio output is at the same level as the ACC2 jack, however the audio input side is routed via the microphone input side so it gets preamp'ed and processed appropriately. That's what I've been using for digital modes - I can divide down the input side to a couple hundred millivolts to keep it all kosher. However - and here's the really annoying part - the mic mute input on ACC2 also mutes the AFSK input line.

Then there's what happens if you leave the microphone connected. If you do leave it connected, even in a quiet room, it seems to present some load that requires a lot more signal on AFSK input to do its thing. If you tune it all up to the right signal levels and then disconnect the microphone, you'll be really overdriving the RF section. Ugh.

So - I don't have to do any of this for the TS-711 and TS-811 - their input values are a lot lower and grounding the mic line actually just quietens the mic input.

If I can score a slightly different radio - like a TS-680S for example - then I can do this stuff with a much lower level line input value. It'll be tricky to get it down to the TS-680S level (it wants it at 10mV!) but at least I can do that with passive, well shielded bits.

On the plus side - yes, this means I at least can do digital modes on my TS-440S. I just have to keep unplugging the microphone line for now. What I may end up doing for now though is adding another switch to the desktop microphone I have to turn /its/ microphone input off so it is fully disconnected. Hopefully that'll be enough to do digital modes without constantly screwing and unscrewing things.

If you're at all curious - https://www.pskreporter.info/pskmap?callsign=kk6vqk&search=Find

More TS-440S hijinx, or "ok, what if you wanna homebrew a digital hookup?"

I've been homebrewing digital hookups between my amateur radios (HF, VHF, UHF) and a FreeBSD PC. It all ... mostly works. There's one or two FreeBSD hiccups though, which are summarised thusly:

The default package selection for audio paths is .. suboptimal. Some can be configured to use OSS and that is nice. Some provide ALSA but FreeBSD's "ALSA" implementation doesn't provide full ALSA device emulation so we don't get a list of ALSA devices by default. You have to put your devices into asound.conf with names for things. However ..

.. FreeBSD doesn't currently make it easy to hard-code say, USB device paths to serial port names or sound devices to something predictable. So every time I reboot or mess with the setup it goes pear shaped.

Then there's a bug where I can run three USB audio devices, but I can't do mic input on the last one. Output works fine. That's going to be amusing to diagnose.

So I do have my TS-440S, TS-711A and TS-811E all doing digital modes. They're just all .. subtly different.

The TS-711A and TS-811E have an accessory jack (ACC2) that has input and output. There's a PTT control line and a mic mute line. The line levels of those signals is a couple hundred millivolts, so it's good enough to build a little resistor divider with a potentiometer to get the computer output down to the right level. I'm using mic input on the USB audio devices, so that also works fine at a couple hundred millivolts.

The TS-440S also has a similar accessory jack, however the audio input in that jack seems to be quite a big higher than a couple hundred millivolts. It looks like it needs to be around 4-5v peak-to-peak for it to be at the right level internally on the IF board. The ACC2 path has a couple of resistor attenuators so it looks like this was intentional - after asking around it looks like it's expecting professional line audio output levels (~4v peak-to-peak) instead of consumer grade levels (~1.5v peak-to-peak.) I'll go dig into it some more. This path bypasses the microphone pre-amplifier entirely and goes straight into the Mic Gain control pot.

The TS-440S also has AFSK input/output RCA jacks on the back. The audio output is at the same level as the ACC2 jack, however the audio input side is routed via the microphone input side so it gets preamp'ed and processed appropriately. That's what I've been using for digital modes - I can divide down the input side to a couple hundred millivolts to keep it all kosher. However - and here's the really annoying part - the mic mute input on ACC2 also mutes the AFSK input line.

Then there's what happens if you leave the microphone connected. If you do leave it connected, even in a quiet room, it seems to present some load that requires a lot more signal on AFSK input to do its thing. If you tune it all up to the right signal levels and then disconnect the microphone, you'll be really overdriving the RF section. Ugh.

So - I don't have to do any of this for the TS-711 and TS-811 - their input values are a lot lower and grounding the mic line actually just quietens the mic input.

If I can score a slightly different radio - like a TS-680S for example - then I can do this stuff with a much lower level line input value. It'll be tricky to get it down to the TS-680S level (it wants it at 10mV!) but at least I can do that with passive, well shielded bits.

On the plus side - yes, this means I at least can do digital modes on my TS-440S. I just have to keep unplugging the microphone line for now. What I may end up doing for now though is adding another switch to the desktop microphone I have to turn /its/ microphone input off so it is fully disconnected. Hopefully that'll be enough to do digital modes without constantly screwing and unscrewing things.

If you're at all curious - https://www.pskreporter.info/pskmap?callsign=kk6vqk&search=Find

Postgres in FreeBSD Jails

In the past we had to use the old SysV IPC sysctls and change the UID that each PostgreSQL server ran as under in each Jail. Which was annoying and error prone.

I started down this path because I was trying to figure out which tweaks are needed to allow PostgreSQL to run. Initially I was seeing the service postgresql initdb error out with this error:

creating template1 database in /usr/local/pgsql/data/base/1 ... FATAL: could not create shared memory segment: Function not implemented

In digging into this problem and the fact that PostgreSQL should be able to use posix shared memory now, I discovered some new jail knobs. In FreeBSD 11.0 and later it is very easy to run PostgreSQL in a FreeBSD jail.

In your jail.conf, just add:

sysvmsg=new
sysvsem
sysvshm

Or if like me you are using ezjail, just add to config file for that specific jail on the _parameters line, like so:

export jail_mon_example_com_parameters="sysvmsg=new sysvsem sysvshm"

Then start the jail and install PostgreSQL:

pkg install postgresql95-server

Verify that we can initialize the database correctly:

service postgresql initdb

If that completes without error, as it should, go ahead and start up the PostgreSQL service:

service postgresql start

I really like this method as it keeps the database service isolated with the service it is running for and keeps the shared memory and the shared memory config isolated into that one jail.

FreeBSD and APRS, or "hm what happens when none of this is well documented.."

Here's another point along my quest for amateur radio on FreeBSD - bring up basic APRS support. Yes, someone else has done the work, but in the normal open source way it was .. inconsistently documented.

First is figuring out the hardware platform. I chose the following:

  • A Baofeng UV5R2, since they're cheap, plentiful, and do both VHF and UHF;
  • A cable to do sound level conversion and isolation (and yes, I really should post a circuit diagram and picture..);
  • A USB sound device, primarily so I can whack it into FreeBSD/Linux devices to get a separate sound card for doing radio work;
  • FreeBSD laptop (it'll become a raspberry pi + GPS + sensor + LCD thingy later, but this'll do to start with.)
The Baofeng is easy - set it to the right frequency (VHF APRS sits on 144.390MHz), turn on VOX so I don't have to make up a PTT cable, done/done.

The PTT bit isn't that hard - one of the microphone jack pins is actually PTT (if you ground it, it engages PTT) so when you make the cable just ensure you expose a ground pin and PTT pin so you can upgrade it later.

The cable itself isn't that hard either - I had a baofeng handmic lying around (they're like $5) so I pulled it apart for the cable. I'll try to remember to take pictures of that.

Here's a picture I found on the internet that shows the pinout:


Now, I went a bit further. I bought a bunch of 600 ohm isolation transformers for audio work, so I wired it up as follows:

  • From the audio output of the USB sound card, I wired up a little attenuator - input is 2k to ground, then 10k to the input side of the transformer; then the output side of the transformer has a 0.01uF greencap capacitor to the microphone input of the baofeng;
  • From the baofeng I just wired it up to the transformer, then the output side of that went into a 0.01uF greencap capacitor in series to the microphone input of the sound card.
In both instances those capacitors are there as DC blockers.

(I'd draw up a circuit diagram but for some reason there's no easy tool here in blogger to do that in-line! Sigh.)

Ok, so that bit is easy.

Then on to the software side.

The normal way people do this stuff is "direwolf" on Linux. So, "pkg install direwolf" installed it. That was easy.

Configuring it up was a bit less easy. I found this guide to be helpful:

https://andrewmemory.wordpress.com/tag/direwolf/

FreeBSD has the example direwolf config in /usr/local/share/doc/direwolf/examples/direwolf.conf . Now, direwolf will run as a normal user (there's no rc.d script for it yet!) and by default runs out of the current directory. So:

$ cd ~
$ cp /usr/local/share/doc/direwolf/examples/direwolf.conf .
$ (edit it)
$ direwolf

Editing it isn't that hard - you need to change your callsign and the audio device.

OK, here is the main undocumented bit for FreeBSD - the sound device can just be /dev/dsp . It isn't an ALSA name! Don't waste time trying to use ALSA names. Instead, just find the device you want and reference it. For me the USB sound card shows up as /dev/dsp3 (which is very non specific as USB sound devices come and go, but that's a later problem!) but it's enough to bring it up.

So yes, following the above guide, using the right sound device name resulted in a working APRS modem.

Next up - something to talk to it. This is called 'xastir'. It's .. well, when you run it, you'll find exactly how old an X application it is. It's very nostalgically old. But, it is enough to get APRS positioning up and test both the TCP/IP side of APRS and the actual radio radio side.

Here's the guide I followed:

https://andrewmemory.wordpress.com/2015/03/22/setting-up-direwolfxastir-on-a-raspberry-pi/

So, that was it! So far so good. It actually works well enough to decode and watch APRS traffic around me. I managed to get out position information to the APRS network over both TCP/IP and relayed via VHF radio.

FreeBSD and APRS, or "hm what happens when none of this is well documented.."

Here's another point along my quest for amateur radio on FreeBSD - bring up basic APRS support. Yes, someone else has done the work, but in the normal open source way it was .. inconsistently documented.

First is figuring out the hardware platform. I chose the following:

  • A Baofeng UV5R2, since they're cheap, plentiful, and do both VHF and UHF;
  • A cable to do sound level conversion and isolation (and yes, I really should post a circuit diagram and picture..);
  • A USB sound device, primarily so I can whack it into FreeBSD/Linux devices to get a separate sound card for doing radio work;
  • FreeBSD laptop (it'll become a raspberry pi + GPS + sensor + LCD thingy later, but this'll do to start with.)
The Baofeng is easy - set it to the right frequency (VHF APRS sits on 144.390MHz), turn on VOX so I don't have to make up a PTT cable, done/done.

The PTT bit isn't that hard - one of the microphone jack pins is actually PTT (if you ground it, it engages PTT) so when you make the cable just ensure you expose a ground pin and PTT pin so you can upgrade it later.

The cable itself isn't that hard either - I had a baofeng handmic lying around (they're like $5) so I pulled it apart for the cable. I'll try to remember to take pictures of that.

Here's a picture I found on the internet that shows the pinout:


Now, I went a bit further. I bought a bunch of 600 ohm isolation transformers for audio work, so I wired it up as follows:

  • From the audio output of the USB sound card, I wired up a little attenuator - input is 2k to ground, then 10k to the input side of the transformer; then the output side of the transformer has a 0.01uF greencap capacitor to the microphone input of the baofeng;
  • From the baofeng I just wired it up to the transformer, then the output side of that went into a 0.01uF greencap capacitor in series to the microphone input of the sound card.
In both instances those capacitors are there as DC blockers.

(I'd draw up a circuit diagram but for some reason there's no easy tool here in blogger to do that in-line! Sigh.)

Ok, so that bit is easy.

Then on to the software side.

The normal way people do this stuff is "direwolf" on Linux. So, "pkg install direwolf" installed it. That was easy.

Configuring it up was a bit less easy. I found this guide to be helpful:

https://andrewmemory.wordpress.com/tag/direwolf/

FreeBSD has the example direwolf config in /usr/local/share/doc/direwolf/examples/direwolf.conf . Now, direwolf will run as a normal user (there's no rc.d script for it yet!) and by default runs out of the current directory. So:

$ cd ~
$ cp /usr/local/share/doc/direwolf/examples/direwolf.conf .
$ (edit it)
$ direwolf

Editing it isn't that hard - you need to change your callsign and the audio device.

OK, here is the main undocumented bit for FreeBSD - the sound device can just be /dev/dsp . It isn't an ALSA name! Don't waste time trying to use ALSA names. Instead, just find the device you want and reference it. For me the USB sound card shows up as /dev/dsp3 (which is very non specific as USB sound devices come and go, but that's a later problem!) but it's enough to bring it up.

So yes, following the above guide, using the right sound device name resulted in a working APRS modem.

Next up - something to talk to it. This is called 'xastir'. It's .. well, when you run it, you'll find exactly how old an X application it is. It's very nostalgically old. But, it is enough to get APRS positioning up and test both the TCP/IP side of APRS and the actual radio radio side.

Here's the guide I followed:

https://andrewmemory.wordpress.com/2015/03/22/setting-up-direwolfxastir-on-a-raspberry-pi/

So, that was it! So far so good. It actually works well enough to decode and watch APRS traffic around me. I managed to get out position information to the APRS network over both TCP/IP and relayed via VHF radio.

FreeBSD: Using Open-Xchange on FreeBSD

If you go looking for a usable webmail application, then you might end up with Open-Xchange (OX for short). Some larger ISP's are using OX as their webmail application for customers. It has a multitude of options available, using multiple email accounts, caldav/carddav included (not externally (yet?)) etc. There are commercial options available for these ISP's, but also for smaller resellers etc.

But, there is also the community edition available. Which is the installation you can run for free on your machine(s). It does not have some of the fancy modules that large setups need and require, and some updates might follow a bit later which are more directly delivered to paying customers, but it is very complete and usable.

I decided to setup this for my private clients who like to use a webmail client to access their email. At first I ran this on a VM using Bhyve on FreeBSD. The VM ran on CentOS6 and had the necessary bits installed for the OX setup (see: https://oxpedia.org/wiki/index.php?title=AppSuite:Open-Xchange_Installation_Guide_for_CentOS_6). I modified the files I needed to change to get this going, and there, it just worked. But, running on a VM, with ofcourse limited CPU and Memory power assigned (There is always a cap) and it being emulated, I was not very happy with it. I needed to maintain an additional installation and update it, while I have this perfectly fine FreeBSD server instead. (Note that I am not against using bhyve at all, it works very well, but I wanted to reduce my maintenance base a bit :-)).

So a few days ago I considered just moving the stuff over to the FreeBSD host instead. And actually it was rather trivial to do with the working setup on CentOS.

At this moment I do not see an easy way to get the source/components directly from within FreeBSD. I have asked OX for help on this, so that we can perhaps get this sorted out and perhaps even make a Port/pkg out of this for use with FreeBSD.

The required host changes and software installation

The first thing that I did was to create a zfs dataset for /opt. The software is normally installed there, and in this case I wanted to have a contained location which I can snapshot, delete, etc, without affecting much of the normal system. I copied over the /opt/open-xchange directory from my CentOS installation. I looked at the installation on CentOS and noticed that it used a specific user 'open-xchange', which I created on my FreeBSD host. I changed the files to be owned by this user. Getting a process listing on the CentOS machine also revealed that it needed Java/JDK. So I installed the openjdk8 pkg (''pkg install openjdk8''). The setup did not yet start, there were errors about /bin/bash missing. Obviously that required installing bash (''pkg install bash'') and you can go with two ways, you can alter every shebang (#!) to match /usr/local/bin/bash (or better yet #!/usr/bin/env bash), or you can symlink /usr/local/bin/bash to /bin/bash, which is what I did (I asked OX to make it more portable by using the env variant instead).

The /var/log/open-xchange directory does not normally exist, so I created that and made sure that ''open-xchange'' could write to that. (mkdir /var/log/open-xchange && chown open-xchange /var/log/open-xchange).

I was able to startup the /opt/open-xchange/sbin/open-xchange process with that. I could not yet easily reach it, on the CentOS installation there are two files in the Apache configuration that needed some attention on my FreeBSD host. The Apache include files: ox.conf and proxy_http.conf will give away hints about what to change. In my case I needed to do the redirect on the Vhost that runs OX (RedirectMatch ^/$ /appsuite/) and make sure the /var/www/html/appsuite directory is copied over from the CentOS installation as well. You can stick it in any location, as long as you can reach it with your webuser and Alias it to the proper directory and setup directory access).

Apache configuration (Reverse proxy mode)

The proxy_http.conf file is more interesting, it includes the reverse proxy settings to be able to connect to the java instance of OX and service your clients. I needed to add a few modules in Apache so that it could work, I already had several proxy modules enabled for different reasons, so the list below can probably be trimmed a bit to the exact modules needed, but since this works for me, I might as well just show you;

LoadModule slotmem_shm_module libexec/apache24/mod_slotmem_shm.so
LoadModule deflate_module libexec/apache24/mod_deflate.so
LoadModule expires_module libexec/apache24/mod_expires.so
LoadModule proxy_module libexec/apache24/mod_proxy.so
LoadModule proxy_connect_module libexec/apache24/mod_proxy_connect.so
LoadModule proxy_http_module libexec/apache24/mod_proxy_http.so
LoadModule proxy_scgi_module libexec/apache24/mod_proxy_scgi.so
LoadModule proxy_wstunnel_module libexec/apache24/mod_proxy_wstunnel.so
LoadModule proxy_ajp_module libexec/apache24/mod_proxy_ajp.so
LoadModule proxy_balancer_module libexec/apache24/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module libexec/apache24/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module libexec/apache24/mod_lbmethod_bytraffic.so
LoadModule lbmethod_bybusyness_module libexec/apache24/mod_lbmethod_bybusyness.so

After that it was running fine for me. My users can login to the application and the local directory's are being used instead of the VM which ran it first. If you notice previous documentation on this subject, you will notice that there are more third party packages needed at that time. It could easily be that there are more modules needed than that I wrote about. My setup was not clean, the host already runs several websites (one of them being this one) and ofcourse support packages were already installed.

Updating is currently NOT possible. The CentOS installation requires running ''yum update'' periodically, but that is obviously not possible on FreeBSD. The packages used within CentOS are not directly usable for FreeBSD. I have asked OX to provide the various Community base and optional modules as .tar.gz files (raw) so that we can fetch them and install them on the proper location(s). As long as the .js/.jar files etc are all there and the scripts are modified to start, it will just work. I have not (yet) created a startup script for this yet. For the moment I will just start the VM and see whether there are updates and copy them over instead. Since I did not need to do additional changing on the main host, it is a very easy and straight forward process in this case.

Support

There is no support for OX on FreeBSD. Ofcourse I would like to see at least some support to promote my favorite OS more, but that is a financial situation. It might not cost a lot to deliver the .tar.gz files so that we can package them and spread the usage of OX on more installations (and thus perhaps add revenue for OX as commercial installation), but it will cost FTE's to support more then that. If you see a commercial opportunity, please let them know so that this might be more and more realistic.

The documentation written above is just how I have setup the installation and I wanted to share it with you. I do not offer support on it, but ofcourse I am willing to answer questions you might have about the setup etc. I did not include the vhost configuration in it's entirely, if that is a popular request, I will add it to this post.

Open Questions to OX:

So as mentioned I have questioned OX for some choices:

  • Please use a more portable path for the Bash shell (#!/usr/bin/env bash)
  • Please allow the use of a different localbase (/usr/local/open-xchange for example)
  • Please allow FreeBSD packagers to fetch a "clean" .tar.gz, so that we can package this for OX and distribute it for our end-users.
  • Unrelated to the post above: Please allow the usage of external caldav/carddav providers

Edit:

I have found another thing that I needed to change. I needed to use gsed (Gnu-sed) instead of FreeBSD-sed so that the listuser scripts work. Linux does that a bit differently but if you replace sed with gsed those scripts will work fine.

I have not yet got some feedback from OX.

Color themes support^Whack for vt(4)

I was updating my laptop to the latest HEAD today and noticed that my bash prompt looks ugly in default console color scheme. So what with one thing and another I ended up writing color themes support for vt(4). Just because it was fun thing to do. The idea is that you can redefine any ANSI color in console using variable in /boot/loader.conf, i.e.:

kern.vt.color.0.rgb="0,0,0" # color 0 is black
# or
kern.vt.color.15.rgb="#ffffff" # color 15 is white

Here is how my Tomorrow Night theme looks like:

kern.vt.color.0.rgb="#1d1f21"
kern.vt.color.1.rgb="#d77c79"
kern.vt.color.2.rgb="#c2c77b"
kern.vt.color.3.rgb="#f4cf87"
kern.vt.color.4.rgb="#93b2ca"
kern.vt.color.5.rgb="#c0a7c7"
kern.vt.color.6.rgb="#9ac9c4"
kern.vt.color.7.rgb="#d0d2d1"
kern.vt.color.8.rgb="#d0d2d1"
kern.vt.color.9.rgb="#d77c79"
kern.vt.color.10.rgb="#c2c77b"
kern.vt.color.11.rgb="#f4cf87"
kern.vt.color.12.rgb="#93b2ca"
kern.vt.color.13.rgb="#c0a7c7"
kern.vt.color.14.rgb="#9ac9c4"
kern.vt.color.15.rgb="#ffffff"

It works only with framebuffer-based console like efifb or i915kms.
Patch: console-color-theme.diff

gqrx, direct sampling configuration, shortwave/AM reception, etc

This is one of those articles that I write specifically to, you know, not forget it the next time.

I picked up a v3 RTL-SDR dongle from http://www.rtl-sdr.com/rtl-sdr-blog-v-3-dongles-user-guide/ a few weeks ago. It's a solidly looking aluminium can dongle with a much more useful RF connector. But, it still only tunes down by default to the same RTL-SDR limits as .. well, the rest.

So I go digging. I bought this thing because on Windows you're supposed to be able to tune down to a couple hundred KHz. I found out some useful stuff after a bit of google abuse:

  • it's called "direct sampling mode";
  • it's supported by the rtl_sdr driver/library that you normally use;
  • you have to configure things up in a special way to get GQRX, etc to actually do the right thing;
  • There's a specific thing called "Q-Branch" that you have to care about.
Ok, so the actual details.

Firstly, the NIC needs to go into direct sampling mode. So instead of being mixed with a VCO, you're bypassing all of that and just acting as a really fast ADC. Which, yes, will alias signals everywhere.

Secondly, you have to tell it to only give you "Q" samples, rather than both I and Q. If it's direct sampling, mixing doesn't come into it, so the hardware is just giving you ADC samples.

Then, you just tell GQRX, etc to ignore enforcing tuning limits (tick "No limits") and you're golden.

The specific string to add to the device string when you set up which rtl_tcp instance to talk to is:

direct_samp=2

(0 = no direct sampling; 1 = direct sampling on I, 2 = direct sampling on Q.)

Ok, so does it work? I'm still testing it. I will need to acquire some RF filters to try and filter out bands of HF frequencies to avoid aliasing as there is limited to on actual passband filtering going on here. But it did tune to AM radio and I picked up some data transmissions in 3MHz.

Baby steps!


gqrx, direct sampling configuration, shortwave/AM reception, etc

This is one of those articles that I write specifically to, you know, not forget it the next time.

I picked up a v3 RTL-SDR dongle from http://www.rtl-sdr.com/rtl-sdr-blog-v-3-dongles-user-guide/ a few weeks ago. It's a solidly looking aluminium can dongle with a much more useful RF connector. But, it still only tunes down by default to the same RTL-SDR limits as .. well, the rest.

So I go digging. I bought this thing because on Windows you're supposed to be able to tune down to a couple hundred KHz. I found out some useful stuff after a bit of google abuse:

  • it's called "direct sampling mode";
  • it's supported by the rtl_sdr driver/library that you normally use;
  • you have to configure things up in a special way to get GQRX, etc to actually do the right thing;
  • There's a specific thing called "Q-Branch" that you have to care about.
Ok, so the actual details.

Firstly, the NIC needs to go into direct sampling mode. So instead of being mixed with a VCO, you're bypassing all of that and just acting as a really fast ADC. Which, yes, will alias signals everywhere.

Secondly, you have to tell it to only give you "Q" samples, rather than both I and Q. If it's direct sampling, mixing doesn't come into it, so the hardware is just giving you ADC samples.

Then, you just tell GQRX, etc to ignore enforcing tuning limits (tick "No limits") and you're golden.

The specific string to add to the device string when you set up which rtl_tcp instance to talk to is:

direct_samp=2

(0 = no direct sampling; 1 = direct sampling on I, 2 = direct sampling on Q.)

Ok, so does it work? I'm still testing it. I will need to acquire some RF filters to try and filter out bands of HF frequencies to avoid aliasing as there is limited to on actual passband filtering going on here. But it did tune to AM radio and I picked up some data transmissions in 3MHz.

Baby steps!


gqrx on freebsd

I'm still tinkering with my own SDR code in C, but I decided I'd give gqrx a go. It sports a simple UI to visualise things as well as a simple control network protocol.

FreeBSD has packages for it, so it really was as simple as:

# pkg install gqrx

I already have the rtl-sdr bits installed, so that was pretty easy:

# rtl_tcp

.. to bring up the network service for it.

But running gqrx .. didn't work. It just stood there and did nothing when I ran it - no UI, no useful log output, nothing.

So out came truss. Step one in the "what is going on" showed no useful logging. Step 2 is "ok, what syscalls is it making." It turns out it was trying to connect to the IPv6 localhost address - but rtl_tcp defaults on FreeBSD to IPv4 localhost. gqrx doesn't handle connect() errors gracefully, so it was just stuck in an infinite connect() loop.

So I cleared the config, started gqrx again, and ensured this time it was set to '127.0.0.1' instead of 'localhost'. Voila! It worked!

Now - I need an SDR unit that can go down closer to DC so I can pick up shortwave. This RTL dongle I have doesn't tune below 27MHz.

gqrx on freebsd

I'm still tinkering with my own SDR code in C, but I decided I'd give gqrx a go. It sports a simple UI to visualise things as well as a simple control network protocol.

FreeBSD has packages for it, so it really was as simple as:

# pkg install gqrx

I already have the rtl-sdr bits installed, so that was pretty easy:

# rtl_tcp

.. to bring up the network service for it.

But running gqrx .. didn't work. It just stood there and did nothing when I ran it - no UI, no useful log output, nothing.

So out came truss. Step one in the "what is going on" showed no useful logging. Step 2 is "ok, what syscalls is it making." It turns out it was trying to connect to the IPv6 localhost address - but rtl_tcp defaults on FreeBSD to IPv4 localhost. gqrx doesn't handle connect() errors gracefully, so it was just stuck in an infinite connect() loop.

So I cleared the config, started gqrx again, and ensured this time it was set to '127.0.0.1' instead of 'localhost'. Voila! It worked!

Now - I need an SDR unit that can go down closer to DC so I can pick up shortwave. This RTL dongle I have doesn't tune below 27MHz.

Bringing up 802.11ac on FreeBSD

I've been chipping away at bringing up 802.11ac on FreeBSD. I've been meaning to write this post for a while, but you know, life got in the way.

net80211 has reasonably good 802.11n support, but no 802.11ac support. I decided a while ago to start adding basic 802.11ac support. It was a good exercise in figuring out what the minimum set of required features are and another excuse to go find some of the broken corner cases in net80211 that needed addressing. I'll talk about a few here.

802.11ac introduces a few new concepts that the stack needs to understand. I decided to use the QCA 802.11ac parts because (a) I know the firmware and general chip stuff from the first generation 11ac parts well, and (b) I know that it does a bunch of stuff (like rate control, packet scheduling, etc) so I don't have to do it. If I chose, say, the Intel 11ac parts then I'd have to implement a lot more of the fiddly stuff to get good behaviour.

So, the first step was to survey what I needed to at bring up a very small subset of functionality. The typical first target is "monitor" mode, since that requires channel configuration to work, but doesn't require the rest of the stack to know about anything (ie, no 11ac aware IE negotiation.)

Step one - adding VHT channels. I decided in the shorter term to cheat and just add VHT channels to the already very large ieee80211_channel map. The linux way of there being a channel context rather than hundreds of static channels to choose from is better in the long run, but I wanted to get things up and running. So, that's what I did first - I added VHT flags for 20, 40, 80, 80+80 and 160MHz operating modes and I did the bare work required to populate the channel lists with VHT channels as well.

This required adding support to lib80211 and ifconfig to support loading VHT channels from the regulatory domain (regdomain.xml.) And yes, this let me get VHT channels statically configured.

Then I needed to add VHT capabilities to the driver structure (ieee80211com) and the virtual interface structure (ieee80211vap.) This is so that drivers can be told that yes, we can do VHT things.

Then I needed to glue it into an 11ac driver. My ath10k port was far enough along to attempt this, so I added enough glue to say "I support VHT" to the ic_caps field and propagated it to the driver for monitor mode configuration. And yes, after a bit of dancing, I managed to get a VHT channel to show up in ath10k in monitor mode and could capture 80MHz wide packets. Success!

So at this point I took a big of a segue and decided to ensure that the receive packet flags made sense for VHT operation. I did a bunch of work to tidy this up so I could mark frames as 20/40/80/etc and had all of that mostly done in a few days.

Next up was station operation. So this required a few things to get right:
  • Channel promotion (ie, going from 11a -> 11n -> 11ac)
  • Knowing about VHT IEs, both for transmit and receive
  • Negotiating VHT capabilities to announce to the AP during association request/response and then telling the driver about what's going on.
By far the most fiddly was getting channel promotion to work. net80211 supports the concept of dumb NICs (like atheros 11abgn parts) very well, where you can have multiple virtual interfaces but the "driver" view of the right configuration is what's programmed into the hardware. For firmware NICs which do this themselves (like basically everything sold today) this isn't exactly all that helpful. So, for now, it's limited to a single VAP, and the VAP configuration is partially derived from the global state and partially derived from the negotiated state. It's annoying, but it is adding to the list of things I will have to fix later.

Sorry, I went off on a bit of a tangent. So one of the things net80211 does is it looks at the list of desired channels for each VAP and determines what the "NIC" channel should be in order to support it. So yes, ic_curchan is being used and it's what gets updated. the vap->iv_des_chan is the "I'd like to be this channel" but the active channel is actually vap->iv_bss->ni_chan (yes, the BSS node current channel.) That's what gets promoted. I need to uhm, "fix" the per-VAP state to have a "current channel" that gets updated with the BSS channel - so that's high on the list of things to fix.

Then the next part for channel promotion is how it's done. IT's not done via a call to "promote channel" - it instead was done by a call to net80211's 802.11n routines which just parse HT (11n) IEs. Yes, parsing an 11n IE also upgraded the channel. So I spent a bunch of time refactoring that, and now the bits that parse IEs are separate from channel promotion. Fun times.

So, once channel promotion worked, I associated fine as an open mode 11ac station. All of the negotiated pieces were wrong, so I spent a few days looking at packet captures to ensure I negotiated VHT capabilities correctly, but it seems to work fine.

Next was adding in support for things that stopped me doing encrypted data. The TL;DR here is - net80211's crypto key handling model needed overhauling.

Firstly the key management code assumes everything is synchronous and can't sleep. For firmware based devices this really isn't true. So, a lot of drivers defer key management into taskqueues to run things. This sounds fine, but then you realise that once the key addition succeeds, net80211 then will transmit frames expecting them to be encrypted. Userland too has this problem - it will do an ioctl() to set a key, then when it returns OK, it will queue further key negotiation frames. So, I had to do some uhm, clever things to try and work around this, and I'll cover how I need to fix it properly later.

Next up - the QCA chips/firmware do 802.11 crypto offload. They actually pretend that there's no key - you don't include the IV, you don't include padding, or anything. You send commands to set the crypto keys and then you send unencrypted 802.11 frames (or 802.3 frames if you want to do ethernet only.) This means that I had to teach net80211 a few things:

  • frames decrypted by the hardware needed to have a "I'm decrypted" bit set, because the 802.11 header field saying "I'm decrypted!" is cleared
  • frames encrypted don't have the "i'm encrypted" bit set
  • frames encrypted/decrypted have no padding, so I needed to teach the input path and crypto paths to not validate those if the hardware said "we offload it all."
So yeah, that's now done.

(Now, there's a whole separate discussion about hostap that I'll defer for later. Suffice to say - at this point I got hostap mostly working.)

Now comes the hard bit of fixing the shortcomings before I can commit the driver. There are .. lots.

The first one is the global state. The ath10k firmware allows what they call 'vdevs' (virtual devices) - for example, multiple SSID/BSSID support is implemented with multiple vdevs. STA+WDS is implemented with vdevs. STA+P2P is implemented with vdevs. So, technically speaking I should go and find all of the global state that should really be per-vdev and make it per-vdev. This is tricky though, because a lot of the state isn't kept per-VAP even though it should be.

So, I need to fix the following to be per-vdev rather than global:
  • Slot time configuration
  • ERP configuration (ie, 11g BSS overlap with 11b)
  • Frame protection configuration
  • QoS/WMM configuration
  • 20/40 channel width configuration - which needs extending to 11ac channel widths too
Then there are some things which are just plainly not implemented. A-MPDU offload for example is fine, but A-MSDU offload pretends that there are multiple 802.11 frames with the same sequence number / crypto information. This is problematic for the sequence number de-duplication code and also A-MPDU offload - in both cases, all but one A-MSDU frames are just tossed. This drastically kills performance. I'm almost done implementing A-MSDU in A-MPDU offload support, and when that's done, I'll implement A-MSDU offload de-duplication awareness and performance should drastically rise.

Another thing as mentioned is channel width, as well as other 802.11n things that need extending for 802.11ac. A bit one that I'd like to get right is SMPS (spatial multiplexing power save.) If the chip supports it in firmware then we just need to allow it to be configured and controlled, but that isn't even an option right now. I'd like to implement SMPS for Atheros 11n parts in ath(4), but that's a lot more work.

Anyway, so far so good. I need to do some of the above and land it in FreeBSD-HEAD so I can finish off the ath10k port and commit what I have to FreeBSD. There's a lot of stuff coming - including all of the wave-2 stuff (like multiuser MIMO / MU-MIMO) which I just plainly haven't talked about yet.

Viva la FreeBSD wireless! 

Bringing up 802.11ac on FreeBSD

I've been chipping away at bringing up 802.11ac on FreeBSD. I've been meaning to write this post for a while, but you know, life got in the way.

net80211 has reasonably good 802.11n support, but no 802.11ac support. I decided a while ago to start adding basic 802.11ac support. It was a good exercise in figuring out what the minimum set of required features are and another excuse to go find some of the broken corner cases in net80211 that needed addressing. I'll talk about a few here.

802.11ac introduces a few new concepts that the stack needs to understand. I decided to use the QCA 802.11ac parts because (a) I know the firmware and general chip stuff from the first generation 11ac parts well, and (b) I know that it does a bunch of stuff (like rate control, packet scheduling, etc) so I don't have to do it. If I chose, say, the Intel 11ac parts then I'd have to implement a lot more of the fiddly stuff to get good behaviour.

So, the first step was to survey what I needed to at bring up a very small subset of functionality. The typical first target is "monitor" mode, since that requires channel configuration to work, but doesn't require the rest of the stack to know about anything (ie, no 11ac aware IE negotiation.)

Step one - adding VHT channels. I decided in the shorter term to cheat and just add VHT channels to the already very large ieee80211_channel map. The linux way of there being a channel context rather than hundreds of static channels to choose from is better in the long run, but I wanted to get things up and running. So, that's what I did first - I added VHT flags for 20, 40, 80, 80+80 and 160MHz operating modes and I did the bare work required to populate the channel lists with VHT channels as well.

This required adding support to lib80211 and ifconfig to support loading VHT channels from the regulatory domain (regdomain.xml.) And yes, this let me get VHT channels statically configured.

Then I needed to add VHT capabilities to the driver structure (ieee80211com) and the virtual interface structure (ieee80211vap.) This is so that drivers can be told that yes, we can do VHT things.

Then I needed to glue it into an 11ac driver. My ath10k port was far enough along to attempt this, so I added enough glue to say "I support VHT" to the ic_caps field and propagated it to the driver for monitor mode configuration. And yes, after a bit of dancing, I managed to get a VHT channel to show up in ath10k in monitor mode and could capture 80MHz wide packets. Success!

So at this point I took a big of a segue and decided to ensure that the receive packet flags made sense for VHT operation. I did a bunch of work to tidy this up so I could mark frames as 20/40/80/etc and had all of that mostly done in a few days.

Next up was station operation. So this required a few things to get right:
  • Channel promotion (ie, going from 11a -> 11n -> 11ac)
  • Knowing about VHT IEs, both for transmit and receive
  • Negotiating VHT capabilities to announce to the AP during association request/response and then telling the driver about what's going on.
By far the most fiddly was getting channel promotion to work. net80211 supports the concept of dumb NICs (like atheros 11abgn parts) very well, where you can have multiple virtual interfaces but the "driver" view of the right configuration is what's programmed into the hardware. For firmware NICs which do this themselves (like basically everything sold today) this isn't exactly all that helpful. So, for now, it's limited to a single VAP, and the VAP configuration is partially derived from the global state and partially derived from the negotiated state. It's annoying, but it is adding to the list of things I will have to fix later.

Sorry, I went off on a bit of a tangent. So one of the things net80211 does is it looks at the list of desired channels for each VAP and determines what the "NIC" channel should be in order to support it. So yes, ic_curchan is being used and it's what gets updated. the vap->iv_des_chan is the "I'd like to be this channel" but the active channel is actually vap->iv_bss->ni_chan (yes, the BSS node current channel.) That's what gets promoted. I need to uhm, "fix" the per-VAP state to have a "current channel" that gets updated with the BSS channel - so that's high on the list of things to fix.

Then the next part for channel promotion is how it's done. IT's not done via a call to "promote channel" - it instead was done by a call to net80211's 802.11n routines which just parse HT (11n) IEs. Yes, parsing an 11n IE also upgraded the channel. So I spent a bunch of time refactoring that, and now the bits that parse IEs are separate from channel promotion. Fun times.

So, once channel promotion worked, I associated fine as an open mode 11ac station. All of the negotiated pieces were wrong, so I spent a few days looking at packet captures to ensure I negotiated VHT capabilities correctly, but it seems to work fine.

Next was adding in support for things that stopped me doing encrypted data. The TL;DR here is - net80211's crypto key handling model needed overhauling.

Firstly the key management code assumes everything is synchronous and can't sleep. For firmware based devices this really isn't true. So, a lot of drivers defer key management into taskqueues to run things. This sounds fine, but then you realise that once the key addition succeeds, net80211 then will transmit frames expecting them to be encrypted. Userland too has this problem - it will do an ioctl() to set a key, then when it returns OK, it will queue further key negotiation frames. So, I had to do some uhm, clever things to try and work around this, and I'll cover how I need to fix it properly later.

Next up - the QCA chips/firmware do 802.11 crypto offload. They actually pretend that there's no key - you don't include the IV, you don't include padding, or anything. You send commands to set the crypto keys and then you send unencrypted 802.11 frames (or 802.3 frames if you want to do ethernet only.) This means that I had to teach net80211 a few things:

  • frames decrypted by the hardware needed to have a "I'm decrypted" bit set, because the 802.11 header field saying "I'm decrypted!" is cleared
  • frames encrypted don't have the "i'm encrypted" bit set
  • frames encrypted/decrypted have no padding, so I needed to teach the input path and crypto paths to not validate those if the hardware said "we offload it all."
So yeah, that's now done.

(Now, there's a whole separate discussion about hostap that I'll defer for later. Suffice to say - at this point I got hostap mostly working.)

Now comes the hard bit of fixing the shortcomings before I can commit the driver. There are .. lots.

The first one is the global state. The ath10k firmware allows what they call 'vdevs' (virtual devices) - for example, multiple SSID/BSSID support is implemented with multiple vdevs. STA+WDS is implemented with vdevs. STA+P2P is implemented with vdevs. So, technically speaking I should go and find all of the global state that should really be per-vdev and make it per-vdev. This is tricky though, because a lot of the state isn't kept per-VAP even though it should be.

So, I need to fix the following to be per-vdev rather than global:
  • Slot time configuration
  • ERP configuration (ie, 11g BSS overlap with 11b)
  • Frame protection configuration
  • QoS/WMM configuration
  • 20/40 channel width configuration - which needs extending to 11ac channel widths too
Then there are some things which are just plainly not implemented. A-MPDU offload for example is fine, but A-MSDU offload pretends that there are multiple 802.11 frames with the same sequence number / crypto information. This is problematic for the sequence number de-duplication code and also A-MPDU offload - in both cases, all but one A-MSDU frames are just tossed. This drastically kills performance. I'm almost done implementing A-MSDU in A-MPDU offload support, and when that's done, I'll implement A-MSDU offload de-duplication awareness and performance should drastically rise.

Another thing as mentioned is channel width, as well as other 802.11n things that need extending for 802.11ac. A bit one that I'd like to get right is SMPS (spatial multiplexing power save.) If the chip supports it in firmware then we just need to allow it to be configured and controlled, but that isn't even an option right now. I'd like to implement SMPS for Atheros 11n parts in ath(4), but that's a lot more work.

Anyway, so far so good. I need to do some of the above and land it in FreeBSD-HEAD so I can finish off the ath10k port and commit what I have to FreeBSD. There's a lot of stuff coming - including all of the wave-2 stuff (like multiuser MIMO / MU-MIMO) which I just plainly haven't talked about yet.

Viva la FreeBSD wireless! 

(finally) investigating how to get dynamic WDS (DWDS) working in FreeBSD!

I sat down recently to figure out how to get dynamic WDS working on FreeBSD-HEAD. It's been in FreeBSD since forever, and it in theory should actually have just worked, but it's extremely not documented in any useful way. It's basically the same technology in earlier Apple Airports (before it grew into what the wireless tech world calls "Proxy-STA") and is what the "extender" technology on Qualcomm Atheros chipsets implement.

A common question I get from people is "why can't I bridge multiple virtual machines on my laptop and have them show up over wifi? It works on ethernet!" And my response is "when I make dynamic WDS work, you can just make this work on FreeBSD devices - but for now, use NAT." That always makes people sad.

So what is it?

With normal station / access point setups, wireless frames have up to three addresses. In the header it's "address 1", "address 2", and "address 3".

Depending upon the packet type, these can be a variety of addresses:

  • SA - source address - source of the packet (eg the STA address)
  • TA - transmitter address - STA/AP that transmitted the frame
  • RA - receiver address - immediate destination of the packet 
  • DA - finally recipient of the data
  • BSSID - BSS ID (ie, AP mac address)
There are a lot of addresses. There are, in fact, more than the number of address fields in a normal 802.11 frame.

Now, if you want to understand when each of these are used in which frames, you can totally find blog posts from people which describe things (eg http://80211notes.blogspot.com/2013/09/understanding-address-fields-in-80211.html) will fill you in. But the TL;DR for normal AP/STA traffic is:

  • From an AP, the frame has BSSID, SA of the MAC (eg ethernet behind the AP bridge) sending data, and DA is the STA MAC address
  • From a STA, the frame has BSSID, TA is the STA that transmits, and DA is the final destination of the frame (eg ethernet behind the AP bridge.)
The big note here is that there's not enough MAC addresses to say "please send this frame to a station MAC address, but then have them forward it to another MAC address attached behind it in a bridge." That would require 4 mac addresses in the 802.11 header, which we don't get.

.. except we do. There's a separate address format where from-DS and to-DS bits in the header set to 1, which means "this frame is coming from distribution system to a distribution system", and it has four mac addresses. The RA is then the AP you're sending to, and then a fourth field indicates the eventual destination address that may be an ethernet device connected behind said STA.

If you don't configure up WDS, then when you send frames from a station from a MAC address that isn't actually your 802.11 interface MAC address, the system would be confused. The STA wouldn't be able to transmit it easily, and the AP wouldn't know how to get back to your bridged ethernet addresses.

Ok, so how does this work with WDS? The above from/to-DS mode is actually the technical hilarity behind "Wireless Distribution System",  which is a fancy way of saying "an AP connects to another AP and can relay frames for you." It's what was used for extending wireless networks before true meshing solutions came into existence.

The original WDS was a statically configured thing. You'd configure up a particular device as a WDS extender, and both sides would need configuring:

  • The central AP would need to know the MAC address of a WDS master, so it would know that frames from/to that particular AP needed the four-address frame format, and
  • The extender AP would need to be configured to talk to the central AP to act as a WDS master - it would then associate as a station to that central AP, and would use 4-address frames to relay traffic to it.
 So for static configurations, this works great. You'd associate your extender AP as a station of the central AP, it'd use wpa_supplicant to setup encryption, then anything between that central AP and that extender AP (as a station) would be encrypted as normal station traffic (but, 4-address frame format.)

But that's not very convenient. You have to statically configure everything, including telling your central AP about all of your satellite extender APs. If you want to replace your central AP, you have to reprogram all of your extenders to use the new MAC addresses.

So, Sam Leffler came up with "dynamic WDS" - where you don't have to explicitly state the list of central/satellite APs. Instead, you configure a central AP as "dynamic WDS", and when a 4-address frame shows up from an associated station, it "promotes" it to a WDS peer for you. On the satellite AP, it will just find an AP to communicate to, and then assume it'll do WDS and start using 4-address frames. It's still a bit clunky (there's no beacon, probe request, etc IEs that say "I do dynamic WDS!" so you'd better make ALL your central APs a different SSID!) but it certainly is better than what we had.

(Yes, one of the things I'll be doing to FreeBSD now that this works is adding that concept of "I'm a DWDS primary!" node concept so satellites can just "find" a DWDS primary enabled AP to associate to. Baby steps..)

But, I tried it - and ... let's just say, the documentation didn't say very much. So I couldn't really get it to work.

Then a friend pointed out he figured it out. (Thankyou Edward!)

Firstly, there are scripts in src/tools/tools/net80211/ - setup.wdsmain and setup.wdsrelay. These scripts are .. well, the almost complete documentation on a dynamic WDS setup. The manpage doesn't go into anywhere near enough information.

So I dug into it. It turns out that dynamic WDS uses a helper daemon - 'wlanwds' -  which listens for dynamic WDS configuration changes and will do things for you. This is what runs on the central AP side. Then it started making sense!
  • For dynamic WDS, there are no WDS interface types created by default
  • net80211 will post a routing socket message if a 4-address frame shows up on a "dwds" enabled interface, which is the signal to userland to plumb up a DWDS interface for that particular peer
  • wlanwds is then responsible for creating that virtual interface with the right configuration
  • Then it runs a shell script that you provide which lets you do things like assign it to a bridge group so it can bridge traffic
  • Finally, if the station goes away, wlanwds will get another notification from net80211 saying the station has gone, and wlanwds will destroy the virtual interface for that peer.
So far, so good. I followed that script, modified it a bit to use encryption, and .. well, it half worked. Association worked fine, but no traffic was passing.

A little tcpdump'ing later showed what was going on!
  • 4-address frames from the extender side was successfully being encrypted and transmitted to the central AP
  • 4-address frames from the central AP were being send, but unencrypted
  • .. so the station dropped them as well, unencrypted when they should've been encrypted.
A little more digging showed the actual problem - the dynamic WDS example scripts are for an open/unencrypted network. If you are using an encrypted network, the central AP side needs to enable privacy on the virtual interfaces so traffic gets encrypted with the parent interface encryption keys. So adding this:

ifconfig $DEV wepmode mixed

.. to the shell script for when an interface was created made everything work.

Now, I've only done enough testing to show that indeed it is working. I haven't done anything like pass lots of traffic via iperf, or have a mix of DWDS and normal STA peers, nor actually run it for longer than 5 minutes. I'm sure there will be issues to fix. However - I do need it at home, as I can't see the home AP from the upstairs room (and now you see why I care about DWDS!) and so when I start using it daily I'll fix whatever hilarity ensues.