OpenBSD 6.2 on EdgeRouter Lite with bsd.mp symmetric multiprocessing kernel
Wil Knoll · 2017 Oct 20, 17:17 · 1738 words · 9 minutes read
TL;DR
Installed OpenBSD 6.2 on EdgeRouter Lite. I believe the installer did not install the multiprocessor kernel, ignoring the second core.
To fix, swap the kernel on the “i” FAT32 disk partition, not root partition. Be warned that this looks like it breaks KARL.
a tiny bastion
I always run little OpenBSD jump boxen. Before Algo and Wireguard my home network only ever had a single exposed port on an OpenBSD host. I borrowed the idea. OpenBSD on an unpopular architecture with only OpenSSH exposed. You don’t just have to exploit OpenSSH on OpenBSD, you then had to write macppc shellcode.
The box had to be small, quiet, spilling little heat and sipping power. I had used a G4 MacMini for years. But 2.5” PATA drive replacements were getting scarce, and that big power brick put out a fair bit of heat.
I had fallen in love with the Ubiquiti EdgeRouter lineup. Even if not for the software but then the hardware. I’ve got EdgeRouter X installs running EdgeOS and LEDE, and had read about the Edge Router Lite being able to run OpenBSD. Perfect. Small. Dual core MIPS 64bit with 512 of RAM for around $130. USB boot support. Fanless.
installing OpenBSD
I followed the OCETEON.install for 6.2 with some help from Hamza Sheikh’s post on a ERL 6.1 install.
Hamza’s post is wonderful and thorough. If you’ve never set up OpenBSD as a router before there is great stuff in there.
The key item to note from Hamza’s writeup is that the box is running the GENERIC kernel. That’s single processor. Only one CPU is spun up.
OpenBSD 6.1 (GENERIC) #0: Mon Apr 3 07:47:02 UTC 2017
visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC
real mem = 536870912 (512MB)
avail mem = 524255232 (499MB)
warning: no entropy supplied by boot loader
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
But then looking at Frederic Cambus’ brief post about OpenBSD on ERL, he’s got 6.2 beta running the GENERIC.MP core. So SMP is running on the ERL. Right?
OpenBSD 6.2-beta (GENERIC.MP) #0: Wed Aug 23 05:12:05 UTC 2017
visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC.MP
real mem = 536870912 (512MB)
avail mem = 523943936 (499MB)
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
cpu1 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu1: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
But GENERIC.MP wasn’t running on mine. And that’s not good enough. I hate spending money on shit and not getting it to do the thing I bought it for. If I can’t get this working there’s going to be an extra core sitting there going even more unused for weeks on end.
testing bsd.mp
Having the old OpenBSD jumpbox still around, I pulled down the octeon 6.2 directory, and served it up over NFS and TFTP. I rebooted and pointed the ERL at it to TFTP boot the bsd.mp kernel. IP addresses sanitized.
Octeon ubnt_e100# setenv serverip 10.10.10.3
Octeon ubnt_e100# tftpboot 0 bsd.mp
Using octeth0 device
TFTP from server 10.10.10.3; our IP address is 10.10.10.176
Filename 'bsd.mp'.
Load address: 0x9f00000
Loading: #########################################
done
Bytes transferred = 5768620 (5805ac hex), 1466 Kbytes/sec
Octeon ubnt_e100# bootoctlinux
<SNIP>
OpenBSD 6.2 (GENERIC.MP) #0: Wed Oct 4 05:08:15 UTC 2017
visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC.MP
real mem = 536870912 (512MB)
avail mem = 523943936 (499MB)
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
clock0 at mainbus0: int 5
No joy on that. GENERIC.MP was booting but a single CPU spun up. And for some reason this coremask: 0x1 stuck out. Something I had passed while googling…
RTFM
I reviewed the INSTALL.oceon document, and in the install section for both 6.1 and 6.2 there is a line that seems kinda relevant.
On dual-core systems, boot with both CPU cores enabled:
# bootoctlinux rootdev=rd0 coremask=0x3
Duh.
Also, let’s run setenv bootcmd 'usb reset; fatload usb 0 $loadaddr bsd; bootoctlinux rootdev=sd0 coremask=0x3" and savenv for the next boot. Just like the doc says.
Octeon ubnt_e100# setenv bootcmd 'usb reset; fatload usb 0 $loadaddr bsd; bootoctlinux rootdev=sd0 coremask=0x3'
Octeon ubnt_e100# savenv
Octeon ubnt_e100# tftpboot 0 bsd.mp
Using octeth0 device
TFTP from server 10.10.10.3; our IP address is 10.10.10.176
Filename 'bsd.mp'.
Load address: 0x9f00000
Loading: #########################################
done
Bytes transferred = 5768620 (5805ac hex), 1657 Kbytes/sec
Octeon ubnt_e100# bootoctlinux rootdev=rd0 coremask=0x3
<SNIP>
OpenBSD 6.2 (GENERIC.MP) #0: Wed Oct 4 05:08:15 UTC 2017
visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC.MP
real mem = 536870912 (512MB)
avail mem = 523943936 (499MB)
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
cpu1 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu1: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
clock0 at mainbus0: int 5
Bingo. So, copy over the bsd.mp kernel which was not installed, tell the bootloader to set the coremask, and everything should be fine. Right?
# mnt -t NFS 10.10.10.3:/home/wintr/openbsd/ /mnt
# mv /bsd /bsd.sp
# cp /mnt/bsd.mp /bsd
# reboot
syncing disks... done
System restart.
Time to go get a celebration drink.
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
clock0 at mainbus0: int 5
What the fuck?
# uname -a
OpenBSD array.unit2.ca 6.2 GENERIC#0 octeon
I am indeed running GENERIC. But I replaced that kernel right?
I copy over bsd.mp again. Something isn’t making sense here. Let’s be specific about which files are what.
# sha256 bsd
SHA256 (bsd) = 217dd111d55178df9f2092f2111a6549b9cbeb92bd11c57f551448929fa55f7d
# sha256 bsd.mp
SHA256 (bsd.mp) = 217dd111d55178df9f2092f2111a6549b9cbeb92bd11c57f551448929fa55f7d
# sha256 bsd.sp
SHA256 (bsd.sp) = 1e8ed50fa0453990af581ad0e7334e57a0675cb3fad4de83950a7778c58b5868
# sha256 bsd.booted
SHA256 (bsd.booted) = 1e8ed50fa0453990af581ad0e7334e57a0675cb3fad4de83950a7778c58b5868
What. The. Fuck.
bsd is the GENERIC.MP kernel. Same hash as bsd.mp. But it’s booting the GENERIC kernel, the hashes match between bsd.sp and bsd.booted. The bootcmd is pointed at bsd…
So I delete bsd, bsd.sp, bsd.mp, and obsd. I replace them all with a fresh copy of bsd.mp. Reboot and wondering if I’m hallucinating. I had been awake for a while.
# sha256 bsd.booted
SHA256 (bsd.booted) = 1e8ed50fa0453990af581ad0e7334e57a0675cb3fad4de83950a7778c58b5868
…
Where the hell is this kernel coming from?
RTFM 2
I review INSTALL.octeon again.
${bootcmd} is run by U-Boot when ${autoload} is enabled. Now create a new
${bootcmd} which will load an ELF file called 'bsd' from the first active FAT
partition on the first CF card or USB device. The FAT partition has been created
by the installer.
Right, this port does not use the OpenBSD bootloader. It has a ubiquiti compiled u-boot. It’s going to read an ELF on a FAT partition.
I don’t remember a FAT partition.
The FAT partition has been created by the installer.
I don’t have any FAT mounted…
# mount
/dev/sd0a on / type ffs (local)
/dev/sd0l on /home type ffs (local, nodev, nosuid)
/dev/sd0d on /tmp type ffs (local, nodev, nosuid)
/dev/sd0f on /usr type ffs (local, nodev)
/dev/sd0g on /usr/X11R6 type ffs (local, nodev)
/dev/sd0h on /usr/local type ffs (local, nodev, wxallowed)
/dev/sd0k on /usr/obj type ffs (local, nodev, nosuid)
/dev/sd0j on /usr/src type ffs (local, nodev, nosuid)
/dev/sd0e on /var type ffs (local, nodev, nosuid)
The FAT partition has been created by the installer.
Then it clicked. Out loud I said “fuck off” to no none.
# disklabel /dev/rsd0c
# /dev/rsd0c:
type: SCSI
disk: SCSI disk
label: Flash Drive FIT
duid: e3798b25063c707d
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 7801
total sectors: 125337600
boundstart: 65600
boundend: 125337600
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 2097152 65600 4.2BSD 2048 16384 12958 # /
b: 1572864 2162752 swap # none
c: 125337600 0 unused
d: 8388608 3735616 4.2BSD 2048 16384 12958 # /tmp
e: 10485760 12124224 4.2BSD 2048 16384 12958 # /var
f: 4194304 22609984 4.2BSD 2048 16384 12958 # /usr
g: 2097152 26804288 4.2BSD 2048 16384 12958 # /usr/X11R6
h: 18741024 28901440 4.2BSD 2048 16384 12958 # /usr/local
i: 65536 64 MSDOS
j: 4194304 47642464 4.2BSD 2048 16384 12958 # /usr/src
k: 10633696 51836768 4.2BSD 2048 16384 12958 # /usr/obj
l: 62867104 62470464 4.2BSD 2048 16384 12958 # /home
“i”?
#mount /dev/sd0i /mnt
#ls /mnt
bsd bsd.rd
Fuck Me.
# sha256 bsd
SHA256 (bsd) = 1e8ed50fa0453990af581ad0e7334e57a0675cb3fad4de83950a7778c58b5868
Huh… That’s where that kernel keeps coming from.
# cp /bsd.mp /mnt/bsd
# reboot
<SNIP>
OpenBSD 6.2 (GENERIC.MP) #0: Wed Oct 4 05:08:15 UTC 2017
visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC.MP
real mem = 536870912 (512MB)
avail mem = 523943936 (499MB)
mainbus0 at root
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
cpu1 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu1: cache L1-I 32KB 4 way D 8KB 64 way, L2 128KB 8 way
clock0 at mainbus0: int 5
So that’s how it works on oceton. As OpenBSD on this platform needs to use u-boot, which can’t read OpenBSD’s filesystem, none of the kernels on / are touched. Tucked away on this little slice of FAT is the kernel that is actually booted.
To be fair… The documentation did basically tell me everything I needed to know.
Dangling threads
bsd.rd or miniroot.fs does not detect multiple cores.
I did some more testing the following day. This should all be automated right? The installer script should detect multiple cores and then install the proper bsd.mp at /bsd.
chmod og-rwx /mnt/bsd{,.mp,.rd} 2>/dev/null
if [[ -f /mnt/bsd.mp ]] && ((NCPU > 1)); then
_kernel=$_kernel.MP
echo "Multiprocessor machine; using bsd.mp instead of bsd."
mv /mnt/bsd /mnt/bsd.sp 2>/dev/null
mv /mnt/bsd.mp /mnt/bsd
fi
Even with coremask=0x3 set when booting the install kernel, this is copying the bsd.sp kernel to the root partition. I think. And that’s not the part that’s responsible for copying the kernel to the FAT slice. Probably has something to do with whatever is different with the script to put the kernel on the FAT slice. Need to poke at that a bit more.
This fix breaks KARL
Tue Oct 17 23:14:27 MDT 2017
reorder_kernel: kernel relinking failed; see /usr/share/compile/GENERIC.MP/relink.log
Not going to worry about that right now.