Guide: Run FreeBSD 13.1-RELEASE for ARM64 in QEMU on Apple Silicon Mac (MacBook Pro M1, etc) with HVF acceleration (Hypervisor.framework)

README.md

Guide: Run FreeBSD 13.1-RELEASE for ARM64 in QEMU on Apple Silicon Mac (MacBook Pro M1, etc) with HVF acceleration (Hypervisor.framework)

This guide was adapted from https://gist.github.com/niw/e4313b9c14e968764a52375da41b4278#running-ubuntu-server-for-arm64

Running FreeBSD 13.1-RELEASE for ARM64

1. Install Xcode from App Store or install Command Line Tools on your Mac running on Apple Silicon.

   xcode-select --install
   

2. Install Homebrew and QEMU.

   https://brew.sh/

   brew install qemu

   rehash

   qemu-system-aarch64 --version

   QEMU emulator version 7.0.0
   Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
   

3. Download pre-build EDK II OVMF EFI image for QEMU.

   This EFI image is built from stable202011 tag with additional resolutions in QemuRamfb.c.

   https://gist.github.com/niw/4f1f9bb572f40d406866f23b3127919b/raw/f546faea68f4149c06cca88fa67ace07a3758268/QEMU_EFI-cb438b9-edk2-stable202011-with-extra-resolutions.tar.gz

   To build it from the source code for adding more resolutions, see the following section.

4. Prepare pflash for non-volatile variable store, such as screen resolution.

   mkdir ~/qemu-vm/
   cd ~/qemu-vm/
   tar xvf ~/Downloads/QEMU_EFI-cb438b9-edk2-stable202011-with-extra-resolutions.tar.gz
   dd if=/dev/zero of=pflash0.img bs=1m count=64
   dd if=/dev/zero of=pflash1.img bs=1m count=64
   dd if=QEMU_EFI.fd of=pflash0.img conv=notrunc
   dd if=QEMU_VARS.fd of=pflash1.img conv=notrunc
   

   • This step is optional, you can use -bios QEMU_EFI.fd instead of -drive ...if=pflash lines in the next step, but in that case, any changes in EFI will not be persistent.

5. Download FreeBSD 13.1-RELEASE for ARM64 raw VM image xz-compressed file

   https://ftp2.uk.freebsd.org/pub/FreeBSD/releases/VM-IMAGES/13.1-RELEASE/aarch64/Latest/FreeBSD-13.1-RELEASE-arm64-aarch64.raw.xz

   (See https://docs.freebsd.org/en/books/handbook/mirrors/ for a list of mirrors to choose from.)

6. Decompress xz-compressed file, keeping a copy of the original compressed file

   Keeping a copy of the original file is convenient because then you can use it if you want to create additional VMs later. Just be careful not to overwrite the image of your first VM when you want to make a second VM though :P

   mv ~/Downloads/FreeBSD-13.1-RELEASE-arm64-aarch64.raw.xz .
   unxz -k FreeBSD-13.1-RELEASE-arm64-aarch64.raw.xz
   

7. Grow the disk image

   After you've decompressed the disk image, it'll be about 5 GiB in size. Depending on what you plan to do the amount of available space may be a bit on the low side. Let's grow the disk image by another 30 GiB.

   qemu-img resize -f raw FreeBSD-13.1-RELEASE-arm64-aarch64.raw +30G
   

   By performing this resize before you boot the VM for the first time, FreeBSD will automatically adjust the partition size during first boot.

8. Run your FreeBSD 13.1-RELEASE for ARM64 VM

   qemu-system-aarch64 \
     -M virt \
     -accel hvf \
     -cpu host \
     -smp 4 \
     -m 4096 \
     -drive file=pflash0.img,format=raw,if=pflash,readonly=on \
     -drive file=pflash1.img,format=raw,if=pflash \
     -device virtio-gpu-pci \
     -display default,show-cursor=on \
     -device qemu-xhci \
     -device usb-kbd \
     -device usb-tablet \
     -device intel-hda \
     -device hda-duplex \
     -drive file=FreeBSD-13.1-RELEASE-arm64-aarch64.raw,format=raw,if=virtio,cache=writethrough \
     -nographic \
     -serial mon:stdio
   

sysbench CPU benchmark comparison

Running natively in macOS Monterey Version 12.5 (21G72) on MacBook Pro (13-inch, M1, 2020)

(Installed via Homebrew.)

Events per second: 25,445,903.16 or ~25.4 Mio events/sec

sysbench cpu --threads=2 run

sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 2
Initializing random number generator from current time


Prime numbers limit: 10000

Initializing worker threads...

Threads started!

CPU speed:
    events per second: 25445903.16

General statistics:
    total time:                          10.0000s
    total number of events:              254468472

Latency (ms):
         min:                                    0.00
         avg:                                    0.00
         max:                                    0.12
         95th percentile:                        0.00
         sum:                                 8916.15

Threads fairness:
    events (avg/stddev):           127234236.0000/15768.00
    execution time (avg/stddev):   4.4581/0.00

Running in FreeBSD 13.1-RELEASE for ARM64 in QEMU on MacBook Pro (13-inch, M1, 2020)

(Installed via FreeBSD ports.)

Events per second: 15,467,914.47 or ~15.5 Mio events/sec

sysbench cpu --threads=2 run

sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 2
Initializing random number generator from current time


Prime numbers limit: 10000

Initializing worker threads...

Threads started!

CPU speed:
    events per second: 15467914.47

General statistics:
    total time:                          10.0003s
    total number of events:              154689213

Latency (ms):
         min:                                    0.00
         avg:                                    0.00
         max:                                    0.21
         95th percentile:                        0.00
         sum:                                 5684.03

Threads fairness:
    events (avg/stddev):           77344606.5000/169564.50
    execution time (avg/stddev):   2.8420/0.00

DNS is working but ping gives weird result:

Probably this is related(?): https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237441

I have one of these running in a actual nano pi r2s but the problem is that they only way I can reach it is using ssh but i dunno the image's default user and password, does anybody know it, by any chance?

Username is root and there is no password.

Good news for everyone! There is some update in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237441 regarding the network issue:

It appears this is a libslirp issue on macOS.
Rebuilding QEMU with slirp updated to include https://gitlab.freedesktop.org/slirp/libslirp/-/commit/7271345efe182199acaeae602cb78a94a7c6dc9d fixes this issue for me.
See also https://gitlab.freedesktop.org/slirp/libslirp/-/issues/35
I've filed Homebrew/homebrew-core#73517

@BrettDong

I've filed Homebrew/homebrew-core#73517

I tried to build qemu with the most recent hvf patches now and at the same time specifying --enable-slirp=system when running configure.

But I am still getting the same kind of errors inside of the FreeBSD VM when I try to ping 1.1.1.1.

Is there something else one needed to do in order to fix this issue that I missed? My homebrew installed libslirp is version 4.6.1 so it should have the patch.

Funnily enough, this tutorial uses the linux-style dd numeric syntax (lowercase M) which fails on macOS (I think it inherited dd from freebsd a looong time ago) and needs bs=1M (not bs=1m).

Running the commands as-is on current macOS results in:

dd: invalid number: ‘1m’
dd: invalid number: ‘1m’

@sneak What version of macOS are you running? The dd commands given in the guide run as they should on macOS Monterey Version 12.5

Also, what is the output for you of the following?

which dd

You should get the following output:

/bin/dd

If you are getting a different output, and you are using the same version of macOS that I mentioned, then you are not using the dd that ships with macOS. In which case, make sure you are using the dd that ships with macOS.

This guide will not work on M1 macs. Qemu 7.0 has a regression (6.2 worked just fine) and you have to use another build:

# Install QEmu for Mac M1:
#
brew tap uenob/qemu-hvf
brew install --head qemu-hvf

# and then you may want to start headless vm like this:

/opt/homebrew/opt/qemu-hvf/bin/qemu-system-aarch64 \
    -M virt,accel=hvf,highmem=off \
    -m 12288 \
    -smp cores=4 \
    -cpu cortex-a72 \
    -drive file=edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on \
    -drive file=freebsd13.qcow2,format=qcow2 \
    -nographic \
    -device virtio-net-device,netdev=en0 \
    -netdev user,id=en0,hostfwd=tcp::4445-:22 \
    -serial tcp::4444,server,telnet,nowait \
    -pidfile freebsd13.pid

@dmilith What do you mean it won't work? I am running qemu 7.0 as shown in the output of the guide, running on MacBook Pro (13-inch, M1, 2020).

I see that you are using the highmem=off argument. With qemu 7.0 you must run without that part.

Please install qemu 7.0 and follow the guide step by step and you will see that it does work.

Tested on MacBook Air M2, works fine.
I just tested OpenBSD too:
https://gist.github.com/astreknet/4860e3362ad98e1116f0a970b99e2afc

This guide will not work on M1 macs. Qemu 7.0 has a regression (6.2 worked just fine) and you have to use another build:

# Install QEmu for Mac M1:
#
brew tap uenob/qemu-hvf
brew install --head qemu-hvf

# and then you may want to start headless vm like this:

/opt/homebrew/opt/qemu-hvf/bin/qemu-system-aarch64 \
    -M virt,accel=hvf,highmem=off \
    -m 12288 \
    -smp cores=4 \
    -cpu cortex-a72 \
    -drive file=edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on \
    -drive file=freebsd13.qcow2,format=qcow2 \
    -nographic \
    -device virtio-net-device,netdev=en0 \
    -netdev user,id=en0,hostfwd=tcp::4445-:22 \
    -serial tcp::4444,server,telnet,nowait \
    -pidfile freebsd13.pid

With QEMU 7.2 it works perfectly fine but probably it's a bit outdated.

Ran exactly like explained in the guide, but only on qemu 7.2. While idle, FreeBSD creates load ~200% CPU on my host machine. Is it supposed to be like this?

Update:
I found a solution - simply add this line: kern.hz=100 in /boot/loader.conf
Source: https://gitlab.com/qemu-project/qemu/-/issues/959

Ran exactly like explained in the guide, but only on qemu 7.2. While idle, FreeBSD creates load ~200% CPU on my host machine. Is it supposed to be like this?

Update: I found a solution - simply add this line: kern.hz=100 in /boot/loader.conf Source: https://gitlab.com/qemu-project/qemu/-/issues/959

That's inside FreeBSD root filesystems isn't it?

That's inside FreeBSD root filesystems isn't it?

Yes, on the guest machine (inside qemu) that runs FreeBSD. So this is a boot file config for the guest FreeBSD.

For some reason, I had to do
root@freebsd:~ # efibootmgr -a -c -l /boot/efi/EFI/BOOT/bootaa64.efi -L FreeBSD-14
to get a boot entry for FreeBSD. Not sure why this wasn't the default after the first boot...

Has anyone gotten X running with or without acceleration?

On a M1 Mac with 14.1 absolut perfect