The vision

I had high hopes for this technology. Firstly, I was looking for a way to scale-down my existing electricity-guzzling servers by replacing them with Pi's, probably saving me 70 or 80 watts. Secondly, I wanted a solution to present production interfaces of guest VMs to my DMZ whilst keeping the management of the host on my core network. ESXi could certainly do that.

My setup

Back in November 21, I piloted build 16966451 for a couple of weeks on some spare lab hardware I had to hand:

  • A Raspberry Pi 4 4gb
  • Official PSU
  • A 64gb USB 3 Pendrive
  • An 'endurance' 32gb SD Micro Card, as a boot device
  • Official Case
  • A high-performance but extremely noisy cooling fan, powered from the 5v GPIO pin

I built the server as per the official guidance using 8gb of the 64gb Pendrive for the ESXi system. The remainder I used for a local VMFS datastore. ESXi consumes about 1.2GB RAM leaving 2.8gb available for VMs on a 4gb Pi. I created a couple of PhotonOS VMs  and observed they operated as expected for the two week pilot. I don't have a vCenter license so all admin was performed via the Web GUI and SSH.

On the basis of this proof, I decided to go ahead and invest in dedicated production hardware consisting:

  • A Raspberry Pi 4 8gb
  • Official PSU
  • A Crucial 128gb SSD
  • A 'Ugreen' USB 3 enclosure to house the above SSD
  • A USB 3 GBE adapter (with compatible Realtek chipset)
  • An 'endurance' 32gb SD Micro Card, as a boot device
  • A generic aftermarket vented case
  • A silent cooling fan, powered from the 3v GPIO pin

The above Bill of Materials cost around GBP£130.

Installation and Configuration

To install, I followed the official guidance again differing only to restrict the ESXi system partition to 8GB by interrupting the installer with Ctrl-O and appending autoPartitionOSDataSize=8192.
After the installer finished, I logged-on to the DCUI and set a static IPv4 address, disabled IPv6, set my local DNS server and host name for the server. I checked the management LAN was the internal, not the USB, network adapter. I rebooted, disconnecting the video console and keyboard.

Via the Web GUI, I then proceeded to:

  • Set NTP servers and started the NTP service (as the Raspberry Pi lacks RTC hardware),
  • Create a local VMFS datastore on the remainder of the SSD, to house VMs
  • Create an NFS datastore by attaching to an exported share on my lab NAS, to provide a target backup location
  • Assign my 'free' license
  • Join the server to my AD
  • Integrate the administrative access to the host with my AD role-based access control structure
  • Wrote a very simple VM backup shell script, to backup VMs to the NAS datastore
  • Appended vim-cmd hostsvc/enable_ssh tp local.sh to always enable SSH at reboot - an acceptable risk in my lab
  • Configured syslog to log to my lab syslog server
  • Enabled SNMP for monitoring via the Telegraf SNMP integration on another host

At this point I was ready to host some VMs.

Experiences with Virtual Machines

In addition to PhotonOS I built Debian 10, Ubuntu Server, Fedora and Alpine VMs, with and without GUIs. Apart from PhotonOS, all required a manual build of open-vm-tools as per this guidance. When built, I took a copy of the VMDK files for use as a template for future machines.

I settled on Debian 10, that being my OS of choice elsewhere in my lab. I built:

  • A test server for proving applications before entering production
  • A media server with Jellyfin and Airsonic running in containers.
  • A web server with various instances of Wordpress, MySql and Ghost, again all in Docker containers
  • A PBX server to host an instance of FusionPBX

The test server worked just fine, as did the web server.

FusionPBX failed to work. The installer ran, reported ARM unsupported but proceeded anyway. After the installation had finished the application would not run as some of the dependencies were not available for ARM. An equivalent install on x64 worked perfectly.

I had mixed results with the media server. Installation in Docker worked fine. Jellyfin continually consumed disk space until it was all exhausted. Airsonic crashed frequently corrupting its internal database each time requiring a restore. Jellyfin and Airsonic worked fine in equivalent installs on an x64 VM.

So what went well?

  • It is VMWare running on a Raspberry Pi! Even after a year I still can't believe how cool this really is.
  • It is a great, cheap and very accessible way of playing with and learning VMWare, which is a wonderful thing.
  • It is just about reliable enough to be used in a non-production environment for testing and development.
  • It runs silently and consumes very little power versus my other lab servers.

What went badly?

  • Builds prior to 18427252 suffer frequent disconnections of USB-attached datastores. This is a major barrier to adoption and prevents usage in a service delivery environment, even for my home lab. Upon disconnection, all VMs crash and the host must be rebooted. So far, all of my guest VMs have recovered successfully. At worst, the datastore was disconnecting every 2 or 3 days; at best I managed 25 days uptime. You can see a pattern of downtime in the following Grafana chart, where absent data appears as gaps in the time series. This appears much improved since build 18427252 and I have now achieved over 85 days uptime.
  • Poorer performance versus my other lab servers. For services, this doesn't really matter, but if you are planning to use interactive desktops then you might be disappointed by performance. For comparison, my HPE Gen8 Microserver with a spinning disk and a Celeron G1610T CPU is more responsive despite hosting 8 other VMs.
  • The SNMP engine is missing in build 18427252 onwards - despite the service being listed in the web interface. This breaks my Telegraf monitoring and means I lack confidence to use it in production. It's a known bug that will fixed in the future, but when..?
  • Available guest OSs lack the engineering maturity and range of packages of their x64 equivalents. This means, for example, manually compiling Open VMTools and therefore not benefiting from updates via package management.
  • ARM builds of applications lack the range and engineering maturity of their x64 equivalents. In my experience this led to less opportunity to host the range services I wanted, and lower availability of services that I could.
  • No patches. VMWare does not issue patches for ESXi on Arm so any updated builds require a re-install and the import of your existing datastore. In practice, this isn't a big deal so long as you have documented your configuration, but it doesn't align well with the regular patching regime I apply to the remainder of my lab.

Updates and Improvements

Over the life of my Pi VMWare Server, I have:

  • Worked-around the missing SNMP module by writing a monitoring shell script called from my Ansible host.
  • Upgraded my guests to Debian 11; enabling the backports repository provides access to packaged open-vm-tools.
  • Upgraded my host several times, as in place upgrade is possible from later builds.
  • Considered purchasing a powered USB hub for my datastore disk as potential fix for the frequent disconnection.
  • Written a simple but effective shell script for snapshot backups.
  • Observed that the maturity and availability of ARM-based software has improved significantly. 

My conclusions

Would I do it again? Perhaps; it has been great fun to build and play with, and has kept my VMWare skills active. This was very welcome as I have been running a project at work to replace VMWare with Hyper-V.

It has certainly not replaced or even supplemented my x64 VM hosts. In fact, I have purchased another x64 hosting server during the period of review due, in part, to its shortcomings.

That said, I'm really hopeful that VMWare will continue to improve the product, along with a maturing of the general ARM ecosystem to include OSs and applications. Over time, this is likely to become a really useful and interesting solution... but not just yet.

Should you buy one? Firstly, good luck finding a Raspberry Pi at the time of writing as stock levels are non-existent. A quick eBay search finds a refurbished small form-factor Lenovo M93 Tiny PC with i5-4570T processor, 8GB RAM and a 240GB SSD for the same price. This is likely to offer a similar capacity for learning VMWare and running VMs but without the problems described above.

Next steps

I'm not ready to give up just yet. When time allows I might see how much of a complete virtualised datacentre infrastructure I can fit on a Pi. I might even look at using KVM on Fedora as an alternative hypervisor to ESXi.