Airgapped Testing - VMware Edition
Recently, I got a bug report for the
kitchen-vcenter driver, which allows lifecycle management of testing VMs on VMware vCenter environments. Apparently, a customer tried to create a VM without any network interface.
The problem was that this crashed in a very unintuitive way. But it made me wonder: Would it be possible to use non-networked machines for tests? It turns out: That’s absolutely possible!
First off, this is probably not what you want to do. At least some degree of software installation is likely to take place - and without network access, you are a bit limited in that regard. But maybe there are some valid use cases in your project?
kitchen-vcenter bugfix version 2.11.4 replaces the noisy crash with a friendly warning that this VM will not have network access.
Create a Non-Network Machine
To finish the creation of a VM, the
kitchen-vcenter driver expects some IP addresses to be present.
To set a static IP address for detection only, you need to create a dummy device. Be careful not to assign a “link-local” address1 because this is usually an indicator for a DHCP-related problem. In this case, the driver will warn you about this issue and not continue.
/etc/netplan/01-netcfg.yaml will create a dummy bridge including a private IP address2:
network: version: 2 renderer: networkd bridges: dummy0: dhcp4: no dhcp6: no accept-ra: no interfaces:  addresses: - 192.0.2.1/24
Now we set the minimum configuration for the driver in
kitchen.yml and can run
kitchen create successfully:
driver: name: vcenter vcenter_host: vcsa.lab.local vcenter_username: firstname.lastname@example.org vcenter_password: "..." vcenter_disable_ssl_verify: true datacenter: "Datacenter"
Transfer Files and Execute Commands
I can imagine people wondering about the purpose of this. We can provision (and de-provision) machines now, which are only accessible via the vCenter web console. But for testing with Test Kitchen, files need to be transferred, and commands need some way of execution.
Well, as I described in a post back then about Instant Clones and VMware Guest Operations that is not a problem. VMware comes with a handy side-channel for all that, as it needs to inspect machines even without using their network interfaces - how else would you be able to use the Web console in the end?
This type of access is done via a combination of RPC calls and mapped memory regions, handily providing API functionality to execute commands and transfer files (with proper OS authentication, though!).
kitchen-vcenter driver also uses this functionality for its Active IP Discovery feature, and it got ported into the train-vsphere-gom transport.
Even though Test Kitchen lent inspiration to the Train framework, there is no official way to connect both software pieces. But, the inofficial kitchen-transport-train provides an adapter to use any Train transport with Test Kitchen. This includes exotic ones like Telnet, Serial/USB and AWS Systems Manager.
Install both via
chef gem install train-vsphere-gom kitchen-transport-train, then configure your
kitchen.yml section for file transfer/execution:
transport: name: train backend: vsphere-gom username: root password: "..." vcenter_host: vcsa.lab.local vcenter_username: email@example.com vcenter_password: "..."
Notice that this does not provide the
kitchen login command. But it also means that you can use Test Kitchen VMs even without routing/firewalling!
You might start thinking about the provisioner now that we can provision non-networked machines and connect to them. We don’t want to transfer the whole Chef Infra packages over the transport, so I recommend preinstalling it inside the VMware templates.
Alternatively, you can toy around with something like the
shell provisioner, which is part of Test Kitchen itself:
provisioner: name: shell command: "uname -a"
Debugging might be tricky without any access to the machine or merely via the vCenter console. But there are alternatives:
govc is a tool managed by VMware themselves, which enables performing most VMware-related tasks via command line. It also includes support for running commands via
GOVC_URL=https://...:...@vcsa.lab.local/sdk GOVC_INSECURE=1 govc guest.run -vm my-vm "uname -a"
But what if you want interactive access? Well, I was in this specific situation a while ago when a customer had access to development machines limited too much. After trying different things for some time, I decided to write an interactive shell based on Train to connect to remote systems: TrainSH.
- use all installed Train transports (including
- open multiple sessions in parallel, even with different backends
- interactively execute commands
- upload and download files
- locally view and edit remote files, for example, with
- copy files between active sessions
VI_USERNAME="..." VI_PASSWORD="..." VI_SERVER=vcsa.lab.local trainsh connect vsphere-gom://root:...@my-vm
Have fun exploring machines without network connectivity!