It should be noted that we have been using the in-kernel implementation of the TCP NVMe Target module (
nvmet-tcp) to export the volume from the target system. The very same individual decided to repeat the experiment with the SPDK userspace implementation. Interestingly, it worked immediately and without issue. The only real difference (from the perspective of the ESXi host) being that the SPDK framework supports Fused commands.
Defined in the NVMe specification (1.2.1), a fused command allows for the execution of two commands as a single atomic unit. For example: Compare and Write (0x05 / 0x01) which is comparable to the SCSI COMPARE AND WRITE (0x89).
Anyway, currently, the in-kernel implementation does not support this feature and it is unlikely that it will in the near future. In 2021, there was an attempt to implement this in the kernel but it was met with a lot of skepticism.
More Google searches turned up similar results:
The linux nvme target does not support fused commands so there is no support for VMware currently.
Assuming this to be the case, I decided to look to the company that developed the in-kernel TCP NVMe Target implementation: Lightbits Labs. An official announcement was made in September of 2021 that their flagship operating system, LightOS, supports NVMe Target over TCP to VMware ESXi 7 Update 3. And a whole webpage has been dedicated to provide users with additional but limited details.
But how are they doing it? Are they leveraging SPDK and not their very own kernel module? Also, does VMware ever plan to support NVMe Targets without fused command support?