VMware, Lightbits Labs and NVMe over TCP

by Posted on

Somewhat recently, a very committed contributor to one of my open source projects noticed an issue when trying to import NVMe exported Targets from a Linux machine into a VMware ESXi host. It didn’t work. He was using version 7.0U3 which supposedly supports importing NVMe Targets over TCP. Quite some time was spent going down this rabbit hole. Parsing the cryptic system logs only raised more questions. Google search after Google search revealed that we weren’t the only ones experiencing this exact same issue. The ESXi host was able to see the NQN (NVMe Qualified Name) but set the status of the path to the drive to “Dead.”

It should be noted that we have been using the in-kernel implementation of the TCP NVMe Target module (nvmet-tcp) to export the volume from the target system. The very same individual decided to repeat the experiment with the SPDK userspace implementation. Interestingly, it worked immediately and without issue. The only real difference (from the perspective of the ESXi host) being that the SPDK framework supports Fused commands.

Defined in the NVMe specification (1.2.1), a fused command allows for the execution of two commands as a single atomic unit. For example: Compare and Write (0x05 / 0x01) which is comparable to the SCSI COMPARE AND WRITE (0x89).

Anyway, currently, the in-kernel implementation does not support this feature and it is unlikely that it will in the near future. In 2021, there was an attempt to implement this in the kernel but it was met with a lot of skepticism.

More Google searches turned up similar results:

The linux nvme target does not support fused commands so there is no support for VMware currently.

Assuming this to be the case, I decided to look to the company that developed the in-kernel TCP NVMe Target implementation: Lightbits Labs. An official announcement was made in September of 2021 that their flagship operating system, LightOS, supports NVMe Target over TCP to VMware ESXi 7 Update 3. And a whole webpage has been dedicated to provide users with additional but limited details.

But how are they doing it? Are they leveraging SPDK and not their very own kernel module? Also, does VMware ever plan to support NVMe Targets without fused command support?

 

Related Posts:

Published by Petros

Petros Koutoupis is the self appointed BDFL of the RapidDisk project. Most of his career has been in software development in the data storage industry. He is deeply involved in open source software development and for years has written code for the Linux kernel, various open source device drivers and applications in both the embedded and server spaces.

4 Replies to “VMware, Lightbits Labs and NVMe over TCP”

  1. Hi Petros,

    Since you asked… Lightbits does it by having our own target implementation that does support fused commands and was extensively validated in VMware environments. In fact, unlike SPDK — which we also use and love, but in other contexts — Lightbits SDS is a VMware certified solution. I hope this helps.

    Cheers,
    Muli (Lightbits Co-Founder and Chief Scientist)

    1. Thank you very much for clarifying. It is much appreciated. Is there any plan to merge that code upstream sometime in the future?

      1. Hi Petros, ,my pleasure. Unfortunately we do not have any plans at this time to implement fused commands in the Linux kernel NVMe target or to open source the Lightbits SDS. If you are interested in using Lightbits in your environment and are not averse to a proprietary offering, feel free to shoot me an email or hit me up on twitter.

        Cheers,
        Muli

Comments are closed.