DESC:FORM 2
THE PATENTS ACT, 1970
[39 of 1970]
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(Section 10; Rule 13)

METHOD TO IMPLEMENT RDMA NVMe DEVICE

SAMSUNG R&D INSTITUTE INDIA – BANGALORE Pvt. Ltd.
# 2870, ORION Building, Bagmane Constellation Business Park,
Outer Ring Road, Doddanekundi Circle,
Marathahalli Post,
Bangalore -560037, Karnataka, India
Indian Company

The following Specification particularly describes the invention and the method it is being performed

RELATED INVENTION
The present invention claims benefit of the Indian Provisional Application Nos. 3494/CHE/2015 titled “ METHOD TO IMPLEMENT RDMA NVMe DEVICE” by Samsung R&D Institute India – Bangalore Private Limited, filed on 8th July 2015, which is herein incorporated in its entirety by reference for all purposes.

FIELD OF THE INVENTION
The present invention generally relates to data storage over network and more particularly to a method for integrating remote direct memory access (RDMA)-enabled network interface controller (RNIC) and Non-Volatile Memory Express (NVMe) at the controller/device side.

BACKGROUND OF THE INVENTION

In a conventional computing arrangement, a client and a server include respective network interface controllers (NICs) or network (NW) input/output (I/O) devices that are capable of communicating with each other using a Remote Direct Memory Access (RDMA) protocol. The server includes a host processor that executes the server's operating system and associated drivers. The server may also include a storage controller that manages access to storage maintained at or by the server. The client's NW I/O device issues requests to the server's NW I/O device to write data to and read data from the storage maintained by the server. The server's operating system, associated drivers, and host processor process the requests received by the server's NW I/O device, and issues corresponding requests to the storage controller. The storage controller receives and executes these corresponding requests. After executing the corresponding requests, the storage controller issues request completion information (and associated data if data has been read from the storage) to the server's operating system and associated drivers. From this, the server's operating system, associated drivers, and host processor generate corresponding request completion information and associated data, and issue the corresponding request completion information and associated data to the server's NW I/O device. The server's NW I/O device then issues the corresponding request completion information and associated data to the client's NW I/O device.

The Non-Volatile Memory Express (NVMe) forum is currently standardizing the NVMe over fabrics specification which targets different transport for NVMe protocol. The forum is aiming on low latency data transfer over network using Remote DMA with NVMe. With increase in need for faster data storage on the network, the usage of low latency NVMe based solid-state drive (SSD) using RDMA is highly anticipated.

According to the current art, the memory is managed separately where the command and data are copied from RNIC memory to NVMe memory and vice versa. Each of the modules register their own interrupts and there is redundant usage of host resources. The problems faced by the current method include incompatible interfaces, increased redundancy, conversion module increases cost, inefficient memory usage and multiple interrupt handlers.

In view of the foregoing, there is a need for a method and system for handling the data over network for seamless data transfer between the initiator device and the controller/device.

The above mentioned shortcomings, disadvantages and problems are addressed herein and which will be understood by reading and studying the following specification.

SUMMARY OF THE INVENTION

The various embodiments of the present invention disclose a method to implement a Remote Direct Memory Access (RDMA) Non-Volatile Memory Express (NVMe) device. According to the present invention, a NVMe solid-state drive (SSD) is implemented which employs RDMA to support block device data storage over a network. The present invention enables a user to access the device remotely over the network and perform all the disk operations like it is connected locally to a machine.

According to an embodiment of the present invention, a method of enabling access to a storage device remotely over the network is disclosed. The method comprises of initializing a Non-Volatile Memory Express (NVMe) controller, by a network device coupled to a server, configuring, by the network device, a NVMe queue pair for handling a remote device discovery process, receiving, at the network device, a request from the remote device to access the storage device controlled by the NVMe controller maintained at the server, initiating, by the network device, the discovery process for locating the remote device and establishing, by the network device, a connection with the remote device by mapping the NVMe queue pair with a Remote Direct Memory Access (RDMA) queue pair, once the remoted device is discovered.

According to an embodiment of the present invention, the method of establishing the connection with the remote device comprises of mapping NVMe queue pair associated with the network device with the RDMA queue pair associated with the remote device and mapping an admin submission queue base address and completion queue base addresses with a RDMA admin queue pair.

According to an embodiment of the present invention, the method further comprises of transmitting, by the remote device, a NVMe command to the network device, fetching by the NVMe controller, the NVMe command, where the NVMe command comprises a reference of a queue pair receive completion event, decoding the NVMe command by the NVME controller and triggering the queue pair completion event.

According to an embodiment of the present invention, the NVMe command comprises one of a flush, a write, a read, a write uncorrectable or a compare instruction. The NVMe command further comprises of a Physical Region Page (PRP) field, where the PRP field comprises of a data identity which is to be sent to the network device.

According to an embodiment of the present invention, the method of initializing the controller comprises of booting up of the processor and invoking firmware, initializing by firmware one or more system modules comprising Special Function Registers (SFRs), memory units and initializing RNIC.

According to an embodiment of the present invention, the storage device connected to the network device is implemented with an NVMe storage protocol using a Remote Direct Memory Access (RDMA). The NVME storage protocol comprises one of Internet Wide Area RDMA protocol (iWARP), Infiniband or RDMA over Converged Ethernet (RoCE). Further a Peripheral Component Interconnect Express (PCIe) based RDMA Network interface Card (RNIC) and the network devices are coupled together by mapping the queue pairs of the network device and the remote device and registering same memory for both the network device and the remote device.

According to an embodiment of the present invention, the method further comprises of indicating a status of the NVMe command in a completion indication field to the remote device. The completion indication indicates that the received command to access the storage device is executed by the NVMe controller.

According to an embodiment of the present invention, the storage device comprises one of a hard disk drive (HDD) or a solid state drive (SSD), the SSD having non-volatile memory comprising at least one of a 3-dimensional cross-point memory, flash memory, ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory, nanowire, ferroelectric transistor random access memory (FeTRAM or FeRAM), nanowire or electrically erasable programmable read-only memory (EEPROM).

Embodiments herein further disclose a system for providing access to a storage device remotely over the network. The system comprising a storage device, remote device, a Non-Volatile Memory Express (NVMe) controller and a network device coupled to a server. The network device is adapted for configuring a NVMe queue pair for handling a remote device discovery process, receiving a request from the remote device to access the storage device controlled by the NVMe controller maintained at the server, initiating the discovery process for locating the remote device and establishing a connection with the remote device by mapping the NVMe queue pair with a Remote Direct Memory Access (RDMA) queue pair, once the remoted device is discovered.

The foregoing has outlined, in general, the various aspects of the invention and is to serve as an aid to better understanding the more complete detailed description which is to follow. In reference to such, there is to be a clear understanding that the present invention is not limited to the method or application of use described and illustrated herein. It is intended that any other advantages and objects of the present invention that become apparent or obvious from the detailed description or illustrations contained herein are within the scope of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS
The other objects, features and advantages will occur to those skilled in the art from the following description of the preferred embodiment and the accompanying drawings in which:

Figure 1 is a schematic representation illustrating an interface mechanism, according to the prior art illustration.

Figure 2 is a block diagram illustrating system for providing access for remote device data storage over network, according to an embodiment of the present invention.

Figure 3 is a block diagram of a RDMA NVMe subsystem, according to an embodiment of the present invention.

Figure 4 is a block diagram illustrating interfacing of an RDMA NVMe device, according to an embodiment of the present invention.

Figure 5 is a block diagram illustrating interfacing of an RDMA NVMe device, according to another embodiment of the present invention.

Figure 6 is a block diagram illustrating a method of enabling access to a storage device remotely over the network using an RDMA NVMe device, according to an embodiment of the present invention.

Although specific features of the present invention are shown in some drawings and not in others, this is done for convenience only as each feature may be combined with any or all of the other features in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The various embodiments of the present invention disclose a method to implement a Remote Direct Memory Access (RDMA) Non-Volatile Memory Express (NVMe) device. In the following detailed description of the embodiments of the invention, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

The specification may refer to “an”, “one” or “some” embodiment(s) in several locations. This does not necessarily imply that each such reference is to the same embodiment(s), or that the feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments.

As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless expressly stated otherwise. It will be further understood that the terms “includes”, “comprises”, “including” and/or “comprising” when used in this specification, specify the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations and arrangements of one or more of the associated listed items.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Figure 2 is a block diagram illustrating an NVMe SSD which employs RDMA to support remote device data storage over a network, according to an embodiment of the present invention. The system comprises of a remote device 201, a network 202, a server 203, a network device 204, and an NVMe Controller 205 coupled to the server and a storage device 206.

The network 204 can be any network allowing transport of data by any means between the remote device 201, server 203, network device 204 and the storage device. In general, data can take the form of, but is not limited to, images, video, music, and documents. The network 204 can be private or public, wired or wireless, in whole or in part.

The server 203 can be any computer system or equivalent that provides services to other computing systems over a network 204. Generally, the server 203 manages information and/or has a mechanism for validation of information. In the system, the server 203 manages information and/or has a mechanism for validation of information. In an embodiment of the invention, the server 203 provides identification
of devices connected to the network 202.

In another embodiment of the invention, the server 203 validates credentials sent by the devices to the network 202 with credentials sent by another device coupled to the network 202. In yet another embodiment of the present invention, the server 203 stores data and manages access to the stored data over the network 130.

The remote device 201 can be, but is not limited to, a personal computer, a digital lifestyle device such as a digital still image camera, digital video camera, television, or digital reading/display device that is communicatively coupled to the network 202. In
general, the remote device 201 creates remote access information to transfer to the network device 204 over the network 202.

The storage device 206 comprises one of a hard disk drive (HDD) or a solid state drive (SSD), the SSD having non-volatile memory comprising at least one of a 3-dimensional cross-point memory, flash memory, ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory, nanowire, ferroelectric transistor random access memory (FeTRAM or FeRAM), nanowire or electrically erasable programmable read-only memory (EEPROM). The storage device 206 connected to the network device 204 is implemented with a NVMe storage protocol using a Remote Direct Memory Access (RDMA). Here the NVME storage protocol comprises one of Internet Wide Area RDMA protocol (iWARP), Infiniband or RDMA over Converged Ethernet (RoCE).

Figure 3 is a block diagram of a RDMA NVMe subsystem, according to an embodiment of the present invention. The interfaces comprising of the building blocks for efficient integration of RDMA and NVMe controllers are as described herein. The RDMA Queue Pair will be registered as NVMe queues. The NVMe uses the same Direct Memory Access (DMA) which is used by a RDMA-enabled network interface controller (RNIC) and place the data directly. The internal RAM is registered with the RDMA as Non-Shared Memory Region (NSMR) and the same is used for handling RDMA_SEND requests which consist of NVMe commands and posting responses using RDMA_SEND. Also, the internal RAM is registered with the RDMA for direct data placement used for RDMA_WRITE and RDMA_READ_RES. The data from the initiator is directly placed into the Random Access Memory (RAM) of Firmware (F/W) to process as per the respective command.

Figure 4 is a block diagram illustrating interfacing of an RDMA NVMe device, according to an embodiment of the present invention. The process involved in the mechanism which covers the sequences from connection establishment between an remote device and the network device for handling of one full Read and Write transaction is as follows. Once the network device is powered on, NVMe controller initialization takes places which comprises of booting up of the processor and invoking Firmware, initializing by firmware all the modules including Special Function Registers (SFRs), memory units, etc and lastly RNIC initialization. Upon initialization, RNIC configures itself as server. Further, the network device by default configures a master queue pair for handling the discovery process. Upon configuring the master queue pair, the network device listens for incoming connection requests from any remote devices.

Further, the network device initiates the discovery process for locating the remote device. Once the remote device is discovered, the network device binds itself with the remote device and the connection is established. The connection establishment indicates that the master queue pairs at both the ends are configured. The master queue pair is mapped with the admin queue for NVMe and the master queue pair at the network device has the attributes where the Admin commands are requested by the network device using this queue pair and a default NSMR is registered exclusively for this queue pair.

Still further, the network device creates a NVMe identify command and posts an RDMA_SEND. The NVMe command will be decoded, which is nothing but processing the command by the NVMe controller. The NVMe command is fetched with the reference of queue pair receive completion event. Upon fetching the NVMe command, the command is directly read by the NVMe controller as the queue tail pointer is always in sync with queue pair completion event. The completion event triggers the firmware in the similar manner as ring doorbell operation. For data movement, data is directly DMAed into RDMA Scatter/Gather Lists (SGLs) which are registered for a particular operation.

According to the present invention, the command capsule also contains the Physical Region Page (PRP) field which is a host registered shared memory region. The identify data will then be directly read from the firmware and DMAed into the RDMA SGL which is to be sent to the network device. The above steps are same for any command which performs data transfer from the remote device to the network device. In case of data transfer from the remote device to the storage device, the respective write command is decoded by the NVMe controller. The firmware will register the address where it wants to store the data with RDMA. Further, the SGL will be formed for respective command with the registered address and sent to the network device. Upon receiving, the network device will form the data packet addressing it to the requested SGL and post it. Once the respective RDMA packet is received, RNIC will directly put the data into firmware’s buffer. The firmware will be notified once the receive completion event is triggered by RNIC.

Figure 5 is a block diagram illustrating interfacing of an RDMA NVMe device, according to another embodiment of the present invention. The storage device is connected over network is implemented with the NVMe storage protocol using RDMA. The network protocols include iWARP, infiniband, RoCE etc. Further, the NVMe queues are mapped with the RDMA queue pair, the admin submission queue base address and completion queue base addresses are mapped with RDMA admin queue pair. Similarly for multiple IO queues as well. The completion event for any queue pair implements the NVMe controller for processing the requested operation. No Tail doorbell update is required. The data movement from the controller and host will happen through memory registered with RDMA as different memory regions.

According to another embodiment of the invention, the standard interfaces provided by the Peripheral Component Interconnect Express (PCIe) based NVMe controller needs to be modified to be interfaced with the RNIC efficiently.

According to an embodiment of the present invention, the existing PCIe based RNIC and NVMe devices are coupled together which utilizes less memory by mapping the queue pairs and registering same memory for both the drivers. The interrupts registered for both the devices are reduced by utilizing RNIC interrupts to invoke NVMe controller operation and vice versa.

Figure 6 is a block diagram illustrating a method of enabling access to a storage device remotely over the network using an RDMA NVMe device, according to an embodiment of the present invention. At step 602, initialize a Non-Volatile Memory Express (NVMe) controller, by a network device coupled to a server. At step 604, the network device configures a NVMe queue pair for handling a remote device discovery process. At step 606, the network device receives a request from the remote device to access the storage device controlled by the NVMe controller maintained at the server. At step 608, the network device initiates a discovery process for locating the remote device. At step 610, the network device establishes a connection with the remote device by mapping the NVMe queue pair with a Remote Direct Memory Access (RDMA) queue pair, once the remoted device is discovered.

The embodiments as disclosed by the present invention provides for low latency of NVMe and speed of RDMA are combined to provide faster network storage solutions, standalone SSD which has network capability and Portability and scalability of storage.

The present embodiments have been described with reference to specific example embodiments; it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the various embodiments. Furthermore, the various devices, modules, and the like described herein may be enabled and operated using hardware circuitry, for example, complementary metal oxide semiconductor based logic circuitry, firmware, software and/or any combination of hardware, firmware, and/or software embodied in a machine readable medium. For example, the various electrical structure and methods may be embodied using transistors, logic gates, and electrical circuits, such as application specific integrated circuit.

Although the embodiments herein are described with various specific embodiments, it will be obvious for a person skilled in the art to practice the invention with modifications. However, all such modifications are deemed to be within the scope of the claims. It is also to be understood that the following claims are intended to cover all of the generic and specific features of the embodiments described herein and all the statements of the scope of the embodiments which as a matter of language might be said to fall there between.

,CLAIMS:CLAIMS
We Claim:

1. A method of enabling access to a storage device remotely over the network, the method comprising:
initializing a Non-Volatile Memory Express (NVMe) controller, by a network device coupled to a server;
configuring, by the network device, an NVMe queue pair for handling a remote device discovery process;
receiving, at the network device, a request from the remote device to access the storage device controlled by the NVMe controller maintained at the server;
initiating, by the network device, the discovery process for locating the remote device; and
establishing, by the network device, a connection with the remote device by mapping the NVMe queue pair with a Remote Direct Memory Access (RDMA) queue pair, once the remoted device is discovered.

2. The method of claim 1, wherein establishing the connection with the remote device comprises of :
mapping the NVMe queue pair associated with the network device with the RDMA queue pair associated with the remote device; and
mapping an admin submission queue base address and completion queue base addresses with a RDMA admin queue pair.

3. The method of claim 1, further comprising:
transmitting, by the remote device, an NVMe command to the network device;
fetching by the NVMe controller, the NVMe command, where the NVMe command comprises a reference of a queue pair receive completion event;
decoding the NVMe command by the NVME controller; and
triggering the queue pair completion event.

4. The method of claim 3, wherein the NVMe command comprises one of a flush, a write, a read, a write uncorrectable or a compare instruction.

5. The method of claim 3, wherein the NVMe command further comprises of a Physical Region Page (PRP) field, where the PRP field comprises of a data identity which is to be sent to the network device.

6. The method of claim 1, wherein the initializing the controller comprises of:
booting up of the processor and invoking firmware;
initializing by firmware one or more system modules comprising Special Function Registers (SFRs), memory units; and
initializing a RDMA Netwrork interface Card (RNIC).

7. The method of claim 1, wherein the storage device connected to the network device is implemented with a NVMe storage protocol using a Remote Direct Memory Access (RDMA).

8. The method of claim 7, wherein the NVME storage protocol comprises one of Internet Wide Area RDMA protocol (iWARP), Infiniband or RDMA over Converged Ethernet (RoCE).

9. The method of claim 1, wherein a Peripheral Component Interconnect Express (PCIe) based RDMA Netwrork interface Card (RNIC) and the network devices are coupled together by mapping the queue pairs of the network device and the remote device and registering same memory for both the network device and the remote device.
10. The method of claim 1, further comprising indicating a status of the NVMe command in a completion indication field to the remote device, where the completion indication indicates that the received command to access the storage device is executed by the NVMe controller.

11. The method of claim 1, wherein the storage device comprises one of a hard disk drive (HDD) or a solid state drive (SSD), the SSD having non-volatile memory comprising at least one of a 3-dimensional cross-point memory, flash memory, ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory, nanowire, ferroelectric transistor random access memory (FeTRAM or FeRAM), nanowire or electrically erasable programmable read-only memory (EEPROM).

12. A system for providing access to a storage device remotely over the network, the system comprising:
a remote device;
a Non-Volatile Memory Express (NVMe) controller;
a network device coupled to a server adapted for:
configuring a NVMe queue pair for handling a remote device discovery process;
receiving a request from the remote device to access the storage device controlled by the NVMe controller maintained at the server;
initiating the discovery process for locating the remote device; and
establishing a connection with the remote device by mapping the NVMe queue pair with a Remote Direct Memory Access (RDMA) queue pair, once the remoted device is discovered.

13. The method of claim 1, wherein initializing the controller comprises of:
booting up of the processor and invoking firmware;
initializing by firmware one or more system modules comprising Special Function Registers (SFRs), memory units; and
initializing RNIC.

Dated the 25th day of November, 2015

Signature

Keerthi J S
Patent Agent
Agent for applicant