FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
(See Section 10 and Rule 13)
Title of invention:
LOW LATENCY FIFO MESSAGING SYSTEM
Applicant
TATA Consultancy Services Limited A company Incorporated in India under The Companies Act, 1956
Having address:
Nirmal Building, 9th Floor,
Nariman Point, Mumbai 400021,
Maharashtra, India
The following specification describes the invention and the manner in which it is to be performed.

FIELD OF THE INVENTION
The present invention relates to the field of inter processor messaging and more particularly relates to low latency remote messaging assisted by Remote Direct Access Memory based flrst-in-first-out system.
BACKGROUND OF THE INVENTION
With the advent of computing acceleration, the exchange of data between two different software threads or processor cores demands to be fast and efficient. In general, the existing methods of remote messaging over a typical TCP/IP schemes have the disadvantage of high CPU utilization for sending and receiving of messages. In the messaging TCP/IP paradigm, a software thread does not share any common mcmon space with another software thread it desires to communicate with. Instead, sending and receiving a message to and from another software thread requires the use of socket send () and socket recv () system call respectively.
This aspect of communication via typical TCP/IP scheme involves a lot of software instructions that are to be executed by CPU cores residing on both the sending and remote hosts. Additionally, every time a send () system call is executed there is a change of context from user level to system level which amounts to a high CPU overhead. So is the case for the receive system call on the receiving end.
Since the amount of data that should be exchanged between two different software threads has swelled up, the message FIFO between two processor cores needs to be low latency so that the processors need not slow down due to frequent communication. With TCP/IP protocol in place, it is very difficult to achieve low latency for messaging at high message rates because of the system calls that needs to be executed by the application process in order to facilitate message exchange between the sending and receiving peers.

This implies that the messaging infrastructure (including software) should be capable of processing very large workloads. Very large workloads imply more than a million messages per second. Accordingly, keeping in view of the amount of work load the current messaging system presently demands and future anticipated workload, new system which ensures low latency messaging and optimized throughput is urgently required.
Thus, in the light of the above mentioned background of the art, it is evident that, there is a need for a system and method which:
• Provides high throughput and low latency messaging technique for an interprocesses communication between at least two processes running on at least two nodes.
• increases the throughput optimization of the messaging system;
• reduces the latencies of the messaging system;
• requires minimum infrastructure;
• reduces the cost of the hardware setup to improve throughput and reduce the latency of the messaging system; and
• is easy to deploy on existing systems.
OBJECTS OF THE INVENTION
The principle object of the present invention is to provide a system for messaging high throughput optimization and latency reduction in an inter-processes communication between the processes running on the remote nodes.
Another significant object of the invention is to provide a high throughput and low latency messaging system in an inter-processes communication between the multiple processes running remote nodes.

It is another object of the present invention to provide a cost effective high throughput and low latency messaging system in an inter-processes communication between the processes running on the remote nodes.
Another object of the invention is to provide a system employing minimal computational resources by reducing CPU intervention for high throughput and low latency messaging and making it more available for application programs.
Yet another object of the invention is to provide an inter-process messaging system requiring minimal infrastructure support by eliminating the need for additional receiver required for receiving messages at remote host thereby reducing one latency introducing component.
Yet another object of the invention is to reduce the number of additional message copies required for realizing high throughput and low latency message passing technique in an inter-processes communication.
SUMMARY OF THE INVENTION
Before the present methods, systems, and hardware enablement are described, it is to be understood that this invention in not limited to the particular systems, and methodologies described, as there can be multiple possible embodiments of the present invention and which are not expressly illustrated in the present disclosures. It is also to be understood that the terminology used in the description is for the purpose of describing the particular versions or embodiments only, and is not intended to limit the scope of the present invention which will be limited only by the appended claims.
The present invention envisages a system and method for low latency and high throughput messaging in inter-process communication between the processes running on remote nodes.

In the preferred embodiment of the invention the system implements an asynchronous lockless FIFO message queue between two server hosts using Random Direct Memory Access (RDMA) technology. The inter-process communication is implemented by using RDMA write operations accessed through infiniband verbs library that obviates the use of TCP/IP scheme provided by the operating system for remote messaging which involves a high system call overhead.
The present system, on the contrary, provides a direct access to the RDMA enable Network Interface Cards (NIC) without a system call overhead which is a key to achieving very low latency messaging. The RDMA NIC converts RDMA write operations into a series of RDMA protocol messages over TCP/IP which is acted upon by the RDMA NIC in the remote host and makes the necessary updates to the memory of the remote host.
According to one of the preferred embodiments of the present invention a system for lockless remote messaging in an inter-process communication between at least two processes running on at least two nodes implemented by RDMA supported Network Interface Card configured to synchronize a memory mapped file hosted on each of the said node is provided, the system comprising of:
a) a first host node communicatively coupled with a second host node for sending and receiving messages over a computing network;
b) RDMA supported Network Interface Card deployed on each of said host nodes for executing RDMA commands;
c) a storage hosted on each host node adapted to store inter process messages invoked by either of the communicatively coupled host nodes;
d) a first memory mapped file hosted on the first host node configured to synchronize static circular queue of messages with second memory mapped file hosted on the second host node; and
e) at least one remote sender process running on the remote sender first host node, asynchronously sending at least one batch of message along with corresponding RDMA work requests from the said queue to the second host node.

According to one of the other preferred embodiments of the present invention, a memory mapped structure comprising computer executable program code is provided, wherein the said structure is configured to synchronize static circular queue of messages between the sending and receiving host nodes, the structure comprising:
a) plurality of messages bundled together to form at least one batch, each batch comprising of a sequence of payload section, each payload intermittently followed by a corresponding node counter element to constitute a contiguous memory region and coupled with a corresponding common queue data and continuously arranged headers;
b) the common queue data comprising of data pointing element adapted to point to the next message to be received by the receiving host node; a free pointing element adapted to point to the message written by the sending host node; an insertion counter to count number of messages sent by the sender host node and a deletion counter to count the number of messages read by the receiving host node since queue initiation;
c) rdma free pointing element adapted to point to the message buffer in which the sending host node inserts new message;
d) a rdma insertion counter to count number of messages inserted by a remote sender process in accordance with corresponding rdma work request; and
e) last message node pointing element to point to the last message sent from the remote sender process to the receiving host node.
In one of the other embodiments of the present invention, a method for lockless remote messaging in an inter-process communication between at least two processes running on at least one sending and one receiving host node implemented by RDMA supported Network Interface Card configured to synchronize queue of messages via memory mapped file hosted on each of the said node is provided, the said method comprising: a) initializing the transfer of first batch of message from data buffer of sender host
node to the corresponding memory mapped file whenever the message queue is
indicated of sufficient space;

b) updating rdma free pointing element to initialize last message node pointing element to point to the last message sent through the remote sender process;
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing summary, as well as the following detailed description of preferred embodiments, are better understood when read in conjunction with the appended drawings. For the purpose of illustrating the invention, there is shown in the drawings example constructions of the invention; however, the invention is not limited to the specific methods and system disclosed. In the drawings;
Figure 1 illustrates a typical layout of Memory Mapped File as known in the prior art.
Figure 2 shows circular queue of messages as represented in a Memory Mapped File Layout.
Figure 3 illustrates a system for inter process communication between two server hosts with memory mapped files synchronized using RDMA.
Figure 4 shows a design layout of Memory Mapped File in accordance with the preferred embodiment of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
Some embodiments of this invention, illustrating all its features, will now be discussed in detail.
The words "comprising;" "having," "containing," and "including;" and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or

items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.
It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, the preferred, systems and methods are now described.
The disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms.
Definitions;
Throughput: The numbers of messages read or write operations that can be from the
queue per second are called the Throughput.
Latency - The time elapsed between the sending of a message by the sender process and receiving of that message by the receiver process is the latency experienced by that message.
RDMA write operation: RDMA write operation, interchangeably called as RDMA write work request is a command issued to the RDMA capable NIC. It is a user level call which notifies the local NIC about where the data is placed in RDMA registered memory (RAM) and its length. The NIC then (asynchronously) fetches the data and transfers it across the network using the relevant (IWARP) RDMA protocol. In effect, it writes the specified data from the local memory location to the memory location on the remote host. The NIC on the remote host responds to the iWARP messages by placing the data in its RDMA registered memory in its host, thus carrying out an RDMA write work request. The RDMA write operations are accessed through infiniband verbs library.

Memory registration: These are API's provided by RDMA to make a local memory region available to remote hosts. This is essential for the use of RDMA write operations.
RDMA technology allows applications to access memory on remote hosts as if it was available on the same host where the application runs. RDMA was first introduced in Infiniband networks and later supported on Ethernet using iWARP. In both networks the network interface cards (NIC) are capable of executing RDMA write commands which cause the placement of data in the memory area of the remote host. A RDMA interlace allows an application to read from and/or write into memory (RAM) locations of a remote host. This is very much unlike sending and receiving messages. It gives the application the illusion of a shared memory between the sender and receiver processes even though they run on separate hosts.
The device drivers of NICs supporting RDMA operation provide a direct interface to application programs to send data bypassing the operating system. The costly overhead of switching from user mode to system mode is avoided thereby allowing the application to be executed by the CPU more efficiently. Further, The RDMA NIC implements the complex networking tasks required to transfer the message from the local host to the remote host without any CPU intervention, make the CPU more available for applications programs.
The other advantageous feature of the present invention is eliminating the need of performing an additional copy operation as required in a typical system call assisted communication. For an RDMA write operation the NIC does a Direct Memory Access transfer with the source data as the registered memory region which the application running in user mode can directly write the message into, thereby obviating the need for that additional copy.
Also, as the RDMA write operation takes up the entire responsibility of the making the data available in the registered memory region of the remote host where the receiver process runs, there is no need for a separate receiver process to manage the receiving of

messages from the network which effectively contributes to the reduction of one latency introducing component in the system.
As shown in Figure 1 & 2, a typical layout of memory mapped file contains a static circular queue of messages. Each message structure in the file has a header section 101 and a payload section 102. The payload section 102 contains the raw message as passed by the application. It is also referred to as message buffer. The header section 101 contains the pointer to the header of the next message thus creating a circular queue of messages. The initial part of the file contains data specific to the queue. Some of the important variables in this section are:
• data_anchor 103 - points to the next message to be read by the receiver.
• free_node 104 - points to the message will be written to by the sender.
• number_of_inserts 105 - number of messages sent by the sender (since queue
creation) number_of_deletes 106- number of messages read by the receiver (since queue creation)
Referring to Figure 2, additional free node pointer called rdma_free_node 201 and a new counter variable called rdma_inserts 202 have been introduced which splits the typical message structure into two. The messages pointed by free_node 104 up to the messages pointed by rdma_free_node 201 represents the messages that have been queued by the sender process, but are yet to be transferred to the remote host (server B) via RDMA. The messages from free node to data_anchor 103 are already in transit to (or have reached) the remote host, waiting to be acknowledged by the receiver process (on server B) thru the update of the data_anchor pointer.
Now, referring to Figure 3 a system for inter process communication between two server hosts with memory mapped files synchronized using RDMA is presented. The system 300, according to one of the preferred embodiments of the present invention comprises of:

• Physical Server 301 - This is the host on which the sender application process
runs.
Physical Server 302 - This is the host on which runs the receiver application runs.
• RDMA capable network interface card NIC on server 301 - This NIC is capable of executing RDMA commands from the local or remote host
• RDMA capable network interface card on server 302 - This NIC is capable of executing RDMA commands from the local or remote host
• Messaging Library- This library contains the message send and receives functions which are linked in & invoked by the sender and receiver application processes.
• Memory mapped file 303 on server 301 - This memory mapped file contains the FIFO queue which is used for sending and receiving messages. It is synchronized with server 302.
• Memory mapped file 304 on server 302 - This memory mapped file contains the FIFO queue which is used for sending and receiving messages. It is synchronized with server 301.
• Remote Sender process 305 running on server 301. This is the component which is responsible for batching incoming messages via RDMA. It will group all messages from free_node 104 to rdma_free_node 201 and issue RDMA work requests for the entire group of messages.
• Ethernet or Infiniband switch (optional) - The switch that connects servers 301 and 302.
Referring next to Figure 4, a design layout of Memory Mapped File is shown. The figure represents a separate section for buffer headers which point to the payload areas of the buffers. The headers are contiguously allocated in one memory area and so are the payload sections. There is also an area where the data common to the queue stored. Data anchor 103, free node pointer 104, number_of_inserts 105, number_of_deletes 106 are all example of such common queue data 401. Within the common queue data area 401. a structure which groups the free node pointer 104 and the data anchor pointer 103 is also presented.

Further, in accordance with one of the embodiments of the present invention, common data part of the memory mapped file contains two variables free_node 104 and number_of_inserts 105 that have been grouped together in a single structure to eliminate the latency reducing component. This helps send the entire structure in 1 RDMA write work request instead of sending them in separate work requests. This structure will now be known as node_counter structure 402.
The newly added variables and new meanings of modified variables are as follows:
• rdma_free_node - points to the message buffer in which the sender will insert the next new message.
• rdma_inserts - number of messages inserted by the sender process since the queue was created
• node_counter.free_node - points to the next message starting which the remote sender process will start batching messages in order to send the messages as part of one RDMA write work request.
• node_counter.number_of_inserts - The number of messages that have been updated to the remote host (via RDMA write work requests) ever since the creation of the queue.
The optimization achieved by introducing newly added variables for establishing low latency high message throughput is presented below wherein the number of RDMA work request have been reduced to two from the remote sender process 305.
1. Registering the local memory mapped file with remote host 302 and performing the following operations:
a. If the rdma_free_node equals free_node and rdmainserts equals
number_of_inserts keep checking; else proceed to next step
b. Node_var = free_node (Here Node_var refers to a variable used for iterating
to the nodes of messages that needs to be updated to the remote server)
c. Initialize message_group to null
d. Initialize group_size to 0

e. While node_var does not equal rdma_free_node
i. Add the message pointed to by node into the message_group ii. Increment group_size
f. Add group_size to node_counter.number_of_inserts
g. Update node_counter.free_node to point the the message buffer next to the
last message in message_group
h. Check status of previous RDMA write work requests and clear them if any
of them completed i. Issue 2 RDMA work requests
i. One for updating the "group_size" number of messages in "message_group" that was copied to the local memory mapped file on server A to the memory mapped file on server B ii. The next one for updating the node_counter structure.
There is only a minor change in the receiver process adopted, i.e. compressing 2 RDMA work requests into 1, just like in the remote sender process.
1. Register the local memory mapped file with remote host 301 and performing the following operations:
a. Check status of previous RDMA write work requests and clear them if any
of them completed
b. If node_counter.free_node equals data_anchor go to keep checking else
continue
c. Copy message from local memory mapped file to user buffer
d. Update the data anchor pointer to the next data buffer
e. Increment number_of_deletes counter
f. Issue 1 RDMA work request for updating the node_counter structure
The above adopted approach reduces the number of work requests in the remote sender process 305 to 2 by grouping the variables number of inserts and free_node pointer into a single structure. The deciding factor of performance for a system could be the maximum

number of work requests that can be executed per sec by the NIC supporting RDMA. With this in mind it should be ensured that the number of work requests per update is optimized. By batching the messages and grouping variables as in the previous optimization the number of work requests that go into one update has been reduced. However if the number of work requests are reduced from 2 to 1 then an increase in throughput can be expected further.
In every update issued by the remote sender process RS 305 there are two work requests. One work request points to the payload region in the area. The other work request points to the node_counter structure. These work requests cannot be combined because one work request can point only to a contiguous region of memory. To reduce the two work requests required per update to one request it is necessary to combine the two sets of data in a different way.
Figure 4 depicts an optimized memory layout wherein the node_counter structure 402 has been repeated at the end of the payload section of every message. Thus it is now possible to combine the message payload plus the node_counter structure into one work request as they are both in contiguous memory region.
BEST MODE/EXAMPLE OF WORKING OF THE INVENTION
The invention is described in the example given below which is provided only to illustrate the invention and therefore should not be construed to limit the scope of the invention.
Referring to Figure 4, it is assumed that the remote sender process 305 has batched 3 messages C, D and E and wants to update the remote host 302 end using RDMA. The memory region to be updated in the batch is marked in the referred figure. Note that this memory region will include the node_counter structure for messages B, C, D and E. Also important to note is that the only node_counter strtuctures that need to be updated are the ones attached to payloads of messages B and E. The reasoning for this is as follows:

• Node_counter structure attached to B. Prior to message C, D and E. the last message sent from the remote sender to the receiver was B. The node_counter structure of B was also updated as part of that last message. So the receiver will be checking the free_node pointer in the node_counter structure attached to B to determine if any new message has come.
• Node counter structure attached to E. Once the batch is updated to the remote and the messages C, D and E are read by the receiver, the node_counter structure attached to the payload of E is checked by the receiver to know that there are no further messages. Only the next batch update from the remote sender will update this node_counter structure in C to indicate that there are more messages inserted in the queue.
Thus the new optimized approach for the remote sender is as follows:
1. Registering the local memory mapped file with remote host 302 and perform the following operations:
a. If the rdma_free_node equals free_node and rdma_inserts equals
number_of_inserts keep checking else proceed to next step
b. Node_var = free_node
c. Prev_node = NULL
d. Initialize message_group to null
e. Initialize group_size to 0
f. While node_var does not equai rdma_free_node
i. Add the message pointed to by node into the
message_group ii. Increment group_size iii. Prev_node = node_var iv. Node_var = next node
g. Add group_size to number_of_inserts into the node_counter structure of
nodes pointed by last_message_node_counter_pointer and prevnode

h. Update free_node (in the node_counter structure of nodes pointed by
last_message_node_counter_pointer and prevjiode) to point the message
buffer next to the last message in message group i. last_message_node_counter_pointer = prev_node j. Check status of previous RDMA work requests and clear them if any of
them completed k. Issue 1 RDMA work request for the payload section of messages in
message_group.
Wherein the variables "last_sent_message_node_counter_pointer" of the first node in queue is introduced in the common queue data area. This variable points to the node_counter structure of the last message which was last sent to the remote server B. For the case of illustration it will point to the node_counter belonging message node A in the above figure. This is done during queue creation and similar is the case for data_anchor, rdma_free_node and node_counter.free_node, where they are made to point to message node A during queue creation as it was for the previous implementations. The counters node_counter.no_of_inserts in all the message nodes and no_of_deletes in the common data area are initialized to zero during queue creation as it was for the previous implementations. This initialization has not been explicitly mentioned before. It is being mentioned now for convenience.
The variable "last_received_message_node_counter_poiritef' of the first node in queue is introduced in the common queue data area. This variable point to the nodecounter structure of the last message which was last received the remote server A.
The optimized algorithm of the receiver process is as follows:
1. Register the local memory mapped file with remote host (server A) and performance the following operations:
a. Check status of previous RDMA work requests and clear them if any of them completed

b. If last received message__node_counter_pointer.free_node equals
data_anchor keep checking else continue
c. Copy message from local memory mapped file to user buffer
d. fast_received_message_node_counter__pointer = dataanchor
e. Update the data anchor pointer to the next data buffer
f. Increment number_of_deletes counter
g. Issue 1 RDMA work request
h. for updating the free_node pointer together with the number of inserts counter just updated in the local memory mapped file on server 301 to the memory mapped file on server 302, This has been achieved by combining both these variables into a single structure in C language.
The optimization done using the above adopted approach could achieve as much as 780000 messages per second with a one way latency not exceeding 55 microseconds.
The preceding description has been presented with reference to various embodiments of the invention. Persons skilled in the art and technology to which this invention pertains will appreciate that alterations and changes in the described structures and methods of operation can be practiced without meaningfully departing from the principle, spirit and scope of this invention.

We Claim:
1) A system for lockless remote messaging in an inter-process communication between at least two processes running on at least two nodes implemented by RDMA supported Network Interface Card configured to synchronize a memory mapped file hosted on each of the said node, the system comprising of:
a) a first host node communicatively coupled with a second host node for sending and receiving messages over a computing network;
b) RDMA supported Network Interface Card deployed on each of said host nodes for executing RDMA commands;
c) a storage hosted on each host node adapted to store inter process messages invoked by either of the communicatively coupled host nodes;
d) a first memory mapped file hosted on the first host node configured to synchronize static circular queue of messages with second memory mapped file hosted on the second host node; and
e) at least one remote sender process running on the remote sender first host node, asynchronously sending at least one batch of message along with corresponding RDMA work requests from the said queue to the second host node.
2) The system of claim 1, wherein the RDMA is supported on Ethernet using
iWARP to connect the first and the second host nodes.
3) The system of claim 1, wherein the RDMA supported NIC provides a direct
interface for direct memory access on a second or a remote node.
4) The system of claim 1, wherein the memory mapped file contains a static circular queue of messages between the first and second host nodes.
5) The system of claim 1, wherein the memory mapped file comprises of a sequence of payload section wherein each payload intermittently is followed by a

corresponding node counter element to constitute a contiguous memory region and coupled with a corresponding common queue data and continuously arranged headers; a rdma free pointing element to point to the message buffer in which the sending host node inserts new message; a rdma insertion counter to count number of messages inserted by a remote sender process; and a last message node pointing element to point to the last message sent from the remote sender process to the second host node.
6) The system of claim 5, wherein the common queue data comprises of data pointing element adapted to point to the next message to be received by the second host node; a free pointing element adapted to point to the message written by the first host node; an insertion counter to count number of messages sent by the first host node and a deletion counter to count the number of messages read by the second host node since queue initiation.
7) The system of claim 1, wherein the layout of memory mapped file is optimized to form a contiguous memory region adapted to batch the plurality of messages and group the constituting elements such that the number of RDMA work request is reduced to one.
8) The system of claim 1, wherein the remote sender process is configured to batch the message based upon an indication of the previous message received by the second host node.
9) The system of claim 1, wherein the remote sender process updates the node counter element for insertion of additional messages in the circular queue to make it accessible to the second host node.
10) A memory mapped structure operable by computer executable instructions, wherein the said memory mapped structure is configured to synchronize a static circular queue of messages between the sending and receiving host nodes, the memory mapped structure comprising:

a) plurality of messages bundled together to form at least one batch. each batch comprising of a sequence of payload section, each payload intermittently followed by a corresponding node counter element to constitute a contiguous memory region, wherein the payload section is further coupled with a corresponding common queue data and continuously arranged headers; the common queue data comprising of; a data pointing element adapted to point to the next message to be received by the receiving host node, a free pointing element adapted to point to the message written by the sending host node, an insertion counter to count number of messages sent by the sender host node, and a deletion counter to count the number of messages read by the receiving host node since queue initiation;
b) a rdma free pointing element adapted to point to a message buffer in which the sending host node inserts new message;
c) a rdma insertion counter to count number of messages inserted by a remote sender process in accordance with corresponding rdma work request; and
d) last message node pointing element to point to the last message sent from the remote sender process to the receiving host node.
11)The structure of claim 10, wherein the layout of memory mapped file is optimized to form the contiguous memory region adapted to batch the plurality of messages and group the constituting elements such that the number of RDMA work request is reduced to one.
12) The structure of claim 10, wherein the remote sender process is configured lo batch the message based upon an indication of the previous message received by the second host node.
13) The structure of claim 10, wherein the remote sender process updates the node counter element for insertion of additional messages in the circular queue to make it accessible to the second host node.

14) A method for lockless remote messaging in an inter-process communication between at least two processes running on at least one sending and one receiving host node implemented by RDMA supported Network Interface Card configured to synchronize a queue of messages via a memory mapped file hosted on each of the said node, the said method comprising:
a) initializing transfer of a first batch of message from a data buffer of the sender host node to the corresponding memory mapped file whenever the message queue is indicated of sufficient space;
b) updating a rdma free pointing element to initialize a last message node pointing element to point to a last message sent through a remote sender process;
c) checking iteratively status of a first rdma work request for the last message node pointing element being equal to a previous node; and
d) issuing a second RDMA work request for transmitting a contiguous payload associated therewith the first batch of message.

15) The method of claim 14, wherein the memory mapped file is priori registered with the sending and receiving host nodes.
16) The method of claim 14, wherein asynchronous updation of a free pointing element with an insertion counter is carried out at the sender and the receiver nodes by means of a combined structure variable.