FIELD OF THE TECHNOLOGY

The present invention relates to communication technology, especially relates to a method of snapshot management for persistent data.

BACKGROUND

In applications use data and metadata, where the metadata is kept in memory for faster access and the data is stored on the disk, there is a certain amount of metadata which is in memory but needs to be persisted to a permanent storage like a HDD/SSD to take care of scenarios like power failures, process crashesetc. The size of this metadata can be in GBs. Writing this to the disk is a time consuming task.

Existing Technology Case 1: Mechanisms like "mmap and msync" are used to save this metadata to a permanent storage. Taking snapshot using mmap: The mapped memory is synchronized to disk by the Linux OS but the time at which each modified memory page is synchronized to disk is decided by the OS. So there is a problem here if the information related to one particular file is spanning two memory pages, then if one page is synchronized and other is not synchronized and the process restarts, the data in the disk is inconsistent. So it cannot be used for recovery.

Hence the technology which taking snapshot using mmap and fork to save this metadata to a permanent storage appears. The advantage of this approach is that the data which is stored in the disk is consistent. Linux "fork" system call uses copy on write mechanisms to ensure that the memory does not increase significantly after the fork. But in this method, if metadata is called during the process of "save this metadata to a permanent storage", the process will be blocked. A US patent named "saving snapshot of knowledge base without blocking" provides the solution. In this case the user operations are not blocked but disk is the bottleneck hence other read or write operations to the disk is blocked or delayed.

SUMMARY A first aspect of the invention is a method of snapshot management for persistent data, comprising:

when a page is dirty or modified, marking the page using mprotect function in memory, wherein metadata is stored in the memory, then writing the marked page to a disk using mmap function and fork function.

A second aspect of the invention is a device of snapshot management for persistent data, comprising:
a marking module, configured to mark a page using mprotect function in memory when the page is dirty or modifiedDwherein metadata is stored in the memory;
a writing module, configured to write the marked page to a disk using mmap function and fork function.

A third aspect of the invention is a device of snapshot management for persistent data, comprising:
a processor and a memory coupled to the processor;
wherein the processor is configured to: mark a page using mprotect function in memory when the page is dirty or modified, wherein metadata is stored in the memory, then write the marked page to a disk using mmap function and fork function.

According to the invention, the mechanism uses Linux API "mprotect" to detect the dirty pages, and uses the efficient way of marking the page to dirty, so that only one page is changed and only the changed page is synchronized to disk. Compared to existing technology, the data that is synchronized every time to disk is smaller, less time is spent on synchronization of the dirty data.

BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWING(S)

Fig. 1 illustrates the process of snapshot management for persistent data.

Fig. 2 illustrates the process of marking dirty pages using mprotect.

Fig. 3a illustrates a schematic view of structure of a physical device for executing the procedure of a method provided by an embodiment of the invention.

Fig. 3b illustrates the process of taking snapshot which is based on marking dirty pages using mprotect.

Fig. 4 is a schematic view of structure of one example usage scenario of the invention.

Fig. 5 is a schematic view of structure of another example usage scenario of the invention.

Fig. 6 is a schematic view of structure of a device of snapshot management for persistent data provided by the present invention.

Fig. 7 is a schematic view of structure of the marking module.

Fig. 8 is another schematic view of structure of a device of snapshot management for persistent data provided by the present invention.

DETAILED DESCRIPTION

In order to make the objects, technical solutions and merits of the present invention clearer, a further detailed description of embodiments of the present invention is given by reference to accompanying drawings.

Fig. 1 illustrates the process of snapshot management for persistent data. As shown in Fig. 1, the method comprises:
Step101. When a page is dirty or modified, marking the page using mprotect function in memory, wherein metadata is stored in the memroy. The action related to marking dirty pages using mprotect in the memory.

The metadata is data about data. Generally, the metadata is kept in the memory for faster access, and the data is usually stored on the disk. There is a certain amount of metadata which is in the memory but needs to be persisted to a permanent storage like a HDD/SSD to take care of scenarios like power failures, process crashes etc. SO at scenario of the invention, the metadata that need to be wrote to the disk is the persistent data.

mprotect function is a Linux API function, configured to modify the protection mode of memory image, for details please refer to the prior art. In the present embodiment, when a page is dirty or modified, the protection mode of the area of the memory where the dirty page occupies is modified by mprotect function, such that to mark the page which is dirty. It should be noted that, the memory where store the metadata also can be called the region of the metadata. step 102. writing the marked page to a disk using mmap function and fork function.

The present embodiment uses mmap function and fork function in order to increase the efficiency of write operations. For details of mmap function and fork function, please refer to the prior art. For application of mmap function and fork function in the present embodiment, please refer to the description of embodiment shown in Fig. 3.

As shown in Fig. 2, the process of stepl0l comprises:
step 1011. protecting the memory using mprotect function, and creating a bitmap for the page which is dirty in the memory. The process protects the memory using mprotect. A bitmap for dirty pages is created in the meimory by the process. Further, the process registers a signal handler to handle signal SIGSEGV. Ptr is pointing to any address in the memory.

Stepl012. checking whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, If so, marking the page corresponding to the address as dirty in the bitmap and unprotecting the page corresponding to the address. The process initializes ptr as aD*Ptr=aD. Since the memory is protected, OS will give a signal SIGSEGV to the process. When the signal SIGSEGV is given, by the signal handler handling the signal SIGSEGV, such that to check whether the address, which the ptr is pointing to, is for the area which is protected in the memory. If so, the process marks the page as dirty in the bitmap and will unprotect that page. Now the dirty page is identified, and this page will be synchronized next time.

The procedure of the method provided by the present embodiment can be executed by a physical device, the physical device can be a computer, a server, and a terminal device etc. The terminal device can be a smartphone. Fig. 3a illustrates a schematic view of structure of a physical device for executing the procedure of a method provided by an embodiment of the invention. As shown in Fig. 3 a, the hardware layer of the physical device comprises CPU, memory and disk etc, and further comprises input/output device, memory controller, network interface etc; the input device can comprise keyboard, mouse and touch screen etc; the output device can comprise display device such as LCD, CRT, Holographic and Projector etc. An operating system such as Linux and other application programs can be operated above the hardware layer. The application program comprises father process and son process which is configured to carry out the method provided by the embodiment of the invention. In addition, the physical device further comprises a driver layer. The driver layer can comprise CPU driver, display controller driver and network driver etc.

Take Fig. 3a as an example, Fig. 3b illustrates the process of taking snapshot which is based on marking dirty pages using mprotect function. As shown in Fig. 3 b, the process comprises:

step301. a process calls fork function at the first time Tl, where the process can be called father process.
step302. OS creates a new process. The new process can be called the first son process.
step303. the first son process accesses memory and writes all the mapped data to disk.
step304. the first son process exits.
step305. the father process modifies metadata in the memory.
step306. the father process marks dirty pages using mprotect in the memory.

Specifically, the father process protects the memory using mprotect function, and creates a bitmap for the whole memory to mark dirty pages. The father process registers a signal handler to handle signal SIGSEGV. Then initialize ptr=a. After OS has given a signal SIGSEGV to the father process, the signal handler checks whether the address, which ptr is pointing to, is for the area which is protected in the memory. If so, the father process marks the page as dirty in the bitmap and will unprotect that page. Now the dirty page is identified.
step307. the father process calls fork function at the first time T2.
step308. OS creates a new process. The new process can be called the second son process. step309. the second son process accesses the bitmap to find the dirty pages, and writes the dirty pages to disk using mmap function.

Above illustrates creating son process by calling the fork function, in order to access the bitmap to find the marked page, then calls the mmap function one or more times to write the marked page to the disk.
Further, the second son process can call the mmap function one or more times to write the marked page to the disk using disk 10 operations.
step310. the second son process exits.

According to the invention, the mechanism uses Linux API "mprotect" to detect the dirty pages, and uses the efficient way of marking the page to be dirty, so that only one page is changed and only the changed page is synchronized to disk. Compared to existing technology, the data that is synchronized every time to the disk is smaller, less time is spent on synchronization of the dirty data. Further, the method can also accelerate the response to the user by copy-on-write technique. And the invention can also reduce the effect of other read or write operations to the disk, that is to say, time delay of the disk read request is less when the synchronization is going on, and consistency of the disk read request can be ensured. Time delay of the metadata request coming to cache is less when the synchronization is going on in the invention, and the invention can also improve the 10 efficient as only dirty pages are synchronized.

For example, the metadata for typical cache is 150 bytes. In a cache storing 10M records, the metadata that needs to be synchronized is 10M * 150 bytes = ~1.5 GB. Typical HDD performance is 0.2 MB/s (Direct 10 without disk cache), so time taken to store 1.5GB is 7500s, about2hrs (Assuming no sharing of disk). During this time, the disk will be shared for other read/writes which will again increase the time for persistence. According to the invention, typical synchronization time is 30 minutes. Average metadata changes is 2500/sec (assuming 2500 injects/sec into the cache), in a typical synchronization time worst case, the change will be 2500*30*60 = 4.5M records. Thus it can be seen, this is a saving of 55% time.

One example usage scenario of the invention, please refer to Fig. 4: Assuming the cache is distributed cache, the distributed cache use this methodology to take the snapshot of the persistent data of MDS and OSD for periodic backup.

Another example usage scenario of the invention, please refer to Fig. 5: In memory database is a common use case where the application will use this database for the speedup of the processing. In these kinds of applications there is a huge demand for identifying the changed records in a database and just synchronization only the changes to the disk.

Fig. 6 is a schematic view of structure of a device of snapshot management for persistent data provided by the present invention. As shown in Fig. 6, the device comprises: a marking module 61 and a writing module 62.

The marking module 61, configured to mark a page using mprotect function in memory when the page is dirty or modified, wherein metadata is stored in the memory;
the writing module 62, configured to write the marked page to a disk using mmap function and fork function.

Further, as shown in Fig. 7, an implementation of the marking module 61 comprises: a creating unit 611 and a marking unit 612.
The creating module 611, configured to protect the memory using mprotect function, and creat a bitmap for the page which is dirty in the memory;

The marking module 612, configured to check whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, If so, marking the page corresponding to the address as dirty in the bitmap and unprotecting the page corresponding to the address.

Further, as shown in Fig. 7, the marking module 61 also comprises: registering unit 613. The registering unit 613, configured to register a signal handler to handle a signal SIGSEGV when the creating unit 611 protecting the memory using mprotect function, and creating a bitmap for the page which is dirty in the memory.

On the basis of this, the marking unit 612 is specifically configured hat when the signal SIGSEGV is given, by the signal handler handling the signal SIGSEGV, such that to check the address, which the pointer of the memory is pointing to, is for the area which is protected in the memory, if so, marking the page corresponding to the address as dirty in the bitmap and unprotecting the page corresponding to the address.

In an alternative implementing embodiment, the writing module 62 is specifically configured to create son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then call the mmap function one or more times to write the marked page to the disk.

Further, the writing module 62 is specifically configured to create a son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then calling the mmap function one or more times to write the marked page to the disk using disk 10 operations.

According to the invention, the device uses Linux API "mprotect" to detect the dirty pages, and uses the efficient way of marking the page to be dirty, so that only one page is changed and only the changed page is synchronized to disk. Compared to existing technology, the data that is synchronized every time to the disk is smaller, less time is spent on synchronization of the dirty data. Further, the device can also accelerate the response to the user by copy-on-write technique. And the device can also reduce the effect of other read or write operations to the disk, that is to say, time delay of the disk read request is less when the synchronization is going on, and consistency of the disk read request can be ensured. Time delay of the metadata request coming to cache is less when the synchronization is going on in the invention, and the device can also improve the 10 efficient as only dirty pages are synchronized.

Fig. 8 is another schematic view of structure of a device of snapshot management for persistent data provided by the present invention. As shown in Fig. 8, the device comprises: a processor 82 and a memory 81 coupled to the processor 82.

The memory 81, configured to store program. Specifically, the program can includes program code, the program code comprises computer operating instruction.

The processor 82, configured to executive the program in order to mark a page using mprotect function in memory when the page is dirty or modified, wherein metadata is stored in the memory, then write the marked page to a disk using mmap function and fork function.

The memory 81 includes High speed RAM and non-volatile memory.
the processor 82 can be a Central Processing Unit (CPU), or can be Application Specific Integrated Circuit (ASIC), or can be configured to one or more ASIC.

Further, wherein in the step of marking the page using mprotect function in memory when a page is dirty or modified, the processor is specifically configured to: protect the memory using mprotect function, and creat a bitmap for the page which is dirty in the memory; check whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, If so, mark the page corresponding to the address as dirty in the bitmap and unprotect the page corresponding to the address.

Further, wherein the processor 82 is further configured to: register a signal handler to handle a signal SIGSEGV when protecting the memory using mprotect function, and creating a bitmap for the page which is dirty in the memory.

In the step of checking whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, the processor 82 is further configured that: when the signal SIGSEGV is given, by the signal handler handling the signal SIGSEGV, such that to check the address, which the pointer of the memory is pointing to, is for the area which is protected in the memory.

On the basis of this, wherein in the step of writing the marked page to a disk using mmap function and fork function, the processor 82 is specifically configured to: create a son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then calling the mmap function one or more times to write the marked page to the disk.

Further, wherein in the step of calling the mmap function one or more times to write the marked page to the disk, the processor 82 is specifically configured to: call the mmap function one or more times to write the marked page to the disk using disk 10 operations. Further, as shown in Fig. 8, the device also comprises: communication interface 83, configured to complete the communication between the device and other devices.

As shown in Fig. 8, the device also comprises: disk 84. the disk 84, configured to store the marked page.

Alternatively, in specific implementation, if the memory 81, the processor 82 the communication interface 83 and the disk 84 can be implemented individually, then the memory 81, the processor 82 the communication interface 83 and the disk 84 can be in communication connection via BUS. The BUS can be Industry Standard Architecture (ISA) BUS, Peripheral Component (PCI) BUS or Extended Industry Standard Architecture (EISA) BUS etc. The BUS can be divided into address BUS, data BUS and control BUS etc. For convenient representation, the BUS is only represented by a single thick line, but does not mean there is only one BUS or one kind of BUS.

Alternatively, in specific implementation, if the memory 81, the processor 82 the communication interface 83 and the disk 84 can be integrated in a single chip, then the memory 81, the processor 82 the communication interface 83 and the disk 84 can be in communication connection via internal interface.

According to the invention, the device uses Linux API "mprotect" to detect the dirty pages, and uses the efficient way of marking the page to be dirty, so that only one page is changed and only the changed page is synchronized to a disk. Compared to existing technology, the data that is synchronized every time to the disk is smaller, less time is spent on synchronization of the dirty data. Further, the device can also accelerate the response to the user by copy-on-write technique. And the device can also reduce the effect of other read or write operations to the disk, that is to say, time delay of the disk read request is less when the synchronization is going on, and consistency of the disk read request can be ensured. Time delay of the metadata request coming to cache is less when the synchronization is going on in the invention, and the device can also improve the 10 efficient as only dirty pages are synchronized.

Many variations of the above described embodiments will be available to one skilled in the art without deviating from the essence of the invention as set out herein and in the claims. In particular, some operations could be reordered, combined, or interleaved, or executed in parallel, and many of the data structures could be implemented differently. When one element, step, or object is specified, in many cases several elements, steps, or objects could equivalently occur. Steps in flowcharts could be implemented, e.g., as state machine states, logic circuits, or optics in hardware components, as instructions, subprograms, or processes executed by a processor, or as a combination of these and other techniques.

It is to be understood that the aspects and embodiments of the invention described herein may be used in any combination with each other. Several of the aspects and embodiments may be combinedtogether to form a further embodiment of the invention, and not all features, elements, or characteristics of an embodiment necessarily appear in other embodiments. A method, a computer, or a computer program product which is an aspect of the invention may comprise any number of the embodiments or elements of the invention described herein. Separate references to "an embodiment" or "one embodiment" refer to particular embodiments or classes of embodiments (possibly different embodiments in each case), not necessarily all possible embodiments of the invention.

WE CLAIM :

1. A method of snapshot management for persistent data, comprising:
when a page is dirty or modified, marking the page using mprotect function in a memory, wherein metadata is stored in the memory, then writing the marked page to a disk using mmap function and fork function.

2. The method as claimed in claim 1, wherein the process of marking the page using
mprotect in the memory comprises:
protecting the memory using mprotect function, and creating a bitmap for the page which is dirty in the memory;
checking whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, if so, marking the page corresponding to the address as dirty in the bitmap and unprotecting the page corresponding to the address.

3. The method as claimed in claim 2, further comprising:
registering a signal handler to handle a signal SIGSEGV when protecting the memory using mprotect function, and creating a bitmap for the page which is dirty in the memory,;
wherein the process of checking whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory comprises:
when the signal SIGSEGV is given, by the signal handler handling the signal SIGSEGV, such that to check whether the address, which the pointer of the memory is pointing to, is for the area which is protected in the memory.

4. The method as claimed in claim 2 or 3, wherein the process of writing the marked
page to a disk using mmap function and fork function comprises:
creating a son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then calling the mmap function one or more times to write the marked page to the disk.

5. The method as claimed in claim 4, wherein the process of calling the mmap function
one or more times to write the marked page to the disk comprises:
calling the mmap function one or more times to write the marked page to the disk using disk 10 operations.

6. A device of snapshot management for persistent data, comprising:
a marking module, configured to mark a page using mprotect function in memory when the page is dirty or modified wherein metadata is stored in the memory;
a writing module, configured to write the marked page to a disk using mmap function and fork function.

7. The device as claimed in Claim 6, characterized in that, the marking module
comprises:
a creating unit, configured to protect the memory using mprotect function, and creat a bitmap for the page which is dirty in the memory;
a marking unit, configured to check whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, If so, marking the page corresponding to the address as dirty in the bitmap and unprotecting the page corresponding to the address.

8. The device as claimed in Claim 7, characterized in that, the marking module further
comprises:
a registering unit, configured to register a signal handler to handle a signal SIGSEGV when the creating unit protecting the memory using mprotect function, and creating a bitmap for the page which is dirty in the memory;
the marking unit is specifically configured that when the signal SIGSEGV is given, by the signal handler handling the signal SIGSEGV, such that to check the address, which the pointer of the memory is pointing to, is for the area which is protected in the memory, if so, marking the page corresponding to the address as dirty in the bitmap and unprotecting the page corresponding to the address.

9. The device as claimed in Claim 7 or 8, characterized in that, the writing module is specifically configured to create son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then calling the mmap function one or more times to write the marked page to the disk.

10. The device as claimed in Claim 9, characterized in that, the writing module is specifically configured to create a son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then calling the map function one or more times to write the marked page to the disk using disk 10 operations.

11. A device of snapshot management for persistent data, comprising:
a processor and a memory coupled to the processor;
wherein the processor is configured to: mark a page using mprotect function in memory when the page is dirty or modified, wherein metadata is stored in the memory, then write the marked page to a disk using mmap function and fork function.

12. The device as claimed in Claim 11, wherein in the step of marking the page using mprotect function in memory when a page is dirty or modified, the processor is specifically configured to:
protect the memory using mprotect function, and creat a bitmap for the page which is dirty in the memory;
check whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, If so, mark the page corresponding to the address as dirty in the bitmap and unprotect the page corresponding to the address.

13. The device as claimed in Claim 12, wherein the processor is further configured to:
register a signal handler to handle a signal SIGSEGV when protecting the memory using
mprotect function, and creating a bitmap for the page which is dirty in the memory;
in the step of checking whether an address, which a pointer of the memory is pointing to, corresponds to an area which is protected in the memory, the processor is further configured that: when the signal SIGSEGV is given, by the signal handler handling the signal SIGSEGV, such that to check the address, which the pointer of the memory is pointing to, is for the area which is protected in the memory.

14. The device as claimed in Claim 12 or 13, wherein in the step of writing the marked page to a disk using mmap function and fork function, the processor is specifically configured to: create a son process by calling the fork function in order to access the bitmap to find the marked page, wherein the bitmap is created for the page which is dirty in the memory, then calling the mmap function one or more times to write the marked page to the disk.

15. The device as claimed in Claim 14, wherein in the step of calling the mmap function one or more times to write the marked page to the disk, the processor is specifically configured to: call the mmap function one or more times to write the marked page to the disk using disk 10 operations.