DESCRIPTION
! DISTRIBUTED STORAGE SYSTEM, CONTROL APPARATUS, CLIENT
TERMINAL, LOAD BALANCING METHOD AND PROGRAM
5
TECHNICAL FIELD
!I [OOOl]
i
(DESCRIPTION OF RELATED APPLICATION)
The p r e s e n t a p p l i c a t i o n claims p r i o r i t y from Japanese Patent
10 Application No. 2012-079952 ( f i l e d on March 3 0 , 2012), t h e content of
which is hereby incorporated i n its entirety by reference i n t o t h i s
disclosure.
[0002]
The present invention relates to a d i s t r i b u t e d s t o r a g e system, a
I
I 15 c o n t r o l a p p a r a t u s , a client t e r m i n a l , a load balancing method and a
program and, more p a r t i c u l a r l y , to a d i s t r i b u t e d storage system, a
c o n t r o l a p p a r a t u s , a client t e r m i n a l , a load balancing method and a
program, in which key-value type d a t a a r e supervised.
BACKGROUND
20 [0003]
I n P a t e n t Literature 1, there is disclosed a network system i n
which t h e load imposed o n a Web server(s) within a site is alleviated by
a load balancer to enable high-speed path switching even on occurrence
of f a i l u r e s o n a path g e t t i n g to the load b a l a n c e r . According to Patent
25 Literature 1, when a router disposed halfway between a c l i e n t terminal
and a s e r v e r has received a packet from the client t e r m i n a l , t h e router
decides on a forwarding destination server based on a d e s t i n a t i o n L4
port number as well as a hash value of a source address contained in a
packet header. By so doing, it is possible t o a l l o c a t e processing to
30 servers that d i f f e r from one s e r v i c e recognizable on t h e l a y e r 4 (L4) to
another.
[0004]
I n P a t e n t Literature 2, there is disclosed a load balancer which
will prevent d e t e r i o r a t i o n of a response c h a r a c t e r i s t i c t o t h e c l i e n t
terminal as uniqueness is positively maintained. In this Patent
L i t e r a t u r e , a r e l a y apparatus(es) arranged between the load balancer and
t h e s e r v e r references a response from t h e s e r v e r t o the client terminal to
post-write o r add a path identifier ( a n i d e n t i f i e r a l l o c a t e d from one
r e l a y apparatus to another) in L7 d a t a . When the c l i e n t terminal
re-sends the request, the path i d e n t i f i e r , received l a s t t i m e , is embedded
in t h e r e q u e s t . The l o a d balancer t h a t has received the packet transfers
i t t o a n appropriate o n e o f the relay apparatuses.
[OOOS]
In Patent Literature 3, there is disclosed a computer system i n
which t h e processing of deciding a forwarding d e s t i n a t i o n is
divisionally taken charge of by a p l u r a l i t y of apparatuses so as to
reduce the s t o r a g e capacity or t h e amount o f c a l c u l a t i o n s of a URL
(Uniform Resource Locator) s w i t c h .
[0006]
In Non-Patent L i t e r a t u r e s 1 and 2, there is disclosed a c e n t r a l i z e d
c o n t r o l type network c a l l e d an OpenFlow exploiting a c e n t r a l i z e d
c o n t r o l configuration network a r c h i t e c t u r e in which a control apparatus
termed an OpenFlow c o n t r o l l e r c o n t r o l s the behavior of a switch termcd
an OpenFlow s w i t c h . The behavior of a number of t h e s e switches,
making up a v i r t u a l network, can be f l e x i b l y modified in keeping with
t h e s t a t u s of the entire network. Thus, with the use of the OpenFlow, a
highly f l e x i b l e v i r t u a l network can be constructed with much ease
CITATION LIST
PATENT LITERATURE
[0007]
P a t e n t L i t e r a t u r e 1:
JP P a t e n t Kokai Publication No. JP2003-131961A
P a t e n t L i t e r a t u r e 2:
30 JP Patent Kokai Publication No. JP2011-041006A
Patent L i t e r a t u r e 3 :
JP P a t e n t Kokai Publication No. JP2006-309383A
NON-PATENT LITERATURE
[OOOS]
Non-Patent L i t e r a t u r e 1:
Niclc McKeown and seven o t h e r s : "OpenFlow: Enabling Innovation i n
Campus Networks," [ o n l i n e ] , [searched on February 14, 20121, Internet
.
1
! 5 Non-Patent L i t e r a t u r e 2:
"OpenFlow Switch Specification, Version 1.1 .O, Implemented (Wire
Protocol 0x02)," [ o n l i n e ] , [searched on February 14, 20121, Internet
.
SUMMARY
I 10 TECHNICAL PROBLEM
[0009]
The following a n a l y s i s is given by the p r e s e n t i n v e n t i o n . I n a
d i s t r i b u t e d s t o r a g e system that supervises key-value configuration data,
it is difficult to accomplish s e l e c t i n g t h e forwarding d e s t i n a t i o n server
15 by a s i n g l e s e l e c t i o n apparatus and decreasing delay in request
I processing in combination. Selecting t h e forwarding d e s t i n a t i o n
i server by a s i n g l e s e l e c t i o n device means carrying o u t the processing of
selecting, at a s i n g l e node, to which s e r v e r s the total of the requests
from the c l i e n t t e r m i n a l is to be t r a n s f e r r e d . I f such processing is
c a r r i e d o u t by a p l u r a l i t y of nodes, and a server(s) is added or removed,
it is necessary to use a proper scheme to maintain the uniqueness of the
s e r v e r information from o n e node to another, thus complicating the
processing.
[OO l o ]
On the o t h e r hand, i f t h e s e l e c t i o n of t h e forwarding destination
server is made by a s i n g l e node, it becomes necessary f o r the single
selecting node t o r e c e i v e t h e t o t a l o f the requests from t h e t o t a l of the
c l i e n t terminals and to select the forwarding d e s t i n a t i o n s e r v e r to
perform t h e forwarding processing. This may l e a d t o a n increased load
on t h e s i n g l e node t o aggravate t h e delay i n r e q u e s t processing.
[OOll]
In the network system of P a t e n t L i t e r a t u r e 1, if the r e q u e s t s from
a s p e c i f i e d client terminal to the same service are made f r e q u e n t l y , t h e
r e q u e s t s a r e forwarded to t h e same server, so that load balancing may
not be sufficient.
[OO 121
In P a t e n t L i t e r a t u r e 2, it is again necessary f o r t h e r e l a y
apparatus to reference t h e t o t a l of t h e responses from the server to the
5 c l i e n t t e r m i n a l , t h u s possibly increasing t h e processing delay.
[00 131
I n P a t e n t Literature3, t h e forwarding d e s t i n a t i o n is decided using
a forwarding d e s t i n a t i o n s e l e c t i o n t a b l e t h a t coordinates t h e hash value
of a f i x e d l e n g t h , c a l c u l a t e d from the identifier of contents requested
10 from the client t e r m i n a l , with the forwarding d e s t i n a t i o n . If the
r e q u e s t s a r e concentrated on s p e c i f i e d c o n t e n t s , t h e same s e r v e r is
n e c e s s a r i l y s e l e c t e d , so that load balancing may s i m i l a r l y n o t be
maintained.
[0014]
15 It is therefore an object of t h e p r e s e n t invention to provide a
d i s t r i b u t e d s t o r a g e system, a c o n t r o l a p p a r a t u s , a client terminal, a load
I
I d i s t r i b u t i o n method and a program which w i l l contribute to
accomplishing s e l e c t i o n of the forwarding d e s t i n a t i o n s e r v e r s b y a
s i n g l e s e l e c t i o n apparatus and reduction in the delay in t h e r e q u e s t
20 processing i n combination.
SOLUTION TO PROBLEM
[00 151
According to a f i r s t a s p e c t of the p r e s e n t i n v e n t i o n , t h e r e is
provided a d i s t r i b u t e d s t o r a g e system. The d i s t r i b u t e d s t o r a g e system
25 comprises: a p l u r a l i t y of servers that s t o r e d a t a t h a t is associated with
key information, r e s p e c t i v e l y ; a packet forwarding apparatus t h a t , o n
receipt of a new packet t h a t contains t h e key information and i s
addressed t o o n e of the plurality of s e r v e r s , r e q u e s t s a c o n t r o l apparatus
to decide a forwarding destination from among t h e p l u r a l i t y of servers;
30 and t h e c o n t r o l a p p a r a t u s . The c o n t r o l apparatus comprises: a
forwarding d e s t i n a t i o n s e l e c t i o n s e c t i o n t h a t decides a forwarding
destination of the paclcet based on key information in t h e p a c k e t ; and an
e n t r y s e t t i n g s e c t i o n t h a t sets, in a paclcet forwarding apparatus(es) on a
path to t h e forwarding d e s t i n a t i o n , a flow e n t r y f o r forwarding a
subsequent packet(s) with same key information to t h e forwarding
destination. The packet forwarding apparatus(es) forwards a packet(s)
with t h e same key i n f o r m a t i o n t o t h e forwarding destination using the
I set flow entry.
5 [0016]
According to a second a s p e c t of the p r e s e n t i n v e n t i o n , there is
I
'I provided a c o n t r o l apparatus connected to a p l u r a l i t y of servers and a
!
packet forwarding apparatus. The plurality o f s e r v e r s stores data that
!
! is associated with key information, r e s p e c t i v e l y . The packet
:! 10 forwarding apparatus r e q u e s t s t h e c o n t r o l apparatus to decide a
forwarding d e s t i n a t i o n from among the plurality o f s e r v e r s on receipt of
a new packet t h a t contains t h e key information and is addressed t o o n e
of t h e p l u r a l i t y of servers. The c o n t r o l apparatus comprises: a
forwarding destination selection section that decides a forwarding
15 destination of the packet based on key information in the packet; and an
I
I
'1 e n t r y s e t t i n g s e c t i o n t h a t s e t s , i n a packet forwarding apparatus(es) on a I path to the forwarding d e s t i n a t i o n , a flow entry for forwarding a
subsequent packet(s) with same key information to t h e forwarding
destination.
[00 171
According to a third aspect of t h e p r e s e n t i n v e n t i o n , t h e r e is
provided a c l i e n t t e r m i n a l . The c l i e n t terminal comprises a n address
setting section t h a t generates a r e q u e s t packet with a d e s t i n a t i o n
address generated using a hashed value of key i n f o r m a t i o n . The client
terminal r e q u e s t s a s e r v e r using t h e r e q u e s t p a c k e t .
[0018]
According to a fourth aspect of t h e p r e s e n t i n v e n t i o n , t h e r e is
provided a load balancing method. The load balancing method
comprises: by a control a p p a r a t u s , receiving from a packet forwarding
apparatus a r e q u e s t f o r deciding a forwarding d e s t i n a t i o n o f a new
packet with k e y i n f o r m a t i o n paired to a value(s); deciding a forwarding
destination of the packet from among a p l u r a l i t y of servers based on the
key information in the packet; and s e t t i n g , i n a packet forwarding
apparatus(es) on a path to the forwarding d e s t i n a t i o n , a flow e n t r y f o r
forwarding a subsequent packet(s) with same key information to the
forwarding d e s t i n a t i o n . The p r e s e n t invention is bound up with a
specified machine which is a computer t h a t p r e s e n t s i t e m s t o a user and
t h a t accepts s e l e c t i o n o f the items from the user.
5 [0019]
i I According to a f i f t h aspect of t h e p r e s e n t i n v e n t i o n , t h e r e is
provided a program for a computer loaded on a control apparatus t h a t is
connected to a p l u r a l i t y of servers and a packet forwarding a p p a r a t u s .
The plurality of servers storing data that is associated with key
10 information, r e s p e c t i v e l y . The packet forwarding apparatus r e q u e s t s
the c o n t r o l apparatus to decide a forwarding destination from among the
plurality of the s e r v e r s o n receipt of a new packet t h a t contains the key
information and is addressed t o o n e o f the plurality of servers. The
program causes the computer to execute: deciding a forwarding
15 destination of the packet from among a p l u r a l i t y o f s e r v e r s based on the
key information in t h e p a c k e t ; and s e t t i n g , i n a packet forwarding
apparatus(es) on a path to the forwarding d e s t i n a t i o n , a flow entry for
forwarding a subsequent packet(s) with same key information to the
forwarding d e s t i n a t i o n . It is noted that the program can be recoded cn
20 a computer-readable, that i s , n o n - t r a n s i e n t , recording medium. That i s ,
the present invention may be implemented as a computer program
product.
ADVANTAGEOUS EFFECTS OF INVENTION
[0020]
2 5 According to the p r e s e n t i n v e n t i o n , it is possible, in a d i s t r i b u t e d
s t o r a g e system s u p e r v i s i n g k e y - v a l u e type data, to contribute to
accomplishing selection of a forwarding destination server by a s i n g l e
selection node a n d reduction of t h e delay in r e q u e s t processing at the
same time.
30 BRIEF DESCRIPTION OF THE DRAWINGS
[0021]
Fig. 1 is a block diagram showing a configuration of an exemplary
embodiment of the p r e s e n t i n v e n t i o n .
F i g . 2 is a block diagram showing a configuration of a d i s t r i b u t e d
s t o r a g e system according to a f i r s t exemplary embodiment of t h e p r e s e n t
invention.
F i g . 3 shows a n example o f a s e r v e r table retained by a c o n t r o l
apparatus according to the f i r s t exemplary embodiment of t h e p r e s e n t
5 invention.
Fig. 4 is a flowchart for i l l u s t r a t i n g an operation ( a t the t i m e o f a w r i t e
request) of the f i r s t exemplary embodiment of the present invention.
Fig. 5 is a f l o w c h a r t f o r i l l u s t r a t i n g an operation ( a t t h e time of a
response from t h e s e r v e r ) of the f i r s t exemplary embodiment of the
10 p r e s e n t i n v e n t i o n .
Fig. 6 is a block diagram showing a configuration of a d i s t r i b u t e d
s t o r a g e system according to a second exemplary embodiment of the
p r e s e n t i n v e n t i o n .
Fig. 7 is a schematic view f o r i l l u s t r a t i n g a method for generating a
15 dummy address according to t h e second exemplary embodiment of the
p r e s e n t i n v e n t i o n .
!
I Fig. 8 shows a n example o f a s e r v e r table retained by a c o n t r o l
apparatus of the second exemplary embodiment of the p r e s e n t i n v e n t i o n .
Fig. 9 is a flowchart for i l l u s t r a t i n g an operation (at the t i m e o f a w r i t e
20 request) of the second exemplary embo'diment of the present invention.
Fig. 10 is a flowchart for i l l u s t r a t i n g a n o p e r a t i o n ( a t t h e t i m e o f
selection of the forwarding d e s t i n a t i o n s e r v e r ) of t h e second exemplary
embodiment of the present invention.
Fig. 11 is a flowchart for i l l u s t r a t i n g an operation (at the time of a
25 response from t h e s e r v e r ) o f t h e second exemplary embodiment of the
p r e s e n t i n v e n t i o n .
Fig. 12 i s a block diagram showing a configuration of a d i s t r i b u t e d
s t o r a g e system according to a third exemplary embodiment of the
p r e s e n t i n v e n t i o n .
30 Fig. 13 is a flowchart for i l l u s t r a t i n g an operation (at the time of a w r i t e
request) of the third exemplary embodiment of the present invention.
Fig. 14 is a flowchart for i l l u s t r a t i n g an operation ( a t t h e time of a
response from t h e server) of t h e t h i r d exemplary embodiment of the
p r e s e n t invention
DESCRIPTION OF EMBODIMENTS
[0022]
Initially, a summary of an exemplary embodiment of the present
invention w i l l be described with reference to t h e drawings. It i s noted
that the symbols used in the summary for referring to the drawings a r e
merely f o r a s s i s t i n g i n understanding and a r e n o t intended to restrict
the p r e s e n t invention to the modes i l l u s t r a t e d .
LO0231
With reference to Fig. 1 , the present invention a c c o r d i n g t o a n
exemplary embodiment may be implemented by a configuration
comprising: a p l u r a l i t y o f servers 1 that s t o r e d a t a that is associated
with key information, r e s p e c t i v e l y ; a packet forwarding apparatus 2A
that, on receipt of a new packet that contains the key information and is
addressed t o o n e o f the plurality of s e r v e r s , r e q u e s t s a c o n t r o l apparatus
3A t o decide a forwarding destination from among the plurality of
servers, and the control apparatus 3A.
[0024]
More specifically, the c o n t r o l apparatus 3A comprises: a
forwarding destination s e l e c t i o n s e c t i o n 33A t h a t decides a forwarding
destination of the packet based on key information in the packet; and an
e n t r y s e t t i n g section 34A that s e t s , i n a packet forwarding apparatus(es)
2A on a path to t h e forwarding d e s t i n a t i o n , a flow e n t r y f o r forwarding
a subsequent packet(s) with same key information to t h e forwarding
destination. The packet forwarding apparatus(es) 2A forwards a
packetcs) with t h e same key information to the forwarding d e s t i n a t i o n
using the set flow entry.
[0025]
4 s described above, i t is possible to s e l e c t t h e forwarding
d e s t i n a t i o n s e r v e r by a s i n g l e s e l e c t i o n node ( t h e above mentioned
c o n t r o l apparatus 3A) and to decrease the delay i n r e q u e s t processing
(realized by the above f l o w entry) in combination.
[0026]
[ F i r s t Exemplary Embodiment]
A f i r s t exemplary embodiment according to the present invention will
now be d e t a i l e d with reference to the drawings. F i g . 2 depicts a block
diagram showing a configuration including a d i s t r i b u t e d s t o r a g e system
according to the f i r s t exemplary embodiment of the present invention.
Referring to Fig. 2, there is shown a configuration including a s e r v e r 1
5 t h a t s t o r e s d a t a , a switch 2 t h a t forwards a packet (equivalent to the
above mentioned packet forwarding a p p a r a t u s ) , a c o n t r o l l e r 3 that
I
1 controls the switch 2 and a c l i e n t terminal 4 t h a t s e n d s o u t a r e q u e s t to
the server. It is noted t h a t , in actuality, one or more servers and one or
more c l i e n t terminals 4 a r e connected to the switch 2 and o n e o r more
10 switches 2 a r e connected to t h e c o n t r o l l e r 3 .
[0027]
The switch 2 includes a f l o w t a b l e 21, a packet forwarding
section 22 and a c o n t r o l l e r connection section 23.
[0028]
15 The flow table 21 is comprised of entries stating packet
,I
forwarding r u l e s . Each entry is made u p o f a r u l e p a r t that stores
i match conditions, and a processing p a r t that s t o r e s processing contents
applied to a packet that matches the match conditions. I n t h e rule
s e c t i o n , t h e r e may be s t a t e d , as match c o n d i t i o n s , a s w i t c h p o r t numbcr
20 received, a destination a d d r e s s , a source a d d r e s s , a V-LAN-ID, an L4
source port number, an L4 d e s t i n a t i o n p o r t number or otherwise. In
the processing p a r t , the forwarding switch port number, multicast and so
forth may be s t a t e d .
[0029]
2 5 The packet forwarding section 22 s e a r c h e s , from the flow table
21, an e n t r y having match conditions matching t h e packet r e c e i v e d ,
forwards a p a c k e t , or otherwise. S p e c i f i c a l l y , t h e packet forwarding
section 22 collates a header p a r t of t h e packet against the rule part of
each e n t r y a n d , in c a s e o f match, executes processing contents stored in
the processing p a r t . The f l o w t a b l e search as well as execution of the
processing by the flow table 21 and the packet forwarding section 22
may be performed at high speed by using a configuration employing an
ASIC (Application Specific I n t e g r a t i o n C i r c u i t ) . I n case the packet
matched none of the e n t r i e s , t h e packet forwarding section 22 sends out
the packet to the c o n t r o l l e r 3 v i a t h e c o n t r o l l e r connection s e c t i o n 23
with a r e q u e s t for setting an entry. In short, the packet may be
forwarded a t a higher speed in case match occurred than in case no
match occurred.
5 [0030]
I
The c o n t r o l l e r connection section 23 forwards the p a c k e t , o u t of
,i t h e received p a c k e t s , t h a t matches none of the e n t r i e s , to the controller l
3.
[003 I ]
;I 1 0 The c o n t r o l l e r 3 includes a s e r v e r t a b l e 3 1 , a s e r v e r t a b l e
management section 32, a forwarding d e s t i n a t i o n s e r v e r s e l e c t i o n
section 33 and an entry setting s e c t i o n 3 4 .
[0032]
The s e r v e r t a b l e management section 32 manages the s e r v e r t a b l e
15 31 t h a t supervises the correspondence between the categories and
s e r v e r s i n charge of the categories. In each of the categories in the
server t a b l e 3 1 , t h e r e is r e g i s t e r e d t h e information concerning the
server(s) 1 in charge of the category. Examples of the server
information include addresses of the servers and the port numbers of the
20 switch 2 t h e s e r v e r s a r e connected 'LO. In each category, there is
registered the information on at least one s e r v e r . F i g . 3 shows an
example s e r v e r table.
[0033]
On receipt of a packet from the switch 2, the forwarding
25 d e s t i n a t i o n s e r v e r selection section 33 s e l e c t s a forwarding d e s t i n a t i o n
server based on the packet received and the server table 31. I n t h e
s u b j e c t exemplary embodiment, an upper order b i t ( s ) of the forwarding
d e s t i n a t i o n address of t h e packet is mapped a g a i n s t the category IDS in
t h e s e r v e r table 31 to s e l e c t t h e s e r v e r 1 r e g i s t e r e d i n the relevant
30 category ID as t h e forwarding d e s t i n a t i o n server.
[0034]
The entry s e t t i n g s e c t i o n 34 generates an e n t r y f o r t h e flow table
21 so t h a t t h e packet received will be forwarded to the server 1 selected
by the forwarding destination server s e l e c t i o n s e c t i o n 3 3 . The entry
setting s e c t i o n s e t s t h e s o generated e n t r y i n the switch 2 . Also, when
t h e s e r v e r 1 sends out a response of the r e s u l t s o f processing to the
c l i e n t terminal 4, t h e e n t r y s e t t i n g section generates an entry for the
flow table 21 so that t h e packet w i l l be forwarded to t h e c l i e n t terminal
4, a n d s e t s the so generated e n t r y i n t h e switch 2 .
[0035]
It is noted that the e n t r y s e t t i n g section 34 is able t o d i s t i n g u i s h a
r e q u e s t packet from the client terminal 4 to the server I and a response
packet from t h e s e r v e r 1 to t h e c l i e n t terminal 4 from e a c h other. That
i s , when the source address of t h e p a c k e t , received from t h e switch 2,
coincides w i t h o n e of t h e addresses of the servers 1, managed by the
s e r v e r t a b l e 31, the packet received can be decided to bc a response
packet from the s e r v e r 1 to t h e c l i e n t terminal 4. If t h e source address
coincides with none of t h e addresses managed by the s e r v e r t a b l e 3 1 , t h e
packet received can be decided to be a r e q u e s t packet from the client
terminal 4 to t h e s e r v e r 1.
[0036]
The switch 2 as well as the controller 3, described above, may be
implemented by an OpenFlow switch disclosed i n Non-Patent L i t e r a t u r e
1 and an OpenFlow c o n t r o l l e r , disclosed in Non-Patent L i t e r a t u r e 2, as
r e s p e c t i v e b a s i c configurations
[0037]
The c l i e n t t e r m i n a l 4 may be any one of various terminals each
including a r e q u e s t section 41, an address setting section 42 and a
packet sendinglreceiving section 43.
[0038]
The request section 41 generates a r e q u e s t packet, such as a
key-value type data writelreadout request for t h e s e r v e r 1.
[0039]
The address setting section 42 sets a d e s t i n a t i o n address and a
source address of t h e r e q u e s t p a c k e t . As t h e d e s t i n a t i o n a d d r e s s , a
dummy a d d r e s s , c o n s i s t e n t with t h e key information contained i n the
request or with t h e s o r t of the request, is set. Here, the key contained
in t h e r e q u e s t is i n p u t a s a l e t t e r string i n t o a preset hash f u n c t i o n , and
a hash value of a preset number of b i t s , such as 32 b i t s , is calculated.
The hash v a l u e , thus calculated, is used as t h e dummy a d d r e s s . As the
source address, a correct address of t h e c l i e n t terminal 4 is set.
[0040]
The packet sending/receiving section 43 sends out a request
packet from the request section 41 to the switch 2 , while n o t i f y i n g t h e
r e q u e s t s e c t i o n 41 about the response packet received from the switch 2 .
[0041]
The r e s p e c t i v e s e c t i o n s (processing means) of the switch 2,
controller 3 and t h e c l i e n t terminal 4, shown in F i g . 2 , may be
implemented by a computer program t h a t causes a computer, making up
t h e s e apparatuses, to execute t h e above mentioned respective
processing operations using the computer's hardware.
[0042]
An operation of the p r e s e n t exemplary embodiment will now be
explained i n d e t a i l with reference to the drawings. In the following
explanation, it is assumed that a request to write data of ( k e y l , v a l u e l )
from the client terminal 4 is to be forwarded t o a n optimum o n e o f four
servers 1, where the request is then processed. It is also assumed t h a t
t h e c o n t r o l l e r 3 has already detected t h e f o u r servers 1 and that its
s e r v e r t a b l e management section 32 has formed a s e r v e r t a b l e such as
one shown in Fig. 3.
COO431
Initially, t h e operation of sending o u t t h e w r i t e r e q u e s t from the
c l i e n t terminal 4 to t h e s e r v e r 1 is explained. Fig. 4 depicts a
flowchart f o r i l l u s t r a t i n g an operation ( a t t h e time of a w r i t e r e q u e s t ) of
the f i r s t exemplary embodiment of the p r e s e n t i n v e n t i o n .
[0044]
Referring t o F i g . 4, t h e r e q u e s t s e c t i o n 41 of the client terminal 4
generates a ( k e y l , v a l u e l ) w r i t e r e q u e s t ( s t e p A l ) . The address
s e t t i n g s e c t i o n 42 calculates a hash value of a p r e s e t number of bits,
such as 32 b i t s , a s t h e address s e t t i n g s e c t i o n regards t h e k e y l a s a
l e t t e r s t r i n g ( s t e p A2). It is now assumed t h a t the upper two bits of the
hash value c a l c u l a t e d a r e '10'. The address s e t t i n g s e c t i o n s e t s above
12
hash value as the packet's d e s t i n a t i o n a d d r e s s , while setting a c o r r e c t
address of the client terminal as the source address (step A3). The
packet sendinglreceiving section 43 t h e n s e n d s out the request to the
switch 2 (step A4).
[0045]
When the switch 2 has received the packet ( s t e p A5), the packet
forwarding section 22 references the f l o w t a b l e 21 to confirm if there
exists an e n t r y t h a t matches a received packet (step A6). If no entry is
found (No of the step A6), t h e c o n t r o l l e r connection section 23 sends
o u t t h e above mentioned packet to t h e c o n t r o l l e r 3 (step A7).
[0046]
When t h e c o n t r o l l e r 3 has received the packet (stcp A8), the
forwarding destination s e r v e r s e l e c t i o n section 33 selects the
forwarding destination server based on the forwarding destination
address of t h e received packet and the server table 31 (step A9).
Specifically, the forwarding d e s t i n a t i o n s e r v e r s e l e c t i o n s e c t i o n 33
selects, from t h e s e r v e r table 31, the forwarding destination server to
which t h e category ID corresponding to the upper b i t s o f t h e d e s t i n a t i o n
address has been assigned. Since here t h e upper two b i t s o f t h e hash
value a r e '10' (= 2), the forwarding d e s t i n a t i o n s e r v e r s e l e c t i o n s e c t i o n
33 selects the server S2 of the category ID= 2 as the forwarding
destination server.
[0047]
The entry s e t t i n g s e c t i o n 34 then generates an entry, so that the
packet w i l l be forwarded to the forwarding destination s e r v e r s e l e c t e d
by the forwarding destination s e r v e r s e l e c t i o n section 33, and sets the
so generated e n t r y on the switch 2 on t h e path between the client
terminal 4 and t h e s e l e c t e d s e r v e r 1 (step A10). For i n s t a n c e , i f t h e
server S2 has been selected, the entry setting section generates an entry
in which the flow e n t r y condition p a r t is the "destination a d d r e s s : upper
two b i t s being ' 1 0 , ' " and t h e processing p a r t is " t h e forwarding
destination switch port number: P2," where P2 is t h e switch p o r t number
the server S 2 i s connected t o . The entry s e t t i n g s e c t i o n sets the so
generated e n t r y i n t h e switch S 2 . The packet forwarding s e c t i o n 2 2
then again confirms i f t h e r e is an entry t h a t matches t h e packet ( s t e p
A6). Since the entry has now been s e t i n t h e step A10, the matched
entry is found o u t this time.
[0048]
It is noted t h a t , i f a matched e n t r y has been found i n t h e step A6
(Yes of the step A6), the switch 2 forwards the packet i n accordance
with t h e contents of its processing p a r t ( s t e p S11). For i n s t a n c e , i f t h e
above mentioned e n t r y has been set in the step A10, and t h e packet has
matched the entry, the packet is s e n t from the port number P2 to the
server S2. On receipt of the r e q u e s t , t h e s e r v e r 1 performs processing
consistent with t h e r e q u e s t ( s t e p A12).
[0049]
I f thenceforth t h e switch 2 has received the packet corresponding
to " t h e source address: upper two bits being '10,"' the switch sends o u t
t h e packet from i t s port number P2, without regard to the address of the
source client terminal 4, without i n q u i r i n g the controller 3 about the
packet.
[OOSO]
An operation of sending o u t a response from the server 1 that has
received the request packet to t h e c l i e n t terminal 4 w i l l now be
explained. Fig. 5 depicts a flowchart f o r i l l u s t r a t i n g a n operation of
t h e f i r s t exemplary embodiment of t h e p r e s e n t invention at the time of
responding from the s e r v e r .
[005 l ]
Referring to Fig. 5, i n i t i a l l y the server 1 sends o u t a response
packet ( s t e p B l ) . When the switch 2 has received the packet (step B2),
t h e packet forwarding section 22 references t h e f l o w table 21 to confirm
i f t h e r e e x i s t s a n entry that matches the received packet ( s t e p B3). If
no such entry is found (No o f i h e step B3), the c o n t r o l l e r connection
section 23 sends o u t the above mentioned response packet to the
controller 3 (step B4).
[0052]
The entry s e t t i n g s e c t i o n 34 of the controller 3 generates an entry
of the f l o w t a b l e 21 so that t h e packet will be forwarded to the client
terminal 4, and sets t h e s o generated e n t r y on the switch 2 on t h e path
between t h e s e r v e r 1 and t h e d e s t i n a t i o n client terminal 4 (step B5).
The packet forwarding s e c t i o n 2 2 t h e n a g a i n c o n f i r m s if there is an
e n t r y t h a t matches the packet ( s t e p B3).
[0053]
I f t h e matched entry has been found in the step B3 (Yes of the
step B3), t h e switch 2 forwards t h e packet in accordance with t h e
contents of the processing p a r t of the e n t r y ( s t e p B6). On r e c e i p t of a
response, t h e client terminal 4 performs processing c o n s i s t e n t with the
response ( s t e p B7).
[0054]
I n t h e f i r s t exemplary embodiment of the p r e s e n t i n v e n t i o n ,
described above, the packet that matches the e n t r y s e t by t h e packet
forwarding section 22 is forwarded at a high speed based on the flow
table 21. On the other hand, t h e c o n t r o l l e r 3 is the sole s e l e c t i o n node
t h a t s e l e c t s t h e forwarding d e s t i n a t i o n s e r v e r . Therefore, t h e
forwarding d e s t i n a t i o n s e r v e r is selected by a s i n g l e s e l e c t i o n node
without compromising the reduction i n the delay in the request
processing.
[0055]
In first the exemplary embodiment of the p r e s e n t i n v e n t i o n , the
e n t r y s e t t i n g section 34 generates an entry based not on the entire
d e s t i n a t i o n a d d r e s s , that i s , on the hash value in i t s entirety, but o n i t s
upper b i t s . Hence, the probability of a h i t on t h e e n t r y of the flow
t a b l e 21 at the t i m e o f forwarding a packet becomes h i g h , thereby
f u r t h e r reducing the delay in t h e packet forwarding.
[0056]
Furthermore, i n t h e f i r s t exemplary embodiment of the present
invention, the a d d r e s s s e t t i n g s e c t i o n 42 sets a dummy address as the
d e s t i n a t i o n a d d r e s s , while it is unnecessary to set the real address of the
server 1. Hence, in adding a new client terminal 4, it is unnecessary to
set t h e s e r v e r information at the time of setup, so that the new c l i e n t
terminal may be introduced more readily.
[0057]
[Second Exemplary Embodiment]
A second exemplary embodiment of t h e p r e s e n t i n v e n t i o n , i n which
replication is made at t h e w r i t e time, w i l l now be d e t a i l e d with
reference to the drawings. Fig. 6 d e p i c t s a configuration of a
d i s t r i b u t e d s t o r a g e system of t h e second exemplary embodiment of the
p r e s e n t i n v e n t i o n .
[0058]
Referring to Fig. 6, there is shown a configuration which, as in
the f i r s t exemplary embodiment of t h e p r e s e n t i n v e n t i o n , includes a
server 1, a switch 2, a c o n t r o l l e r 3 and a client terminal 4A. It is noted
t h a t t h e c l i e n t terminal 4A differs from t h e corresponding component of
t h e f i r s t exemplary embodiment of t h e p r e s e n t invention in further
including a synchronization c o n t r o l section 44.
[0059]
When the r e q u e s t s e c t i o n 41 sends o u t a write r e q u e s t , t h e
synchronization c o n t r o l s e c t i o n 44 p o s t - w r i t e s a r e q u e s t ID in the write
request. When t h e packet sendinglreceiving section 43 r e c e i v e s a
response to the w r i t e r e q u e s t , the synchronization c o n t r o l s e c t i o n
references the request ID contained i n the response. The
synchronization c o n t r o l s e c t i o n s u p e r v i s e s t h e number of t h e responses,
so f a r r e t u r n e d , from o n e request ID to another. The synchronization
c o n t r o l s e c t i o n r e t u r n s the response to t h e r e q u e s t section 41 only when
the number of t h e responses so far returned has reached a p r e s e t number
of replications. In r e t u r n i n g t h e response, i f even o n e or more of an
error(s) is contained in the response, t h e synchronization c o n t r o l
s e c t i o n r e t u r n s the response corrupted with an e r r o r ( s ) . I n t h e r e is no
error, t h e synchronization c o n t r o l section r e t u r n s t h e normal response.
The synchronization control section 44 performs no operations when the
request section 41 sends o u t a readout request o r has received a
response to i t s readout r e q u e s t .
[0060]
The address s e t t i n g section 42 sets a destination address and a
source address of the r e q u e s t p a c k e t . As t h e d e s t i n a t i o n a d d r e s s , a
dummy address c o n s i s t e n t with t h e key information contained i n t h e
request or with t h e r e q u e s t species is s e t . The p r e s e n t exemplary
embodiment is similar to t h e exemplary embodiment as long as the
p o i n t s j u s t s t a t e d a r e concerned. However, in the present exemplary
embodiment, a hashed value of a p r e - s e t number of b i t s , such as 32 bits,
is c a l c u l a t e d from the key contained i n t h e r e q u e s t , and the lower most
bit of the hashed value is overwritten by the r e q u e s t s p e c i e s , which is 1
f o r a w r i t e r e q u e s t and 0 f o r a readout r e q u e s t , to yield a dummy
address.
[0061]
Fig. 7 i l l u s t r a t e s a method for generating a dummy address i n t h e
second exemplary embodiment of the present invention. The present
exemplary embodiment is similar to the f i r s t exemplary eillbodiment in
using t h e hashed value of the k e y l , however, the t r a i l i n g bit of the
hashed value is overwritten with t h e r e q u e s t species to yield the dummy
address.
[0062]
On r e c e i p t of the packet from the switch 2, the forwarding
d e s t i n a t i o n s e r v e r s e l e c t i o n s e c t i o n 33 of the controller 3 s e l e c t s a
forwarding destination server based on t h e received packet and the
server t a b l e 3 1 . Specifically, as in the f i r s t exemplary embodiment of
the present i n v e n t i o n , a server assigned to the category ID
corresponding to t h e upper b i t s o f the packet's destination address is
s e l e c t e d from t h e s e r v e r table 31. In addition, in the subject
exemplary embodiment, if the r e q u e s t s p e c i e s of t h e lower most b i t is 1
representing a write r e q u e s t , t h e forwarding destination s e r v e r s e l e c t i o n
section 33 s e l e c t s a number of forwarding destination s e r v e r s equal to
the preset number of times of r e p l i c a t i o n s . I f t h e request species of
t h e lower most bit is 0 representing a readout r e q u e s t , t h e forwarding
d e s t i n a t i o n s e r v e r s e l e c t i o n section s e l e c t s a s i n g l e forwarding
d e s t i n a t i o n s e r v e r .
[0063]
Otherwise, the configuration of the second exemplary
embodiment of the p r e s e n t invention is similar to that of t h e above
described f i r s t exemplary embodiment, and hence the description
therefor is dispensed with.
[0064]
An operation of the p r e s e n t exemplary embodiment w i l l be
explained in detail with reference to the drawings. I n t h e following
explanation, i t is assumed t h a t a w r i t e request for data ( k e y l , v a l u e l )
from t h e c l i e n t terminal 4 is forwarded t o a n appropriate one 1 of 12
servers 1, where the request is then processed. It is a l s o assumed that
t h e number o f t i m e s of replications of 3 has been set, and a s e r v e r t a b l e
shown i n F i g . 8 has already been formulated by the s e r v e r t a b l e
management s e c t i o n 3 2 .
[0065]
Initially, a n o p e r a t i o n of sending a w r i t e r e q u e s t from the client
terminal 4 to t h e s e r v e r 1 will be explained. Fig. 9 d e p i c t s a flowchart
f o r i l l u s t r a t i n g a n o p e r a t i o n of t h e second exemplary embodiment of the
p r e s e n t invention a t t h e time of a w r i t e r e q u e s t .
[0066]
Referring to Fig. 9, the request section 41 of the client terminal 4
generates a write r e q u e s t f o r ( k e y l , v a l u e l ) in a step C I . The
synchronization control section 44 then p o s t - w r i t e s a r e q u e s t ID in the
w r i t e r e q u e s t ( s t e p C2). The address s e t t i n g section 42 then calculates
a hash value of a preset number of b i t s , f o r example 32 b i t s , as the
address s e t t i n g section regards t h e key1 as a l e t t e r s t r i n g ( s t e p C3). It
is here assumed t h a t t h e upper two b i t s o f t h e above mentioned hash
value is '10'. The address s e t t i n g section s e t s t h e above mentioned
hash value and t h e r e q u e s t s p e c i e s (= 1) as the packet's destination
address ( s e e Fig. 7 ) , while s e t t i n g t h e client t e r m i n a l ' s c o r r e c t address
as the source address ( s t e p C4). The packet sendinglreceiving section
43 then sends o u t the request to the switch 2 (step C5).
[0067]
On receipt of the packet ( s t e p C6), t h e switch 2 references the
flow t a b l e 21 to check to s e e i f t h e r e is an entry that matches the
received packet (step C7). If no such e n t r y is found (No of the step
C7), the c o n t r o l l e r connection s e c t i o n 23 sends o u t the packet to the
controller 3 (step C8).
When the packet is received by t h e c o n t r o l l e r 3 (step C9), the
forwarding destination server selection section 33 selects the
forwarding destination server based on the forwarding destination
address of t h e received packet and the server table 31 (step C10).
[0069]
F i g . 1 0 d e p i c t s a f l o w t a b l e showing the detail of the above
mentioned step C10. Referring t o F i g . 1 0 , the forwarding d e s t i n a t i o n
s e r v e r s e l e c t i o n section selects the category ID by mapping the upper
bits of the forwarding d e s t i n a t i o n address of the packet a g a i n s t t h e
category IDS of t h e s e r v e r t a b l e 31 (step D l of Fig. 10). The
forwarding destination s e r v e r s e l e c t i o n s e c t i o n t h e n references t h e
r e q u e s t s p e c i e s represented by t h e lower most b i t ( s t e p D2). If the
r e q u e s t s p e c i e s is 1 representing a write request (Yes of the step D2), a
number of t h e forwarding d e s t i n a t i o n s e r v e r s equal to the preset number
of t i m e s o f the r e p l i c a t i o n s is s e l e c t e d ( s t e p D3). On the other hand, a
sole forwarding destination server is s e l e c t e d i n a step D4 if the request
species is 0 representing a readout r e q u e s t (No of the step D2). For
i n s t a n c e , i f the upper two b i t s o f the hash value a r e ' 1 0 ' (= 2) and the
lower most bit is '1,' t h e s e r v e r s S 6 , S7 and S 8 , r e g i s t e r e d in the
category ID2, are selected as the forwarding destination servers.
[0070]
Referring again to Fig. 9, t h e e n t r y s e t t i n g section 34 generates
an entry so that the packet w i l l be forwarded to t h e forwarding
d e s t i n a t i o n s e r v e r as selected by the forwarding destination server
selection section 33, a n d s e t s the so generated e n t r y in the switch 2
(step C l l ) . For i n s t a n c e , i f t h e servers s e l e c t e d a r e S6, S7 and S8,
such an entry in which the conditional part is "forwarding d e s t i n a t i o n
address: upper two bits being ' 1 0 , ' the lower most bit being '1"' and the
processing p a r t is "forwarding destination switch p o r t numbers: P 6 , P7,
P8" ( i n d i c a t i n g switch port numbers connecting to the servers S6, S7,
S8, respectively) is generated and set in t h e switch 2.
[0071]
The packet forwarding s e c t i o n 2 2 t h e n again confirms i f t h e r e is
an entry that matches the packet ( s t e p C7). Since the entry has now
been s e t in t h e s t e p C11, t h e matched entry is found out t h i s t i m e .
[0072]
I f t h e matched e n t r y has been found i n the step C7 (Yes in the
step C7), the switch 2 forwards t h e packet in accordance with t h e
contents of the processing p a r t of the entry t h u s found ( s t e p C12). For
i n s t a n c e , i f t h e above mentioned e n t r y has been s e t in the step Cl1 and
t h e packet has matched t h e entry, t h e packet is sent by multi-cast from
the p o r t numbers P6, P7, P8 to t h e s e r v e r s S 6 , S 7 , S8, respectively.
[0073]
On r e c e i p t of the r e q u e s t , t h e s e r v e r s 1 perform processing
consistent with the r e q u e s t ( s t e p C13).
[0074]
If thenceforth the switch 2 has received the packet corresponding
to " t h e d e s t i n a t i o n a d d r e s s : upper two b i t s being '10' and the lower most
bit being '1,"' t h e switch sends o u t t h e packet from i t s ports P6, P7, P8,
without regard to t h e address of the source c l i e n t t e r m i n a l 4, without
i n q u i r i n g t h e c o n t r o l l e r 3 about the packet.
[0075]
An operation of sending o u t a response from the server 1 t h a t h a s
received t h e r e q u e s t packet to t h e c l i e n t terminal 4 w i l l now be
explained. Fig. 11 depicts a flowchart f o r i l l u s t r a t i n g an operation of
the second exemplary embodiment of the present invention at the time
of responding from the server.
[0076]
Referring to Fig. 11, t h e s e r v e r 1 i n i t i a l l y sends o u t a response
inclusive of the request ID contained i n t h e r e q u e s t (step El of Fig. 11).
[0077]
When the switch 2 has received a packet ( s t e p E2), the packet
forwarding s e c t i o n 22 references the f l o w t a b l e 21 to check to see if
there is any e n t r y that matches the received packet (step E3). If no
entry has been found (No of the step E3), t h e c o n t r o l l e r connection
section 23 sends out the packet to t h e c o n t r o l l e r 3 (step E4).
[0078]
The entry s e t t i n g s e c t i o n 34 of the controller 3 generates an entry
of t h e flow t a b l e 21 so t h a t t h e packet w i l l be forwarded to t h e c l i e n t
terminal 4, and sets the so generated e n t r y in the switch 2 (step E5).
The packet forwarding s e c t i o n 22 t h e n a g a i n checks to see i f t h e r e is
any e n t r y that matches the packet (step E3).
[0079]
If t h e matched entry has been found in t h e s t e p E3 (Yes of the
step E7), the switch 2 forwards the packet in accordance with the
contents of t h e processing part of t h e e n t r y thus found (step E6).
[OOSO]
On r e c e i p t of the response ( s t e p E7), the synchronization c o n t r o l
section 44 of the client terminal 4 references the request ID to confirm
i f t h e number of responses t h a t returned the r e q u e s t s ID has reached t h e
number of the r e p l i c a t i o n s ( s t e p ES). In case the number of the
responses returned has reached t h e number of the replications (Yes of
the step ES), the synchronization c o n t r o l section 44 returns the response
to the request section 41 (step E9). The request section 41 performs
processing c o n s i s t e n t with t h e response ( s t e p E10). I f conversely the
number of t h e responses returned is not up to the number of replications
(No of the step E8), the synchronization control section 44 performs no
processing.
[OOSl]
An operation a t t h e time of a readout request will now be
explained. It is assumed here that a r e q u e s t t o read o u t d a t a
corresponding to key1 is to be forwarded to an appropriate one of 12
servers 1, where the request is to be processed.
[0082]
Initially, t h e processing of sending o u t a request from the client
terminal 4 to t h e s e r v e r 1 will be explained. An operation of sending a
readout r e q u e s t i s t h e same a s t h e operation of sending the request
according to the f i r s t exemplary embodiment shown in Fig. 4.
[0083]
However, in the step A3, t h e c o n t r o l l e r 3 sets the packet's
forwarding destination address based on the above mentioned hash
value as found in the step A2 and on the request species (= 0), while
setting the c l i e n t t e r m i n a l ' s correct address as the source address.
[0084]
Also, in the step A9, the forwarding destination server selection
section 33 selects the forwarding destination server based on the
destination address of the received packet and on the server table 31.
This will now be detailed with reference t o Fig. 10. For instance, if
the upper three bits and the lower most b i t o f the hashed value are
respectively '100' and '0,' the forwarding destination server selection
section selects the server S6, from among the category IDS 2 , as the
forwarding destination server. I f the upper three bits and the lower
most bit of the hashed value a r e respectively 1 0 1 ' and 0 the
forwarding destination server selection s e c t i o n s e l e c t s the server S7 as
the forwarding destination server. In this manner, readout requests t o
the same category ID may be selected such as to assure load balancing
because data must have been replicated within the same ID category.
[OOSS]
The operation other than that described above is similar t o the
response sending operation of the f i r s t exemplary embodiment of the
present invention and hence the corresponding explanation is dispensed
with.
[0086]
The operation of sending the response t o the readout request from
the server 1 to the client terminal 4 is the same as the operation of
sending the response in the first exemplary embodiment of the present
invention and hence the corresponding explanation is dispensed with.
[0087]
In the above described second exemplary embodiment of the
present invention, in doing data replication, data is sent out not by
multi-cast from the client terminal 4 but rather by multicast from the
switch 2 to the server. It is thus possible to suppress the total amount
of the packets flowing in the entire network so that flooding is less
l i a b l e t o be produced.
[0088]
I n a d d i t i o n , in the second exemplary embodiment of t h e p r e s e n t
i n v e n t i o n , i n which the load by t h e readout request is d i s t r i b u t e d among
t h e s e r v e r s having s t o r e d t h e same r e p l i c a t i o n s , it is possible to
suppress the load otherwise imposed on the servers.
[0089]
[Third Exemplary Embodiment]
A third exemplary embodiment of the p r e s e n t i n v e n t i o n , in which a r e l a y
node ( f r o n t end 6 ) i s arranged between the client terminal 4 and the
switch 2, and the dummy address is generated in this relay node, will
now be d e t a i l e d with reference to the drawings. Fig. 12 d e p i c t s a
block diagram showing a configuration of a d i s t r i b u t e d s t o r a g e system
of the third exemplary embodiment of t h e p r e s e n t invention.
[0090]
Referring to Fig. 12, there is shown an arrangement i n c l u d i n g , i n
addition to a s e r v e r 1, a switch 2 , a c o n t r o l l e r 3 and a client terminal 4B,
shown in the f i r s t exemplary embodiment, a load balancer 5 and a f r o n t
end(s) 6.
[0091]
The load balancer 5 is a load d i s t r i b u t i o n apparatus provided with
a packet forwarding section 51. The packet forwarding s e c t i o n 51
forwards t h e r e q u e s t , received from the client terminal 4B, to one of the
f r o n t ends 6 connected thereto. In forwarding the packet, the packet's
d e s t i n a t i o n address is changed to the address of the f r o n t e n d 6 of the
forwarding d e s t i n a t i o n . The packet forwarding section 51 also
forwards the response, received from the f r o n t e n d 6, to the client
terminal 4B.
[0092]
The f r o n t e n d 6 is t h e apparatus performing pre-processing f o r
the r e q u e s t processing and includes an address setting section 61 and a
packet sendinglreceiving section 62.
[0093]
The address setting section 61 sets a d e s t i n a t i o n a d d r e s s of the
r e q u e s t received from the load balancer 5. As the destination address,
a dummy address c o n s i s t e n t with t h e key information contained in the
request as well as t h e r e q u e s t s p e c i e s is set. Here, it is assumed t h a t a
hashed value of a p r e s e t number of bits, such as 32 b i t s , i s calculated
from t h e key contained in the request, and the hashed value is to be the
dummy address. The source address remains to be same as the address
5 of the client terminal 4B i n h e r e n t l y s e t in the packet.
[0094]
The packet sendinglreceiving section 62 receives the request
from t h e load balancer 5 and sends o u t t h e request processed by the
address s e t t i n g s e c t i o n 61 to the switch 2, while sending t h e response
10 received from the switch 2 to the load balancer 5.
100951
The address s e t t i n g s e c t i o n 42 of the client terminal 4B s e t s t h e
address of t h e load balancer 5 as the d e s t i n a t i o n a d d r e s s , while s e t t i n g
the c o r r e c t address of t h e c l i e n t terminal 4B as the source address.
15 [0096]
The p r e s e n t exemplary embodiment is otherwise t h e same i n
1 configuration as the f i r s t exemplary embodiment of the present
invention, and hence the corresponding explanation is dispensed w i t h .
[0097]
2 0 An operation of t h e p r e s e n t exemplary embodiment will now be
detailed with reference to the drawings. I n t h e explanation to follow,
it is assumed t h a t a write request f o r d a t a ( k e y l , v a l u e l ) from the client
terminal 4 is sent to an appropriate one 1 of four s e r v e r s 1 , where t h e
request is processed. It is a l s o assumed t h a t the four servers 1 have
25 already been detected by t h e c o n t r o l l e r 3 and that a server table shown
i n F i g . 3 has already been formulated by t h e s e r v e r table management
section 32.
[0098]
Initially, a n o p e r a t i o n o f sending a write request from the client
30 terminal 4 to t h e s e r v e r 1 is explained. Fig. 13 d e p i c t s a flow table for
i l l u s t r a t i n g an operation a t t h e time of a w r i t e r e q u e s t according to the
third exemplary embodiment of t h e p r e s e n t invention.
[0099]
Referring to Fig. 13, i n i t i a l l y t h e r e q u e s t section 41 of t h e c l i e n t
terminal 4 generates a w r i t e r e q u e s t for ( k e y l , v a l u e l ) i n a step F1.
The address setting section 42 t h e n s e t s a n address of the load balancer
5, as a p a c k e t ' s destination a d d r e s s , while setting the client terminal's
c o r r e c t address as a source address ( s t e p F 2 ) . The packet
sendinglreceiving section 43 then sends o u t a r e q u e s t t o the load
balancer 5 (step F3).
[OIOO]
On r e c e i p t of the r e q u e s t ( s t e p F4), the load balancer 5 forwards
t h e r e q u e s t to o n e o f the f r o n t e n d s 6 connected thereto (step F5).
[OlOl]
When the f r o n t e n d 6 has received t h e r e q u e s t ( s t e p F6), the
address s e t t i n g section 61 c a l c u l a t e s a hashed value of a p r e s e t number
of b i t s , such as 32 bits, from keyl contained in the r e q u e s t ( s t e p F7).
It is assumed here that the upper two b i t s o f the hashed value a r e '10'.
The front end s e t s t h e above mentioned hashed value as the packet's
d e s t i n a t i o n a d d r e s s , while s e t t i n g , as the source a d d r e s s , the correct
address of the c l i e n t t e r m i n a l 4 as i n h e r e n t l y s e t in the packet (step F8).
The front end 6 sends out the request to the switch 2 (step F9).
[O 1021
When the switch 2 has received t h e packet ( s t e p FlO), the packet
forwarding s e c t i o n 2 2 references t h e f l o w t a b l e 21 to check to see if
there is the entry that matches the packet received (step F11). If no
such e n t r y h a s been found (No of the step F l l ) , t h e c o n t r o l l e r
connection section 23 sends o u t the packet to t h e c o n t r o l l e r 3 ( s t e p
F12).
[0 1031
When t h e c o n t r o l l e r 3 has received the packet (step F13), the
forwarding destination s e r v e r s e l e c t i o n s e c t i o n 3 3 s e l e c t s the
forwarding destination s e r v e r based on the destination address of the
received packet and on the s e r v e r t a b l e 3 1 . Here, t h e upper bits of the
p a c k e t ' s d e s t i n a t i o n address a r e mapped a g a i n s t t h e category IDS of the
s e r v e r t a b l e 31 t o s e l e c t t h e forwarding destination s e r v e r i n keeping
with t h e upper bits of t h e d e s t i n a t i o n address ( s t e p F14). Since t h e
upper two bits of the hashed value a r e '10' (= 2), the forwarding
d e s t i n a t i o n s e r v e r s e l e c t i o n section 33 selects the server S 2 , r e g i s t e r e d
in t h e category ID2, as t h e forwarding destination server.
[0 1041
The entry s e t t i n g s e c t i o n 34 generates an entry so that the packet
will be forwarded to the forwarding d e s t i n a t i o n s e r v e r as selected by
t h e forwarding d e s t i n a t i o n s e r v e r s e l e c t i o n section 33, and s e t s t h e
r e s u l t i n g e n t r y in the switch 2 ( s t e p F15). If the s e r v e r S 2 , f o r
example, h a s been s e l e c t e d , t h e e n t r y s e t t i n g s e c t i o n generates such
entry the conditional part of which is "destination address: upper two
b i t s being '10'" and the processed p a r t o f which is "forwarding
destination s w i t c h p o r t number: P2", representing the switch port
number t h e s e r v e r 52 is connected t o , a n d s e t s the r e s u l t i n g e n t r y i n t h e
switch 2.
[0105]
The packet forwarding s e c t i o n 2 2 t h e n a g a i n checks to s e e i f
there is an entry that matches the packet ( s t e p F11). Since the entry
has now been s e t i n the step F15, the matched e n t r y is found this time.
[0106]
If t h e matched e n t r y has been found in t h e s t e p F11 (Yes in the
step F l l ) , the switch 2 forwards the packet in accordance with the
contents of t h e processing p a r t of t h e e n t r y thus found ( s t e p F16). For
instance, if the above mentioned e n t r y has been s e t in the step F15 and
the packet matched the entry, t h e packet is forwarded from the port
number P2 to the server S2.
[0107]
On r e c e i p t o f t h e r e q u e s t , t h e s e r v e r 1 performs processing i n
keeping with the request (step F17).
[0108]
If thenceforth t h e switch 2 has received the packet corresponding
to t h e "source address: upper two b i t s being '10,"' t h e switch sends o u t
the packet from i t s port P2, without regard to the address of the source
c l i e n t terminal 4, without i n q u i r i n g t h e c o n t r o l l e r 3 about t h e p a c k e t .
[0109]
An operation of sending o u t a response from the server 1 that has
received t h e r e q u e s t packet to t h e c l i e n t terminal 4 w i l l now be
explained. Fig. 14 depicts a flowchart f o r i l l u s t r a t i n g a n operation of
the t h i r d exemplary embodiment of t h e p r e s e n t invention at the t i m e o f
responding from the server.
[OllO]
Referring to Fig. 14, i n i t i a l l y the server 1 sends o u t a response to
the r e q u e s t ( s t e p G1 of Fig. 14).
[ O l l l ]
When t h e switch 2 has received the packet (Step G2), t h e packet
forwarding s e c t i o n 2 2 references t h e flow table 21 to check t o s e e if
there is an entry that matches the received packet ( s t e p G3). If no such
entry is found (No of the step G3), the c o n t r o l l e r connection section 23
sends o u t the packet to t h e c o n t r o l l e r 3 (step G4).
[0112]
The entry s e t t i n g s e c t i o n 34 in t h e c o n t r o l l e r 3 generates an entry
of the f l o w t a b l e 21 so that the packet will be forwarded to the client
terminal 4, a n d s e t s the r e s u l t i n g e n t r y in the switch 2 ( s t e p G5).
[0113]
The packet forwarding s e c t i o n 22 of the switch 2 then agairi
checks to see if there is an entry t h a t matches the packet (step 6 3 ) .
[0114]
If the matched e n t r y is found i n the step 6 3 (Yes of the step G3),
the switch 2 forwards t h e packet in accordance with the c o n t e n t s o f the
processing part of the entry thus found ( s t e p G6).
[0115]
On receipt of the response, the front end 6 forwards i t t o the load
balancer 5 ( s t e p G7). On r e c e i p t of the response, the load balancer
forwards the r e s p o n s e t o the client terminal 4 (step G8).
[0116]
On receipt of the response, the c l i e n t terminal 4 performs t h e
processing in keeping with t h e response (G9).
[0117]
I n t h e t h i r d exemplary embodiment of the p r e s e n t i n v e n t i o n ,
described above, the dummy address for the destination address is
formed by t h e f r o n t end 6, disposed closer than t h e load balancer 5 to
the server 1 . Hence, t h e c l i e n t terminal 4 may be connected, at a
location ahead of the load balancer 5, to a switch of the type which is
n o t c o n t r o l l e d by the controller 3.
[0118]
It should be noted that, while p r e f e r r e d exemplary embodiments
of the present invention are described above, t h e p r e s e n t i n v e n t i o n is
not to be r e s t r i c t e d to t h e s e p a r t i c u l a r modes, such t h a t f u r t h e r changes,
substitutions or adjustments may be made within the range n o t departing
from t h e b a s i c technical concept of the invention. For instance, in the
above described exemplary embodiments, i t has been i n d i c a t e d that the
address to be collated a g a i n s t t h e r u l e p a r t o f the flow entry is to be
formed using hashing c a l c u l a t i o n s . It is however a l s o possible to use a
method f o r forming a flow identifier or otherwise from the key by other
techniques than those shown above.
[0119]
Moreover, in the above described exemplary embodiments, it has
been i n d i c a t e d t h a t the address s e t t i n g s e c t i o n is provided a t t h e client
terminal 4 o r a t the front end 6 and the hash value is generated at the
c l i e n t terminal 4 or at the front end 6 . Alternatively, the key
information may be sent through without the c l i e n t t e r m i n a l 4 or the
front end 6 generating t h e hashed v a l u e . In s u c h c a s e , it may be t h e
controller 3 t h a t generates the hashed value from the key information or
generates a f l o w i d e n t i f i e r t h a t c a n be uniquely found from the key
information. Similar control may be implemented by the controller 3
causing t h e switch 2 on t h e path to perform the processing of embedding
the hashed value or the flow identifier in a header p a r t of the relevant
packet or otherwise, or t h e processing of appending another header.
[O 1201
The above mentioned flow identifier may be of a v a r i e t y of
different configurations. For i n s t a n c e , t h e key may be rendered into
numerical figures i n accordance with the ASCII code and these
numerical figures may then be p u t to modulo c a l c u l a t i o n s , f o r example,
c a l c u l a t e d modulo 232, to f i n d a remainder, which may then be used as
an address of t h e d e s t i n a t i o n .
[0121]
The e n t i r e disclosures of the above P a t e n t L i t e r a t u r e s and
Non-Patent L i t e r a t u r e s a r e incorporated h e r e i n by reference t h e r e t o .
5 Modifications and adjustments of the exemplary embodiment a r e
p o s s i b l e within the scope of the overall disclosure (including t h e
claims) of the present invention and based on the basic technical
concept of t h e p r e s e n t i n v e n t i o n . Various combinations a n d selections
of various disclosed elements (including e a c h element o f e a c h claim,
1 0 e a c h element o f e a c h exemplary embodiment, each element of each
drawing, e t c . ) a r e possible within the scope of the claims of the present
i n v e n t i o n . That i s , t h e p r e s e n t invention of course includes various
variations and modifications t h a t could be made by those skilled in the
a r t according to the o v e r a l l d i s c l o s u r e including t h e claims and the
15 technical concept.
REFERENCE SIGNS LIST
[O 1221
1 server
2 switch
20 2A packet forwarding apparatus
3 c o n t r o l l e r
3A c o n t r o l apparatus
4, 4A, 4B c l i e n t t e r m i n a l
5 load balancer
25 6 front end
2 1 f l o w t a b l e
22, 51 packet forwarding section
2 3 c o n t r o l l e r connection section
3 1 server table
30 32 s e r v e r t a b l e management section
33 forwarding destination s e r v e r s e l e c t i o n s e c t i o n
33A forwarding destination selection section
34 e n t r y s e t t i n g section
34A entry s e t t i n g s e c t i o n
4 1 r e q u e s t s e c t i o n
42, 61 address setting section
43, 62 packet sendinglreceiving section
44 synchronization c o n t r o l s e c t i o n
Translation of Amendment of the claims under article 19(1)
PCT/JP2013/059567
Amended Claims
[Received by the international Bureau a s o f August 9, 2013
(09.08.2013)]
5 1. (Amended) A d i s t r i b u t e d s t o r a g e system, comprising:
a p l u r a l i t y of servers that s t o r e d a t a t h a t is associated with key
information, r e s p e c t i v e l y ;
a packet forwarding apparatus t h a t , on receipt of a new packet
that contains t h e key information and is addressed to the server(s),
10 r e q u e s t s a control apparatus to decide a forwarding d e s t i n a t i o n from
among t h e p l u r a l i t y of servers; and
t h e c o n t r o l apparatus ;
t h e c o n t r o l apparatus comprising: a forwarding d e s t i n a t i o n
selection s e c t i o n t h a t decides a forwarding d e s t i n a t i o n o f the packet
15 based on key information i n a header of the p a c k e t ; a n d a n e n t r y s e t t i n g
s e c t i o n t h a t s e t s , in a packet forwarding apparatus(es) on a path to the
forwarding d e s t i n a t i o n , a flow e n t r y f o r forwarding a subsequent
packet(s) with same lcey information to the forwarding destination,
wherein
2 0 t h e packet forwarding apparatus(es) forwards a packet(s) with the
same key i n f o r m a t i o n t o t h e forwarding d e s t i n a t i o n u s i n g the set flow
entry.
2. The d i s t r i b u t e d storage system according to claim 1, wherein
a d e s t i n a t i o n address is s e t i n t h e packet, the d e s t i n a t i o n address
25 being generated using a hashed value of the key information, and
a preset bit(s) of the hashed value is used as a match condition i n
the flow entry.
3. The d i s t r i b u t e d s t o r a g e system according to claim 1, wherein
t h e forwarding d e s t i n a t i o n s e l e c t i o n section generates a hashed
30 value from t h e key information and decides a forwarding destination of
t h e packet using a p r e s e t b i t ( s ) of the generated hashed v a l u e .
4. (Amended) The d i s t r i b u t e d s t o r a g e system according to any one
of claims 1 to 3, wherein
a header p a r t of t h e packet includes r e q u e s t s o r t information,
Translation of Amendment of the claims under article 19(1)
PCT/JP2013/059567
in case t h e r e q u e s t sort information is a w r i t e requcst, the
forwarding destination selection section s e l e c t s a p r e s e t number of
forwarding destination(s), and
a number of replication(s) equal to t h e p r e s e t number is
5 generated.
5. The d i s t r i b u t e d s t o r a g e system according to claim 4, wherein
a c l i e n t terminal comprises a synchronization c o n t r o l l e r t h a t
checks whether or n o t generation of the p r e s e t number of replication(s)
is completed.
10 6 . The d i s t r i b u t e d s t o r a g e system according t o a n y o n e o f claims 1,
2, 4 and 5, comprising:
a r e l a y node provided between a source of the packet and the
packet forwarding a p p a r a t u s , wherein
the r e l a y node transforms the packet received from the source
15 thereof into a packet that is addressed to the server and has a
d e s t i n a t i o n address generated using a hashed value of the key
information.
7. (Amended) A control apparatus connected to a p l u r a l i t y of
servers and a packet forwarding a p p a r a t u s , the plurality of servers
20 s t o r i n g d a t a t h a t is associated with key information, respectively, the
packet forwarding apparatus requesting t h e c o n t r o l apparatus to decide
a forwarding destination from among t h e p l u r a l i t y of servers on receipt
of a new packet t h a t c o n t a i n s the key information and i s addressed to
the server(s), the control apparatus comprising:
2 5 a forwarding d e s t i n a t i o n s e l e c t i o n s e c t i o n t h a t decides a
forwarding d e s t i n a t i o n o f t h e packet based on key information in a
header part of t h e p a c k e t ; and
an entry s e t t i n g s e c t i o n that sets, i n a packet forwarding
apparatus(es) on a path to t h e forwarding d e s t i n a t i o n , a flow e n t r y f o r
30 forwarding a subsequent packet(s) with same key information to the
forwarding d e s t i n a t i o n .
8. A c l i e n t t e r m i n a l , comprising:
an addrkss s e t t i n g s e c t i o n t h a t generates a r e q u e s t packet with a
d e s t i n a t i o n address generated using a hashed value of key information,
Translat~ono f Amendment of the clalms under art~cle1 9(1)
PCT/JP20131059567
wherein
the client terminal r e q u e s t s a server using the request pacltet.
9. (Amended) A load balancing method, comprising:
by a c o n t r o l a p p a r a t u s , r e c e i v i n g from a paclcei forwarding
apparatus a r e q u e s t f o r d e c i d i n g a forwarding d e s t i n a t i o n of a new
paclcet with ltey information paired to a value(s);
deciding a forwarding d e s t i n a t i o n of the paclcet from among a
plurality of servers based on the ltey information in a header p a r t o f the
paclcet; and
setting, in a pacltet forwarding apparatus(es) on a path to the
forwarding d e s t i n a t i o n , a flow entry f o r forwarding a subsequent
pacltet(s) with same ltey i n f o r m a t i o n t o the forwarding d e s t i n a t i o n .
10. (Amended) A program f o r a computer loaded on a c o n t r o l
apparatus t h a t is connected to a p l u r a l i t y of s e r v e r s a n d a paclcei
forwarding a p p a r a t u s , the plurality of servers storing data that is
associated with ltey information, r e s p e c t i v e l y , the paclcet forwarding
apparatus requesting the control apparatus to decide a forwarding
d e s t i n a t i o n from among the plurality of the s e r v e r s o n r e c e i p t of a new
pacltet that contains the ltey i n f o r m a t i o n a n d is addressed to the
server(s), the program causing the computer to execute:
deciding a forwarding d e s t i n a t i o n of the paclcet from among a
plurality of servers based on the ltey information in a header part of the
packet; and
s e t t i n g , i n a pacltet forwarding apparatus(es) on a path to the
forwarding d e s t i n a t i o n , a fiow e n t r y f o r forwarding a subsequent
paclcet(s) with same ltey i n f o r m a t i o n t o the forwarding d e s t i n a t i o n .