WE CLAIM:
1. An apparatus, comprising:
a link interface to support bi-directional communication over a link with a link peer, the link interface including a replay buffer (1806), the link interface to,
store in the replay buffer (1806) a copy of a packet to be transmitted over the link to the link peer;
transmit the packet over the link to the link peer;
determine that the packet has been received without error at the link peer without use of an explicit Acknowledgement (ACK) received from the link peer;
allow the copy of the packet in the replay buffer (1806) to be overwritten if it is determined the packet has been received at the link peer without error; and
determine an implicit ACK that a packet has been received by the link peer without error prior to allowing the copy of the packet in the replay buffer (1806) to be overwritten,
wherein the buffer comprises a circular buffer having a depth, and in which copies of packets are written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and wherein the link interface to detect an implicit ACK by performing operations comprising:
tracking, for the packet, a slot at which a copy of the packet was written to in the circular buffer; and
detecting that a retry request for a retransmission of a packet copied to a given slot has not been received by the time the sequence for writing copies of packets into the circular buffer has returned to the given slot.
2. The apparatus as claimed in claim 1, wherein the link interface is to:
determine that a number of slots in the circular buffer that would be written to
during a roundtrip transmission between interface ports of the apparatus and the link peer is less than or equal to a depth of the circular buffer.
3. An apparatus, comprising:
a link interface to support bi-directional communication over a link with a link peer, the link interface including a replay buffer (1806), the link interface to,

44

INTL-7097-IN

store in the replay buffer (1806) a copy of a packet to be transmitted over the link to the link peer;
transmit the packet over the link to the link peer;
determine that the packet has been received without error at the link peer without use of an explicit Acknowledgement (ACK) received from the link peer;
allow the copy of the packet in the replay buffer (1806) to be overwritten if it is determined the packet has been received at the link peer without error; and
determine an implicit ACK that a packet has been received by the link peer without error prior to allowing the copy of the packet in the replay buffer (1806) to be overwritten, wherein the buffer comprises a circular buffer having a depth, and in which copies of packets are written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and wherein the link interface is to detect an implicit ACK by,
tracking, for each of multiple transmitted packets, a slot at which a copy of the packet was written to in the circular buffer; and
detecting that a retry request for a retransmission of a packet copied to a given slot has not been received by the time the sequence for writing copies of packets into the circular buffer has returned to the given slot plus n additional packet transfer cycles, wherein n is an integer >0.
4. The apparatus as claimed in claim 3, wherein the link interface is to determine the n additional packet transfer cycles by performing operations comprising:
sending a first packet to the link peer, the first packet including indicia to solicit return of a roundtrip marker from the link peer in response to receipt of the first packet;
beginning at a starting slot, transmitting packets retrieved from slots in the replay buffer to the link peer, wherein each packet transfer cycle advances the slot in the replay buffer by one until a last slot in the replay buffer is reached, at which point the replay buffer slot wraps to a first slot;
while transmitting the packets, detecting receipt of the roundtrip marker from the link peer; and
if the buffer slot has returned to the starting slot prior to receiving the roundtrip marker, starting a counter and transmitting additional packets to the link peer until the roundtrip marker is received, wherein the counter is incremented by one for each additional packet that is transmitter and wherein n is the value of the counter when the roundtrip marker is received.
45 INTL-7097-IN

5. The apparatus as claimed in claim 3, wherein the link interface is to determine
the n additional packet transfer cycles by performing operations comprising:
sending a first packet to the link peer, the first packet including indicia to solicit return of a roundtrip marker from the link peer in response to receipt of the first packet;
beginning at a starting slot, transmitting packets retrieved from slots in the replay buffer to the link peer, wherein each packet transfer cycle advances the slot in the replay buffer by one until a last slot in the replay buffer is reached, at which point the replay buffer slot wraps to a first slot;
while transmitting the packets, detecting receipt of the roundtrip marker from the link peer; and
if the roundtrip marker is received prior to the buffer slot returning to the starting slot, starting a counter at a value m and transmitting additional packets to the link peer until the buffer slot has returned to the starting slot, wherein the counter is decremented by one for each additional packet that is transmitted and wherein m is the value of the counter when the roundtrip marker is received.
6. The apparatus as claimed in claim 1, wherein the link interface is to:
receive and process packets transmitted from the link peer when operating in a normal active link transfer state;
detect sequential received packets are errant packets; and
in response to detecting sequential errant packets, performing at least one of taking the link offline, initiating retraining of the link, and initiating reinitialization of the link;
and wherein the link interface is to send link transfer control packets in sequential pairs such that at least one of the sequential pair of link transfer control packets is either received without error or the link is taken offline, retraining of the link is initiated, or reinitialization of the link is initiated.
7. The apparatus as claimed in claim 1, wherein the link interface is to:
receive packets transmitted from the link peer;
detect that an errant packet containing errant data has been received; and send a retry request to the link peer requesting the link peer resend the errant packet.

46

INTL-7097-IN

8. The apparatus as claimed in claim 7, wherein the link interface is to:
determine a buffer location in a replay buffer in the link peer at which a non-errant
copy of the errant packet is stored; and
embed indicia in the retry request identifying the buffer location.
9. The apparatus as claimed in claim 7, wherein the link interface is to:
in response to detecting the errant packet,
switch a state of the link interface to a tossing mode under which received packets are discarded; and
return the state of the link interface to a normal receive mode to receive a retransmission of the errant packet from the link peer.
10. The apparatus as claimed in claim 9, wherein the link interface is to:
receive a retry marker sent from the link peer while the receive port (1802) is in tossing mode; and
prepare to return the state of the link interface to a normal receive mode after receiving the retry marker, wherein the retry marker indicates how many packets following transmission of the retry marker will be transmitted from the link peer prior to retransmission of the non-errant copy of the errant packet.
11. The apparatus as claimed in claim 10, wherein the link interface is to:
in response to detecting receipt of a retry marker, transmit a roundtrip marker to the link peer.
12. The apparatus as claimed in claim 7, wherein the link interface is to:
receive a retry request sent from the link peer, the retry request identifying a slot in the replay buffer storing a non-errant copy of the packet that was received at the link peer as an errant packet; and
sequentially retransmit copies of packets in the replay buffer, beginning with the copy of the packet stored in the slot identified by the retry request.
13. The apparatus as claimed in claim 12, wherein the link interface is to transmit a
retry marker identifying a number of packets that are to be transmitted between the retry
marker and retransmission of the non-errant copy of the errant packet.

47

INTL-7097-IN

14. The apparatus as claimed in claim 1, the link interface comprising a transmit
port (1800) to transmit data over the link to the link peer and a receive port (1802) to
receive data over the link from the link peer.
15. The apparatus as claimed in claim 1, wherein the apparatus comprises a
memory (2908) and at least one processor core (2940).
16. The apparatus as claimed in claim 15, wherein the link interface, the at least one processor core and the memory are located within the same chip package.
17. The apparatus as claimed in claim 15, wherein the link interface, the at least one processor core and the memory are integrated on a System on a Chip.
18. A method, comprising:
storing, in a buffer (1806), a copy of each of a plurality of packets to be transmitted over a link to a receiver (2350);
transmitting the packets over the link to the receiver;
determining that the packets have been received without error at the receiver without use of explicit Acknowledgements (ACKs) from the receiver;
overwriting copies of previously transmitted packets for which it has been determined were received at the receiver without error,
detecting an implicit ACK that a packet has been received without error prior to overwriting the copy of the packet in the buffer, wherein the buffer comprises a circular buffer having a depth, and in which copies of packets are written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and wherein detecting an implicit ACK comprises,
tracking, for a given transmitted packet, a slot at which a copy of the packet was written to in the circular buffer;
determining that a number of slots in the circular buffer that would be written to during a link roundtrip transmission is less than or equal to a depth of the circular buffer; and
detecting that a retry request for a retransmission of a packet copied to a given slot has not been received by the time the sequence for writing copies of packets into the circular buffer has returned to the given slot.

48

INTL-7097-IN

19. The method as claimed in claim 18, wherein the buffer comprises a circular
buffer having a depth, and in which copies of reliable packets are written into respective
slots in a sequence that is restarted when the end of the circular buffer is reached, the
method comprising:
determining a number of non-reliable packets to transmit after cycling through the circular buffer;
tracking, for each transmitted reliable packet, a slot at which a copy of the reliable packet was written to in the circular buffer; and
detecting the implicit ACK by detecting that a retry request for a retransmission of a reliable packet copied to a given slot has not been received by the time the sequence for writing copies of reliable packets into the circular buffer has returned to the given slot and the determined number of non-reliable packets have been retransmitted.
20. The method as claimed in claim 18, wherein the link comprises a bi-directional
link coupled between a first link interface and a second link interface, the method
comprising:
receiving packets transmitted from the second link interface at the first link interface;
detecting that an errant packet containing errant data has been received at the first link interface; and
sending a retry request from the first link interface to the second link interface requesting the second link interface resend the errant packet.
21. The method as claimed in claim 20, comprising:
determining, at the first link interface, a buffer location in a buffer in the second link interface at which a copy of errant packet is temporarily stored; and
including indicia in the retry request indicating the buffer location.
22. A set of components, comprising:
first and second integrated circuits including respective first and second link interfaces, each of the first and second link interfaces configured to perform bi-directional communication with a link peer including a link interface with a similar configuration when installed in a system and operating, the system including a fabric (102) comprising a plurality of point-to-point fabric links (134, 1361, 136n, 138, 1401, 140n, 142, 142, 144);

49

INTL-7097-IN

wherein each of first and second integrated circuits are configured to be implemented in respective fabric endpoints coupled to a fabric link, comprise respective fabric endpoints coupled to a fabric link, or be implemented in a fabric switch to which a plurality of fabric links are coupled;
and wherein the first and second link interfaces are to perform operations when the first and second integrated circuits are implemented in the system including,
transmitting packet data from the first link interface over a fabric link to the second link interface; and
detecting at the first link interface the packet data has been received without error at the second link interface without using explicit ACKnowledgements (ACKs),
wherein the first link interface is configured to,
generate a plurality of a first type of packets;
divide each first type of packet into a plurality of data units;
bundle the data units into groups to form a plurality of a second type of packets; and transmit the plurality of second type of packets onto a fabric link coupled to the first link interface;
and wherein the second link interface is configured to,
receive second type of packets transmitted over a fabric link coupled to the second link interface;
for each second type of packet that is received, detect whether it has any errors; and extract data units from second type of packets for which an error is not detected and reassemble the data units to regenerated the plurality of first type of packets,
comprising a fabric switch having third and fourth link interfaces, wherein the first link interface that is connected to the third link interface on the fabric switch via a first fabric link and the fourth link interface on the fabric switch is connected to the second link interface via a second fabric link, and wherein,
the first link interface is configured to bundle data units from the first type of packets into a first plurality of second type of packets and transmit the first plurality of second type of packets over the first fabric link to the third link interface;
the fabric switch is configured to extract data units from the first plurality of second type of packets received at the third link interface, and rebundle the data units to generate a second plurality of second type of packets and transmit the second plurality of second type of packets from the fourth link interface over the second fabric link to the second link interface; and
50 INTL-7097-IN

the second link interface is configured to extract error-free data units from the second plurality of second type of packets and reassemble the error free data units to regenerate the first type of packets,
and wherein each of the first and fourth link interfaces are configured to determine each second type of packet transmitted from that link interface has been received without error by a peer link interface without use of explicit ACKs.
23. The set of components as claimed in claim 22, wherein each link interface is
configured to transmit reliable and non-reliable second type of packets and includes a
replay buffer in which copies of each reliable second type of packet transmitted from that
link interface is temporarily buffered, wherein each link interface is configured to send a
retry request back to a link interface peer in response to detection of a second type of
packet that has been received with an error; and
wherein each link interface is configured to retransmit copies of reliable second type of packets from its replay buffer in response to receiving a retry request from its peer link interface.
24. The set of components as claimed in claim 22, wherein the first and second integrated circuits comprise host fabric interface integrated circuits including an integrated link interface.
25. The set of components as claimed in claim 22, wherein at least one of the first and second integrated circuits comprises a System on a Chip (SoC) including a processor (2906) operatively coupled to a host fabric interface including an integrated link interface.
26. An apparatus, comprising means for:
storing, in a buffer (1806), a copy of each of a plurality of packets to be transmitted over a link to a receiver (2350);
transmitting the packets over the link to the receiver;
determining that the packets have been received without error at the receiver without use of explicit Acknowledgements (ACKs) from the receiver;
overwriting copies of previously transmitted packets for which it has been determined were received at the receiver without error; and

51

INTL-7097-IN

detecting an implicit ACK that a packet has been received without error prior to overwriting the copy of the packet in the buffer,
wherein the buffer comprises a circular buffer having a depth, and in which copies of packets are written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, and wherein detecting an implicit ACK comprises,
tracking, for a given transmitted packet, a slot at which a copy of the packet was written to in the circular buffer;
determining that a number of slots in the circular buffer that would be written to during a link roundtrip transmission is less than or equal to a depth of the circular buffer; and
detecting that a retry request for a retransmission of a packet copied to a given slot has not been received by the time the sequence for writing copies of packets into the circular buffer has returned to the given slot.
27. An apparatus, comprising means for:
storing, in a buffer (1806), a copy of each of a plurality of packets to be transmitted over a link to a receiver (2350);
transmitting the packets over the link to the receiver;
determining that the packets have been received without error at the receiver without use of explicit Acknowledgements (ACKs) from the receiver;
overwriting copies of previously transmitted packets for which it has been determined were received at the receiver without error; and
detecting an implicit ACK that a packet has been received without error prior to overwriting the copy of the packet in the buffer,
wherein the buffer comprises a circular buffer having a depth, and in which copies of reliable packets are written into respective slots in a sequence that is restarted when the end of the circular buffer is reached, the apparatus comprising means for:
determining a number of non-reliable packets to transmit after cycling through the circular buffer;
tracking, for each transmitted reliable packet, a slot at which a copy of the reliable packet was written to in the circular buffer; and
detecting the implicit ACK by detecting that a retry request for a retransmission of a reliable packet copied to a given slot has not been received by the time the sequence for

52

INTL-7097-IN

writing copies of reliable packets into the circular buffer has returned to the given slot and the determined number of non-reliable packets have been retransmitted.
28. The apparatus as claimed in claim 26, wherein the link comprises a bi¬
directional link coupled between a first link interface and a second link interface, the
apparatus comprising means for:
receiving packets transmitted from the second link interface at the first link interface;
detecting that an errant packet containing errant data has been received at the first link interface; and
sending a retry request from the first link interface to the second link interface requesting the second link interface resend the errant packet.
29. The apparatus as claimed in claim 28, comprising means for:
determining, at the first link interface, a buffer location in a buffer in the second
link interface at which a copy of errant packet is temporarily stored; and
including indicia in the retry request indicating the buffer location.
30. A method implemented in a system including a fabric (102) having a plurality
of point-to-point fabric links (134, 1361, 136n, 138, 1401, 140n, 142, 142, 144) to which
first and second fabric endpoints are coupled, the method comprising:
transmitting packet data from the first fabric endpoint to the second fabric endpoint via at least one fabric link; and
determining, for each fabric link, that packet data has been received without error using implicit ACKnowledgements (ACKs).
31. The method as claimed in claim 30, comprising:
generating a plurality of fabric packets at the first fabric endpoint; dividing each fabric packet into a plurality of flits;
bundling the data units into groups to form a plurality of a link transfer packets (LTPs); transmitting the plurality of LTPs onto a fabric link coupled to the first fabric endpoint; receiving a second plurality of LTPs from a fabric link coupled to the second fabric endpoint;
for each LTP that is received, detecting whether it has any errors; and

53

INTL-7097-IN

extracting flits from LTPs for which an error is not detected and reassembling the flits to regenerate the plurality of first fabric packets.
32. The method as claimed in claim 31, wherein the first and second fabric
endpoints include respective first and second link interfaces and the fabric includes at least
one fabric switch having a plurality of link interfaces, the method comprising:
transmitting, from a transmit port (1800) of the first link interface, the first plurality of LTPs onto a first fabric link coupled between the first link interface and a third link interface on the fabric switch;
receiving the fust plurality of LTPs at the third link interface;
checking errors for each LTP, and extracting flits from the LTPs for which an error is not detected; and
rebundling flits to form the second plurality of LTPs and transmitting the second plurality of LTPs from a transmit port (1800) of a fourth link interface on the fabric switch over a second fabric link coupled between the fourth link interface and the second link interface of the second fabric endpoint,
wherein the transmit ports for each of the first and fourth link interfaces determine each LTP transmitted from that transmit port (1800) has been received without error by a receive port (1802) of a peer link interface using implicit ACKs.
33. The method as claimed in claim 31, wherein each transmit port transmits
reliable and non-reliable LTPs and temporarily buffers each reliable LTP it transmits in a
replay buffer, wherein each link interface sends a retry request back to a link interface peer
in response to detection of an LTP has been received with an error; and
wherein each transmit port retransmits copies of reliable LTPs from its replay buffer in response to receiving a retry request at the receive port of the link interface comprising that transmit port.
Dated this 27th day of August 2019
(P. Dileep Kumar) Registration Number: IN/PA-1364 For Law Firm of Naren Thappeta Agent for Applicant

54

INTL-7097-IN