SYSTEM AND METHOD FOR POPULATING A VIRTUAL SHOPPING CART BASED ON VIDEO OF A CUSTOMER’S SHOPPING SESSION AT A PHYSICAL

STORE

TECHNICAL FIELD

This disclosure relates generally to remote monitoring techniques, and more particularly, to a system and method for populating a virtual shopping cart based on video of a customer’s shopping session at a physical store.

BACKGROUND

During a traditional shopping session in a physical store, a customer selects items from shelves located within the store and then presents those items to a cashier. The cashier generates a bill for the items and receives payment from the customer. Any cameras located within the store are typically present for security purposes.

SUMMARY

Shopping sessions in traditional stores may be associated with several inefficiencies for both the customers and the store owners. For example, during busy periods within a store, a customer may spend a considerable amount of time waiting in line to pay the cashier for the items he/she selected. The time spent waiting may even exceed the total amount of time that the customer spent selecting the items. This may lead to customer frustration and potentially to a loss of repeat customer business. As another example, traditional stores typically rely on the presence of one or more employees to act as cashiers within the stores. Even when the store is otherwise empty, such employees are nevertheless present, in case a customer happens to enter the store to make a purchase. As a result, outside of peak business hours, much of a cashier’s time within a store may be spent idle.

This disclosure contemplates a virtual store tool that addresses one or more of the above technical problems. The tool generates a virtual store configured to emulate a physical store. The tool also generates a set of videos from camera feeds received from cameras located in the physical store, to track a customer during a shopping session in the physical store. In certain embodiments, the tool then uses the virtual store and the videos of the shopping session in the physical store to generate a virtual shopping cart, storing a set of items configured to emulate the items selected by the customer in the physical store. Accordingly, the tool may use the virtual shopping cart to charge the customer for his/her purchases. In some embodiments, the tool may also be used in conjunction with an algorithm trained to determine the items selected by a customer during a shopping session in a physical store, based on inputs received from sensors located in the physical store. In such embodiments, the tool uses the virtual store and the videos of the shopping session in the physical store to verify the determination made by the algorithm. Certain embodiments of the tool are described below.

According to one embodiment, an apparatus includes an interface, a display, a memory, and a hardware processor communicatively coupled to the memory and the display. The interface receives a first video feed. The first video feed includes a first camera feed corresponding to a first camera located in a physical store and a second camera feed corresponding to a second camera located in the physical store. The first camera is directed at a first location in the physical store. The second camera is directed at a second location in the physical store. The hardware processor stores a first video segment in the memory. The first video segment is assigned to a first person and captures a portion of a shopping session of the first person in the physical store occurring during a time interval between a starting timestamp and an ending timestamp. The first video segment includes a first camera feed segment corresponding to a recording of the first camera feed from the starting timestamp to the ending timestamp, and a second camera feed segment corresponding to a recording of the second camera feed from the starting timestamp to the ending timestamp. The processor also assigns a first slider bar to the first video segment.

Playback of the first camera feed segment and the second camera feed segment is synchronized and the first slider bar controls a playback progress of the first camera feed segment and the second camera feed segment. The processor additionally displays the first camera feed segment and a first copy of the first slider bar in a first region of the display. The processor further displays the second camera feed segment and a second copy of the first slider bar in a second region of the display. The processor also receives an instruction from at least one of the first copy of the first slider bar and the second copy of the first slider bar to adjust the playback progress of the first camera feed segment and the second camera feed segment. In response to receiving the instruction, the processor adjusts the playback progress of the first camera feed segment and the second camera feed segment.

According to another embodiment, an apparatus includes a display, an interface, and a hardware processor communicatively coupled to the display. The interface receives a rack camera feed from a rack camera located in a physical store. The rack camera is directed at a first physical rack of a set of physical racks located in the physical store. The hardware processor displays, in a first region of the display, a virtual layout of a virtual store. The virtual layout is configured to emulate a physical layout of the physical store. The virtual layout includes a first virtual rack assigned to a first physical rack and a second virtual rack assigned to a second physical rack. Here, an arrangement of the first virtual rack and the second virtual rack in the virtual layout is configured to emulate an arrangement of the first physical rack and the second physical rack in the physical layout.

The processor also receives an indication of an event associated with the first physical rack. The event includes a person located in the physical store interacting with the first physical rack. In response to receiving the indication of the event associated with the first physical rack, the processor displays, in a second region of the display, the first virtual rack. The first virtual rack includes a first virtual shelf and a second virtual shelf. The first virtual shelf includes a first virtual item and the second virtual shelf includes a second virtual item. The first virtual item includes a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item includes a graphical representation of a second physical item located on a second physical shelf of the first physical rack. The processor additionally displays, in a third region of the display, a rack video segment corresponding to a recording of the rack camera feed from a starting timestamp to an ending timestamp. The rack video segment depicts the event associated with the first physical rack.

According to another embodiment, an apparatus includes a display, an interface, and a hardware processor communicatively coupled to the display. The interface receives a rack video from a rack camera located in a physical store. The rack camera is directed at a first physical rack of a set of physical racks located in the physical store. The rack camera captures video of the first physical rack during a shopping session of a person in the physical store. The processor displays, in a first region of the display, a first virtual rack that emulates the first physical rack. The first virtual rack includes a first virtual shelf and a second virtual shelf. The first virtual shelf includes a first virtual item and the second virtual shelf includes a second virtual item. The first virtual item includes a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item includes a graphical representation of a second physical item located on a second physical shelf of the first physical rack.

The processor also displays, in a second region of the display, the rack video. The rack video depicts an event including the person interacting with the first physical rack. The processor additionally displays, in a third region of the display, a virtual shopping cart. The processor further receives information associated with the event. The information identifies the first virtual item, and the rack video depicts that the person selected the first physical item while interacting with the first physical rack. In response to receiving the information associated with the event, the processor stores the first virtual item in the virtual shopping cart.

According to another embodiment, an apparatus configured to create a virtual layout of a virtual store to emulate a physical layout of a physical store includes a memory and a hardware processor communicatively coupled to the memory. The hardware processor receives a first physical position and a first physical orientation associated with a first physical rack located in the physical store. In response to receiving the first physical position and the first physical orientation, the processor places a first virtual rack at a first virtual position and with a first virtual orientation on the virtual layout. The first virtual position of the first virtual rack on the virtual layout represents the first physical position of the first physical rack on the physical layout and the first virtual orientation of the first virtual rack on the virtual layout represents the first physical orientation of the first physical rack on the physical layout. The processor also receives a first virtual item associated with a first physical item located on a first physical shelf of the first physical rack. In response to receiving the first virtual item, the processor places the first virtual item on a first virtual shelf of the first virtual rack. The first virtual shelf of the first virtual rack represents the first physical shelf of the first physical rack.

The processor additionally receives a second virtual item associated with a second physical item located on a second physical shelf of the first physical rack. In response to

receiving the second virtual item, the processor places the second virtual item on a second virtual shelf of the first virtual rack. The second virtual shelf of the first virtual rack represents the second physical shelf of the first physical rack. The processor further assigns a first rack camera located in the physical store to the first virtual rack. The first rack camera captures video that includes the first physical rack. The processor also stores the virtual layout in the memory.

According to another embodiment, an apparatus includes a hardware processor. The processor receives an algorithmic shopping cart that includes a first set of items. The first set of items is determined by an algorithm to have been selected by a first person during a shopping session in a physical store, based on a set of inputs received from sensors located within the physical store. The processor also receives a virtual shopping cart that includes a second set of items associated with the shopping session. Video of the shopping session was captured by a set of cameras located in the physical store. The video depicts the person selecting the second set of items. The processor additionally compares the algorithmic shopping cart to the virtual shopping cart. In response to comparing the algorithmic shopping cart to the virtual shopping cart, the processor determines that a discrepancy exists between the algorithmic shopping cart and the virtual shopping cart. The processor further determines a subset of the set of inputs associated with the discrepancy. The processor also attaches metadata to the subset. The metadata explains the discrepancy. The processor additionally uses the subset to train the algorithm.

Certain embodiments provide one or more technical advantages. For example, an embodiment reduces the processing resources spent when reviewing surveillance video of a customer in a store, by presenting multiple camera views of the store at once, synchronized with one another, and configured to capture the shopping session of the customer. As another example, an embodiment increases the efficiency of a shopping session through the use of automation and remote monitoring techniques. As a further example, an embodiment provides an independent verification of a machine learning tracking algorithm, configured to track a customer in a physical store. The system described in the present disclosure may particularly be integrated into a practical application of a remote monitoring system for a physical location, such as a store, where inputs from sensors located in the store may be used to monitor and track events occurring within the store.

Certain embodiments may include none, some, or all of the above technical advantages. One or more other technical advantages may be readily apparent to one skilled in the art form the figures, descriptions, and claims included herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:

FIGURES 1 A and IB present a comparison between a physical store and a virtual store;

FIGURES 2A and 2B present a comparison between a physical layout of a physical store and a virtual layout of a virtual store;

FIGURES 3A and 3B present a comparison between a physical rack in a physical store and a virtual rack in a virtual store;

FIGURE 4 illustrates an example system according to the present disclosure;

FIGURE 5 A illustrates example locations in a physical store of cameras configured to capture regions of the store for use in the system illustrated in FIGURE 4;

FIGURE 5B illustrates an example of the regions of a physical store captured by the layout cameras of the system illustrated in FIGURE 4;

FIGURE 6 illustrates the video processor component of the virtual store tool of the system illustrated in FIGURE 4;

FIGURES 7 A through 7C present an example illustrating the manner in which the virtual store tool of the system illustrated in FIGURE 4 displays camera feed segments associated with the layout cameras and the rack cameras of the system illustrated in FIGURE 4;

FIGURE 8 presents a flowchart illustrating the process by which the virtual store tool of the system illustrated in FIGURE 4 generates and displays camera feed segments associated with the layout cameras and the rack cameras of the system illustrated in FIGURE 4;

FIGURES 9A through 9D present examples illustrating the manner in which the virtual store tool of the system illustrated in FIGURE 4 may virtually emulate a shopping session occurring in a physical store;

FIGURE 10 presents a flowchart illustrating the manner in which the virtual store tool of the system illustrated in FIGURE 4 may virtually emulate a shopping session occurring in a physical store;

FIGURES 11A and 11B illustrate an example embodiment of a graphical user interface generated by the virtual store tool of the system illustrated in FIGURE 4, which may be used to generate a virtual layout configured to emulate a physical layout of a physical store;

FIGURE 12 presents a flowchart illustrating the manner in which the virtual store tool of the system illustrated in FIGURE 4 may generate a virtual layout configured to emulate a physical layout of a physical store;

FIGURES 13 A and 13B present examples of sensors that may be used to provide input to an algorithm configured to determine items selected by a customer during a shopping session in a physical store;

FIGURES 13C and 13D illustrate an example of the use of sensors coupled to a physical shelf in a physical store to define zones of the physical shelf and its corresponding virtual shelf;

FIGURE 14 illustrates a resolution component of the virtual store tool of the system illustrated in FIGURE 4;

FIGURE 15 illustrates a machine learning component of the virtual store tool of the system illustrated in FIGURE 4; and

FIGURE 16 presents a flowchart illustrating the manner by which the virtual store tool of the system illustrated in FIGURE 4 may provide feedback to an algorithm configured to determine the items selected by a customer during a shopping session in a physical store.

DETAILED DESCRIPTION

Embodiments of the present disclosure and its advantages may be understood by referring to FIGURES 1 through 16 of the drawings, like numerals being used for like and corresponding parts of the various drawings. Additional information is disclosed in U.S.

Patent Application No. _ entitled, “Scalable Position Tracking System For Tracking

Position In Large Spaces” (attorney docket no. 090278.0176) and U.S. Patent Application

No. _ entitled, “Topview Object Tracking Using a Sensor Array” (attorney docket no.

090278.0180) which are both hereby incorporated by reference herein as if reproduced in their entirety.

I. Introduction to Virtual Emulation

This disclosure is generally directed to generating a virtual store that is configured to emulate a physical store, and using the virtual store, along with videos of a shopping session occurring within the physical store, to virtually emulate the physical shopping session. Although this disclosure describes virtual emulation of a physical store, this disclosure contemplates that any type of physical space (e.g., a warehouse, a storage center, an amusement park, an airport, an office building, etc.) may be virtually emulated using the tool described in the present disclosure. For example, the physical store may be a convenience store or a grocery store. This disclosure also contemplates that the physical store may not be a physical building, but a physical space or environment in which shoppers may shop. For example, the physical store may be a grab and go pantry at an airport, a kiosk in an office building, or an outdoor market at a park, etc.

As illustrated in FIGURE 1 A, a physical store 100 is a brick and mortar store — i.e., a store that is located in a physical building. Customers 105 (who may carry mobile devices 125) enter physical store 100 to purchase items. On the other hand, a virtual store 110 is a computerized representation of a physical store, displayed on a computer or other device 115 belonging to a user 120, as illustrated in FIGURE IB. This disclosure contemplates that user 120 may use virtual store 110 to emulate a shopping session of customer 105 in physical store 100. Virtual store 110 may be generated locally on device 115 or generated remotely and transmitted over a network to device 115.

Virtual store 110 may be configured to emulate physical store 100 in several different ways. For example, in certain embodiments, and as illustrated in FIGURES 2A and 2B, the virtual layout 205 of virtual store 110 is configured to emulate the physical layout 200 of physical store 100. In particular, the shape, location, and orientation of virtual display racks 230a, 230b, 230c, and 230d are configured to emulate the shape, location, and orientation of physical display racks 210a, 210b, 210c, and 210d. For example, in the example illustrated in FIGURE 2 A, physical display racks 210a and 210b are located along back wall 235a of physical layout 200 of physical store 100. Accordingly, virtual display racks 230a and 230b are placed along back wall 240a of virtual layout 205 of virtual store 110, to emulate the location and orientation of physical display racks 210a and 210b. Similarly, virtual display rack 230d is placed along side wall 240b of virtual layout 205, to emulate the position and orientation of physical display rack 210d along side wall 235b, and virtual display rack 230c is placed in the center of virtual layout 205, to emulate the position and orientation of physical display rack 210c.

As another example, in some embodiments, the contents of virtual display racks 230a, 230b, 230c, and 230d are configured to emulate the contents of physical display racks 210a, 210b, 210c, and 210d. For example, in certain embodiments, virtual display racks 230a, 230b, 230c, and 230d are each assigned a list of items, wherein the list of items includes those items stored on physical rack 210a, 210b, 210c, and 210d, respectively. In other embodiments, each virtual display rack is assigned a set of virtual shelves, where the number and placement of the virtual shelves on the virtual display rack are configured to emulate the number and placement of the physical shelves on the corresponding physical display rack. Each virtual shelf of the set of virtual shelves then holds a set of virtual items that is configured to emulate the set of physical items stored on a corresponding physical shelf. Here the virtual items may be configured to emulate the physical items in terms of appearance and/or positioning on the virtual shelf.

As a specific example, FIGURES 3A and 3B present a comparison between physical display rack 210a and virtual display rack 230a in one embodiment. As seen in FIGURE 3A, physical display rack 210a includes two physical shelves — first physical shelf 305a and second physical shelf 305b. Accordingly, to emulate physical display rack 210a, virtual display rack 230a also includes two shelves — first virtual shelf 310a and second

virtual shelf 310b. Additionally, each of virtual shelves 310a and 310b includes a set of virtual items configured to emulate the physical items stored on the corresponding physical shelf of physical shelves 305a and 305b. For example, virtual shelf 310a includes first virtual item 320a, located in first virtual zone 330a of virtual shelf 310a, second virtual item 320b, located in second virtual zone 330b of virtual shelf 310a, and third virtual item 320c, located in third virtual zone 330c of virtual shelf 310a, positioned to emulate the positioning of first physical item 315a in first physical zone 325a of physical shelf 305a, second physical item 315b in second physical zone 325b of physical shelf 305a, and third physical item 315c in third physical zone 325c of physical shelf 305a. Similarly, virtual shelf 310b includes fourth virtual item 320d, fifth virtual item 320e, and sixth virtual item 320f, positioned, respectively, in fourth virtual zone 330d, fifth virtual zone 330e, and sixth virtual zone 330f of virtual shelf 310b, to emulate the positioning of fourth physical item 315d, fifth physical item 315e, and sixth physical item 315f in fourth physical zone 325d, fifth physical zone 325e, and sixth physical zone 325f of physical shelf 305b. Additionally, each of virtual items 320a through 320f is configured to emulate the appearance of the corresponding physical item 315a, 315b, 315c, 315d, 315e, or 315f. For example, each virtual item may correspond to a two-dimensional, graphical representation of the corresponding physical item. In this manner, a virtual item may easily be identified based on the appearance of its real world, physical counterpart.

II. System Overview

FIGURE 4 illustrates an example system 400 that includes virtual store tool 405, device 115, display 410, network 430a, network 430b, layout cameras 490, and rack cameras 495. In certain embodiments, system 400 additionally includes external system 485 and sensors 498. Generally, virtual store tool 405 is configured to generate a virtual store 110 that emulates a physical store 100. In certain embodiments, virtual store tool 405 uses virtual store 110 to generate a receipt for a shopping session conducted by a person 105 in physical store 100, based in part on videos tracking the shopping session, received from layout cameras 490 and/or rack cameras 495 located in the physical store 100. In some embodiments, virtual store tool 405 uses virtual store 110 and videos received from

layout cameras 490 and rack cameras 495 to validate a determination made by an algorithm 488 of the items selected by person 105 during the shopping session in physical store 100.

Device 115 includes any appropriate device for communicating with components of system 400 over network 430a. For example, device 115 may be a telephone, a mobile phone, a computer, a laptop, a wireless or cellular telephone, a tablet, a server, an IoT device, and/or an automated assistant, among others. This disclosure contemplates device 115 being any appropriate device for sending and receiving communications over network 430a. Device 115 may also include a user interface, such as a microphone, keypad, or other appropriate terminal equipment usable by user 120. In some embodiments, an application executed by a processor of device 115 may perform the functions described herein.

Device 115 may include or be coupled to display 410. Display 410 is a screen used by device 115 to display information received from virtual store tool 405. In certain embodiments, display 410 is a standard display used in a laptop computer. In certain other embodiments, display 410 is an external display device connected to a laptop or desktop computer. In further embodiments, display 410 is a standard touch-screen liquid crystal display found in a typical smartphone or tablet.

As illustrated in FIGURE 4, in certain embodiments, display 410 may present camera feed segments 415a through 415f, virtual layout 205, virtual rack 230, virtual shopping cart 420, and/or rack camera feed segment 425. Camera feed segments 415a through 415f are video recordings of camera feeds received by virtual store tool 405 from layout cameras 490 located in physical store 100, and are assigned to a person 105 conducting a shopping session in physical store 100. The method by which virtual store tool 405 generates camera feed segments 415a through 415f and displays camera feed segments 415a through 415f on display 410 is described in further detail below, in the discussion of FIGURES 5 through 8.

Virtual layout 205 is assigned to the particular physical store 100 from which virtual store tool 405 received the camera feeds associated with camera feed segments 415a through 415f, and is configured to emulate the physical layout 200 of that physical store. The method by which virtual store tool 405 generates virtual layout 205 is described in further detail below, in the discussion of FIGURES 11 and 12.

Virtual rack 230 corresponds to one of the virtual racks included in virtual layout 205 and is configured to emulate a physical rack 210 of physical store 100. Accordingly, virtual rack 230 displays a set of virtual items 320, with each virtual item 320 representing a physical item 315 stored on the corresponding physical rack 210. Virtual shopping cart 420 is used to hold virtual items 320, each of which represents a physical item 315 selected by person 105 during the shopping session in physical store 100. Rack camera feed segment 425 is a recording of a camera feed received by virtual store tool 405 from a rack camera 495. Rack camera 495 is directed at the physical rack 210 of physical store 100 to which virtual rack 230 is assigned. Virtual shopping cart 420 may be populated by virtual items 320 stored on virtual rack 230, based in part on rack camera feed segment 425. The method by which virtual store tool 405 determines a virtual rack 230 to display on display 410 and then uses virtual rack 230 to populate virtual shopping cart 420 is described in further detail below, in the discussion of FIGURES 9 and 10.

In some embodiments, and as described in further detail below, with respect to FIGURES 11A and 11B, display 410 displays a graphical user interface through which a user 120 may generate a virtual layout 205 configured to emulate a physical layout 200 of a physical store 100.

Network 430a facilitates communication between and amongst the various components of system 400 located outside of network 430b, connecting layout cameras 490, rack cameras 495, and external system 485 to virtual store tool 405. This disclosure contemplates network 430a being any suitable network that facilitates communication between such components of system 400. Network 430a may include any interconnecting system capable of transmitting audio, video, signals, data, messages, or any combination of the preceding. Network 430a may include all or a portion of a public switched telephone network (PSTN), a public or private data network, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a local, regional, or global communication or computer network, such as the Internet, a wireline or wireless network, an enterprise intranet, or any other suitable communication link, including combinations thereof, operable to facilitate communication between the components.

Network 430b facilitates communication between and amongst the various components of virtual store tool 405 and layout cameras 490, rack cameras 495, and

external system 485. This disclosure contemplates network 430b being any suitable network that facilitates communication between the components of virtual store tool 405 and layout cameras 490, rack cameras 495, and external system 485. Network 430b may include any interconnecting system capable of transmiting audio, video, signals, data, messages, or any combination of the preceding. Network 430b may include all or a portion of a public switched telephone network (PSTN), a public or private data network, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a local, regional, or global communication or computer network, such as the Internet, a wireline or wireless network, an enterprise intranet, or any other suitable communication link, including combinations thereof, operable to facilitate communication between the components. This disclosure contemplates that network 430b may be the same network as network 430a or a separate network from network 430a.

As seen in FIGURE 4, virtual store tool 405 includes a processor 435, a memory 440, and an interface 445. This disclosure contemplates processor 435, memory 440, and interface 445 being configured to perform any of the functions of virtual store tool 405 described herein. Generally, virtual store tool 405 implements layout creator 460, video processor 465, display controller 470, resolution component 475, and machine learning module 480. Virtual store tool 405 may use layout creator 460 to generate a virtual layout 205 configured to emulate a physical layout 200 of a physical store 100. This function of virtual store tool 405 is described in further detail below, in the discussion of FIGURES 11 and 12. Virtual store tool 405 may use video processor 465 to generate camera feed segments 415 and rack camera feed segments 425, assigned to a person 105 conducting a shopping session in physical store 100, based on camera feeds received from layout cameras 490 and rack cameras 495, respectively. This function of virtual store tool 405 is described in further detail below, in the discussion of FIGURES 5 through 8. Virtual store tool 405 may use display controller 470 to adjust the information displayed on display 410, based on input received from device 115. This function of virtual store tool 405 is described in further detail below, in the discussion of FIGURES 7 through 12. Virtual store tool 405 may use resolution component 475 to compare the contents of virtual cart 420 to an algorithmic shopping cart, determined by an algorithm 488 to contain items selected by customer 105 during a shopping session in physical store 100. Resolution component 475

may identify any discrepancies between virtual cart 420 and the algorithmic cart, resolve such discrepancies, and generate a receipt to send to customer 105. Resolution component 475 will be described in further detail below, in the discussion of FIGURE 14. Finally, virtual store tool 405 may use machine learning module 480 to identify discrepancies between virtual shopping cart 420 and the algorithmic cart and assign metadata to the algorithmic inputs associated with the discrepancies. This metadata may then be used to retrain the algorithm. Machine learning module 480 will be described in further detail below, in the discussion of FIGURES 15 and 16.

Processor 435 is any electronic circuitry, including, but not limited to microprocessors, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and/or state machines, that communicatively couples to memory 440 and controls the operation of virtual store tool 405. Processor 435 may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. Processor 435 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. Processor 435 may include other hardware and software that operates to control and process information. Processor 435 executes software stored on memory to perform any of the functions described herein. Processor 435 controls the operation and administration of virtual store tool 405 by processing information received from network 430a, network 430b, memory 440, device(s) 115, layout cameras 490, rack cameras 495, and external system 485. Processor 435 may be a programmable logic device, a microcontroller, a microprocessor, any suitable processing device, or any suitable combination of the preceding. Processor 435 is not limited to a single processing device and may encompass multiple processing devices.

Memory 440 may store, either permanently or temporarily, data, operational software, or other information for processor 435. Memory 440 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 440 may include random access memory (RAM), read only memory (ROM), magnetic storage devices, optical storage devices, or any other

suitable information storage device or a combination of these devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in memory 440, a disk, a CD, or a flash drive. In particular embodiments, the software may include an application executable by processor 435 to perform one or more of the functions described herein.

Additionally, in certain embodiments, memory 440 may store virtual layouts 205 and sets of videos 450. Each of virtual layouts 205athrough 205n corresponds to a different physical store 100 and is configured to emulate the physical layout 200 of physical store 100. Virtual layouts 205 may be stored in memory 440 according to a store identification number. In this manner, a given virtual layout 205a may be retrieved from memory 440 using the store identification number. This disclosure contemplates that set of videos 450 includes the camera feed segments 415 and rack camera feed segments 425 assigned to a given person 105, for example, through identification number 455. Such segments are video recordings of camera feeds received by virtual store tool 405 from layout cameras 490 and rack cameras 495, respectively. For example, set of videos 450 may include camera feed segments 415a through 415f and rack camera feed segments 425, assigned to a person 105. The manner in which virtual store tool 405 generates sets of videos 450 is described in further detail below, in the discussion of FIGURE 6.

Interface 445 represents any suitable device operable to receive information from networks 430a and 430b, transmit information through networks 430a and 430b, perform suitable processing of the information, communicate to other devices, or any combination of the preceding. For example, interface 445 receives camera feeds from layout cameras 490 and rack cameras 495. As another example, interface 445 receives input from device 115. Interface 445 represents any port or connection, real or virtual, including any suitable hardware and/or software, including protocol conversion and data processing capabilities, to communicate through a LAN, WAN, or other communication systems that allows virtual store tool 405 to exchange information with device 115, layout cameras 490, rack cameras 495, and/or other components of system 400 via networks 430a and 430b.

External system 485 represents any system operable to receive input from sensors 498 located in physical store 100 and to apply an algorithm 488 to this input to track customers 105 in physical store 100 and/or to determine physical items 315 selected by

such customers during shopping sessions in physical store 100. Embodiments of external system 485 are described in U.S. Patent Application No. _ entitled, “Topview Object

Tracking Using a Sensor Array” (attorney docket no. 090278.0180), the contents of which are incorporated by reference herein. This disclosure contemplates that sensors 498 may include any type of suitable sensors, located in physical store 100, and operable to detect customers 105 in physical store 100. For example, physical store 100 may include cameras, light detection and range sensors, millimeter wave sensors, weight sensors, and/or any other appropriate sensors, operable to track a customer 105 in physical store 100 and detect information associated with customer 105 selecting one or more items 315 from physical store 100. This disclosure also contemplates that algorithm(s) 488 may be any suitable algorithm(s) for tracking customers 105 in physical store 100 and determining items 315 selected by customers 105. For example, in certain embodiments, algorithm(s) 488 may be a machine learning algorithm(s).

Layout cameras 490 and rack cameras 495 are located in physical store 100. Each of layout cameras 490a through 490f is directed at a location in physical store 100 and captures video and/or images of a region in space around the location. Each of rack cameras 495 is directed at a physical display rack 210 located in physical store 100 and captures video and/or images of the physical display rack 210 and the region in space around the physical display rack 210. This disclosure contemplates that any number of layout cameras 490 may be installed in physical store 100 and connected to virtual store tool 405 through network 430b. Similarly, any number of rack cameras 495 may be installed in physical store 100 and connected to virtual store tool 405 through network 430b. For example, in some embodiments, physical store 100 contains the same number of rack cameras 495 as physical shelves 210. In other embodiments, physical store 100 contains more rack cameras 495 than physical shelves 210. In certain embodiments, rack cameras 495 are the same as layout cameras 490. In other embodiments, rack cameras 495 are distinct from layout cameras 490. The operation of layout cameras 490 and rack cameras 495 is described in further detail below, in the discussion of FIGURES 5 and 6.

Modifications, additions, or omissions may be made to the systems described herein without departing from the scope of the invention. For example, system 400 may include any number of users 120, devices 115, displays 410, networks 430a and 430b, layout

cameras 490, rack cameras 495, and external systems 485. The components may be integrated or separated. Moreover, the operations may be performed by more, fewer, or other components. Additionally, the operations may be performed using any suitable logic comprising software, hardware, and/or other logic.

III. Customer-Based Video Tracking

As described above, virtual store tool 405 may use virtual layout 205 to emulate a shopping session of a customer 105 in a physical store 100 captured by cameras feed segments 415 and/or 425. FIGURES 5 through 8 are used to describe the method by which virtual store tool 405 generates and displays camera feed segments 415 and/or 425.

a. Cameras Used for Customer-Based Video Tracking

FIGURE 5 A illustrates example locations of layout cameras 490 and rack cameras 495 in a physical store 100. The numbers of layout cameras 490 and rack cameras 495 chosen for a physical store 100 may depend on the size and/or layout of physical store 100. As seen in the example of FIGURE 5 A, physical store 100 may include five layout cameras 490a through 490e. While illustrated as located on the ceiling of physical store 100, this disclosure contemplates that layout cameras 490 may be mounted anywhere in physical store 100. Additionally, in the example of FIGURE 5 A, physical store 100 may include four rack cameras 495a through 495d. While illustrated as located both on the ceiling and sidewalls of physical store 100, this disclosure contemplates that rack cameras 495 may be mounted anywhere in physical store 100. Rack cameras 495 may be separate from layout cameras 490 or the same as layout cameras 490.

Each of rack cameras 495 is directed at a rack 210 located in physical store 100. For example, as illustrated in FIGURE 5 A, rack camera 495a is directed at physical display rack 210a, rack camera 495b is directed at physical display rack 210b, rack camera 495c is directed at physical display rack 210c, and rack camera 495d is directed at physical display rack 21 Od. While FIGURE 5 A illustrates a set of five layout cameras 490 and a set of four rack cameras 495 in physical store 100, this disclosure contemplates that any suitable number of layout cameras 490 and rack cameras 495 may be used in physical store 100,

depending on the size and/or layout of physical store 100. FIGURE 5 A additionally illustrates a set of turnstiles 510 located in physical store 100. Turnstiles 510 may be used to control the entry and exit of customers 105 into or out of physical store 100, as described in further detail below, in the discussion of FIGURE 6.

As illustrated in FIGURE 5B, each of layout cameras 490 is directed at a particular location in physical store 100 and captures a region 505 of the layout 200 of physical store 100, surrounding the location. For example, first layout camera 490a is directed at a first location and captures video and/or images of a first region 505a of physical store 100; second layout camera 490b is directed at a second location and captures video and/or images of a second region 505b of physical store 100; third layout camera 490c is directed at a third location and captures video and/or images of a third region 505 c of physical store 100; fourth layout camera 490d is directed at a fourth location and captures video and/or images of a fourth region 505d of physical store 100; and fifth layout camera 490e is directed at a fifth location and captures video and/or images of a fifth region 505e of physical store 100. In certain embodiments, layout cameras 490 may capture overlapping regions of physical store 100. For example, as illustrated in FIGURE 5B, all of third region 505c is overlapped by portions of first region 505a, second region 505b, fourth region 505d, and fifth region 505e. The overlapping regions of physical store 100 may be a result of the proximity of layout cameras 490 to one another. Generally, by capturing overlapping regions of physical store 100, certain portions of physical layout 200 can be captured by multiple layout cameras 490. This may be desirable, to provide sufficient camera coverage of physical layout 200 in the event that certain of layout cameras 490 malfunction or go offline.

While illustrated in FIGURE 5B as rectangular in shape, this disclosure contemplates that regions 505 may be of any shape or size. For example, in certain embodiments, regions 505 are elliptical in shape. In some embodiments, regions 505 are of uniform size and shape. For example, as illustrated in FIGURE 5B, regions 505a through 505e are all the same shape and size. In other embodiments, regions 505 may include regions 505 of different sizes and shapes.

b. Camera Feed Processing

The videos and/or images of physical store 100 captured by layout cameras 490 and/or rack cameras 495 are transmitted to virtual store tool 405 in the form of camera feeds. Virtual store tool 405 then uses video processor 465 to generate camera feed segments 415 and rack camera feed segments 425, assigned to a person 105 conducting a shopping session in physical store 100, based on these camera feeds. FIGURE 6 illustrates the operation of video processor 465 of virtual store tool 405.

FIGURE 6 presents an example of the operation of video processor 465 of virtual store tool 405, in an embodiment that includes a first layout camera 490a, a second layout camera 490b, and a rack camera 495a. As illustrated in FIGURE 6, video processor 465 receives first camera feed 605a from first layout camera 490a, second camera feed 605b from second layout camera 490b, and rack camera feed 620a from rack camera 495a. In certain embodiments, video processor 465 receives first camera feed 605a, second camera feed 605b, and rack camera feed 620a directly from layout cameras 490a, 490b, and rack camera 495a. In some embodiments, video processor 465 receives first camera feed 605a, second camera feed 605b, and rack camera feed 620a from interface 445.

Prior to processing camera feeds 605a, 605b, and 620a, video processor 465 first determines that a person 105, associated with an identification number 455, entered physical store 100. This disclosure contemplates that video processor 465 may determine that person 105 entered physical store 100 in any suitable manner. For example, in certain embodiments, physical store 100 includes turnstiles 510, which control the entry of persons 105 into the store. A turnstile 510 may open upon person 105 scanning a QR code, located on a physical card or a mobile device 125 belonging to person 105, using a scanner 515 attached to the turnstile 510. Accordingly, the scanning of the QR code may generate a notification, sent to virtual store tool 405, indicating that person 105 entered physical store 100. As another example, in some embodiments, an algorithm 488 may be used to determine that person 105 entered physical store 100, based on information received from sensors 498 located in physical store 100. An example of such an algorithm 488 will be described in further detail below, in the discussion of FIGURES 13 through 16.

This disclosure contemplates that camera feeds 605 and 620 are synchronized in terms of timestamps, such that video associated with a given timestamp from each of camera feeds 605a, 605b, and 620a corresponds to the same real time within physical store 100. Such synchronization may be achieved in any suitable manner. For example, in certain embodiments, layout cameras 490 and rack cameras 495 are plugged into the same ethemet switch. Determining that person 105 entered physical store 100 may then include receiving a starting timestamp 610 corresponding to the timestamp at which person 105 entered physical store 100.

Given that data packets associated with first camera feed 605 a, second camera feed 605b, and rack camera feed 620a may arrive at virtual store tool 405 over network 430b at different times, this disclosure contemplates that rather than virtual store tool 405 streaming first camera feed 605a, second camera feed 605b, and rack camera feed 620a from starting timestamp 610 onwards, video processor 465 of virtual layout tool 405 stores recordings of first camera feed 605 a, second camera feed 605b, and rack camera feed 620a, lasting a predefined amount of time, in memory 440. Such recordings may then be replayed, each synchronized with the others according to timestamps. Accordingly, once video processor 465 determines starting timestamp 610, corresponding to the timestamp at which person 105 entered physical store 100, video processor 465 next prepares segments of each camera feed, starting at starting timestamp 610 and ending at ending timestamp 615. Video processor 465 then stores these segments in memory 440. For example, video processor 465 prepares first camera feed segment 415a, corresponding to a recording of first camera feed 605a from starting timestamp 610 to ending timestamp 615, second camera feed segment 415b, corresponding to a recording of second camera feed 605b from starting timestamp 610 to ending timestamp 615, and rack camera feed segment 425a, corresponding to a recording of rack camera feed 620a from starting timestamp 610 to ending timestamp 615. Video processor 465 then stores each of segments 415a, 415b, and 425a in memory 450.

This disclosure contemplates that the time interval between starting timestamp 610 and ending timestamp 615 may be any predetermined amount of time. For example, in certain embodiments, the time interval is five minutes. In order to capture video of a shopping session lasting more than this predetermined amount of time, once camera feeds

605a, 605b, and 620a reach ending timestamp 615, video processor 465 may store additional recordings of camera feeds 605a, 605b, and 620a, starting at ending timestamp 615 and ending at a new ending timestamp, the new ending timestamp occurring at the predetermined amount of time after ending timestamp 615. Video processor 465 may store any number of additional camera feed segments in memory 440, each corresponding to an additional predetermined interval of time. In certain embodiments, video processor 465 continues to record such additional camera feed segments until it receives an indication that person 105 has left physical store 100.

Video processor 465 may store camera feed segments 415 and 425 for any number of persons 105. Accordingly, video processor 465 may store a collection of camera feed segments 415 and 425 assigned to a person 105 as set of videos 450, where set of videos 450 is assigned identification number 455 associated with person 105. As an example, a first person 105a may enter physical store 100 at a first starting timestamp 610a and a second person 105b may enter physical store 100 at a second starting timestamp 610b after the first starting timestamp 610a, wherein the second starting timestamp 610b is within the predefined time interval after first starting timestamp 610a, such that the camera feed segments recorded for first person 105a will contain video that overlaps with the camera feed segments recorded for second person 105b. Accordingly, video processor 465 may store the camera feed segments recorded for first person 105a, along with an identification number 455a, assigned to first person 105a, in memory 440, as set of videos 450a. Similarly, video processor 465 may store the camera feed segments recorded for second person 105b, along with an identification number 455b, assigned to second person 105b, in memory 440, as set of videos 450b. Virtual store tool 405 may then retrieve from memory 440 the camera feed segments associated with a given person 105, using the identification number 455 assigned to that person.

Video processor 465 may be a software module stored in memory 440 and executed by processor 435. An example of the operation of video processor 465 is as follows: (1) receive camera feeds 605 and 620 from cameras 490 and 495, respectively; (2) determine that a person 105 entered physical store 100; (3) determine the timestamp 610 corresponding to the time at which person 105 entered physical store 100; (4) record camera feed segments 415 and 425 from camera feeds 605 and 620, respectively, where the camera feed segments correspond to recordings of camera feeds 605 and 620 from timestamp 610, corresponding to the time at which person 105 entered physical store 100, and lasting a predetermined amount of time to ending timestamp 615; and (5) store camera feed segments 415 and 425 in memory 440 according to an identification number 455 of person 105, as set of videos 450.

c. Displaying Camera Feed Segments

Once video processor 465 has recorded set of videos 450 from camera feeds 605 and 620, virtual store tool 405 may then use display controller 470 to display set of videos 450 on display 410 of device 115. In certain embodiments, virtual store tool 405 may display set of videos 450 on display 410 of device 115 in the form of a graphical user interface 700. FIGURES 7A through 7C present an example illustrating the manner in which virtual store tool 405 displays set of videos 450 on display 410.

FIGURE 7A illustrates an embodiment in which virtual store tool 405 instructs display 410 to display four camera feed segments 415a through 415d. Virtual store tool 405 displays first camera feed segment 415a in a first region 750 of display 410, second camera feed segment 415b in a second region 755 of display 410, third camera feed segment 415c in a third region 760 of display 410, and fourth camera feed segment 415d in a fourth region 765 of display 410. Virtual store tool 405 may instruct display 410 to display any number of camera feed segments 415. For example, in certain embodiments, virtual display tool 405 may instruct display 410 to display the same number of camera feed segments 415 as stored in set of videos 450. In some embodiments, virtual display tool 405 may instruct display 410 to display fewer camera feed segments 415 than stored in set of videos 450. This may be desirable in embodiments in which physical store 100 is a large store that includes a large number of layout cameras 490. In such embodiments, displaying all of camera feed segments 415 on display 410 may make it difficult for a user 120 to view specific features of physical store 100 in any one of the displayed camera feed segments 415. Accordingly, virtual store tool 405 may display a subset of camera feed segments 415 on display 410. Virtual store tool 405 may select a subset of camera feed segments 415 to display on display 410 in any suitable manner. As an example, in certain embodiments,

virtual store tool 405 may display a subset of camera feed segments 415 that includes, at any given time, those camera feed segments 415 capturing regions of physical store 100 closest to the location of person 105, to whom set of videos 450 is assigned. In such embodiments, when set of videos 450 depicts person 105 moving to a new location in physical store 100, virtual store tool 405 may replace the subset of camera feed segments 415 currently displayed on display 410 with a new subset of camera feed segments 415, which includes those camera feed segments 415 that capture regions of physical store 100 closest to the new location of person 105. Virtual store tool 405 may determine the subset of camera feed segments 415 that capture regions of physical store 100 closest to the location or person 105 in any suitable manner. For example, in certain embodiments, virtual store tool 405 may receive an indication of the location of person 105 from a machine-learning algorithm 488 configured to track the locations of a person 105 in physical store 100, based on inputs received from a set of sensors 498 located in physical store 100.

As illustrated in FIGURE 7 A, in addition to displaying camera feed segments 415, virtual store tool 405 also assigns a slider bar 705 to set of videos 450 and displays copies of slider bar 705 along with each camera feed segment 415. For example, virtual store tool 405 displays a first copy 705a of slider bar 705 along with first camera feed segment 415a, a second copy 705b of slider bar 705 along with second camera feed segment 415b, a third copy 705c of slider bar 705 along with third camera feed segment 415c, and a fourth copy 705d of slider bar 705 along with fourth camera feed segment 415d. Each copy of slider bar 705 may contain a slider 710 configured to control the playback progress of the associated camera feed segment 415. For example, the position of slider 710 on slider bar 705 indicates the current playback progress of the associated camera feed segment 415. The position of slider 710 may be manually adjusted (e.g., by auser 120) to anew position corresponding to a new playback time. Such adjustment may result in the playback of the associated camera feed segment adjusting to the new playback time.

In certain embodiments, the playback of each camera feed segment 415 is synchronized with that of the other camera feed segments 415, such that an adjustment of the slider 710 on any of the copies of slider bar 705 leads to a corresponding adjustment of the playback progress of all of the displayed camera feed segments 415. For example, if slider 710 is adjusted on first copy 705a of slider bar 705 from a first playback time to a second playback time, slider 710 on second copy 705b of slider bar 705, slider 710 on third copy 705 c of slider bar 705, and slider 710 on fourth copy 705d of slider bar 705 will all similarly adjust from the first playback time to the second playback time. This may be desirable for a user 120 using camera feed segments 415 to observe a shopping session of a customer 105 in physical store 100. User 120 may adjust the playback progress of camera feed segments 415 until user 120 determines that camera feed segments 415 have reached a point of interest to user 120, rather than viewing the entire, uninterrupted playback of camera feed segments 415.

In certain embodiments, slider bar 705 may include one or more markers 715. For example, as illustrated in FIGURE 7A, slider bar 705 may include a first marker 715a, located at a first marker position on slider bar 705 and corresponding to a first marker playback time, as well as a second marker 715b, located at a second marker position on slider bar 705 and corresponding to a second marker playback time. First marker 715a is associated with a first event occurring at the first marker playback time and second marker 715b is associated with a second event occurring at the second marker playback time. The first event and the second event may include any type of events occurring within physical store 100. For example, the first event may be associated with a person 105a selecting a physical item 315a from a physical shelf 305a located in a physical rack 210a in physical store 100. Similarly, the second event may be associated with person 105a selecting a second physical item 315b from a second physical shelf 305b located in a second physical rack 210b in physical store 100.

The locations for first marker 715a and second marker 715b on slider bar 705 may be determined in any suitable manner. As an example, in certain embodiments, the first event, associated with first marker 715a, and the second event, associated with second marker 715b, may be determined by an algorithm 488, based on a set of inputs received from sensors 498 located within physical store 100. For example, algorithm 488 may determine that the first event takes place at a first time, corresponding to a first timestamp, and that the second event takes place at a second time, corresponding to a second timestamp. Virtual store tool 405 may then use the first and second timestamps to place first marker 715a and second marker 715b on slider bar 705, at positions corresponding to the timestamps. An example algorithm 488, used to determine the timing of the first and second events, is described in further detail below, in the discussion of FIGURES 13 through 16. The use of markers 715 may be desirable for a user 120 using camera feed segments 415 to observe a shopping session of customer 105 in physical store 100. Rather than viewing the entire, uninterrupted playback of camera feed segments 415, user 120 may adjust the playback progress of camera feed segments 415 until slider 710 reaches one of the events associated with first marker 715a or second marker 715b, to, for example, observe customer 105 selecting a physical item 315 from a physical rack 210 in physical store 100.

As described above, in the discussion of FIGURE 6, each of camera feed segments 415 is of a predetermined time interval, lasting from a starting timestamp 610 to an ending timestamp 615. Accordingly, in certain embodiments in which customer 105 remains within physical store 100 for longer than the predetermined time interval, multiple camera feed segments may exist, from each of layout cameras 490. For example, virtual store tool 405 may store in memory 440 camera feed segments 415 for a first time interval, a second time interval, a third time interval, and a fourth time interval. Memory 440 stores any number of camera feed segments 415 for any number of time intervals. In such embodiments, when slider 710 reaches the end of slider bar 705, virtual store tool 405 may replace those camera feed segments 415 currently displayed on display 410, with the next set of camera feed segments 415, corresponding to the time interval immediately following the time interval captured by the currently displayed set of camera feed segments 415. This process of replacing the currently displayed camera feed segments 415 with a new set of camera feed segments 415, corresponding to the time interval immediately following the time interval captured by the currently displayed set of camera feed segments 415 may continue until virtual store tool 405 determines that customer 105 has left physical store 100.

Virtual store tool 405 may determine that customer 105 has left physical store 100 in any suitable manner. As an example, in certain embodiments, virtual store tool 405 may determine that customer 105 has left physical store 100 based on input received from user 120. For example, in embodiments in which set of videos 450 are displayed on display 410 in the form of a graphical user interface 700, the graphical user interface 700 may include an interactive button 730 (e.g., an exit customer button) through which user 120 may indicate that he/she observed customer 105 exiting physical store 100, on camera feed segments 415, as illustrated in FIGURE 7B. As another example, virtual store tool 405 may determine that customer 105 has left physical store 100 based on information received from an algorithm 488 configured to track customers 105 within physical store 100. Such as algorithm 488 is described in further detail below, in the discussion of FIGURES 13 through 16. As a further example, virtual store tool 405 may determine that customer 105 has left physical store 100 based on information received from physical store 100. For example, physical store 100 may include a set of turnstiles 510 near the exit of physical store 100. In order to open a turnstile 510 and leave physical store 100, a customer 105 may be asked to scan the same QR code that he/she used to enter physical store 100. Scanning the QR code may then send a signal to virtual store tool 405, indicating that customer 105 has exited physical store 100.

In certain embodiments, in order to assist a user 120 in determining which of camera feed segments 415 may include information of interest, virtual store tool 405 is configured to highlight certain camera feed segments 415, at certain times, based on events depicted in those camera feed segments 415, at those certain times. For example, as illustrated in FIGURE 7B, virtual store tool 405 may be configured to determine that a given camera feed segment 415a depicts customer 105 at a first time. Accordingly, virtual store tool 405 may highlight camera feed segment 415a in response to determining that slider 710 on slider bar 705 reached that first time. Here, highlighting camera feed segment 415a may include any manner by which virtual store tool 405 may draw attention toward camera feed segment 415a. For example, as illustrated in FIGURE 7B, highlighting camera feed segment 415a may include placing a frame 720 around camera feed segment 415a. As another example, highlighting camera feed segment 415a may include increasing the size of camera feed segment 415a, depicted on display 410, relative to the other camera feed segments 415.

In certain embodiments, the graphical user interface 700 displayed on display 410 may be used by a user 120 to monitor a shopping session of a customer 105a in physical store 100. To aid such a user 120 in monitoring a particular customer 105a in a physical store that includes several other customers 105, virtual store tool 405 may additionally display an image 725 of customer 105a, captured when customer 105a entered physical store 100. For example, in certain embodiments in which physical store 100 includes turnstiles 510 to control the entry of persons 105 into the store, physical store 100 may include a camera configured to take an image 725 of customer 105a as customer 105a passes through a turnstile 510.

In certain embodiments in which slider bar 705 includes one or more markers 715, each marker 715 may include metadata 740 describing the event associated with the marker 715. An example of one such embodiment is illustrated in FIGURE 7C. As described above, in the discussion of FIGURE 7A, each marker 715a and 715b may be associated with an event consisting of customer 105a selecting a physical item 315 from a physical shelf 305 of a physical rack 210 located in physical store 100. Accordingly, each marker may include metadata 740 indicating an identification number 745 assigned to the physical item 315 selected by customer 105a, an identification number 750 assigned to the physical shelf 305 from which customer 105a selected the physical item 315, and/or an identification number 755 assigned to the physical rack 210 that includes the physical shelf 305 from which customer 105a selected the physical item 315. In certain embodiments, item identification number 745 may correspond to a zone identification number 745, identifying a zone of physical shelf 305 from which customer 105a selected the physical item 315. The use of shelf zones will be described in further detail below, in the discussion of FIGURES 13C and 13D.

Virtual store tool 405 may use metadata 740 in any suitable manner. For example, in certain embodiments, when slider 710 on slider bar 705 reaches first marker 715a, virtual store tool 405 may use metadata 740 to determine that customer 105 selected a physical item 315 from physical rack 210. Accordingly, virtual store tool 405 may display rack camera segment 425a on display 410, where rack camera segment 425a depicts video of physical rack 210. Rack camera segment 425a may be synchronized with camera feed segments 415athrough 415d, such that an adjustment of the slider 710 on any of the copies of slider bar 705 leads to a corresponding adjustment of the playback progress of rack camera segment 425a. Automatically displaying rack camera segment 425a, in response to slider 710 reaching marker 715 on slider bar 705 may be desirable, to provide a user 120 with a view of physical rack 210 through which user 120 is able to observe customer 105 selecting a physical item 315 from physical rack 210. In certain embodiments, user 120 may be able to use a second graphical user interface to choose a rack camera 495 from among several potential rack cameras 495 to assign to physical rack 210, to provide user 120 with a rack camera segment 425a that displays the best view of physical rack 210, as determined by user 120. This aspect of virtual store tool 405 will be described in further detail below, in the discussion of FIGURES 11 and 12.

FIGURE 8 presents a flowchart illustrating the process by which virtual store tool 405 generates camera feed segments 415 and 425 and displays such segments on display 410. In step 805, virtual store tool 405 receives a set of layout camera feeds 605 from a set of layout cameras 490 and a set of and rack camera feeds 620 from a set of rack cameras 495 located in physical store 100. In step 810, virtual store tool 405 determines whether a person 105 entered physical store 100. This disclosure contemplates that virtual store tool 405 may determine that person 105 entered physical store 100 in any suitable manner. For example, in certain embodiments, physical store 100 includes turnstiles 510, which control the entry of persons 105 into the store. A turnstile 510 may be opened upon person 105 scanning a QR code, located on a physical card or a mobile device 125 belonging to person 105. Accordingly, the scanning of the QR code may generate a notification, sent to virtual store tool 405, to indicate that person 105 entered physical store 100. As another example, in some embodiments, an algorithm 488 may be used to determine that person 105 entered physical store 100, based on information received from sensors 498 located in physical store 100.

If, in step 810, virtual store tool 405 determines that person 105 entered physical store 100, in step 815, virtual store tool 405 stores a set of camera feed segments 415 and 425 in memory 440. Each camera feed segment of camera feed segments 415 corresponds to a recording of one of the camera feeds 605 from a starting timestamp 610 to an ending timestamp 615. Similarly, each rack camera feed segment of rack camera feed segments 425 corresponds to a recording of one of the rack camera feeds 620 from starting timestamp 610 to ending timestamp 615. Starting timestamp 610 corresponds to the time at which person 105 entered physical store 100. Ending timestamp 615 corresponds to a predetermined time interval after starting timestamp 610.

In step 820, virtual store tool 405 assigns copies of a slider bar 705 to each camera feed segment 415 and 425. Slider 710 on each copy of slider bar 705 moves forward as the corresponding camera feed segment 415 and/or 425 progresses. In certain embodiments, the copies of slider bar 705 are synchronized with one another such that all of camera feed segments 415 and 425 progress together, at the same pace. Additionally, in such embodiments, an adjustment of the slider 710 on any of the copies of slider bar 705 leads to a corresponding adjustment of the playback progress of all of camera feed segments 415 and 425. This may be desirable for a user 120 using camera feed segments 415 to observe a shopping session of a customer 105 in physical store 100. User 120 may adjust the playback progress of camera feed segments 415 until user 120 determines that camera feed segments 415 have reached a point of interest to user 120, rather than viewing the entire, uninterrupted playback of camera feed segments 415.

In step 825, virtual store tool 405 presents one or more camera feed segments 415 and/or 425 on display 410, along with corresponding copies of slider bar 705. For example, virtual store tool 405 may display first camera feed segment 415a, along with first copy 705a of slider bar 705 in a first region of display 410, second camera feed segment 415b, along with second copy 705b of slider bar 705 in a second region of display 410, third camera feed segment 415c, along with third copy 705c of slider bar 705 in a third region of display 410, and fourth camera feed segment 415d, along with fourth copy 705d of slider bar 705 in a fourth region of display 410. Virtual store tool 405 additionally plays camera feed segments 415 and/or 425, such that slider 710 on each copy of slider bar 705 progresses.

In step 830, virtual store tool 405 next determines whether an adjustment occurred for any slider 710 in a copy of slider bar 705, from a first position on slider bar 705 to a second position on slider bar 705, where the first position corresponds to a first playback time and the second position corresponds to a second playback time. If, in step 830, virtual store tool 405 determines that an adjustment occurred, virtual store tool 405 next adjusts the playback progress of each of camera feed segments 415 and 425 from the first playback time to the second playback time.

In step 840, virtual store tool 405 determines whether person 105 has left physical store 100. Virtual store tool 405 may determine that customer 105 has left physical store 100 in any suitable manner. As an example, in certain embodiments, virtual store tool 405 may determine that customer 105 has left physical store 100 based on input received from user 120. For example, in embodiments in which camera feed segments 415 and/or 425 are displayed on display 410 in the form of a graphical user interface 700, the graphical user interface 700 may include an interactive button 730 (e.g., an exit customer button) through which user 120 may indicate that he/she observed customer 105 exiting physical store 100 on one or more camera feed segments 415. As another example, virtual store tool 405 may determine that customer 105 has left physical store 100 based on information received from an algorithm 488 configured to track customers 105 within physical store 100. Such as algorithm 488 is described in further detail below, in the discussion of FIGURES 13 through 16. As a further example, virtual store tool 405 may determine that customer 105 has left physical store 100 based on information received from physical store 100. For example, physical store 100 may include a set of turnstiles 510 near the exit of physical store 100. In order to open a turnstile 510 and leave physical store 100, a customer 105 may be asked to scan the same QR code that he/she used to enter physical store 100. Scanning the QR code may then send a signal to virtual store tool 405, indicating that customer 105 has exited physical store 100.

If, in step 840, virtual store tool 405 determines that person 105 has not left physical store 100, in step 845, virtual store tool 405 determines whether camera feed segments 415 and 425 have reached ending timestamp 615. If, in step 845, virtual store tool 405 determines that camera feed segments 415 and 425 have not reached ending timestamp 615, virtual store tool returns to step 830, to determine whether an adjustment occurred for any slider 710 in a copy of slider bar 705, from a first position on slider bar 705 to a second position on slider bar 705. On the other hand, if, in step 845, virtual store tool 405 determines that camera feed segments 415 and 425 have reached ending timestamp 615, virtual store tool 405 returns to step 825 and displays a new set of camera feed segments 415 and/or 425 on display 410, where the new set of camera feed segments corresponds to recordings of camera feeds 605 and/or 620 over a time interval immediately following the previous time interval associated with the previous set of camera feed segments 415 and/or 425.

Modifications, additions, or omissions may be made to method 800 depicted in FIGURE 8. Method 800 may include more, fewer, or other steps. For example, steps may be performed in parallel or in any suitable order. While discussed as virtual store tool 405 (or components thereof) performing the steps, any suitable component of system 400, such as device(s) 115 for example, may perform one or more steps of the method.

IV. Virtual Emulation of a Shopping Session

As described above, camera feed segments 415 and 425 may be used in conjunction with virtual layout 205 in order to virtually emulate a shopping session occurring in physical store 100 and captured by camera feed segments 415 and/or 425. For example, in certain embodiments, camera feed segments 415 and 425, along with virtual layout 205, may be presented to a user 120, in the form of a graphical user interface 700. Here, camera feed segments 415 and 425 may be assigned to a customer 105 and capture a shopping session of customer 105 in physical store 100. User 120 may monitor camera feed segments 415 and 425 to view customer 120 selecting physical items 315 from physical racks 210. Accordingly, user 120 may populate a virtual shopping cart 420 with virtual items 320 that represent the physical items 315 selected by customer 105, such that at the end of customer 105’s shopping session, virtual shopping cart 420 may include a virtual item 320 for each physical item 315 selected by customer 105.

FIGURES 9A through 9D present further examples of a graphical user interface 700, displayed on display 410, that may be used to virtually emulate a shopping session occurring in physical store 100 and captured by camera feed segments 415 and 425. As illustrated in FIGURE 9A, virtual store tool 405 may display camera feed segments 415 in a first region 955 of display 410, as described above in the discussion of FIGURES 7A through 7C. Virtual store tool 405 may additionally display virtual layout 205 in a second region 960 of display 410. Virtual layout 205 is configured to emulate the physical layout 200 of physical store 100. As illustrated in FIGURE 9A, virtual layout 205 includes a set of virtual racks 230. This disclosure contemplates that virtual layout 205 may include any number of virtual racks 230, where the number of virtual racks 230 displayed on virtual layout 205 corresponds to the number of physical racks 210 in physical store 100. The

layout of virtual racks 230 in virtual layout 205 is configured to emulate the arrangement of the corresponding physical racks 210 in physical store 100.

a. Receiving an Indication of an Event

As illustrated in FIGURE 9B, virtual store tool 405 may receive an indication of an event associated with a physical rack 210a located in physical store 100. In certain embodiments, the event associated with physical rack 210a may include customer 105 interacting with physical rack 210a. For example, the event associated with physical rack 210a may include customer 105a approaching physical rack 210a, and/or selecting a physical item 315f from physical rack 210a. The indication of the event may include any suitable indication received by virtual store tool 405. For example, in certain embodiments, the indication of the event may include user 120 selecting virtual shelf 230a in virtual layout 205, in response to viewing customer 105 approaching and/or interacting with physical rack 210a. As another example, the indication of the event may include slider 710 on slider bar 705 reaching a marker 715, where the marker 715 indicates the physical rack 210 associated with the event, through metadata 740. As a further example, in certain embodiments, the indication of the event may include receiving information from an algorithm 488 configured to determine that customer 105 approached and/or selected an item 315 from physical rack 210a, based on inputs received from sensors 498 located in physical store 100

In certain embodiments, in which the graphical user interface 700 displayed on display 410 may be used by a user 120 to monitor a shopping session of a customer 105 in physical store 100, virtual store tool 405 may display a predicted location 950 of customer 105 on virtual layout 205, based on the current playback progress of camera feed segments 415 and/or 425. Predicted location 950 may correspond to the probable location of customer 105 in physical layout 200, as determined by an algorithm 488 configured to track customers 105 in physical store 100, based on inputs received from sensors 498 located in physical store 100, at a physical time corresponding to the current playback progress of camera feed segments 415 and/or 425. This may aid a user 120 in monitoring a particular customer 105a in a physical store that includes several other customers 105. While illustrated in FIGURE 9B as dot 950 on virtual layout 205, the predicted location of

customer 105 may be presented on virtual layout 205 in any suitable manner. For example, the predicted location may be a line, including the predicted path of customer 105. In such embodiments, the indication of the event may include user 120 selecting virtual shelf 230a in virtual layout 205, in response to viewing customer 105 approaching and/or interacting with physical rack 210a and/or viewing predicted location 950 of customer 105 on virtual layout 205 indicating customer 105’s proximity to physical rack 210a.

In response to receiving the indication of the event, virtual store tool 405 may display the virtual rack 230a corresponding to the physical rack 210a associated with the event, in a third region 905 of display 410, where virtual rack 230a is configured to emulate physical rack 210a. In certain embodiments, third region 905 of display 410 may be located to the right of virtual layout 205. In certain embodiments, virtual store tool 405 may additionally highlight virtual rack 230a, in virtual layout 205, in response to receiving the indication of the event associated with physical rack 210a. Highlighting virtual rack 230a may include any method of distinguishing virtual rack 230a from the other virtual racks 230b through 230k. For example, as illustrated in FIGURE 9B, highlighting virtual rack 230a may include placing a frame around virtual rack 230a. Highlighting virtual rack 230a may additionally include applying a color to virtual rack 230a, and/or any other suitable method of distinguishing virtual rack 230a from the remaining virtual racks 230b through 230k.

As illustrated in FIGURE 9B, virtual rack 230a, displayed in third region 905 of display 410 includes a set of virtual items 320a through 320h. Virtual items 320a through 320h are configured to emulate the physical items stored on physical rack 210a. In certain embodiments, virtual items 320a through 320h are displayed in third region 905 as a list of items, where the names of the items in the list correspond to the names of the physical items 315a through 315h stored on physical rack 210a. In other embodiments, the appearance of virtual rack 230a, displayed in third region 905, is configured to emulate the appearance of physical rack 210a. For example, first virtual shelf 310a is configured to emulate first physical shelf 305a, second virtual shelf 310b is configured to emulate second physical shelf 305b, and third virtual shelf 310c is configured to emulate third physical shelf 305c. In particular, first virtual item 320a is located in a first zone 330a of first virtual shelf 310a to emulate the location of first physical item 315a in a first zone 325a of first physical shelf 305a. Similarly, second virtual item 320b is located in a second zone 330b of first virtual shelf 310a, to the right of first virtual item 320a, to emulate the location of second physical item 315b in a second zone 325b of first physical shelf 305a, and third virtual item 320c is located in a third zone 330c of first virtual shelf 310a, to the right of second virtual item 320b, to emulate the location of third physical item 315c in a third zone 325c of first physical shelf 305a. Virtual items 320d through 320f are similarly located on second virtual shelf 310b to emulate the locations of the physical items 315d through 315f, located on second physical shelf 305b, and virtual items 320g and 320h are located on third virtual shelf 310c to emulate the locations of physical items 315g and 315h located on third physical shelf 305c. To further emulate physical items 315, each of virtual items 320 may include a graphical representation of the corresponding physical item 315.

In addition to displaying virtual rack 230a in region 905 of display 410, in response to receiving the indication of the event associated with physical rack 210a, virtual store tool 405 may also display rack camera segment 425a in a fourth region 970 of display 410, as illustrated in FIGURE 9C. In certain embodiments, the fourth region 970 of display 410 is to the right of third region 905. Rack camera segment 425a depicts physical rack 210a, during the time interval in which the event occurs. For example, in embodiments in which the event includes customer 105 approaching physical rack 210a, rack camera segment 425a depicts customer 105 approaching physical rack 210a. As another example, in embodiments in which the event includes customer 105 selecting an item 315f from physical rack 210a, rack camera segment 425a depicts customer 105 selecting item 315f from physical rack 210a.

Rack camera segment 425a may be synchronized with camera feed segments 415a through 415f, such that an adjustment of the slider 710 on any of the copies of slider bar 705 leads to a corresponding adjustment of the playback progress of rack camera segment 425a. Displaying rack camera segment 425a, in response to receiving the indication of the event may be desirable, to provide a user 120 with a view of physical rack 210a through which user 120 is able to observer customer 105 approaching and/or interacting with physical rack 210a. For example, rack camera segment 425a may help user 120 to see if customer 105 selected an item 315 from physical rack 210a. User 120 may then use this information to populate virtual cart 420, as described in further detail below, in the

discussion of FIGURE 9D. In certain embodiments, user 120 may be able to select a rack camera 495 to assign to physical rack 210 to provide user 120 with a rack camera segment 425a that displays the best view of physical rack 210a, as determined by user 120. This aspect of virtual store tool 405 will be described in further detail below, in the discussion of FIGURES 11 and 12.

b. Receiving Information Identifying a Selected Item

In certain embodiments in which the event includes person 105 selecting an item from physical shelf 210a, the indication of the event may include information identifying the item selected by person 105. For example, if the event includes person 105 selecting physical item 315f from physical rack 210a, the indication of the event received by virtual store tool 405 may include information identifying physical item 315f and/or virtual item 320f. As an example, in certain embodiments, each physical shelf 305 of physical rack 210a includes a set of weight sensors 1300, coupled to zones 325 of the physical shelf 305, as described below, in the discussion of FIGURES 13B through 13D. When person 105 removes an item 315 from physical shelf 305, the weight sensor 1300 coupled to the zone 325 of physical shelf 305 on which the item 315 is located may send information to virtual store tool 405 (either directly, or through other components of system 400, such as external system 485), indicating that the item 315 has been selected from physical shelf 305 of physical rack 210a. Virtual store tool 405 may use this information to highlight the corresponding virtual item 320 on virtual rack 230a, displayed in third region 905 of display 410. For example, a weight sensor coupled to a third zone of second physical shelf 305b of physical rack 210a may send information to virtual store tool 405 indicating that item 315f has been removed from the third zone of second physical shelf 305b of physical rack 210a.

As another example, in certain embodiments, the indication of the event may include slider 710 on slider bar 705 reaching a marker 715. Markers 715 may include metadata 740, as described above, in the discussion of FIGURE 7C. Metadata 740 may include information indicating an identification number 745 assigned to the physical item 315 selected by customer 105, an identification number 750 assigned to the physical shelf 305 from which customer 105 selected the physical item 315, and/or an identification

number 755 assigned to the physical rack 210 that includes the physical shelf 305 from which customer 105 selected the physical item 315. When, for example, slider 710 on slider bar 705 reaches marker 715a, virtual store tool 405 may read metadata 740 assigned to marker 715a, to identify that person 105 selected physical item 315f from second physical shelf 305b of physical rack 210a. Markers 715 may be added to slider bar 705 in any suitable manner. For example, in certain embodiments, virtual display tool 405 adds markers 715 to slider bar 705 based on information received from an algorithm 488 configured to track customers 105 in physical store 100 and to determine the physical items 315 selected by each customer 105, based on inputs received from sensors 498 located in physical store 100.

In response to receiving information identifying physical item 315f as being the physical item selected by person 105 from physical rack 210a, virtual store tool 405 may highlight sixth virtual item 320f, located on second virtual shelf 310b of virtual rack 230a. Highlighting sixth virtual item 320f may include any method of distinguishing sixth virtual item 320f from the remaining virtual items 320. For example, highlighting sixth virtual item 320f may include placing a frame around sixth virtual item 320f, as illustrated in FIGURE 9C, enlarging sixth virtual item 320f compared to the other virtual items 320, and/or any other suitable method of distinguishing sixth virtual item 320f from the remaining virtual items 320.

c. Populating a Virtual Cart

In certain embodiments, the graphical user interface 700 displayed by virtual store tool 405 on display 410 may additionally include a virtual shopping cart 420, as illustrated in FIGURE 9D. Virtual shopping cart 420 may be used to further emulate a shopping session of a customer 105 in physical store 100, by storing virtual items 320 corresponding to the physical items 315 selected by person 105 during his/her shopping session. Virtual store tool 405 may display virtual shopping cart 420 in a fifth region 965 of display 410. In certain embodiments, the fifth region 965 of display 410 is located between virtual rack 230b, displayed in third region 905 of display 410, and rack camera segment 425a.

In certain such embodiments, receiving information identifying physical item 315f as being the physical item selected by person 105 from physical rack 210a, may include

receiving information associated with dragging and dropping virtual item 320f, corresponding to physical item 315f, from virtual rack 230a, displayed in region 905, to virtual shopping cart 420. For example, a user 120 may observe customer 105 selecting physical item 315f on camera feeds segments 415a through 415f and/or rack camera feed segment 425a. Accordingly, user 120 may select virtual item 320f from virtual rack 230a, where virtual item 320f corresponds to physical item 315f and is configured to emulate physical item 315f. User 120 may then drag virtual item 320f to virtual shopping cart 420 and drop virtual item 320f in virtual shopping cart 420. In order to help aid user 120 in observing customer 105 selecting a physical item 315 on camera feed segments 415a through 415f and/or rack camera feed segment 425 a, in certain embodiments, user 120 can make any of the displayed camera feed segments 415a through 415f and/or rack camera feed segment 425a larger than the others, by selecting the camera feed segments 415a through 415f and/or rack camera feed segment 425a. For example, user 120 can click on a given camera feed segment 415 or 425, to instruct virtual store tool 405 to increase the size of the segment presented on display 410.

In response to receiving information identifying physical item 315f as the physical item selected by person 105 from physical rack 210a — either from metadata 740, weight sensors 1300 coupled to physical shelf 305b, a dragging and dropping of virtual item 320f into virtual shopping cart 420, and/or any other suitable method of receiving information identifying physical item 315f — virtual store tool 405 may store virtual item 320f, corresponding to physical item 315f, in virtual shopping cart 420. Virtual shopping cart 420 may store any number of virtual items 320. For example, as the playback of camera feed segments 415 and 425 progresses, virtual store tool 405 may receive further information identifying an additional, different physical item 315 as having been selected by person 105 from a physical rack 210. Physical rack 210 may be the same as physical rack 210a or different from physical rack 210a. In response to receiving the information identifying the additional physical item 315, virtual store tool 405 may store an additional virtual item 320, corresponding to the additional physical item 315, in virtual shopping cart 420. This process may repeat any number of times, such as a number of times corresponding to the number of times the camera feed segments 415 and 425 indicate that a person 105 selected a physical item 315 from a physical rack 210.

As illustrated in FIGURE 9D, in certain embodiments, virtual shopping cart 420 may display each virtual item 320 as a graphical representation of the corresponding physical item 315 and/or a textual description 910 of the corresponding physical item 315. Virtual shopping cart 420 may also indicate a quantity 915 of each virtual item 320f contained in the virtual shopping cart 420. For example, virtual shopping cart 420 may indicate a quantity 915 of two virtual items 320f, to emulate the fact that customer 105 selected two physical items 315f from physical rack 210a. Quantity 915 of each virtual item 320 may be increased in any suitable manner. For example, in certain embodiments, quantity 915 of virtual item 320f may be increased by dragging and dropping virtual item 320f, corresponding to physical item 315f, from virtual rack 230a, displayed in region 905, to virtual shopping cart 420 multiple times. As another example, in some embodiments, quantity 915 of virtual item 320f may be increased by a user 120 interacting with graphical user interface 700 through an addition button 925. Similarly, quantity 915 of virtual item 320f may be decreased by user 120 interacting with graphical user interface 700 through a subtraction button 925. User 120 may also remove virtual item 320f from virtual shopping cart 420 by interacting with graphical user interface 700 through a trash button 930.

At the end of the shopping session of customer 105 in physical store 100 (i.e., when virtual store tool 405 determines that customer 105 has exited physical store 100), virtual shopping cart 420 may be used to charge customer 105 for physical items 315 selected by customer 105 during his/her shopping session, and to send a receipt to customer 105. Additionally, virtual shopping cart 420 may be used to validate a determination made by an algorithm 488, based on inputs received from sensors 498 located in physical store 100, of the physical items 315 selected by customer 105 during his/her shopping session. These aspects of virtual store tool 405 will be described in further detail below, in the discussion of FIGURES 13 through 16.

d. Method for Virtually Emulating a Physical Shopping Session

FIGURE 10 presents a flowchart illustrating the manner in which virtual store tool 405 emulates a shopping session of a customer 105 in a physical store 100, using virtual layout 205 and camera feed segments 415 and/or 425 received from physical store 100, and capturing the shopping session. In step 1005, virtual store tool 405 displays virtual layout 205 of virtual store 110. Virtual layout 205 is configured to emulate a physical layout 200 of physical store 100. In particular, the arrangement of virtual racks 230 on virtual layout 205 is configured to emulate the physical layout 200 of physical racks 210 in physical store 100

In step 1010, virtual store tool 405 determines whether the tool has received an indication of an event associated with a person 105 interacting with a physical rack 210 of physical store 100, during a shopping session in physical store 100. This event may include customer 105 approaching a physical rack 210 and/or selecting a physical item 315 from physical rack 210. The indication of the event may include any suitable information that indicates that customer 105 interacted with physical rack 210. For example, in certain embodiments, the indication of the event may include user 120 selecting virtual shelf 230 in virtual layout 205, in response to viewing customer 105 approaching and/or selecting physical item 315 from physical rack 210 on a set of camera feed segments 415, generated from camera feeds 605 received from layout cameras 490, located in physical store 100 and capturing the shopping session of customer 105. As another example, in certain embodiments, the indication of the event may include slider 710 on slider bar 705, assigned to camera feed segments 415, reaching a marker 715. Marker 715 may include metadata 740 indicating the physical rack 210 associated with the event. As a further example, the indication of the event may include receiving information from an algorithm 488 configured to determine that customer 105 approached and/or selected an item 315 from physical rack 210, based on inputs received from sensors 498 located in physical store 100.

If, in step 1010, virtual store tool 405 receives an indication of an event associated with person 105 interacting with physical rack 210, in step 1015, virtual store tool 405 displays the virtual rack 230 corresponding to physical rack 210 (i.e., configured to emulate physical rack 210), in region 905 of display 410. Additionally, in step 1015, virtual store tool 405 displays a rack camera segment 425 generated from a rack camera feed 620 received from a rack camera 495 assigned to physical rack 210. Rack camera segment 425 depicts physical rack 210 during the time interval in which the event occurs.

In step 1020, virtual store tool 405 determines whether the tool has received information identifying a first virtual item 320. As an example, in certain embodiments, each physical shelf 305 of physical rack 210 includes a set of weight sensors 1300, coupled to zones of the physical shelf 305, as described below, in the discussion of FIGURES 13B through 13D. When person 105 removes an item 315 from physical shelf 305, the weight sensor 1300 coupled to the zone of physical shelf 305 on which the item 315 is located may send information to virtual store tool 405 (either directly, or through other components of system 400, such as external system 485), indicating that the item 315 has been selected from physical shelf 305 of physical rack 210a. As another example, in certain embodiments, the indication of the event may include slider 710 on slider bar 705 reaching marker 715a or 715b. Markers 715a and 715b may include metadata 740, as described above, in the discussion of FIGURE 7C. Metadata 740 may include information indicating an identification number 745 assigned to the physical item 315 selected by customer 105, an identification number 750 assigned to the physical shelf 305 from which customer 105 selected the physical item 315, and/or an identification number 755 assigned to the physical rack 210 that includes the physical shelf 305 from which customer 105 selected the physical item 315. Accordingly, when slider 710 on slider bar 705 reaches a marker 715, virtual store tool 405 may receive information identifying physical item 315, by reading metadata 740 assigned to marker 715, to identify that person 105 selected physical item 315 from physical shelf 305 of physical rack 210. Markers 715 may be added to slider bar 705 in any suitable manner. For example, in certain embodiments, virtual display tool 405 adds markers 715 to slider bar 705 based on information received from an algorithm 488 configured to track customers 105 in physical store 100 and to determine the physical items 315 selected by each customer 105, based on inputs received from sensors 498 located in physical store 100. As a further example, receiving information identifying physical item 315/virtual item 320 may include receiving information associated with dragging and dropping virtual item 320, configured to emulate physical item 315, from virtual rack 230, displayed in region 905 of display 410, to virtual shopping cart 420.

WHAT IS CLAIMED IS:

1. An apparatus comprising:

a display;

an interface configured to receive a rack video from a rack camera located in a physical store, the rack camera directed at a first physical rack of a set of physical racks located in the physical store, the rack camera configured to capture video of the first physical rack during a shopping session of a person in the physical store; and

a hardware processor communicatively coupled to the display, the hardware processor configured to:

display, in a first region of the display, a first virtual rack configured to emulate the first physical rack, wherein the first virtual rack comprises a first virtual shelf and a second virtual shelf, the first virtual shelf comprising a first virtual item, the second virtual shelf comprising a second virtual item, wherein the first virtual item comprises a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item comprises a graphical representation of a second physical item located on a second physical shelf of the first physical rack;

display, in a second region of the display, the rack video, wherein the rack video depicts an event comprising the person interacting with the first physical rack;

display, in a third region of the display, a virtual shopping cart; receive information associated with the event, the information identifying the first virtual item, wherein the rack video depicts that the person selected the first physical item while interacting with the first physical rack; and

in response to receiving the information associated with the event, store the first virtual item in the virtual shopping cart.

2. The apparatus of Claim 1, wherein:

the interface is further configured to receive a second rack video from a second rack camera located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks located in the physical store, the second rack camera

configured to capture video of the second physical rack during the shopping session of the person in the physical store; and

the processor is further configured to:

display, in the first region of the display, a second virtual rack configured to emulate the second physical rack, wherein the second virtual rack comprises a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack;

display, in the second region of the display, the second rack video, wherein the second rack video depicts a second event comprising the person interacting with the second physical rack;

receive information associated with the second event, the information identifying the third virtual item, wherein the second rack video depicts that the person selected the third physical item while interacting with the second physical rack; and

in response to receiving the information associated with the second event, store the third virtual item in the virtual shopping cart.

3. The apparatus of Claim 2, wherein the processor is further configured to:

determine that the shopping session has ended;

in response to determining that the shopping session has ended:

compare the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within the physical store;

determine that the virtual shopping cart matches the algorithmic shopping cart; and

in response to determining that the virtual shopping cart matches the algorithmic shopping cart:

generate a receipt comprising a first price assigned to the first virtual item and a second price assigned to the third virtual item; and send the receipt to the person.

4. The apparatus of Claim 3, wherein the sensors comprise:

an array of algorithmic cameras, different from the rack cameras; and

a set of weight sensors, wherein each item of the items determined by the algorithm to have been selected during the shopping session is coupled to a weight sensor of the set of weight sensors.

5. The apparatus of Claim 4, wherein in response to determining that the algorithmic shopping cart comprises an error, the processor is further configured to: determine a subset of the set of inputs associated with the error, wherein the subset comprises inputs received from the array of algorithmic cameras;

attach metadata to the subset, the metadata explaining the discrepancy; and use the subset to train the algorithm.

6. The apparatus of Claim 2, wherein the processor is further configured to:

determine that the shopping session has ended;

in response to determining that the shopping session has ended:

compare the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected by the person during the shopping session, the algorithmic shopping cart determined based on inputs received from sensors located within the physical store;

determine that the virtual shopping cart does not match the algorithmic shopping cart; and

in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:

determine that the algorithmic shopping cart comprises an error; and

in response to determining that the algorithmic shopping cart comprises an error:

generate a receipt based on the virtual shopping cart, the receipt comprising a first price assigned to the first virtual item and a second price assigned to the third virtual item; and send the receipt to the person.

7. The apparatus of Claim 1, wherein the processor is further configured to:

determine that the shopping session has ended;

in response to determining that the shopping session has ended:

compare the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on inputs received from sensors located within the physical store;

determine that the virtual shopping cart does not match the algorithmic shopping cart; and

in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:

determine that the virtual shopping cart comprises an error; and

in response to determining that the virtual shopping cart comprises an error:

generate a receipt based on the algorithmic shopping cart, the receipt comprising prices of the items determined by the algorithm to have been selected by the person during the shopping session; and

send the receipt to the person.

8. The apparatus of Claim 1 , wherein the information associated with the event comprises information associated with dragging and dropping the first virtual item from the first virtual rack into the first virtual cart.

9. A method comprising:

receiving a rack video from a rack camera located in a physical store, the rack camera directed at a first physical rack of a set of physical racks located in the physical store, the rack camera configured to capture video of the first physical rack during a shopping session of a person in the physical store;

displaying, in a first region of a display, a first virtual rack configured to emulate the first physical rack, wherein the first virtual rack comprises a first virtual shelf and a second virtual shelf, the first virtual shelf comprising a first virtual item, the second virtual shelf comprising a second virtual item, wherein the first virtual item comprises a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item comprises a graphical representation of a second physical item located on a second physical shelf of the first physical rack;

displaying, in a second region of the display, the rack video, wherein the rack video depicts an event comprising the person interacting with the first physical rack;

displaying, in a third region of the display, a virtual shopping cart;

receiving information associated with the event, the information identifying the first virtual item, wherein the rack video depicts that the person selected the first physical item while interacting with the first physical rack; and

in response to receiving the information associated with the event, storing the first virtual item in the virtual shopping cart.

10. The method of Claim 9, further comprising:

receiving a second rack video from a second rack camera located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks located in the physical store, the second rack camera configured to capture video of the second physical rack during the shopping session of the person in the physical store; displaying, in the first region of the display, a second virtual rack configured to emulate the second physical rack, wherein the second virtual rack comprises a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical

shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack;

displaying, in the second region of the display, the second rack video, wherein the second rack video depicts a second event comprising the person interacting with the second physical rack;

receiving information associated with the second event, the information identifying the third virtual item, wherein the second rack video depicts that the person selected the third physical item while interacting with the second physical rack; and

in response to receiving the information associated with the second event, storing the third virtual item in the virtual shopping cart.

11. The method of Claim 10, further comprising:

determining that the shopping session has ended;

in response to determining that the shopping session has ended:

comparing the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within the physical store;

determining that the virtual shopping cart matches the algorithmic shopping cart; and

in response to determining that the virtual shopping cart matches the algorithmic shopping cart:

generating a receipt comprising a first price assigned to the first virtual item and a second price assigned to the third virtual item; and sending the receipt to the person.

12. The method of Claim 11, wherein the sensors comprise:

an array of algorithmic cameras, different from the rack cameras; and

a set of weight sensors, wherein each item of the items determined by the algorithm to have been selected during the shopping session is coupled to a weight sensor of the set of weight sensors.

13. The method of Claim 12, further comprising, in response to determining that the algorithmic shopping cart comprises an error:

determining a subset of the set of inputs associated with the error, wherein the subset comprises inputs received from the array of algorithmic cameras;

attaching metadata to the subset, the metadata explaining the discrepancy; and using the subset to train the algorithm.

14. The method of Claim 10, further comprising:

determining that the shopping session has ended;

in response to determining that the shopping session has ended:

comparing the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected by the person during the shopping session, the algorithmic shopping cart determined based on inputs received from sensors located within the physical store;

determining that the virtual shopping cart does not match the algorithmic shopping cart; and

in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:

determining that the algorithmic shopping cart comprises an error; and

in response to determining that the algorithmic shopping cart comprises an error:

generating a receipt based on the virtual shopping cart, the receipt comprising a first price assigned to the first virtual item and a second price assigned to the third virtual item; and

sending the receipt to the person.

15. The method of Claim 9, further comprising:

determining that the shopping session has ended;

in response to determining that the shopping session has ended:

comparing the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on inputs received from sensors located within the physical store;

determining that the virtual shopping cart does not match the algorithmic shopping cart; and

in response to determining that the virtual shopping cart does not match the algorithmic shopping cart:

determining that the virtual shopping cart comprises an error; and in response to determining that the virtual shopping cart comprises an error:

generating a receipt based on the algorithmic shopping cart, the receipt comprising prices of the items determined by the algorithm to have been selected by the person during the shopping session; and

sending the receipt to the person.

16. The method of Claim 9, wherein the information associated with the event comprises information associated with dragging and dropping the first virtual item from the first virtual rack into the first virtual cart.

17. A system comprising:

a display element;

an interface operable to:

receive a rack video from a rack camera located in a physical store, the rack camera directed at a first physical rack of a set of physical racks located in the physical store, the rack camera configured to capture video of the first physical rack during a shopping session of a person in the physical store; and

receive a second rack video from a second rack camera located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks located in the physical store, the second rack camera configured to capture video of the second physical rack during the shopping session of the person in the physical store; and

a processing element communicatively coupled to the display element, the processing element operable to:

display, in a first region of the display, a first virtual rack configured to emulate the first physical rack, wherein the first virtual rack comprises a first virtual shelf and a second virtual shelf, the first virtual shelf comprising a first virtual item, the second virtual shelf comprising a second virtual item, wherein the first virtual item comprises a graphical representation of a first physical item located on a first physical shelf of the first physical rack and the second virtual item comprises a graphical representation of a second physical item located on a second physical shelf of the first physical rack;

display, in a second region of the display, the rack video, wherein the rack video depicts an event comprising the person interacting with the first physical rack;

display, in a third region of the display, a virtual shopping cart; receive information associated with the event, the information identifying the first virtual item, wherein the rack video depicts that the person selected the first physical item while interacting with the first physical rack;

in response to receiving the information associated with the event, store the first virtual item in the virtual shopping cart;

display, in the first region of the display, a second virtual rack configured to emulate the second physical rack, wherein the second virtual rack comprises a third virtual shelf and a fourth virtual shelf, the third virtual shelf comprising a third virtual item, the fourth virtual shelf comprising a fourth virtual item, wherein the third virtual item comprises a graphical representation of a third physical item located on a third physical shelf of the second physical rack and the fourth virtual item comprises a graphical representation of a fourth physical item located on a fourth physical shelf of the second physical rack;

display, in the second region of the display, the second rack video, wherein the second rack video depicts a second event comprising the person interacting with the second physical rack;

receive information associated with the second event, the information identifying the third virtual item, wherein the second rack video depicts that the person selected the third physical item while interacting with the second physical rack; and

in response to receiving the information associated with the second event, store the third virtual item in the virtual shopping cart;

determine that the shopping session has ended;

in response to determining that the shopping session has ended:

compare the virtual shopping cart to an algorithmic shopping cart comprising items determined by an algorithm to have been selected during the shopping session, the algorithmic shopping cart determined based on a set of inputs received from sensors located within the physical store, wherein the sensors comprise:

an array of algorithmic cameras, different from the rack cameras;

a set of weight sensors, wherein each item of the items determined by the algorithm to have been selected during the shopping session is coupled to a weight sensor of the set of weight sensors; and

in response to comparing the virtual shopping cart to the algorithmic shopping cart:

generate a receipt for the shopping session; and

send the receipt to the person.

18. The system of Claim 17, wherein the receipt comprises a first price assigned to the first virtual item and a second price assigned to the third virtual item.

19. The system of Claim 17, wherein in response to comparing the virtual shopping cart to the algorithmic shopping cart, the processing element is further operable to:

determine that the algorithmic shopping cart comprises an error;

determine a subset of the set of inputs associated with the error, wherein the subset comprises inputs received from the array of algorithmic cameras;

attach metadata to the subset, the metadata explaining the discrepancy; and use the subset to train the algorithm.

20. The system of Claim 17, wherein the information associated with the event comprises information associated with dragging and dropping the first virtual item from the first virtual rack into the first virtual cart.