Claims:An apparatus comprising:
a processor to:
receive a first compressed content bitstream from a host server device;
capture input content to utilize as overlay content for the first compressed content bitstream;
determine whether one of a first mode of pass-through encoding is enabled or a second mode of look-ahead encoding is enabled; and
responsive to the first mode of pass-through encoding being enabled:
generate a second content bitstream representing the input content;
multiplex the first compressed content bitstream with the second content bitstream to generate a third combined content bitstream; and
transmit the third combined content bitstream to a streaming server device for streaming to one or more users.
, Description:RELATED APPLICATION
[0001] The present application claims priority to U.S. Non-Provisional Patent Application No. 17/210,977 filed March 24, 2021 and titled “VIDEO STREAMING FOR CLOUD GAMING” the entire disclosure of which is hereby incorporated by reference.

FIELD
[0002] This application relates generally to data processing and more particularly to video streaming for cloud gaming.

BACKGROUND
[0003] Current parallel graphics data processing includes systems and methods developed to perform specific operations on graphics data such as, for example, linear interpolation, tessellation, rasterization, texture mapping, depth testing, etc. Traditionally, graphics processors used fixed function computational units to process graphics data; however, more recently, portions of graphics processors have been made programmable, enabling such processors to support a wider variety of operations for processing vertex and fragment data.
[0004] To further increase performance, graphics processors typically implement processing techniques such as pipelining that attempt to process, in parallel, as much graphics data as possible throughout the different parts of the graphics pipeline. Parallel graphics processors with single instruction, multiple data (SIMD) or single instruction, multiple thread (SIMT) architectures are designed to maximize the amount of parallel processing in the graphics pipeline. In a SIMD architecture, computers with multiple processing elements attempt to perform the same operation on multiple data points simultaneously. In a SIMT architecture, groups of parallel threads attempt to execute program instructions synchronously together as often as possible to increase processing efficiency.
[0005] Parallel graphics data processing may be utilized in a variety of technologies, including cloud gaming and game streaming. In cloud gaming, a game is run by remote cloud server, frames/audio are captured, encoded, and sent to customer devices through a network. The user’s control data is sent back to the remote cloud server. Game streaming is an approach where a user runs a game, captures video/audio, annotate the captured video/audio with overlays (e.g., self-camera, chats, voice, etc.), and streams this captured content though streaming services for viewing or watching. Currently, streaming client devices (e.g., user device running the game) performs the overlaying/composition, capturing, and encoding locally, even for games running remotely. Such an approach utilizes high throughput hardware on the client end. Moreover, cloud gaming is positioned as a technology to play games on thin client devices.
[0006] Conventionally, a standard game streaming approach is to capture a gaming display with audio and mix this gaming display with overlays/self-camera/voice using software. Using the conventional approaches, streamers run games locally and capture video frames from a graphics adapter. With cloud gaming, users start to capture display/audio after rendering of encoded sequence coming from cloud. Conventional streaming software development kits (SDKs) have to capture and fully re-encode content coming from cloud gaming platforms and do not use any information about video sequence that can be obtained from an original cloud gaming stream.
[0007] A conventional approach in streaming gaming video content includes executing a game remotely in a cloud server device. In this scenario, the streamer’s device receives a bitstream that represents encoded game frames, decodes the frames, and renders the frames. The streamer device can further composite the original game frames with self-camera/overlays, capture the display, and encode it for subsequent streaming (e.g., by a streaming cloud server device). As such, the streamer device runs transcoding, which includes decoding of the original content, video processing/composition, and encode. Thus, the streamer device should have the capability to perform each of these operations in runtime, which can be problematic and/or difficult as cloud gaming scenarios are targeted to thin end user-devices, such as mobile phones, tablets, pads, and so on, that can lack the hardware to sufficiently support such capabilities.
 
BRIEF DESCRIPTION OF THE DRAWINGS
[0009] So that the manner in which the above recited features of the present embodiments can be understood in detail, a more particular description of the embodiments, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments and are therefore not to be considered limiting of its scope.
[0010] FIG. 1 is a block diagram of a processing system.
[0011] FIG. 2A-2D illustrate computing systems and graphics processors.
[0012] FIG. 3A-3C illustrate block diagrams of additional graphics processor and compute accelerator architectures.
[0013] FIG. 4 is a block diagram of a graphics processing engine of a graphics processor.
[0014] FIG. 5A-5B illustrate thread execution logic including an array of processing elements employed in a graphics processor core.
[0015] FIG. 6 illustrates an additional execution unit.
[0016] FIG. 7 is a block diagram illustrating a graphics processor instruction formats.
[0017] FIG. 8 is a block diagram of an additional graphics processor architecture.
[0018] FIG. 9A-9B illustrate a graphics processor command format and command sequence.
[0019] FIG. 10 illustrates example graphics software architecture for a data processing system.
[0020] FIG. 11A is a block diagram illustrating an IP core development system.
[0021] FIG. 11B illustrates a cross-section side view of an integrated circuit package assembly.
[0022] FIG. 11C illustrates a package assembly that includes multiple units of hardware logic chiplets connected to a substrate (e.g., base die).
[0023] FIG. 11D illustrates a package assembly including interchangeable chiplets.
[0024] FIG. 12 is a block diagram illustrating an example system on a chip integrated circuit.
[0025] FIG. 13A-13B are block diagrams illustrating example graphics processors for use within an SoC.
[0026] FIG. 14 is a block diagram illustrating a communication system for video streaming for cloud gaming using a pass-through mode, according to embodiments.
[0027] FIG. 15 is a block diagram depicting a content and overlay combined bitstream 1500, in accordance with embodiments.
[0028] FIG. 16 is a block diagram illustrating a communication system for video streaming for cloud gaming using a look ahead encoding mode, according to embodiments.
[0029] FIG. 17 is a block diagram illustrating a system for look ahead encoding in accordance with embodiments.
[0030] FIG. 18 is a flow diagram illustrating an embodiment of a method for content streaming with overlay utilizing a pass-through mode.
[0031] FIG. 19 is a flow diagram illustrating an embodiment of a method for content streaming with overlay utilizing a look ahead encoding mode.
[0032] FIG. 20 is a flow diagram illustrating an embodiment of a method for content streaming with overlay utilizing a look ahead encoding with multiplexing mode.
[0033] FIG. 21 is an illustrative diagram of an example system or device for video streaming and coding, arranged in accordance with embodiments.
 
DETAILED DESCRIPTION
[0034] A graphics processing unit (GPU) is communicatively coupled to host/processor cores to accelerate, for example, graphics operations, machine-learning operations, pattern analysis operations, and/or various general-purpose GPU (GPGPU) functions. The GPU may be communicatively coupled to the host processor/cores over a bus or another interconnect (e.g., a high-speed interconnect such as PCIe or NVLink). Alternatively, the GPU may be integrated on the same package or chip as the cores and communicatively coupled to the cores over an internal processor bus/interconnect (i.e., internal to the package or chip). Regardless of the manner in which the GPU is connected, the processor cores may allocate work to the GPU in the form of sequences of commands/instructions contained in a work descriptor. The GPU then uses dedicated circuitry/logic for efficiently processing these commands/instructions.
[0035] In the following description, numerous specific details are set forth to provide a more thorough understanding. However, it will be apparent to one of skill in the art that the embodiments described herein may be practiced without one or more of these specific details. In other instances, well-known features have not been described to avoid obscuring the details of the present embodiments.