DESC:BACKGROUND
[0001] Embodiments of the present specification relate generally to audio-visual control systems, and more particularly to systems and methods for real-time automated audio-visual control of ambience in response to musical input.
[0002] Various systems and methods for controlling lights, speakers, and other audio-visual equipment have been developed to improve entertainment and create environmental ambience in different settings. Lighting, sound, and other special effects have been used to create a theatrical setting for entertainment. For example, musical fountains are a type of choreographed fountains where the water flow and direction of the water flow are synchronized to music and also include lighting effects via use of lights, lasers, video projection, and other imagery to create an audio-visual choreographed spectacle. In the musical fountains, the timing and sequence of water flow set to music is typically pre-recorded and controlled via a computerized system. Similarly, light and sound shows provide theatrical entertainment, typically in an outdoor setting, using lighting, sound, audio, and other effects to relate the historical significance of a place. In this example, the light and sound shows are generally pre-recorded and controlled via a control console.
[0003] In another example, disco lighting effects and dance floor lighting and audio have been utilized to enhance the audio-visual experience in a discotheque or a club. Flashing multicolored lights, strobe lights, a light shining on a mirror ball are often synchronized with music to create a visual immersive experience in the discotheque. The flashing of the lights to the music is generally pre-programmed and controlled by a computerized system. In yet another example, a light and sound show in a vehicle is a choreographed show in which the lights in the vehicle are illuminated based on musical input to provide in- vehicle ambience generation. However, in a vehicle, typically, the light show is pre-programmed to correspond to a particular musical input and hence may not be flexible to adapt to real-time musical input.
[0004] It may be noted that in some of the examples of controlling lights, speakers, and other audio-visual equipment noted hereinabove, the choreography of the lighting to the music is still painstakingly pre-programmed by hand. Also, in some examples, the choreographed shows are manually controlled via a control console based on previously programmed choreography and hence are unable to perform in response to real-time musical input. By way of example, conventional musical fountains are manually pre-programmed moment-to-moment of the choreographed show, and hence the musical fountain may not be able to adapt in real-time to a new musical input.
[0005] Recent advances in technology have facilitated unattended control of the lights, speakers, and other audio-visual equipment via pre-recorded choreography. However, none of these audio-visual control systems are flexible or customizable to effectively utilize the properties of musical input automatically or in real-time to create a desired ambience. Hence, there is a need for automated audio-visual control of ambience in response to musical input in real-time to create an enhanced immersive ambience. There is also a growing need for dynamically adjusting the ambience in response to new musical input to create a rich, immersive, and responsive ambience in various settings.
BRIEF DESCRIPTION
[0006] In accordance with aspects of the present specification, a system for real-time automated audio-visual control of ambience based on musical input is presented. The system includes an ambience control system that includes an acquisition subsystem configured to receive the musical input and a processing subsystem in operative association with the acquisition subsystem and including an audio-visual ambience control platform configured to process the musical input to generate Mel Frequency Cepstrum Coefficients, identify features, patterns, timing, musical expressions, or combinations thereof in the musical input based on the Mel Frequency Cepstrum Coefficients, assign each Mel Frequency Cepstrum Coefficient to a different control channel, generate a control signal corresponding to each control channel based on the Mel Frequency Cepstrum Coefficients, the identified features, patterns, timing, and musical expressions in the musical input, or combinations thereof, create a timecode file based on the Mel Frequency Cepstrum Coefficients and the control signals, and control a plurality of ambience elements based on the timecode file to dynamically create a rich, immersive, and responsive ambience in a setting.
[0007] In accordance with another aspect of the present specification, a method for real-time automated audio-visual control of ambience based on musical input is presented. The method includes receiving the musical input. Moreover, the method includes processing the musical input to generate Mel Frequency Cepstrum Coefficients. The method also includes identifying features, patterns, timing, musical expressions, or combinations thereof in the musical input based on the Mel Frequency Cepstrum Coefficients. In addition, the method includes assigning each Mel Frequency Cepstrum Coefficient to a different control channel. Furthermore, the method includes generating a control signal based on the Mel Frequency Cepstrum Coefficients, the features, the patterns, the timing, the musical expressions in the musical input, or combinations thereof. Additionally, the method includes creating a timecode file based on the Mel Frequency Cepstrum Coefficients and the control signals. Also, the method includes controlling a plurality of ambience elements based on the timecode file to dynamically create a rich, immersive, and responsive ambience in a setting.
[0008] In accordance with yet another aspect of the present specification, a processing system for real-time automated audio-visual control of ambience based on a musical input is presented. The processing system includes an audio-visual ambience control platform. The audio-visual ambience control platform includes a pre-processing unit configured to receive the musical input from one or more sources, parse the musical input based on a format and extract audio channels in the musical input, decode data in the musical input to extract raw audio samples, and mix the extracted audio channels based on the data in the musical input. Furthermore, the audio-visual ambience control platform includes a Mel Frequency Cepstrum Coefficients extraction unit configured to transform the musical input from the time domain to the frequency domain to extract the Mel Frequency Cepstrum Coefficients and identify features, patterns, timing, musical expressions, or combinations thereof in the musical input based on the Mel Frequency Cepstrum Coefficients. Moreover, the audio-visual ambience control platform includes a channel assignment unit configured to assign each Mel Frequency Cepstrum Coefficient to a different control channel based on the properties of the Mel Frequency Cepstrum Coefficients, the plurality of ambience elements to be controlled, a setting, or combinations thereof. Additionally, the audio-visual ambience control platform includes a control signal generation unit configured to process each assigned control channel to generate a control signal, where the control signal is configured to control the plurality of ambience elements to create a rich, immersive, and responsive ambience in a setting, and where the control signals are generated based on the properties of the Mel Frequency Cepstrum Coefficients, the plurality of ambience elements to be controlled, the setting, or combinations thereof. Also, the audio-visual ambience control platform includes a timecode file generation unit configured to generate a timecode file, where the timecode file includes information for controlling the ambience elements in response to the musical input. In addition, the audio-visual ambience control platform includes a device control unit configured to control the operation of a plurality of ambience elements based on the control signals and the timecode file to dynamically create the rich, immersive, and responsive ambience in the setting.

DRAWINGS
[0009] These and other features and aspects of embodiments of the present specification will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:
[0010] FIG. 1 is a schematic representation of an exemplary system for real-time automated audio-visual control of ambience based on musical input, in accordance with aspects of the present specification;
[0011] FIG. 2 is a flow chart illustrating a method for real-time automated audio-visual control of ambience based on musical input, in accordance with aspects of the present specification;
[0012] FIG. 3 is a schematic illustration of some steps of the method for real-time automated audio-visual control of ambience based on musical input of FIG. 2, in accordance with aspects of the present specification;
[0013] FIG. 4 is a diagrammatical illustration of one embodiment of the implementation of the system for real-time automated audio-visual control of ambience based on musical input, in accordance with aspects of the present specification;
[0014] FIG. 5 is a diagrammatical illustration of one embodiment of a real-time audio-visual ambience control platform for use in the system of FIG. 1, in accordance with aspects of the present specification; and
[0015] FIG. 6 is a schematic representation of one embodiment of a digital processing system implementing a real-time audio-visual ambience control platform for use in the system of FIG. 1, in accordance with aspects of the present specification.
DETAILED DESCRIPTION
[0016] The following description presents exemplary systems and methods for real-time automated audio-visual control of ambience based on musical input. In particular, the systems and methods presented herein facilitate the real-time automated control of the audio-visual ambience in response to musical input using Mel Frequency Cepstrum Coefficients (MFCCs). The MFCCs represent the human hearing sensitive frequencies and correspond to different features and/or patterns of the musical input. Accordingly, the MFCCs are suited for the automated audio-visual control of ambience based on musical input. The systems and methods presented herein are configured to receive a time domain signal in the form of a musical input from a source. Moreover, the systems and methods are configured to transform the received musical input from the time domain to the frequency domain and process the frequency domain representation of the musical input to extract the MFCCs. Also, the frequency domain representation of the musical input may be parsed to identify specific features, timing, musical expressions, and/or patterns in the musical input. Additionally, in accordance with aspects of the present specification, the MFCCs are used as the basis for the subsequent generation of control signals for controlling various devices or ambience elements based on the identified features, musical expressions, timing, and/or patterns in the musical input. These control signals are employed to control one or more devices or ambience elements such as lights, speakers, horn, lasers, actuators, dancing lights, laser lights, flashing multicolored lights, strobe lights, fountain jets, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, sound and light shows, and more. It may be noted that the various settings may be generally referred to as a light and sound orchestration. In one example, the control signals may be used to convert the audio signal of the musical input into a visual display that is synchronized with the ambience elements to create an enhanced immersive experience/ambience. It may be noted that the terms “ambience” and “experience” may be used interchangeably. In addition, the systems and methods are also configured to dynamically adjust the control signals based on real-time changes in the audio signal of the musical input, thereby allowing the systems and methods to be dynamically customizable and adaptable to new musical input in real-time.
[0017] Embodiments described hereinafter present exemplary systems and methods that are designed to be highly efficient and robust, thereby ensuring a high-quality output. Use of the present systems and methods presents significant advantages in reliably enhancing the immersive audio-visual experience where the ambience may be controlled in real-time via automated audio-visual control of the ambience based on the musical input, and where the ambience dynamically responds to real-time changes in the audio signal of the musical input, thereby overcoming the drawbacks of currently available methods of controlling audio-visual equipment to improve entertainment and ambience.
[0018] For ease of understanding, the exemplary embodiments of the present systems and methods are described in the context of an in-vehicle ambience generation system designed to effectively utilize properties of musical input to provide real-time automated audio-visual control of the in-vehicle ambience, and where the system is configured to dynamically adjust the ambience in the vehicle in response to real-time changes in the audio signal of the musical input. It may be noted that although the various systems and methods presented herein are generally described in the context of an in-vehicle ambience generation system, the systems and methods may also be used in other settings such as, but not limited to, light and sound shows, dance floor audio and lighting, theaters, music halls, musical fountains, live music and dance shows, and the like. An exemplary environment that is suitable for practising various implementations of the present systems and methods is discussed in the following sections with reference to FIG. 1.
[0019] As used herein, the term “user” refers to a person using the system 100 for in-vehicle ambience generation. In the present example, the user may include a driver of a vehicle, passengers in the vehicle, and the like. Further, as used herein, the term “ambience” is used to refer to a feeling or mood associated with a particular place, person, or thing. Additionally, the term “music,” “music file,” or “musical input” is used to generally refer to sound arranged in time to express ideas and emotions via a combination of form, harmony, melody, rhythm, or otherwise expressive content. Some non-limiting examples of music include people singing with their voices, people playing musical instruments, such as the piano, guitar, drums, flute, clarinet, or violin, and the like. The term “setting” is used to refer to an environment, framework, or location such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, light and sound shows, live concerts, live music and dance shows, and the like. Further, the term “device” or “ambience element” may be used to refer to any element that is configured to provide or enhance the ambience of any setting. Some non-limiting examples of the devices or ambience elements include lights, speakers, actuators, dance lights, lasers, laser lights, flashing multicolored lights, strobe lights, fountain jets, horns, and the like. It may be noted that the terms devices and ambience elements may be used interchangeably. Also, in the context of the in-vehicle ambience generation system, the term “lights” or “lighting” in the vehicle may be used to refer to headlights, taillights, parking lights, signal lights, brake lights, backup lights, fog lights, blinker lights, door lights, hazard lights, daytime driving lights, reading lights, interior lights, strip lights, recessed lights, dancing lights, laser lights, and the like.
[0020] Referring now to the drawings, FIG. 1 illustrates an exemplary system 100 for real-time automated audio-visual control of ambience based on musical input. For ease of explanation, in the present specification, this system 100 may generally be referred to as an in-vehicle ambience generation system 100 configured to provide real-time automated audio-visual control of ambience based on musical input in a setting that includes a vehicle. The in-vehicle ambience generation system 100 is designed to effectively utilize properties of a musical input to create a rich, immersive, and responsive ambience using MFCCs associated with the musical input.
[0021] It may be noted that the system 100 may be integrated in a vehicle (not shown). In one example, the system 100 may be integrated into the infotainment system of the vehicle. In another example, the system 100 may be a standalone unit that is communicatively coupled to the vehicle and/or the infotainment system in the vehicle. In a presently contemplated configuration, the system 100 includes an ambience control system 102. The ambience control system 102 is configured to receive musical input 104 and effectively utilize the properties associated with the musical input 104 such as MFCCs to dynamically create a rich, immersive, and responsive ambience in the vehicle. In addition, the ambience control system 102 is configured to dynamically adjust the ambience in response to changes in the musical input 104 in various settings. Consequently, the ambience control system 102 is configured to automatically create, in real-time, an immersive audio-visual experience that is dynamically generated based on a presently selected musical input 104.
[0022] The musical input 104 may be in the form of a music file. The musical input 104 may include one or more music files corresponding to a variety of genres. Also, the music file 104 may include instrumental music, people singing, combinations thereof, and the like. Furthermore, in one example, these music files 104 may be loaded or stored in an infotainment system of the vehicle. Additionally, or alternatively, the ambience control system 102 may be configured to receive the music files 104 from various sources such as, but not limited to, a storage device such as a Universal Serial Bus (USB) device, a compact disc (CD), a cassette, a digital versatile disc (DVD), a Blu-ray disc, digital files, live performances, online streaming platforms, a data repository, or any other source of musical output.
[0023] Once a user such as a driver of the vehicle or a passenger in the vehicle desires to create an enhanced immersive experience in the vehicle, the user may initiate the enhanced immersive experience via a light and sound show application. In one example, the light and sound show application may be accessed via an infotainment system in the vehicle. Also, it may be noted that in one example the enhanced immersive experience in the vehicle may include a light and sound show that is generated in response to the music file 104. The light and sound show may entail use of one or more combinations of lights, speakers, and the horn in the vehicle.
[0024] Further, to initiate the light and sound show, the user may select a desired music file 104. In one embodiment, the music file 104 may be stored in the infotainment system, while in other embodiments, the user may select the music file 104 from other sources that are communicatively coupled to the ambience control system 102. As previously noted, the other sources include digital files, live performances, online streaming platforms, USBs, CDs, DVDs, Blu-ray discs, or any other source of musical output.
[0025] Moreover, in one embodiment, to initiate the light and sound show that is responsive to the selected music file 104, the user may select or touch a button corresponding to the light and sound show application on a display of the infotainment system of the vehicle. Subsequently, the ambience control system 102 may be configured to process the music file 104 to create an enhanced immersive audio-visual experience in the vehicle in real-time.
[0026] In a presently contemplated configuration, the ambience control system 102 includes an acquisition subsystem 106 and a processing subsystem 108. The acquisition subsystem 106 is configured to receive the selected music file 104 from a source. Moreover, the acquisition subsystem 106 is configured to process the music file 104 to be compatible with a range of audio formats and sources to ensure versatility and broad application. The acquisition subsystem 106 is configured to communicate the music file 104 to processing subsystem 108. The processing subsystem 108 is configured to process the music file 104 to create in real-time a unique in-vehicle ambience based on the selected music file 104 to provide an enhanced immersive experience to the user.
[0027] In a non-limiting example, the processing subsystem 108 may include one or more application-specific processors, digital signal processors, microcomputers, graphical processing units, microcontrollers, Application Specific Integrated Circuits (ASICs), Programmable Logic Arrays (PLAs), Field Programmable Gate Arrays (FGPAs), and/or any other suitable processing devices. In some embodiments, the processing subsystem 108 may also be configured to retrieve the music file 104 from a data repository 118. The data repository 118 may include a hard disk drive, a floppy disk drive, a read/write CD, a DVD, a Blu-ray disc, a flash drive, a solid-state storage device, a local database, and the like.
[0028] In addition, the examples, demonstrations, and/or process steps performed by certain components of the system 100 such as the processing subsystem 108 may be implemented by suitable code on a processor-based system, where the processor-based system may include a general-purpose computer or a special-purpose computer. Also, different implementations of the present specification may perform some or all of the steps described herein in different orders or substantially concurrently.
[0029] In one embodiment, the processing subsystem 108 includes an audio-visual ambience control platform 110 that is configured to receive a time domain signal in the form of the music file 104 that may be selected by the user. The audio-visual ambience control platform 110 is configured to process the music file 104 to control the ambience elements in the vehicle such as lighting, speakers, and/or the horn to generate an enhanced and more immersive audio-visual experience based on properties of the music file 104 such as MFCCs. One example of an enhanced and more immersive audio-visual experience may include a light and sound show using the lights, speakers, and/or the horn in the vehicle in response to the music file 104. Additionally, the audio-visual ambience control platform 110 is configured to control the generation of the immersive experience such that the ambience dynamically responds in real-time to changes in the music file 104 being played.
[0030] As noted hereinabove, the audio-visual ambience control platform 110 is configured to generate a unique in-vehicle ambience based on the selected music file 104 using MFCCs associated with the music file 104. As will be appreciated, the Mel scale is a scale that relates the perceived frequency of a tone to the actual measured frequency. The Mel scale scales the frequency in order to match more closely what the human ear can hear. Also, the Mel-frequency cepstrum (MFC) is a representation of the short-term power spectrum of a sound, which in turn is based on a linear cosine transform of a low power spectrum on a non-linear Mel scale of frequency. Furthermore, in the MFC, the frequency bands are equally spaced on the Mel scale, which approximates the human auditory system’s response more closely than the linearly spaced frequency bands used in the normal spectrum. Moreover, the MFCCs are coefficients that collectively make up an MFC. Also, the MFCCs of a signal are a small set of features (usually about 10-20) which concisely describe the overall shape of a spectral envelope.
[0031] Furthermore, as will be appreciated, the MFCC technique is an audio processing method used to extract features from an audio signal such as the music file 104. It may be noted that the MFCCs contain information about the rate changes in the different spectrum bands. Moreover, if an MFCC has a positive value, the majority of the spectral energy is concentrated in the low-frequency regions. However, if an MFCC has a negative value, most of the spectral energy is concentrated at high frequencies.
[0032] In accordance with exemplary aspects of the present specification, the audio-visual ambience control platform 110 is configured to facilitate real-time automated control of the audio-visual ambience in a given setting in response to the music file 104 using MFCCs associated with the music file 104. As previously noted, the MFCCs represent the human hearing sensitive frequencies and correspond to different features and/or patterns of the music file 104 and hence are suitable for the real-time automated audio-visual control of ambience based on musical input. Accordingly, the audio-visual ambience control platform 110 is configured to process the music file 104 to extract the MFCCs. In some embodiments, prior to extracting the MFCCs, the music file 104 may be pre-processed. In one example, the music file 104 may be parsed to extract the audio channels and the audio data in the music file 104 may be decoded to get the raw samples. Additionally, the music file 104 may also be subject to other pre-processing to modify the sampling rate and mixing.
[0033] To extract the MFCCs from the music file 104, the audio-visual ambience control platform 110 is configured to frame the music file 104 into short frames. Subsequently, the short frames of the music file 104 are transformed from the time domain to the frequency domain via use of a Fourier transform, in one example. The powers of the spectrum obtained subsequent to the processing via the Fourier transform may be mapped onto the Mel scale via use of a Mel filter bank applied to the power spectrum. Further, logarithms of the powers at each of the Mel frequencies may be taken. Moreover, the Mel log powers may be processed via a discrete cosine transform (DCT) to generate MFCCs. In one example, coefficients 2-13 may be retained and the rest of the coefficients may be discarded. The MFCCs so generated are representative of the amplitudes of the resulting spectrum. It may be noted that the zeroth MFCC may correspond to lower frequencies while the higher MFCCs may correspond to higher frequencies.
[0034] In addition, the audio-visual ambience control platform 110 is configured to use the MFCCs at each time instant to identify specific features, timing, musical expressions, and/or patterns in the music file 104. Accordingly, at each time instant the MFCCs may be parsed to identify properties such as specific features, patterns, timing, and/or musical expressions in the music file 104. Some non-limiting examples of the specific features, patterns, and/or musical expressions of the music file 104 include tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, and the like. It may be noted that the time interval used may be determined based on the maximum rate at which the ambience elements may be controlled.
[0035] According to aspects of the present specification, these MFCCs form the basis for the subsequent generation of control signals for controlling the ambience elements to create a rich, immersive, responsive experience/ambience in a given setting. It may be noted that the MFCC extraction process is optimized to ensure accuracy and efficiency.
[0036] Subsequent to the extraction of the MFCCs, the audio-visual ambience control platform 110 is configured to assign control channels based on the generated MFCCs. In one embodiment, the audio-visual ambience control platform 110 is configured to assign each MFCC to a different control channel. Furthermore, this assignment of MFCCs to different control channels may be performed in a variety of ways. In one embodiment, the audio-visual ambience control platform 110 may be configured to assign each MFCC to a different control channel in a sequential fashion, a random fashion, or based on a predetermined mapping. By way of example, the channel corresponding to the third MFCC may be assigned to a control channel configured to control the headlights of the vehicle, while the channel corresponding to the eleventh MFCC may be assigned to a control channel configured to control the fog lights of the vehicle. It may be noted that in some embodiments, the audio channels may be pre-assigned prior to the extraction of the MFCCs.
[0037] Additionally, in certain embodiments, the audio-visual ambience control platform 110 may also be configured to assign each MFCC to a different control channel based on the properties of the MFCCs and/or the requirements of the ambience elements that are to be controlled. For example, the lighting configuration of different vehicle models may be considered in the assignment of the MFCCs to the different control channels. Also, the configuration or setup of the ambience elements to be controlled may be accounted for in the channel assignment process based on the MFCCs. In one example, the configuration of the ambience elements such as the lights, dance lights, laser lights, speakers, actuators, flashing lights, strobe lights, and/or horn corresponding to different vehicle models may be obtained from a data repository. Considering the properties of the MFCCs and the requirements of the ambience elements to be controlled ensures that the control signals generated from the MFCCs are best suited for the specific ambience elements in use.
[0038] With continuing reference to FIG. 1, in accordance with aspects of the present specification, subsequent to the channel assignment of the MFCCs, the audio-visual ambience control platform 110 may be configured to further process each assigned control channel to generate a control signal. As noted hereinabove, the extracted MFCCs and the identified features, patterns, timing, and/or musical expressions of the music file 104 such as tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, in particular form the basis for the generation of control signals for controlling the ambience elements. Further, in some embodiments, the processing of each assigned control channel to generate a control signal based on the extracted MFCCs may include normalization to ensure that all signals are within a determined range, modulation to create more complex signals, filtering to remove unwanted frequencies, or any other suitable transformation.
[0039] The control signal generation process is designed to be flexible and adaptable, allowing for a wide range of control signals to be generated depending on the needs of the specific application or setting. In particular, the properties of the MFCCs and the requirements of the ambience elements to be controlled are considered in the generation of the control signals. Accordingly, the control signals generated from the MFCCs are adaptable for the specific ambience elements in use. By way of example, identified MFCC features, patterns, timing, and/or musical expressions corresponding to the music file 104 such as tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, and the like are considered in the generation of the control signals. Further, as each MFCC corresponds to a different frequency, each MFCC may be employed as a control signal to facilitate actuation of the ambience elements in the vehicle. In one example, the value of an MFCC at a given time instant may be used to control an intensity of actuation of the lights, speakers, actuators, and/or the horn. In another example, the value of the MFCC may be used to determine an ON state or an OFF state of a light, an intensity of light/sound, and the like.
[0040] Accordingly, the audio-visual ambience control platform 110 may be configured to map the musical expressions, features, timing, and/or patterns obtained from the MFCCs to the control of the ambience elements. More particularly, the mapping of the musical expressions, patterns, timing, and/or features to the ambience elements corresponding to a specific application/setting may be generated. By way of example, in a setting that includes in-vehicle ambience generation, a mapping of musical expressions, features, timing, and/or patterns to the ambience elements corresponding to one or more vehicle models may be generated. The information related to the vehicle model may be obtained via an application configuration file. For example, a particular vehicle model may include a large number of lights. However, the musical expressions may be smaller in number. Hence, in this example, the audio-visual ambience control platform 110 is configured to group the numerous lights in the vehicle into a plurality of light groups. Also, in one example, the number of light groups may be based on the number of musical expressions. Furthermore, the musical expressions obtained for each time interval may be mapped to the light groups to control the operation of these lights in accordance with the music file 104. Accordingly, a mapping of the musical expressions/features/patterns/timing to the ambience elements to be controlled corresponding to a particular setting may be generated. In one example, the mapping may include one or more lighting sequences designed to control the operation of the lights in the vehicle in accordance with the music file 104. Consequent to the control signal generation process, in one non-limiting example, the control signals, a mapping of the lighting groups to musical expressions, light control sequences, speaker control sequences, horn control sequences, and the like may be generated.
[0041] Subsequently, the control signals and/or the mapping may be employed to control one or more ambience elements such as lights, dance lights, laser lights, speakers, horns, lasers, actuators, fountain jets, strobe lights, flashing lights, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, light and sound shows to create an enhanced immersive experience. In one example, the control signals may be used to convert the audio signal of the music file 104 into a visual display that is synchronized with the ambience elements such as lights, dance lights, strobe lights, flashing lights, laser lights, speakers, actuators, and/or horn based on the mapping or the identified features, patterns, timing, and/or musical expressions in the music file 104 to create an immersive light and sound show experience in the vehicle. It may also be noted in some embodiments, that the generation of the control signals based on the MFCCs may be performed prior to the assignment of the audio channels.
[0042] Implementing the control signal generation process as described hereinabove allows the control signal generation process to be flexible and adaptable to the music file 104, which in turn facilitates a wide range of control signals to be generated depending on the specific needs of the application or setting such as in-vehicle ambience generation, dance floor audio and lighting, musical fountains, light and sound shows, and the like.
[0043] Furthermore, in one embodiment, subsequent to MFCC extraction, the channel assignment, and control signal generation, in some embodiments, the audio-visual ambience control platform 110 may be configured to generate a timecode file. The timecode file includes information that is essential for controlling the ambience elements in response to the selected music file 104. In the present example of the in-vehicle ambience generation responsive to the input music file 104 based on the MFCCs, the timecode file may include information regarding a list of all the lights available in the particular vehicle model, a grouping of the lights in the vehicle, the MFCCs, the control signals, a mapping of the lighting groups to musical expressions, a light control time stamp (LCTS) and status of the light groups during each LCTS, the light control sequences, speaker control sequences, horn control sequences, and the like. In accordance with aspects of the present application, the audio-visual ambience control platform 110 may be configured to use the timecode file to control the various ambience elements in synchrony with the music file 104 to create an immersive audio-visual experience. More particularly, using the timecode file, the audio-visual ambience control platform 110 may be configured to control the various ambience elements in synchrony with the music file 104 such that the ambience dynamically responds to any changes in the music file 104. In one embodiment, the audio-visual ambience control platform 110 may be configured to communicate or issue control commands to the ambience elements to control the ambience elements based on the timecode file.
[0044] In accordance with aspects of the present specification, , once the music file 104 is selected by the user and received by the ambience control system 102, the audio-visual ambience control platform 110 is configured to verify if the selected music file 104 has a corresponding timecode file stored in the infotainment system 112, the ambience control system 102, or the data repository 118, for example. If it is determined that a timecode file corresponding to the music file 104 exists in the ambience control system 102, then the ambience control system 102 is configured to wait for a command from the user to start a light and sound show based on the selected music file 104. However, if a timecode file corresponding to the selected music file 104 does not exist in the ambience control system 102, then the audio-visual ambience control platform 110 is configured to generate a timecode file corresponding to the selected music file 104 and store the generated file and wait for a command from the user to start the light and sound show based on the selected music file 104. In one example, the generated timecode file may be stored in the infotainment system 112, the ambience control system 102, or the data repository 118. The generation of the timecode file will be described in greater detail with reference to FIG. 3.
[0045] Subsequent to the identification and/or generation of the timecode file, the ambience control system 102 is ready to perform the in-vehicle light and sound show to generate an immersive experience for the driver and/or passengers of the vehicle based on the selected music file 104. Once the audio-visual ambience control platform 110 receives a command such as a “START” command from the user or the ambience control system 102, the audio-visual ambience control platform 110 is configured to play the selected music file 104. Additionally, the audio-visual ambience control platform 110 is configured to use the corresponding timecode file to initiate and control the light and sound show in the vehicle using the various ambience elements such as lights, speakers, and/or the horn of the vehicle. More particularly, the controls signals generated based on the MFCCs/features/patterns/music expressions/timing and stored in the timecode file are utilized to control the various ambience elements to create the in-vehicle immersive experience. Specifically, the control signals are configured to cause the ambience elements to respond to the control signals by altering a current state to create an enhanced desired immersive ambience. Moreover, in one example, controlling the ambience elements via the control signals to alter a current state may include turning the lights on or off, changing the color, intensity, or brightness of the lights, adjusting the volume or tone of speakers and/or horns, moving actuators, and the like. The ambience elements could be simple or complex and may be controlled individually or controlled in groups. It may be noted that the ambience control system 102 is designed to work with a wide variety of ambience elements and to execute complex control sequences.
[0046] Additionally, the audio-visual ambience control platform 110 is configured to dynamically adjust the control signals based on real-time changes in the audio signal of the music file 104, thereby allowing the ambience control system 102 to be dynamically customizable and adaptable to new musical input in real-time.
[0047] Further, in certain embodiments, to control the ambience elements via the control signals, the audio-visual ambience control platform 110 may be configured to communicate the control signals to the corresponding ambience elements. In the example of FIG. 1, the audio-visual ambience control platform 110 may be configured to communicate the control signals to the lights, speakers, actuators, and/or the horn via a Controller Area Network (CAN) bus 120 in the vehicle. As will be appreciated, the vehicle includes a CAN bus 120 that enables communication across various components in the vehicle. Further, each vehicle includes a plurality of electronic control units (ECUs) 122 that are configured to keep the engine or vehicle running smoothly. The ECUs 122 are communicatively coupled to one another via the CAN bus 120. Reference numeral 124 is used to generally refer to the various lights in the vehicle, while the horn is generally represented by reference numeral 126. Also, reference numeral 128 is used to refer to the speakers in the vehicle.
[0048] As noted hereinabove, once the user initiates the light and sound show, the light and sound orchestration entails playing the music file 104 and controlling the ambience elements based on a corresponding timecode file in a synchronous manner. However, based on a setting, in certain situations, there is a perceptible delay in synchronization between the time the audio of the selected music file 104 starts playing and the time of control of corresponding ambience elements. In the example of in-vehicle ambience generation, the time taken from issuing a command via a light and sound application to the time taken to control an ambience element such as a light (ON/OFF) and the time at which the light turns ON/OFF, scheduling delay, CAN bus driver delay, delay due to communication over the CAN bus 120, and the like may adversely impact the synchronization between the time the audio of the selected music file 104 starts playing and the time of control of ambience elements. Additionally, there exists a delay due to the time taken from initiating the music file play request from the light and sound application to the sound starting to play at the speaker due to file parsing delay, decoding delay, buffering delay, audio driver delay, and the like. Another challenge to the synchronization between the time the audio of the selected music file 104 starts playing and the time of control of ambience elements is the notion that the time for audio play and the application time used to control the ambience elements are different. Consequently, although the audio play and the control of the ambience elements are initiated at the same time, over a period of time the audio play and the control of the ambience elements may drift apart. Hence it is desirable to minimize or optimize any synchronization delays by synchronizing in time, the control of the ambience elements with the audio play of musical expression.
[0049] To address the synchronization issues, in one embodiment, the audio-visual ambience control platform 110 may be configured to initiate playing of the music file 104, reading of the timecode file, and the controlling of the ambience elements at the same time. In another embodiment, to tackle the synchronization issues, depending on the system and/or the setting, the audio-visual ambience control platform 110 may be configured to introduce a configurable delay element. In this approach, a presentation time stamp of the audio being played may be matched with a corresponding timecode in the associated timecode file to select the appropriate control commands to control the ambience elements. These ambience element control commands may be advanced or delayed based on the configurable delay element with respect to the music file 104 being played if there is a synchronization issue. Moreover, the configurable delay element may be calibrated and included as part of the timecode file, in one example.
[0050] In accordance with one embodiment, in the example of the in-vehicle ambience generation, the audio-visual ambience control platform 110 is configured to communicate the control signals via the CAN bus 120 to an ECU 122 associated with the lights 124, the horn 126, and/or the speakers 128. Moreover, the ECUs 122 in turn are configured to control the switching on or off of the lights 124, control the intensity of the lights/horn/speakers 124, 126, 128 based on the control signal, and the like. Further, in one example, the ECU 122 may be a light control module (LCM) in the vehicle that is configured to control the complete lighting system including cabin lights, front lights, side lights, rear lights, working lights, and any other additional lamps or LED modules, and/or the speakers and horn function of the vehicle.
[0051] As noted hereinabove, in the example of the in-vehicle ambience generation, the audio-visual ambience control platform 110 is configured to communicate the control signals via the CAN bus 120. However, in different settings, different protocols and/or controls may be employed to facilitate the communication. By way of example, in the musical fountains setting, an actuator control may be used to facilitate the communication. Also, in a theater setting, a protocol adapter for a CAN bus may be utilized to facilitate the communication.
[0052] With continuing reference to FIG. 1, in certain embodiments, the ambience control system 102 may be integrated with an infotainment system 112 of the vehicle. However, in other embodiments, the ambience control system 102 may be a standalone unit and may be communicatively coupled to the infotainment system 112. In a presently contemplated configuration depicted in FIG. 1, the ambience control system 102 is depicted as being integrated with the infotainment system 112 of the vehicle. The ambience control system 102 may include a display 114 and a user interface 116. It may be noted that in some embodiments the display 114 and the user interface 116 may be representative of the display and user interface in the infotainment system 112 in the vehicle.
[0053] The display 114 and the user interface 116 may overlap in some embodiments such as a touch screen. Further, in some embodiments, the display 114 and the user interface 116 may include a common area. The display 114 may be configured to visualize or present any relevant information to the driver and passengers in the car. In one example, the relevant information may include a list of music files selected by the driver/passengers, the name of the song in the music file 104, the artiste performing the song, duration of the song, and other such information.
[0054] The user interface 116 of the ambience control system 118 may include a human interface device (not shown) that is configured to aid the driver/passengers in providing inputs or manipulating the outcomes visualized on the display 114. By way of example, the driver or passenger may select one or more music files 104 using the user interface 116. Additionally, the driver and passengers may also enter the commands such as “START,” “STOP,” “PAUSE,” and “RESUME” to be performed on the music file 104 via the user interface 116 to control and/or modify the in-vehicle ambience. In certain embodiments, the human interface device may include a trackball, a joystick, a stylus, a mouse, or a touch screen. It may be noted that the user interface 116 may be configured to aid the driver and/or a passenger in navigating through the inputs and/or outputs generated by the ambience control system 102.
[0055] Implementing the system 100 and the ambience control system 102 that includes the audio-visual ambience control platform 110 as described hereinabove aids in controlling various ambience elements such as lights, speakers, lasers, actuators, horns, fountain jets, dancing lights, strobe lights, and other audio-visual equipment to improve entertainment and provide an enhanced ambience in various settings based on the MFCCs associated with the music file 104. In particular, the system 100 is designed to effectively utilize properties of musical input to provide real-time automated audio-visual control of the in-vehicle ambience, and where the system 100 is configured to dynamically adjust the ambience in the vehicle in response to real-time changes in the audio signal of the musical input. Additionally, the ambience control system 102 utilizes the properties of the musical input 104 in the form of MFCCs to create a rich, immersive, and responsive ambience in a given setting. Each MFCC is assigned to a different control channel. Control signals corresponding to each control channel are employed to control various types of ambience elements, such as, but not limited to conventional lights, horns, lasers, speakers, musical fountains, disco lights, and actuators. Moreover, the ambience control system 102 is configured to dynamically adjust the ambience in response to any changes in the music file 104 and can be applied in various settings, including in-vehicle ambience generation, dance floor audio and lighting control, musical fountains, sound and light shows, live dance and music shows, and more. Additionally, the ambience control system 102 is configured to dynamically adjust the control signals based on real-time changes in the audio signal of the music file 104, thereby allowing the systems and methods to be customizable and adaptable to new musical input in real-time. Additionally, the system 100 is configured to optimize/minimize any delays in synchronization between the time the audio of the selected music file 104 starts playing and the time of control of ambience elements by synchronizing in time, the control of the ambience elements with the playing of the music file 104.
[0056] The overall operation of the ambience control system 102 is designed to be highly efficient and robust, thereby ensuring a reliable and high-quality output. Furthermore, the ambience control system 102 is designed to be highly flexible and customizable, thereby allowing the ambience control system 100 to be adapted for a wide range of applications and settings. The working of the system 100 may be better understood with reference to FIGs. 2-7.
[0057] It may be noted that the ambience control system 102 described hereinabove may be applied in a variety of settings. In the example of FIG. 1, the ambience control system 102 may be installed in vehicles to generate a unique in-vehicle immersive experience based on the music playing in the vehicle. The ambience control system 102 may also be used to control lighting and audio in a dance floor setting to create a responsive and immersive environment. Other applications of the ambience control system 102 include musical fountains, where the ambience control system 102 may be employed to control water movement and light effects based on music. Additionally, the ambience control system 102 may be used in sound and light shows, where various audio-visual effects can be synchronised with the music.
[0058] Embodiments of the exemplary method of FIG. 2 may be described in a general context of computer executable instructions on computing systems or a processor. Generally, computer executable instructions may include routines, programs, objects, components, data structures, procedures, modules, functions, and the like that perform particular functions or implement particular abstract data types.
[0059] Moreover, the embodiments of the exemplary methods may be practised in a distributed computing environment where optimization functions are performed by remote processing devices that are linked through a wired and/or wireless communication network. In the distributed computing environment, the computer executable instructions may be located in both local and remote computer storage media, including memory storage devices.
[0060] Additionally, in FIG. 2 the exemplary method is illustrated as a collection of blocks in a logical flow chart, which represents operations that may be implemented in hardware, software, firmware, or combinations thereof. It may be noted that the various operations are depicted in the blocks to illustrate the functions that are performed. In the context of software, the blocks represent computer instructions that, when executed by one or more processing subsystems, perform the recited operations.
[0061] Moreover, the order in which the exemplary methods are described is not intended to be construed as a limitation, and any number of the described blocks may be combined in any order to implement the exemplary methods disclosed herein, or equivalent alternative methods. Further, certain blocks may be deleted from the exemplary methods or augmented by additional blocks with added functionality without departing from the spirit and scope of the subject matter described herein.
[0062] Turning now to FIG. 2, a flow chart 200 of an exemplary method for real-time automated audio-visual control of ambience based on musical input is presented. For ease of explanation, in the present specification, the method 200 may generally be referred to as a method for in-vehicle ambience generation. Moreover, in accordance with aspects of the present specification, the method 200 for real-time automated audio-visual control of ambience based on musical input for in-vehicle ambience generation is designed to effectively utilize properties of musical input to create a rich, immersive, and responsive ambience using MFCCs associated with the musical input. The method 200 entails receiving musical input and effectively utilizing the properties of the musical input to create an enhanced in-vehicle ambience using the MFCCs corresponding to the musical input. In addition, the method 200 facilitates dynamically adjusting the ambience in response to any changes in the musical input in various settings, thereby allowing the method 200 to be dynamically customizable and adaptable to new musical input in real-time.
[0063] The method 200 of FIG. 2 is described with reference to the components of FIG. 1. Moreover, in certain embodiments, the method 200 may be performed by the ambience control system 102 that includes the audio-visual ambience control platform 110.
[0064] In accordance with aspects of the present application, a user such as a driver of the vehicle or a passenger in the vehicle may initiate an enhanced immersive experience via a light and sound show application. The enhanced immersive experience in the vehicle may include a light and sound show that is generated in response to the input music file 104. Also, the light and sound show may entail control of one or more combinations of various ambience elements such as lights, speakers, actuators, and the horn in the vehicle based on the musical input to create the enriched immersive and responsive ambience/experience.
[0065] In one example, the light and sound show application may be provided on an infotainment system 112 of the vehicle. Further, to initiate the light and sound show the user may select a desired music file. In some other embodiments, the user may select the music file stored in the infotainment system 112, the data repository 118, or from a storage device that is communicatively coupled to the ambience control system 102. Subsequently, the user may launch the light and sound show by touching a button corresponding to the light and sound show application on the display 114 of the infotainment system 112 of the vehicle.
[0066] The method starts at step 202, where musical input selected by the user is received by the ambience control system 102. The musical input may be in the form of a music file 104. The musical input 104 may include one or more music files corresponding to a variety of genres. Also, the music file 104 may include instrumental music, people singing, and the like. Furthermore, in one embodiment, the ambience control system 102/audio-visual ambience control platform 110 may receive these music files 104 from the infotainment system 112 of the vehicle or from other sources such as a USB device, a CD, a cassette, a DVD, a Blu-ray disc, digital files, live performances, online streaming platforms, the data repository 118, or any other source of musical output. It may also be noted that in certain embodiments, the ambience control system 102 may be configured to automatically select the music file 104.
[0067] Subsequent to the selection of the music file 104 and the initiation of the light and sound show application, the acquisition subsystem 106 is configured to receive the music file 104 from the source. Moreover, the music file 104 may be processed to be compatible with a range of audio formats and sources to ensure versatility and broad application. Furthermore, the music file 104 is communicated to the processing subsystem 108 for further processing. In one embodiment, the processing subsystem 108 and the audio-visual ambience control platform 110 in particular is configured to process the selected music file 104 to control the ambience elements in the vehicle to create an enhanced and more immersive audio-visual experience based on MFCCs extracted from the music file 104. Additionally, the audio-visual ambience control platform 110 is configured to control the ambience such that the ambience dynamically responds to any changes in the selected music file 104.
[0068] Subsequently, as indicated by step 204, the music file 104 may be processed by the audio-visual ambience control platform 110 to extract the MFCCs. In one example, a time domain signal of the music file 104 is transformed into the frequency domain and processed to extract the MFCCs.
[0069] Moreover, at step 206, the frequency domain representation of the music file 104 is parsed to identify specific features, musical expressions, timing, and/or patterns corresponding to the MFCCs in the music file 104. Some non-limiting examples of the specific features, patterns, and/or musical expressions of the music file 104 include tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, and the like. It may be noted that the MFCC extraction process is optimized to ensure accuracy and efficiency.
[0070] Subsequent to the extraction of the MFCCs, the audio-visual ambience control platform 110 is configured to assign control channels based on the generated MFCCs, as depicted by step 208. In one embodiment, the audio-visual ambience control platform 110 is configured to assign each MFCC to a different control channel based on the properties of the MFCCs and/or the requirements of the ambience elements that are to be controlled. Also, in certain embodiments, each MFCC may be assigned to a different control channel in a sequential fashion, a random fashion, or based on a predetermined mapping.
[0071] Furthermore, at step 210, the audio-visual ambience control platform 110 may be configured to process the audio/control channels to generate control signals based on the MFCCs. In particular, the control signals for controlling the devices or ambience elements are generated based on the identified features, musical expressions, timing, and/or patterns in the music file 104. The control signals are employed to control one or more devices or ambience elements such as lights, dance lights, laser lights, speakers, horns, lasers, actuators, fountain jets, flashing lights, strobe lights, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, and light and sound shows.
[0072] Further, the control signals may be customized by generating a mapping of the musical expressions/features/patterns/timing to the ambience elements to be controlled that corresponds to a particular setting. By way of example, a mapping of musical expressions, features, timing, and/or patterns to the ambience elements corresponding to each vehicle model may be generated. Also, the control signals are dynamically adjusted based on real-time changes in the audio signal of the music file 104, thereby allowing the method 200 to be dynamically customizable and adaptable to new musical input in real-time. Consequent to the control signal generation process of step 210, in one non-limiting example, the control signals, a mapping of the lighting groups to musical expressions, light control sequences, speaker control sequences, horn control sequences, and the like may be generated.
[0073] In accordance with further aspects of the present specification, following the MFCC extraction, the control channel assignment, and the control signal generation, in some embodiments, the audio-visual ambience control platform 110 may be configured to generate a timecode file. The timecode file includes information that is essential for controlling the ambience elements in response to the selected music file 104. In one example, the timecode file may include information regarding a list of all the lights available in the particular vehicle model, a grouping of the lights in the vehicle, the MFCCs, the control signals, a mapping of the lighting groups to musical expressions, a light control time stamp (LCTS) and status of the light groups during each LCTS, the light control sequences, the speaker control sequences, the horn control sequences, fountain jet control sequences, and the like. The timecode file may be employed to control the various ambience elements in synchrony with the music file 104 to create an immersive audio-visual experience in a setting, where the ambience dynamically responds to the music file 104.
[0074] Accordingly, a check is carried out to verify if a timecode file corresponding to the music file 104 exists, as indicated by step 212. At step 212, if it is determined that a timecode file corresponding to the music file 104 exists in the ambience control system 102, then control is passed to step 216. However, at step 212, if it is determined that a timecode file corresponding to the selected music file 104 does not exist in the ambience control system 102, a timecode file corresponding to the selected music file 104 is created/generated and stored, as indicated by step 214. In one example, the generated timecode file may be stored in the infotainment system 112, the ambience control system 102, or the data repository 118. Once the timecode file is generated, the ambience control system 102 is ready to perform the in-vehicle light and sound show. The generation of the MFCCs, the assignment of control channels, the generation of control signals, and the creation of the timecode file of steps 202-214 will be described in greater detail with reference to FIG. 3.
[0075] Subsequently, as depicted by step 216, the ambience control system 102 is configured to wait for a command from the user to initiate the in-vehicle light and sound show to generate an immersive experience for the driver and/or passengers of the vehicle based on the selected music file 104. By way of example, the ambience control system 102 is configured to wait for a “START” command from the user to start the light and sound show based on the selected music file 104.
[0076] In response to receipt of the “START” command from the user, the selected music file 104 is played by the ambience control system 102, as indicated by step 218. In some embodiments, the selected music file 104 may be automatically played by the ambience control system 102 without waiting for the “START” command from the user. Additionally, at step 218, information stored in the timecode file is utilized by the ambience control system 102 to initiate and control the light and sound show in the vehicle using the various ambience elements of the vehicle to create a rich, immersive, and responsive ambience in the vehicle.
[0077] In particular, the control signals and/or the mapping in the timecode file may be employed to control one or more ambience elements such as lights, dance lights, laser lights, speakers, horns, lasers, actuators, fountain jets, strobe lights, flashing lights, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, and light and sound shows to create an enhanced immersive experience for the driver and/or passengers in the vehicle. In one example, the control signals may be used to convert the audio signal of the music file 104 into a visual display that is synchronized with the ambience elements such as lights, dance lights, laser lights, speakers, actuators, and/or horn based on the mapping or the identified features, patterns, timing, and/or musical expressions in the music file 104 to create an immersive light and sound show experience in the vehicle.
[0078] As noted hereinabove, once the user initiates the light and sound show, based on a setting, there may a perceptible delay in synchronization between the time the audio of the selected music file 104 starts playing and the time of control of corresponding ambience elements in the light and sound orchestration. Hence it is desirable to minimize or optimize any synchronization delays by synchronizing in time, the control of the ambience elements with the audio play of musical expression. Accordingly, at step 218, to address the synchronization issues, playing of the music file 104, reading of the timecode file, and controlling of the ambience elements may be initiated at the same time, in one example. Moreover, in another example, to tackle the synchronization issues, depending on the system and/or the setting, a configurable delay element may be introduced. In this approach, a presentation time stamp of the audio being played may be matched with a corresponding timecode in the associated timecode file to select the appropriate control commands to control the ambience elements. If there is a synchronization issue these ambience element control commands may be advanced or delayed based on the configurable delay element with respect to the music file 104 being played Also, the configurable delay element may be calibrated and included as part of the timecode file, in one example.
[0079] Moreover, in one example, the control signals may be communicated from the audio-visual ambience control platform 110 via the CAN bus 120 to one or more ECUs 122. The ECUs 122 in turn are configured to control the actuation of the lights, speakers, and/or horn based on the MFCC based control signals.
[0080] Implementing the method for in-vehicle ambience generation as described hereinabove facilitates utilization of properties of musical input to create a rich, immersive, and responsive ambience using MFCCs corresponding to the musical input. In particular, the method 200 is designed to effectively utilize properties of the musical input to provide real-time automated audio-visual control of the in-vehicle ambience. Additionally, the method 200 is configured to dynamically adjust the ambience in the vehicle in response to real-time changes in the audio signal of the musical input. Moreover, the method 200 efficiently provides control of lights, speakers, lasers, horns, and other audio-visual equipment to improve entertainment and provide an enhanced ambience in various settings based on the MFCCs associated with the music file 104. Furthermore, the method 200 employs the properties of the musical input 104 in the form of MFCCs to create a rich, immersive, and responsive ambience. Moreover, the method 200 is configured to dynamically adjust the ambience in response to the music file 104 and can be applied in various settings, including in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, sound and light shows, and more. Furthermore, the overall operation of the method 200 is designed to be highly efficient and robust, ensuring a reliable and high-quality output. Also, since the method 200 is designed to dynamically adjust the ambience in response to any music file input 104, the method 200 is highly flexible and customizable, thereby allowing the method 200 to be adapted for a wide range of applications and settings.
[0081] FIG. 3 is a schematic illustration 300 of steps 202-214 of FIG. 2. In particular, FIG. 3 illustrates one example of the generation of a timecode file based on MFCCs, specific features, musical expressions, timing, and/or patterns associated with a music file. The method 300 of FIG. 3 is described with reference to the components of FIGs. 1-2. Also, in one embodiment, the steps 202-214 depicted in FIG. 3 may be performed by the ambience control system 102 and the audio-visual ambience control platform 110, in particular.
[0082] In accordance with aspects of the present specification, a timecode file is created for each music file 302. It may be noted that the music file 302 may generally be representative of the music file 104 of FIG. 1. The timecode file includes information that is essential for controlling the various ambience elements in response to the input music file 302. In the present example of the in-vehicle ambience generation, responsive to the input music file 302 and based on the extracted MFCCs, specific features, musical expressions, timing, and/or patterns associated with the music file 302, the timecode file may include information regarding a list of all the lights available in the particular vehicle model, a grouping of the lights in the vehicle, the MFCCs, the control signals, a mapping of the lighting groups to musical expressions, a light control time stamp (LCTS) and status of the light groups during each LCTS, light control sequences, speaker control sequences, horn control sequences , and the like. Furthermore, in accordance with aspects of the present application, the audio-visual ambience control platform 110 may be configured to use the timecode file to create an immersive audio-visual experience, where the ambience dynamically responds to the music file 302.
[0083] Furthermore, with continuing reference to the example of in-vehicle ambience generation, a timecode file is generated for each music file for each vehicle model as the configuration of lights, speakers, and/or horn and the list of available lights in each vehicle model may be different. Consequently, the timecode file content is customized for each music file to correspond to a given vehicle model. Moreover, a timecode file is typically generated off-line before starting the light and sound show. However, if a timecode file corresponding to a music file is not available, a timecode file may be generated in real-time. These timecode files may be stored in the ambience control system 102, the data repository 118, the infotainment system 112, and the like.
[0084] As will be appreciated, the various music files 302 are available in different formats, such as, but not limited to MP3, AIFF, AAC, WAV, WMA, OGG, FLAC, ALAC, and the like. Accordingly, as indicated by block 304, to facilitate the generation of the timecode file corresponding to an input music file 302, the audio-visual ambience control platform 110 may be configured to parse the music file 302 based on the format to extract audio channels 306 present in the music file 302. It may be noted that the number of audio channels 306 in the music file 302 is based on the type of the music file 302. In one example, the number of audio channels 306 may be 1 or 2. Also, in one embodiment, an application programming interface (API) may be employed to extract the audio channels 306 from the music file 302. Further, as will be appreciated, data in the audio channels 306 is typically encoded to reduce the size of the music file 302. Accordingly, as depicted by block 304, the audio in the music file 302 may be decoded to get the raw audio samples. In one example, an audio Codec API may be used to decode the audio data.
[0085] Moreover, the audio-visual ambience control platform 110 may be configured to pre-process and/or mix the extracted audio channels 306 based on the audio content/data in the music file 302, as indicated by block 308. In one example, the sampling rate of the audio data may be different for different music files. Accordingly, the audio data in the various music files 302 may be pre-processed to standardize the sampling rate across the music files 302 to one rate before any further processing. Also, the music files 302 may have one or more audio channels. For example, stereo music has two channels. Accordingly, all the audio channels may be mixed into one channel to make further processing easier and less processing intensive.
[0086] In accordance with exemplary aspects of the present specification, as depicted by block 310, the audio-visual ambience control platform 110 may be configured to process the music file(s) 302 to obtain MFCCs 312 from the music file 302. To that end, the music file 302 is framed into short frames. Subsequently, the short frames of the music file 302 are transformed from the time domain to the frequency domain via use of a Fourier transform. Also, the powers of the spectrum obtained subsequent to the processing via the Fourier transform may be mapped onto the Mel scale via use of a Mel filter bank applied to the power spectrum. Moreover, logarithms of the powers at each of the Mel frequencies may be taken. Additionally, the Mel log powers may be processed via a discrete cosine transform (DCT) to generate MFCCs 312. Coefficients 2-13 may be retained and the rest of the coefficients may be discarded. These MFCCs 312 are representative of the amplitudes of the resulting spectrum. In accordance with exemplary aspects of the present specification, these MFCCs 312 form the basis for the subsequent generation of control signals for controlling the ambience elements such as lights, speakers, actuators, and/or the horn in the vehicle.
[0087] With continuing reference to block 310, the audio-visual ambience control platform 110 is further configured to extract musical expressions, features, patterns, timing, or combinations thereof from the music file 302 based on the MFCCs 312. In one embodiment, at each time instant, the frequency domain representation of the music file 104 may be parsed to identify specific features, musical expressions, timing, and/or patterns in the music file 104. As previously noted, some non-limiting examples of the specific features, patterns, timing, and/or musical expressions of the music file 104 include tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, and the like. Also, the time interval used may be decided based on the maximum rate at which the lights, speakers, and/or horn can be controlled.
[0088] Furthermore, as depicted by block 314, subsequent to the generation of the MFCCs 312, the audio-visual ambience control platform 110 may be configured to assign each MFCC 312 to a control channel. This assignment of MFCCs 312 to different channels may be performed in a variety of ways. By way of example, the audio-visual ambience control platform 110 may be configured to assign each MFCC 312 to a different control channel in a sequential fashion, a random fashion, or based on a predetermined mapping.
[0089] Also, in some embodiments, the audio-visual ambience control platform 110 may be configured to assign each MFCC 312 to a different control channel based on the properties of the MFCCs 312, the requirements of the ambience elements that are to be controlled, and/or the setting. In the example of in-vehicle ambience generation, information regarding the different ambience elements associated with a variety of vehicle models may be obtained from a vehicle model configuration file stored in a device configuration data repository 316, for example. For example, the lighting configuration of different vehicle models may be obtained from the device configuration data repository 316 and may be considered in the assignment of the MFCCs 312 to the different control channels. Also, the configuration or setup of the ambience elements to be controlled such as lights, horns, speakers, actuators, and the like may be accounted for in the channel assignment process of the MFCCs 312. Taking into consideration the properties of the MFCCs 312, the requirements associated with the ambience elements to be controlled, and/or the setting ensures that the control signals generated from the MFCCs 312 are suited for the specific ambience elements in use.
[0090] Subsequently, as indicated by block 318, the audio-visual ambience control platform 110 may be configured to further process each assigned control channel to generate a control signal 320. As noted hereinabove, the extracted MFCCs 312 and the identified features, patterns, timing, and/or musical expressions of the music file 104 such as tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, in particular form the basis for the generation of control signals 320 for controlling the ambience elements. The processing of each assigned control channel to generate a control signal 320 based on the extracted MFCCs 312 and associated properties of the MFCCs may include normalization to ensure that all signals are within a determined range, modulation to create more complex signals, filtering to remove unwanted frequencies, or any other suitable transformation.
[0091] In accordance with aspects of the present specification, the control signals 320 are generated so as to be flexible and adaptable. Consequently, a wide range of control signals 320 may be generated depending on the needs of the specific application or setting. More particularly, since the process for generating the control signals 320 takes into consideration the properties of the MFCCs 312, the requirements of the ambience elements to be controlled, and/or the setting, the control signals 320 generated from the MFCCs 312 are designed to be adaptable for specific ambience elements and settings.
[0092] Accordingly, to generate control signals 320 that are adaptable for specific ambience elements and/or settings, the audio-visual ambience control platform 110 may be configured to map the musical expressions, patterns, timing, and/or features obtained from the MFCCs 312 to the ambience elements to provide customized control of the ambience elements. More particularly, the mapping of the musical expressions, patterns, timing, and/or features to the ambience elements corresponding to a specific application/setting may be generated. In the example of in-vehicle ambience generation, a mapping of the musical expressions, patterns, timing, and/or features to the ambience elements corresponding to one or more vehicle models may be generated. The information related to the vehicle model may be obtained via an application configuration file in the device configuration data repository 316. By way of example, a particular vehicle model may include a large number of lights. However, the musical expressions may be smaller in number. Hence, in this example, the audio-visual ambience control platform 110 is configured to group the numerous lights into a plurality of light groups. Also, in one example, the number of light groups may be based on the number of musical expressions. Furthermore, the musical expressions obtained for each time interval may be mapped to the light groups to control the operation of these lights in accordance with the music file 104. In one example, the control signals 320 may include light control sequences, horn control sequences, speaker control sequences, and the like.
[0093] The control signals 320 and/or the mapping may be employed to control one or more ambience elements such as lights, dance lights, laser lights, strobe lights, flashing lights, speakers, horns, lasers, actuators, fountain jets, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, and light and sound shows to create an enhanced immersive experience. In one example, the control signals 320 may be used to convert the audio signal of the music file 104 into a visual display that is synchronized with the ambience elements such as lights, dance lights, laser lights, strobe lights, flashing lights, speakers, actuators, and/or horn based on the mapping of the identified features, patterns, timing, and/or musical expressions in the music file 104 to the ambience elements to create an immersive light and sound show experience in the vehicle.
[0094] Subsequently, as depicted by block 322, a timecode file 324 may be generated. In one embodiment, the timecode file 324 may be generated in a pre-defined format. Accordingly, in one example, the timecode file 324 may include a list of all the ambience elements such as lights, dance lights, laser lights, flashing lights, strobe lights, speakers, horns, lasers, actuators, fountain jets, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, and light and sound shows, a grouping of the lights, the MFCCs 312, the control signals 320, a mapping of the lighting groups to musical expressions, a light control time stamp (LCTS) and status of the light groups during each LCTS, light control sequences, horn control sequences, speaker control sequences, fountain jet control sequences, and the like.
[0095] As noted hereinabove, the timecode file 324 so generated includes information that is essential for controlling the various ambience elements in different settings in response to the input music file 302. The audio-visual ambience control platform 110 may use the timecode file 324 to control the various ambience elements to generate an enhanced immersive experience for the driver and/or passengers in the vehicle in various settings.
[0096] FIG. 4 is a diagrammatical illustration 400 of one embodiment of the implementation of the system for real-time automated audio-visual control of ambience based on musical input, in accordance with aspects of the present specification.
[0097] In the example depicted in FIG. 4, a schematic illustration 400 of the infotainment system 112 of FIG. 1 is presented. The user may select a music file to be played by selecting the button 406, in one example. Further, in the example embodiment of FIG. 4, a button 408 corresponding to a light and sound show application is visualized on a display 402 of the infotainment system 112 of the vehicle. The user may initiate the light and sound show based on the selected music file by selecting this button 408 to create an enhanced immersive experience in the vehicle. Reference numeral 404 is used to refer to other controls on the infotainment system 112. It may be noted that the example of the visual representation presented in FIG. 4 is for illustrative purposes. Other designs are also anticipated.
[0098] Turning now to FIG. 5, an example embodiment 500 of the audio-visual ambience control platform 110 of FIG. 1, in accordance with aspects of the present specification, is depicted. The embodiment 500 of FIG. 5 is described with reference to the components of FIGs. 1-4.
[0099] In a presently contemplated configuration, the audio-visual ambience control platform 110 is depicted as including a pre-processing unit 502, an MFCC extraction unit 504, a channel assignment unit 506, a control signal generation unit 508, a timecode file generation unit 510, and a device control unit 512. It may be noted the embodiment 400 depicted in FIG. 5 is for illustrative purposes. Other designs are also contemplated.
[0100] The pre-processing unit 502 is configured to receive the music file 104 from various sources. Further, the pre-processing unit 502 is configured to parse the input music file 104 based on a format and extract the audio channels in the music file 104. Also, the pre-processing unit 502 is configured to decode the data in the music file 104 to extract raw audio samples.
[0101] Moreover, the MFCC extraction unit 504 is configured to process the input music file 104 to extract the MFCCs. To extract the MFCCs, the MFCC extraction unit 504 is configured to apply a Fourier transform to convert the time-domain signal of the music file 104 to the frequency domain, followed by a Mel filter bank application. The logarithm is then taken, and a DCT is applied to extract the MFCCs. The MFCCs represent the power spectrum of a sound and form the basis for the subsequent generation of control signals. The MFCC extraction unit 504 is also configured to parse the MFCCs at each time instant to identify properties such as specific features, patterns, timing, and/or musical expressions in the music file 104. Some non-limiting examples of the specific features, patterns, timing, and/or musical expressions of the music file 104 include tempo, rhythm, pitch, timbre, dynamics, melody, harmony, beats, and the like.
[0102] In addition, the channel assignment unit 506 is configured to assign each MFCC to a different control channel. This assignment of control channels may be performed in various ways, such as sequentially, randomly, or based on a predetermined mapping. Moreover, the channel assignment unit 506 may also consider the properties of the MFCCs and the requirements of the ambience elements that are to be controlled in the channel assignment process, thereby ensuring that the control signals generated from the MFCCs are suited for the specific ambience elements in use.
[0103] The control signal generation unit 508 is configured to process each assigned control channel to generate a control signal, where the control signal is configured to control the ambience elements to create the rich, immersive, and responsive ambience/experience. As previously noted, the MFCCs form the basis for the generation of control signals. The processing may include normalization to ensure all signals are within a certain range, modulation to create more complex signals, filtering to remove unwanted frequencies or any other suitable transformation. It may be noted that the control signal generation process is designed to be flexible and adaptable, thereby allowing for a wide range of control signals to be generated depending on the specific application’s needs.
[0104] Additionally, the timecode generation file unit 510 is configured to generate a timecode file which includes information that is essential for controlling the ambience elements in response to the selected music file 104. In the example of the in-vehicle ambience generation, the timecode file may include information regarding a list of all the lights available in the vehicle, a grouping of the lights in the vehicle, the MFCCs, the control signals, the musical expressions, a mapping of the lighting groups to the musical expressions, a light control time stamp (LCTS) and status of the light groups during each LCTS, light control sequences, speaker control sequences, horn control sequences, and the like. The timecode file is used to create an immersive audio-visual experience, where the ambience dynamically responds to the music file 104.
[0105] Also, the device control unit 512 is configured to use the control signals to control various types of ambience elements, such as lights, dance lights, laser lights, flashing lights, strobe lights, speakers, horns, lasers, actuators, fountain jets, and the like in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, and light and sound shows. The device control unit 512 may be configured to communicate the control signals to the corresponding ambience elements to control the operation of the ambience elements based on the input music file. In one example, control of the ambience elements based on the control signals may entail turning lights on or off, changing the color and/or brightness of the lights, adjusting the volume or tone of speakers, actuating the fountain jets, moving actuators, and the like. The ambience elements to be controlled may be simple or complex and may be controlled individually or in groups. Further, the device control unit 512 is designed to work with a wide variety of ambience elements and to execute complex control sequences.
[0106] As previously, once the user initiates the light and sound show, based on a setting, there may a perceptible delay in synchronization between the time the audio of the selected music file 104 starts playing and the time of control of corresponding ambience elements in the light and sound orchestration. Accordingly, the device control unit 512 may be configured to minimize or optimize any synchronization delays by synchronizing in time the control of the ambience elements with the audio play of musical expression. In particular, to address the synchronization issues, the device control unit 512 may be configured to initiate playing of the music file 104, reading of the timecode file, and controlling of the ambience elements at the same time, in one example. Moreover, in another example, to tackle the synchronization issues, the device control unit 512 may be configured to introduce a configurable delay element depending on the system and/or the setting. In this approach, the device control unit 512 may be configured to match a presentation time stamp of the audio being played with a corresponding timecode in the associated timecode file to select the appropriate control commands to control the ambience elements. Furthermore, if there is a synchronization issue, the device control unit 512 may be configured to advance or delay based on the configurable delay element these ambience element control commands with respect to the music file 104 being played. Moreover, the device control unit 512 may also be configured to calibrate the configurable delay element and include the configurable delay element as part of the timecode file, in one example.
[0107] Referring now to FIG. 6, a schematic representation 600 of one embodiment 602 of a digital processing system implementing the implementing the audio-visual ambience control platform 110 (see FIG. 1), in accordance with aspects of the present specification, is depicted. Also, FIG. 6 is described with reference to the components of FIGs. 1-5.
[0108] It may be noted that while the audio-visual ambience control platform 110 is shown as being a part of the ambience control system 102, in certain embodiments, the audio-visual ambience control platform 110 may also be integrated into end user systems such as, but not limited to, the infotainment system 112 in the vehicle. Moreover, the example of the digital processing system 602 presented in FIG. 6 is for illustrative purposes. Other designs are also anticipated.
[0109] The digital processing system 602 may contain one or more processors such as a central processing unit (CPU) 604, a random access memory (RAM) 606, a secondary memory 608, a graphics controller 610, a display unit 612, a network interface 614, and an input interface 616. It may be noted that the components of the digital processing system 602 except the display unit 612 may communicate with each other over a communication path 618. In certain embodiments, the communication path 618 may include several buses, as is well known in the relevant arts.
[0110] The CPU 604 may execute instructions stored in the RAM 606 to provide several features of the present specification. Moreover, the CPU 604 may include multiple processing units, with each processing unit potentially being designed for a specific task. Alternatively, the CPU 604 may include only a single general-purpose processing unit.
[0111] Furthermore, the RAM 606 may receive instructions from the secondary memory 608 using the communication path 618. Also, in the embodiment of FIG. 6, the RAM 606 is shown as including software instructions constituting a shared operating environment 620 and/or other user programs 622 (such as other applications, DBMS, and the like). In addition to the shared operating environment 620, the RAM 606 may also include other software programs such as device drivers, virtual machines, and the like, which provide a (common) run time environment for execution of other/user programs.
[0112] With continuing reference to FIG. 6, the graphics controller 610 is configured to generate display signals (e.g., in RGB format) for display on the display unit 612 based on data/instructions received from the CPU 604. The display unit 612 may include a display screen to display images defined by the display signals. Furthermore, the input interface 616 may correspond to a keyboard and a pointing device (e.g., a touchpad, a mouse, and the like) and may be used to provide inputs. In addition, the network interface 614 may be configured to provide connectivity to a network (e.g., using Internet Protocol), and may be used to communicate with other systems connected to a network, for example.
[0113] Moreover, the secondary memory 608 may include a hard drive 626, a flash memory 628, and a removable storage drive 630. The secondary memory 608 may store data generated by the system 100 (see FIG. 1) and software instructions (for example, for implementing the various features of the present specification), which enable the digital processing system 602 to provide several features in accordance with the present specification. The code/instructions stored in the secondary memory 608 may either be copied to the RAM 606 prior to execution by the CPU 604 for higher execution speeds or may be directly executed by the CPU 604.
[0114] Some or all of the data and/or instructions may be provided on a removable storage unit 632, and the data and/or instructions may be read and provided by the removable storage drive 630 to the CPU 604. Further, the removable storage unit 632 may be implemented using medium and storage format compatible with the removable storage drive 630 such that the removable storage drive 630 can read the data and/or instructions. Thus, the removable storage unit 632 includes a computer readable (storage) medium having stored therein computer software and/or data. However, the computer (or machine, in general) readable medium can also be in other forms (e.g., non-removable, random access, and the like.).
[0115] It may be noted that as used herein, the term “computer program product” is used to generally refer to the removable storage unit 632 or a hard disk installed in the hard drive 626. These computer program products are means for providing software to the digital processing system 602. The CPU 604 may retrieve the software instructions and execute the instructions to provide various features of the present specification.
[0116] Also, the term “storage media/medium” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may include non-volatile media and/or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, or solid-state drives, such as the secondary memory 608. Volatile media include dynamic memory, such as the RAM 606. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
[0117] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, the transmission media may include coaxial cables, copper wire, and fiber optics, including the wires that include the communication path 618. Moreover, the transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
[0118] Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present specification. Thus, appearances of the phrases “in one embodiment,” “in an embodiment” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.
[0119] Furthermore, the described features, structures, or characteristics of the specification may be combined in any suitable manner in one or more embodiments. In the description presented hereinabove, numerous specific details are provided such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, and the like, to provide a thorough understanding of embodiments of the specification.
[0120] The aforementioned components may be dedicated hardware elements such as circuit boards with digital signal processors or may be software running on a general-purpose computer or processor such as a commercial, off-the-shelf personal computer (PC). The various components may be combined or separated according to various embodiments of the invention.
[0121] Furthermore, the foregoing examples, demonstrations, and process steps such as those that may be performed by the system may be implemented by suitable code on a processor-based system, such as a general-purpose or special-purpose computer. It should also be noted that different implementations of the present specification may perform some or all of the steps described herein in different orders or substantially concurrently, that is, in parallel. Furthermore, the functions may be implemented in a variety of programming languages, including but not limited to C++, Python, and Java. Such code may be stored or adapted for storage on one or more tangible, machine readable media, such as on data repository chips, local or remote hard disks, optical disks (that is, CDs or DVDs), memory or other media, which may be accessed by a processor-based system to execute the stored code. Note that the tangible media may include paper or another suitable medium upon which the instructions are printed. For instance, the instructions may be electronically captured via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner if necessary, and then stored in the data repository or memory.
[0122] The aforementioned components may be dedicated hardware elements such as circuit boards with digital signal processors or may be software running on a general-purpose computer or processor such as a commercial, off-the-shelf personal computer (PC). The various components may be combined or separated according to various embodiments of the invention.
[0123] Embodiments of the systems and methods for real-time automated audio-visual ambience control based on musical input described hereinabove advantageously present a robust framework for controlling various ambience elements such as lights, dance lights, laser lights, strobe lights, flashing lights, speakers, horns, lasers, actuators, fountain jets, and other audio-visual equipment to improve entertainment and provide an enhanced rich, responsive, and immersive ambience in various settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, and light and sound shows using MFCCs corresponding to musical input. In particular, the systems and methods are designed to effectively utilize properties of the musical input to provide real-time automated audio-visual control of a given setting such as in-vehicle ambience. Additionally, the systems and methods are configured to dynamically adjust the ambience in a given setting in response to real-time changes in the audio signal of the musical input. Also, the systems and methods use the properties of the musical input in the form of MFCCs to create a rich, immersive, and responsive ambience. Further, the systems and methods are configured to dynamically adjust the ambience in response to any changes in the musical input in real-time. Also, a reliable and high-quality output is ensured since the overall operation of the system is designed to be efficient and robust. In addition, the systems and methods are designed to be highly flexible and customizable, thereby allowing the systems to be adapted for a wide range of applications and settings such as in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, and/or light and sound shows. Moreover, the systems and methods are configured to optimize/minimize any delays in synchronization between the time the audio of the selected music file 104 starts playing and the time of control of ambience elements by synchronizing in time, the control of the ambience elements with the playing of the music file 104.
[0124] Although specific features of embodiments of the present specification may be shown in and/or described with respect to some drawings and not in others, this is for convenience only. It is to be understood that the described features, structures, and/or characteristics may be combined and/or used interchangeably in any suitable manner in the various embodiments.
[0125] While only certain features of the present specification have been illustrated and described herein, many modifications and changes will occur to those skilled in the art. It is, therefore, to be understood that the present specification is intended to cover all such modifications and changes as fall within the true spirit of the invention.

,CLAIMS:1. A system (100) for real-time automated audio-visual control of ambience based on a musical input (104, 302), the system (100) comprising:
an ambience control system (102) comprising:
an acquisition subsystem (106) configured to receive the musical input (104, 302) ;
a processing subsystem (108) in operative association with the acquisition subsystem (106) and comprising:
an audio-visual ambience control platform (110) configured to:
process the musical input (104, 302) to generate Mel Frequency Cepstrum Coefficients (312);
identify features, patterns, timing, musical expressions, or combinations thereof in the musical input (104, 302) based on the Mel Frequency Cepstrum Coefficients (312);
assign each Mel Frequency Cepstrum Coefficient (312) to a different control channel;
generate a control signal (320) corresponding to each control channel based on the Mel Frequency Cepstrum Coefficients (312), the identified features, patterns, timing, and musical expressions in the musical input (104, 302), or combinations thereof;
create a timecode file (324) based on the Mel Frequency Cepstrum Coefficients (312) and the control signals (320); and
control a plurality of ambience elements based on the timecode file (324) to dynamically create a rich, immersive, and responsive ambience in a setting.

2. The system (100) of claim 1, wherein the ambience control system (102) further comprises an interface unit (112, 114, 116) configured to facilitate the real-time automated audio-visual control of ambience in the setting based on the musical input (104, 302).

3. The system (100) of claim 1, wherein the musical input (104, 302) is received from one or more sources comprising a Universal Serial Bus (USB) device, a compact disc (CD), a cassette, a Digital Versatile Disc (DVD), a Blu-ray disc, digital files, live performances, online streaming platforms, a data repository (118), or combinations thereof.

4. The system (100) of claim 1, wherein the plurality of ambience elements comprises lights, speakers, horn, lasers, actuators, dancing lights, laser lights, lasers, strobe lights, flashing lights, fountain jets, or combinations thereof.

5. The system (100) of claim 1, wherein the setting comprises in-vehicle ambience generation, dance floor lighting and audio control, musical fountains, theaters, music halls, sound and light shows, or combinations thereof.

6. The system (100) of claim 1, wherein to generate the control signal (320) corresponding to each control channel the audio-visual ambience control platform (110) is configured to map the identified features, patterns, timing, and musical expressions in the musical input (104, 302) obtained from the Mel Frequency Cepstrum Coefficients (312) to the plurality of ambience elements corresponding to the setting, and wherein the control signals (320), the mapping, or a combination thereof are configured to dynamically control operation of the plurality of ambience elements in real-time in response to the musical input (104, 302).

7. The system (100) of claim 1, wherein to control the plurality of ambience elements based on the timecode file (324), the audio-visual ambience control platform (110) is configured to:
play the musical input (104, 302) in response to an input from a user, automatically, or a combination thereof; and
initiate a light and sound orchestration by controlling the plurality of ambience elements based on the timecode file (324) to create the rich, immersive, and responsive ambience in the setting.

8. The system (100) of claim 1, wherein to control the plurality of ambience elements based on the timecode file (324), the audio-visual ambience control platform (110) is configured to synchronize in time the control of the ambience elements with the playing of the musical input (104, 302).

9. The system (100) of claim 1, wherein the audio-visual ambience control platform (110) is configured to be dynamically customizable and adaptable to new musical input (104, 302) in real-time.

10 The system (100) of claim 1, wherein the audio-visual ambience control platform (110) is configured to:
dynamically adapt the control signals (320) in real-time in response to changes in the musical input (104, 302) or to a new musical input (104, 302) to generate modified control signals; and
dynamically modify the operation of the plurality of ambience elements in real-time based on the modified control signals to enable the audio-visual ambience control platform (110) to be customizable and adaptable to the changes in the musical input (104, 302) or to the new musical input (104, 302) in real-time.

11. The system (100) of claim 1, wherein the audio-visual ambience control platform (110) is configured to utilize the control signals (320) to convert an audio signal of the musical input (104, 302) into a visual display that is synchronized with the plurality of ambience elements to create the rich, immersive, and responsive ambience in the setting.

12. The system (100) of claim 1, wherein the timecode file comprises a list of ambience elements, a grouping of the ambience elements, the Mel Frequency Cepstrum Coefficients (312), the control signals (320), the musical expressions, the patterns, the features, the mapping of the plurality of ambience elements to the musical expressions, a light control time stamp and status of the ambience elements during each time stamp, light control sequences, speaker control sequences, horn control sequences, fountain jet control sequences, or combinations thereof, and wherein the timecode file is utilized to create an immersive audio-visual ambience in the setting such that the ambience dynamically responds to the musical input (104, 302)

13. The system (100) of claim 1, wherein the timecode file (324), the Mel Frequency Cepstrum Coefficients (312), the control signals (320), or combinations thereof are stored in an infotainment system (112), the ambience control system (102), a data repository (118), or combinations thereof.

14. The system (100) of claim 1, wherein the audio-visual ambience control platform (110) comprises:
a pre-processing unit (502) configured to:
receive the musical input (104, 302) from one or more sources;
parse the musical input (104, 302) based on a format and extract audio channels (306) in the musical input (104, 302);
decode data in the musical input (104, 302) to extract raw audio samples;
mix the extracted audio channels (306) based on the data in the musical input (104, 302);
a Mel Frequency Cepstrum Coefficients extraction unit (504) configured to:
transform the musical input (104) from the time domain to the frequency domain to extract the Mel Frequency Cepstrum Coefficients (312);
identify features, patterns, timing, musical expressions, or combinations thereof in the musical input (104, 302) based on the Mel Frequency Cepstrum Coefficients (312);
a channel assignment unit (506) configured to assign each Mel Frequency Cepstrum Coefficient (312) to a different control channel based on the properties of the Mel Frequency Cepstrum Coefficients (312), the plurality of ambience elements to be controlled, the setting, or combinations thereof;
a control signal generation unit (508) configured to process each assigned control channel to generate a control signal (320), wherein the control signal (320) is configured to control the plurality of ambience elements to create the rich, immersive, and responsive ambience in the setting, and wherein the control signals (320) are generated based on the identified features, patterns, timing, and musical expressions in the musical input (104, 302) obtained from the Mel Frequency Cepstrum Coefficients (312), the plurality of ambience elements to be controlled, the setting, or combinations thereof;
a timecode file generation unit (510) configured to generate a timecode file (324), wherein the timecode file (324) comprises information for controlling the plurality of ambience elements in response to the musical input (104, 302); and
a device control unit (512) configured to control the operation of the plurality of ambience elements based on the control signals (320) and the timecode file (324) to dynamically create the rich, immersive, and responsive ambience in the setting.

15. The system (100) of claim 14, wherein to extract the Mel Frequency Cepstrum Coefficients (312), the Mel Frequency Cepstrum Coefficients extraction unit (504) is configured to:
apply a Fourier transform to convert the time domain signal of the musical input (104, 302) to the frequency domain;
process the frequency domain signal via a Mel filter bank to map the frequency domain signal onto the Mel scale;
process the Mel filter bank signal via a logarithm; and
process the logarithm signal via a discrete cosine transform to extract the Mel Frequency Cepstrum Coefficients (312).

16. The system (100) of claim 14, wherein the device control unit (512) is configured to:
generate a configurable delay element based on the setting, wherein the configurable delay element is configured to optimize a synchronization delay between a time that the musical input (104, 302) starts playing and a time of control of corresponding ambience elements in the light and sound orchestration; and
advance or delay one or more control commands to the corresponding ambience elements with respect to the musical input (104, 302) to optimize the synchronization delay based on the configurable delay element.

17. A method (200) for real-time automated audio-visual control of ambience based on a musical input (104, 302), the method (200) comprising:
receiving (202) the musical input (104, 302) from one or more sources;
processing (204) the musical input (104, 302) to generate Mel Frequency Cepstrum Coefficients (312);
identifying (206) features, patterns, timing, musical expressions, or combinations thereof in the musical input (104, 302) based on the Mel Frequency Cepstrum Coefficients (312);
assigning (208) each Mel Frequency Cepstrum Coefficient (312) to a different control channel;
generating (210) a control signal (320) based on the Mel Frequency Cepstrum Coefficients (312), the features, the patterns, the timing, the musical expressions in the musical input (104, 302), or combinations thereof;
creating (214) a timecode file (324) based on the Mel Frequency Cepstrum Coefficients (312) and the control signals (320); and
controlling (218) a plurality of ambience elements based on the timecode file (324) to dynamically create a rich, immersive, and responsive ambience in a setting.

18. The method (200) of claim 17, further comprising receiving (216) an input from a user, automatically, or a combination thereof to initiate controlling of the plurality of ambience elements based on the musical input (104, 302) to create the rich, immersive, and responsive ambience in the setting.

19. The method (200) of claim 17, wherein generating (210) the control signal (320) corresponding to each control channel comprises:
mapping the identified features, patterns, timing, and musical expressions in the musical input (104, 302) obtained from the Mel Frequency Cepstrum Coefficients (312) to the plurality of ambience elements corresponding to the setting; and
dynamically controlling the operation of the plurality of ambience elements in real-time in response to the musical input (104, 302) based on the control signals (320), the mapping, or a combination thereof.

20. The method (200) of claim 17, further comprising:
dynamically adapting the control signals (320) in real-time in response to changes in the musical input (104, 302) or to a new musical input (104, 302) to generate modified control signals; and
dynamically modifying operation of the plurality of ambience elements in real-time based on the modified control signals to enable a system to be customizable and adaptable to the changes in the musical input (104, 302) or to the new musical input (104, 302) in real-time.

21. The method (200) of claim 17, further comprising converting, based on the control signals (320), an audio signal of the musical input (104, 302) into a visual display that is synchronized with the plurality of ambience elements to create the rich, immersive, and responsive ambience in a setting.

22. The method (200) of claim 17, further comprising:
parsing (304) the musical input (104, 302) based on a format and extracting audio channels (306) in the musical input (104, 302);
decoding (304) data in the musical input (104, 302) to extract raw audio samples;
mixing (308) the extracted audio channels (306) based on the data in the musical input (104, 302);
transforming (310) the musical input (104, 302) from the time domain to the frequency domain to extract the Mel Frequency Cepstrum Coefficients (312);
assigning (314) each Mel Frequency Cepstrum Coefficient (312) to a different control channel based on the properties of the Mel Frequency Cepstrum Coefficients (312), the plurality of ambience elements to be controlled, the setting, or combinations thereof;
generating (318) a control signal (320) by processing each assigned control channel, wherein the control signal (320) is configured to control the plurality of ambience elements to create the rich, immersive, and responsive ambience in the setting;
creating (322) a timecode file (324), wherein the timecode file (324) comprises information for controlling the plurality of ambience elements in response to the musical input (104, 302); and
controlling the operation of the plurality of ambience elements based on the control signals (320) and the timecode file (324) to dynamically create the rich, immersive, and responsive ambience in the setting.

23. The method (200) of claim 22, wherein transforming (310) the musical input (104, 302) from the time domain to the frequency domain to extract the Mel Frequency Cepstrum Coefficients (312) comprises:
applying a Fourier transform to convert the time domain signal of the musical input (104, 302) to the frequency domain;
processing the frequency domain signal via a Mel filter bank to map the frequency domain signal onto the Mel scale;
processing the Mel filter bank signal via a logarithm; and
processing the logarithm signal via a discrete cosine transform to extract the Mel Frequency Cepstrum Coefficients (312).

24. The method (200) of claim 22, wherein controlling (218) the operation of the plurality of ambience elements based on the control signals (320) and the timecode file (324) to dynamically create the rich, immersive, and responsive ambience in the setting comprises:
playing the musical input (104, 302) in response to an input from a user, automatically, or a combination thereof; and
initiating a light and sound orchestration by controlling the plurality of ambience elements based on the timecode file (324) to create the rich, immersive, and responsive ambience in the setting; and
synchronizing in time the control of the ambience elements with the playing of the musical input (104, 302).

25. The method (200) of claim 24, wherein synchronizing in time the control of the ambience elements with the playing of the musical input (104, 302) comprises:
generating a configurable delay element based on the setting, wherein the configurable delay element is configured to optimize a synchronization delay between a time that the musical input (104, 302) starts playing and a time of control of corresponding ambience elements in the light and sound orchestration; and
advancing or delaying one or more control commands to the corresponding ambience elements with respect to the musical input (104, 302) to optimize the synchronization delay based on the configurable delay element.

26. A processing system (108) for real-time automated audio-visual control of ambience based on a musical input (104, 302), the processing system (108) comprising:
an audio-visual ambience control platform (110) comprising:
a pre-processing unit (502) configured to:
receive the musical input (104, 302) from one or more sources;
parse the musical input (104, 302) based on a format and extract audio channels (306) in the musical input (104, 302);
decode data in the musical input (104, 302) to extract raw audio samples;
mix the extracted audio channels (306) based on the data in the musical input (104, 302);
a Mel Frequency Cepstrum Coefficients extraction unit (504) configured to:
transform the musical input (104) from the time domain to the frequency domain to extract the Mel Frequency Cepstrum Coefficients (312);
identify features, patterns, timing, musical expressions, or combinations thereof in the musical input (104, 302) based on the Mel Frequency Cepstrum Coefficients (312);
a channel assignment unit (506) configured to assign each Mel Frequency Cepstrum Coefficient (312) to a different control channel based on the properties of the Mel Frequency Cepstrum Coefficients (312), the plurality of ambience elements to be controlled, the setting, or combinations thereof;
a control signal generation unit (508) configured to process each assigned control channel to generate a control signal (320), wherein the control signal (320) is configured to control the plurality of ambience elements to create a rich, immersive, and responsive ambience in a setting, and wherein the control signals (320) are generated based on the properties of the Mel Frequency Cepstrum Coefficients (312), the plurality of ambience elements to be controlled, the setting, or combinations thereof;
a timecode file generation unit (510) configured to generate a timecode file (324), wherein the timecode file (324) comprises information for controlling the plurality of ambience elements in response to the musical input (104, 302); and
a device control unit (512) configured to control the operation of a plurality of ambience elements based on the control signals (320) and the timecode file (324) to dynamically create the rich, immersive, and responsive ambience in the setting.