Generation of panoramic images

description

The present invention relates to the generation of panoramic images, in particular special to the generation of panoramic images with reduced artifacts.

A large viewing angle is covered in panoramic images, which does not necessarily include a field of view of 360. Panoramic images are usually created by combining several images that were recorded with different viewing directions.

In a variant of the generation of a panorama image, several individual images are generated one after the other. The camera pans a little further between the individual images. The individual or partial images are then combined to form a panorama image using a so-called stitching technique. Errors can occur in the stitching process. Errors caused by the fact that the entrance pupil does not coincide with the axis of rotation of the camera can be kept to a minimum in terms of design. Problems that cannot be eliminated easily are, however, those that arise due to the movement of objects in the recorded scene.

The combination of individual recordings to form an overall image can, according to a further variant, not be carried out on the basis of images recorded in time division multiplex, but rather on those recorded in channel multiplex. Here the individual images are recorded simultaneously through several channels with different viewing directions and then combined. Parallax problems occur here when stitching due to the side-by-side arrangement of the channels. In both cases there are problems with stitching due to inconsistencies between the partial recordings, on the one hand due to the different recording times and the other time due to the distance between the optical channels.

The object of the present invention is to generate panoramic images which have an improved quality, for example due to a reduced inconsistency between the individual partial recordings.

This object is achieved by the subject matter of the independent patent claims. Refinements are defined in the dependent claims.

The present application is based on the idea that two types of single images are recorded, which lead to a set of simultaneously recorded single images - a first set of single images - and a further set of temporally successively recorded single images - a second set of single images. For this purpose, simultaneous recordings - first recordings - generated by means of several optical channels of a camera system, offset in time - to the first individual images and / or to each other - at least one further individual image - called second individual image (s) - those with one or more of these channels are generated. The set of time-shifted recordings can therefore be made from the same "location" or with a small base distance, so that they have no or only a negligible parallax. The other set of simultaneously generated recordings does not suffer from inconsistencies in motion between the recordings. Due to their complementary advantages, an improved combination to form a panorama image, ie a construction of an improved panorama image, can be achieved from these two sets of individual images. In combination, a panorama image can result that is free of artifacts than with just one set of recordings, regardless of the type of overall scene, namely a scene in the overall field of view with or without movement. The additional design effort can be very low or even non-existent, depending on the existing design of the camera system. The other set of simultaneously generated recordings does not suffer from inconsistencies in motion between the recordings. Due to their complementary advantages, an improved combination to form a panorama image, ie a construction of an improved panorama image, can be achieved from these two sets of individual images. In combination, a panorama image can result that is free of artifacts than with just one set of recordings, regardless of the type of overall scene, namely a scene in the overall field of view with or without movement. The additional design effort can be very low or even non-existent, depending on the existing design of the camera system. The other set of simultaneously generated recordings does not suffer from inconsistencies in motion between the recordings. Due to their complementary advantages, an improved combination to form a panorama image, ie a construction of an improved panorama image, can be achieved from these two sets of individual images. In combination, a panorama image can result that is free of artifacts than with just one set of recordings, regardless of the type of overall scene, namely a scene in the overall field of view with or without movement. The additional design effort can be very low or even nonexistent, depending on the existing design of the camera system. Due to their complementary advantages, an improved combination to form a panorama image, ie a construction of an improved panorama image, can be achieved from these two sets of individual images. In combination, a panorama image can result that is free of artifacts than with just one set of recordings, regardless of the type of overall scene, namely a scene in the overall field of view with or without movement. The additional design effort can be very low or even non-existent, depending on the existing design of the camera system. From these two sets of individual images, an improved combination to form a panorama image, ie a construction of an improved panorama image, can be achieved due to their complementary advantages. In combination, a panorama image can result that is free of artifacts than with just one set of recordings, regardless of the type of overall scene, namely a scene in the overall field of view with or without movement. The additional design effort can be very low or even non-existent, depending on the existing design of the camera system. and regardless of the type of overall scene, namely a scene in the overall field of vision with or without movement. The additional design effort can be very low or even nonexistent, depending on the existing design of the camera system. and regardless of the type of overall scene, namely a scene in the overall field of vision with or without movement. The additional design effort can be very low or even non-existent, depending on the existing design of the camera system.

when the first difference is greater than the second difference. The first and second overlap areas can coincide.

In a further embodiment, the device is set up to determine the first and second differences and to select the first or second pair based thereon along a second direction which is perpendicular to a first direction along which the fields of view covered by the pair of individual images lie next to one another to be carried out in such a way that the determination is subdivided into cross-sections / strips, so that the selection for different cross-sections that run transverse to the second direction can be different.

In one embodiment, the recording of the at least one second individual image is carried out in such a way that the individual images of the second set of individual images are recorded with different viewing directions, and a viewing direction variation is carried out between the individual images of the second set of individual images, which is achieved by rotating an optical channel of or by rotating the entire camera system and / or switching a switchable optical element that causes a deflection of a beam path of one optical channel or a beam path of several channels of the camera system, and / or rotating a mirror or prism for deflecting a beam path of the one of the multiple optical channels or a beam path of the multiple channels of the camera system is caused.

In a further embodiment, the system comprises a second camera system and is designed for the simultaneous recording of a third set of individual images, which together cover the panoramic field of view, with several optical channels of the second camera system, the second camera system being arranged offset relative to the first camera system, for recording at least one fourth single image, with one of the multiple optical channels of the second camera system, so that the at least one fourth single image together (ie the at least one fourth single image taken by itself) or together with one through the one of the multiple optical channels The recorded third single image gives a fourth set of single images that are recorded with a time delay and also cover the panoramic field of view,and for constructing a further panorama image or a depth map for the panorama image on the basis of the third set of third individual images and the at least one fourth individual image.

Embodiments of the present invention are explained in more detail below with reference to the figures, among which:

1 shows a schematic representation of a device for generating panorama images according to an embodiment;

2 shows a schematic representation of a camera system according to an exemplary embodiment;

3 shows a schematic representation of a camera system according to an exemplary embodiment, in which the object to be photographed is moving;

Fig. 4 is a schematic illustration of overlapping pairs of individual images relating to a common overlap area for analysis of the same with respect to the pair to be preferred for generating panoramic images; and

Fig. 5 is a schematic representation of overlapping pairs of individual images relating to a common overlap area for the analysis of the same with respect to the pair to be preferred for generating panoramic images, the analysis being refined in sections.

Before exemplary embodiments are described with reference to the figures, it is once again briefly illustrated and motivated, which results in advantages from the exemplary embodiments described below.

As mentioned in the introduction to the description, it can be useful for various reasons to combine several recordings with a comparatively small field of view into one recording with a larger field of view. For example, the overall resolution of the image can be higher, or the dimensions of the camera system are smaller, or the form factor of the system can be cheaper, or it can be built particularly flat.

There are essentially two methods available for recording the individual images;

1. Time-shifted recording of the individual images with a single optical channel. To do this, the direction of view is changed between the recordings, for example by rotating the entire camera or by deflecting the beam path with a movable mirror. All recordings are made from a common point of view. 2. Simultaneous recording of the individual images with several optical channels that have different fixed viewing directions. In this case, the optical channels are inevitably not in the same place, but are laterally offset from one another. A beam splitter would enable several optical channels with a common nodal point, but can lead to an undesirable loss of light.

In the first method, the individual images differ in their recording time, in the second method in the camera location. This can lead to complications when joining the individual images to form the panorama image. Artifact-free joining is only possible for scenes with the following properties:

1. Static scenes, ie there is no movement of the objects in the scene between the recording times.

2. All objects in the scene are at the same distance from the camera system or at a distance that is large compared to the offset of the optical channels, depending on the angular resolution of the camera system by a factor of 100 or more.

The reconstruction of other scenes can also be attempted, but requires an increased algorithmic effort. If the reconstruction fails, this manifests itself in ghost images or in mismatched edges and image details.

Artifacts can be reduced, for example, by quickly switching the viewing direction or by keeping the optical channels close together.

The exemplary embodiments described below make use of the individual advantages or scene-specific suitability of the two variants by recording two sets of individual images, one recorded at the same time and one recorded with a delay, so that the construction of the panorama image is based on the both sets can be improved overall, regardless of the type of scene, ie with movement and / or with strong deep variations in the individual image overlap areas.

1 shows a schematic representation of an exemplary embodiment of a device 100 for generating a panoramic image. The device 100 comprises an arrangement of several optical channels 110a-110d of a camera system 110. In the present example, the camera system comprises four optical channels, but the present invention is not limited to four optical channels, but can have two or more. The camera system 110 can comprise one or more lenses or one or more other optical elements 115 per channel 110a-110c, which are used to image a respective sub-area or sub-field of view 150a-150d belonging to the respective channel from an entire panoramic field of view 150 onto a respective image sensor or respective image sensor section of an image sensor 11 1. The channels can we sentlichen parallel to each other from the image sensor 11 to a deflecting device, such as a mirror, which deflects the beam paths of the channels or their fields of view. It is pivotable between a first position, shown in FIG. 1, in which the total field of view 150 is, and a second position in which the total field of view 150 'is, which is displaced in relation to the 150 in the first position so that the Partial fields of view 150d, 150b of channels 1, 10d, b in the second position 150 'essentially overlap with partial fields of view 150c, a of channels 1, 10c, a in the first position. For this purpose, the partial fields of view 150a-d are arranged here in each position, by way of example, in a 2x2 array, overlapping one another in order to cover the entire field of view, so that the total fields of view 150, 150 ' in the two positions are offset from one another by a partial field of view in a direction parallel to the beam paths in front of the deflection device 140 and perpendicular to the pivot axis of the deflection device 140. The channels 110d, 110b are thus able to cover the entire field of view from the first position 150 if the sequence of positions 150 and 150 'is considered. The division into a 2x2 cover and the number of partial fields of view is not critical and the number of positions could also be increased. The positions still play a role when taking a panorama picture, as is described below.

The camera system 1 10 or its image sensor (s) 1 1 1 is / are connected to a decision device 120 which selects a set of individual images from a set of individual images that are recorded according to a specific procedure described in more detail below, in order to then be put together to form a panorama image, the selection of the set of images also being able to vary, for example in a direction transverse to the mutual partial field of view offset, as will be described below. The composition / construction of the panorama image on the basis of the record selection is performed by the panorama image generator 130. Furthermore, in some exemplary embodiments, the device can comprise a device 140 which changes the “viewing direction” of the camera system. As described, the entire field of view of the panorama camera can be swiveled to produce a panorama image. This pivoting, which could be carried out manually by the user, can of course also be carried out in exemplary embodiments by a device which changes the alignment of the camera system or its overall field of view alignment in the course of a panorama recording operation. As an alternative to Fig. 1, this could be done, for example, by rotating the entire camera system 1 10, but here is accomplished by rotating the deflecting direction, such as a mirror or prism, so that the beam path of the optical channels of the camera system is deflected, like this is indicated by the double arrow on the deflection device 140 in Fig. 1. Alternatively, it would be possible to change the direction of view or change of position by switching a switchable optical element, the optical element causing a deflection of the beam path. As said, the variation in the viewing direction of the camera system could also be carried out manually by the user of the camera.

It should be emphasized that the presence of a deflection device 140 can have a purpose that already results from a different requirement than that of the change of position, as used here below for the purpose of creating panoramic images, such as image stabilization, so that in one such a case, an already existing camera system can be expanded without major structural measures to make a Pano ramabilderbildung according to the present application. In the case of the double use of the deflecting device 140 for image stabilization in the direction transverse to the pivot axis, there is also an advantage that in this case the change of the viewing direction can be made in a very short time, so that the deflecting device can execute the position changes quickly, so that between the Recordings,

FIG. 2 shows a schematic illustration which is intended to illustrate how different sets of individual images could be recorded in an exemplary embodiment. Fig. 2 increases for illustrative purposes, the number of positions and also the number of channels in the direction of the relative offset of the Gesamtge field of view between the positions to three compared to 2, as shown in Fig. 1, although the division into two adjacent partial fields of view which are transverse to the direction of the total field of view offset between the positions in FIG

runs, doesn't matter. The sequence of recordings, as it will now be described, could be run through automatically by the camera system 110, such as controlled by a controller of the system 110, such as upon actuation of a recording button by the user, possibly in a certain panoramic recording mode.

The shown camera system 110 thus has in this example of FIG. 2 three optical cal channels A, B and C, but a different number of optical channels is also conceivable. The optical channels can belong to a single camera, ie the channels can be permanently installed in a housing, as illustrated in FIG. 1, or there can also be three individual cameras arranged next to one another, of which at least one, namely here at least the Channel A forming, is pivotable to look into the partial face fields of the other two channels A and C. As can be seen from the figure, the three channels A, B and C are arranged next to one another so that they can simultaneously record individual images in the direction of a panoramic field of view, the respective viewing directions of the individual optical channels differing slightly from one another, so that the individual images recorded have overlap areas 220, 230. This happens, for example, when taking a picture in a position 1, which leads to pictures 1A, 1 B and 1 C, which cover the entire field of view with mutual overlap between 1 A and 1 B and between 1 B and 1 C.

Channel A can be aligned in position 2 so that it looks in the direction of view of channel B in position 1, for example by pivoting to the left. Ideally, it is assumed in FIG. 2 that the pivoting of the channel A between the positions takes place in such a way that its nodal point remains constant, but this is only one example, and pivoting according to FIG was described there, would also be possible and change little in the concept and nature of the following description of a panorama image generation. In position 2, a receptacle 2A can be generated through channel A, which essentially covers the same partial field of vision as receptacle 1 B through channel B in position 1. In addition, a position 3 is also illustrated,

To explain the advantages and functioning of the subsequent processing of the recordings 1 A-1 C, 2A and 3B to form an overall field of view, letters A and B are illustrated in FIG. 2 as examples of objects in the scene. They lie at different distances in the scene, which is illustrated by their different sizes. Based on the objects, the influence of different recording times between positions 1, 2, and 3 and the influence of the scene depth, ie the different depth or object distance from objects in the scene, and the way in which they are prevented or reduced are explained shows how this influence can have a negative effect on a panorama image generated from the images 1A-1 C, 2A and 3A. The letter B is in another,

Furthermore, FIG. 2 shows circled areas 220 and 230 in which the recording areas of cameras A and B or A and C overlap in position 1. This means, for example in the case of the overlap 220, that the individual image 1A has an overlap with the individual image 1B. More precisely, single image 1 A has an overlap with the right area of ​​single image 1 B in its left area. This means that the same image content is included in this area. However, these image contents can differ due to parallax effects, so that the image contents may not be identical. Similarly, there is the overlap area 230 between the individual images 1A and 1C,

In the example of FIG. 2, channel A of the camera system 1 10 thus records the image 1 A, while the image 1 B from channel B and the image 1 C from camera C is recorded simultaneously. This happens at position 1. Since the cameras or the optical channels A, B and C are not in the same place, but are arranged somewhat offset relative to one another, the recorded images can have a parallax. This is indicated by the different positions of the aforementioned letters A and B, which are located in the panoramic field of view.

In single image 1 A, the letter B appears to be in the middle behind the letter A. The optical channel B, which records the partial field of view to the left of channel A in the viewing direction and which generates the single image 1 B, sees the letter B laterally offset to the left relative to the letter A, since its nodal point or its entrance pupil is laterally to the left of that of the channel A lies. He sees the scene a little from the left. Analogously, the optical channel C records the single image 1 C in which the letter B appears to be offset somewhat to the right relative to the letter A. This effect, the parallax effect, occurs when the optical channels are laterally offset from one another and the scene has a depth, ie the objects are arranged in it at different distances.

It should be emphasized that the order in which the positions 1, 2 and 3 are run through is not relevant and could also be changed, but now the second position is 2, in which the optical channel A is rotated to the left the single image 2A, and the position 3, in which the optical channel A is rotated to the right in order to record the single image 3A. With regard to the partial field of view of the recording, they correspond to the individual images 1 B and 1 C, but unlike the individual images 1 B and 1 C, the recordings 2A and 3A are recorded from the same perspective as is available for the individual recording 1B. It can accordingly be seen in FIG. 2 that since the location of the optical channel A does not change, no parallax effect occurs here, ie

In general, it would therefore be advantageous to generate a panoramic image based on the set of recordings 1A, 2A and 3A, since there the arrangement of the letters A and B does not differ relative to one another and thus no artifacts occur.

However, it is also clear that the set of recordings 1A, 1B and 1C are recorded simultaneously, whereas recordings 1A, 2A and 3A form a set of recordings that have to be generated with a time offset from one another, since between The camera or the optical channel A must be swiveled to the left or right for the individual recordings or another manipulation / switching of the viewing direction must take place.

If in the example of FIG. 2 the arrangement of the letters A and B in the panoramic view field is static, it would not be necessary, for example, to take the simultaneously recorded or individual images 2A and 3A, but it would be more advantageous to create an artifact-free panoramic image to generate, to use the set of single images 1A, 1B and 1C recorded one after the other. However, if the objects in the panoramic field of view are not static, but rather have a movement, or if the scene is dynamic, the situation can look different, as shown in FIG. 3.

Fig. 3 shows schematically a similar arrangement as that shown in Fig. 2, only in this case, the letter A is subject to movement in the specific example, this book rod rotates clockwise.

The three individual images 1A, 1 B and 1 C that were recorded at the same time each show the letter A with the same orientation, since the images were recorded at the same time. The individual images 2A and 3A subsequently recorded with a time delay, which are recorded through the optical channel A after the optical channel A has been pivoted to the left and right, show the letter A slightly rotated. The single image 2A shows that the letter A is rotated slightly to the right, relative to the orientation of the letter A in the recordings 1A, 1 B and 1 C, since the single image 2A is temporally after the single images 1A, 1 B and 1 C was recorded. Since the single image 3A took place even after the recording of the single image 2A, the letter A in the single image 3A is shown rotated a little further in the clockwise direction. Fig.

If the panorama image were now to be generated from the set of individual images 1A, 2A and 3A, there would be a clear optical artifact due to the successive rotation of the letter A. In this case, it would be more advantageous to base the panorama image on the set of individual images 1A, 1 B or 1 C.

Depending on the scene, it can happen that different scene situations occur at the overlapping areas 220 and 230, i.e. movement and / or scene depth, so that another set of individual images could also be the most favorable for generating the panorama image, such as a generation based on the Set of frames 1A, 2A or 3A, or the set of frames 1A, 2A and 1 C.

In order to decide which combination of individual images is more advantageous for generating the panorama image, it is possible in exemplary embodiments to take a closer look at the aforementioned overlap areas 220, 230 of the individual images or to examine pairs of individual images that are located in these overlap areas over lap.

As shown in FIGS. 2 and 3, the individual image 1A has an overlap with the individual image 1B on the left-hand side shown here. The same applies to the pairing between image 1 A and single image 2A. Analogously, the individual image 1 A has an area of ​​overlap with the individual images 1 C and 3A in the right-hand area of ​​the image 1 A.

4 shows a schematic illustration which is intended to explain in more detail the concept of the decision about the compilation of the set of recordings as the basis for the generation of panoramic images on the basis of an analysis of the overlapping areas, as carried out by the decision maker 120. Individual images 1A, 1B and 2A from FIG. 2 are shown, with individual images 1A and 1B being shown next to one another in the left-hand area of ​​FIG. 4, and individual images 1A and 2A being shown in the right-hand area.

In Fig. 4 sind zu den Einzelbildern die Überlappungsbereiche Ü1 und Ü2 markiert. Dies sind die Bereiche, die sich in den Einzelbildern 1A und 2A auf den gleichen Teil des Pano ramablickfeldes beziehen. Es ist augenfällig, dass der Überlappungsbereich Ü1 die glei che Bildinformation trägt wie der Überlappungsbereich Ü2. Dies ist nicht verwunderlich, da ja der Panoramabildbereich der Fig. 2 keine Bewegung aufweist und somit die zeitver setzt aufgenommenen Einzelbilder 1 A und 2A sich im Überlappungsbereich stark ähneln, da ja in diesem Fall der Parallaxeneffekt entfällt, weil diese aus den gleichen Perspektive aufgenommen wurden, d. h. der Standort des Kanals war jeweils der gleiche. Die Einzel bilder 1 A und 1 B sind in Fig. 4 auch einander gegenübergestellt. Unter diesen Einzelbil dern sind die Überlappungsbereiche Ü1 und Ü3 markiert. Hier ist die Ähnlichkeit im Über lappungsbereich nicht so ausgeprägt verglichen zu der Situation in dem Bilderpaar 1 A und 2A. Die Bildinformation über die linke Hälfte des Buchstabens A ist zwar in beiden Überlappungsbereichen sehr ähnlich, allerdings ist aufgrund des Parallaxeneffektes im Bild 1 A mehr die linke Hälfte des Buchstabens B zu sehen, wohingegen in dem Überlap pungsbereich Ü3 der Buchstabe B fast gar nicht zu sehen ist.

When comparing the similarity of the overlapping areas U1 and U2 in comparison to the areas U1 and U3, the single image pair 1A and 2A have the greater similarity. To create a panorama image, the pair of single images 1A and 2A would be the preferred combination.

In general, in one example, in order to decide whether the combination of the individual image 1 A with 1 B or the combination of the individual image 1 A with individual image 2B is more advantageous for the overall panoramic image, a difference in the overlap area between

Image 1A and Image 1 B can be determined, and a difference between single image 1 A and single image 2A in the overlap area can also be determined. Depending on which of the two differences is smaller, a combination of two individual images is selected to generate the panorama image.

To form a difference or a difference value in the overlap area of ​​two individual images, it is conceivable, for example, to compare the individual images in the overlap area pixel by pixel and to add up the difference values ​​in terms of amount are determined, or the evaluation of a correlation maximum in the event of mutual displacement of the individual images pairs along the direction along which the fields of view lie next to one another.

Fig. 5 shows schematically a further embodiment in which the individual images are divided into segments or cross-sections, for which a separate decision is made as to whether the corresponding simultaneously recorded sections / cross-sections are used to generate the panorama image or the cross-sections recorded consecutively in time, but which then have no parallax.

The panoramic field of view of this example is characterized in that the letter A is subject to movement, ie it shows a change over time due to clockwise rotation, whereas the letter B is static. In this case, the letter B is arranged above the letter A, as a result of which the panoramic image area can be divided into an upper area without movement and a lower area with movement.

In Fig. 5, therefore, each frame is divided into an upper and lower area, e.g. B. the single image 1 A is divided into the lower area 1 Au and the upper area 1Ao, for the other single images 1 B, 1 C, 2A and 3A the same applies accordingly.

In choosing the top or bottom fields to be used in constructing the panoramic image, the top area will now be discussed first. In the upper area, due to the lack of movement, an analysis that essentially corresponds to that of FIG. 2 applies. Starting from the upper image area 1 Ao, the individual image cross-sections 2Ao and 3Ao are preferable to sections 1 Bo and 1 Co, since the individual image cross-sections

Sections 2Ao and 3Ao do not show any parallax and it is therefore to be expected that these result in the more consistent panorama image.

In the lower area of ​​the individual images, it is more advantageous in the present case to generate the panorama image based on the individual image cross-sections 1Au, 1 Bu and 1 Cu, since the individual image cross-sections 2Au and 3Au have a rotated letter A. Since a rotation can often be perceived as more disturbing than a parallax effect in human perception and the similarities in the overlapping areas are also less due to the rotation, in the present case the parallax-afflicted individual image cross-sections are preferably used for the construction of the panorama image.

In a case not shown, in which the rotation of the letter A takes place much more slowly, ie that the individual image 2Au would show an only slightly rotated letter A, it would be conceivable that in this case the individual image cross-section 2Au would be more similar to the corresponding one Overlap area with the single image cross-section 1 Au, and that in this case the combination of 1Au and 2Au would be more advantageous compared to the parallax-afflicted combination 1Au and 1 Bu, however, it would then also be possible that in the single image 3A the twisting of the letter A can be perceived more clearly and in this case the combination of 1 Au with 1 Cu is preferable to the combination of 1 Au and 3Au.

In general, it can be stated that the assessment of static scenes, of scenes with objects at a relatively great distance and scenes with objects in a restricted distance range is relatively unproblematic.

It is also conceivable, in continuation of the subdivision of the individual images into individual image cross-sections, as shown in FIG. 5, that the individual images themselves can be broken down into further cross-sections, for example into three, four or even more cross-sectional areas. This can be useful in cases in which a single image is divided into areas in a direction perpendicular to the pan direction of the panoramic image, which are either static or have movement.

In order to assess whether a further subdivision into refined cross-sections makes sense, such a subdivision can be carried out on a trial basis and the similarity of the different cross-sections can be calculated. In the event that the respective differences in the cross-sections turn out to be insignificant, a further refined subdivision into cross-sections can be omitted, in the event that clear differences become apparent, it can be concluded that a finer subdivision into cross-sections makes sense and this can be done accordingly. Such a consideration works for a larger number of scenes and is only problematic for image details that contain both movement and a gradation of depth. The staggered depth leads to the previously discussed parallax effects.

In the literature, arrangements are described which are intended to measure and compensate for parallax effects. This involves the simultaneous recording of individual images with several optical channels that have different fixed viewing directions. However, each of these channels is available twice, the base width being deliberately chosen as large as possible. In this way, each partial field of view is recorded from two locations. This double recording of the scene with two cameras with a certain base width / base distance makes it possible to create a disparity between these two views, and this disparity can be used to compensate for the parallax for the entire field of view in order to ultimately obtain an artifact-free overall image.

However, there are scenes in which gaps arise due to masking effects either in the depth estimate or in compensation. There are then parts of the image for which either no depth estimation is possible or for which the parallax compensation exposes scene areas that were not recorded by any of the optical channels. In this case, parallax compensation is not possible without errors. Even if the scene is static, no artifact-free or error-free panorama can be created. If, however, at least one channel is additionally rotated or deflected in such a way that this channel covers all areas of the field of view, a panorama image can be generated in static scenes without parallax compensation. This is useful in the case of concealment described above, but can also be done without concealment, to avoid the complex parallax compensation. A device for recording the individual images can contain a mirror or some other device for deflecting the viewing direction. In this case, the use of the method described above is particularly advantageous since the mechanical requirements are already in place. Changes are only necessary when controlling the mirror or when evaluating the image.

Furthermore, it can be provided in exemplary embodiments not only to record all image areas with one channel, as in the aforementioned examples with channel A, but also with the other optical channels B and C. The viewing directions

of the other optical channels changes at the same time as the first channel, this does not increase the recording time.

A quick switchover of the viewing direction, for example to record the individual images 2A and 3A, can be achieved by rotating the entire system or at least the camera system, by a switchable optical element, for example by means of a tunable lens or a liquid crystal lens or a liquid lens, or by Rotating or tilting a mirror or prism in the beam path. Such a mirror or prism could be driven by means of a piezo control or an electric motor.

In particular, by using switchable optical elements or by rotating or tilting a mirror or prism, switching times in the range of a few tens of milliseconds can be achieved, which is comparable to the usual exposure times with weak lighting.

In all cases it must be ensured that no parallax is introduced by the rotation or tilting or that this is at least minimized. It can be sufficient that it is achieved that the shift due to parallax is less than the depth of field.

If the camera system used is equipped with an autofocus, all recordings can and should be made with a setting determined during the first recording. This reduces the time between shots to the time it takes to switch the viewing direction. An identical focus setting for all partial images is also beneficial for composing the panorama image.

Such a camera system according to the invention can be used in mobile telephones and vehicle cameras, and can also be used in robotics, video conference systems, remote sensing, medical applications and machine vision. The method according to the invention is also suitable for multi-channel camera systems in which a mirror or some other mechanism for controlling the viewing direction is already present. This is the case, for example, in systems with a folded beam path.

Although some aspects have been described in connection with a device, it is understood that these aspects also include a description of the corresponding

driving, so that an element of a device is also to be understood as a corresponding method step or as a feature of a method step. Analogously to this, aspects that have been described in connection with or as a method step also represent a description of a corresponding block or details or features of a corresponding device. Some or all of the method steps can be carried out by a hardware apparatus (or using a hardware Device), such as a microprocessor, a programmable computer or an electronic circuit. In some exemplary embodiments, some or more of the most important method steps can be carried out by such an apparatus.

Depending on the specific implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, a hard disk or other magneti electronic or optical memory, on which electronically readable control signals are stored, which can interact or cooperate with a programmable computer system in such a way that the respective method is carried out. Therefore, the digital storage medium can be computer readable.

Some exemplary embodiments according to the invention thus comprise a data carrier which has electronically readable control signals which are able to interact with a programmable computer system in such a way that one of the methods described herein is carried out.

In general, exemplary embodiments of the present invention can be implemented as a computer program product with a program code, the program code being effective to carry out one of the methods when the computer program product runs on a computer.

The program code can, for example, also be stored on a machine-readable carrier. Other exemplary embodiments include the computer program for performing one of the methods described herein, the computer program being stored on a machine-readable carrier. In other words, an exemplary embodiment of the method according to the invention is thus a computer program that

a program code for performing one of the methods described herein when the computer program runs on a computer.

A further exemplary embodiment of the method according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for performing one of the methods described herein is recorded. The data carrier, the digital storage medium or the computer-readable medium are typically tangible and / or non-perishable or not temporary.

Another exemplary embodiment includes a processing device, for example a computer or a programmable logic component, which is configured or adapted to carry out one of the methods described herein.

Another exemplary embodiment comprises a computer on which the computer program for performing one of the methods described herein is installed.

In some exemplary embodiments, a programmable logic component (for example a field-programmable gate array, an FPGA) can be used to carry out some or all of the functionalities of the methods described herein. In some exemplary embodiments, a field-programmable gate array can interact with a microprocessor in order to carry out one of the methods described herein. In general, in some exemplary embodiments, the methods are performed by any hardware device. This can be universally applicable hardware such as a computer processor (CPU) or hardware specific to the method, such as an ASIC.

The devices described herein can be implemented, for example, using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

The devices described herein, or any components of the devices described herein, can be implemented at least partially in hardware and / or in software (computer program).

The methods described herein can be implemented, for example, using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

The methods described herein, or any components of the methods described herein, can be carried out at least partially by hardware and / or by software.

The above-described embodiments are merely illustrative of the principles of the present invention. It is to be understood that modifications and variations of the arrangements and details described herein will be apparent to other skilled persons. It is therefore intended that the invention be limited only by the scope of protection of the following patent claims and not by the specific details presented herein with reference to the description and the explanation of the exemplary embodiments.
Claims

1. Device for generating a panoramic image of a panoramic field of view, which is designed for

simultaneous recording of a first set of first single images (1A, 1 B, 1 C), which together cover the panoramic field of view, with several optical channels of a first camera system;

Recording of at least one second single image (2A, 3A) with one of the multiple optical channels of the first camera system, so that the at least one second single image (2A, 3A) together or together with a first recorded through the one of the multiple optical channels Single image results in a second set of single images that are recorded with a time offset from one another and cover the panoramic field of view;

Selecting, for at least one image area of ​​the panorama image, one or more images from a subset of the first individual images (1 A, 1 B, 1 C) and the at least one second individual image (2A, 3A) which overlaps with the at least one image area; and

Constructing the panoramic image using the one or more images in the at least one image area.

2. Device according to claim 1, wherein the subset comprises at least one individual image from the first set and at least the at least one second individual image.

3. Apparatus according to claim 1 or claim 2, further configured for a

Determining a first difference between a first pair of first individual images which relate to at least a section of the panoramic field of view and overlap one another in the panoramic field of view in a first overlap area, in the first overlap area,

Determining a second difference between a second pair of individual images from the second set of individual images which meet the section of the panoramic field of view and overlap in the panoramic field of view in a second overlap area, in the second overlap area,

Selecting the first pair for constructing the panoramic image in the section when the first difference is less than the second difference, and

Selecting the second pair for constructing the panoramic image in the section when the first difference is greater than the second difference.

4. The device according to claim 3, wherein the first individual images lie next to one another along a first direction in the panoramic field of view, the device being designed to determine the first and second differences and to select the first or second pair based thereon the first direction perpendicular to the second direction divided into cross-sections, so that the selection for different cross-sections can be different.

5. Device according to one of claims 1 to 4, wherein the first individual images lie next to one another along a first direction in the panoramic field of view, the device being designed to select a first set of one or more predetermined images for a first partial image area and for a second partial image area which overlaps with the first partial image area along the first direction, but is offset from the first partial image area in a second direction perpendicular to the first direction, to select a second set of one or more predetermined images that is different from the first subset.

6. Device according to one of claims 1 to 5, wherein the first individual images lie next to one another along a first direction in the panoramic field of view, the device being designed to carry out the selection divided into cross-sections along a second direction perpendicular to the first direction, so that the selection for different cross-sections can be different.

7. The device according to claim 4 or 6, which is designed to adaptively refine the division into cross sections.

8. Device according to one of the preceding claims, wherein the recording of the at least one second single image (2A, 3A) is carried out such that the second set of single images is recorded with a smaller parallax compared to the first set of first single images.

9. The device according to one of claims 1 to 8, wherein the recording of the at least one second single image (2A, 3A) is carried out so that the single images of the second set of single images are recorded with different viewing directions, and the device is designed to to perform a line of sight variation between the individual images of the second set of individual images by:

Rotating one optical channel or the camera system, and / or

Switching a switchable optical element which causes a deflection of a beam path of the one optical channel or a beam path of the multiple channels of the camera system; and or

Rotating a mirror or prism to deflect a beam path of one of the multiple optical channels or a beam path of the multiple channels of the camera system.

10. Device according to one of the preceding claims, comprising a second camera system and also designed for a

simultaneous recording of a third set of third individual images (1 A, 1 B, 1 C), which together cover the panoramic field of view, with several optical channels of the two th camera system, the second camera system being arranged offset relative to the first camera system;

Recording of at least one fourth single image (2A, 3A) with one of the multiple optical channels of the second camera system, so that the at least one fourth single image (2A, 3A) together or together with a third single image recorded through the one of the multiple optical channels results in fourth set of individual images that are recorded with a time lag and cover the panoramic field of view; and

Constructing a further panorama image or a depth map for the panorama image on the basis of the third set of third individual images and the at least one fourth individual image.

1 1. The device according to claim 10, further designed for a

Estimating a parallax based on the depth map; and

Compensate for parallax based on the estimated parallax.

12. The apparatus of claim 10, further configured for a

Select, for the at least one image area of ​​the further panorama image, one or more images from a subset of the third individual images (1 A, 1 B, 1 C) and the at least one fourth individual image (2A, 3A), the selection for the image area of the further panorama image corresponds to the selection for the image area of ​​the panorama image from the subset of the first individual images (1 A, 1 B, 1 C) and the at least one second individual image.

13. Device according to one of claims 1 to 10, comprising a second camera system and also designed for a

simultaneous recording of a third set of third individual images (1 A, 1 B, 1 C), which together cover the panoramic field of view, with several optical channels of the two th camera system, the second camera system being arranged offset relative to the first camera system; and

Compensating for parallax effects in the panoramic image using the third set of third frames.

14. Device according to one of claims 1 to 13, further designed for a

Recording of at least one further single image with another of the plurality of optical channels of the first camera system.

15. Method for generating a panoramic image of a panoramic field of view, including:

simultaneous recording of a first set of first individual images (1 A, 1 B, 1 C), which together cover the panoramic field of view, with several optical channels of a first camera system;

Recording of at least one second single image (2A, 3A) with one of the several optical channels of the first camera system, so that the at least one second single image (2A, 3A) together or together with a first single image recorded through the one of the several optical channels results in a second set of individual images which are recorded with a time delay and which cover the panoramic field of view;

Selecting, for at least one image area of ​​the panorama image, one or more images from a subset of the first individual images (1A, 1 B, 1 C) and the at least one second individual image (2A, 3A) which overlaps with the at least one image area; and

Constructing the panoramic image using the one or more images in the at least one image area.