Publikationen

Browse

Now showing 1 - 20 of 48

A random finite set approach for dynamic occupancy grid maps
(Universität Ulm, 2017-06-01) Nuss, Dominik; Dietmayer, Klaus; Koch, Wolfgang
Reliable vehicle environment perception is a basic prerequisite for advanced driver assistant systems and autonomously driving cars. A common environment representation form is an occupancy grid map. It divides the environment into single grid cells and estimates for each cell whether the space it represents is occupied or free, assuming grid cells are independent of each other. The mathematical framework is based on the binary Bayes filter (BBF), which combines sensor measurements from different sensors and potentially from different points in time. Since an occupancy grid map does not employ a concept of individual objects, it is able to represent arbitrarily shaped obstacles. A classical occupancy grid map is not eligible for estimating dynamic environments, because it does not apply a process model. A much-noticed extension to a static occupancy grid map is the Bayesian occupancy filter (BOF). In contrast to a classical occupancy grid, the BOF estimates a velocity distribution for the occupancy of each grid cell based on a histogram filter. Since the BOF is computationally extremely demanding, recent publications suggest to represent the dynamic state of grid cells with particles. This allows to calculate dynamic grid maps in real-time applications with increased grid cell size and resolution. Today, dynamic occupancy grid maps are still a younger research area and not as well-studied as object-tracking approaches are. Up to now, the BOF has been addressed as a research field with little connection to other tracking methods. This work presents a new concept of dynamic grid mapping as an approximation of a random finite set (RFS) filter. A random finite set is a general, probabilistic representation of a random but limited number of objects and their states. The finite set statistics (FISST) describe Bayesian filtering of random finite sets and are basis for a number of multi-object tracking approaches like the probability hypothesis density (PHD) filter. Describing the grid as a random finite set allows transferring advanced concepts from the well-established field of random finite set filtering to the field of dynamic grid mapping. The thesis derives a filter called probability hypothesis density / multi-instance Bernoulli (PHD/MIB) filter, which represents and propagates the dynamic grid map in alternating forms as a PHD and as multiple instances of Bernoulli filters. Additionally, the thesis presents a sequential Monte Carlo (SMC) realization of the PHD/MIB filter and an approximation in the Dempster-Shafer domain called Dempster-Shafer PHD/MIB (DS-PHD/MIB) filter, which requires a smaller number of particles than the original PHD/MIB filter. The thesis describes in detail an efficient, massively parallel implementation of the DS-PHD/MIB filter and outlines the algorithm in pseudo code. Finally, the thesis describes characteristics of the DS-PHD/MIB filter and discusses its advantages and disadvantages compared to object-based tracking approaches using practical application examples. A quantitative evaluation with real-world data shows that the DS-PHD/MIB filter provides consistent state estimation results and that it appropriately models the stochastic multi-object transition process and the stochastic multi-object observation process. Furthermore, the evaluation confirms the real-time capability of the parallelized implementation of the DS-PHD/MIB filter and its usefulness for state estimation of a dynamic vehicle environment.
Autonomous measurement robot
(Universität Ulm, 2024-11-15) Forstenhäusler, Marc; Dietmayer, Klaus; Hanss, Michael
In the era of Industry 4.0, which emphasizes customized production processes, there is a growing demand for flexible and autonomous measurement technology. However, current stationary and centralized metrology systems fall short of meeting these requirements. This work presents the development and implementation of the Autonomous Measuring Robot (AuMeRo) system, designed specifically for industrial metrology and inspection, as a solution to address these limitations. The primary objective of the AuMeRo system is to establish robot-based metrology as a pivotal technology in future production environments. It offers precision, autonomy, and flexibility while considering safety, mobility, and integrability into existing production systems. The AuMeRo system encompasses a mobile platform equipped with a robotic arm, cameras, and a measuring sensor. It is capable of independent navigation within the production facility, identification of measurement objects, precise visual localization, placement of the measurement sensor using the robotic arm, and ondemand measurement data acquisition. The system’s autonomy and real-time data collection are facilitated by advanced perception algorithms. One critical aspect of system autonomy is accurate object recognition and state estimation during interactions. To achieve this, a pose graph-based fusion approach is introduced, aiming to enhance localization precision by combining individual measurements through a smoothing technique. The algorithm leverages pre-processed pose estimates and odometry information for state estimation. By constructing a graph representation with nodes representing various pose variables (such as robot and object poses) and edges representing their relationships, the fusion algorithm optimizes the graph topology using nonlinear least squares optimization. To account for the uncertainty in the robot arm’s kinematic chain, Lie theory is employed to model and propagate uncertainties associated with three-dimensional poses. The evaluation of the AuMeRo system focuses on assessing the accuracy and repeatability of its algorithms and hardware modules. The results demonstrate the effectiveness and potential of the proposed fusion approach in improving state estimation, even in challenging and flexible configurations. Compared to classical filter-based methods, improved robustness and accuracy can be observed. Integrating camera-based pose estimation, the fusion algorithm, and uncertainty modeling provides a modular and efficient solution for achieving accurate robot localization. Thus, this work contributes to the further development of autonomous measurement technology and shows its promising potential for future production environments.
Behavior prediction for autonomous driving using graph neural networks
(Universität Ulm, 2024-11-22) Schmidt, Julian; Dietmayer, Klaus; Flohr, Fabian
Predicting the future behavior of surrounding agents, i.e., their trajectories and intents, allows an autonomous vehicle to plan a safe and comfortable path. Human behavior is complex and interactive, making behavior prediction a difficult task that is usually solved using machine learning-based approaches. Current approaches for behavior prediction mostly assume that there are hard-to-obtain and maintain high definition (HD) maps given. As an alternative, this thesis firstly proposes the map-free and multi-modal trajectory prediction model CRystal ATtention Prediction (CRAT-Pred), which combines a graph neural network (GNN) and self-attention. The evaluation shows that CRAT-Pred is able to achieve state-of-the-art performance for map-free trajectory prediction despite significantly fewer model parameters than other approaches. Secondly, freely available navigation maps are considered instead of HD maps, resulting in a model-agnostic approach to integrate navigation maps into existing learning-based trajectory prediction models. The experiments show that using navigation maps outperforms map-free trajectory prediction significantly and achieves results close to the original HD map-reliant models. These results are promising, because they illustrate that accurate behavior prediction is possible without expensive HD maps. The approaches described above and most approaches known from literature are limited in their ability to encode entities beyond dynamic agents, roads, and lanes, as they use mechanisms tailored to these entity types and their corresponding relations. In its most general form, however, behavior prediction should encompass and reason about all available knowledge that may either be perceived or provided by an HD map. The generic method SCene Encoding NEtwork (SCENE) developed in this thesis overcomes this limitation by encoding a traffic scene in a heterogeneous scene graph that allows for the representation of arbitrary entities and relationships between these entities. A developed heterogeneous GNN operating on this graph is then used for prediction. The evaluation of SCENE demonstrates the effectiveness of combining heterogeneous graphs and GNNs: It manages to outperform all task-specific and other generic baselines on two different intent prediction tasks and even transfers to applications beyond the domain of traffic scenes.
Bewegungsplanung für automatisierte Fahrzeuge in strukturierten Umgebungen unter Nutzung von Fahrermodellen und kontinuierlicher Optimierung
(Universität Ulm, 2021-12-03) Graf, Maximilian; Dietmayer, Klaus; Maurer, Markus
Die autarke Bewegung eines Roboters durch seine Umwelt erfordert die Planung von Sollzuständen auf Basis gemessener und aufbereiteter Daten der Umgebung bzw. des Roboters selbst. Die zeitliche Abfolge von Zuständen wird dabei als Trajektorie bezeichnet und die entsprechende Berechnung als Trajektorien- bzw. Bewegungsplanung. Eine nachgelagerte Regelung setzt die Sollzustände schließlich in Stellsignale zur Ansteuerung der Aktorik um. Dieses allgemeine regelungstechnische Prinzip greift auch beim automatisierten Fahren. Dabei muss eine Reihe von Anforderungen an die Bewegungsplanung gestellt werden: Zum einen muss die Sicherheit der Fahrzeuginsassen stets gewährleistet sein. Entsprechend sind Kollisionen jeglicher Art sowie das Erreichen fahrdynamischer Grenzbereiche durch die Bewegungsplanung zu unterbinden. Außerdem sind Verhaltenskonventionen sowie örtliche Verkehrsregeln zu beachten. Weiterhin soll ein vorausschauendes Fahrverhalten sowie ein möglichst hoher Fahrkomfort für die Fahrzeuginsassen gewährleistet sein. Ein vielversprechender Ansatz in nicht sicherheitskritischen Verkehrssituationen besteht in der Nutzung von Fahrermodellen, welche aus dem Bereich der Verkehrswissenschaften stammen und im Wesentlichen einer Heuristik der Fahrzeugbewegung entsprechen. Die Verbindung mit kontinuierlichen Optimierungsverfahren erlaubt die Planung flexibler Manöver und ermöglicht eine vorausschauende Fahrweise. In sicherheitskritischen Situationen hat der Fahrkomfort hingegen nachrangige Bedeutung und das Fahrzeug ist unter Einhaltung fahrdynamischer Restriktionen schnellstmöglich in den Stillstand zu überführen. In diesem Fall ist die Nutzung rein optimierungsbasierter Methoden mit entsprechenden Beschränkungen zielführend.
Causal modeling and reasoning for identifying functional deficiencies of automated driving systems
(Universität Ulm, 2021-10-22) Chen, Meng; Dietmayer, Klaus; Winner, Hermann
Similar to the transition from horse-drawn to horseless vehicles, safety is one of the crucial aspects during the development of automated driving vehicles for series production. When Software and Hardware failures are taken into consideration, ISO 26262 as an established functional safety standard can guide the development processes of an Automated Driving System (ADS). Beside Software and Hardware failures, other hazard sources exist. Bad weather conditions like fogs or heavy rains can drastically influence the precision of vision-based machine perception. Gesture communications between human drivers or abnormal driving scenarios are challenging for situation comprehension by an ADS. Considering the underlying open world challenge, unknown critical operating conditions would persist by nature. If some critical conditions are not anticipated or the specified functions are not sufficient for handling them, deviations from the intended functionality are possible. These deviations are understood as functional deficiencies of an ADS in this thesis. In the development of Driving Assistance Systems, functional deficiencies are handled mainly by ensuring sufficient controllability for the human drivers. In relation to ADSs, the changing role of humans challenges this approach. Hence, needs for integrating new methods into the development methodology have emerged. For ensuring safety of ADSs, functional deficiencies need to be identified, mitigated and safety assurance must be validated before release. The identification of functional deficiencies is the focus of this work. The survey of Flight Guidance Systems implies that a pragmatic starting point for identifying critical operating conditions is experience-based checklists and knowledge about rare-normal conditions in air transportation is still being accumulated systematically. Researches of autonomous robots suggest to model environment uncertainties in a mathematically rigorous way when it comes to an open operating area. Inspired from these, this work presents a method framework which models the causality knowledge and explicitly quantifies the uncertainties. The method framework consists of three analyses used in the concept and design phase of development. The Hazard Assessment and Risk Analysis (HARA) from ISO 26262 is reused with minor adaptions to identify the effects (deficient behaviors). Then, the Fault Tree Analysis (FTA) as an established method in the field of safety engineering is reusable for identifying the propagation paths. These propagation paths connect deficient behaviors with trigger-events, which are caused by operating conditions. The causes are derived by a novel method called Causal Scenario Analysis (CSA). The cause is interpreted as boundary in a use case (abbr. as boundary) in a CSA. A fuzzy relational model is adopted for capturing the uncertainties of expert opinions. Utilizing this method framework in the concept and design phase of an ADS, known functional deficiencies can be modeled in causality chains between deficient behaviors, trigger-events, and boundaries. Additionally, a diagnosis concept is presented for identifying unknown functional deficiencies during the analysis of testing data. The concept assumes an observation of some trigger-events as input and seeks for plausible boundaries from the CSA as diagnosis solutions. Moreover, measurements or human annotations of plausible boundaries are needed for evaluating their presence. The plausibility of the best solution is used to quantify the explainability of input observations, resulting in data points labeled with either fail unknown or fail known. Consequently, the fail unknown ones contain potentially unknown functional deficiencies to be identified in further investigation. Finally, the method framework and the diagnosis concept are applied using practical examples. The reused methods HARA and FTA are well-suitable for analyzing functional deficiencies. The advantages and disadvantages of the CSA are discussed compared to a reference approach (brainstorming and FTA-modeling). The CSA is proven to be more systematic and results in a well-organized knowledge structure. Using the same experiment setup, the diagnosis concept is implemented and executed based on various types of input data sources. The experiments gradually proved the feasibility of the concept and also confirmed that the scalability is promising with support of human annotations and some automation scripts for preprocessing the raw testing data.
Deep learning on point clouds with applications in vehicle self-localization
(Universität Ulm, 2023-04-20) Engel, Nico; Dietmayer, Klaus; Stiller, Christoph
The precise identification of the vehicle pose, which is comprised of the position and orientation within a local or global coordinate system, is a critical backbone module and an essential prerequisite for the safe and responsible operation of automated vehicles. In well-structured environments with low complexity, for example, on highways, series-produced vehicles already operate in an automated manner in specific scenarios using environment perception only. However, the deployment in highly dynamic and urban areas requires sophisticated perception and action planning algorithms to calculate and perform a driving maneuver appropriate to the current situation. For this purpose, the pose is estimated using self-localization approaches, which, combined with accurate digital map information acts as an additional sensor. For example, an intersection that the sensors cannot correctly perceive can be loaded from the digital map, and valuable and required information is extracted accordingly. Then, appropriate trajectories can be planned, which would not be possible without knowing the vehicle's position on the map. Subsequently, localization methods can be divided into several categories, often characterized by the type of map and measurement representation. This thesis focuses on processing pure point cloud-based measurements and landmarks, which means that the environment is represented by two or three-dimensional unordered and unstructured point sets. Traditional filter-based localization methods are well-explored in the literature, matching multi-modal sensor measurements, such as radar, camera, or lidar measurements, with information from a previously acquired high-accuracy digital map to infer the pose. This process requires a so-called association and matching algorithm, which is usually implemented by heuristic or hand-tuned algorithms. The application of deep neural networks has already revolutionized many areas of computer vision, especially the processing of camera-based image information. Based on these advances, this work aims to present state-of-the-art methods in the area of data-based point cloud processing, as well as novel techniques that are applied to landmark-based self-localization. First, this thesis introduces the Point Transformer network architecture, a general method that processes geometric information from three-dimensional point clouds and learns shape and local dependencies. A novel sorting module is presented, which learns a latent feature representation of the input that induces permutation-invariance. Furthermore, the widespread attention mechanism is adapted to deal with the difficult task of processing arbitrary point sets. The network can be applied to solve common computer vision tasks, such as object classification or part segmentation. The second part of this thesis focuses on a deep neural network that adapts well-known and established point cloud processing architectures for vehicle self-localization. Here, two unstructured point sets represent the input, i.e., the landmarks and multi-modal measurements. The goal is to approximate the traditional self-localization task comprised of multiple subtasks such as measurement-to-landmark association and point cloud registration, using a permutation-invariant neural network that processes the landmarks and measurement sets to infer the vehicle's pose. Moreover, different improvements, a novel training strategy, and two system architectures that allow the deployment in real-world scenarios are presented. The methods are evaluated using a dataset creation pipeline that is able to generate automatically labeled data on the fly. Finally, the introduced Point Transformer approach is adapted for self-localization, where high-density lidar point clouds that contain semantic features in addition to geometric information are processed. Again, an automatically generated and annotated dataset is presented for this method that uses state-of-the-art lidar segmentation networks. The evaluation of all methods presented in this thesis is first performed in synthetically generated scenarios, allowing systematic analysis of different influences and interfering factors of the localization task. Subsequently, the methods are evaluated in experiments using real-world data acquired with an automatically operated vehicle. Compared to traditional filter-based localization algorithms, significant improvements in localization accuracy can be achieved, meeting the requirements for safe and reliable automated operation in urban environments.
Dreidimensionale Rekonstruktion und Verfolgung von Elektrophysiologiekathetern aus asynchronen biplanaren Fluoroskopiebildsequenzen
(Universität Ulm) De Jong, Marcel
The presented work provides a contribution to the field of computer-assisted catheter reconstruction and tracking on fluoroscopy sequences. During a minimally invasive catheter ablation intervention, various catheters are navigated inside the heart under display of two-dimensional X-ray image sequences. The physician is faced with the challenge of properly reconstructing the spatial position of the catheters in his mind. If the geometry of the fluoroscopy system and the positions of the catheter projections in the two image planes are known, the automatic determination of the spatial position of the catheter becomes possible by triangulation. This results in applications such as the direct 3D representation of the procedure and the measurement of the cardiac motion. Overall, the support of the catheter ablation intervention in this form has the potential to simplify, speed up, improve and assure the quality of the procedure. In the presented work, two novel approaches are being investigated for this purpose. They allow a temporal tracking and spatial reconstruction of the catheter using image processing algorithms. The first approach is based on the detection of the catheter tip in the image using a template-matching method and the temporal tracking of its three-dimensional reconstruction by a model-based state filtering method. The special image acquisition situation is dealt with by an implicit triangulation in the state filter. The temporal tracking and spatial reconstruction of a longer course of the catheter body is solved in the second approach by an iterative curve deformation method. In addition, a novel approach to simplify the initialization of the tracking algorithms is presented based on the curve deformation algorithm. The evaluation of the algorithms on a collection of several patient records as well as on simulations show an acceptable accuracy of reconstruction and robustness of the catheter tracking for clinical practise.
Dynamic occupancy grid mapping with recurrent neural networks for autonomous driving
(Universität Ulm, 2024-07-19) Schreiber, Marcel; Dietmayer, Klaus; Stiller, Christoph
A reliable environment perception is a fundamental prerequisite for the realization of automated driving functions. One possible form of environment modeling is the dynamic occupancy grid map, which represents the environment in bird’s eye view. In this environment representation, the vehicle surrounding is divided into grid cells, each containing an occupancy probability and a velocity estimate. Current state-of-the-art approaches use a particle-based algorithm for the estimation of the dynamic occupancy grid map. In this work, a novel learning-based approach is developed for the estimation of dynamic occupancy grid maps. The input data are occupancy grid maps calculated based on light detection and ranging (lidar) measurements of a single measurement cycle using an inverse sensor model. The basic idea is to train a recurrent neural network using sequences of such measurement grid maps as input data to capture motion. The network architecture has an encoder-decoder structure while combining convolutional and recurrent layers. The convolutional layers have the ability to process the image-like grid map data and use spatial context, whereas the recurrent network layers capture movements in a sequence of input data. In this configuration, with measurement grid maps as input data, the recurrent neural network can be seen as a direct substitute for the particle filter algorithm. A main contribution of this work is the development of an ego-motion compensation method, which is applicable in the used network architecture with recurrent states working on different resolutions. During training, sequences of input data are used, which leads to high memory consumption and slow processing time. Instead, during deployment, only the current input data and the recurrent states of the previous time step are used to make the current prediction. The ego-motion compensation method ensures that the network architecture can be deployed in this configuration. Thus, the developed approach achieves real-time capability with comparable large grid map data. The developed approach is evaluated and compared to a state-of-the-art particle-based algorithm. The results show that the developed model significantly reduces the amount of false velocity estimates, i.e. velocity estimates in static areas. In addition, the learning-based approach provides more accurate velocity predictions in more dynamic driving situations, for example turning or acceleration. The evaluation results are also discussed considering the systemic differences between the learning-based approach and the particle-based approach. Compared to the particle-based approach, the recurrent neural network can also be extended to additionally perform a semantic classification of the occupied cells, leading to a more comprehensive environment model. A further extension is the usage of raw lidar data as input instead of measurement grid maps. Therefore, the preprocessing steps consisting of a ground point removal and measurement grid map calculation are not required. Instead, the raw lidar data is represented as a binary 3D grid, containing the value one in a cell with at least one lidar measurement, otherwise zero. This input data is then used in the same neural network architecture. The network output in this setting is a measurement grid map, which is augmented with object labels, i.e. the 2D bounding boxes are represented as occupied areas. The experiments show that the model learns to predict the free space based on the raw lidar points by internally detecting the ground measurements. In addition, the network is capable to predict the object shapes as occupied areas and is superior in the semantic classification compared to a model with measurement grid maps as input data. Moreover, the model can be extended with the task of road classification. The work concludes with a comparative summary of the different evaluated dynamic occupancy grid mapping approaches. The recurrent neural network using the measurement grid maps as input data relies on a classical inverse sensor model but solves some limitations of the particle-based approach. In this setting, the neural network has the ability to generalize well to a changed sensor setup and environment. Instead, the model using the raw lidar data as input has the advantage of fewer preprocessing steps and improved performance in the task of semantic classification. However, the model is then more dependent on the sensor setup and the environment used during training.
Efficient human gesture recognition in automated driving
(Universität Ulm, 2025-02-25) Holzbock, Adrian; Belagiannis, Vasileios; Ortmanns, Maurits
In recent years, the research in automated driving has made significant progress, leading to automated prototype vehicles navigating driverless through urban traffic. The automated vehicle has to interact not only with other vehicles but also with pedestrians or cyclists. In urban traffic, these human traffic participants often communicate with gestures. The automated vehicles must understand the gestures for a safe interaction with the human traffic participants, wherefore this thesis improves relevant components in the gesture recognition procedure. In the first step, the procedure extracts the 2D human pose from a camera image and then tracks the humans’ 2D detections over time to form 2D skeleton sequences. These sequences are converted to 3D skeleton sequences and are taken for gesture recognition, enabling the automated vehicle to understand humans. Overall, this thesis improves the gesture recognition procedure regarding performance, robustness, efficiency, and reliability. The improvement of the gesture recognition performance is done in the pre-processing and the skeleton-based gesture recognition. First, a pedestrian environment model is proposed, containing the pedestrians’ position and pose. The pedestrians’ pose is extracted with a human pose estimation neural network, and the skeletons are tracked over time with a simple tracking approach. The skeleton sequences are used to determine the pedestrian’s location in the world coordinate system, using geometric dependencies between the time steps and a position refinement step. Second, the skeleton-based gesture recognition is enhanced. Here, the temporal data in the skeleton sequences is not processed with the standard recurrent modules but is handled with a multilayer-perceptron-only unit. A transpose operation is applied to process the skeleton data in the spatial and temporal dimensions, and a squeeze-and-excitation block weights the single time steps according to their importance to the final gesture. Afterward, this thesis tackles the robustness of gesture recognition by fusing skeleton and radar data, making gesture recognition insensitive to environmental influences. The different modalities are first processed in separate streams to extract modality-specific information. In a middle fusion, the information of both streams is combined for robust gesture recognition. During training, a proposed auxiliary loss function makes the fusion neural network more robust against single sensor failures. The efficiency of the gesture recognition procedure is improved starting at the human pose estimation by reducing the number of parameters required for the human pose extraction. In this thesis, a data-free and task-agnostic pruning method is introduced that can be used to reduce the parameter set of the neural network’s backbone. Therefore, the neural network is divided into a backbone and a head model. The knowledge lost during pruning is recovered by fine-tuning the backbone with synthetic images. These images are generated by backpropagating through the neural network onto noise images. The parameter optimization of the pruned backbone is performed with a reconstruction loss between the feature maps from the pruned and unpruned model. During inference, the original head is added to the pruned backbone. Finally, the reliability of the pruned neural network is examined in the thesis after the proposed pruning method. Therefore, an investigation of the effect of pruning on uncertainty estimation is presented. Here, the influence of five pruning methods on three uncertainty estimation approaches is studied to exclude harmful influences of pruning on uncertainty estimation. Further, a new evaluation metric is introduced to evaluate the pruning’s influence on uncertainty estimation. All methods presented in this thesis are evaluated on standard benchmarks and reach promising performance.
Environment modeling for automated driving using grid-based and object-based representations
(Universität Ulm, 2023-07-28) Gies, Fabian; Dietmayer, Klaus; Stiller, Christoph
One of the biggest challenges for automated driving vehicles is the environment perception. This task requires a robust, accurate and error-free detection and modeling static obstacles and dynamic objects in the local environment. In the literature, there are a variety of different approaches that detect and model objects, and estimate their states over a period of time. Two of the most well-known and widely used approaches are multi-object tracking and dynamic occupancy grid mapping. In multi-object tracking, the local environment is represented in an object-based representation, while the dynamic occupancy grid mapping uses a gridbased representation. The dynamic occupancy grid mapping allows location-based modeling of arbitrary shaped objects independent of their type. In contrast, multiobject tracking algorithms model an object explicitly by estimating its kinematic and dynamic states. Due to their different computational methods, each has certain advantages and disadvantages depending on the current scene and sensors used. This work presents a functional system architecture, that enables combination of all necessary software components for a robust environment perception. A central module, the environment model, is thereby introduced that fuses the results from multi-object tracking and the dynamic occupancy grid mapping. Therefore, a method is presented that extracts dynamic objects from the grid-based representation. Due to subsequent fusion of objects based on object-based and grid-based representations, a redundant modeling of traffic participants is realized. Thus, a more robust and consistent representation of the environment is achieved. The results of the centralized environment model are used to plan and execute the vehicle’s future trajectory. Thus, in order to be error-free, the likelihood is evaluated whether the estimated objects exist in reality or are false estimations. Therefore, an extended existence probability is introduced, which relates the estimated objects based on the different contextual constraints from the digital maps and checks their plausibility. In summary, the presented approach provides a modular system architecture to integrate algorithms for environment perception. A comprehensive environment model fuses the object-based and grid-based representations which results in an accurate, robust and consistent perception of the local environment. The presented methods are evaluated based on the real-world data from an experimental vehicle.
Extrinsic calibration for intelligent vehicles and infrastructure sensors using dual quaternions
(Universität Ulm, 2024-03-07) Horn, Markus; Dietmayer, Klaus; Sawodny, Oliver
The objective of extrinsic sensor calibration is to estimate transformations of sensors relative to other sensors or coordinate frames. These transformations are required for any system that fuses measurements of multiple sensors. This thesis develops a framework to unify multiple formulations for extrinsic sensor calibration: point set registration, pose set registration, hand-eye calibration, and hand-eye robot-world calibration. The transformations are represented as unit dual quaternions, making it possible to formulate all covered optimization problems as quadratically constrained quadratic programs that only require a few equality constraints to ensure valid solutions. The proposed methods are applied in this thesis for calibrating intelligent vehicles and infrastructure sensors. Especially vehicles pose an additional challenge since, due to their restricted motion, no unique solution is possible with hand-eye calibration without including additional a priori information. The same restriction applies to hand-eye robot-world calibration. Hence, multiple extensions for handling planar-only motions and poses are presented in this work. Either the common ground plane of all sensors or the measured translation norm between the sensors is included in the calibration process. These extensions further make it possible to apply the proposed hand-eye robot-world calibration for automatic, geo-referenced calibration of infrastructure sensors. The proposed methods are evaluated and compared quantitatively to the state of the art on artificial measurements and sensor data simulated in the CARLA simulator. The latter is used to simulate the sensor setup of an automated vehicle, an intersection with multiple cameras, and all calibration targets, allowing a realistic evaluation with ground-truth transformations. Furthermore, all proposed methods are evaluated using measurements of an experimental vehicle and infrastructure sensors at a test site for connected and cooperative mobility technology. The results on all scenarios demonstrate that the proposed methods can obtain accurate transformations which can be used for all described applications.
Fahrzeugübergreifende Informationsfusion für ein Kreuzungsassistenzsystem
(Universität Ulm, 2017-12-14) Seeliger, Florian; Dietmayer, Klaus; Trommer, Gert
Fahrzeuglokale Perzeptionssysteme liefern die Datenbasis für aktuelle Fahrerassistenzsysteme wie zum Beispiel Adaptive Cruise Control (ACC) oder Notbremsassistenten. Für neue Anwendungen, wie die Kreuzungsassistenz, reicht die Informationsbasis fahrzeuglokaler Perzeptionssysteme in manchen Situationen nicht aus, z.B. wenn die Sicht auf einen Konfliktpartner durch andere Verkehrsteilnehmer oder statische Objekte verdeckt ist. Die Erfassung des Konfliktpartners mit der fahrzeuglokalen Sensorik ist dann nur eingeschränkt oder gar nicht möglich. Zukünftige Fahrerassistenzsysteme die den Fahrer in Situationen unterstützen sollen, in denen häufig Sichtverdeckungen auftreten können, benötigen daher eine erweiterte Informationsbasis. In dieser Arbeit wird eine Erweiterung des lokalen Fahrzeugumfeldmodells durch den Austausch von Lokalisierungs- und Perzeptionsinformation zwischen mehreren Fahrzeugen und einem Kreuzungsperzeptionssystem realisiert. Die Arbeit war Teil des Projektes Ko-PER der Forschungsinitiative Ko-FAS, das sich explizit mit der Fahrerassistenz in Verdeckungssituationen beschäftigt hat. Der Austausch der Information erfolgt mit Hilfe von C2X-Nachrichten, die in Ko-PER entwickelt wurden. Zentraler Beitrag dieser Arbeit ist ein echtzeitfähiges fahrzeugübergreifendes Informationsfusionssystem, welches die empfangene Information mit fahrzeuglokaler Information zu einem konsistenten Umfeldmodell zusammenfügt. Bei der ausgetauschten Perzeptionsinformation handelt es sich um bereits zeitlich gefilterte Objektzustände, die mit Hilfe von Track-2-Track-Fusionsmethoden fusioniert werden. Hiermit wird berücksichtigt, dass in den zu fusionierenden Daten identische Information in Form von Modellwissen enthalten ist, wodurch inkonsistente Schätzergebnisse vermieden werden. Weiterhin wird bei der Kommunikation und Fusion die räumliche Ausdehnung der Objekte mit Hilfe eines Bezugspunktes berücksichtigt. Da die Bezugspunkte der Objektdaten, je nach Perspektive der Informationsquellen, verschieden sein können, werden die Bezugspunkte im fahrzeugübergreifenden Informationsfusionssystem bei Bedarf angeglichen. Die Güte der Fusion wird im Rahmen eines Vergleichs von Track-2-Track-Fusionsmethoden evaluiert. Aufbauend auf dem entwickelten fahrzeugübergreifenden Informationsfusionssystem wurde in Ko-PER ein prototypisches Kreuzungsassistenzsystem realisiert. Dieses System wird genutzt um die Eignung des generierten erweiterten Umfeldmodells für die Kreuzungsassistenz nachzuweisen. Es werden sowohl die Echtzeitfähigkeit des fahrzeugübergreifenden Informationsfusionssystems in Grenzsituationen hinsichtlich der Objektassoziation betrachtet als auch die Realisierbarkeit von frühzeitigen informativen Warnungen. Letztere ermöglichen es dem Fahrer, eigenständig eine Konfliktsituation durch einen Bremseingriff zu entschärfen.
Fahrzeugumfelderfassung und Fußgängerschutz unter Nutzung mehrzeiliger Laserscanner
(Universität Ulm) Fürstenberg, Kay Christian
Komplexität und erhebliche Kosten eines umfassenden Sicherheitskonzepts auf Basis vorausschauender Sensorsysteme werden als Hindernisse einer ausgedehnten Verbreitung identifiziert. Das in dieser Arbeit vorgestellte universelle Fahrzeugumfeldmodell auf Basis eines Laserscanners kann sowohl die Komplexität als auch die Kosten signifikant reduzieren, da diese einer Vielzahl von Applikationen als Datenbasis dienen. Das Gesamtsystem umfasst Algorithmen zur Segmentierung, Eigenbewegungsermittlung, Tracking und Klassifikation. Zusätzlich wurde eine Prototypenimplementierung zum Fußgängerschutz realisiert und umfassend evaluiert. Insbesondere die Fußgängererkennung ist durch die FFT-basierte parallele Auswertung der zeitlichen Amplitudenverläufe des linken und des rechten Beines in allen vier Layern des Laserscanners besonders robust. Diese Beinpendelanalyse kann mit dem eingesetzten Laserscanner sowohl longitudinal als auch lateral bewegte Fußgänger vor dem Fahrzeug erfolgreich klassifizieren. Die Fußgängererkennung zeigt eine Zuverlässigkeit mit nur vier Fehlklassifikationen in den mehr als zwei Stunden Testfahrten, bei gleichzeitig hoher Detektionsrate (82 %). Es wurde ein innovativer Ansatz zur Unfallfolgenminderung auf Basis einer RONE (Region of no Escape) entwickelt. Diese RONE beschreibt einen dynamisch definierbaren Bereich vor dem Fahrzeug, in dem das Vorhandensein eines Fußgängers zu einem unvermeidbaren Fahrzeug-Fußgänger-Unfall führt. Diese Applikation wurde in Fahrten von mehr als 14.000 km ohne einen einzigen Fehlalarm getestet. Die durchgeführten Positivtests zeigen Detektionsraten von über 80 %. Diese hohe Zuverlässigkeit macht eine signifikante Verbesserung des Fußgängerschutzes möglich.
Feature-level fusion of laser scanner and video data for advanced driver assistance systems
(Universität Ulm) Kämpchen, Nico
Advanced driver assistance systems aim at an improved traffic safety, enhanced comfort and driving pleasure. Sensors perceive the objects surrounding the vehicle and produce an environment description. The assistance systems support the driver by assessing the situation recognized by this vehicle environment description. Current research in the area of advanced driver assistance systems aims at increased functionality. Comfort systems, such as the ACC, are expected to support the driver not only in normal driving phases, but also in more complex situations such as traffic jams. Safety systems will trigger collision avoidance or mitigation measures in a number of potential crash configurations and not only in well defined rear end collision situations. These future advanced applications impose strong requirements on the sensors and demand novel and complex signal processing algorithms for vehicle detection, tracking and situation assessment. The present work addresses challenging traffic scenarios. Novel sensor signal processing, sensor data fusion and tracking algorithms were developed which provide precise and consistent motion estimates for complex intersection scenarios, lane change maneuvers of vehicles, trucks on neighbouring lanes and highly dynamic situations. A novel emergency brake application offers a general solution to the situation assessment and aims in particular at so far unresolved intersection scenarios.
Filtersynthese zur simultanen Minimierung von Existenz-, Assoziations- und Zustandsunsicherheiten in der Fahrzeugumfelderfassung mit heterogenen Sensordaten
(Universität Ulm) Mählisch, Mirko
This work analyzes the classical system architecture for environment perception consisting of separate modules for the uncertainty domains of detection, data association, and state estimation with respect to early decision making and the usage of thresholds and heuristics. Within this common processing chain error sources are identified. The reason behind these errors is the inability of the state space model underlying recursive Bayesian state estimation to inherently represent all three existing uncertainty domains. For the first time in an automotive context, a knowledge representation based on Mahler’s Random Finite Set Statistics theory is used successfully. This approach allows the simultaneous representation of all three uncertainty domains in a single knowledge base. For the chosen parametric model, the Multi-Object Multi-Bernoulli distribution on random sets, automotive transition and measurement models get developed. It is shown that under certain assumptions, the resulting FISST filtering equations correspond to the Joint Integrated Probabilistic Data Association Filter. The presented filter handles all uncertainty domains within a holistic algorithm and replaces the classical modularized system architecture. Particularly, the algorithm is free of early decisions, thresholds and heuristics for the price of an increased modelling effort. Besides the elimination of error sources, there is also the benefit of a less expensive optimization procedure due to the reduced number of system parameters. A prototype perception system is demonstrated with a video and a Lidar sensor. The test car implementation tracks other cars up to distances of 200 meters regardless of whether they are moving or standing. A superior perception performance of the JIPDAF outputs compared to the raw detections is revealed with ROC charts. The work also analyzes the transmission behaviour of the JIPDAF.
Fully bayesian vehicle tracking using extended object models
(Universität Ulm, 2019-12-05) Scheel, Alexander; Dietmayer, Klaus; Stiller, Christoph
Environment perception systems for automated vehicles and driver assistance systems commonly use radar and lidar sensors as well as cameras for tracking other vehicles. While these sensors exhibit different strengths and weaknesses, they share the ability to resolve multiple measurements for a single object: lidar sensors typically provide multiple distance measurements for a vehicle, modern high-resolution radar sensors yield multiple detections, and objects cover a set of pixels in camera images. Such dense data is valuable for obtaining a detailed and precise representation of the environment. From a tracking perspective, however, it poses three major challenges: First, tracking algorithms have to process multiple measurements for one object and thus face the extended object problem. This conflicts with the classical assumption of at most one measurement per cycle and object. Secondly, the increased amount of data further complicates multi-object problems which involve measurement-to-object associations in the presence of clutter, occlusion, and misdetections. Thirdly, the data from several heterogeneous sensors needs to be fused into a consistent estimate in order to benefit from complementary sensor strengths or to either achieve increased sensor fields of view or redundancy. This thesis presents a tracking framework that is based on finite set statistics (FISST) and tackles these challenges for the application of vehicle tracking in a Bayesian fashion. The employed multi-object filter uses labeled multi-Bernoulli distributions and allows for combining different sensor modules with sensor-specific update routines through centralized fusion. Furthermore, extended object measurement models which work on the full raw sensor data are developed. In particular, it is demonstrated how separable likelihood approaches can be used to incorporate accurate vehicle models for lidar data and semantically labeled camera images. Additionally, two radar vehicle models are developed. They are able to process multiple radar detections and allow for tracking vehicles in arbitrary maneuvers, including turning vehicles or cross-traffic. While the first radar model is based on expert knowledge, the second uses variational Gaussian mixtures and is learned from actual measurements. The methods are evaluated on experimental data with accurate ground truth. The results demonstrate that the extended object models achieve precise estimates, that the data-driven variational radar model is able to outperform the manually designed model, and how different sensor combinations improve the performance.
Generic sensor data fusion in information space and a new approach to processing dense sensor data
(Universität Ulm, 2018-02-05) Wilking, Benjamin; Dietmayer, Klaus; Wanielik, Gerd
Sensor data fusion is the key to a comprehensive environment perception by today’s and future systems basing on object tracking. Early and rather simple advanced driver assistance systems (ADAS) are still using a sensor setup with a single sensor where the problem of fusing different data sources does not arise. With the intent on fusing data from multiple heterogeneous sensors into one common object tracking system, various fusion methods are conceivable. Probabilistic data association (PDA) is one of these methods and it has been shown to be feasible and effective in former publications [Mäh09; Mun11]. In very complex scenarios, algorithms based on finite set statistics [Mah03] have become popular over the last few years. These allow the modeling of interactions between the tracked objects to resolve ambiguities comparable to the multi-object Bayes filter. Independent of the fusion method, all procedures share one major drawback: the interchangeability of the sensors is not possible. In many systems it is necessary to transmit knowledge about the sensors to the fusion system. Thus, a change of the sensor setup can entail comprehensive consequences and can be expensive and costly in development time. Besides increasing the sensor interchangeability, the anonymization of the sensor is desirable. This allows sensors to be used without knowledge of the sensors’ theoretical principles and enables the sensor manufacturer to maintain secrecy about the details. The generic linkage of sensors to object tracking systems as well as the anonymization of sensors is the purpose of this work. Therefore, a mathematically equivalent alternative to the Kalman filter, the information filter, is used. The focus is set on probabilistic data association and the successful adaption of it to use the information filter is evaluated in simulation and real-data scenarios. Additionally, it is shown how to use the information filter approach in many other fusion systems. In the further course of this work a novel approach to preprocessing high density data from distance measuring sensors is presented. This new approach meets the requirements on generically linked sensors. It allows the use of the information space and simultaneously increases the perception performance markedly in comparison to former attempts. This is achieved by filtering the raw sensor data over time and generating reliable object hypotheses using the filtered data. The performance of the achieved sensor model is demonstrated in various real-data scenarios.
Generisches Sensorfusionsframework zur gleichzeitigen Zustands- und Existenzschätzung für die Fahrzeugumfelderkennung
(Universität Ulm) Munz, Michael
The presented work provides a contribution in the field of automotive environment perception based on sensor fusion systems. In contrast to other existing fusion systems, a generic framework is realized, which allows for exchanging any sensor module without the need to adapt the fusion algorithm. This is achieved by modeling sensor specific properties like detection performance or reliability in a probabilistic way, which allows the definition of abstract interfaces between sensor and fusion module. The fusion process is carried out by simultaneously estimating state and existence of objects based on probabilistic information by the sensors. In addition, the resulting environment model can be processed by different driver assistance applications at the same time, which may have unequal needs for reliability of object existence. The requirements for a generic fusion system are analyzed and the need for novel sensor fusion algorithms is described. Existing data fusion algorithms are extended for the claims of the generic fusion system and novel data fusion methods are introduced. Approximations for handling the computationally expensive calculations in real-time are developed which allow ensuring a maximum computing time of the fusion process. Moreover, methods for modeling sensor specific properties, object birth, and object classification are presented. The proposed sensor fusion system is implemented for a specific sensor combination consisting of a laser scanner and a video camera for detecting and tracking vehicles in dense environments like inner city areas. Practical considerations concerning cross-calibration of sensors and timing aspects are provided. The probabilistic sensor models are described in detail. Finally, an exhaustive evaluation of the system based on real-world data is presented and several algorithms are compared. Relevant system parameters like detection performance, tracking time, estimation consistency and state precision are examined.
Hierarchische modellprädiktive Betriebsstrategie für Elektrofahrzeuge mit redundanten Antriebssträngen
(Universität Ulm, 2020-12-18) Bächle, Thomas; Dietmayer, Klaus; Sawodny, Oliver
Mobilitätskonzepte der Zukunft zeichnen sich durch einen wachsenden Elektrifizierungsgrad aus. Einerseits betrifft dies klassische Verbrennerkonzepte, die sich mit Elektromotoren und batteriebasierten Energiespeichern zu Hybridantrieben wandeln. Andererseits entstehen durch die beträchtlichen Fortschritte in der Batterietechnologie zunehmend auch rein batterieelektrisch betriebene Fahrzeuge. Beiden Varianten ist gemein, dass durch die elektrifizierten Antriebsstränge eine Energieübertragung innerhalb des Fahrzeugs einfach und effizient möglich wird. Gekoppelt mit der hohen Leistungsdichte moderner Elektromotoren entstehen so vielfältige Antriebstopologien mit individuell angetriebenen Achsen oder auch einzelnen Rädern. Neben fahrdynamischen Vorteilen wird eine Redundanz erreicht, wodurch geeignete Ansteuerverfahren beispielsweise einen Weiterbetrieb bei Ausfall einzelner Maschinen darstellen können. Jedoch stellen derartige Fahrzeuge hohe Anforderungen an den Bauraum und das Gewicht der Antriebsstrangkomponenten, woraus sich ein hoher Ausnutzungsgrad der Elektromaschinen und erhöhte Anforderungen an das Kühlsystem ergeben. Auch hier ermöglicht eine vorhandene Redundanz, gezielt Einfluss auf die Erwärmung der Maschinen zu nehmen. Motiviert aus diesen Gegebenheiten befasst sich die vorliegende Arbeit mit einer modellprädiktiven Betriebsstrategie für überaktuierte Fahrzeuge mit elektrifizierten Antriebssträngen. In einer hierarchischen Struktur kommt zunächst eine auf kurzfristige Optimalität ausgelegte Momentenverteilungsstrategie zum Einsatz. In dieser wird die Aufgabe des situationsabhängigen Einsatzes der verschiedenen Antriebe unter Beachtung systemweiter Aspekte wie der Einhaltung von Beschränkungen gelöst und für übergeordnete Kontrollstrategien abstrahiert. Aufbauend darauf zielt eine langfristig orientierte Strategie auf eine thermische Konditionierung der verbauten Antriebe ab. Unter Einbeziehung der voraussichtlichen Fahraufgabe erfolgt eine geeignete Beeinflussung der unterlagerten Momentenverteilung, so dass eine Absenkung der Maschinentemperaturen erzielt werden kann. Während sich die resultierende, echtzeitfähige Gesamtbetriebsstrategie für eine Vielfalt an Antriebstopologien eignet, wird eine radindividuelle Allradtopologie als konkretes Anwendungsbeispiel betrachtet. Ein entsprechend aufgebautes Versuchsfahrzeug dient als Basis und Validierungswerkzeug.
Hochgenaue Positionierung und Kartographie mit Laserscannern für Fahrerassistenzsysteme
(Universität Ulm) Weiss, Thorsten-Tobias
Driver assistance systems depend on an accurate environmental model containing the states of moving and stationary objects. The main goal of this work is to provide a detailed model of the stationary environment and the accurate pose of the observing vehicle in it. This is an important component for future driving assistance systems such as autonomous driving and intersection safety. The consistent movement estimation of the host vehicle based on serial sensors is a basic algorithm used by many applications. Quantitative analyses show the accuracy and the limitations of the approach. Online Maps provide detailed representations of the stationary environment based on laser scanner data and enable a robust separation of moving and stationary objects. The occupancy grid mapping approach was extended to get accurate results in real traffic environments. It considers for uncertainties of the sensor measurements and the movement estimation. SLAM (Simultaneous Localization And Mapping) algorithms are proposed, which are designed to improve the accuracy of the movement estimation in real traffic scenarios using laser scanners. gridSLAM compensates small errors in the movement estimation in common driving situations. The hybridSLAM approach is used to determine the movement of the vehicle in skidding and drift manoeuvres using both point style landmarks and expanded stationary objects with arbitrary outer contours, which can be found in most traffic scenarios. The approaches are switched by a robust skidding detection algorithm. Based on the Online Map a new approach for driving path detection is proposed, which detects lane boundaries and moving objects driving in front of the host vehicle. Stationary objects are extracted from grid maps with high position accuracy. Infrastructure elements and lane boundaries are detected automatically. Positioning algorithms allow for a highly accurate determination of the position and orientation relative to the maps.

Browse

Browsing Publikationen by Serial Ulm "Schriftenreihe des Instituts für Mess-, Regel- und Mikrotechnik"

Results Per Page

Sort Options