This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.
The idea of letting a machine do the driving for you may put a lot of people off autonomous vehicles. But research could make it possible to backseat-drive an autonomous vehicle just as you might with a human driver.
Self-driving cars carefully balance a host of parameters to ensure a smooth ride, including things like speed, acceleration, and the smoothness of turns. But human driving preferences can often vary depending on how much of a rush they’re in, whether they’re feeling carsick, or how busy the traffic is.
These cars have a software component called the motion planner, which is responsible for choosing a safe and efficient path through traffic. The motion planner is normally tuned by engineers before the vehicles hit the road so that there’s little scope for passengers to adjust a vehicle’s driving style on the fly. But now researchers at the Delft University of Technology (TU Delft) in the Netherlands have developed a system that uses a large language model (LLM) to translate natural-language user requests such as “I am running late, go fast” into adjustments to a self-driving control system. The researchers posted their preprint on arXiv and are presenting the work at the IEEE Intelligent Transportation Systems Conference in September.
LLMs Personalize Autonomous Driving
The system doesn’t give users direct control over the vehicle’s driving decisions; it simply tunes the parameters of a safety-aware motion-planning algorithm, which helps to keep the vehicle’s behavior within safe bounds. And the system keeps the human in the loop by describing how it’s going to alter its behavior in nontechnical language, and by asking the passenger to confirm before making changes. When the system was tested in simulation, the researchers found it adjusted the speed and smoothness of driving in line with natural-language instructions.
“The motion-planning problem is not only about reaching a place while avoiding collisions, it’s also how you do it,” says lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft. “The motivation here is trying to make the way the autonomous car drives adaptable by end users easily, just by talking to the car.”
Previous research has investigated the potential of using LLMs and video-language models (VLMs) to direct decision-making for self-driving vehicles, but the researchers deliberately targeted driving style instead. Using LLMs and VLMs to directly control vehicles faces several challenges, says Martinez-Baselga. These include relatively slow response times, which can make these models unsuitable for the fast-paced decision-making required in driving, and the fact that they can’t provide concrete performance guarantees in the way a deterministic motion planner can.
Instead, the researchers used an LLM’s language and reasoning capabilities to translate fuzzy human preferences into something a vehicle’s motion planner can use. The system relies on a model predictive-path integral controller previously developed by the researchers, which identifies multiple paths the vehicle could take to reach its goal and then judges them on various criteria, including speed, steering angle, and collision probability. It then finds an optimal path that is a combination of the trajectories that scored best on those judging criteria.
The team combined this with OpenAI’s GPT-4o-mini model to parse passengers’ natural-language suggestions and use them to tune how the controller chooses its path. The model is given the users’ prompt and a natural-language description of the scenario the vehicle is operating in. The description was handwritten by the researchers for the purposes of the study, but it could ultimately be provided directly by a car’s perception system, says Martinez-Baselga.
The model doesn’t directly tweak the settings of the controller; it uses the prompt to rate the relative importance of the judging criteria the controller uses to assess trajectories. This rating is then used to adjust each criteria up or down either side of a safe baseline set by the researchers. So, if a user says they are feeling dizzy, the LLM will dial up parameters that encourage smooth steering and gentle acceleration to make the vehicle favor more sedate travel.
Prior to making any changes, however, the model first presents the user with a natural-language description of the adjustments it plans to implement. The user can then sign off on the plan or make further suggestions. The system is also interactive, so the user can request further adjustments if the vehicle’s behavior doesn’t match expectations or the user‘s preferences change.
Martinez-Baselga says this human-in-the-loop system allows the passenger to catch instances when the model misinterprets prompts. But it also helps deal with the inherent subjectivity of suggestions like “go faster” or the possibility that models don’t accurately describe changes they plan to make. In that case the passenger can simply follow up with additional prompts “as you would do if you were in a taxi or with a friend that is driving,” says Martinez-Baselga.
The researchers tested the system in the popular self-driving simulator nuPlan in scenarios that involved merging onto a busy highway. Across eight different prompts, the system changed the controller’s parameters in ways matching user intent, with requests for a more comfortable ride dialing up smoothness and those indicating urgency leading to higher speeds.
This isn’t the first time LLMs have been used to tune a self-driving car’s motion planner. Nicolas Baumann, a Ph.D. student at ETH Zurich in Switzerland, published research last year in which an LLM tweaked the parameters of a model racing-car controller, allowing the user to alter driving style but also give more concrete instructions like “reverse the car” or “maintain a specific speed.”
The strength of the approach, says Baumann, is that separating the LLM from the main controller means that even if the model hallucinates, it can’t do anything dangerous. “You get the possibility of language interaction, but you can guarantee that it is going to be within the constraints of this classical controller, so you can bake in safety,” he says. However, setting these constraints requires considerable engineering work, he adds.
And if you want provable safety, you need to go a step further, says Matthias Althoff, a professor of cyberphysical systems at the Technical University of Munich. His group built a system that gets an LLM to suggest driving decisions, but then uses a mathematical process to check them against traffic rules and predictions about the behavior of other road users. This makes it possible to verify their safety before committing to them, something the Delft paper doesn’t provide. “As with any LLM, it is not guaranteed that the result is correct,” says Althoff. “For that reason, we safeguard the decisions of the LLM in our works.”
Edd Gent is a freelance science and technology writer based in Bengaluru, India. His writing focuses on emerging technologies across computing, engineering, energy and bioscience. He's on Twitter at @EddytheGent and email at edd dot gent at outlook dot com. His PGP fingerprint is ABB8 6BB3 3E69 C4A7 EC91 611B 5C12 193D 5DFC C01B. His public key is here. DM for Signal info.
Facts Only
* Researchers at Delft University of Technology (TU Delft) developed a system using a large language model (LLM) to adjust autonomous vehicle driving styles.
* The system utilizes OpenAI’s GPT-4o-mini model.
* The LLM translates natural-language user requests into parameter adjustments for a model predictive-path integral controller.
* The system was tested using the nuPlan self-driving simulator.
* Scenarios tested involved merging onto a busy highway.
* The system requires passenger confirmation via natural-language descriptions before implementing changes.
* The research was posted as a preprint on arXiv.
* The work is being presented at the IEEE Intelligent Transportation Systems Conference in September.
* Diego Martinez-Baselga is the lead author and a postdoctoral researcher at TU Delft.
* Separate research by Nicolas Baumann at ETH Zurich involved an LLM tweaking a racing-car controller.
* Matthias Althoff at the Technical University of Munich developed a system that verifies LLM driving decisions against traffic rules using a mathematical process.
Executive Summary
Researchers at TU Delft have introduced a method to personalize autonomous vehicle behavior by integrating a large language model (LLM) with a safety-aware motion planner. Unlike systems that give users direct control over steering or braking, this approach uses an LLM to translate vague human preferences—such as "I am running late"—into specific weight adjustments for the vehicle's existing safety parameters. By adjusting the relative importance of criteria like speed and smoothness, the vehicle can alter its driving style while remaining within predefined safety bounds.
To mitigate the risks of LLM hallucinations or misinterpretations, the system employs a human-in-the-loop design, requiring passengers to confirm proposed changes before they are enacted. While simulation results in the nuPlan environment indicate successful alignment with user intent, external experts note a distinction between "safe bounds" and "provable safety." Some suggest that while separating the LLM from the core controller prevents dangerous direct actions, additional mathematical verification is necessary to ensure strict adherence to traffic laws and predictable safety guarantees.
Full Take
This research operates in ACADEMIC MODE, focusing on the intersection of stochastic language models and deterministic control systems.
1. METHODOLOGY CHECK: The study relies on the nuPlan simulator, which provides a controlled environment but lacks the edge-case volatility of real-world urban driving. The "scenario descriptions" provided to the LLM were handwritten by researchers; replacing these with real-time perception system data introduces a significant layer of potential noise and error not addressed in the preprint.
2. CLAIMS vs EVIDENCE: The findings demonstrate that the system can successfully map natural language to parameter shifts (e.g., "urgency" to "higher speed"). However, the claim of "keeping behavior within safe bounds" is relative to the motion planner's baseline. It demonstrates functional adaptability, not a guarantee of absolute safety.
3. LITERATURE CONTEXT: The work extends the field by shifting the LLM's role from a decision-maker to a "tuner." It acknowledges the latency and non-deterministic nature of LLMs as a primary constraint, positioning the LLM as a translation layer rather than a driver.
4. REAL-WORLD IMPLICATIONS: If successful, this restores a sense of agency to the passenger, transitioning the experience from being a passive cargo to an active supervisor. However, it introduces a new cognitive load: the passenger must now audit the vehicle's proposed logic before every adjustment.
5. BRIDGE QUESTIONS: How does the system handle contradictory or paradoxical prompts (e.g., "get there as fast as possible but make it incredibly smooth")? Would the introduction of emotional stress in a user's voice alter the LLM's parameter weighting in an unpredictable way?
COUNTERSTRIKE SCAN: A coordinated campaign would frame this as "giving humans total control" to mask the underlying reliance on black-box LLMs. The actual content is clean, explicitly detailing the constraints and the necessary human-in-the-loop safeguards.