Lightweight End-to-End Multimodal Model for Autonomous Driving (2025)

III-A VLM Selection

We select state-of-the-art VLMs from both open-source and commercial offerings, covering 6 model types with a total of 12 models. For each model type, we evaluate two variants: a basic version and an advanced version. All models used are the latest publicly available versions (as of the end of this project) that support both textual and image inputs. This setup enables comprehensive performance comparisons both across different models and between variants within the same model family. The selected models are: GPT-4o, GPT-4.1 [25], Gemini-2.0-Flash, Gemini-2.5-Pro [14], Claude-3.5-Sonnet, Claude-3.7-Sonnet [26], DeepSeek-VL2-16B, DeepSeek-VL2-28B [27], LLaMA-3.2-11B-Vision-Instruct, LLaMA-3.2-90B-Vision-Instruct [28], Qwen2.5-VL-7B-Instruct, and Qwen2.5-VL-72B-Instruct [29].

For commercial models, we access them via paid APIs. This approach simplifies deployment by eliminating the need to manage local hardware, software updates, and scalability, as these tasks are handled directly by the providers.

For open-source models, we download them from HuggingFace and deploy them locally using H100 GPUs. Most models require only a single H100 GPU, although larger models may require more; we report the minimum number of GPUs needed in TableI. To facilitate multi-GPU deployments, we leverage PyTorch’s automatic device mapping for efficient GPU utilization.

III-B Camera Input

When the front-view camera image is input into the VLM, we do not employ any visual encoders such as CLIP [30], nor do we apply preprocessing techniques to modify the image before feeding it to the model. Our findings indicate that VLMs effectively describe the scene and accurately identify objects directly from raw visual inputs, demonstrating robustness in handling unprocessed visual data.

In line with this design approach, we also choose to use only the current driving scene image as input, rather than concatenating multiple past frames as performed in previous studies [13, 15]. Our preliminary experiments indicate that incorporating additional frames does not yield noticeable performance gains. Instead, the model tends to redundantly extract identical features across multiple frames rather than capturing meaningful spatiotemporal dynamics. Additionally, adding more frames results in a roughly linear increase in processing time and computational cost without clear performance benefits.

Alternatively, models such as VideoBERT [31] and VideoMAE [32] support video inputs through specialized temporal encodings rather than simply treating videos as frame sequences. Such models inherently adopt different architectures and could potentially capture richer temporal information. However, exploring these specialized video-oriented models falls outside the scope of our current study.

III-C Driving History Input

Our work represents vehicle actions using speed and curvature, an interpretable format in which speed captures longitudinal motion and curvature describes lateral movement. We adopt this representation instead of explicit trajectory points because VLMs often struggle to reason effectively about coordinates that involve implicit physical constraints.

III-D VLM Promoting

We adopt a straightforward CoT approach to guide the VLM in scene understanding and action generation, where the output from each stage is integrated into the subsequent stage along with additional prompt:

III-E Example Workflow

In this section, we demonstrate an example workflow and results from the CoT process using the images shown in Fig.1. The responses and the trajectory are generated by Claude-3.7-Sonnet. Due to space limitations, we preserve the original wording but omitted some less critical details.

Scenario Description: The image shows an urban street scene with active road construction or maintenance work…A police officer is standing in the middle of the road directing traffic…There are several construction vehicles visible including a black dump truck parked on the left side of the road along with other work vehicles…Traffic appears to be controlled through this construction zone with the presence of the officer and possible lane restrictions.

High-Level Intent: Given the current scene with active construction and a police officer directing traffic in the middle of the road, the ego vehicle should significantly decelerate by approximately $2-3m/s^{2}$ to prepare for possible stopping, reducing speed to around $3-4m/s$ in the next 3 seconds. The vehicle should continue following the lane while being prepared to come to a complete stop if directed by the officer.

Low-Level Commands: [(6.0, -0.001), (5.0, -0.001), (4.0, 0.0), (3.5, 0.0), (3.0, 0.0), (3.0, 0.0)].

IV-A Inference Time

A summary of inference times is presented in TableI, showing the average processing time per image frame. Gemini-2.0-Flash achieves the fastest inference at only 4.5 seconds per frame, while LLaMA-3.2-90b is the slowest at 40.8 seconds. Qwen-2.5-72B and Gemini-2.5-Pro also exhibit relatively slow performance, each requiring over 30 seconds per frame. The remaining models typically operate around 10 seconds per frame, with basic versions generally running faster than their advanced counterparts.

Note that even the fastest model, Gemini-2.0-Flash, has a processing time significantly slower than the real-time update frequency. To be genuinely effective for real-world deployment, these models would need to operate one to two orders of magnitude faster. Additionally, API-based commercial models rely on stable internet connections, which might be unreliable in a moving vehicle. Conversely, local deployment faces constraints from limited computing power and energy consumption, further restricting their practicality.

IV-B Input and Output Tokens

We compute the average number of input and output tokens per frame using the official instructions provided by each model. As shown in TableI, the number of input tokens is significantly higher than the output tokens, typically around 6000 input tokens compared to roughly 300 output tokens. This aligns with expectations, as inputs include image data while outputs are purely textual.

However, there are some exceptions. The LLaMA models report only about 1000 input tokens per frame. Upon further investigation, we discovered that the official LLaMA token counting method excludes image tokens, counting only text. Despite extensive efforts, we could not identify a reliable approach to accurately estimate image-related tokens; therefore, we present these results as provided by the official method.

Additionally, the token counts for Gemini-2.5-Pro clearly contain errors in both input and output token calculations, as they significantly deviate from the results of comparable models. Notably, Gemini-2.0-Flash, calculated using the identical token counting setup, produced consistent and reasonable results, indicating an issue specific to Gemini-2.5-Pro that needs correction.

IV-C Cost

The cost section applies exclusively to commercial APIs. To ensure accurate measurement and reporting, billing history is cross-referenced with official pricing tables, based on input and output token usage. For clarity, all results presented in TableI are shown in cents per frame.

Gemini-2.0-Flash is the cheapest at only 0.07, making its cost negligible. GPT-4o and GPT-4.1 exhibit similar costs, approximately 1.3. Claude-3.7-Sonnet is significantly more expensive than Claude-3.5-Sonnet, and notably pricier than the GPT models. Due to inaccuracies in Gemini-2.5-Pro’s token calculations, an exact estimate is difficult. Thus, the value reported here is based solely on billing history after running the model.

IV-D Response Error

In the final model output stage, we observed diverse response formatting errors. Although we prompt the VLM to strictly return outputs in the format [(v1, c1), (v2, c2), (v3, c3), (v4, c4), (v5, c5), (v6, c6)] without additional text, we occasionally encountered deviations such as missing brackets or commas, additional explanation or punctuation, and incorrect list lengths. Specific examples are omitted here but are available in our GitHub repository.

As shown in TableII, Qwen‑2.5‑72B exhibits the highest error rate at 62.9%, while its basic counterpart, Qwen‑2.5‑7B, produced no errors. GPT‑4.1 also demonstrates a problematic error rate of 28.9%, and GPT‑4o shows fewer errors at 7.8%. The remaining models perform reliably, displaying either zero or less than 1% error rates.

We argue that, given identical prompting and workflow across all models, these random failures reflect inherent model limitations rather than systematic flaws in our framework. While many formatting errors could be mitigated through post-processing, additional prompts, or other enhancement techniques, our objective is to evaluate, not optimize, individual model performance. Therefore, we maintain a consistent experimental design and report observed error rates without modification.

IV-E Prediction Accuracy

The prediction accuracy follows the standard evaluation approach adopted in the nuScenes prediction task, reporting the L2 loss at 1s, 2s, and 3s intervals, along with their average value. Due to response errors, each model yields predictions for a different subset of the original frames. To ensure a fair comparison, we exclude any frame from evaluation for all models if any model fails to produce a valid prediction for it. Because Qwen-2.5-72B and GPT-4.1 exhibit particularly high failure rates, we exclude these two models entirely from this analysis to retain a sufficiently large set of frames. Ultimately, this filtering results in a subset of 3506 frames out of the original 3908, preserving 90 percent of the data.

The L2 loss results are summarized in TableII. For simplicity and ease of comparison, our analysis primarily focuses on the average L2 loss (in meters); Overall, GPT-4o achieves the best performance at 1.07 m, closely followed by Claude-3.5-Sonnet and Claude-3.7-Sonnet, whose results are only slightly inferior. The Gemini models perform comparatively worse; notably, Gemini-2.5-Pro exhibits substantially poorer performance than Gemini-2.0-Flash. Overall, open-source models underperform relative to commercial ones, with the two DeepSeek models demonstrating the worst performance.

IV-F L2 Loss Baseline

While the L2 loss provides a straightforward method to evaluate model prediction performance, it may not fully capture the complexity of driving scenarios, as discussed by [33]. To mitigate this issue, we introduce a simple yet effective baseline: extending the latest AV action unchanged for the next three seconds. Trajectories generated from these constant actions are then evaluated by computing their L2 losses against the ground truth.

Our results indicate that this trivial baseline achieves an average L2 loss of 1.10 m, closely matching the best-performing VLM results from GPT-4o (1.07 m) and Claude 3.7-Sonnet (1.09 m), and significantly outperforming many other models. This comparison highlights the current limitations of zero-shot VLM approaches in trajectory planning tasks, indicating that existing models may struggle to adequately handle driving-specific complexities. Consequently, it emphasizes the need for targeted enhancements, such as designing VLM architectures specifically for driving contexts or fine-tuning models using domain-specific driving datasets.

Despite achieving sub-optimal L2 loss results in the trajectory prediction tasks, it is important to acknowledge that VLMs frequently demonstrate meaningful driving intelligence and often behave differently compared to the simple baseline—though not necessarily better. We further explore this point in the subsequent section.

Case 1: Trajectory Bias from Historical Actions

Fig.2.1 illustrates a scenario in which the ground-truth trajectory involves driving straight, yet the predicted trajectory erroneously suggests a strong right turn, failing to recognize an obstacle positioned on the right. Although initially counterintuitive, this behavior is consistently observed across all models. It occurs because the AV had just completed a right turn at an intersection immediately preceding this frame. Consequently, the historical actions reflect a pronounced curvature to the right. However, the VLMs struggle to identify the updated road conditions based solely on the current front-view image, mistakenly projecting the preceding turning behavior forward. Notably, shortly after the vehicle resumes a straight path, the models correctly adjust and begin predicting straight trajectories again. Such errors are prevalent among models and occur frequently, not only with right turns but similarly with left turns.

Case 2: Insufficient Context from Visual Cues

Fig.2.2 demonstrates another scenario in which all models consistently fail. In this case, the ground-truth trajectory involves turning left, yet all models incorrectly predict continuing straight. Although this scenario is inherently challenging—given the absence of explicit left-turn markings on the pavement or dedicated traffic lights—there are still implicit indicators available. For instance, the AV occupies the leftmost lane, whereas vehicles in the adjacent lane to the right are positioned to continue straight. To reliably overcome this issue, models could incorporate additional contextual information, such as explicit navigation instructions clearly indicating a left turn at the intersection.

Case 3 & 4: Divergent Responses to Stop Signals

Fig.2.3 illustrates a scenario highlighting notable divergences in VLM responses. In this case, the AV gradually approaches a stopped vehicle at an intersection controlled by a red traffic signal. The ground-truth trajectory demonstrates the AV smoothly and progressively decelerating until it reaches a complete stop behind the leading vehicle. However, the VLM predictions diverge into two distinct categories, neither accurately replicating the ground-truth behavior.

The first category typically includes models with relatively lower L2 loss. These models correctly identify the presence of the red traffic signal and the stopped vehicle ahead, as reflected in their scenario descriptions. Nevertheless, they predict an immediate and abrupt braking action instead of the controlled, gradual deceleration observed in reality. This behavior indicates that while these models effectively recognize critical visual cues and associate them with appropriate driving actions, they lack nuanced spatial reasoning based solely on visual input. Consequently, their responses appear event-triggered—instantly reacting to visual signals such as a red light—rather than demonstrating a comprehensive understanding of the developing scenario.

The second category comprises models generally characterized by higher L2 loss. These models inaccurately predict that the AV will continue straight through the intersection without slowing or stopping, effectively ignoring both the stationary vehicle and the red traffic signal. Such predictions reveal fundamental shortcomings in the models’ capability to interpret critical visual cues and link them appropriately to driving actions, underscoring significant opportunities for further improvement.

A similar pattern is observed in Fig.2.4. Here, the ground-truth behavior again involves the AV approaching an intersection with a red traffic signal, where a pedestrian is actively crossing. VLM predictions either anticipate an abrupt emergency stop, despite ample distance available ahead, or entirely overlook the pedestrian and traffic signal, forecasting that the AV will maintain its speed and pass through without decelerating.

Case 5: Divergent Responses to Go Signals

Fig.2.5 depicts a scenario in which the AV is initially stationary, waiting at an intersection controlled by a traffic signal. Upon the traffic signal changing from red to green, the ground-truth behavior involves the AV promptly initiating acceleration and smoothly traversing the intersection. Models exhibiting lower L2 loss closely replicate this behavior, accurately recognizing the green signal as a clear indicator to proceed and consequently predicting appropriate acceleration trajectories. Conversely, models characterized by higher L2 loss remain stationary, failing to establish the crucial link between the green signal and the corresponding action of accelerating. Their responses are indistinguishable from our simplistic constant-action baseline, highlighting their inability to effectively interpret dynamic visual cues to initiate appropriate vehicle movements.

Case 6: Conflicting Visual Cues and Model Responses

The final example, shown in Fig.2.6, presents an intriguing scenario where even models demonstrating low L2 loss exhibit differing behaviors. Analogous to the situation in Fig.2.5, the traffic signal has just transitioned from red to green. One set of models observes the green light and predicts immediate acceleration, disregarding the vehicle directly ahead. Conversely, another group of models accurately recognizes the conflicting cues—acknowledging that despite the green signal, the AV must remain stationary due to the obstructing vehicle. This scenario further extends the observations from Fig.2.5, highlighting how different VLMs respond when confronted with conflicting visual information.

Furthermore, such divergent responses from VLMs underscore the inherent instability in their decision-making processes when applied to autonomous driving tasks. These inconsistencies can lead directly to hazardous situations, such as unintended acceleration or collision risks, emphasizing the necessity for robust safety mechanisms or guardrails.