SMART: Scalable Multi-agent Real-time Simulation via Next-token Prediction (2024)

1 Introduction

In the context of autonomous driving, leveraging vectorized maps and vehicle trajectory data facilitates various motion generation tasks, including motion planning hu2023planning ; cheng2023rethinking ; huang2023gameformer ; huang2023dtpp ; cui2021lookout , motion prediction wilson2023argoverse ; ettinger2021large ; song2020pip , and Sim Agents gulino2024waymax . Previous research cui2019multimodal ; chen2023interaction ; nayakanti2023wayformer has predominantly employed encoder networks to represent driving scenes and decoder networks to generate multi-modal motions. These generated motions are then directly regressed to continuous trajectory distributions using Gaussian chai2019multipath or Laplace zhou2023qcnext mixture loss functions. While this framework demonstrates strong performance in prediction tasks that prioritize regression accuracy, it often underperforms in motion generative tasks that emphasize the safety and reasonableness of driving behavior, such as planning caesar2021NuPlan or Sim Agents montali2024waymo . The primary reasons for this underperformance are as follows: First, the framework does not represent future interactions between the motions of different agents, leading to inconsistent scene-level forecasting. Second, the model generates multi-modal motion by initializing multiple intention queries in the decoder, which is typically limited by GPU memory, resulting in a fixed number of motion modalities. Consequently, it is uncertain whether the generated modalities sufficiently represent the diversity of future behaviors. Thirdly, these models struggle to generalize across different datasets, requiring new data collection for training in new urban environments or maps.

The advent of autoregressive large language models (LLMs) floridi2020gpt ; touvron2023llama has ushered in a new era in artificial intelligence. Drawing inspiration from this, some studies in the driving motion generation domainphilion2023trajeglish ; seff2023motionlm , have tokenized agent trajectories into discrete motion tokens and employed a Next Token Prediction (NTP) task based on cross-entropy loss for autoregression. These models continue to utilize an encoder-decoder architecture, encoding continuous vectorized map and historical trajectory data with an encoder, and decoding discrete tokens solely in the decoder module. Compared to continuous distribution regression methods, the autoregressive paradigm of NTP has the following advantages: the model adopts a step-by-step next token prediction, allowing it to model interactions between agents’ motions at each time step, and the number of modalities is not limited, leading to better diversity in generative tasks.

However, existing NTP-based motion models still fail to address the aforementioned issues of generalizability and scalability, which have a critical impact on industrial applications. Generalizability means achieving satisfactory results across diverse datasets through zero-shot and few-shot learning, while scalability involves improving model performance as dataset size or model parameters increase, following scaling laws defined by hoffmann2022training . This shortfall is due to two main factors: First, current model architectures lack generalizability under the constraints of limited data scale. Due to the high cost of acquiring extensive driving data, open-source datasets typically cover only a few hundred hours of driving in specific urban areas, with significant domain gaps caused by perceptual and regional differences. Second, unlike tasks involving the serialization of a single dimension, motion generation requires the serialization of both the temporal dimension of trajectories and the spatial interactions between maps and agents. To tackle these challenges, this paper introduces the SMART model: Scalable Multi-Agent Real-Time Motion Generation via Next-token Prediction. The model incorporates a tokenizer for map data and proposes an autoregressive prediction task for the next road token prediction to enhance the model’s spatial comprehension. Subsequently, a GPT-style approach is adopted, tokenizing agent trajectories across the entire time series to establish a decoder-only transformer model. The decoder-only transformer allows SMART to compute the next token for the upcoming frame at the current moment during inference, eliminating the need to re-encode historical motion tokens with each inference, which significantly improves inference efficiency for real-time interactive autonomous driving simulation.

In summary, our contributions to the community include:(1) We propose a novel framework for motion generation, incorporating a tokenization scheme for both vectorized road and agent trajectories and utilizing a decoder-only transformer for training on the next token prediction task. This approach offers new insights into the design of motion generation algorithms for autonomous driving. (2) In the field of driving motion generation, we have pioneered a focus on the model’s zero-shot generalizability across different datasets. Notably, the model trained solely on the NuPlan dataset performed well on the WOMD test dataset, despite the lack of overlap between the map areas of these two datasets. An empirical validation of SMART models’ scalability emulates the appealing properties of large fundamental models. (3) SMART achieves state-of-the-art performance across most metrics in the generative Sim Agents challenge, ranking 1^th on the WOMD leaderboards¹¹1https://waymo.com/open/challenges/2024/sim-agents/. Furthermore, SMART’s single-frame inference time is within 15ms, meeting the real-time requirements for interactive simulation in autonomous driving.

2 Related work

3 Method

In this section, we introduce SMART, an autoregressive generative model for dynamic driving scenarios. While both language and agent motions are sequential, they differ in their representation—natural language consists of words from a finite vocabulary, whereas agent motions are continuous real-valued data. This distinction necessitates the unique design outlined in Sec.3.1 for agent motion and road vector tokenizer, including the construction of vocabulary and the tokenization of motion sequences. Sec.3.2 provides a comprehensive description of the model’s architecture. Sec.3.3 elaborates on the training tasks designed for the proposed model to learn the distribution of the motion token within the temporal sequence and the distribution of the road token within the spatial sequence.

3.1 Tokenization

Agent Motion tokenization

To apply discrete sequence modeling in continuous domains, prior works typically follow one of two approaches: either use a pre-trained tokenizer, such as VQVAE van2017neural or VQGAN esser2021taming , to encode continuous features into discrete tokens, or normalize continuous features and divide continuous values into discrete slots at equal intervals ansari2024chronos ; seff2023motionlm . For the former approach, establishing a latent vocabulary often requires a large amount of raw data to train the tokenizer; otherwise, the tokenizer itself will be biased towards the pre-training dataset. Since our work aims to enable the model to generalize effectively when trained on a small number of data samples, SMART opts to discretize explicit trajectory and map features. Specifically, similar to philion2023trajeglish , we segment the continuous trajectories of all agents in the dataset into trajectory sets by fixed time intervals $t=0.5s$. Then, we cluster the trajectory sets using the k-disks algorithm. As shown in Figure1(b), the sampled trajectories serve as our final agent motion token vocabulary $V_{a}$.

As shown in Figure1(a), the blue box represents the tokens obtained after discretizing the ground truth trajectory. At every 0.5-second interval, a search is conducted within the token vocabulary for candidate tokens, from which an appropriate(closest) token is selected to represent the current moment. Note that to prevent matching errors that may occur during the tokenization process of the agent motion sequence, we implement a rolling matching approach for the entire continuous motion sentence in a given period $T$. This implies that the token for the next time step is matched by referring to the position of the token currently matched, rather than relying on the actual correct position. However, due to the transformer decoder must perform sequential inference step by step, this approach inevitably leads to out-of-distribution issues due to compounding errorsrawte2023survey . Especially, in the field of autonomous driving, these accumulated errors may result in collisions and off-map events9636795 . To address this issue, we introduce noise into the tokenization process to enable the model to simulate distribution shifts during training. Specifically, we perturb the currently matched token by selecting one from the top-k tokens closest to the ground truth token in the vocabulary. Then, in the next time step, we match the motion token based on the perturbed vehicle state. This data augmentation method allows the model to effectively handle issues such as distribution shifts and accumulated errors, thereby enhancing robustness in generative tasks. Finally, the agent motion token is represented as $A\in\mathbb{R}^{N_{A}\times N_{T}\times F_{A}}$, where $N_{A}$ denotes the total number of agents, and $N_{T}$ represents the number of time steps, with a feature size of $F_{A}$, containing coordinates, heading, and shapes.

Road vector tokenization

To enhance the model’s generalization capabilities, we have applied a similar tokenization process to road vectors as we did with agent motion. Each road vector is a directed lane segment with features including start and end positions, length, turn direction, and other semantics from the dataset. To obtain fine-grained inputs for the road network, all road vectors are segmented into tokens spanning no longer than 5 meters in length. Unlike the motion sequence, the tokenization process of the road sentence does not have a time-series dependency. As shown in Figure1(c), the tokenization of the road sentence is performed in parallel, directly tokenizing all the original road vector segments. The road vector token is represented as $R\in\mathbb{R}^{N_{R}\times F_{R}}$, where $N_{R}$ denotes the total number of road vectors, and $F_{A}$ represents the token features.

3.2 Model Architecture

Figure2 illustrates the simple but expressive model architecture of SMART. The model comprises an encoder for road map encoding and a motion decoder that predicts a category distribution based on motion token embeddings.

RoadNet: road token encoder

We employ multi-head self-attention (MHSA) to model the relationships among road tokens, after which the updated road token encodings will assist motion token decoding. For the $i^{th}$ road token, we derive a query from its embedding $r_{i}$ and let it attend to the neighboring tokens ${r_{j}}\in R_{i}$:

$\displaystyle r_{i^{\prime}}=$

$\displaystyle\text{MHSA}\left(q(r_{i}),\right.\left.k(r_{j},\text{RPE}_{ij}),v$

(1)

where $M_{i}$ denotes the neighbor set of the road tokens. To incorporate spatial awareness for map encoding, we generate the $j^{th}$ key/value vector from the concatenation of $r_{j}$ and the relative positional embedding $\text{RPE}_{ij}$cui2023gorela .

MotionNet: factorized agent motion decoder

Prevailing methods for encoding agents prioritize capturing the temporal dynamics of an agent’s movements, followed by the integration of agent-map and agent-agent interactions, as highlighted by shi2022motion . Factorized attention effectively captures detailed agent-map interactions across temporal scales ngiam2021scene . In our work, we leverage a factorized Transformer architecture with multi-head cross-attention (MHCA) to decode complex road-agent and agent-agent relationships along the time series. Akin to query-centric methodologies zhou2023query , we utilize relative positional embeddings to differentiate between agents’ local coordinate frames, enabling symmetric encoding. Take the $i^{th}$ agent at time step $t$ as an example. Denoted as Eq.2a, given the query derived from the agent motion token’s embedding $e_{i}^{t}$, we employ temporal attention by computing the key and value based on which are the $i^{th}$ agent’s token embeddings from time step $t-\tau$ to time step $t-1$ and the corresponding relative positional embeddings.

	$$e_{i^{\prime}}=\text{MHSA}\left(q(e_{i}^{t}),k(e_{i}^{t-\tau}+\text{RPE}_{i}^{$$		(2a)
	$$e_{i^{\prime}}=\text{MHCA}\left(q(e_{i}^{t}),k(r_{j},\text{RPE}_{ij}),v(r_{j},$$		(2b)
	$$e_{i^{\prime}}=\text{MHSA}\left(q(e_{i}^{t}),k(e_{j}^{t},\text{RPE}_{ij}^{t}),$$		(2c)

Likewise, in Eq.2b and Eq.2c, the key and value for agent-map and agent-agent attention are derived from road token $r_{j},j\in N_{i}$ and agents’ motion token $e_{j}^{t},j\in N_{i}$ in the neighborhood respectively, where the neighbor set $N_{i}$ is determined by a distance threshold of 50 meters. We stack the temporal, the agent-agent, and the agent-map attention sequentially as one fusion block and repeat such blocks $K$ times.

4 Experiments

To validate the generalizability and scalability of the SMART model, we conducted extensive experiments and trained models across various scales. On the official WOMD Sim Agents Challenge (WOSAC), we employed the SMART 8 Million parameters (8M) model, which was exclusively trained on the WOMD dataset. Concurrently, the SMART 8M model was also utilized for generalization experiments and ablation studies. In the scale law experiments, we integrated additional datasets and trained on models of multiple scales. For all experiments, the testing datasets employed the split validation dataset from WOMD. Detailed hyperparameters for the SMART architecture can be found in Section A.1. In the following sections, Section 4.1 presents the results of rollouts generated by SMART on the WOSAC benchmark montali2024waymo . Evaluations of SMART’s generalizability and scalability are detailed in Sections 4.2 and 4.3, respectively. Finally, an ablation analysis of our design methods is conducted in Section 4.4.

4.2 Generalization

Zero-shot generalization on different dataset

Zero-shot generation is the ability of models to generate motions for time series from different datasets. In this work, we use the training data from NuPlan dataset to train SMART models and the test data from WOMD validation dataset.As shown in Table3, SMART^* still achieves good performance in the overall metrics. Due to significant differences in the accuracy of the calibrated ground truth values for agent position and heading between different datasets, there may be a larger gap in the agent kinematic metrics, resulting in lower scores. However, SMART^* demonstrated excellent generalization in the metrics of agent interaction and drivable map. It is worth mentioning that the size of the two datasets does not differ greatly, so the SMART model can have good generalization ability based on a small number of data training.

Method	Kinematic metrics$\uparrow$	Interactive metrics$\uparrow$	Map-based metrics$\uparrow$	minADE $\downarrow$
SMART	0.4537	0.8034	0.8514	1.5127
SMART*	0.4161	0.7853	0.7970	2.3041
SMART**	0.4310	0.8087	0.8559	1.5671

Zero-shot generalization on unseen scenarios

Multiple map scenarios as shown in Figure 3 are present only in the WOMD but not in the NuPlan dataset. Without modifications to the network architecture or tuning parameters, SMART trained only on NuPlan has achieved decent results in these scenarios, substantiating the generalization ability of SMART.

4.3 Scalability

Prior research kaplan2020scaling ; touvron2023llama have established that scaling up large language models (LLMs) leads to a predictable decrease in test loss $L$. This trend correlates with parameter counts $N$, training tokens $T$, following a power-law:

$\displaystyle log(L)=\beta log(X)+\alpha$

(4)

where $X$ can be any of $N$, $T$. The exponent $\alpha$ reflects the smoothness of power-law, and $L$ denotes the reducible loss normalized by irreducible loss. The data sources for validating scaling laws are detailed in the A.3. Overall, we trained models across four sizes, ranging from 1M to 100M parameters, on a training set containing 2.2M scenarios (or 1B motion tokens under 0.5s agent motion tokenization).

Scaling laws with model parameters

We investigate the test loss trend as the model size increases. We assessed the final test cross-entropy loss $L$ on the validation set of 100,000 traffic scenarios. The results are plotted in Figure4, where we observed a clear power-law scaling trend for Loss $L$ as a function of model size $N$. The power-law scaling laws can be expressed as:

$\displaystyle log(L)=-0.157log(X)+1.52$

(5)

These results verify the strong scalability of SMART, providing valuable insights into how model performance scales with dataset size.

5 Conclusions

In this paper, we have introduced SMART, a novel paradigm for autonomous driving motion generation that leverages vectorized map and agent trajectory data, processed through a decoder-only transformer architecture in a GPT-style framework. We have observed that SMART emulates two critical properties: scalability and zero-shot generalization, which are essential for advancing large models. We believe that our findings and the release of all codes will encourage further exploration and development of models for motion generation in the autonomous driving field, ultimately contributing to more reliable autonomous driving systems.

References

• [1]Elmira Amirloo, Amir Rasouli, Peter Lakner, Mohsen Rohani, and Jun Luo.Latentformer: Multi-agent transformer-based interaction modeling and trajectory prediction.arXiv preprint arXiv:2203.01880, 2022.
• [2]AbdulFatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, SyamaSundar Rangapuram, SebastianPineda Arango, Shubham Kapoor, etal.Chronos: Learning the language of time series.arXiv preprint arXiv:2403.07815, 2024.
• [3]Holger Caesar, Juraj Kabzan, KokSeang Tan, WhyeKit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari.nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021.
• [4]Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov.Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction.arXiv preprint arXiv:1910.05449, 2019.
• [5]Yongli Chen, Shen Li, Xiaolin Tang, Kai Yang, Dongpu Cao, and Xianke Lin.Interaction-aware decision making for autonomous vehicles.IEEE Transactions on Transportation Electrification, 2023.
• [6]Jie Cheng, Yingbing Chen, Xiaodong Mei, Bowen Yang, BoLi, and Ming Liu.Rethinking imitation-based planner for autonomous driving.arXiv preprint arXiv:2309.10443, 2023.
• [7]Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, and Raquel Urtasun.Lookout: Diverse multi-future prediction and planning for self-driving.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16107–16116, 2021.
• [8]Alexander Cui, Sergio Casas, Kelvin Wong, Simon Suo, and Raquel Urtasun.Gorela: Go relative for viewpoint-invariant motion forecasting.In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7801–7807. IEEE, 2023.
• [9]Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric.Multimodal trajectory predictions for autonomous driving using deep convolutional networks.In 2019 International Conference on Robotics and Automation (ICRA), pages 2090–2096. IEEE, 2019.
• [10]Patrick Esser, Robin Rombach, and Bjorn Ommer.Taming transformers for high-resolution image synthesis.In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
• [11]Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, CharlesR Qi, Yin Zhou, etal.Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710–9719, 2021.
• [12]Luciano Floridi and Massimo Chiriatti.Gpt-3: Its nature, scope, limits, and consequences.Minds and Machines, 30:681–694, 2020.
• [13]Junru Gu, Chen Sun, and Hang Zhao.Densetnt: End-to-end trajectory prediction from dense goal sets.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303–15312, 2021.
• [14]Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, etal.Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research.Advances in Neural Information Processing Systems, 36, 2024.
• [15]Zhiming Guo, Xing Gao, Jianlan Zhou, Xinyu Cai, and Botian Shi.Scenedm: Scene-level multi-agent trajectory generation with consistent diffusion models.arXiv preprint arXiv:2311.15736, 2023.
• [16]Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego deLas Casas, LisaAnne Hendricks, Johannes Welbl, Aidan Clark, etal.Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022.
• [17]Yihan Hu, Jiazhi Yang, LiChen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, etal.Planning-oriented autonomous driving.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023.
• [18]Zhiyu Huang, Peter Karkus, Boris Ivanovic, Yuxiao Chen, Marco Pavone, and Chen Lv.Dtpp: Differentiable joint conditional prediction and cost evaluation for tree policy planning in autonomous driving.arXiv preprint arXiv:2310.05885, 2023.
• [19]Zhiyu Huang, Haochen Liu, and Chen Lv.Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving.arXiv preprint arXiv:2303.05760, 2023.
• [20]Chiyu Jiang, Andre Cornman, Cheolho Park, Benjamin Sapp, Yin Zhou, Dragomir Anguelov, etal.Motiondiffuser: Controllable multi-agent motion prediction using diffusion.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9644–9653, 2023.
• [21]Xiaoyong Jin, Youngsuk Park, Danielle Maddix, Hao Wang, and Yuyang Wang.Domain adaptation for time series forecasting via attention sharing.In International Conference on Machine Learning, pages 10280–10297. PMLR, 2022.
• [22]Jared Kaplan, Sam McCandlish, Tom Henighan, TomB Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020.
• [23]Ilya Loshchilov and Frank Hutter.Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017.
• [24]Osama Makansi, Eddy Ilg, Ozgun Cicek, and Thomas Brox.Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7144–7153, 2019.
• [25]Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen.Finite scalar quantization: Vq-vae made simple.arXiv preprint arXiv:2309.15505, 2023.
• [26]Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, etal.The waymo open sim agents challenge.Advances in Neural Information Processing Systems, 36, 2024.
• [27]Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, KhaledS Refaat, and Benjamin Sapp.Wayformer: Motion forecasting via simple & efficient attention networks.In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2980–2987. IEEE, 2023.
• [28]Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-TienLewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, etal.Scene transformer: A unified architecture for predicting multiple agent trajectories.arXiv preprint arXiv:2106.08417, 2021.
• [29]BernardoPérez Orozco and StephenJ Roberts.Zero-shot and few-shot time series forecasting with ordinal regression recurrent neural networks.arXiv preprint arXiv:2003.12162, 2020.
• [30]Jonah Philion, XueBin Peng, and Sanja Fidler.Trajeglish: Learning the language of driving scenarios.arXiv preprint arXiv:2312.04535, 2023.
• [31]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, etal.Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019.
• [32]Vipula Rawte, Amit Sheth, and Amitava Das.A survey of hallucination in large foundation models, 2023.
• [33]Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone.Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16, pages 683–700. Springer, 2020.
• [34]Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, KhaledS Refaat, Rami Al-Rfou, and Benjamin Sapp.Motionlm: Multi-agent motion forecasting as language modeling.In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579–8590, 2023.
• [35]Shaoshuai Shi, LiJiang, Dengxin Dai, and Bernt Schiele.Motion transformer with global intention localization and local movement refinement.Advances in Neural Information Processing Systems, 35:6531–6543, 2022.
• [36]Shaoshuai Shi, LiJiang, Dengxin Dai, and Bernt Schiele.Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying.arXiv preprint arXiv:2306.17770, 2023.
• [37]Chonghao Sima, Katrin Renz, Kashyap Chitta, LiChen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li.Drivelm: Driving with graph visual question answering.arXiv preprint arXiv:2312.14150, 2023.
• [38]Haoran Song, Wenchao Ding, Yuxuan Chen, Shaojie Shen, MichaelYu Wang, and Qifeng Chen.Pip: Planning-informed trajectory prediction for autonomous driving.In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 598–614. Springer, 2020.
• [39]Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun.Trafficsim: Learning to simulate realistic multi-agent behaviors.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10400–10409, 2021.
• [40]Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao.Drivevlm: The convergence of autonomous driving and large vision-language models.arXiv preprint arXiv:2402.12289, 2024.
• [41]Luís Torgo and João Gama.Regression by classification.In Advances in Artificial Intelligence: 13th Brazilian Symposium on Artificial Intelligence, SBIA’96 Curitiba, Brazil, October 23–25, 1996 Proceedings 13, pages 51–60. Springer, 1996.
• [42]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, etal.Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023.
• [43]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, etal.Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023.
• [44]Aaron Van DenOord, Oriol Vinyals, etal.Neural discrete representation learning.Advances in neural information processing systems, 30, 2017.
• [45]Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, KhaledS Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, ChiPang Lam, Dragomir Anguelov, etal.Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction.In 2022 International Conference on Robotics and Automation (ICRA), pages 7814–7821. IEEE, 2022.
• [46]YuWang, Tiebiao Zhao, and Fan Yi.Multiverse transformer: 1st place solution for waymo open sim agents challenge 2023.arXiv preprint arXiv:2306.11868, 2023.
• [47]Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, JhonyKaesemodel Pontes, etal.Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:2301.00493, 2023.
• [48]Jiahui Yu, Yuanzhong Xu, JingYu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, BurcuKaragol Ayan, etal.Scaling autoregressive models for content-rich text-to-image generation.arXiv preprint arXiv:2206.10789, 2(3):5, 2022.
• [49]Lijun Yu, José Lezama, NiteshB. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, AlexanderG. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, DavidA. Ross, and LuJiang.Language model beats diffusion – tokenizer is key to visual generation, 2024.
• [50]Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone.Guided conditional diffusion for controllable traffic simulation.In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560–3566. IEEE, 2023.
• [51]Jinyun Zhou, Rui Wang, XuLiu, Yifei Jiang, Shu Jiang, Jiaming Tao, Jinghao Miao, and Shiyu Song.Exploring imitation learning for autonomous driving with feedback synthesizer and differentiable rasterization.In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1450–1457, 2021.
• [52]Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang.Query-centric trajectory prediction.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17863–17873, 2023.
• [53]Zikang Zhou, Zihao Wen, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang.Qcnext: A next-generation framework for joint multi-agent trajectory prediction.arXiv preprint arXiv:2306.10508, 2023.

Appendix A Appendix