Kangrui Ruan (Darren)

Applied Scientist, AWS AI Lab  |  RL for LLM Agents and Code Agents

I am an Applied Scientist at AWS AI Lab, where I work on RL and post-training for agentic LLMs, with a focus on multi-turn tool-integrated reasoning and code agents, including Kiro and Amazon Q. I received my Ph.D. from Columbia University, advised by Prof. Xuan Di and Prof. Elias Bareinboim. Before Columbia, I received my Master's degree from Carnegie Mellon University in 2020.

My research interests include RL for LLM agents, agentic post-training, RLHF/RLAIF/RLEF, tool-integrated reasoning, code generation, and causality. Representative papers are highlighted.

Email  /  CV (last updated: July 2023)  /  LinkedIn  /  Google Scholar

Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization

Training LLMs for multi-turn Tool-Integrated Reasoning (TIR) remains challenging for existing RL approaches. We propose Group Turn Policy Optimization (GTPO), a novel RL algorithm specifically designed for training LLMs on multi-turn TIR tasks. GTPO introduces turn-level reward assignment, return-based advantage estimation, and self-supervised reward shaping to densify sparse binary outcome-based rewards. Our evaluation shows that GTPO outperforms GRPO across math reasoning, commonsense reasoning, and program synthesis tasks while incurring negligible overhead.

Causal Imitation for Markov Decision Processes: A Partial Identification Approach

In this paper, we investigate robust imitation learning within the framework of canonical Markov Decision Processes (MDPs) using partial identification, allowing the agent to achieve expert performance even when the system dynamics are not uniquely determined from the confounded expert demonstrations. Specifically, we first theoretically demonstrate that when unobserved confounders (UCs) exist in an MDP, the learner is generally unable to imitate expert performance.

From Twitter to Reasoner: Understand Mobility Travel Modes and Sentiment Using Large Language Models

In this study, we introduce a novel methodological framework using LLMs to infer the mentioned travel modes from social media posts, and reason people's attitudes toward the associated travel mode, without the need for manual annotation. We compare different LLMs along with various prompting engineering methods in light of human assessment and LLM verification.

S2E: Towards an End-to-End Entity Resolution Solution from Acoustic Signal

Traditional cascading Entity Resolution (ER) pipeline suffers from propagated errors from upstream tasks. We address this issue by formulating a new end-to-end (E2E) ER problem, Signal-to-Entity (S2E), resolving query entity mentions to actionable entities in textual catalogs directly from audio queries instead of audio transcriptions in raw or parsed format.

Causal Imitation Learning via Inverse Reinforcement Learning

This paper has 2 key contributions. First, the paper analyzes structural conditions on the causal model under which learning the expert policy is possible in the presence of unobserved confounding. Second, the authors further exploit knowledge of the graphical structure to extend IRL algorithms such as GAIL or MWAL to the confounded settings.

Learning Human Driving Behaviors with Sequential Causal Imitation Learning

We develop a sequential causal template that generalizes the default MDP settings to one with Unobserved Confounders (MDPUC-HD).

Autonomous navigation at unsignalized intersections: A coupled reinforcement learning and model predictive control approach

This paper develops an integrated safety-enhanced RL and MPC framework for autonomous vehicles (AVs) to navigate unsignalized intersections.

CVLight: Decentralized Learning for Adaptive Traffic Signal Control with Connected Vehicles

This paper develops a decentralized RL scheme for multi-intersection adaptive traffic signal control (TSC), called “CVLight”, that leverages data collected from connected vehicles (CVs).