Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion.

As a solution Philip Schroeder and Thomas Weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks.

To learn more, watch Episode 111 of RoboPapers, with Michael Cho and Jiafei Duan!

Abstract

Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: this https URL

Learn More

Project page: https://philip-mit.github.io/sole-r1/

ArXiV: https://arxiv.org/abs/2603.28730

Transcript

Transcripts are automatically generated by Riverside and are lightly edited by ChatGPT.

Michael Cho [00:06]

Hello guys, welcome to another episode of RoboPapers. Today we are really happy to have Philip Schroeder and Thomas Weng, two of the co-authors of this paper, SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning. Thank you so much for making the time, guys. Maybe as a start, Philip, you can share a bit about yourself and then Thomas.

Philip Schroeder [00:27]

Yeah, so my name is Philip. I’m a final-year PhD student at MIT in computer science. My work has mainly focused on improving the reasoning capabilities of language and vision-language models in challenging embodied settings. And over the last couple years, my work has mainly focused on reward reasoning over video, which is what we do for this paper.

Thomas Weng [00:55]

Cool. I’m Thomas. I’m a research scientist at RAInstitute. And more recently my work has been focusing on the pieces required to achieve like post-training and deployment for robotics. So reward modeling is one piece of that. On-robot reinforcement learning where you have data from deployment is also a piece of that. But that’s what I do.

Michael Cho [01:22]

All right, great to have both of you on. Yeah, maybe Philip you can take us through.

Philip Schroeder [01:27]

Yeah, yeah. Okay. So the goal of this paper was to develop a vision-language model that has strong enough reward reasoning that we can train policies from scratch through online reinforcement learning. So the model we developed is called SOLE-R1. And we show that it can serve as the sole reward signal for learning over 20 different unseen manipulation tasks through online RL. And I guess the motivating problem for this paper was that if you use just general vision-language models or VLM-based reward models that have recently come out, you get this reward hacking behavior when you try to train a policy from scratch using the predicted rewards.

So here, the task is to open the drawer, and you can see that these VLM-based reward models give very high rewards just for the policy fiddling with the drawer and just slightly opening it. And then here’s another example with the Pick Cube task in RoboSuite. Again, you see that the policy is getting these very high rewards by just finding these odd camerangles and positions that trick the VLM into thinking the task is being accomplished without actually doing the task. And you can see here the SOLE-R1 predicted rewards are plotted here in blue, and the ground-truth rewards are plotted in red. And you can see that solar SOLE-R1. Much more robust to this type of reward hacking.

And you kind of think of this paper as a recipe for how to train a reward model that is robust to reward hacking. And there’s kind of two major components to this that differentiate our work from prior work and really help with this reward hacking robustness. And I can quickly mention those and then go over them in more detail later. So the first is our training data mixture. We do I think three unique things with our training data mixture. One is that we include a lot of general spatial reasoning data. And a lot of this data is new. It came out with NeurIPS 2025 and ICLR 2026 papers. So at the time of this work it was data sets that were just released.

So it includes just general spatial re spatial reasoning data SSR chain-of-thought And some robotics data sets, some recent one called From Seeing to Doing, and some very famous ones like Embody chain-of-thought and Robo2VLM. And we combine this with data that forces the model to reason over multiple images within the same space, capturing different moments in time. So this includes general spatial reasoning data or spatiotemporal reasoning data spot the diff and robot-specific video data that we generate and then. Also curate from Open X-Embodiment. And our video data includes failure datand I think much more failure data than prior work because we generate a lot of it ourselves in RoboCasa. So that’s one big thing.

And then another big thing is that we annotate the time each time step of the videos with dense progress values and dense chain-of-thought reasoning traces to help the model explicitly reason about progress with these failure trajectories or suboptimal trajectories. And then we mix this with these artificial failure data from Open X So that’s the training data mixture. And then another novel thing we do is have the model do this chain-of-thought reasoning at each time step of the video. And this has been shown to significantly improve language model performance and visual language model performance. So and we show ablations where it helps significantly.

So the models con it at each time step is reasoning about what’s changing over time between the current and previous time step and predicting a dense progress value. Relative to the previous progress value that it predicted at the last time step. And then we do a lot of experiments that I can that I can show. But let me know if you have any questions so far. I can go over in more detail the both how we generate our training datand then how we train this chain-of-thought reasoning over video.

Jiafei Duan [05:56]

Can you let me yeah. Yeah, I’m gonna go ahead. Yeah, no, you go ahead.

Michael Cho [05:56]

Yeah, let’s go over the training data. Sorry, Jiafei. No, I was gonnask about the training data. I mean it’s a nontrivial amount of training data. It’s curious how you get that, like the dense annotation. Jiafei whether that’s your question.

Philip Schroeder [06:10]

Yeah, so we w most of the novel data we’re generating is in sim. So we’re leveraging the ground-truth simulator state at each moment. And I have some slides for sh walking through that and I can I can show that next. Jiafei if you have a question, I can answer that before I show that.

Jiafei Duan [06:26]

Yeah, so maybe like I’m curious about the reward hacking, right? So you showed the first example, where I guess it’s like it’s non-trivial in the sense whereby you are trying to showcase a task where it looks like it’s grasping, but when the from a different angle it might look slightly different. Is that my understanding? Okay. So is it a perception problem or like do you think it’s like the a result of perception that leads to reward hacking or you know?

Philip Schroeder [06:31]

Yeah. Yes. Yeah, so it

Jiafei Duan [06:55]

or what do you think is the you know, how do we quantify this? Yeah.

Philip Schroeder [06:59]

Yeah, so it’s definitely a perceptual failure. Where the policy is just finding out-of-distribution visual input by just ‘cause it has so many degrees of freedom with moving the camera into any position. And since we’re already a little bit out-of-distribution with this task setup because there’s probably not a lot of rope like the robotics data inside video data inside the training data for these visual language models. But especially when you give the policy this opportunity to optimize against the VLM’s sense of progress. And it’s always easier to just find these weird camerangles than it is to just do the task. It’s always easier to get these false positive states than to get the true positive.

Jiafei Duan [07:41]

I see. I see. Maybe my question would be like humans also have this, right? Like in the sense whereby it’s like you know, if you look at things from a certain angle, you have an illusion that, you know, certain things has been done correctly, or it might not be the case. Like, is this like is this a would you say this like something that you have to fix to ensure that you know you are able to train a real model that is, you know, able to really accurately sort of predict progress regardless of this kind of, you know, camera positioning? Yeah.

Philip Schroeder [08:15]

Yeah, I mean, I think, for example, this time step right here, we want models that can look at this. I think any I think almost any human that watches this video will see clearly that the robot is not picking it up. And to the extent that we want our models to have this human-like reasoning capability visually, I think we want to fix this problem. If that’s I don’t I don’t know. I mean, there might be ways of using the model that help it help it avoid these scenarios. Like for example, we start with a randomly initialized policy, so we give it like this full capability to explore and explothese weaknesses in the model. But I think I don’t know if that answers your question, but I think fundamental I think this something fundamental to the model.

It’s not just some long tail of tiny narrow failure cases. It’s there are very obvious examples where it’s hallucinating success when it’s given the full opportunity to recognize that success is not being achieved. Does that answer your question? I don’t know if I’ve answered your question.

Jiafei Duan [09:24]

Yeah, yeah. Yeah, no, yeah, I get it, yeah, yeah. Thanks.

Thomas Weng [09:28]

I think to add on to that, it’s also the fact that we’re in this like pretty difficult regime of reinforcement learning, like specifically online RL, where we are acting as kind of the sole reward signal for this model. And in that situation, if you have any kinds of false positives then the policy will learn to exploit that pretty quickly. So that’s part of it. But I think in general with any kind of

Jiafei Duan [09:41]

Mm.

Thomas Weng [09:58]

reward harness if you don’t address these issues of like overestimation or overconfidence, then you can quickly run into some pretty bad scenarios.

Jiafei Duan [10:07]

I see, I see. Yeah. I guess maybe the reason I wasking this that I’ve been also thinking a lot about reward modeling from norm like visual perception. Like if you add in a tactile sensor or you add in some kind of force feedback, then you could easily tell for sure that the robot is not touching the cube or that it’s not grasping. So then you might not be able to see, but the other signal could inform you of the, you know, progress of the of the reward that’s tracking. So I’m just saying that, you know, in no like thinking forward, you know, w would this be like is vision the only thing that o that we sort of rely on or do we look at other modalities that, you know, potentially the way that reward hacking could look like could also be different given the kind of census that you give.

Yeah.

Philip Schroeder [10:50]

yeah, it’s a great point. I we considered at the beginning of this work whether we should just have like one scalar value representing the contact the gripper is making with the different objects it’s trying to pick up or manipulate. And I agree that there are a lot of scenarios where that would help. I think the policy would still, I think, find ways of just triggering that sensor and still not doing the task. It would make the so I think it would make the reward hacking job harder for the policy. But I think it would like for example here, you could see that even if it’s like it could start to grip the cube here and trigger some sort of sensor that’s as if it’s picking it up, but just never lift it off the table. But you’re right, it does make the reward hacking harder. The more the more

Jiafei Duan [11:21]

Mm. Mm. Makes sense, yeah.

Philip Schroeder [11:46]

The more datand the and there’s other things you do with reward hacking as well. Like for example, you can change the you can constantly change the camerangle. Like for example, the external camerangle here, you can constantly flip sides of the table or orient it differently. So the could so the policy never knows what external camera view the VLM is watching it from. So it can’t it can’t like re find reliable paths to exploiting any visual shortcoming of the VLM.

Jiafei Duan [12:14]

Mm.

Philip Schroeder [12:14]

And it’s like we do that, for example, with the ManiSkill setup. It was all really the only way we could get things to work. We had a flip between alternating sides of the of the table. But yeah, so I think yeah, any extra information or changing information or sources of information will make it harder to do the reward hacking. But fundamentally, I mean like if you if you get an image and a human can watch or get a video and a human can watch it and say, okay, clearly not success here.

Jiafei Duan [12:23]

Hm.

Philip Schroeder [12:42]

And but the VLMs reliably predict success. I think we want to solve that problem from the visual side. If as long as humans can do it from the visual side, I guess. I g I guess that’s the bar.

Jiafei Duan [12:53]

Makes yeah, yeah. Makes sense.

Thomas Weng [12:56]

I think I mean vision alone is I mean we have like all the bootstrapped like capabilities off of VLMs to be able to use vision. But I also kind of think that if we had other proprioception or perceptual information that could be used to prevent reward hacking, I don’t think we’re necessarily opposed to that. It’s just that you know, if you have a tactile sensor that does help you prevent reward hacking, then that’s great. But it does need to probably vision will still be part of this picture.

Philip Schroeder [13:32]

Yeah, and a nice part about vision is it’s agnostic to the embodiment. And you can also like integrate by manual, et cetera. So it you don’t have to like retrain it on a new set of joints or two versus one arm or it or any other changes you make. So that does make it a little bit more general. And if you a lot I think a lot of people might not want to have to send extra sensor. I don’t know, it depends on what the data is, but It’s nice being able to just send videos if you have like a big data set of videos. So yeah. But To answer Michael’s question, I can walk through the data generation in more detail. And see if you guys have any questions about the actual steps that we take.

Okay, so again, this our overview of our training data mixture that I showed earlier. We have this general spatial reasoning data where the model’s just reasoning over single image and Producing this chain-of-thought reasoning along with final answers to questions that talk about spatial relationships. And then we add these robotics data sets, like I mentioned. And then we also have this data that requires the model to reason over multiple images or multiple time steps of a video representing the same space at different moments in time. So we have this general data from spot the diff, and then our curated data from both RoboCasand Open X-Embodiment.

And I will walk through how we generate the non-expert trajectories inside RoboCasa now, along with the dense progress estimates and the per time step chain-of-thought. Okay, so I think most of us are familiar with this the source of our data, which is RoboCasa. I think most of us are familiar with it. It’s great resource from UT Austin, and I think they have updated versions of this now, but we’re using just the original RoboCasa where they release thousands of demonstrations, of expert demonstrations of many different manipulation tasks inside randomized kitchen scenes that they generate. And the demonstrations, although there’s multiple robots shown here, the demonstrations are all provided with the Franca. And so here’s some examples.

So these are the single-stage tasks or the atomic tasks. that they that they categorize. And so it ranges from picking up and placing things to opening and closing drawers to turning on the stove to turning on the sink. And then here are some multi-stage tasks like picking up cans and putting them inside the pantry and then taking the coffee mug outside the cabinet and putting inside the coffee machine and pressing the start button. Okay, so RoboCasa provides a this great resource of diverse Expert demonstrations across many different manipulation tasks. And the question is, how do we get the non-expert trajectories from these expert trajectories? So the task example I’ll show here is picking up the salt shaker, and the expert states are shown in blue.

So we get non-expert trajectories by inserting random actions inside these trajectories and then recovering and then we can still recover to the expert trajectory through just linear state interpolation between the deviate the final deviated state and a future expert state. And you have to be a little bit careful with how you do this so you don’t get like floating objects or the gripper moving through the cabinet. But if you’re careful you can generate pretty realistic suboptimal trajectories or just total failures by injecting these deviations. And because we know the simulator state at each moment, we can measure the distance between the robot and different goal states throughout the trajectory.

So here we would measure the distance between the gripper and the salt shaker at each time step. And when we do that, we can get these very reliable progress estimates that we use to ultimately train our vision-language model. And then in addition to These progress estimates, we have all of this ground-truth information that we can translate into just natural language statements about the about each time step. And we leverage this to generate chain-of-thought reasoning traces that are explaining kind of why the progress is what it is at each time step. And the final training data looks something like this. So these are two videos from the actual training mixture. We change the resolution a lot and we rotate the camera on the wrist to help just build robustness of our model.

And we change the lighting, et cetera. But you can see the progress values that are plotted on the right. And the tasks here are just to pick up this object and put it in the cabinet. I think the one on the top fails to pick it up, and the one on the bottom picks it up but then fails to place it. And you can see the chain-of-thought reasoning traces we generate. So that the I’ll play this back again one more time. So the chain-of-thought reasoning traces come from just pure templated language mixed with templated language that we enrich with VLMs that see the time steps and see the templated language and then just add more characterization of the scene of the movements Without deviating from the ground-truth information that’s in the templated language.

So I think this probably the most important thing that leads to our model performance is generating these suboptimal trajectories with very good progress estimates and then the reliable chain-of-thought reasoning. So definitely let me know if you have any questions about this, and I’m happy to go back to any previous slides.

Michael Cho [19:40]

Yeah, Philip, I’m curious, like I guess there’s a concept of memory here, right? Like, you know, how ha have you done an application studies or like h how many frames or how often do you take the frames? Is it like when is it too much for the VLM? Yeah. When you do this CoT. Mm.

Philip Schroeder [19:58]

Yeah, so yeah, I that’s a great question. So we have a very simple solution to this, which is we just our all the tasks that we care about are short enough that we just include the first, previous, and current time steps in the context window of our VLM. And that makes it really fast for inference and our inference scales cost scales linearly with the video length because of that. But I think you can play with this a lot. You could do some sort of dynamic frame pruning or dynamic like dropout of frames depending on what the structure of the task. Or you can just try to concatenate like 10 plus frames to get a bigger picture of what’s going on if you have longer horizon tasks. I think our framework’s pretty agnostic to this.

We just chose, I think pretty simple like three frame three time step framework because our we know our tasks aren’t very long. And we show in the mod so the model gets a total of six frames because it gets three time steps and then two camera views. So it’s getting the wrist and the external view. But yeah I think memory is a really important thing and I think The easiest solution is just increasing the number of frames you include in the context. It’ll slow down the model, but there’s other ways to speed it up.

Michael Cho [21:30]

Got it. Yeah. And am I right that in a way, you know, your templated text here is in a way capturing like the memory. It’s just in text form. As you cycle through

Philip Schroeder [21:42]

Yeah, I mean Yeah, so I think most of the template language or most of the yeah, most of the chain-of-thought language is talking about the previous time step relative to the current time step. And it can infer a lot about the trajectory just based on the or I based on these time steps. And then it’s also it also sees the very first time step of the trajectory, so it sees the starting position of the robot. But yeah, in some sense I think you could leverage if you didn’t want to can ca continue to just concatenate frames which are expensive. You could somehow be summarizing along the way in text and just keeping that in context. But our model doesn’t see the chain-of-thought reasoning from the previous time step. It only sees the progress predicted the previous time step.

So all of this text that it’s generating at each time step is essentially forgotten at the next one. But I think there’s probably a way to leverage the text that it’s generating along the way. To serve as a some form of memory that’s far more compressed than trying to just concatenate a lot of frames together.

Michael Cho [22:55]

Got it. Yeah. I’m also actually also curious, I don’t know whether this the right time to ask. It’s for let’s say slightly longer horizon tasks where maybe the way to get to the final outcome is you need to do a little bit of detour. You know, it could be one of those like okay, you it you have to do a little bit of detour. So I wonder how ha have you have you done those kind of tasks where and then how does it square with the task progress?

Philip Schroeder [23:14]

Yeah, yeah. Yeah,

Michael Cho [23:26]

You know, maybe temporary it’s the only way to solve the final task is you need to do a big detour in yeah.

Philip Schroeder [23:32]

Yes. Yeah, so yes, this a common problem, especially when the tasks get more complicated. You can’t always just use like Cartesian distance as a progress proxy because you’re right. Sometimes you have to avoid obstacles and you have to temporarily deviate from going to the target location in order to avoid some midway collision. And there are so I essentially just handle that by hand in our data generation. Like I know all these like edge cases where there’s a pick and place task in RoboCasa where it has to maybe like work around the sink or it has to go from below the cabinet to the top of it, but it temporarily has to move away from the final target position.

And I just flatten out the progress values at those points, but this a very like hand-engineered solution that doesn’t scale. So I think if you just allowed that type of imperfection in the progress values, the VLM would probably still perform well because it’s still going to get points as soon as the object gets closer to the target position after avoiding the obstacle. But it might really affect policy learning where the because the policies getting penalized for avoiding a collision with some middle thing, it never really learns to get out of that and place low place the actual object.

Yeah, so I mean I guess I think this why, yeah, I think ultimately we want a VLM to be able to reason about okay, you want to include videos where there are examples of having to avoid an obstacle. and the VLM would be reasoning about, it’s moving further from the target position, but obviously it’s trying to avoid this obstacle, so reward should go up. You want that. I mean, you want this human-like rational reasoning. But I don’t, yeah. For our data, I just hand I hand engineer solutions to all these specific cases where this might happen. And that was good enough because it’s a small enough task set, but yeah, it’s a really important problem.

Thomas Weng [25:52]

I guess specifically for our data, we looking at the datand the simulator like information coming out of RoboCasa, I think we borrowed some from our previous work of ROVER, where were using essentially like the Cartesian distance information in like 3D geometry to try to determine our task progress indications. And so

Jiafei Duan [25:52]

Right.

Thomas Weng [26:19]

For the scale that were working at, like it was fine to kind of go in and make sure that those were not a problem. But if you didn’t use that kind of ground-truth metric of the Cartesian distance, then you wouldn’t necessarily have these have to deal with this like artifact. Now it does mean maybe you have to do some more intense reasoning or like maybe you start using some kind of motion planning or collision optimization in order to pre process your data. And I think that’s what Philip is getting at when he’s saying like, you know, right now we basically are dealing with the fact that we know our heuristic is Cartesian distance between like the object’s current location and the goal location and things like that. But ultimately we can move away from that kind of heuristic.

Jiafei Duan [27:11]

I have a question. I think more high level and philosophical. So I think like one thing, you know, we start to realize with all these frontier companies that companies that who really trains pretty good, like you know, I mean ChatGPT, you know, version of ChatGPT tends to do RL better. And smaller models like those companies that do smaller models struggles a lot with this because there’s a missing component where you’re not able to have user feedback. Okay, you’re not able to have that kind of

Michael Cho [27:12]

Yeah, makes sense. Yep.

Jiafei Duan [27:41]

real-time val viable reward in that sense. And you guys are generating this trajectory or discount reward using simulation. I think robot meter does differently. Like what do you think is the most ideal scenario to get some of this annotation? Will it be like if you have a massive deployment of real-world robots and you know consistent evaluation and people are able to sort of collect evaluation rollouts, get it labeled pretty well, get all the human verifiable you know, reward out of that. That would be like the perfect scenario to be used to train a reward model. Like do you guys see that as the way, or do you see like scanning up simulation or the robot meter approach? Yeah. Red philosophy can you do the paper, just a r yeah

Philip Schroeder [28:25]

Yeah it’s Yeah, it’s a good question. And we also include Open X-Embodiment data to mix in with this. And we’d use the RoboMeter approach of just shuffling frames and using time step as a proxy for progress with the like creating essentially these artificial failure data. But yeah, so I mean I think one of the questions you asked during the RoboMeter episode was do you need any robot datat all to train a perfect reward model? Or a human-level reward prediction model. And the answer to that question might be no. If you just get a good enough sense of the physical world through other means, I think you could do good enough zero-shot reward prediction better than any model we have so far. And I think maybe, I don’t know, maybe GPT-6 Astra. Is almost there.

And who knows how much robot data they have in their training, but probab they probably don’t have data this and it probably does very good job of generalizable progress prediction. But yeah I think to answer your but to answer your question I think the more reliable path I mean so yes we might not need robot datat all. That I mean that could be long term the actual final outcome. But in the near term I think The brute force approach of just scaling up real-world data collection and then just getting human annotators to label videos with very reliable progress values. I think and I and you don’t need nearly as many videos to train a good reward model as you do to train a good policy. So it I think this actually is feasible.

And as long as you get a diverse enough task and set of trajectories that represent true failures, I think that brute force approach could actually work. Work in the near term and produce either a human level or a superhuman level reward prediction model.

Michael Cho [30:36]

Yeah, I actually like on a side note we just did an episode a couple of days ago with guys from Empire. I don’t know whether you guys yeah, so and they during the episode they updated the whole thing with Astrand obviously Astra is phenomenal. And so I suppose the closer you can get like some of these like just like huge foundational model that can help you with reset, then you can actually have really truly explore a lot of play and then just get the reward that way. They actually just ask I think us ChatGPT to figure out what’s the reward. And then remove the human in the loop. It was quite crazy. Yeah.

Philip Schroeder [31:21]

Yeah, I think yeah, so you’re right. I think ultimately we will no matter how good Astra gets, we will always want to distill a smaller model that is less expensive and just more suitable for as a final form model for robotics. And I think the question then becomes how do you do this distillation? And I think in some ways this our paper is showing how you can do distillation in a way that builds robustness to reward hacking. So it would be nice to maybe use Astra but still integrate this authentic failure datand other data that you find and annotate it with Astra’s reasoning traces and then distill from that.

Because I think in the long term I think I mean I’m sure OpenAI will work on creating faster, cheaper versions of Astra’s with Astra level reasoning and provide an API that can maybe be used for robotics. But at least in its current version, it’s just too expensive, too slow.

Michael Cho [32:17]

Yeah. Mm. Mm.

Philip Schroeder [32:27]

to be a realistic I think for deployed robots or online learning of that requires like fifty thousand plus rollouts to train a model or a fleet of models. So but yeah I mean I think Yeah, I think Astra is definitely a step up of I and I wish we could have included Astra in the baselines for this. We definitely would have if were doing this today. But it’s amazing how big of a step up it is relative to the previous models.

Michael Cho [33:01]

I’m curious like you know, d this failure trajectory that you guys like generate, like how what is the proportion relative like ‘cause in theory you can generate like le almost like an unlimited amount of failure trajectories so called. Like how much do you go into it or do you just maybe from every successful you only have one failure or like you have like one hundred other failure trajectories?

Philip Schroeder [33:26]

Yeah, so we’re lim we’re only limited well I mean unless we want to do extra work, we’re initially limited by the number of tasks settings that RoboCasa release. So we so RoboCasa initially released a little over 20 different manipulation tasks. And had I think 3,000 trajectory, expert trajectories per task. And the vast majority of those are generated with MimicGen starting from a source data set of human teleoperated examples. And then we generate, I think, a total of 40,000 videos from RoboCasacross these different tasks. But yes, I mean one version of this could be to just use Astra.

Michael Cho [33:43]

Well just.

Philip Schroeder [34:11]

to generate diverse sim environments. And then you might have to manually collect demonstrations inside those environments or use Astra to collect demonstrations potentially in those environments. And then generate the failure data off of those expert trajectories. So there’s a lot of creativity potentially in the space to really scale up the diversity of the in the in the in the number of sim environments you create. But yeah, we kind of just stuck with the with the constraints of the original source data RoboCasa provided, which gave us about 40,000 videos. And then we take about 200,000 videos from the Open X-Embodiment data set.

So we have about a quarter million robot videos in our training mixture and total and all of them have these dense chain-of-thought reasoning traces and dense progress estimates.

Michael Cho [35:03]

Got it. But b but the I still wanna get a sense of the scale of the so called failure trajectory that you guys so w it’s it

Philip Schroeder [35:11]

So I guess what do you mean by so the failure almost all of the tr so I think about ninety percent of the RoboCasa trajectories, which is so about thirty five thousand, thirty-six thousand are all suboptimal trajectories. Does that answer your question?

Michael Cho [35:29]

I see. Okay. I so is something like s something like w ‘cause I mean you made the point earlier that di this failure trajectory really helped, right? So I that’s why I wanna get a sense of what is the relative scale to the so-called ‘cause you start what with like the successful tele op trajectory first, right? And then you guys generate some so called suboptimal trajectory. I’m you know, so I’m just curious how many of

Philip Schroeder [35:51]

Yes.

Michael Cho [35:59]

those suboptimal so called failure, maybe not entire failure, but suboptimal trajectories that you that you manage to do.

Philip Schroeder [36:07]

Yeah, so for the vast majority of the RoboCasa trajectories, we’re not actually including the expert trajectory inside the training data. We’re just converting that into a suboptimal trajectory. And that’s about ninety percent of them. So we could we maintain about five thousand out of the forty thousand trajectories as the true expert trajectories. And then the rest are these different levels of deviation from that trajectory. So ranging all the way from like it just barely doesn’t accomplish the task to it doesn’t even approach the object, for example. And it just outputs random actions. This I yeah, I okay, gotcha.

Michael Cho [36:21]

Mm. Mm. Got it. Yeah. Yeah.

Thomas Weng [36:49]

I guess kind of returning to the high level like questions. There’s this question of like, do you only need like a data flywheel of real data that is maybe suboptimal from human teleoperation or from policy deployments on robots that are like getting not perfect success rates? Or do you actually want something like a simulator where you can extract certain features of ground-truth and use those to like I don’t know set up a template like what we have here of like making sure we know what the ground-truth state is now or maybe with the memory that we had in the past, et cetera, et cetera. Or is it just that you only need kind of real-world datand are you inducing more problems by taking on this like all the complications of trying to Create a simulator for individual tasks.

I think in this work we definitely make good use of simulation. And it is possible that like bigger and better models in the future will be able to help us create better real-to-sim or digital twin type situations. That’s getting really popular in the field now as well. So yeah, I think it’s kind of I just wanted to flag that as like an open question. Like, We’re finding cooler and cooler and more effective uses of simulators. But is it necessary or do you actually want a data flywheel from robots interacting in the real-world and being deployed? Yeah.

I think in either case, this paper is really contributing kind of again the recipe of how to avoid reward hacking so that you can get reasonably good pr task progress estimates that are better distilled than say just Directly querying some large model.

Philip Schroeder [38:43]

Yeah, I mean, I think the most obvious answer would be you would want both. And they do complement each other nicely. Like for example, with our data, you could you could say that the sim data has lower task diversity, but high trajectory diversity by showing these f different failure types. And then the Open X-Embodiment data has much higher task diversity, but much lower trajectory diversity. And then when you combine them, you get kind of the best of both worlds. I think so I think unless we get to a point where sim is indistinguishable from the real-world graphics or visuals, I think we’re always gonna benefit from the diversity you get from real-world. Even though how pa even though it’s extremely painful to collect the real-world data relative to sim.

But yeah, it’s a good I think question. Okay, I can go on to show the training of the reasoning. If unless you guys have more questions about the let me know if you have any more questions about the training data generation. Okay. Okay, so and just briefly to close on this, we mix up this RoboCasa data generation with the Open X-Embodiment data that I mentioned. And yeah, just like RoboMeter, we’re doing this artificial failure generation with shuffling the frames and then using time step as a proxy. But this just really like playing the video forward and backward for the model and pretending like that’s a failure day trajectory. And we know that’s not really reflective of true failure behavior. But yeah, okay. So that’s an overview of our training data mixture.

So with this tr with this with this training data, we train the model to do this chain-of-thought reasoning at each time step, where the input of the model is just the task description and the video. And at each time step, it’s doing this reasoning about what’s changing over time between the previous and current time step. And then it outputs this progress value relative to the previous progress value that it predicted. And this progress value is directly used as a reward during online RL for learning different manipulation tasks.

And to train this reason, we have this two-stage approach, which is becoming quite common, especially after the DeepSeek paper in early 2025, where we first do supervised fine-tuning, which is just the standard cross-entropy loss with next token prediction, with that’s common with language models, and then we follow that up with a Reinforcement learning stage where we’re training the model to specifically predict these progress values. So the loss for the supervised fine-tuning is dominated by the tokens inside the thinking trace, and only a small part of the signal is actually coming from the final answer. And then reinforcement learning is kind of the opposite, where all the tokens inside the reasoning trace are rewarded and are rewarded entirely based on.

How close the answer, the final answer is to the ground-truth answer. So they complement each other nicely. So the model builds good space good spatiotemporal chain-of-thought during SFT, and then it leverages this good chain-of-thought to get increasingly better rewards during RL for predicting the progress values that are in the training data. And for our model, SFT takes about a week, and then RL takes another couple of days. So just to give you a sense of how long it’s taking to train on all this data. Okay. So we test on we evaluate on about on forty total tasks, and we include a real-world tabletop setting along with four different sim environments. And for our main results, like I like I mentioned, we’re starting from a randomly initialized policy.

So and so all the tasks are quite simple. And I think I think Most people who have tried starting from randomly initialized policies know that it can be even hard to learn from the ground-truth reward, even very simple tasks. So eventually, like I think future versions of this work would scale up task complexity. And we do some higher complexity tasks with a pre-trained VLA, which we’ve done in more recent experiments. But all our main experiments are super simple, single-step tasks, pretty much. But there’s tremendous diversity. in the actual task setting and even the embodiment in some cases and camerangles, et cetera. And are we don’t do any fine-tuning of our model, of our reward model on any demonstrations of the tasks. We don’t use any sparse rewards from the simulator state.

There’s it’s all zero-shot and the behavior is totally emergent from the reasoning of the VLM. So that’s the nice part about stripping everything away is that it’s the purest way of testing. Arguably the purest way of testing the reasoning capabilities of your VLM because there’s no supplemental aids to compensate for any weaknesses that it has. Okay, so this a bar chart showing the success rates across the 40 tasks, across the five different settings. SOLE-R1 is shown in red. The we include all of the VLM-based reward models as baselines, including RoboMeter, RoboReward, Top Reward, and then some other reward models like Rewind, VLAC, Live. And the strongest baselines are just the general purpose VLMs like GPT-5 and Gemini.

And SOLE-R1 accomplishes at least 50% success rate on 24 of the 40 tasks. The strongest baseline, which is GPT-5, gets over 50% success rate on seven tasks. So there’s a significant gap here between what SOLE-R1 is accomplishing and the baselines, which is really nice to see. Do you guys have let me know if you have any questions about this plot. I can move on to the next figure. Okay.

Michael Cho [45:26]

Yeah, I’m trying to eyeball the tasks, yeah. Looks like across the board, like you guys beat everyone. Yeah.

Jiafei Duan [45:30]

I’m trying to I’m trying to read the performance it’s because there’s too much bar here. Okay.

Michael Cho [45:39]

No, but I it’s if I’m not wrong, like basically every single task, like s so our one is just beating everyone. Just say i is there any task that is not even the case. The door open is surprisingly really, really low though, like for everyone. It’s it door open? Mm-hmm.

Philip Schroeder [45:55]

Yeah. Yeah. So in Meta-World, yeah. Yeah. It’s cause you have to yeah, you have to I think the door looks a little bit weird. It doesn’t look like a standard I can show I think it’s this I don’t know if you can see my mouse? It’s this scene right here. It’s a weird looking door and I think the VLMs don’t recognize that this the handle. And that’s what you have to get the gripper in to then pull it open.

Michael Cho [46:12]

Mm-hmm. Yep. Mm-hmm. Mm.

Philip Schroeder [46:24]

so I think that’s the it’s just so Meta-World it just looks weird. I think it confuses the VLMs very frequently. And I think I don’t know, I these experiments were super expensive to run. I think we spent like something like five to ten thousand dollars on API cost for OpenAI alone just to get these just to get these results for RL the RL runs.

Michael Cho [46:25]

it’s okay. Mm-hmm. Yeah. Mm-hmm.

Philip Schroeder [46:54]

Gemini is much cheaper. And the and the runs take like five days a piece, because the APIs are very slow. ‘cause we have to send every single episode that the agent collects to OpenAI and then getting it densely annotated with GPT. So it’s a little bit insane to do this. But we really want it to have I think these open Th these general purpose VLMs as baselines ‘cause I think they are probably the strongest baselines for this type of zero-shot reasoning.

Michael Cho [47:32]

Yeah. Actually these labs should pay you, man. You’re helping them benchmark things. So very interesting. Mm-hmm. Mm-hmm.

Philip Schroeder [47:35]

Yeah, I wonder how much I don’t know if they do any of this stuff internally. But it would be interesting if they did. Okay. So we dig a little bit deeper into the results by plotting the what we call the true success versus the perceived success of each task. And the true success here is the average ground-truth reward that the policy achieves on the task.

Michael Cho [47:47]

Okay.

Philip Schroeder [48:05]

And the perceived success is the average predicted reward that the VLM predicts for the task. And this bottom right quadrant is what we call these reward hacking failures, where the policy is getting the model Yeah. So yeah, the policy is getting the model to think that the task is being accomplished and predict high rewards, but without actually doing the task.

Michael Cho [48:19]

Mm. False positive.

Philip Schroeder [48:34]

And this the same task setup that I was showing earlier where the policy is just finding these weird camerangles in RoboSuite. And we provide more examples like this across different tasks in the appendix of paper. And we see that I think the major models that are susceptible here are the general VLMs, which is Gemini and GPT, and then you see some yellow dots here for RoboMeter. And then a lot of reward models just fall in this bottom left quadrant, where which we call these signal limited failures, where the policy is failing at the task, but the VLM is at least aware that it’s failing. It but it’s not able to predict rewards that are shaped well enough to get the policy to actually learn the task. So we it’s a little bit less clear what’s happening here than what’s happening here.

Michael Cho [49:19]

Mm-hmm.

Philip Schroeder [49:31]

But we just generally categorize these as signal limited failures where the p the VLM’s just not providing a strong enough signal for the policy to learn the task. And then we do some ablation. So this the same figure that I was showing just before. But now instead of plotting the different baselines, we plot the different ablations of SOLE-R1. So if we remove the non-expert videos or we remove the general spatial reasoning data from the data mixture, we see this movement towards the bottom right quadrant where we get more of these reward hacking failures.

And then if you remove the reasoning that happens before the final progress prediction, it doesn’t move the results towards reward hacking, but it but it does move things closer to the lower left quadrant of these signal limited failures. So it’d be nice. I think potentially I if you really scale up the data, maybe the thinking becomes less and less critical. And you potentially could achieve a larger scale of data if you didn’t have to worry about generating the chain-of-thought reasoning traces. But at least with our scale of data, there’s a significant benefit to ha including these chain-of-thought reasoning traces in the training data mixture, as opposed to just having the progress estimates without the reasoning beforehand.

Michael Cho [50:55]

Mm.

Thomas Weng [50:56]

I guess to Michael’s earlier question about how important was the suboptimal data, the yellow plot kinda gives an indication of like what happens if you only use success or successful expert trajectories.

Michael Cho [51:04]

Mm.

Philip Schroeder [51:10]

Yes. And for those

Michael Cho [51:12]

Yeah, it’s always my hunch that sorry, yep.

Philip Schroeder [51:15]

Yeah, you’re good.

Michael Cho [51:17]

Yeah, I was gonna say like it was it was always my hunch, like once these you know, whatever AI policies you know, once they are beyond like let’s say something like a level two autonomy, usually they learn a lot more just from like failures, you know. So instead of showing them more success, it’s showing what fails, it’s like they learn better from negative examples. I it’s just a general hunch up. Mm-hmm.

Philip Schroeder [51:37]

Yeah. Yeah, and I think you don’t need that many of many failures. We could have done a different experiment where we just slowly increase the amount of failure data. Cause I think there are diminishing gains you’re gonna get from that. I think after a certain set, a the model just kind of just starts to get the point. And I think so you think you could have a large corpus of expert trajectories and then just do a little bit of extra Work to get some non-expert trajectories and it will work almost just as well. And for these yellow dots, just to clarify, we’re still including these artificial failure data where you just shuffle the video frames. So the model’s not learning to just monotonically predict increasing rewards.

It still does see increasing and decreasing progress estimates during training. It’s just all these artificial examples where you’re just reversing the video essentially. That is common for

Thomas Weng [52:40]

Not the like suboptimal deviation data, right?

Philip Schroeder [52:44]

Yeah, it’s not actually changing the trajectories. It’s just playing the video back for back backwards and forwards essentially for the just to pretend like it’s the robot’s actually deviating from what you expect for the expert trajectory. Okay. And then lastly, we do this kind of scaling law experiment where if you just Subset our data to gradually increase the number of tasks you include in the training data for the reward model. You see these dramatic jumps in the number of unseen tasks you can solve through online RL. So this the different pods show the different levels of success of success you get with these different number of tasks.

So I guess the idea is if you were to put the work into scale up your training data to hundreds of tasks potentially on the x-axis, maybe that gets you up to a very meaningful number of tasks you can learn now through reinforcement learning. So in some sense you get if you learn one extra task during the reward prediction training, you get more than one potentially task that you can learn now through RL. It just I think The vision-language models just generalize way better with progress prediction than we see with VLAs or other action models predicting actual actions instead of just rewards. Okay. And then finally, I think this are will be results we add to the paper after went through the NeurIPS reviews and did more experiments.

We In addition, so our original results, we start with the randomly initialized policy. Here we just start from a strong pre-trained VLA and we show that you can use the SOLE-R1 rewards to get close to close to a hundred percent success rate or average a ninety-eight percent success rate on these more complicated tasks that involve multiple steps, multiple objects, and these are all LIBERO tasks that we’re familiar with. So still Sims tasks, but this just to show that you don’t That the model still works even if you start from a much more competent policy. And this a more realistic setting for doing RL than starting from random. And it’s much more sample efficient and you can do tasks that are much more useful by starting from the s the stronger base policy. Okay.

So I think that’s it for the slides. Let me know if you guys have any questions. But yeah, I think Overall, the like I said, the biggest things that differentiate this from RoboMeter and other VLM based reward models are is that the authentic failure or the failure data that we include in the RoboCasa data generation and then this chain-of-thought reasoning that we trained with the two stage approach of supervised fine-tuning and RL with verifiable rewards.

Thomas Weng [56:01]

I think to emphasize that last point, because we didn’t really talk about it. I mean we basically took the training recipe directly from like what was working from the world of like large models and GPTs and applied it in a kind of robotics setting of using chain-of-thought reasoning and using like reinforcement learning with verifiable rewards. And that seemed to work really well for us for this with our particular data mixture. Yeah.

Michael Cho [56:36]

Very cool. What would be sorry. Jiafei.

Jiafei Duan [56:37]

Maybe I guess maybe like the question is yeah. So I guess maybe what would be next, right? ‘Cause like, you know, I taking the as what you just say, as what Thomas just said, like taking the recipe from what it works in the in the world of LMs and GPT, you know, then does it mean to that we need to have a GPT equivalent model to make this thing really useful, or sort of like ca capitalize on its hundred percent capability? Like is that the next thing that we actually need?

Philip Schroeder [57:08]

Yeah, I mean, I think okay, so I think it can be a combination of just purely distilling from the spatial reasoning capabilities of Astrand future models. But then also leveraging things like sim creation, dynamic sim creation to use the advances in graphics and because I mean sim is particularly useful for training reward models because it doesn’t care about the physics. And the graphics are so good and they’re g only getting better, that you can get a long way with sim. So I think a combination of those two things can get to near perfect reward. I think we’re super close to reward models that are pretty perfect.

I think I wouldn’t say the same about pol like actual robot policies, but the existing VLMs have This incredible image captioning abilities, and it’s just a matter of translating that into this video native reasoning that is very sensitive to these incremental changes over time. But yeah, it’s definitely within reach, especially being able to distill from Astra.

Thomas Weng [58:26]

I think kind of coming back to what you said earlier, Jiafei about multimodality, like there’s definitely possibly something left on the table there if we don’t really look into other forms of robot proprioception or somehow distilling that or not distilling, I guess having some kind of access to that as part of this like reasoning chain. Of course you could have an external viewer, someone who just says whether a task succeeds or not. But maybe f when you’re introspecting and trying to determine if you’ll you as the agent have succeeded or not, being able to use multimodal information ideally without having to fully retrain everything, you know, what does that look like?

I mean maybe as a first pass that’s what you do, but yeah, what does it look like to take say like raw tactile signals and actually reason about maybe you have things that are summarizing what’s going on with tactile sensing or force feedback. But then at some point that needs to come back to your reward models in order to determine, like use that as part of the computation to determine whether you’ve succeeded at a task or not, or where you are in a task.

Michael Cho [59:45]

Yeah, I just to echo what you were saying earlier as well, Thomas, like I think the real-to-sim thing, it’s gonna be a big unlock. I think f you for the longest time like simulation it’s just it’s just hard to create the actual assets that kind of reflect reality. But if real-to-sim start to really scale, then you get the benefit of simulation while you have scenarios that actually is based on reality. I think that’s gonna be huge. Yeah.

Thomas Weng [01:00:12]

Yeah. Yeah, and there it seems like there’s a lot more you could do with that. Like once you have that simulator, right?

Michael Cho [01:00:20]

Hmm. Yeah, I mean you get all the ground-truth. I mean, yeah, the you can you can do all sorts of like counterexamples or counterfactual like you guys did here. Alright, very cool. Yeah, I guess you know maybe just to wrap things up we always end with a l like a typical last question. Are there any other papers that you guys came across recently you want to give a shout out for? Doesn’t have to be, you know, reward, it could be anything robotics related.

Philip Schroeder [01:00:51]

Yeah, I mean so yeah, there I think there’s very obvious ones that we’ve already talked about the baselines. I mean RoboMeter was a great contribution to this area. I think they I think have some philosophical differences with us where they have a way of training the training the dense reward prediction without actually explicitly having to label progress values for non-expert trajectories. And we do the opposite. We just brute force label dense progress values for these non-expert trajectories. But yeah so it’s it methodologically it was very good contribution, I think, and it scales better than our method potentially if you don’t have to do the dense progress prediction. And then also there’s RoboReward, obviously, and Of course all of Jason Baugh’s early work with GVL and Live.

So there’s a I’d say it’s a pretty short history, I guess, because it only goes back around like five, ten years, but there’s a there’s a long big family of reward model papers that this building on top of. But I don’t know, Thomas, do you have other papers that you would want to mention?

Thomas Weng [01:02:08]

I guess this I guess this my own opinion and not like a Ray sponsored thing. But I mean for my own opinion I think some of the Play-to-Perfect stuff is really cool for like dexterous manipulation. And I feel like there’s a lot more that’s gonna be coming in that space. I’m kind of curious of how large scale simulation of like dexterous manipulation really scales because I think to your point about assets, like maybe we will get better and better assets with better geometry and friction properties that are enough to do some interesting transfer. And the current level of like procedural generation is sufficient for achieving that. And if that’s the case, is that kind of bridging the gap between kind of large scale RL training and deployment in the real-world.

But that’s very different from I mean, and then I guess the questions there are like, you know, a lot of this work that we’ve done on progress estimation, task estimation is with these like parallel-jaw grippers. And does anything significant really change when you have dexterous hands? Because you know, from the one hand they’re maybe closer to human hands, so there’s a bit of

Michael Cho [01:03:30]

Mm.

Thomas Weng [01:03:36]

nice like retargeting that you might be able to get just by saying, this a hand, but it’s a robot. But then on the other hand, you have more occlusion potentially. A lot of things involving hands. The things you really care about are the things that are like very finicky and require a lot of like fiddling and maybe you’re doing a lot more feeling than you are of like a visual result. Maybe it’s like you fiddle with it and then once you’re done you let go and you can see that it’s completed. Well then during that whole fiddly part, you know, is it just kind of pending in terms of task progress. I’d love to see something along those lines, but I don’t know if really in terms of reward papers there’s anything that’s like close to that. But yeah, I think that’s what I’d have to say.

Philip Schroeder [01:04:19]

Yeah, there’s a new simulator, I think call it’s I don’t know I guess I don’t know how new it is, but it’s superdex. I don’t know if you guys have heard of it. It’s built for this dexterous manipulation with human hands, et cetera. So it’s it looks like a big it has like a lot of soft objects, strings, etc. So it’s kind it’s definitely a step up I think for trying to simulate highly dexterous tasks and could be really good for training reward models.

Michael Cho [01:04:36]

Yeah. Mm. I see. Yeah, I’m looking at it now. From Meta, I think. Cool.

Philip Schroeder [01:04:55]

Yes.

Michael Cho [01:04:58]

it looks cool. Awesome.

Philip Schroeder [01:05:01]

Yeah, it’s really cool, yeah.

Michael Cho [01:05:05]

Alright. Yeah, I well thank you so much, Philip and Thomas for coming on. Really appreciate the time. Yeah, I guess until your next paper, yeah, we’ll see you again.

Thomas Weng [01:05:17]

Yeah, thank you so much. This so fun.

Philip Schroeder [01:05:19]

Yeah, thank you so much for having us on. Yeah, it was a lot of fun and I really love all the episodes you’re producing, so keep it up, please.

Michael Cho [01:05:20]

Yeah. Yeah. yeah, yeah. Thanks for listening. Okay. Yeah, we w we’ll try our best, yeah. Okay, cool. All right, see you guys.

Philip Schroeder [01:05:33]

Yeah.

Thomas Weng [01:05:33]

Thank you.

Michael Cho [01:05:34]

Thanks, Thomas. Bye. Thanks, Philip. Yep.

Thomas Weng [01:05:36]

Thank you.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com

Ep#111: SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

RoboPapers

Ep#111: SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning