Today we introduce Robostral Navigate, our first model designed for integrated navigation. It is an 8B model that takes RGB images and understandable instruction and moves a robot through an environment:
“Exit the lobby, go through the corridor, enter the supply room and stop to face the second shelf.”
To accomplish such tasks, other models often use depth sensors, LiDAR or multiple cameras working together. Using only a regular RGB camera and no depth sensors, Robostral Navigate still achieves 76.6% in R2R-CE (Room-to-Room in Continuous Environments) validation, the benchmark for following instructions in environments where no training occurs. This beats the best single-camera approach by 9.7 points and the best depth-of-field or multi-camera system by 4.5 points, despite using neither.
Table of Contents
navigation
Our model is designed for robotic navigation, enabling robots to autonomously navigate complex environments, including offices, residential and commercial buildings, and outdoor areas.
Robostral Navigate runs completely autonomously on a long teaching route through a work office.
This technology opens up numerous applications in manufacturing, delivery, logistics and hospitality, making it one of the most sought-after capabilities for our customers today. Give Robostral Navigate an instruction and it completes the entire task on its own, moving through a living space full of people and obstacles that it has never been shown and being able to adapt to any environment.
Highlights
State-of-the-art performance on R2R-CE
Powered by a single RGB camera, without LiDAR or depth sensors
8B model, self-built and fully trained in simulation
Runs on wheeled, legged, and flying robots and generalizes across robot sizes
Robust to differences in camera characteristics
Token efficient training using prefix caching
Navigation by pointing
Based on a task and an observation history, Robostral Navigate predicts where the robot should move next showing: It derives the image coordinates of the destination location in the robot’s current camera view along with the desired orientation upon arrival. Unlike commands based on metric shifts, pointing makes the policy naturally robust to changes in camera properties and world scale.
However, this method cannot handle cases where the target location is outside the current field of view. If pointing is not applied, the model will resort to displacements in the robot’s local coordinate frame, such as:
“Walk 2 meters forward, 1.5 meters to the left and turn 25 degrees to the left.”
Built from the ground up
Robostral Navigate is developed entirely in-house and does not rely on existing open source VLMs.
The model is initialized from our Vision-Language model, which specializes in grounding tasks such as pointing, counting and object localization. Navigation is a natural extension of these skills: once she understands where things are, she learns to move.
We have built an efficient data generation pipeline entirely in simulation. This allowed for rapid iteration of the data, resulting in a data set of approx Collected 400,000 trajectories across 6,000 scenes.
Efficient supervised training
An essential part of Robostral Navigate is an efficient training algorithm based on prefix caching. Using a tree-based attention masking strategy, our method compresses an entire episode into a single sequence, enabling training of all time steps in a single forward pass while preventing information loss between time steps.
Compared to training with one sample per time step, our approach reduces the number of training tokens by 22× while retaining all learning signals. In practice this method is converts training runs that would take months into runs that are completed in days.
Online learning for reinforcement
We leverage our knowledge of LLMs after training at scale and use online learning for reinforcement to increase the performance of Robostral Navigate. After the supervised training phase, we improve the performance of the model using CISPO, an online reinforcement learning algorithm. This allows the model to learn through trial and error, recover from errors, and acquire exploratory behavior, effectively mitigating the distribution shift problem when cloning vanilla behavior. This alone improved the success rate by 3.2%. Since we don’t see a plateau, we are confident that more training and more experimentation will continue to push this number higher.
What’s next?
Robostral Navigate is just the first step toward a unified embodied agent.
We believe that navigation is a fundamental capability for general-purpose robotics. By combining large-scale simulation, efficient training, and strong grounding priors, Robostral Navigate demonstrates that state-of-the-art embodied navigation can be achieved with a compact model and a single RGB camera.
Start your journey to embodied frontier AI and speak to our team.
By the way, we are hiring!
The release of our navigation models represents a significant step forward, but our journey is far from over. Our goal is to enable robots to autonomously navigate complex environments – offices, homes, commercial buildings and outdoor areas – and there is still much to do. We are actively expanding our robotics team and are looking for talented researchers and engineers who share our ambitions.
If you are interested in joining us in our mission to enable seamless navigation for robots everywhere, we welcome your applications to join our team!
By Théo Cachet, Arjun Majumdar, Srijan Mishra, Thomas Chabal, Chris Bamford, Elliot Chane-Sane, Benjamin Tibi, Ludovic Ho Fuh, Olivier Duchenne – AI science robotics
https://mistral.ai/news/robostral-navigate/
