Directional Constraints for Efficient Exploration in Safe Reinforcement Learning
(a) Unconstrained
(b) ATACOM
(c) Directional (ours)
Four actions near a constraint (red line) under three handling strategies: (a) raw agent actions without modification; (b) ATACOM scales the orthogonal component of all actions while preserving tangential components; (c) our Directional Constraints selectively scale only the actions directed toward the constraint (a1, a2), leaving safe actions (a3, a4) unchanged.
Reinforcement Learning enables robust learning of complex robotic skills in simulation, but real-world deployment in open-ended environments demands strong safety guarantees. Safe Reinforcement Learning enforces such guarantees through safety constraints — yet learning under constraints typically slows down training and can lead to suboptimal task performance, since the agent must solve a harder constrained optimization problem.
We propose ATACOM Directional Constraints (ATACOM-DC), an extension of the state-of-the-art ATACOM safety layer. The key idea is simple: morph an action through the safety layer only when it moves the system toward a constraint boundary, and leave it untouched when it moves away. This significantly improves the safety–performance trade-off and promotes efficient, safe exploration.
Contributions:
ATACOM achieves safe exploration by turning the constrained problem into an unconstrained one defined over a constraint manifold. Assuming a control-affine system \( \dot{\mathbf{s}} = \mathbf{f}(\mathbf{s}) + \mathbf{G}(\mathbf{s})\,\mathbf{u}_s \) and analytic constraints \( \mathbf{k}(\mathbf{s}) \le \mathbf{0} \), it redefines the action space as the tangent space of the manifold, so that every action keeps the system safe. The safe control input is:
\[ \begin{bmatrix} \mathbf{u}_s \\ \mathbf{u}_\mu \end{bmatrix} = -\,\mathbf{J}_u^{\dagger}\,\boldsymbol{\psi} \;-\; \lambda\,\mathbf{J}_u^{\dagger}\,\mathbf{c} \;+\; \mathbf{B}_u\,\mathbf{u}, \]where \( \boldsymbol{\psi} \) compensates the constraint drift, the contraction term \( \lambda \mathbf{J}_u^{\dagger}\mathbf{c} \) drives the state back to the manifold, and \( \mathbf{B}_u \mathbf{u} \) is the tangential term that lets the agent explore freely.
The limitation is that ATACOM suppresses the action component orthogonal to the constraint symmetrically: it limits motion toward a boundary, but also unintentionally suppresses motion that would move the agent away from it. Once the agent is near a constraint, escaping toward safer regions becomes difficult, hampering exploration and slowing learning.
Directional Constraints scale down only the actions that increase a constraint value. To decide which constraints matter for the sampled action, we evaluate each constraint derivative:
\[ \dot{c}^{\,i}(\mathbf{s}) = \psi^{i}(\mathbf{s}) + \mathbf{J}_k^{i}(\mathbf{s})\,\mathbf{G}(\mathbf{s})\,\mathbf{u}_s . \]A positive derivative means the action drives the system toward the boundary; a negative one means it moves away. We therefore keep only the constraints with \( \dot{c}^{\,i} > 0 \) and disable the rest when computing the action morphing. After compensating the system drift, each constraint derivative is a hyperplane through the origin in action space, so the sampled tangential action determines a quadrant: only the constraints active in that quadrant influence the safe action.
In practice: compensate the drift, sample a residual action, compute the active set from the constraint derivatives, and apply the ATACOM controller using only the active constraints.
Each constraint derivative \( \dot{c}(\mathbf{s},\mathbf{u}) \) is linear in the sampled action. System drift shifts the lines off the origin (left); after drift compensation they pass through the origin (middle); given the sampled residual action \( u_t \), we keep only the constraints with positive derivative and disable the rest (right).
We evaluate ATACOM-DC on three simulated robotic control tasks with safety constraints (joint and workspace limits, collision avoidance, and stability constraints).
KUKA iiwa air hockey
Planar air hockey
Quadrotor navigation
All methods use SAC as the underlying RL algorithm; each experiment is repeated over 15 random seeds. We report a task-performance metric (top row) and the episodic cost — the cumulative constraint violation over an episode (bottom row).
Against vanilla ATACOM, Directional Constraints consistently lead to faster learning and improved final performance, while keeping constraint violations comparable or lower. The effect is most striking on the complex KUKA iiwa air hockey task: less restrictive exploration yields faster, more precise policies, with a higher success rate and faster puck velocities.
KUKA iiwa air hockey: ATACOM-DC (ours) vs. ATACOM.
Planar air hockey (velocity)
Planar air hockey (acceleration)
Quadrotor
Planar air hockey and quadrotor: task performance (top) and safety violations (bottom). ATACOM-DC (ours) vs. ATACOM.
Compared to unconstrained SAC, ATACOM-DC maintains safety throughout the entire training process with similar final performance and generally better or comparable learning curves. Notably, on the quadrotor and on planar air hockey controlled in acceleration, the final SAC policy still violates constraints to reach higher task performance — whereas ATACOM-DC stays safe.
Planar air hockey (velocity)
Planar air hockey (acceleration)
Quadrotor
Task metric (top) and safety (bottom). ATACOM-DC (ours) vs. unconstrained SAC.
If you find this work useful, please consider citing:
@inproceedings{magliano2026directional,
title={Directional Constraints for Efficient Exploration in Safe Reinforcement Learning},
author={Magliano, Paolo and Liu, Puze and Peters, Jan and Tateo, Davide and Camoriano, Raffaello},
booktitle={2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages={1-8},
year={2026},
organization={IEEE}
}This study was carried out within the FAIR – Future Artificial Intelligence Research and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 – D.D. 1555 11/10/2022, PE00000013). This manuscript reflects only the authors' views and opinions, neither the European Union nor the European Commission can be considered responsible for them.
