Teaser video coming soon
placeholder: static/videos/teaser.mp4
(ST)2 lets a human teacher split a long-horizon task into segments
while demonstrating it, so the robot can be taught, corrected, and re-taught one step at a time.
Abstract
Learning from demonstration has proved itself useful for teaching robots complex skills with high
sample efficiency. However, teaching long-horizon tasks with multiple skills is challenging as
deviations tend to accumulate, the distributional shift becomes more evident, and human teachers
become fatigued over time, thereby increasing the likelihood of failure.
To address these challenges, we introduce (ST)2, a sequential method for learning
long-horizon manipulation tasks that allows users to control the teaching flow by specifying key
points, enabling structured and incremental demonstrations. Using this framework, we study how users
respond to two teaching paradigms: (i) a traditional monolithic approach, in which users
demonstrate the entire task trajectory at once, and (ii) a sequential approach, in which the
task is segmented and demonstrated step by step.
We conducted an extensive user study on the restocking task with 16 participants in a realistic retail
store environment, evaluating the user preferences and effectiveness of the methods. User-level
analysis showed superior performance for the sequential approach in most cases (10 users), compared
with the monolithic approach (5 users), with one tie. Our subjective results indicate that some
teachers prefer sequential teaching, as it allows them to teach complicated tasks iteratively, while
others prefer teaching in one go due to its simplicity.
Video
Paper video coming soon
placeholder: static/videos/st2_paper_video.mp4 or YouTube embed
(ST)2: Sequentially Teaching Sequential Tasks
Kinesthetic teaching is the most common way to demonstrate a manipulation skill, but for a
long-horizon task it is physically tiring and mentally demanding: the teacher has to think several
steps ahead while guiding the arm, and the quality of the demonstration drops as fatigue builds up.
Monolithic trajectories are also prone to error accumulation and distributional shift, and a small
mistake late in the demonstration can mean redoing everything.
(ST)2 gives the teacher explicit control over the teaching flow. The task
T = {T1, …, Tm} is demonstrated as a sequence of user-defined segments:
the teacher inserts a key-point whenever a sub-goal is reached (a top grasp, a collision-free reach,
a placement, a gripper release), and a separate
SIMPLe policy
(a time-dependent Gaussian-process policy with an uncertainty-aware Cartesian impedance controller) is
learned for each segment. The robot then executes the segments in sequence, and the teacher can
replay the segments that already work, take over at any key-point, and provide a new demonstration
from there. Five high-level commands, issued by spoken language, steer the process:
c_demoSwitch to demonstration mode: the arm becomes compliant and the teacher guides it kinesthetically while the segment is recorded.
c_insertKPInsert a key-point: close the current segment, add it to the dataset, and start a new one.
c_nextExecute the current policy autonomously until the next key-point; kinesthetic corrections during execution are appended to the segment.
c_resetReturn the robot and the environment to the start of the episode.
c_autoExecute the complete task autonomously by running the segment policies one after another.
After every interaction the policies are retrained on the updated dataset, which takes a few seconds
on a laptop without a GPU. Because a new demonstration always starts from the end point of the
previous segment, no extra machinery is needed to make the transitions between segments smooth, and
the teacher can change the granularity of the segmentation at any time by overwriting a segment and
everything after it.
Teaching flow of user #03 with (ST)2. The top line is the full restocking task with the
ten sub-task boundaries T1–T10; crosses mark the key-points this user inserted. Each branch is a
correction round: the robot executed the earlier segments autonomously up to the branch point,
and the user demonstrated new segments from there.
Supermarket Restocking Task
The user study uses a real-world restocking task in a mock retail store: a Franka Emika Panda has to
move a milk or yogurt carton from a storage box to a shelf. The box only allows a top grasp, but a
top-grasped carton cannot be placed on the upper shelf, so the robot has to put the carton down on
the table, re-grasp it from the side, and only then move it to the shelf. The task is long enough to
make the differences between monolithic and sequential teaching visible, while remaining safe and
reproducible across participants.
The restocking task. A) Initial state, carton in the box. B) Intermediate placement on the table for
re-grasping. C) Final state, carton on the shelf. Arrows on the left mark the sub-tasks listed below.
Task ID
Description
Component
T1
Move to reach carton
arm
T2
Top grasp carton
gripper
T3
Move out of the box
arm
T4
Move above the location on the table
arm
T5
Release the grasp
gripper
T6
Reposition end-effector for side grasp
arm
T7
Side grasp carton
gripper
T8
Move to the shelf
arm
T9
Release the object
gripper
T10
Move out of the shelf
arm
One possible breakdown of the task into ten sub-tasks. Participants never saw this table; they
segmented the task according to their own intuition.
The task was analysed with Failure Mode and Effects Analysis (FMEA) principles before the study. We
identified sixteen error types across teaching (e.g., touching other objects, reaching joint limits,
non-perpendicular placing), task planning (skipping the re-grasp, placing at the reachability limit),
segmenting (too few or meaningless key-points), correcting (missing the moment to correct), and
setup (gripper malfunctions). Their frequencies during the study are reported in the results.
User Study
Sixteen participants (3 female, 13 male, 22–40 years old) took part; 14 had no prior experience
with learning from demonstration and 9 had never interacted with a robot. Each session consisted of a
15-minute familiarisation with the compliant robot and both teaching methods, a 30-minute
experimental session in which the participant taught the restocking task with the monolithic method
and with (ST)2 in random order (up to three attempts from scratch per method), and a
20-minute questionnaire session with the NASA Task Load Index and open-ended questions. The study was
approved by the TU Delft Human Research Ethics Committee.
RQ1: Is the (ST)2 (sequential) method more user-friendly or effective than the monolithic method?
RQ2: Under what conditions does one teaching method become preferable over the other?
Our hypothesis was that users who manage to demonstrate the whole task in one attempt would prefer the
monolithic method for its simplicity and speed, while users whose robot fails to reproduce the
demonstration reliably would shift toward the sequential approach, which offers more control and more
opportunities to correct.
Autonomous execution of the learned (ST)2 policies
Local correction
placeholder: correction.mp4
Kinesthetic correction of a single segment
Results
16
participants
10 vs. 5
users who did better with (ST)2 vs. monolithic (1 tie)
65.5 %
success rate with (ST)2 (51.1 % monolithic)
8 vs. 6
users who preferred (ST)2 vs. monolithic (2 both)
User preferences
Users were asked which method they preferred regarding comfort, trust, expectations, and their
perception of the robot's understanding of the task. Preferences are clearly split between the two
methods, yet most users declared being comfortable with, and trusting, both of them. A notable share
of users favoured (ST)2 for trust, met expectations, and the feeling that the robot
understood the task.
Distribution of users' preferences and subjective evaluation regarding the two teaching methods.
Subjective workload (NASA-TLX)
The monolithic method was rated as less physically demanding (overall p = 0.01), which users
attributed to the single in-one-go demonstration. (ST)2 was perceived as less
temporally demanding (overall p = 0.03): with the monolithic policy, users spent a lot of
time correcting, since finding the right moment to start and stop a correction proved hard for
non-experts. Although one could expect a higher mental demand from having to decide where to put
key-points, no significant difference in mental demand was found (p = 0.42), and there were no
significant differences in perceived performance, effort, or frustration.
(a) Users who preferred monolithic teaching.
(b) Users who preferred (ST)2. NASA-TLX workloads normalised (Z-score) per user, smaller
is better. Users who preferred both methods are counted in both plots.
“It felt more satisfying getting the robot to perform the complete ‘choreography’ correctly in a single movement.”
User #03, on the monolithic method
“Potentially, I would have to re-demonstrate everything to the robot if it fails.”
User #16, on the monolithic method
“This task consists of sub-tasks. So I think I am more comfortable as a demonstrator to achieve the final goal if I can teach tasks separately. Also, I do not need to consider the optimization of the whole task, which reduces the need to think.”
User #02, on (ST)2
“Yes, it allowed me to ‘explain’ the task in segments, which is also how I prefer to do demonstrations to humans. Also, a small mistake felt less ‘stressful’.”
User #07, on (ST)2
Task performance
Task performance is scored as 1 / #trials, the inverse of the number of attempts needed for a
successful execution. Over all users, the scores of the two methods are not significantly different
(p = 0.98), with success rates of 51.1 % for the monolithic method and 65.5 % for
(ST)2. At the user level, 10 users did better with the sequential approach, 5 with the
monolithic approach, and one tied. We attribute this to the teaching flow: (ST)2 lets users
focus their corrections on the problematic segment only, so a successful task is reached in fewer
runs.
Task performance score distribution, split by the user's preferred method (users who preferred both
are counted in both plots). Higher is better.
“I prefer the (ST)2 method because I was able to correct just after a certain step. If everything was good up to that point, I could ‘save my work’.”
User #05, on (ST)2
Quality of the learned policies
Policies learned with (ST)2 produce smoother trajectories, both in Cartesian space and in
the commanded torques, while completing the task in about the same time and with a higher success
rate. This supports the assumption that teachers give better demonstrations and corrections when they
control the segmentation.
Trajectory jerk (×104)
Torque jerk (×1010)
Execution time (s)
Monolithic
10.5 ± 9.99
21.1 ± 32.5
46.1 ± 11.5
(ST)2
8.91 ± 3.05
7.54 ± 4.11
46.3 ± 11.8
Frobenius norms of the jerk product matrices (mean ± std. dev.) and average policy time to complete the task successfully.
Learned behaviour
The mean trajectories and their variability across users are similar for both methods, which shows
that the task design leaves participants freedom in how they segment while keeping the resulting
policies comparable, so the two frameworks can be compared directly.
Monolithic policy: mean trajectory and variability across users.
(ST)2 policy: mean trajectory and variability across users.
Comparison against automatic segmentation
Automatic segmentation methods need a complete demonstration before they can segment it, and they
cluster points by proximity without a guaranteed meaning for the teacher. In the example below, a
demonstration by user #16 is segmented automatically with a physically-consistent dynamical-systems
method; the light-blue segment mixes points from fetching the carton with points from placing it on
the shelf, which would be ambiguous for a policy trained on it. The key-points set by the user, on the
other hand, closely follow the sub-task boundaries of the task breakdown.
Automatic segmentation of user #16's demonstration (segments indicated by colour). The user-defined
key-points (blue crosses, K) often match the sub-task boundaries (red diamonds, T).
FMEA error frequencies
A large share of trials contained at least one of the errors identified in the FMEA analysis (76 % for
monolithic, 54.5 % for (ST)2), which confirms that the task is challenging enough for the
study. Errors involving other objects or several sub-tasks occurred more often with the monolithic
method; with (ST)2, users reached the joint limits more often, which we attribute to
disregarding the path taken between segments.
Error code
ET01
ET02
ET03
ET04
ET05
ET06
ET07
ET08
ET09
EP01
EP02
EG00
Monolithic
16.27
0
13.95
18.60
6.97
4.65
2.32
6.97
0
4.65
2.32
37.2
(ST)2
12.12
0
3.03
9.09
0
0
6.06
0
6.06
0
0
12.12
Frequency of each error (% of trials). ET: teaching errors, EP: task-planning errors, EG00: gripper malfunction. Errors that did not occur are omitted.
Takeaways
Letting the teacher segment a long-horizon task on the fly reduces the perceived temporal demand,
because corrections can be focused on the segment that failed instead of the whole demonstration.
Preferences are split and did not align with task success: some users value the control of
sequential teaching, others the simplicity of a single demonstration. Perceived usability and
actual performance can diverge, so both need to be measured.
(ST)2 policies are smoother than monolithic ones, with the same execution time and a
higher success rate at the user level.
Future work: alternative segmentation interfaces, a broader set of tasks of varying length and
complexity, and long-term studies of how teaching strategies evolve as users gain experience.
BibTeX
@article{ajanovic2026st2,
author = {Ajanovi\'{c}, Zlatan and Prakash, Ravi and de Souza Rosa, Leandro and Kober, Jens},
title = {Sequentially Teaching Sequential Tasks $(ST)^2$: Teaching Robots Long-Horizon Manipulation Skills},
journal = {IEEE Robotics \& Automation Magazine},
year = {2026},
month = {March},
volume = {33},
number = {1},
pages = {130--141},
doi = {10.1109/MRA.2026.3650853}
}
Acknowledgements
The authors would like to thank Giovanni Franzese and Marta Ferraz for their valuable discussions and
assistance at various stages of this work. LSR was sponsored within the SERICS (SEcurity and RIghts in the
CyberSpace) programme and received funding from the European Union Next-Generation EU (Piano Nazionale di
Ripresa e Resilienza (PNRR), Missione 4 Componente 2, Investimento 1.3, D.D. 1555 11/10/2022, PE00000013,
and PE7, CUP J33C22002810001, D.D. 341 15/03/2022, PE00000014). RP is sponsored by ARTPARK, IISc Bangalore.
This project is made possible by a contribution from the National Growth Fund program NXTGEN Hightech. This
work reflects only the authors' views and opinions; neither the European Union nor the European Commission
can be considered responsible for them. This work involved human subjects in its research; approval of all
ethical and experimental procedures and protocols was granted by TU Delft.
