Research Physical Human–Robot Interaction
Deep Reinforcement Learning to Optimize Task Performance in Human-Robot Co-Manipulation
M.Sc. Thesis, Koç University, 2023
@mastersthesis{guler2023comanipulation,
title = {Deep Reinforcement Learning to Optimize Task Performance in Human-Robot Co-Manipulation},
author = {Guler, Berk},
school = {Ko{\c{c}} University},
year = {2023}
}In industry, complex tasks encourage human operators to work together with robots, and carrying heavy objects is one of them. Using the human's dexterity and intelligence and the robot's precision, we can drive and park heavy objects where the human operator wants without the programming in advance. This study's initial aim is to increase efficiency by understanding human motion intention. However, the human force's response is observed by delay due to the object's weight. Therefore, estimated human intention can help make decisions on amplifying or attenuating the human force. Constantly scaling in one way is not efficient for human effort because of the different phases of the co-manipulation task. Amplification is preferable during driving when attenuation becomes more of an issue for parking. The deep learning model provides an adaptive force scaling factor by referring to human motion intention without guaranteeing optimal human effort and comfort. The model-free optimal control algorithm, deep reinforcement learning, adjusts the existing adaptive force scaling factor to decrease human effort and increase comfort.
AI-generated summary · may contain mistakes, see the paper
When a human and a robot carry a heavy object together, a fixed force amplification is a compromise: amplifying the operator's force helps while driving the object, but makes precise parking hard. In this thesis, a neural network reads the operator's recent force and velocity to infer whether they are trying to speed up or slow down, and continuously retunes how strongly the robot amplifies their force. A deep reinforcement learning agent sits on top and adjusts how aggressively that retuning happens, several times a second, trading motion smoothness against physical effort.
Note that the agent does not pick the gain itself; it sets how fast the gain moves toward the value implied by the operator's intent.
Try it: push and park
Push the object into the target zone and hold it there. The robot renders a 40 kg mass with 100 N·s/m damping, so the object keeps moving after you let go. Compare a low, nominal and high force gain, then switch to Adaptive and change the adaptation rate the RL agent would choose.
position force gain target
GH 0.33 GH 1.0 GH 2.68 Adaptive
An illustrative re-implementation of the thesis simulation's admittance model (40 kg virtual mass, 100 N·s/m damping, 125 Hz, the identified velocity loop of the real robot, and GH between 0.33 and 2.68). Two parts are simplified: in Adaptive mode the operator's intent is read from the commanded acceleration (GH·Fh − b·v: pushing harder than needed to hold the current speed means speeding up) instead of the thesis's neural network, and the simulated human follows a minimum-jerk plan instead of the study's learned human model. Effort and jerk follow the definitions used in the paper: human effort is the energy the operator puts in, ∫|Fh·v| dt, and jerk is the integrated absolute jerk of the object, ∫|v̈| dt. Lower is better for both; in this toy, a higher gain usually lowers the effort but raises the jerk. The numbers shown come from this toy, not from the study.
My contributions
- Built an experimental setup for human-robot co-manipulation of a heavy object with a UR5 robot.
- Developed a deep-learning model in
PythonwithTensorFlowto estimate the human's intention to accelerate or decelerate the object. - Developed a deep reinforcement learning agent, working with the intention model, that optimizes task performance with respect to minimum jerk and human effort.
- Designed the pHRI experiments, collected the data, and performed the statistical analysis.