# Workshop at IEEE Humanoids 2011 - Dynamic Models and Optimal Control of Humanoid Robots

## Organizers:

Katja Mombaur, IWR, University of Heidelberg, Germany

katja.mombaur@iwr.uni-heidelberg.de

Wael Suleiman, Electrical and Informatics Engineering Department, Sherbrooke University, Canada, wael.suleiman@USherbrooke.ca

## Time and Place:

October 26, 2011, 9:00 – 17:00

Golf Hotel, Bled Slovenia

More information at www.humanoids2011.org

## Workshop topic:

The goal of this workshop is to gather specialists of dynamic modeling and optimal control problems and efficient feedback control for humanoid robots.

Humanoid robots are highly complex dynamical systems which are subject to many constraints determined by their geometries, their actuators, their control systems - in particular their stability control - as well as their environment. Controlling humanoid robots is a particularly challenging task. The goal in humanoid robotics today is often not only to determine a feasible motion for a given robot, but to find the best possible motion in the given situation. The optimization criterion depends on the circumstances – e.g. the fastest, highest, most efficient, smoothest or most natural motion (i.e. resembling the human role model) will be desired. When performed already during the design process, also design variables can be chosen in an optimal manner. On the other hand, such optimization processes do not only appear off-line – i.e. in the planning process of a motion – but also online, asking for an optimal reaction of the robot to a perturbation.

The generation of the best possible motion – or the optimal trajectory - for humanoid robots results in optimal control problems. Optimal control problems are optimization problems that consider differential equations, such as the dynamical equations of a robot, as constraints. The unknown variables of optimal control problems are not simply n-dimensional vectors as in the case of optimization problems, but vectors of functions, i.e. histories in time, of the state and control variables of the system. There are basically three different approaches to the numerical solution of optimal control problems: (a) Methods based on dynamic programming which require the solution of the Hamilton-Jacobi-Bellman equation and are restricted to small state space dimensions (b) Indirect Methods such as Pontryagin’s Maximum Principle result in problems very difficult to solve for any real application problems.(c) Direct Methods that transform the optimal control problem into a nonlinear programming problem by discretization, are the most suitable methods to solve real world application problems.

Topics of this workshop include but are not limited to:

Modeling dynamic motions of humanoid robots

Optimal control techniques and available tools for off-line optimization of humanoid robot motions

Efficient stability control of humanoid robots

Real-time optimization and model predictive control

Hierarchical problem formulation for optimization problems in humanoid robots

Optimization of walking motions for different objective functions

Optimization of reaching motions

Optimization of interaction between humanoid robots and between humans and humanoids

Optimization of dynamic object manipulation

Biologically inspired optimization criteria for humanoid robots

Imitation of human motions on humanoids

## Workshop program (click here)

## Confirmed invited presentations :

**Chris Atkeson,**CMU, USA :

Efficient Robust Policy Gradient LearningA major focus of reinforcement learning, a form of optimal control, is directly representing and refining a policy. Both model-free and model-based algorithms have been proposed. Stochastic policies are often used to explore and to numerically compute derivatives. We explore model-based algorithms for using deterministic policies. We develop efficient algorithms to calculate first and second order gradients of the cost of a control law with respect to its parameters, to speed up policy optimization. The approach achieves robustness by simultaneously designing one control law for multiple models with potentially different model structures, which represent model uncertainty and unmodeled dynamics. We argue that providing explicit examples of possible unmodeled dynamics during the control design process is easier for the designer and is more effective than providing simulated perturbations to increase robustness, as is currently done in machine learning. Our approach supports the design of nonlinear and time varying controllers for deterministic discrete time nonlinear and time varying systems, including policies with internal state such as observers or other state estimators. We highlight the benefit of control laws made up of collections of simple policies where only one simple policy is active at a time. Controller optimization and learning is particularly fast and effective in this situation because derivatives are decoupled.

**Karim Bouyarmane**, CNRS-AIST, JRL, Tsukuba, Japan:

Humanoid Motion Control through Changing Contact ConfigurationsThe aim of this work is to generate motion of a humanoid robot that is not restricted to cyclic walking. We envision the possibility of a humanoid robot to use various contact spots on its links in order to perform complex motion. The chosen methodlogy is to use multi-objective control, ie. the optimization of a weighted sum of quadratic objectives under linear constraints that include the whole-body dynamics of the robot. The objectives are decided by a finite state machine according to the current contact configuration and the next one we would like to reach. This approach has been tested on simulation, and we will display some results obtained in the form of playback videos.

(Presentation slides)

**Olivier Bruneau**, LISV-UVSQ, Paris, France:

What is a relevant modeling for anthropomorphic robots?Despite the level of technological achievement of complex humanoids, dynamic walking with large inertial effects and with a reactive capability of adaptation remains a major goal at this time in order to envisage a smooth evolution of these robots in environments designed for humans. One way to generate precise and quick walking movements is the use of dynamic models sophisticated enough to handle the nonlinear and coupled phenomena acting on the robot and simple enough to be implemented in real time. Thus, major efforts are still to be done, firstly in terms of effective calculations of dynamic models and the objective justification of their simplifications, secondly in terms of formalization of unified approaches for the generation of various dynamic behaviors. Based on the observation that the dynamic models used for control strategies of bipedal walking robots are mostly either fixed with no explanation or justified in a purely qualitative manner, we have been led to ask ourselves the following question: what is the level of relevance, in quantitative terms, of a dynamic model compared to a reference model? In other words, until what percentage, does a simplified model represent correctly the dynamic behavior of a biped walking for which the complete dynamic model is too complex to control online? To illustrate this latter point, on the one hand the shortcomings of traditional models (Inverse Pendulum model (IPM), Linear IPM, Multi-mass IPM, RMP Reaction Mass Pendulum,...) will be highlighted with respect to criteria of relevance, on the other hand new models more suitable for the generation of highly dynamic behavior will be proposed.

(Presentation slides)

This talk will outline methods for modeling and simulating electrically powered biped robots. The mechanical components are modeled as a rigid multibody system. In order to enable efficient forward dynamics simulations, the equations of motion are derived in a minimal coordinate representation based on the Newton-Euler equations and the principle of virtual power. The foot-ground contact is essential in biped dynamics, since overall linear and angular momentum, and therefore global system dynamics, are only influenced via contact forces. The talk will therefore also describe several methods of modeling the ground contact. There has been quite a bit a research devoted to modeling and identifying friction in Harmonic Drive gears. I will present simple models of load and speed dependent friction and gear elasticity that only depend on cataloged data and do not require additional measurements. Finally, I will show simulation results for the robots**Thomas Buschmann**, TU München, Germany:

Modeling and Simulating Biped Robots*Johnnie*and*Lola*and discuss the relevance of the individual parts of the model with respect to different simulation tasks.

(Presentation slides)

**James Kuffner**, Google and CMU, USA:

Footstep Planning: Discrete Search and Whole-Body DynamicsAutonomous navigation for legged robots in unstructured environments is an important enabling technology for humanoids. The ability for a robot to determine a high-level strategy for navigation based on planning a sequence of footsteps over terrain (a.k.a. "footstep planning") has been an active area of research for ten years. In this talk, I will present an overview of the evolution of footstep planning as it has evolved from simple obstacle overstepping to selecting and executing complex footstep and leg trajectory patterns over rough terrain. Issues related to the workshop themes of dynamic modeling, metrics, and optimality will be discussed.

(Presentation slides)

Emotional facial expressions for humanoid robots have received a lot of attention in recent years. Only very few researchers have investigated emotional body language in the context of humanoid robots. This talk describes a method based on dynamic models and optimal control that can be used to endow humanoid robots with the capability to perform motions which are perceived “emotional”. We study in particular forward walking motion. The underlying assumption of this research is that movements in nature are results of optimization processes, but that different emotions result in different objective functions and therefore produce different movements. Our investigations are based on motion capture and EMG measurements of human forward walking under different emotions. Objective function candidates for each emotion are formulated, and the computed movements are compared to the measurements. Inverse optimal control techniques will used to refine the identification of the objective function. Once correctly identified, optimal control problems can be formulated for the dynamic model of any humanoid robot in order to generate emotional robot walking motions.**Katja Mombaur**, University of Heidelberg, Germany

Using optimal control to generate emotional body language for humanoid robots

(joint work with Martin Felis, Alain Berthoz, Hideki Kadone)

(Presentation slides)

While robust trajectory following is a well-studied problem on mobile robots, the question of how to track accurately a trajectory on a humanoid robot remains open. In this talk, a closed-loop trajectory tracking strategy aimed at humanoid robots will be presented. Compared to approaches from mobile robotics, this control scheme takes into account footsteps alteration, equilibrium constraints and singularities avoidance for humanoids. It provides a robust way to execute long and/or precise motion with the ability of correcting on-line preplanned trajectories in a very reactive manner. Results have been validated on the HRP-2 humanoid platform. Using a motion capture system to localize the robot in real-time, the final robot position has been reached precisely in an highly cluttered environment where the robot has to walk over obstacles to reach its destination.**Thomas Moulard**, LAAS-CNRS, Toulouse, France:

Trajectory following for legged robots

(Presentation slides)

Stochastic optimal control is recently emerging as a model for describing human movement planning. In this talk we will use this mathematical tool to illustrate the possibility of optimally planning the intrinsic system stiffness when performing movements in presence of uncertainties and significant feedback delays.**Francesco Nori,**IIT Genua, Italy:

Active /Passive compliance and stochastic optimal control in movement planning

(Presentation slides)

**Christian Ott**, DLR, Munich, Germany

Humanoid Walking Control using the Capture Point

(joint work with Johannes Englsberger)Bipedal gait generation and stabilization is one of the fundamental tasks for humanoid robot control. A widely used approach is to use a walking pattern generator for computing a nominal desired trajectory, and implement this desired trajectory by a lower level feedback controller for gait stabilization. In this talk, a novel walking control approach will be presented, in which the capture point is used as an intermediate control variable. The system representation based on the capture point leads to an advantageous cascaded system structure in which the stable and unstable parts of the dynamics can easily be separated. Based on the capture point dynamics, two different feedback controllers are discussed and compared with state of the art pattern generation based walking controllers. The presented controllers have been verified in experiments with the DLR-Biped.

(Presentation slides)

Compliant actuation in combination with a proper control system is considered a key property for the realization of human-like bipedal locomotion. In this talk, we introduce our research on the design of the first prototype, BioBiped1, built within the BioBiped project. The goal of the project is to investigate and evaluate hypotheses and results from biomechanics by transfer to a new robot design. Using multibody system dynamics models and simulations, incorporating series elastic actuator models and a realistic nonstiff ground contact model, we compute the forward dynamics of the electrically driven, elastic biped for hopping and running motions. The choice of reference gaits is based on the hypothesis, that the central humanoid locomotion capability should be jogging, and not walking, as opposed to many existing approaches. The obtained ground reaction forces and locomotion performance support the desired dynamic motion goals. The method is applicable to any other robot with series elastic actuation and can be considered as a workaround for the control of a bipedal robot whose sophisticated mechanics still needs to be integrated into an overall control concept.**Katayon Radkhah,**University of Darmstadt, Germany:**Dynamics modeling and simulation of hopping and running gaits for the musculoskeletal robot BioBiped1**

(Presentation slides)

For participation in this workshop, please register at the conference web page www.humanoids2011.org