Hi Lizzy

Good you're making progress with the model, and thank you for your questions.

As regards balance algorithms mixing with reinforcement learning, yes, you can use the two together. One approach is to use successive feedback loop closing. If you first design your balance controller, you can then add the reinforcement as an outer loop, this providing demands to your balance controller. So you'd need to think about what the balance controller should be doing when the robot is commanded to move forwards, for example. When standing still, then the demand for robot centre of gravity needs to be within the contact area mapped out by the feet contact points on the ground, but when walking the demand might need to move outside of this area in the desired direction of travel. Another approach is to switch between controllers based on whether the robot is requested stationary or moving.

I'm not sure what HER is - a quick google search didn't show any AI or reinforcement learning related results. My advice would be to stay within the MathWorks toolset - we have many capabilities within the reinforcement learning toolbox, deep learning toolbox plus support for multiple optimization methods. More important to think about is how to set the problem up for AI/optimization/classical feedback design.

For genetic algorithms, our documentation help should be more than sufficient to figure out how to configure an optimization. The robot example you're starting with already has one training option that used genetic algorithms, so perhaps take a look at that for a good example. If you're thinking about using genetic algorithms, I'd suggest looking for a textbook that gives some insight into what types of problem genetic algorithms are good for and examples of their use.

Most important piece of advice: when looking at/assessing design methods, experiment with a simple problem first. Control of an inverted pendulum is a great place to start, for example.

Hope that helps

Rick

We do not have an on-ramp for Simscape Multibody. However, you should be able to work though the links and examples on this page to get started: Get Started with Simscape Multibody

I suggest you start with modelling something simple like a pendulum rather than starting with the robot example.

Hope that helps

Rick

A few more resources on Simscape Multibody:

• tutorials on Youtube: Physical Modeling Tutorial
• video 47 min: Multibody Simulation with Simscape Multibody

Hi Lizzy
I think you'll find it easier to get the information from the Simscape model rather than the urdf file. Much of this you can get by opening the robot definition in Simulink and interrogating the individual block parameters (bodies and transformations). For the inverse kinematics solve, you might find this useful: https://www.mathworks.com/help/physmod/sm/ref/simscape.multibody.kinematicssolver.html. This creates a MATLAB object from the Simscape Multibody model that you can use to solve the inverse kinematics problem. Hope that helps
-Rick

Hi Lizzy

Some thoughts. When a foot is not in contact with the ground, things you might want to control are landing position and leg landing angle. It is known that, when animals run/walk, leg landing angle is important when trying to regulate the trajectory from one stride to the next. When a foot is in contact with the ground, then things you might want to control are planar (x direction, y direction) forces and force in the direction of the leg.

Inverse kinematics can be used to work out joint angles required to place the foot and leg angle, and this would be one way to get target demands for the individual joint actuators. This is what conventional manufacturing robots do. However, here the actuation is assumed reversible (you can back-drive the motor and sink energy as well as acting as an energy source). This reduces energy consumption, energy being recaptured when the foot its the ground during a walk or run. This makes inverse kinematics less useful, and for better accuracy, you would need to invert the dynamics i.e. include energy stored by the mass and inertia of the limbs. One approach to this is nonlinear dynamic inversion (NDI).

I suggest that, in the first instance, you use inverse kinematics to design a controller that tells you what the individual actuator positions should be to meet a demanded foot position and leg angle, and then repeat for foot forces when in contact with the ground. This gives you a baseline that you can then use with reinforcement learning, it working with demanded foot and leg quantities plus target limb stiffnesses. Once you've done that, then read up on NDI and first apply it to a pendulum and then double pendulum to get familiar with the method. You can then have a go at applying it to the robot leg.

State-space methods could, in theory, be applied to this problem if you gain scheduled your controller with limb angles, but the dimension of your controller quickly becomes large with a humanoid robot given the number of degrees of freedom.

Hope that helps

Rick

I'm not immediately familiar with the AI literature - my background is in conventional control and system modelling. That said, I'm sure there is a lot written about AI in the context of robotics. One thing is to try varying your search criteria, for example using "reinforcement learning" in place of "AI" (which is much more general). I tried "reinforcement learning in robotics" and got lots of useful hits in Google. Try using Google Scholar in place of just Google to find the more useful references. You could try replacing "robotics" with terms like "Locomotion" or "walking". If it is the actual robot-environment interaction you're looking for, then video games are good place to look - my understanding is that the physics in many video games is reasonably accurate.

This looks like a good resource:
https://ethz.ch/content/dam/ethz/special-interest/mavt/robotics-n-intelligent-systems/rsl-dam/documents/RobotDynamics2017/RD_HS2017script.pdf
I suggest discussing with your project supervisor too. NDI would normally be a a topic for a research Masters or PhD, and much depends how much time you have allocated to your project as there is quite a learning curve. You might want instead to consider focusing on inverse kinematics plus some conventional PID control, perhaps gain scheduling PID gains with leg posture if needed.

Here's another useful reference: Dynamics, Modelling, Simulation and Control of a Spacecraft Manipulator System

Hi Jack
Take a look at the Simulink /User-Defined Functions/ sublibrary. There you will find various ways to call MATLAB during a simulation. The Function Caller block is a good place to start. I don't think pausing the simulation will work.
Rick

I'm currently using set_param to 'start' the simulation, then using set_param to 'update' after each time I change the actions, then using pause() to pause the matlab script for a timestep whilst the simulation keeps running, and then taking the new observations. Would this be more likely to work given that the simulation doesn't get paused?

I'm not clear what you're trying to do. Normally you would run the model from Simulink (either from the toolstrip, or invoke the simulation from MATLAB). Then during the simulation, at every controller discrete sample time, the controller gets updated. My suggest relates to this scenario and using the Function Caller block to call your control algorithm. Is this the set-up you have, or are you trying the run the Simulink model as part of a predictor e.g. simulate 5 seconds ahead to see what the state will be given some given control demand?

Perhaps if you could send me a diagram or short Word document summary then that would help me answer your question.

Kind regards

Rick

After some thinking, I believe my problem is actually that I can't find a function that either: pauses a matlab script (but not simulink) until a condition is met (like waitfor but it accepts boolean inputs), or acts as a pause toggle for a matlab script but not simulink (like uiwait and uiresume but without pausing simulink). There are condition-based pause functions but the "condition" always seems to be non-programmatic like a user pushing a key.

Take a look at Simulink enabled subsystems - with these you can pause/run the content of the subsystem based on a logical condition. Another way would be to write your MATLAB code with an enable flag input, and to use this within the MATLAB to control when state is updated or not. Without seeing a summary of your your simulation set-up, it's hard to be more specific.

Hi Lizzy
I believe this is the example: https://www.mathworks.com/help/reinforcement-learning/ug/train-biped-robot-to-walk-using-reinforcement-learning-agents.html
It ships with the Reinforcement Learning Toolbox which is why you couldn't find it on File Exchange.
Rick

You can simulate a model without opening the model canvas and scopes. You can use load_system('modelName') to preload the model into memory, and then call it with the sim command. However, I don't expect it will save you much simulation time - but you can give it a try.
Thanks
Rick

Welcome Anirudha. Please put any questions you have into this discussion board and let us know how you get on.
Many thanks
Rick

I'm no expert on different types of reinforcement learning, so suggest discuss this with your academic supervisor. The one piece of advice I do have is to build what you can into the problem formulation so that the learning part has fewer inputs to work with and/or a smaller search space. An example might be to build in some inverse kinematics into the problem formulation to avoid just presenting raw joint angle commands to the controller. For example, leg landing angle is known to be an important gait control variable in animal locomotion; finding a formulation that exploits this knowledge may result in faster convergence or a better solution.
Rick

Hi, I am a student at university and interested in working and collaborating on this project

Thank you for selecting this project. Please use this discussion board to post questions, and share progress or seek feedback.
Rick

Hi Seshadri, thank you for selecting this project. Please post any questions or thoughts you have about the project on this discussion board. It sounds like you are new to the topics involved in the project. If this is the case, I suggest agreeing a smaller initial task with your university project supervisor before engaging on the a full humanoid robot model and its control e.g. an inverted double pendulum, or similar. -Rick