
Robot language interpreter integration tutorial
This tutorial will try to explain how to integrate or embed a robot language interpreter into V-REP. The procedure is very similar in case you want to integrate an emulator (e.g. a specific microcontroller emulator) into V-REP. Extending V-REP's functionality requires most of the time the development of a plugin. Make sure you have read the tutorial on plugins , and the tutorial on external controllers before proceeding with this tutorial.
The V-REP scene file related to this tutorial is located in V-REP's installation folder "scenes\robotLanguageControl.ttt". The plugin project files, and the server application project files of this tutorial are located in V-REP's installation folder "programming\v_repExtMtb" and "programming\mtbServer". The MTB plugin and MTB server should already be compiled in the installation directory. If you wish to recompile the plugin, remember following:
The plugin is a Qt project, therefore it will require the Qt framework for compilation (most other plugin projects in the programming directory are non Qt projects)
You are required to compile the plugin with the exact same Qt version as the one used to compile V-REP. Have a look at the V-REP [Help --> About] menu bar item for details about Qt version
You should compile the plugin with the same compiler as V-REP (MinGW on Windows, and GCC on Mac OSX and Linux)
First, let's start by loading the related scene file "scenes\robotLanguageControl.ttt":

The "MTB" robot is an imaginary robot ("MTB" stands for "Machine Type B"), that will be controlled with an imaginary robot language. Access the plugin dialog located at [Menu bar --> Plugins --> MTB Robot Properties]. You will only be able to do so if the related plugin (v_repExtMtb.dll, libv_repExtMtb.dylib or libv_repExtMtb.so) was properly loaded (make sure the plugin was located in the same directory as the V-REP executable when you launched V-REP). Select an MTB robot in the scene. This is what the plugin dialog should display:

The plugin dialog shows all recognized MTB robots in the scene (try copy-pasting the robot). If the last selected robot is an MTB robot, then its control program (in the imaginary robot language) is displayed in the lower part of the dialog. That control program can be modified while the simulation is stopped.
As previously stated, the used robot language is imaginary and very very simple. Following commands are supported (one command per line, input is case-sensitive):
"REM" starts a comment line
"SETLINVEL v": sets the prismatic joint velocity for next movements (v is in m/s)
"SETROTVEL v": sets the revolute joint velocity for next movements (v is in degrees/s)
"MOVE p1 p2 p3 p4": moves to joint positions (p1;p2;p3;p4) (in degrees except for p3 in meters)
"WAIT x": waits x milliseconds
"SETBIT y": sets the bit at position y (1-32) in the robot output port
"CLEARBIT y": clears the bit at position y (1-32) in the robot output port
"IFBITGOTO y label": if bit at position y (1-32) in the robot input port is set, jump to "label"
"IFNBITGOTO y label": if bit at position y (1-32) in the robot input port is not set, jump to "label"
"GOTO label": jumps to "label"
Any word different from "REM", "SETLINVEL", "SETROTVEL", "MOVE", "WAIT", "SETBIT", "CLEARBIT", "IFBITGOTO", "IFNBITGOTO" and "GOTO" is considered to be a label. When the dialog checkbox Automatic execution is selected, then the program will execute automatically when simulation starts. Now run the simulation. If the related plugin was not found, following message displays (the display of the message is handled in the child scripts attached to objects "MTB_Robot" and "MTB_Robot#0"):

If the related plugin was found, then the MTB robot language programs will first be compiled before execution. This happens without the user noticing anything, unless the program contains an error, in which case the plugin displays following message (the display of the message is handled in the child scripts attached to objects "MTB_Robot" and "MTB_Robot#0"):

If the compilation was successful, then the robots start executing their respective program. The simulation is a maximum speed simulation, but can be switched to real-time simulation by toggling the related toolbar button:

The execution speed can be even more accelerated by pressing the appropriate toolbar button several times:

Each MTB robot program can be individually paused, stopped or restarted at any time via their displayed custom dialog, which are custom user interfaces:

Above custom user interface is the user-interface of the MTB robot and can be fully customized. Should the MTB robot be copied, then its custom user interface will also be copied. Next to being able to controlling the program execution state, the custom user interface also displays current program line ("Cmd") and the MTB's current joint values. The button located at the bottom of the custom user interface ("Show input/output") allows toggling the input/output dialog:

The above custom user interface allows the user to change the robot's input port bits, and to read the robot's output port bits. Input and output ports can be read and respectively written by the robot language program. Input and output ports can also be written and read by external devices (e.g. the robot's gripper or suction pad) by using appropriate function calls (see further below).
There are two small child scripts attached to the "MTB_Robot" and "MTB_Robot#0" objects. They are in charge of handling the custom dialogs (custom user interfaces) and communicating with the MTB plugin. Most code in the child scripts could be handled by the plugin too.
Now copy and paste one of the MTB robots. Notice how the third MTB robot appears in the plugin dialog:

Run the simulation. Notice how the third MTB robot has also an attached robot program executing. Also notice how all custom dialogs have automatically been copied too. The third robot is actually completely independent and not aware of the others, since one would first have to connect its input/output ports to the other robots. You can also copy/paste the robots during simulation. At any time, you can also load models of other types of robots, furniture, obstacles, etc. by using the model browser. Stop the simulation. At simulation stop, objects added during simulation will be removed automatically, and the initial object's position/orientation restored, but that default behaviour can be modified in the simulation settings dialog.
Try to modify an MTB robot's program to have it perform a different movement sequence. Experiment a little bit. Then save the scene under a different name, and reload it. Notice how the robot language program modification was also saved.
The MTB robots are handled in following way:
the actual robot language program is compiled and executed by the "mtbServer" application. That application also holds the MTB robot's state variables. For each MTB robot in the simulation scene, there will be an instance of the "mtbServer" application launched by the "v_repExtMtb" plugin.
the "v_repExtMtb" plugin handles the user interface (i.e. the MTB robot dialog). It is also in charge of providing custom Lua functions, and handles attaching custom data to scene objects: indeed, the robot language programs are saved/loaded together with scene objects, in the same file! The plugin also launches the "mtbServer" application when needed, and communicates with it via socket communication.
the child scripts attached to "MTB_Robot" and "MTB_Robot#0" check whether the "v_repExtMtb" plugin is loaded, update the custom user interfaces for each robot, and control the robot language execution.
The MTB robot and its simple robot language is a simple prototype meant to demonstrate how to integrate a robot language interpreter into V-REP. It is very easy to extend current functionality for much more complex robots or robot languages. All what is needed is:
Building the model of the robot. This includes importing CAD data, adding joints, etc. This step can be entirely done in V-REP.
Writing a plugin to handle the new robot natively, i.e. to handle the new robot by interpreting its own robot language. Any language capable of accessing C-API functions and capable of being wrapped in a dll can be used to create the plugin (but c/c++ is preferred). The robot language interpreter could be directly embedded in the plugin, or launched as an external application ("mtbServer") as is done in this tutorial.
Writing a small child script responsible for handling custom dialogs and linking the robot with the plugin. This step can be entirely done in V-REP.
Now let's have a look at the MTB's plugin project. There is one prerequisites to embedding a robot language interpreter (or other emulator) into V-REP:
The robot language interpreter should be able to be executed several times in parallel. This means that several interpreter instances should be supported, in order to support several identical, in-parallel operating robots. This can be handled the easiest by launching a new interpreter for each new robot, as is done in this tutorial.
When writing any plugin, make sure that the plugin accesses V-REP's regular API only from the main thread (or from a thread created by V-REP)! The plugin can launch new threads, but in that case those new threads should not be used to access V-REP (they can however be used to communicate with a server application, to communicate with some hardware, to execute background calculations, etc.). This limitation will be relaxed in future.
Following section describes each file contained in the MTB plugin project. The project files themselves contain many comments that help understand its architecture:
"v_repExtMtb.h" and "v_repExtMtb.cpp"
Those files contain the 3 required dll entry points for V-REP: "v_repStart", "v_repEnd" and "v_repMessage". Refer to the section on plugins for more details.
In "v_repStart", the plugin is initialized (when V-REP is launched) and 9 custom Lua commands are registered ("simExtMtbReset", "simExtMtbRun", "simExtMtbGetInput", "simExtMtbSetInput", "simExtMtbGetOutput", "simExtMtbGetJoints", "simExtMtbGetAutoStart", "simExtMtbConnectInput" and "simExtMtbDisconnectInput"). This allows a script (or child script) to call back the plugin with specific parameters and/or to retrieve data from the plugin. Open the child script attached to the MTB robot and try to locate where those custom Lua commands are called.
In "v_repEnd", the plugin releases all resources and memory. This happens when the V-REP application ends.
In "v_repMessage", the plugin handles various messages from V-REP (toggle the plugin dialog, etc.)
"LUA_RESET_CALLBACK" is the function that is called (called-back) when "simExtMtbReset" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will reset the robot language interpreter for the specified robot.
"LUA_RUN_CALLBACK" is the function that is called (called-back) when "simExtMtbRun" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will run the robot language interpreter for the specified robot.
"LUA_GET_INPUT_CALLBACK" is the function that is called (called-back) when "simExtMtbGetInput" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will read 4 bytes of the robot's input port.
"LUA_SET_INPUT_CALLBACK" is the function that is called (called-back) when "simExtMtbSetInput" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will write 4 bytes to the robot's input port.
"LUA_GET_OUTPUT_CALLBACK" is the function that is called (called-back) when "simExtMtbGetOutput" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will read 4 bytes of the robot's output port.
"LUA_GET_JOINTS_CALLBACK" is the function that is called (called-back) when "simExtMtbGetJoints" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will read the state of the robot's 4 axes (joint values).
"LUA_GET_AUTO_START_CALLBACK" is the function that is called (called-back) when "simExtMtbGetAutoStart" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called, then relays the call to the appropriate plugin MTB object for execution. This function will check whether the specified robot should automatically start executing its robot language program at simulation start.
"LUA_CONNECT_INPUT_CALLBACK" is the function that is called (called-back) when "simExtMtbConnectInput" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robots it is called. The function connects a robot's output port bit, to another robot's input port bit.
"LUA_DISCONNECT_INPUT_CALLBACK" is the function that is called (called-back) when "simExtMtbDisconnectInput" is called in the child script. When this function is called, the plugin first needs to figure out for which MTB robot it is called. The function disconnects a bit of the robot's input port.
"mtbGlobal.h"
"mtbGlobal.h" contains 2 important types of data: the developer header and some data tags. Both are used to attach custom data to scene objects (custom data is saved and loaded with scene objects, so this represents a way for a plugin to save custom data with a scene or model). Also refer to the tutorial on plugins for more details.
DEVELOPER_DATA_HEADER: this integer number identifies the company or developer of a plugin. If the same company or the same developer produces several plugins, then the same DEVELOPER_DATA_HEADER should be used. Best is to use the serial number of your first V-REP copy. It is important that you always use the same number, so that your data and other developer's data don't collide.
data tags: data tags are integer numbers that identify different types of custom data saved under a same developer header. For instance, the MTB_PROGRAM tag identifies data that represents an MTB robot's robot language program.
"Access.h" and "Access.cpp"
Represents a static class that holds the plugin's main objects (e.g. dialog & object container). Following routines/data structures are important:
"mtbRobotDialog": the dialog of the plugin
"mtbRobotContainer": the "mtbRobotContainer" of V-REP's current scene.
"insertSerializationData" and "extractSerializationData": functions used to pack several types of data under a same developer header. Refer to the tutorial on plugins for more details.
"mtbRobotContainer.h" and "mtbRobotContainer.cpp"
Represents a container that holds all instances of CMtbRobot for the current scene:
"_allMtbRobots": all instances of CMtbRobot
"actualizeForSceneContent": this function synchronizes objects from V-REP with objects from the plugin; for each MTB robot in the scene, there must be a related CMtbRobot instance in the plugin. When for example the user copy/pastes an MTB robot in the scene, then this function will recognize that a new MTB robot is in the scene, and automatically add a related CMtbRobot instance. In a similar way, if the user deletes an MTB robot in the scene, then this function will remove its related CMtbRobot instance. Following figure illustrates how the plugin should keep its state synchronized with V-REP (the figure corresponds to 2 MTB robots in scene 1, and to 4 MTB robots in scene 2):

"mtbRobot.h" and "mtbRobot.cpp"
Represents an MTB robot related to an MTB robot in the scene (see the figure above).
"attachCustomDataToSceneObject" and "readCustomDataFromSceneObject". Those two functions are in charge of transferring custom data between an MTB robot and its related plugin instance CMtbRobot. If the user changes the robot language program of a given MTB robot, then "attachCustomDataToSceneObject" will be called, in order to actualize the custom data (so that if the user saves the scene, the newest robot language program will also be saved).
"run": calling this function will execute the robot language program of the related MTB robot for "timeStep" seconds, then give control back to V-REP. The function itself will communicate (through socket or pipe communication) with an instance of "mtbServer" (see further below).
"outConnection.h" and "outConnection.cpp"
This is the code that handles the socket or pipe communication with "mtbServer". The communication is handled as a client.
"mtbRobotDialog.h" and "mtbRobotDialog.cpp"
This is the code that handles the MTB plugin dialog that can be accessed with [Menu bar --> Plugins --> MTB Robot Properties].
The actual robot program execution is handled by the "mtbServer" application. For each robot in the scene, an instance of "mtbServer" is launched, and connects to V-REP via socket communication. Following are the main files in the "mtbServer" project:
"inConnection.h" and "inConnection.cpp": handles the socket communication as a server:
"robotLanguageInterpreter.h" and "robotLanguageInterpreter.cpp": this is the actual code that executes a robot language program. It also holds the configuration state of an MTB robot.
"mtbServer.cpp": this code is in charge of communicating with the "v_repExtMtb" plugin and relaying commands to the robot language interpreter (see above)
Now let's have a look at the child script that is attached to the MTB robot. The code might seem quite long or complicated. However most functionality handled in the child script could also be directly handled in the plugin, making the child script much smaller/cleaner. The advantage in handling most functionality in the child script is that modifications can be performed without having to recompile the plugin!
Following is the MTB robot's child script main functionality:
Checking whether the plugin was loaded. If not, an error message is output.
Communicating with the plugin. This means that information is sent to and received from the MTB plugin with the custom Lua functions (see above).
Applying the newly calculated joint values to the MTB robot model. This could also be handled in the MTB's plugin.
Reacting to events on the custom dialogs (custom user interfaces), like button presses. This could also be handled in the MTB's plugin.
Updating the state of the custom dialogs (custom user interfaces), by changing their visual appearance (e.g. displaying the current joint positions, etc.). This could also be handled in the MTB's plugin.
The most important instruction is following:
result,cmdMessage,newJointPositions=simExtMtbRun(robotHandle,dt)
It will execute the robot language program attached to the MTB robot for dt seconds. It is called at each simulation pass, and the robot language program is executed in a stepped fashion.
If the robot language program interpreter could trigger intermediate events that should be handled by V-REP, the above instruction could be handled in an equivalent way with following instructions:
while (dt>0) do result,dt,cmdMessage,newJointPositions=simExtMtbRun(robotHandle,dt) event=simExtMtbGetEvent() while (event~=-1) do -- handle events here event=simExtMtbGetEvent() end end
In that case, the return values of the "simExtMtbRun" should include the remaining execution time, and a new custom Lua function should be registered to retrieve events ("simExtMtbGetEvent").
|