Hand Gesture Control: Movement of the Physical and Digital (Teleoperation)



Introduction: Hand Gesture Control: Movement of the Physical and Digital (Teleoperation)

Hey all,

In this Instructable, I am going to show you how to move a pan and tilt servo with hand movements or gestures through the concept of teleoperation. Also, I am going to show you how attaching a BNO055 orientation sensor to the servos will also move the digital component of this Instructable, which is optional. (If this sounds confusing, just look at the picture to get a sense of what is happening.)

Things you will need (I bought and received these as gifts from the Adafruit Industries: https://www.adafruit.com):

- Access to Processing and Arduino IDE's

- Two Arduino UNO boards https://www.adafruit.com/?q=ardino%20UNO& OR


- Two Breadboards https://www.adafruit.com/products/64

- Jumpers https://www.adafruit.com/products/153

- Pan and Tilt Servos https://www.adafruit.com/products/1967

- LEAPMotion Device https://www.leapmotion.com/ or https://www.adafruit.com/products/2106

Step 1: Set Up Servos

Once you have all the materials necessary, download the LEAPMotion SDK and IDE's for Processing and Arduino. Using the jumpers, wire the Arduino correctly. I referenced the Fritzing schematic in the image above. Depending on which digital pinout (2-13) receives the yellow servo jumper should be noted in the Standard Firmata Arduino Sketch (in the picture, it's 2, 3. Also, be sure that it matches in the IDE/Sketches).

Step 2: Set Up Orientation Device (Optional If Digital Component Is Included in Your Project)

Thanks to Kevin Townsend for the Adafruit wiring picture.

I am going to refer to the Adafruit Adafruit BNO055 Absolute Orientation Sensor Learn page for this part of the project. It's by Kevin Townsend, here is the link.

I attached the sensor to the servos using male-to-female jumpers which is how the sensor is extending from the breadboard. You might do the same depending on your own goals or outcomes of the project.

Step 3: Prepare the Arduino IDE's

1) The very first IDE is called Standard Firmata which is responsible for connecting the data from a microcontroller (Arduino) to host computer (your PC or Mac). This communication is necessary for this sketch because data from the LEAPMotion Processing Sketch will translate that data from the LEAPMotion to the pan-and-tilt servos, hopefully moving the two to the motion of your hand. Run this .ino file first.

The "Standard Firmata" sketch can be found in the "Firmata" examples drop-down menu of the IDE. Make sure the pinouts on the Arduino board are the ones that are in the IDE (it's 2 and 3 now, but you can change that depending on your goals).

One thing to note is that you must make sure that the serial port is correctly synced with the servo Arduino. You can find "ports" in the "tools" drop-down menu of the IDE (when the Arduinos are plugged, they should be labeled as COM1 or COM 2 or something like that. If you are having trouble with this, go to your device manager, find the Arduino, and right-click on them and change the properties so the proper COM serial port is showing. If there is any other troubles refer to this guide: https://www.arduino.cc/en/Guide/Troubleshooting).

2) The second Arduino IDE can be found here https://github.com/adafruit/Adafruit_BNO055. Be sure that you extract the files into the proper place of your PC so that when it is called in the IDE, there are no errors when it is running. Run this .ino file next.

Thank you, Tony Dicola.

Again, this is only necessary if you want a digital aspect of this tutorial.

Step 4: Set Up Processing Sketches (the Visual Aspect)

So here comes the visual aspect of this tutorial.

Once you have everything wired and connected, run the Arduino sketches.

Run Standard Firmata.ino and Bunny.ino file and make sure no errors pop up (make sure the correct serial port are established).

With processing, make sure you download it https://processing.org/download and have these libraries installed.

Arduino (Should be in the Processing library downloads folder).

LeapMotion P5 (for reference of fingers and hand: https://github.com/mrzl/LeapMotionP5)

for digital component (optional)...

saito obj loader (also, have a 3D model. Should be in the library folder for Processing).

1) Here is the LEAPMotion sketch that I have been using with minor edits(such as the com serial port and 3D model for instance). This sketch runs with the Standard Firmata Arduino IDE from the previous step. Again, make sure that files are in the correct assets directory so that when you call it in the sketch, everything runs smoothly. This is the LEAPMotion gesture control that moves the pan and tilt servos. Run this sketch.

2) Here is the sketch that takes care of the sensor information which dictates the orientation of the 3D model. This sketch runs when the bunny.ino file is already running from the previous step. Finally, run this sketch. Make sure that this matches the COM port attached to the bunny.ino from the previous step. This will translate the data from your hand that the LEAPMotion is recording into the servos and the sensor/digital aspect of this tutorial as demonstrated in the GIFS.

Step 5: Things to Remember and Other Stuff...

Just make sure the COM Ports in the Arduino IDE's and the device manager are defined and remember which Arduino pairs with which Processing sketch. All in all, there is only 2 Processing sketches and 2 Arduino IDE's. Make sure your assets are in their proper settings so calling them should not be difficult.

I had fun making this and there were some struggles, obviously, this is a rather meticulous process but it functions and works. Just like all of you reading this, I am still learning.

Feel free to ask questions and I will do my best to answer them. Also, post your successes and/or critiques/criticisms.

If there are any problems with Processing or Arduino, consider downloading previous versions. I had success with Processing 2 but I believe any Arduino version should work.

**Final note, check out my winning solution for John Deere's quest for "connected robotic construction equipment" over on Mindsumo. In my solution, I explain how "motion capture can act as the necessary input of human motion and simultaneously command both backhoe and excavator systems" using the techniques and processes explained in this Instructable. I believe this Instructable demonstrates the capacity of human integration with automation and acts as a perfect physical computing study for anyone who wants to learn more this developing field.

Be the First to Share


    • Anything Goes Contest

      Anything Goes Contest