SlideShare a Scribd company logo

Hacking for salone: drone races

•
•
Emanuele Di Saverio
Emanuele Di Saverio

Drone Race game presented at frog Milan event for the Salone del Mobile 2016, and discussed at Codemotion Milan 2016

Related slideshows

AiRaid: Rise of the Undead
AiRaid: Rise of the UndeadAiRaid: Rise of the Undead
AiRaid: Rise of the Undead
 
Android and OpenNI - NUI Application Treasure Hunter Robot
Android and OpenNI - NUI Application Treasure Hunter RobotAndroid and OpenNI - NUI Application Treasure Hunter Robot
Android and OpenNI - NUI Application Treasure Hunter Robot
 
How to Develop VR Applications and Publish from Phone to Dome - Nils Andersson
How to Develop VR Applications and Publish from Phone to Dome - Nils AnderssonHow to Develop VR Applications and Publish from Phone to Dome - Nils Andersson
How to Develop VR Applications and Publish from Phone to Dome - Nils Andersson
 
1 of 33
HACKING FOR SALONE: DRONE RACES EMANUELE / SIMONE
EMANUELE DI SAVERIO PRINCIPAL DESIGN TECHNOLOGIST Emanuele started writing software for mobile in the pre-iPhone era. Didn’t stop since then. WHO’S ON THE STAGE SIMONE LIPPOLIS SENIOR DESIGN TECHNOLOGIST Simone started writing software when COBOL was a hype word. Now his interests span from web-bots to data-visualization to embedded applications and IOT.
EVERYTHING STARTED ON A BLACK FRIDAY
WE NEEDED MORE ENTERTAININ TO MAKE IT
A CLASSIC GAME CAME IN OUR HELP 7
Hacking for salone: drone races
Hacking for salone: drone races
Hacking for salone: drone races
TODO LIST TALK TO EACH OTHER SENSE EACH OTHER HARDER, BETTER, FASTER, STRONGER
TALK TO EACH OTHER
COMPONENTS ARCHITECTURE • Drones on wheel (JumpingSumo by Parrot) • An open-security 5 Ghz Wi-Fi AP • Android Smartphones • A WebSocket Server • A racing track Controller Mobile App JumpingSumo linux+busybox Game Server NodeJS Controller Mobile App JumpingSumo linux+busybox Controller Mobile App JumpingSumo linux+busybox WebSocket . . .
The Parrot robots work by creating an 802.11ac themselves - smartphones can connect to that and control though applications built with the Parrot SDK. This disables internet access for the smartphones - we needed to “hack” it. HACKING CONNECTION $ telnet 192.168.1.1 accesses the BusyBox environment to edit config files /etc/init.d/init_manager and dragon.conf Drones will connect to your open WiFi at reboot, where they will find Android phones.
The server doesn’t need to provide much functionality, but rather route events in a soft- realtime fashion to the players, with litte computation. • WebSockets help avoid TCP RTT • nodeJS because it can scale to gazillions of events We have 4 players at at time :) A BROKER SERVER
SENSE EACH OTHER
WHERE IS MY DRONE? BLE BEACON/FINGERPRINTING IR MARKER TANGO
LET DRONES “SEE” The hardware platform features a front camera for still photos and streaming. Specs not great - we’ll need to design around. Camera Spec 640x480 15 FPS Fixed focus Motion JPEG
COLOR BLOBS Using OpenCV as resource for highly-optimized algorithm library, we tried to detect color blobs by • thresholding • finding contours • centroids and applying some heuristics
FIDUCIAL MARKERS We also res 20 B&W Square markers are much more robust and can apply adequate Hamming-distance encodings. As an added benefit, they are more robust with respect to luminosity.
Hacking for salone: drone races
IMAGE PROCESSING PIPELINE PARROT SDK IMAGE RE-ENCODE ARTOOLKIT MARKER COORDINATES Each of the incoming frames from the drones has to go through heavy processing to detect a marker. 66 ms 3D MODEL ROTOTRASLATE COORDINATES
HARDER, BETTER, FASTER, STRONGER
OUT OF THE JAVALAND First problem relates to the sheer computation power needed to conver binary image formats - Java implementation is just too slow. We had to port algorithm implementation to RenderScript to have a fully native implementation that performs acceptably. JVM RS RS_MULTI AVG 283.60 1.73 3.24 STDDEV 38.17 2.49 1.52 MEDIAN 262.11 0.86 2.78 MAX 447.57 10.04 15.18
OUT OF THE GC LAND The main source of delays in real-time applications like games is the Garbage Collector - we’re talking half second freeze on recent hardware. This means we cannot allocate memory in the game loop - Android memory profiler tools will help you discover and eliminate: • Bitmap.create() • new Array[] • Object creations / Autoboxing
FULL CONTROL FLOW PARROT SDK (CAMERA) IMAGE RE-ENCODE ARTOOLKIT MARKER COORDINATES 3D MODEL ROTOTRASLATE COORDINATES GAME STATE NETWORK EVENTS GAS PEDAL USER INTERFACE STEERING (GYRO) BROKER SERVER OTHER PLAYERS ACTIONS
FULL CONTROL FLOW PARROT SDK IMAGE RE-ENCODE ARTOOLKIT MARKER COORDINATES 3D MODEL ROTOTRASLATE COORDINATES GAME STATE NETWORK EVENTS GAS PEDAL USER INTERFACE STEERING BROKER SERVER 66 ms 33 ms ~ 15 s 16 ms ~ 50 ms STEERING (GYRO) OTHER PLAYERS ACTIONS
Handling all the different events coming in at different rates would a very complex task to handle in a fully procedural way in a standard game loop: REACTIVE STREAMS FTW while (true) { acquire_image(); process_markers(); get_steer_input(); get_other_players_input(); apply_cmds_to_drone(); apply_game_events(); update_ui(); } . . . .
REACTIVE STREAMS FTW Rx provides common semanthics to express and manipulate a collection of asynchronous events
RXJAVA http://www.slideshare.net/hazam/programming-sideways-asynchronous-techniques-for-android (2014) Plenty of good introductions to RxJava in the InterWebs Exercises https://github.com/jhusain/learnrxjava
KOTLIN MAY BE THE ONE We may have found a winner. Kotlin is a deep language with many modern features, including Functional approaches. The three best features for Kotlin on Android: 1. Overhead is at compile time, Runtime Overhead is minimal 2. Closures and SAM 3. Java Interoperability is a breeze One things I’m still undecided if is good or bad: Nullability FUNCTIONAL REACTIVE PROGRAMMING WITH KOTLIN ON ANDROID - BY GIORGIO NATILI
Hacking for salone: drone races
REFERENCES https://github.com/frog/a-kart http://designmind.frogdesign.com/2016/07/ha cking-a-multi-game-user-drone-race/ SIMONE LIPPOLIS EMANUELE DI SAVERIO @simonelippolis @hazam GAME & UI DESIGN, SERVER DEVELOPMENT MOBILE DEVELOPMENT, COMPUTER VISION FEDERICO BRESSAN ELENA MARENGONI RACETRACK, DRONE SKIN DESIGN RACETRACK, DRONE SKIN DESIGN
THANKS!
THANKS! www.frogdesign.com

More Related Content

Hacking for salone: drone races

  • 1. HACKING FOR SALONE: DRONE RACES EMANUELE / SIMONE
  • 2. EMANUELE DI SAVERIO PRINCIPAL DESIGN TECHNOLOGIST Emanuele started writing software for mobile in the pre-iPhone era. Didn’t stop since then. WHO’S ON THE STAGE SIMONE LIPPOLIS SENIOR DESIGN TECHNOLOGIST Simone started writing software when COBOL was a hype word. Now his interests span from web-bots to data-visualization to embedded applications and IOT.
  • 3. EVERYTHING STARTED ON A BLACK FRIDAY
  • 5. A CLASSIC GAME CAME IN OUR HELP 7
  • 9. TODO LIST TALK TO EACH OTHER SENSE EACH OTHER HARDER, BETTER, FASTER, STRONGER
  • 11. COMPONENTS ARCHITECTURE • Drones on wheel (JumpingSumo by Parrot) • An open-security 5 Ghz Wi-Fi AP • Android Smartphones • A WebSocket Server • A racing track Controller Mobile App JumpingSumo linux+busybox Game Server NodeJS Controller Mobile App JumpingSumo linux+busybox Controller Mobile App JumpingSumo linux+busybox WebSocket . . .
  • 12. The Parrot robots work by creating an 802.11ac themselves - smartphones can connect to that and control though applications built with the Parrot SDK. This disables internet access for the smartphones - we needed to “hack” it. HACKING CONNECTION $ telnet 192.168.1.1 accesses the BusyBox environment to edit config files /etc/init.d/init_manager and dragon.conf Drones will connect to your open WiFi at reboot, where they will find Android phones.
  • 13. The server doesn’t need to provide much functionality, but rather route events in a soft- realtime fashion to the players, with litte computation. • WebSockets help avoid TCP RTT • nodeJS because it can scale to gazillions of events We have 4 players at at time :) A BROKER SERVER
  • 15. WHERE IS MY DRONE? BLE BEACON/FINGERPRINTING IR MARKER TANGO
  • 16. LET DRONES “SEE” The hardware platform features a front camera for still photos and streaming. Specs not great - we’ll need to design around. Camera Spec 640x480 15 FPS Fixed focus Motion JPEG
  • 17. COLOR BLOBS Using OpenCV as resource for highly-optimized algorithm library, we tried to detect color blobs by • thresholding • finding contours • centroids and applying some heuristics
  • 18. FIDUCIAL MARKERS We also res 20 B&W Square markers are much more robust and can apply adequate Hamming-distance encodings. As an added benefit, they are more robust with respect to luminosity.
  • 20. IMAGE PROCESSING PIPELINE PARROT SDK IMAGE RE-ENCODE ARTOOLKIT MARKER COORDINATES Each of the incoming frames from the drones has to go through heavy processing to detect a marker. 66 ms 3D MODEL ROTOTRASLATE COORDINATES
  • 22. OUT OF THE JAVALAND First problem relates to the sheer computation power needed to conver binary image formats - Java implementation is just too slow. We had to port algorithm implementation to RenderScript to have a fully native implementation that performs acceptably. JVM RS RS_MULTI AVG 283.60 1.73 3.24 STDDEV 38.17 2.49 1.52 MEDIAN 262.11 0.86 2.78 MAX 447.57 10.04 15.18
  • 23. OUT OF THE GC LAND The main source of delays in real-time applications like games is the Garbage Collector - we’re talking half second freeze on recent hardware. This means we cannot allocate memory in the game loop - Android memory profiler tools will help you discover and eliminate: • Bitmap.create() • new Array[] • Object creations / Autoboxing
  • 24. FULL CONTROL FLOW PARROT SDK (CAMERA) IMAGE RE-ENCODE ARTOOLKIT MARKER COORDINATES 3D MODEL ROTOTRASLATE COORDINATES GAME STATE NETWORK EVENTS GAS PEDAL USER INTERFACE STEERING (GYRO) BROKER SERVER OTHER PLAYERS ACTIONS
  • 25. FULL CONTROL FLOW PARROT SDK IMAGE RE-ENCODE ARTOOLKIT MARKER COORDINATES 3D MODEL ROTOTRASLATE COORDINATES GAME STATE NETWORK EVENTS GAS PEDAL USER INTERFACE STEERING BROKER SERVER 66 ms 33 ms ~ 15 s 16 ms ~ 50 ms STEERING (GYRO) OTHER PLAYERS ACTIONS
  • 26. Handling all the different events coming in at different rates would a very complex task to handle in a fully procedural way in a standard game loop: REACTIVE STREAMS FTW while (true) { acquire_image(); process_markers(); get_steer_input(); get_other_players_input(); apply_cmds_to_drone(); apply_game_events(); update_ui(); } . . . .
  • 27. REACTIVE STREAMS FTW Rx provides common semanthics to express and manipulate a collection of asynchronous events
  • 28. RXJAVA http://www.slideshare.net/hazam/programming-sideways-asynchronous-techniques-for-android (2014) Plenty of good introductions to RxJava in the InterWebs Exercises https://github.com/jhusain/learnrxjava
  • 29. KOTLIN MAY BE THE ONE We may have found a winner. Kotlin is a deep language with many modern features, including Functional approaches. The three best features for Kotlin on Android: 1. Overhead is at compile time, Runtime Overhead is minimal 2. Closures and SAM 3. Java Interoperability is a breeze One things I’m still undecided if is good or bad: Nullability FUNCTIONAL REACTIVE PROGRAMMING WITH KOTLIN ON ANDROID - BY GIORGIO NATILI
  • 31. REFERENCES https://github.com/frog/a-kart http://designmind.frogdesign.com/2016/07/ha cking-a-multi-game-user-drone-race/ SIMONE LIPPOLIS EMANUELE DI SAVERIO @simonelippolis @hazam GAME & UI DESIGN, SERVER DEVELOPMENT MOBILE DEVELOPMENT, COMPUTER VISION FEDERICO BRESSAN ELENA MARENGONI RACETRACK, DRONE SKIN DESIGN RACETRACK, DRONE SKIN DESIGN

Editor's Notes

  1. We didn’t really plan to buy a drone, it just happened that on a black friday we found one of these toys super discounted on amazon. It wa awesome: a little piece of technology at such a small price, the best way to make a first step into the drone world. As you can see in the video (BTW, is a short clip from an official Parrot’s ad) playing with these things is super funny. But the excitement lasts the time of a battery charge: more or less 20 minutes. If we wanted to use it more, to make it more interesting we needed to do just one thing…
  2. … We thought that you really have fun when you share an experience, and we spent an entire afternoon trying to think how we could improve the experience…
  3. … and at the end, the idea came from a classic game. Wit has everything we needed: speed, competition between different players, it’s super funny and almost everyone knows what it is.
  4. What we are going to tell you is how we transformed a consumer technologic toy into a merciless racer
  5. And a single user toy into a multiuser experience. We obviously had to face a lot of challenges, but as you’ll see we managed to face them in an effective way
  6. This short video shows you how the project evolved, from the first tests to the latest race we help. Because this project that started with an impulsive purchase, at the end, became the main attraction at frog’s Salone Del Mobile party last year. And we’re pretty proud of it.
  7. Machine Learning approaches like Convolutional Neural Networks are on the rise in latest years, but still require robust pre-processing layers (thresholding, centering) and massive a mount of data for training. Even having a trained NN, evaluation can be tricky to perform in Real Time on a mobile device. Marker recognition is a robust technique that requires instrumentation of objects, but is very cheap to implement and fast - algorithms can be ran in real time. Relaxing the position requirement from absolute to relative, we can also avoid complex set up of the environment. Computer vision libraries like OpenCV allow to implement marker detection, but Augmented Reality frameworks include a recognition stage.
  8. argb_to_yuv.rs
  9. CachedBitmapDecoder.kt Controller.mediaStreamer()
  10. PlayActivity.kt Contrller.kt
  11. BmpToYUVToARToolkitConverterRS2.java