75F Machine Learning Engineer Madhushan Tennakoon and Software Intern Myat Mo created a program together called "Deep See," originally developed as part of the 4th Annual IoT Fuse Hack Day. Deep See utilizes the intelligence of Amazon's Alexa to help you see. Madhushan and Myat presented a track on Deep See at IoT Fuse Conference on May 3rd, 2018. It fascinated a lot of the audience who attended their presentation; however, with any live demo comes the risk that something won't cooperate. We were thankful for the forgiving audience that waited patiently in anticipation. So, we decided to film a brief video segment of the Deep See demo at the 75F office and share it here for everyone to be able to experience this amazing passion project by Madhushan and Myat.
How does it work?
This project was based on the premise that Alexa can be made smarter than she already is. Alexa is very sophisticated with voice assistance. She's already been trained on the most complex data set known to us, which is human speech. She can hear what you say and synthesize speech in an almost human like manner. But, what if she could also see?
For this project, the team used Raspberry Pi connected to a camera that is also connected to a pair of glasses. The glasses will take a snapshot when you ask Alexa a question. Then the Raspberry Pi will upload the image to AWS, where it'll do it's analytics. For example, if you wanted to find a friend in the crowd she'll use AWS recognition to scan all the faces and match them in an internal database. You can ask her questions like "what mood is my friend in?" or "what do you see?" and Alexa will describe what she sees in a narrative.