• Project : Gesture-Based Control
• 2015.10.09 - 2016.01.12
• Samsung Software Membership

Gesture-Based Control

1. Project Duration

2015.10.09 - 2016.01.12

2. Purpose and Motivation

This project provides an IoT platform service based on audio recognition using Fast Fourier Transform (FFT). The goal is to include many products in the Internet of Things by connecting switch buttons attached to general products to the network and providing detachable hardware and platform APIs to users.

3. Development Environment

• IDE: Android Studio, Visual Studio

4. Development Schedule

• 2015.10.09 - 2015.11.08: Program planning and environment setup
• 2015.11.09 - 2015.12.08: Core functionality implementation and UI design
• 2015.12.09 - 2016.01.09: Program integration and debugging

5. Background

The Internet of Things (IoT) is the concept of giving objects the ability to identify themselves, communicate, and be controlled over a network. IoT has applications in personal, industrial, and public sectors. Examples include Smart Cars that connect vehicles to the internet for safe and convenient driving, and Smart Homes that connect household appliances to the internet. In the industrial sector, IoT enhances productivity and safety by integrating IoT technology into production, processing, and distribution. In the public sector, IoT information from CCTV, GPS for the elderly, etc., is used to prevent disasters.

FingerSnap aims to provide an IoT platform by connecting many objects to the network through switch buttons, enabling control without physical contact using audio recognition and making it applicable to many systems through open source.

6. Development Purpose

This project provides an IoT platform based on audio recognition using FFT. By controlling switch buttons, many objects can be connected to the network without modification and controlled without touch using audio recognition. It aims to provide open-source APIs applicable to many systems, allowing control through FingerSnap.

The goal is to enable a platform where many objects focus on the user, connect to the network, and communicate with each other.

7. Technology Trends

1) IoT (Internet of Things)

IoT refers to the technology of embedding sensors and communication functions in various objects to connect them to the internet. Examples include home appliances, mobile devices, wearable computers, and other embedded systems. Objects connected to IoT need unique IP addresses and may include sensors for data acquisition from the environment. IoT development must address the security challenges posed by the potential for all connected objects to be hacked.

According to Gartner, the number of objects using IoT technology was 900 million by 2009 and is expected to reach 26 billion by 2020. The vast amount of data generated by connected objects, known as Big Data, requires the development of efficient algorithms for analysis.

2) Audio Recognition

Recent advancements in audio recognition technology enable various applications. The widespread use of smartphones has made media recognition easier. Smartphones come equipped with microphones, sound devices, cameras, and networking capabilities, making them suitable for recording and processing sounds, capturing images, and exchanging data with remote servers. Examples include Google’s and Naver’s voice recognition programs, among other services.

3) Smart Home

A Smart Home supports automation and may include various automatic control techniques (lighting, temperature, doors, security systems, etc.). Smart Homes can be controlled via centralized controllers or wirelessly. They aim to enhance comfort, convenience, and energy efficiency in homes, making Smart Homes a critical part of the IoT landscape.

4) Smart Car

The convergence of automotive and IT technologies has accelerated the emergence of Smart Cars. These vehicles integrate LTE networks, app stores, and entertainment, information, and productivity features for a futuristic driving experience. Smart Cars aim to enhance safety and convenience by integrating mobile and telecommunication technologies. IT and automotive companies are collaborating to create new ecosystems. At CES 2014, several companies showcased their automotive technologies, highlighting the trends of connectivity, wearable devices, eco-friendly vehicles, autonomous driving, smartphone apps, and proprietary app stores, indicating the growing overlap between automobiles and consumer electronics.

Development Goals

1) Architecture

The architecture is divided into Core, Server, Android, Windows, and Hardware components.

• Core: Develop libraries for Fourier transform algorithms and band-pass filters. These libraries are used by Android, Windows, and Hardware code.
• Server: Store user IDs for registered hardware and manage OAuth processes.
• Android: Focus on user convenience, voice recognition, frequency analysis, Bluetooth communication with hardware, and HTTP communication to send user IDs to the server.
• Windows: Log on computers or implement system macros based on recognized voice input from Android (using Fourier transform and band-pass filter).
• Hardware: Perform actions based on signals received from Android. This role is suitable for the current transitional IoT market and will eventually be integrated into building design.

Development Content

1) Server

• It serves as a server to provide OAuth services. Application developers can easily integrate OAuth by applying the open-source API from this project.

• Users can manage user authentication for unique hardware modules and delegate permissions to other users for grouping purposes, using the database and server management.

• An official website will be developed and provided to introduce the open-source and services for crowdfunding, as well as to offer a demo application.

2) Core

• Using audio recognition with Fast Fourier Transform, FingerSnap patterns are recognized. This is a key module that enables remote control of the product in this project.

• FingerSnap produces three distinct snapping sounds: thumb and middle finger (first), middle finger and palm (second), and thumb and index finger (third). This pattern, which occurs in a short time, is unique to humans. An audio sensing algorithm was developed to detect this pattern using a smartphone. Although it currently detects simple captures, the plan is to upgrade the detection algorithm through crowdfunding to distinguish FingerSnap patterns between individuals.

• Additionally, users can not only provide OAuth to many systems but also customize the core to fit their system through open source.

3) Android

• The Android app recognizes audio sensors and controls hardware modules.

• The main screen allows you to view the contents of each macro in a flip format, similar to a music player. To execute a macro, you can either touch the image or snap your fingers.

• You can view all macros in a list format and set lock screen mode and button execution mode for macros in the settings.

• To register each macro, you can select the product type and register the product, whether it is PC hardware or other types. You can also delegate permissions to others using only the user’s phone number.

4) WPF

• 1) Main Screen:
  • 
  • The features in WPF include Windows lock screen, automatic PPT click, automatic web login system, automatic webtoon scroll, and user macro settings. Users can access these features by clicking the respective icons on the main screen.
• 2) Windows Lock Screen: Secure lock screen activated by FingerSnap or a pre-set password.
  • 
• 3) PPT Auto-Slide: Automatically advances slides during presentations with FingerSnap.
  • • 
• 4) Auto Login: Automatically logs into websites like Naver with FingerSnap.
  • 
• 5) Auto Webtoon Scroll: Scrolls webtoons with FingerSnap.
  • 
• 6) User Macro Settings: Allows users to set and execute macros with FingerSnap.
  • 

5) Hardware

Hardware communicates with Android via Bluetooth and allows users to control home devices like lights and door locks using FingerSnap. The system includes a motor for each switch button, an MCU, and a battery. The setup connects to Android via Bluetooth and allows control using predefined protocols.

Development Budget

No	Part Name	Purpose	Price (KRW)	Quantity	Total (KRW)
1	Arduino Pro Mini	Arduino mini board	12,000	5	60,000
2	USB Serial Converter (FTDI-5V type)	USB serial conversion	17,000	2	34,000
3	USB for Serial Converter	USB	1,500	2	3,000
4	Dymond D60 - super micro servo	Servo motor	36,500	2	73,000
5	KR-A2412BS-S1	Servo motor	30,800	1	30,800
6	KR-A410BS	Servo motor	19,800	1	19,800
7	HES-188 metal	Servo motor	14,520	3	43,560
8	HES-1188 metal	Servo motor	19,250	2	38,500
9	4xAA battery holder	Battery holder	2,800	3	8,400
10	Alkaline battery, AA size, 1.5V	Battery	600	12	7,200
11	Serial Port Module HC-06 (DIP)	Bluetooth communication	13,000	3	39,000
Total					357,260

References

Related Books

• Apache Core Guide by Ben Laurie and Peter Laurie
• WPF by Charles Petzold, translated by Se-Young Choi and others

Related Links

• WPF Blog