PROJECTS
Gesture-Based Control
Samsung Software Membership
Gesture-Based Control (FingerSnap)
Samsung Software Membership
Summary
- Goal: Control devices without touching them by using finger snaps (FingerSnap) detected via smartphone microphone and FFT. IoT-style platform: connect “switch” buttons to the network and control them with audio.
- Flow: Phone recognizes snap pattern → sends command via Bluetooth/HTTP → PC runs macros (lock, PPT advance, login, webtoon scroll) or hardware (lights, locks) reacts.
- Stack: Android (sensor, UI, Bluetooth), Windows WPF (macros, lock, auto-login), Server (OAuth, user/hardware management), Core (FFT + band-pass for snap detection), Hardware (Arduino + Bluetooth + servo for physical switches).
Project duration and schedule
- Duration: 2015.10.09 – 2016.01.12
- Phase 1 (2015.10.09 – 11.08): Planning and environment setup
- Phase 2 (2015.11.09 – 12.08): Core logic and UI
- Phase 3 (2015.12.09 – 2016.01.09): Integration and debugging
- Environment: Android Studio, Visual Studio
Purpose & background
- IoT: Objects get identity, connectivity, and control over a network (e.g. Smart Home, Smart Car). FingerSnap adds an IoT-style control layer using audio.
- Idea: Many products have physical switches; connect them to the network and control them with FingerSnap (no touch). Open-source API so others can plug in.
Architecture
- Core: FFT + band-pass filter libraries for snap detection; used by Android, Windows, and hardware.
- Server: OAuth, user IDs for registered hardware, permissions and grouping.
- Android: Mic input, frequency analysis, Bluetooth to hardware, HTTP to server; macro list and settings.
- Windows (WPF): Lock screen, PPT auto-advance, auto web login, webtoon scroll, user macros — triggered by FingerSnap from Android.
- Hardware: Receives commands from Android via Bluetooth; drives servos to actuate physical switches (lights, locks, etc.).
Technology in brief
- IoT: Sensors and connectivity in objects; need security and scalable data handling.
- Audio recognition: FFT on smartphone mic; FingerSnap has three distinct patterns (thumb–middle, middle–palm, thumb–index) in a short window.
- Smart Home / Smart Car: Examples of IoT; FingerSnap fits into “control without touch” use cases.
Development content
Server
- OAuth for app developers; user/hardware DB; permission and grouping. Demo and crowdfunding site planned.
Core (FFT + snap detection)
- FingerSnap pattern recognition via FFT; key enabler for remote control.
- Three snap types; algorithm detects pattern from phone mic. Plan: improve to distinguish users. Core is open for customization.
Android
- Mic → recognize snap → send command to hardware or server.
- Main screen: flip-style macro list; run by touch or snap.
- List view, lock-screen mode, button mode. Register product type (PC hardware or other), delegate permissions by phone number.
Windows WPF
- Main: Icons for lock, PPT, auto-login, webtoon scroll, user macros.
- Lock screen: FingerSnap or password.
- PPT: Auto-advance slides with FingerSnap.
- Auto login: e.g. Naver login with FingerSnap.
- Webtoon scroll: Scroll with FingerSnap.
- User macros: Custom actions triggered by FingerSnap.
Hardware
- Bluetooth link to Android. MCU + servo per switch; controls lights, door locks, etc. via predefined protocol.
Development budget (KRW)
| No | Part Name | Purpose | Price (KRW) | Qty | Total (KRW) |
|---|---|---|---|---|---|
| 1 | Arduino Pro Mini | MCU | 12,000 | 5 | 60,000 |
| 2 | USB Serial (FTDI-5V) | USB serial | 17,000 | 2 | 34,000 |
| 3 | USB for Serial | USB | 1,500 | 2 | 3,000 |
| 4 | Dymond D60 servo | Servo | 36,500 | 2 | 73,000 |
| 5 | KR-A2412BS-S1 | Servo | 30,800 | 1 | 30,800 |
| 6 | KR-A410BS | Servo | 19,800 | 1 | 19,800 |
| 7 | HES-188 metal | Servo | 14,520 | 3 | 43,560 |
| 8 | HES-1188 metal | Servo | 19,250 | 2 | 38,500 |
| 9 | 4xAA battery holder | Battery holder | 2,800 | 3 | 8,400 |
| 10 | Alkaline AA 1.5V | Battery | 600 | 12 | 7,200 |
| 11 | HC-06 (DIP) | Bluetooth | 13,000 | 3 | 39,000 |
| Total | 357,260 | ||||
References
- Apache Core Guide (Ben Laurie, Peter Laurie)
- WPF (Charles Petzold, tr. Se-Young Choi et al.)
- WPF Blog