Vvim - Keyboardless Vim interactions

Related tags

Hardwarevvim
Overview

Vvim - Keyboardless Vim interactions

This is done via a hardware glove that the user wears. The glove detects the finger's positions and translates them into key presses. It's currently a work in progress.

The glove prototype, with 4 sensors on two fingers

Subset of data

The stream of data from the 4 sensors (here each shown in a different colour) has been zeroed so that they all centre around the time when the user pressed the 'y' key.

Current Features

  • Glove prototype has been constructed.

  • Glove can detect finger movements of the right fore finger and right middle finger (With space to expand to more fingers if these first two actually work)

    • This corresponds to the following keys, shown with how often those keys show up in the current dataset: h: 628, u: 291, y: 171, m: 171, b: 155, k: 120, j: 21,
  • Glove records finger movements via an Arduino script vvim.ino on an Uno, and sends them to serial output.

  • Serial output is read by the python script glove_logger.py and saved to the file glove.log along with the Unix milliseconds since epoch.

  • A keylogger is installed on the developer's machine, and logs key presses to the file keys.log along with Unix milliseconds since epoch.

  • Running cleanup.sh cleans up the data from the keylogger and the serial output into one file named sorted.log.

  • A Gradient Boosted tree has been trained and saved to model.pkl. Currently it has a test accuracy of 79.7%.

    • This will hopefully be improved as more data is gathered, as currently there are only 587 keypresses on which to train 9 categories, or about 65 examples per category which is not enough.
  • The file eda.py saves plots to plots/ such as:

Graphs

Each colour is a differently positioned sensor. Each line is one stream of data recorded by a sensor. The streams have each been zeroed so that every instance of pressing a certain key is centred.

Keys on the home row

Some keys are easier to spot, and others less so as my fingers move a lot when pressing a y compared to a k just because of where the keys are positioned on the keyboard.

More or less data

The data has not been normalised, so there's far more data for when common keys like h are pressed compared to when a j is pressed

In Progress

  • Currently there are only about 600 keypresses recorded. Record more examples of typing and add more sensors to the fingers so that fewer keystrokes have to be typed in order to get the data.

To Do

  • If flex sensors aren't enough to predict exactly when a key is pressed, add force sensors to the fingertips.
  • Use an Arduino Nano instead of an Uno, and host the entire thing on the user's hand
  • Connect the glove to the computer via Bluetooth, instead of a wired connection
  • Current models don't have the option of categorizing an sequence of sensor readings as not pressing any key at all. This should be fixed so the model isn't constantly assuming at least one key is being pressed
    • This could be done easily with pressure sensors
  • Write some sort of visualiser to live track sensor data, actual key presses, and predicted key presses

Keys and which finger tends to press them

Note that this list is likely very specific to the author, as different people will type differently. I think I probably use my right ring finger much more than I really should. Also I type a y with my index finger for words like type or you (where I subsequently have to type another letter with me right hand), but I type it with my middle finger for words like yes, yank, or keyboard.

  • Right Hand
    • Thumb: space
    • Index: j, m, n, b, h, y
    • Middle: k, y, u, i, <, (, [
    • Ring: l, :, BACKSPACE, o, p, >, ), ], 0, _, -, +, =, ,, .
    • Pinky: ;, ENTER, /, ?
  • Left Hand (Incomplete as I've not yet built a glove for the left hand)
    • Pinky:
    • Ring:
    • Middle:
    • Index:
    • Thumb:

Here's a picture of my keyboard for reference:

How to Start Recording Data

Probably best to do this all in tmux since handling multiple terminal windows is a pain otherwise. A keylogger (I use Casey Scarborough's keylogger) is also required.

  1. Install requirements
pip3 install -r requirements.txt
  1. Run the command to clear the logfile:
sudo keylogger clear
  1. Start the keylogger:
sudo keylogger ./keys.log
  1. Start recording glove movements:
python3 glove_logger.py
  1. Put the glove on, and start typing things out. I usually do this by opening a text file (like Alice in Wonderland available on Gutenberg) in vim (vim alice.txt), and then splitting the window vertically (:vsp), and then opening a temporary file in which to type in (:e tmp). Finally, type (:set cursorbind) into both frames so that the source text scrolls as you type it. They keystrokes and finger movements will be recorded separately

  2. Remove the glove

  3. Stop the keylogger with CTRL-C

  4. Stop recording the finger movements with CTRL-C

  5. Now the data is recorded, clean it up:

./cleanup.sh
  1. And analyse the data with eda.py
python3 eda.py

The images will be stored to plots/ for your viewing pleasure

License

This work is licensed under GNU GPLv3. See the attached LICENSE. See https://choosealicense.com/licenses/gpl-3.0/# for a non-legalese explanation of the license.

Owner
Boyd Kane
CS and Statistics student at UCT, South Africa. Interested in data science, probability theory and speaking Spanish. ¡Hola!
Boyd Kane
Drobo Status is a python program that will connect to your Drobo and return JSON data regarding your Drobo

This is a simple python script that will run a docker container to pull data from Drobo. It will give information like (Name, serial, firmware, disk-total, disk-used, disk-free and individual disk st

Biofects 1 Jan 15, 2022
Beam designs for infinite Z 3D printers

A 3D printed beam that is as stiff as steel A while ago Naomi Wu 机械妖姬 very kindly sent us one of Creality's infinite-Z belt printers. Lots of people h

RepRap Ltd 105 Oct 22, 2022
Python code written to utilize the Korlan usb2can hardware to send and receive data over the can-bus on a 2008 Nissan 350z

nissan_ecu_hacking Python code written to utilize the Korlan usb2can hardware to send and receive data over the can-bus on a 2008 Nissan 350z My goal

Liam Goss 11 Sep 24, 2022
Estimation of whether or not the persons given information will have diabetes.

Diabetes Business Problem : It is desired to develop a machine learning model that can predict whether people have diabetes when their characteristics

Barış TOKATLIOĞLU 0 Jan 20, 2022
Implemented robot inverse kinematics.

robot_inverse_kinematics Project setup # put the package in the workspace $ cd ~/catkin_ws/ $ catkin_make $ source devel/setup.bash Description In thi

Jianming Han 2 Dec 08, 2022
PlatformIO development platform for GSM modules

PlatformIO development platform for GSM modules Supported Modules Quectel M66 OpenCPU Arduino - TODO other - in progress... Supported Boards Comet M66

Georgi Angelov 5 Aug 06, 2022
Home Assistant custom components MPK-Lodz

MPK Łódź sensor This sensor uses unofficial API provided by MPK Łódź. Configuration options Key Type Required Default Description name string False MP

Piotr Machowski 3 Nov 01, 2022
Controlling fireworks with micropython

Controlling-fireworks-with-micropython How the code works line 1-4 from machine

Montso Mokake 1 Jan 08, 2022
BoneIO is a compact IO controller for home automation.

Project description BoneIO is a compact IO controller for home automation. Main features of this controller are Compact size (27x11x6)cm - 15 DIN modu

Maciej Krasuski 120 Nov 30, 2022
Playing diabolo with two robot arms in ROS + Gazebo

Playing diabolo with robots This repository holds the ROS packages for playing diabolo with two UR5e robot arms on ROS Melodic (Ubuntu 18.04). Read ou

23 Dec 18, 2022
Python library for the Phomemo m02s bluetooth thermal printer

Phomemo M02S Python library This is a basic Python library for controlling the Phomemo M02S bluetooth thermal printer. It probably only works on Mac &

Stargirl Flowers 28 Nov 07, 2022
A Python program that makes it easy to manage modules on a CircuitPython device!

CircuitPython-Bundle-Manager-v2 A Python program that makes it easy to manage modules on a CircuitPython device! The CircuitPython Bundle Manager v2 i

Ckyiu 1 Dec 18, 2021
Segger Embedded Studio project for building & debugging Flipper Zero firmware.

Segger Embedded Studio project for Flipper Zero firmware Установка Добавить данный репозиторий в качестве сабмодуля в корень локальной копии репозитор

25 Dec 28, 2022
Create (templateable) cameras that display qr codes in homeassistant

QRCam This custom component creates cameras displaying qrcodes. The QRCodes can be static or generated from templates. If you use a template as conten

Jannes Müller 5 Oct 06, 2022
DIY split-flap display

The goal is to make a low-cost display that's easy to fabricate at home in small/single quantities (e.g. custom materials can be ordered from Ponoko or similar, and other hardware is generally availa

Scott Bezek 2.5k Jan 05, 2023
Robo Arm :: Rigging is a rigging addon for Blender that helps animating industrial robotic arms.

Robo Arm :: Rigging Robo Arm :: Rigging is a rigging addon for Blender that helps animating industrial robotic arms. It construct serial links(a kind

2 Nov 18, 2021
My self-hosting infrastructure, fully automated from empty disk to operating services

Khue's Homelab Current status: ALPHA This project utilizes Infrastructure as Code to automate provisioning, operating, and updating self-hosted servic

Khue Doan 6.4k Dec 31, 2022
🎃 Some spooky code samples to hack yourself a pumpkin 👻

🎃 Tech Or Treat 👻 It's spooky season for those who celebrate Halloween, and to get in the spirit (spirit - get it? 👻 ) we thought it would be fun t

Jim Bennett 5 Feb 07, 2022
A custom mechanical keyboard inspired by the CFTKB Mysterium

Env-KB A custom mechanical keyboard inspired by the CFTKB Mysterium Build Guide and Parts List What is to do? Right now for the first 5 PCBs I have, i

EnviousData 203 Jan 04, 2023
Resmed_myair_sensors - This is a Home Assistant custom component to pull daily CPAP data from ResMed's myAir service using an undocumented API

resmed_myair This component will set up the following platforms. Platform Description sensor Show info from the myAir API. Installation Using the tool

Preston Tamkin 17 Dec 29, 2022