Vvim - Keyboardless Vim interactions

Related tags

Hardwarevvim
Overview

Vvim - Keyboardless Vim interactions

This is done via a hardware glove that the user wears. The glove detects the finger's positions and translates them into key presses. It's currently a work in progress.

The glove prototype, with 4 sensors on two fingers

Subset of data

The stream of data from the 4 sensors (here each shown in a different colour) has been zeroed so that they all centre around the time when the user pressed the 'y' key.

Current Features

  • Glove prototype has been constructed.

  • Glove can detect finger movements of the right fore finger and right middle finger (With space to expand to more fingers if these first two actually work)

    • This corresponds to the following keys, shown with how often those keys show up in the current dataset: h: 628, u: 291, y: 171, m: 171, b: 155, k: 120, j: 21,
  • Glove records finger movements via an Arduino script vvim.ino on an Uno, and sends them to serial output.

  • Serial output is read by the python script glove_logger.py and saved to the file glove.log along with the Unix milliseconds since epoch.

  • A keylogger is installed on the developer's machine, and logs key presses to the file keys.log along with Unix milliseconds since epoch.

  • Running cleanup.sh cleans up the data from the keylogger and the serial output into one file named sorted.log.

  • A Gradient Boosted tree has been trained and saved to model.pkl. Currently it has a test accuracy of 79.7%.

    • This will hopefully be improved as more data is gathered, as currently there are only 587 keypresses on which to train 9 categories, or about 65 examples per category which is not enough.
  • The file eda.py saves plots to plots/ such as:

Graphs

Each colour is a differently positioned sensor. Each line is one stream of data recorded by a sensor. The streams have each been zeroed so that every instance of pressing a certain key is centred.

Keys on the home row

Some keys are easier to spot, and others less so as my fingers move a lot when pressing a y compared to a k just because of where the keys are positioned on the keyboard.

More or less data

The data has not been normalised, so there's far more data for when common keys like h are pressed compared to when a j is pressed

In Progress

  • Currently there are only about 600 keypresses recorded. Record more examples of typing and add more sensors to the fingers so that fewer keystrokes have to be typed in order to get the data.

To Do

  • If flex sensors aren't enough to predict exactly when a key is pressed, add force sensors to the fingertips.
  • Use an Arduino Nano instead of an Uno, and host the entire thing on the user's hand
  • Connect the glove to the computer via Bluetooth, instead of a wired connection
  • Current models don't have the option of categorizing an sequence of sensor readings as not pressing any key at all. This should be fixed so the model isn't constantly assuming at least one key is being pressed
    • This could be done easily with pressure sensors
  • Write some sort of visualiser to live track sensor data, actual key presses, and predicted key presses

Keys and which finger tends to press them

Note that this list is likely very specific to the author, as different people will type differently. I think I probably use my right ring finger much more than I really should. Also I type a y with my index finger for words like type or you (where I subsequently have to type another letter with me right hand), but I type it with my middle finger for words like yes, yank, or keyboard.

  • Right Hand
    • Thumb: space
    • Index: j, m, n, b, h, y
    • Middle: k, y, u, i, <, (, [
    • Ring: l, :, BACKSPACE, o, p, >, ), ], 0, _, -, +, =, ,, .
    • Pinky: ;, ENTER, /, ?
  • Left Hand (Incomplete as I've not yet built a glove for the left hand)
    • Pinky:
    • Ring:
    • Middle:
    • Index:
    • Thumb:

Here's a picture of my keyboard for reference:

How to Start Recording Data

Probably best to do this all in tmux since handling multiple terminal windows is a pain otherwise. A keylogger (I use Casey Scarborough's keylogger) is also required.

  1. Install requirements
pip3 install -r requirements.txt
  1. Run the command to clear the logfile:
sudo keylogger clear
  1. Start the keylogger:
sudo keylogger ./keys.log
  1. Start recording glove movements:
python3 glove_logger.py
  1. Put the glove on, and start typing things out. I usually do this by opening a text file (like Alice in Wonderland available on Gutenberg) in vim (vim alice.txt), and then splitting the window vertically (:vsp), and then opening a temporary file in which to type in (:e tmp). Finally, type (:set cursorbind) into both frames so that the source text scrolls as you type it. They keystrokes and finger movements will be recorded separately

  2. Remove the glove

  3. Stop the keylogger with CTRL-C

  4. Stop recording the finger movements with CTRL-C

  5. Now the data is recorded, clean it up:

./cleanup.sh
  1. And analyse the data with eda.py
python3 eda.py

The images will be stored to plots/ for your viewing pleasure

License

This work is licensed under GNU GPLv3. See the attached LICENSE. See https://choosealicense.com/licenses/gpl-3.0/# for a non-legalese explanation of the license.

Owner
Boyd Kane
CS and Statistics student at UCT, South Africa. Interested in data science, probability theory and speaking Spanish. ¡Hola!
Boyd Kane
Code for the paper "Planning with Diffusion for Flexible Behavior Synthesis"

Planning with Diffusion Training and visualizing of diffusion models from Planning with Diffusion for Flexible Behavior Synthesis. Guided sampling cod

Michael Janner 310 Jan 07, 2023
Raspberry Pi Pico as a Rubber Ducky

Raspberry-Pi-Pico-as-a-Rubber-Ducky Kurulum Raspberry Pi Pico cihazınız için CircuitPython'u indirin. Boot düğmesine basılı tutarken cihazı bir USB ba

Furkan Enes POLATOĞLU 6 Dec 13, 2022
LT-OCF: Learnable-Time ODE-based Collaborative Filtering, CIKM'21

LT-OCF: Learnable-Time ODE-based Collaborative Filtering Our proposed LT-OCF Our proposed dual co-evolving ODE Setup Python environment for LT-OCF Ins

Jeongwhan Choi 15 Dec 28, 2022
ENC28J60 Ethernet chip driver for MicroPython (RP2)

micropy-ENC28J60 ENC28J60 Ethernet chip driver for MicroPython v1.17 (RP2) Rationale ENC28J60 is a popular and cheap module for DIY projects. At the m

11 Nov 16, 2022
Интеграция Home Assistant с ЛК "Интер РАО"

ЕЛК ЖКХ «Интер РАО» для Home Assistant Предоставление информации о текущем состоянии ваших аккаунтов в ЕЛК ЖКХ. Введение @ TODO @ Установка Посредство

Alexander Ryazanov 27 Nov 05, 2022
Adafruit IO connected smart thermostat based on CircuitPython.

Adafruit IO Thermostat Adafruit IO connected smart thermostat based on CircuitPython. Background and Motivation I have a 24 V Heat-only system with a

Shubham Chaudhary 1 Jan 18, 2022
Raspberry Pi Pico support for VS Code

Pico-Go VS Code Extension Pico-Go provides code auto-completion and allows you to communicate with your Raspberry Pi Pico board using the built-in REP

Chris Wood 114 Dec 28, 2022
Cascade Drone Swarm Physical Demonstration Project

Cascade Drone Swarm Physical Demonstration Project Table of Contents About The Project Built With Getting Started Prerequisites Installation About The

3 Aug 24, 2022
3d printable macropad

Pico Mpad A 3D printable macropad for automating frequently repeated actions. Hardware To build this project you need access to a 3d printer. The mode

Dmytro Panin 94 Jan 07, 2023
Pihole-eink-display - A simple Python script to display PiHole statistics on an eInk Display

Pihole-eink-display - A simple Python script to display PiHole statistics on an eInk Display

Mark McIntyre 64 Oct 11, 2022
Iec62056-21-mqtt - Publish DSMR P1 telegrams acquired over IEC62056-21 to MQTT

IEC 62056-21 Publish DSMR P1 telegrams acquired over IEC62056-21 to MQTT. -21 is

Marijn Suijten 1 Jun 05, 2022
Code and build instructions for Snap, a simple Raspberry Pi and LED machine to show you how expensive the electricyty is at the moment

Code and build instructions for Snap, a simple Raspberry Pi and LED machine to show you how expensive the electricyty is at the moment. On row of LEDs shows the cost of the hour, the other row the co

Johan Jonk Stenström 3 Sep 08, 2022
KIRI - Keyboard Interception, Remapping, and Injection using Raspberry Pi as an HID Proxy.

KIRI - Keyboard Interception, Remapping and Injection using Raspberry Pi as a HID Proxy. Near limitless abilities for a keyboard warrior. Features Sim

Viggo Falster 10 Dec 23, 2022
This is a Virtual Keyboard which is simple yet effective to use.

Virtual-Keyboard This is a Virtual KeyBoard which can track finger movements and lets you type anywhere ranging from notepad to even web browsers. It

Jehan Patel 3 Oct 01, 2021
A PYTHON Library for Controlling Motors using SOLO Motor Controllers with RASPBERRY PI, Linux, windows, and more!

A PYTHON Library for Controlling Motors using SOLO Motor Controllers with RASPBERRY PI, Linux, windows, and more!

SOLO Motor Controllers 3 Apr 29, 2022
Robot Framework keyword library wrapper for atlassian-python-api

Robot Framework keyword library wrapper for atlassian-python-api

Marcin Koperski 3 Jul 29, 2022
🏡 My Home Assistant Configs. Be sure to 🌟 my repo to follow the updates!

Home Assistant Configuration Here's my Home Assistant configuration. I have installed HA on a Lenovo ThinkCentre M93P Tiny with an Intel Dual-Core i5-

iLyas Bakouch 25 Dec 30, 2022
Aqara Camera G3 integration for Home Assistant

Aqara Camera G3 integration for Home Assistant ATTENTION: The component only works after enabled telnet. Only supportd stream. Not support still image

14 Dec 18, 2022
🎃 Some spooky code samples to hack yourself a pumpkin 👻

🎃 Tech Or Treat 👻 It's spooky season for those who celebrate Halloween, and to get in the spirit (spirit - get it? 👻 ) we thought it would be fun t

Jim Bennett 5 Feb 07, 2022
Beam designs for infinite Z 3D printers

A 3D printed beam that is as stiff as steel A while ago Naomi Wu 机械妖姬 very kindly sent us one of Creality's infinite-Z belt printers. Lots of people h

RepRap Ltd 105 Oct 22, 2022