Reproducible nvim completion framework benchmarks.

Overview

Nvim.Bench

Reproducible nvim completion framework benchmarks.

Runs inside Docker. Fair and balanced


Methodology

Note: for all "randomness", they are generated from the same seed for each run, and therefore "fair".

Input

tmux is used to send keys to simulate ideal human typing.

The words typed are naive tokens from parsing current document into (alphanum + "_") delimited by whitespaces and symbols.

This tokenization should work fairly well for c family of languages, which are the industry standard.

A uniform distribution of whitespaces is also generated from the same buffer.

Measurement

n keystrokes of --samples is performed.

Speed

Using --avg-word-len, --wpm and --variance, a Normal Distribution is constructed of the desired delay between keystrokes.

Data

See ./fs/data/

Modularity

Some frameworks will have by default, very little sources enabled, if any.

Other ones will come with more out of the box.

For a fair comparison: All frameworks tested will have to following enabled, on top of whatever else they come enabled by default:

  • buffer

  • lsp

  • path

The reasoning is that: 1) Almost all authors will have written these sources firsthand, and 2) they seem to be the most useful sources.

No default sources will be disabled, because users don't tend to do that.


Cool, pictures

The plots are kernel density estimations, have no idea why they fitted more than 1 curve for some plots.

I usually use R, not used to python ploting. Anyways, they are an estimate of the true probability density function.

Q0, 50, 95, 100?

Mean min, median, 1 in 20, max, respectively.

Without assuming any statistical distribution:

Q50 is a more robust measure than avg, and Q95 is a decent measure of a common bad value.


Analysis

Please keep in mind that this is purely a synthetic benchmark, which definitely is one of those need context to interpret type of things.

There is no good way to measure real speed across frameworks, raw numbers here come with big caveats.

Study design limitations

Streaming completion

Streaming completion is very good for time to first result (TTFR), but it presents us with an issue:

While the fast sources will return right away, the slower ones might never make it before the next keystroke.

This has the funny effect of removing the influence of slower sources entirely, which is disastrous for study integrity.

The mitigation is actually to set typing speed unrealistically slow, enough so that we have confidence that the LSP servers can catch up.

This is obviously not ideal.

Fast on paper != fast IRL

The most responsive frameworks are not necessarily the fastest ones, because humans still have to choose the results.

For example the streaming completion approach actually has severe trade offs infavor of faster TTFR:

Ranking

Having suboptimal ranking is BAD, it pushes work from fast machines onto slow humans.

The streaming approach has to be additive, because its too disruptive to shift existing menu items around.

Therefore it is limited to sorting only within stream batches, and to make things worse, slower batches typically contain higher quality results.

That means better results will often end up at the bottom, necessitating more work for humans.

Limiting

This is a direct consequence of limited ranking optimizations.

Because the framework have no idea how much each source will send, it has the dilemma of either sending too many results or too little.

Sending too many results in early batches from likely inferior sources will waste the users time, and sending too little will obscure potentially useful completions.

Clarity on when / if results will come in

This is a HCI thing:

Having higher quality results come in slower is likely to inadvertently train users to wait for them. This is evidently bad for input speed.

Conclusion

There is never going to be a closed form solution to "what is the fastest framework", because of the trade offs detailed above.

A toy example of a degenerate framework that returns a single fixed 👌 emoji will probably beat anything out there in terms of raw speed, but it is utterly useless.

Before you reach your own conclusion, the results of this repo must be considered alongside inextricably human measure.

Owner
i love my dog
dogs are love dogs are life
i love my dog
Pokemon sword replay capture

pokemon-sword-replay-capture This is an old version (March 2020) pokemon-sword-replay-capture-mar-2020-version of my Pokemon Replay Capture software.

11 May 15, 2022
Beginner Projects A couple of beginner projects here

Beginner Projects A couple of beginner projects here, listed from easiest to hardest :) selector.py: simply a random selector to tell me who to faceti

Kylie 272 Jan 07, 2023
Simple control of Thorlabs Elliptec devices from Python.

Elliptec Simple control of Thorlabs Elliptec devices. No docs yet » Get started · Report a bug · Request a feature About The Project ThorLabs Elliptec

David Roesel 8 Sep 22, 2022
Fetch data from an excel file and create HTML file

excel-to-html Problem Statement! - Fetch data from excel file and create html file Excel.xlsx file contain the information.in multiple rows that is ne

Vivek Kashyap 1 Oct 25, 2021
GA SEI Unit 4 project backend for Bloom.

Grow Your OpportunitiesTM Background Watch the Bloom Intro Video At Bloom, we believe every job seeker deserves an opportunity to find meaningful work

Jonathan Herman 3 Sep 20, 2021
🥦 Send and receive nano with 2 simple functions

easy_nano Send and receive nano (without having to understand the nano protocol).

1 Feb 14, 2022
Pampy: The Pattern Matching for Python you always dreamed of.

Pampy: Pattern Matching for Python Pampy is pretty small (150 lines), reasonably fast, and often makes your code more readable and hence easier to rea

Claudio Santini 3.5k Dec 30, 2022
A tool for checking if the external data used in Flatpak manifests is still up to date

Flatpak External Data Checker This is a tool for checking for outdated or broken links of external data in Flatpak manifests. Motivation Flatpak apps

Flathub 76 Dec 24, 2022
Python MapReduce library written in Cython.

Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.

Brandyn White 243 Sep 16, 2022
A python library what works with numbers.

pynum A python library what works with numbers. Prime Prime class have everithing you want about prime numbers. check_prime The check_prime method is

Mohammad Mahdi Paydar Puya 1 Jan 07, 2022
Refer'd Resume Scanner

Refer'd Resume Scanner I wanted to share a free resource we built to assist applicants with resume building. Our resume scanner identifies potential s

Refer'd 74 Mar 07, 2022
Blender 3.0 Python - Open temporary areas in the Text Editor

PopDrawers When editing text in Blender, it can be handy to have areas like Info, Console, Outliner, etc visible on screen to help with scripting. How

SpectralVectors 7 Nov 16, 2022
Additional useful operations for Python

Pyteal Extensions Additional useful operations for Python Available Operations MulDiv64: calculate m1*m2/d with no overflow on multiplication (TEAL 3+

Ulam Labs 11 Dec 14, 2022
Transparently load variables from environment or JSON/YAML file.

A thin wrapper over Pydantic's settings management. Allows you to define configuration variables and load them from environment or JSON/YAML file. Also generates initial configuration files and docum

Lincoln Loop 90 Dec 14, 2022
Project Guide for ASAM OpenX standards

ASAM Project Guide Important This guide is a work in progress and subject to change! Hosted version available at: ASAM Project Guide (Link) Includes:

ASAM e.V. 2 Mar 17, 2022
Synchrosqueezing, wavelet transforms, and time-frequency analysis in Python

Synchrosqueezing is a powerful reassignment method that focuses time-frequency representations, and allows extraction of instantaneous amplitudes and frequencies

John Muradeli 382 Jan 06, 2023
Hook and simulate global keyboard events on Windows and Linux.

keyboard Take full control of your keyboard with this small Python library. Hook global events, register hotkeys, simulate key presses and much more.

BoppreH 3.2k Jan 01, 2023
Proyectos de ejercicios básicos y avanzados hecho en python

Proyectos Básicos y Avanzados hecho en python Instalación: Tener instalado python 3.x o superior. Tener pip instalado. Tener virtualenv o venv instala

Karlo Xavier Chok 1 Dec 27, 2021
An end-to-end encrypted chat

An end-to-end encrypted chat, that allows users to anonymously talk without ip logs, personal info, or need for registration.

Privalise 1 Nov 27, 2021
A simple PID tuner and simulator.

PIDtuner-V0.1 PlantPy PID tuner version 0.1 Features Supports first order and ramp process models. Supports Proportional action on PV or error or a sp

3 Jun 23, 2022