Search Git commits in natural language

Overview

NaLCoS - NAtural Language COmmit Search

Search commit messages in your repository in natural language.

GitHub Issues Stargazers License Code style: black
GitHub release (latest by date) PyPi
All contributors


NaLCoS (NAtural Language COmmit Search) is a command-line tool for searching commit messages in your repository in natural language.

The key features are:

  • Search commit messages in both local and remote GitHub repositories.
  • Search for commits in a specific branch.
  • Restrict the number of commits to look back in history while searching.
  • Increase the number of retrieved results.

image

Internally, NaLCoS uses Sentence Transformers with pre-trained weights from multi-qa-MiniLM-L6-cos-v1. I chose this particular model because it has a good Performance vs Speed tradeoff. Since this model was designed for semantic search and has been pre-trained on 215M (question, answer) pairs from diverse sources, it is a good choice for tasks such as finding similarity between two sentences.

NaLCoS encodes the query string and all the commits into their corresponding vector embeddings and computes the cosine similarity between the query and all the commits. This is then used to rank the commits.

Why did I build this?

Most of the times when I've used Machine Learning till now, has been in dedicated environments such as Google Colab or Kaggle. I had been learning Natural Language Processing for a while and wanted to use transformers to build something different that is not very resource (read GPU) intensive and can be used like an everyday tool.

Though many Transformer models are far from fitting this description, I found that distilled models are not as hungry as their older siblings are infamous for. Searching for Git commits using natural language was something on which I could not find any pre-existing tool and thus decided to give this a shot.

Though there are various improvements left, I'm happy with what this initially turned out to be. I'm eager to see what further enhancements can be made to this to make it more efficient and useful.

Requirements

NaLCoS uses the following packages:

Installation

Installing with pip (Recommended)

Install with pip or your favourite PyPi manager:

$ pip install nalcos

Run NaLCoS with the --help flag to see all the available options:

$ nalcos --help

Note: When you run the nalcos command for the first time, it will, download the model which would be cached and used the next time you run NaLCoS.

Installing bleeding edge from the GitHub repository

  • Clone the repository:
$ git clone https://github.com/thepushkarp/nalcos.git

This also downloads the model weights stored in the nalcos/models directory so you don't have to download them while running the model for the first time.

  • Create a virtual environment (click here to read about activating virtualenv):
$ virtualenv venv
  • Activate virtualenv (for Linux and MacOS):
  $ source ./venv/bin/activate
  • Activate virtualenv (for Windows):
   $ cd venv/Scripts/
   $ activate
  • Install the requirements:
$ pip install -r requirements.txt
  • Change directory to the nalcos directory:
$ cd nalcos/
  • Run NaLCoS with the --help flag to see all the available options:
$ python nalcos.py --help

Usage

A detailed information about the usage of NaLCoS can be found below:

usage: nalcos [-h] [-g] [-n N_MATCHES] [-b BRANCH] [-l LOOK_PAST] [-v] query location

Search a commit in your git repository using natural language.

positional arguments:
  query                 The query to search for similar commit messages.
  location              The repository path to search in. If `-g` flag is not passed, searches locally in the path specified, else
                        takes in a remote GitHub repository name in the format '{owner}/{repo_name}'

optional arguments:
  -h, --help            show this help message and exit
  -g, --github          Flag to search on GitHub instead of searching in a local repository. Due to API limits currently this
                        allows for around 15 lookups per hour from your IP.
  -n N_MATCHES, --n-matches N_MATCHES
                        The number of matching results to return. Default 10.
  -b BRANCH, --branch BRANCH
                        The branch to search in. If not specified, the current branch will be used by default.
  -l LOOK_PAST, --look-past LOOK_PAST
                        Look back this many commits. Default 100.
  -v, --version         show program's version number and exit

Examples

  • Input:
$ python nalcos.py "improve language" "github/docs" --github
  • Output:
Found 100 commits.

                                        Commits related to "improve language" in "github/docs"
┏━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓
┃ No. ┃ Commit ID ┃ Commit Message                                                        ┃ Commit Author      ┃ Commit Date          ┃
┡━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩
│  1. │ 51bfdbb95 │ Merge branch 'main' into fatenhealy-fix-supportedlanguage             │ Faten Healy        │ 2021-09-12T22:26:31Z │
│  2. │ a9c2c8eea │ fix deprecation label spelling (#21474)                               │ Rachael Sewell     │ 2021-09-13T18:12:03Z │
│  3. │ 94e3c092d │ English search sync (#21446)                                          │ Rachael Sewell     │ 2021-09-13T17:30:08Z │
│  4. │ b048e27e9 │ Merge pull request #9909 from github/fatenhealy-fix-supportedlanguage │ Ramya Parimi       │ 2021-09-12T22:35:19Z │
│  5. │ 73c2717f7 │ Fix typo                                                              │ Adrian Mato        │ 2021-09-13T06:35:27Z │
│  6. │ 86b571982 │ Export changes to a branch for codespaces (#21462)                    │ Matthew Isabel     │ 2021-09-13T14:55:50Z │
│  7. │ 969288662 │ Update diff limit to 500KB (#20616)                                   │ jjkennedy3         │ 2021-09-11T09:12:38Z │
│  8. │ f28ee46d4 │ Update OpenAPI Descriptions (#21447)                                  │ github-openapi-bot │ 2021-09-11T09:22:28Z │
│  9. │ 92af3a469 │ update search indexes                                                 │ GitHub Actions     │ 2021-09-12T09:50:46Z │
│ 10. │ e6018f2aa │ update search indexes                                                 │ GitHub Actions     │ 2021-09-11T02:05:19Z │
└─────┴───────────┴───────────────────────────────────────────────────────────────────────┴────────────────────┴──────────────────────┘

Future plans

Please visit the NaLCoS To Do Project Board to see current status and future plans.

Known issues

Not all retrieved results are always relevant. I could think of two primary reasons for this:

  • The data the model was pre-trained on is not representative of how people write commit messages. Since commit messages usually contain technical jargon, merge commit messages, abbreviations and other non-common terms, the model (which has a limited vocabulary) is not able to generalize well to this data.
  • Two commits may be related even when their commit messages may not be similar and similarly two commit messages maybe unrelated even when their commit messages are similar. We often need more metadata (such as lines changes, files changed) etc. to make the predictions more accurate.

Contributing

Any suggestions, improvements or bug reports are welcome.

Contributors

Thanks goes to these wonderful people (emoji key):


Pushkar Patel

💻 📖 🚧

This project follows the all-contributors specification. Contributions of any kind welcome!

License

This project is licensed under the terms of the MIT license.

Comments
  • Patches

    Patches

    • :truck: Renames .cache directory to models and add Project Board link in README
    • Adds version.py
    • Adds contributing section in README
    • :lipstick: Add black code style
    • Adds back torch cuda support
    • :zap: Improve similarity computation
    • :tada: Upload to PyPi
    opened by thepushkarp 3
  • Version 0.2: Visual changes

    Version 0.2: Visual changes

    • Fix status message typo
    • :sparkles: Adds a flag to show similarity scores of the result
    • :sparkles: Adds an flag to display the entire commit message
    • Module error fixes
    • :sparkles: Adds commit links for results from GitHub
    • :art: Improve download prograss bar display when loading for first time
    • :bookmark: Bump version to 0.2
    opened by thepushkarp 1
  • Add a flag to download the model

    Add a flag to download the model

    Currently, if the model is not downloaded, the program downloads it during the first run, in the middle of the "Retrieving the commits ..." status mesage.

    This can be improved by adding a flag through which the user can download/redownload the model when they need it.

    Additionally, the program should prompt the user when it is run without downloading the weights with a choice to download it now or to abort the program.

    opened by thepushkarp 1
  • Use a Python Wrapper to the GitHub API

    Use a Python Wrapper to the GitHub API

    We can use some Python Wrapper of the GitHub API such as ghapi or PyGitHub instead of using the requests library.

    Additional Reference: https://docs.github.com/en/rest/overview/libraries#python

    This can help with #11

    opened by thepushkarp 1
  • Visual improvements

    Visual improvements

    • :sparkles: Adds a flag to show similarity scores of the result
    • :sparkles: Adds an flag to display the entire commit message
    • Module error fixes
    • :sparkles: Adds commit links for results from GitHub
    opened by thepushkarp 0
  • Add Automated Testing

    Add Automated Testing

    AUtomate the testing of the module (preferably with GitHub Actions for CI).

    NOTE: Considering the large installation size and time of the Torch and HuggingFace modules, the resources allocated may go over the GH Actions limit. This is something we have to take care of.

    Follow up of #13

    opened by thepushkarp 0
  • Adds README and some bug fixes

    Adds README and some bug fixes

    • Adds API limit exceeded warnning
    • :zap: Reverts back to using whole commit msg for serarch; displays only title
    • :memo: Add README
    • :zap: Reduces default value of look_past from 1000 to 100
    • :bug: Retrieves all branch names for GitHub repos and add branch not found Exception
    opened by thepushkarp 0
  • Try out other models.

    Try out other models.

    Currently, we are using multi-qa-MiniLM-L6-cos-v1, which has a speed (sentences encoded/sec on 1 V100 GPU) of 14200 and a model size of 80 MB. We should try out other models to see if we can get better performance and speed out of them.

    Additionally, we can also try using other types of tokenizers.

    Further reading:

    • https://www.sbert.net/docs/pretrained_models.html
    • https://huggingface.co/sentence-transformers
    • https://huggingface.co/transformers/tokenizer_summary.html
    help wanted 
    opened by thepushkarp 0
  • Add personal API token support

    Add personal API token support

    Do #28 before this

    Currently, the project is using an unauthenticated GH API which is capped to 60 requests per hour from an IP address.

    We can add the option to add a user's personal API access token to increase this limit.

    enhancement 
    opened by thepushkarp 0
Releases(v0.2)
  • v0.2(Sep 18, 2021)

    Changelog

    • Adds an option of showing the similarity score for the results using the -s flag.
    • Adds option of viewing the entire commit message instead of just the commit title using the -v flag.
    • Commits in results retrieved from GitHub have links to the commits
    • Improved the model download progress bar display when loading model for the first time
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Sep 14, 2021)

    Changelog

    • Make similarity computation more efficient
    • Add support for computation on CUDA
    • Add Black Code style in requirements
    • Published to PyPi at https://pypi.org/project/nalcos/ 🥳
    Source code(tar.gz)
    Source code(zip)
  • v0.1(Sep 13, 2021)

    Features ✨

    • Search commit messages in both local and remote GitHub repositories.
    • Search for commits in a specific branch.
    • Restrict the number of commits to look back in history while searching.
    • Increase the number of retrieved results.
    Source code(tar.gz)
    Source code(zip)
Owner
Pushkar Patel
Research Intern at SPIRE Labs, IISC Bangalore | GitHub Campus Expert @iiitv
Pushkar Patel
A Chinese to English Neural Model Translation Project

ZH-EN NMT Chinese to English Neural Machine Translation This project is inspired by Stanford's CS224N NMT Project Dataset used in this project: News C

Zhenbang Feng 29 Nov 26, 2022
Automatic privilege escalation for misconfigured capabilities, sudo and suid binaries

GTFONow Automatic privilege escalation for misconfigured capabilities, sudo and suid binaries. Features Automatically escalate privileges using miscon

101 Jan 03, 2023
Code for ACL 2021 main conference paper "Conversations are not Flat: Modeling the Intrinsic Information Flow between Dialogue Utterances".

Conversations are not Flat: Modeling the Intrinsic Information Flow between Dialogue Utterances This repository contains the code and pre-trained mode

ICTNLP 90 Dec 27, 2022
This is the 25 + 1 year anniversary version of the 1995 Rachford-Rice contest

Rachford-Rice Contest This is the 25 + 1 year anniversary version of the 1995 Rachford-Rice contest. Can you solve the Rachford-Rice problem for all t

13 Sep 20, 2022
中文无监督SimCSE Pytorch实现

A PyTorch implementation of unsupervised SimCSE SimCSE: Simple Contrastive Learning of Sentence Embeddings 1. 用法 无监督训练 python train_unsup.py ./data/ne

99 Dec 23, 2022
Simple python code to fix your combo list by removing any text after a separator or removing duplicate combos

Combo List Fixer A simple python code to fix your combo list by removing any text after a separator or removing duplicate combos Removing any text aft

Hamidreza Dehghan 3 Dec 05, 2022
Natural language Understanding Toolkit

Natural language Understanding Toolkit TOC Requirements Installation Documentation CLSCL NER References Requirements To install nut you need: Python 2

Peter Prettenhofer 119 Oct 08, 2022
Generate vector graphics from a textual caption

VectorAscent: Generate vector graphics from a textual description Example "a painting of an evergreen tree" python text_to_painting.py --prompt "a pai

Ajay Jain 97 Dec 15, 2022
Named Entity Recognition API used by TEI Publisher

TEI Publisher Named Entity Recognition API This repository contains the API used by TEI Publisher's web-annotation editor to detect entities in the in

e-editiones.org 14 Nov 15, 2022
COVID-19 Related NLP Papers

COVID-19 outbreak has become a global pandemic. NLP researchers are fighting the epidemic in their own way.

xcfeng 28 Oct 30, 2022
LV-BERT: Exploiting Layer Variety for BERT (Findings of ACL 2021)

LV-BERT Introduction In this repo, we introduce LV-BERT by exploiting layer variety for BERT. For detailed description and experimental results, pleas

Weihao Yu 14 Aug 24, 2022
This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

Nepali-news-notifier This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular in

Sachit Yadav 1 Feb 11, 2022
An official repository for tutorials of Probabilistic Modelling and Reasoning (2021/2022) - a University of Edinburgh master's course.

PMR computer tutorials on HMMs (2021-2022) This is a repository for computer tutorials of Probabilistic Modelling and Reasoning (2021/2022) - a Univer

Vaidotas Šimkus 10 Dec 06, 2022
Create a machine learning model which will predict if the mortgage will be approved or not based on 5 variables

Mortgage-Application-Analysis Create a machine learning model which will predict if the mortgage will be approved or not based on 5 variables: age, in

1 Jan 29, 2022
Twitter bot that uses NLP models to summarize news articles referenced in a user's twitter timeline

Twitter-News-Summarizer Twitter bot that uses NLP models to summarize news articles referenced in a user's twitter timeline 1.) Extracts all tweets fr

Rohit Govindan 1 Jan 27, 2022
Automated question generation and question answering from Turkish texts using text-to-text transformers

Turkish Question Generation Offical source code for "Automated question generation & question answering from Turkish texts using text-to-text transfor

Open Business Software Solutions 29 Dec 14, 2022
In this Notebook I've build some machine-learning and deep-learning to classify corona virus tweets, in both multi class classification and binary classification.

Hello, This Notebook Contains Example of Corona Virus Tweets Multi Class Classification. - Classes is: Extremely Positive, Positive, Extremely Negativ

Khaled Tofailieh 3 Dec 06, 2022
Code for papers "Generation-Augmented Retrieval for Open-Domain Question Answering" and "Reader-Guided Passage Reranking for Open-Domain Question Answering", ACL 2021

This repo provides the code of the following papers: (GAR) "Generation-Augmented Retrieval for Open-domain Question Answering", ACL 2021 (RIDER) "Read

morning 49 Dec 26, 2022
Deduplication is the task to combine different representations of the same real world entity.

Deduplication is the task to combine different representations of the same real world entity. This package implements deduplication using active learning. Active learning allows for rapid training wi

63 Nov 17, 2022
Pretrained Japanese BERT models

Pretrained Japanese BERT models This is a repository of pretrained Japanese BERT models. The models are available in Transformers by Hugging Face. Mod

Inui Laboratory 387 Dec 30, 2022