Cherche (search in French) allows you to create a neural search pipeline using retrievers and pre-trained language models as rankers.

Last update: Nov 29, 2022

Overview

Cherche

Neural search

Cherche (search in French) allows you to create a neural search pipeline using retrievers and pre-trained language models as rankers. Cherche is meant to be used with small to medium sized corpora. Cherche's main strength is its ability to build diverse and end-to-end pipelines.

Installation 🤖

pip install cherche

To install the development version:

pip install git+https://github.com/raphaelsty/cherche

Documentation 📜

Documentation is available here. It provides details about retrievers, rankers, pipelines, question answering, summarization, and examples.

QuickStart 💨

Documents 📑

Cherche allows findings the right document within a list of objects. Here is an example of a corpus.

from cherche import data

documents = data.load_towns()

documents[:3]
[{'id': 0,
  'title': 'Paris',
  'url': 'https://en.wikipedia.org/wiki/Paris',
  'article': 'Paris is the capital and most populous city of France.'},
 {'id': 1,
  'title': 'Paris',
  'url': 'https://en.wikipedia.org/wiki/Paris',
  'article': "Since the 17th century, Paris has been one of Europe's major centres of science, and arts."},
 {'id': 2,
  'title': 'Paris',
  'url': 'https://en.wikipedia.org/wiki/Paris',
  'article': 'The City of Paris is the centre and seat of government of the region and province of Île-de-France.'
  }]

Retriever ranker 🔍

Here is an example of a neural search pipeline composed of a TfIdf that quickly retrieves documents, followed by a ranking model. The ranking model sorts the documents produced by the retriever based on the semantic similarity between the query and the documents.

from cherche import data, retrieve, rank
from sentence_transformers import SentenceTransformer

# List of dicts
documents = data.load_towns()

# Retrieve on fields title and article
retriever = retrieve.TfIdf(key="id", on=["title", "article"], documents=documents, k=30)

# Rank on fields title and article
ranker = rank.Encoder(
    key = "id",
    on = ["title", "article"],
    encoder = SentenceTransformer("sentence-transformers/all-mpnet-base-v2").encode,
    k = 3,
    path = "encoder.pkl"
)

# Pipeline creation
search = retriever + ranker

search.add(documents=documents)

search("Bordeaux")
[{'id': 57, 'similarity': 0.69513476},
 {'id': 63, 'similarity': 0.6214991},
 {'id': 65, 'similarity': 0.61809057}]

Map the index to the documents to access their contents.

search += documents
search("Bordeaux")
[{'id': 57,
  'title': 'Bordeaux',
  'url': 'https://en.wikipedia.org/wiki/Bordeaux',
  'article': 'Bordeaux ( bor-DOH, French: [bɔʁdo] (listen); Gascon Occitan: Bordèu [buɾˈðɛw]) is a port city on the river Garonne in the Gironde department, Southwestern France.',
  'similarity': 0.69513476},
 {'id': 63,
  'title': 'Bordeaux',
  'url': 'https://en.wikipedia.org/wiki/Bordeaux',
  'article': 'The term "Bordelais" may also refer to the city and its surrounding region.',
  'similarity': 0.6214991},
 {'id': 65,
  'title': 'Bordeaux',
  'url': 'https://en.wikipedia.org/wiki/Bordeaux',
  'article': "Bordeaux is a world capital of wine, with its castles and vineyards of the Bordeaux region that stand on the hillsides of the Gironde and is home to the world's main wine fair, Vinexpo.",
  'similarity': 0.61809057}]

Retrieve 👻

Cherche provides different retrievers that filter input documents based on a query.

retrieve.Elastic
retrieve.TfIdf
retrieve.Lunr
retrieve.BM25Okapi
retrieve.BM25L
retrieve.Flash
retrieve.Encoder

Rank 🤗

Cherche rankers are compatible with SentenceTransformers models, Hugging Face sentence similarity models, Hugging Face zero shot classification models, and of course with your own models.

Summarization and question answering

Cherche provides modules dedicated to summarization and question answering. These modules are compatible with Hugging Face's pre-trained models and can be fully integrated into neural search pipelines.

Acknowledgements 👏

The BM25 models available in Cherche are wrappers around rank_bm25. Elastic retriever is a wrapper around Python Elasticsearch Client. TfIdf retriever is a wrapper around scikit-learn's TfidfVectorizer. Lunr retriever is a wrapper around Lunr.py. Flash retriever is a wrapper around FlashText. DPR and Encode rankers are wrappers dedicated to the use of the pre-trained models of SentenceTransformers in a neural search pipeline. ZeroShot ranker is a wrapper dedicated to the use of the zero-shot sequence classifiers of Hugging Face in a neural search pipeline.

Dev Team 💾

The Cherche dev team is made up of Raphaël Sourty and François-Paul Servant 🥳

Comments

Added spelling corrector object

Hello ! I added a spelling corrector base class as well as the original implementation of the Norvig spelling corrector. The spelling corrector can be fitted directly on the pipeline's documents with the '.add(documents)' method. I also provided an optional (defaults to False) external dictionary, the one originally used by Norvig.

I have no issue updating my code for improvements, so feel free to suggest any modification !

opened by NicolasBizzozzero 4
0.0.5
Pull request for Cherche version 0.0.5

RAG: add RAG generator for open domain question answering

RapidFuzzy: New blazzing fast retriever

Retrievers: Provide similarities for each retriever

Union & Intersection: Keep similarity scores
opened by raphaelsty 1
Batch processing
Retrieving documents with batch of queries can significantly speed up things. It is now available for few models using the development version via the batch method.

Models involved are:

TfIdf retriever

Encoder retriever (milvus + faiss)

Encoder ranker (milvus)

DPR retriever (milvus + faiss)

DPR ranker (milvus)

Recommend retriever

Batch is not yet compatible with pipelines.
enhancement
opened by raphaelsty 0
Cherche 1.0.0
Here is an essential update for Cherche. The update retains the previous API and is compatible with previous versions. 🥳

Main additions:

Added compatibility with two new open-source retrievers: Meilisearch and TypeSense.

Compatibility with the Milvus index to use the retriever.Encoder and retriever.DPR models on massive corpora.

Compatibility with the Milvus index to store ranker embeddings in a database rather than in memory.

Progress bar when pre-computing embeddings by Encoder, DPR retrievers and Encoder, DPR rankers.

All pipelines (voting, intersection, concatenation) produce a similarity score. To do so, the pipeline object applies a softmax to normalize the scores, thus allowing us to "compare" the scores of two distinct models.

Integration of collaborative filtering models via adding a Recommend retriever and a Recommend ranker (indexation via Faiss and compatible with Milvus) to consider users' preferences in the search.
opened by raphaelsty 0
"IndexError: index out of range in self "While adding documents to cherche pipeline

I'm using a cherche pipline built of a tfidf retriever with a sentencetransformer ranker as follows : search = (retriever + ranker) While trying to add documents to the pipeline (search.add(documents=documents), I got this error :

"""/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse) 2181 # remove once script supports set_grad_enabled 2182 no_grad_embedding_renorm(weight, input, max_norm, norm_type) -> 2183 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) 2184 2185

IndexError: index out of range in self"""

opened by delmetni 0
incomplete doc about metrics

https://raphaelsty.github.io/cherche/examples/eval_pipeline/ you say that one can find the explnaation about metrics here https://amitness.com/2020/08/information-retrieval-evaluation/ but it doesn't say what "precision" and "r-precision" are.

opened by fpservant 0

Releases(1.0.1)

1.0.1(Oct 27, 2022)

Removed the dependency with grpcio that can cause problems during installation.
Source code(tar.gz)
Source code(zip)
1.0.0(Oct 26, 2022)
What's Changed

Here is an essential update for Cherche! 🥳

Added compatibility with two new open-source retrievers: Meilisearch and TypeSense.

Compatibility with the Milvus index to use the retriever.Encoder and retriever.DPR models on massive corpora.

Compatibility with the Milvus index to store ranker embeddings in a database rather than in memory.

Progress bar when pre-computing embeddings by Encoder, DPR retrievers and Encoder, DPR rankers.

The path parameter is no longer used.

All pipelines (voting, intersection, concatenation) produce a similarity score. To do so, the pipeline object applies a softmax to normalize the scores, thus allowing us to "compare" the scores of two distinct models.

Integration of collaborative filtering models via adding a Recommend retriever and a Recommend ranker (indexation via Faiss and compatible with Milvus) to consider users' preferences in the search.

Cherche is now fully compatible with large-scale corpora and deeply integrates collaborative filtering. Updates retains the previous API and is compatible with previous versions.
Source code(tar.gz)
Source code(zip)
0.1.0(Jun 16, 2022)

Added compatibility with the ONNX environment and quantization to significantly speed up sentence transformers and question answering models. 🏎

It is now possible to choose the type of index for the Encoder and DPR retrievers in order to process the largest corpora while using the GPU.
Source code(tar.gz)
Source code(zip)
0.0.9(Apr 13, 2022)

Voting operator dedicated to retrievers and rankers.
Source code(tar.gz)
Source code(zip)
0.0.8(Mar 7, 2022)

Avoid checking similarities in TF-IDF retrievers while filtering documents.
Source code(tar.gz)
Source code(zip)
0.0.7(Mar 7, 2022)
Significant improvement in the speed of the TF-IDF retriever using sparse CSC matrix.

The setup.py file loads the readme file as UTF-8.

Source code(tar.gz)
Source code(zip)
0.0.6(Mar 3, 2022)
Update documentation

Update retriever Encoder and DPR, path is optionnal

Add deployment documentation

Update similarity type

Avoid round similarity

Source code(tar.gz)
Source code(zip)
0.0.5(Feb 8, 2022)
Loading and Saving tutorial

Fuzzy retriever

Similarities everywhere (retrievers, union, intersection provide similarity scores)

RAG generation

Source code(tar.gz)
Source code(zip)
0.0.4(Jan 20, 2022)

Update of the encoder retriever and the DPR retriever. Documents in the Faiss index will not be duplicated. Query embeddings can now be pre-computed for ranker Encoder and ranker DPR to speed up evaluation without having to compute it again.
Source code(tar.gz)
Source code(zip)
0.0.3(Jan 13, 2022)
Adding:

Translation

DPR retriever

Source code(tar.gz)
Source code(zip)
0.0.2(Jan 12, 2022)

Update of the Cherche dependencies. The previous dependencies were too strict and restrictive as they were limited to a specific version for each package.
Source code(tar.gz)
Source code(zip)
0.0.1(Jan 8, 2022)

Source code(tar.gz)
Source code(zip)

Owner

Raphael Sourty

PhD Student @ IRIT and Renault

GitHub Repository

Code for ACL 2020 paper "Rigid Formats Controlled Text Generation"

SongNet SongNet: SongCi + Song (Lyrics) + Sonnet + etc. @inproceedings{li-etal-2020-rigid, title = "Rigid Formats Controlled Text Generation",

212 Dec 17, 2022

Data loaders and abstractions for text and NLP

torchtext This repository consists of: torchtext.data: Generic data loaders, abstractions, and iterators for text (including vocabulary and word vecto

3.2k Dec 30, 2022

Train GPT-3 model on V100(16GB Mem) Using improved Transformer.

GPT-X using transformer pytorch

24 Sep 11, 2022

Open source code for AlphaFold.

AlphaFold This package provides an implementation of the inference pipeline of AlphaFold v2.0. This is a completely new model that was entered in CASP

9.7k Jan 02, 2023

Grapheme-to-phoneme (G2P) conversion is the process of generating pronunciation for words based on their written form.

Neural G2P to portuguese language Grapheme-to-phoneme (G2P) conversion is the process of generating pronunciation for words based on their written for

11 Nov 16, 2022

Creating a python chatbot that Starbucks users can text to place an order + help cut wait time of a normal coffee.

2 Jan 20, 2022

FB ID CLONER WUTHOT CHECKPOINT, FACEBOOK ID CLONE FROM FILE

* MY SOCIAL MEDIA : Programming And Memes Want to contact Mr. Error ? CONTACT : [ema

9 Jun 17, 2021

Code associated with the "Data Augmentation using Pre-trained Transformer Models" paper

Data Augmentation using Pre-trained Transformer Models Code associated with the Data Augmentation using Pre-trained Transformer Models paper Code cont

44 Dec 31, 2022

In this workshop we will be exploring NLP state of the art transformers, with SOTA models like T5 and BERT, then build a model using HugginFace transformers framework.

Transformers are all you need In this workshop we will be exploring NLP state of the art transformers, with SOTA models like T5 and BERT, then build a

8 Apr 13, 2022

Code to reprudece NeurIPS paper: Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks

Accelerated Sparse Neural Training: A Provable and Efficient Method to FindN:M Transposable Masks Recently, researchers proposed pruning deep neural n

4 Feb 23, 2022

Cherche (search in French) allows you to create a neural search pipeline using retrievers and pre-trained language models as rankers.

Related tags

Overview

Cherche

Installation 🤖

Documentation 📜

QuickStart 💨

Documents 📑

Retriever ranker 🔍

Retrieve 👻

Rank 🤗

Summarization and question answering

Acknowledgements 👏

See also 👀

Dev Team 💾

Comments

Added spelling corrector object

0.0.5

Batch processing

Cherche 1.0.0

"IndexError: index out of range in self "While adding documents to cherche pipeline

incomplete doc about metrics

Releases(1.0.1)

1.0.1(Oct 27, 2022)

1.0.0(Oct 26, 2022)

What's Changed

Here is an essential update for Cherche! 🥳

0.1.0(Jun 16, 2022)

0.0.9(Apr 13, 2022)

0.0.8(Mar 7, 2022)

0.0.7(Mar 7, 2022)

0.0.6(Mar 3, 2022)

0.0.5(Feb 8, 2022)

0.0.4(Jan 20, 2022)

0.0.3(Jan 13, 2022)

0.0.2(Jan 12, 2022)

0.0.1(Jan 8, 2022)

Owner

Raphael Sourty

Code for ACL 2020 paper "Rigid Formats Controlled Text Generation"

Data loaders and abstractions for text and NLP

Train GPT-3 model on V100(16GB Mem) Using improved Transformer.

Open source code for AlphaFold.

Grapheme-to-phoneme (G2P) conversion is the process of generating pronunciation for words based on their written form.

Creating a python chatbot that Starbucks users can text to place an order + help cut wait time of a normal coffee.

FB ID CLONER WUTHOT CHECKPOINT, FACEBOOK ID CLONE FROM FILE

Code associated with the "Data Augmentation using Pre-trained Transformer Models" paper

In this workshop we will be exploring NLP state of the art transformers, with SOTA models like T5 and BERT, then build a model using HugginFace transformers framework.

Code to reprudece NeurIPS paper: Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks

Host your own GPT-3 Discord bot

Code for Editing Factual Knowledge in Language Models

A telegram bot to translate 100+ Languages

Code for the paper "Flexible Generation of Natural Language Deductions"

Count the frequency of letters or words in a text file and show a graph.

HAN2HAN : Hangul Font Generation

Natural Language Processing with transformers

Jupyter Notebook tutorials on solving real-world problems with Machine Learning & Deep Learning using PyTorch

Weaviate demo with the text2vec-openai module

Words-per-minute - A terminal app written in python utilizing the curses module that tests the user's ability to type