Sort By Face

Related tags

Computer VisionSBF
Overview

Sort-By-Face

This is an application with which you can either sort all the pictures by faces from a corpus of photos or retrieve all your photos from the corpus
by submitting a picture of yours.

Setup:

Requirements:

  • python 3.8.5
  • Anaconda 4.9.2+

If anaconda isn't installed, install it from here

  • Clone the repository
  • Download the folder called Models/ from here into the same directory where you cloned the repository.
  • Run conda env create -f environment.yml to create the environment.
  • Run conda activate sorter.
  • Run pip install -r requirements.txt
  • In case you want to run the notebook then make sure Jupyter notebook is installed and accessible for all environments in your system.

Instructions:

  • Put the directory where the folders are located into the project folder.
  • Run python embedder.py -src /path/to/images. Any non image file extensions are safely ignored. This command utilizes all the cores in the system for parallel processing.
  • In case you want to reduce the number of parallel processes, run python embedder.py -src /path/to/images --processes number-of-processes.
  • Both absolute and relative paths work but relative paths are recommended.
  • The above command then calculates all the embeddings for the faces in the pictures. NOTE: It takes a significant amount of time for large directories.
  • The embeddings are saved in a pickle file called embeddings.pickle.

Sort an entire corpus of photos:

  • Run python sort_images.py. This runs the clustering algorithm with the default parameters of threshold and iterations for the clustering algorithm.
  • If you want to tweak the parameters, run python sort_images.py -t threshold -itr num-iterations to alter the threshold and iterations respectively.
  • If you think pictures are missing try reducing the threshold and increasing the iterations. Something like 0.64 and 35 iterations should work.
  • Once the clustering is finished all the images are stored into a folder called Sorted-pictures. Each subdirectory in it corresponds to the unique person identified.

Get pictures of a single person from the corpus:

  • To get pictures of a single person you will need to provide a picture of that person. It is recommended that the picture clears the following requirements for better results:
    • Image must have width and height greater than 160px.
    • Image must consist of only one face (The program is exited when multiple faces are detected)
    • Image must be preferably well lit and recognizable by a human.
  • Run python get_individual.py -src /path/to/person's/image -dest /path/to/copy/images.
  • This script also allows to tweak with the parameters with the same arguments as mentioned before.
  • Once clustering is done all the pictures are copied into the destination

Evaluation of clustering algorithm:

The notebook On testing on the Labeled Faces in the Wild dataset the following results were obtained. (threshold = 0.67, iterations=30)

  • Precision: 0.89
  • Recall: 0.99
  • F-measure: 0.95
  • Clusters formed: 6090 (5749 unique labels in the dataset)

The code for evaluation has been uploaded in this notebook

The LFW dataset has many images containing more than one face but only has a single label. This can have an effect on the evaluation metrics and the clusters formed. These factors have been discussed in detail in the notebook.
For example by running the script get_individual.py and providing a photo of George Bush will result in some images like this.

In Layman terms we have gathered all the 'photobombs' of George Bush in the dataset, but all the labels for the 'photobombs' correspond to a different person.
NOTE: this does not effect the clustering for the original person as the scripts treat each face seperately but refer to the same image.

How it works:

  • Given a corpus of photos inside a directory this application first detects the faces in the photos.
  • Face alignment is then done using dlib, such that the all the eyes for the faces is at the same coordinates.
  • Then the image is passed through a Convolutional Neural Network to generate 128-Dimensional embeddings.
  • These embeddings are then used in a graph based clustering algorithm called 'Chinese Whispers'.
  • The clustering algorithm assigns a cluster to each individual identified by it.
  • After the algorithm the images are copied into seperate directories corresponding to their clusters.
  • For a person who wants to retrieve only his images, only the images which are in the same cluster as the picture submitted by the user is copied.

Model used for embedding extraction:

The project uses a model which was first introduced in this [4] . It uses a keras model converted from David Sandberg's implementation in this repository.
In particular it uses the model with the name 20170512-110547 which was converted using this script.

All the facenet models are trained using a loss called triplet loss. This loss ensures that the model gives closer embeddings for same people and farther embeddings for different people.
The models are trained on a huge amount of images out of which triplets are generated.

The clustering algorithm:


This project uses a graph based algorithm called Chinese Whispers to cluster the faces. It was first introduced for Natural Language Processing tasks by Chris Biemann in [3] paper.
The authors in [1] and [2] used the concept of a threshold to assign edges to the graphs. i.e there is an edge between two nodes (faces) only if their (dis)similarity metric of their representations is above/below a certain threshold.
In this implementation I have used cosine similarity between face embeddings as the similarity metric.

By combining these ideas we draw the graph like this:

  1. Assign a node to every face detected in the dataset (not every image, because there can be multiple faces in a single image)
  2. Add an edge between two nodes only if the cosine similarity between their embeddings is greater than a threshold.

And the algorithm used for clustering is:

  1. Initially all the nodes are given a seperate cluster.
  2. The algorithm does a specific number of iterations.
  3. For each iteration the nodes are traversed randomly.
  4. Each node is given the cluster which has the highest rank in it's neighbourhood.
  5. The rank of a cluster here is the sum of weights between the current node and the neighbours belonging to that cluster.
  6. In case of a tie between clusters, any one of them is assigned randomly.

The Chinese Whispers algorithm does not converge nor is it deterministic, but it turns out be a very efficient algorithm for some tasks.

References:

This project is inspired by the ideas presented in the following papers

[1] Roy Klip. Fuzzy Face Clustering For Forensic Investigations

[2] Chang L, Pérez-Suárez A, González-Mendoza M. Effective and Generalizable Graph-Based Clustering for Faces in the Wild.

[3] Biemann, Chris. (2006). Chinese whispers: An efficient graph clustering algorithm and its application to natural language processing problems.
[4] Florian Schroff and Dmitry Kalenichenko and James Philbin (2015). FaceNet, a Unified Embedding for Face Recognition and Clustering.

Libraries used:

  • NumPy
  • Tensorflow
  • Keras
  • dlib
  • OpenCv
  • networkx
  • imutils
  • tqdm

Future Scope:

  • A Graphical User Interface (GUI) to help users use the app with ease.
  • GPU optimization to calculate embeddings.
  • Implementation of other clustering methods.
A program that takes in the hand gesture displayed by the user and translates ASL.

Interactive-ASL-Recognition Using the framework mediapipe made by google, OpenCV library and through self teaching, I was able to create a program tha

Riddhi Bajaj 3 Nov 22, 2021
This is a tensorflow re-implementation of PSENet: Shape Robust Text Detection with Progressive Scale Expansion Network.My blog:

PSENet: Shape Robust Text Detection with Progressive Scale Expansion Network Introduction This is a tensorflow re-implementation of PSENet: Shape Robu

Michael liu 498 Dec 30, 2022
A collection of resources (including the papers and datasets) of OCR (Optical Character Recognition).

OCR Resources This repository contains a collection of resources (including the papers and datasets) of OCR (Optical Character Recognition). Contents

Zuming Huang 363 Jan 03, 2023
Code for the paper "DewarpNet: Single-Image Document Unwarping With Stacked 3D and 2D Regression Networks" (ICCV '19)

DewarpNet This repository contains the codes for DewarpNet training. Recent Updates [May, 2020] Added evaluation images and an important note about Ma

<a href=[email protected]"> 354 Jan 01, 2023
Developed an AI-based system to control the mouse cursor using Python and OpenCV with the real-time camera.

Developed an AI-based system to control the mouse cursor using Python and OpenCV with the real-time camera. Fingertip location is mapped to RGB images to control the mouse cursor.

Ravi Sharma 71 Dec 20, 2022
LEARN OPENCV IN 3 HOURS USING PYTHON - INCLUDING EXAMPLE PROJECTS

LEARN OPENCV IN 3 HOURS USING PYTHON - INCLUDING EXAMPLE PROJECTS

Murtaza Hassan 815 Dec 29, 2022
Detect textlines in document images

Textline Detection Detect textlines in document images Introduction This tool performs border, region and textline detection from document image data

QURATOR-SPK 70 Jun 30, 2022
CVPR 2021 Oral paper "LED2-Net: Monocular 360˚ Layout Estimation via Differentiable Depth Rendering" official PyTorch implementation.

LED2-Net This is PyTorch implementation of our CVPR 2021 Oral paper "LED2-Net: Monocular 360˚ Layout Estimation via Differentiable Depth Rendering". Y

Fu-En Wang 83 Jan 04, 2023
Camelot: PDF Table Extraction for Humans

Camelot: PDF Table Extraction for Humans Camelot is a Python library that makes it easy for anyone to extract tables from PDF files! Note: You can als

Atlan Technologies Pvt Ltd 3.3k Dec 31, 2022
Slice a single image into multiple pieces and create a dataset from them

OpenCV Image to Dataset Converter Slice a single image of Persian digits into mu

Meysam Parvizi 14 Dec 29, 2022
This is a GUI program which consist of 4 OpenCV projects

Tkinter-OpenCV Project Using Tkinter, Opencv, Mediapipe This is a python GUI program using Tkinter which consist of 4 OpenCV projects 1. Finger Counte

Arya Bagde 3 Feb 22, 2022
CNN+Attention+Seq2Seq

Attention_OCR CNN+Attention+Seq2Seq The model and its tensor transformation are shown in the figure below It is necessary ch_ train and ch_ test the p

Tsukinousag1 2 Jul 14, 2022
A simple component to display annotated text in Streamlit apps.

Annotated Text Component for Streamlit A simple component to display annotated text in Streamlit apps. For example: Installation First install Streaml

Thiago Teixeira 312 Dec 30, 2022
OCR-D-compliant page segmentation

ocrd_segment This repository aims to provide a number of OCR-D-compliant processors for layout analysis and evaluation. Installation In your virtual e

OCR-D 59 Sep 10, 2022
轻量级公式 OCR 小工具:一键识别各类公式图片,并转换为 LaTeX 格式

QC-Formula | 青尘公式 OCR 介绍 轻量级开源公式 OCR 小工具:一键识别公式图片,并转换为 LaTeX 格式。 支持从 电脑本地 导入公式图片;(后续版本将支持直接从网页导入图片) 公式图片支持 .png / .jpg / .bmp,大小为 4M 以内均可; 支持印刷体及手写体,前

青尘工作室 26 Jan 07, 2023
A python screen recorder for low-end computers, provides high quality video output.

RecorderX - v1.0 A screen recorder made in Python with the help of OpenCv, it has ability to record your screen in high quality. No matter what your P

Priyanshu Jindal 4 Nov 10, 2021
BNF Globalization Code (CVPR 2016)

Boundary Neural Fields Globalization This is the code for Boundary Neural Fields globalization method. The technical report of the method can be found

25 Apr 15, 2022
Use Youdao OCR API to covert your clipboard image to text.

Alfred Clipboard OCR 注:本仓库基于 oott123/alfred-clipboard-ocr 的逻辑用 Python 重写,换用了有道 AI 的 API,准确率更高,有效防止百度导致隐私泄露等问题,并且有道 AI 初始提供的 50 元体验金对于其资费而言个人用户基本可以永久使用

Junlin Liu 6 Sep 19, 2022
Text to QR-CODE

QR CODE GENERATO USING PYTHON Author : RAFIK BOUDALIA. Installation Use the package manager pip to install foobar. pip install pyqrcode Usage from tki

Rafik Boudalia 2 Oct 13, 2021