Official implementation of the AAAI 2022 paper "Learning Token-based Representation for Image Retrieval"

Related tags

AuthenticationToken
Overview

Token: Token-based Representation for Image Retrieval

PyTorch training code for Token-based Representation for Image Retrieval. We propose a joint local feature learning and aggregation framework, obtaining 82.3 mAP on ROxf with Medium evaluation protocols. Inference in 50 lines of PyTorch.

Token

What it is. Given an image, Token first uses a CNN and a Local Feature Self-Attention (LFSA) module to extract local features $F_c$. Then, they are tokenized into $L$ visual tokens with spatial attention. Further, a refinement block is introduced to enhance the obtained visual tokens with self-attention and cross-attention. Finally, Token concatenates all the visual tokens to form a compact global representation $f_g$ and reduce its dimension. The aggreegated global feature is discriminative and efficient.

About the code. Token is very simple to implement and experiment with. Training code follows this idea - it is not a library, but simply a train.py importing model and criterion definitions with standard training loops.

mAP performance of the proposed model

We provide results of Token. mAP is computed with Medium and Hard evaluation protocols. model will come soon.

Token

Requirements

  • Python 3
  • cuda 11.0
  • PyTorch tested on 1.8.0, torchvision 0.9.0
  • numpy
  • matplotlib

Usage - Representation learning

There are no extra compiled components in Token and package dependencies are minimal, so the code is very simple to use. We provide instructions how to install dependencies via conda. Install PyTorch 1.8.0 and torchvision 0.9.0:

conda install -c pytorch pytorch torchvision

Data preparation

Before going further, please check out Google landmarkv2 github. We use their training images. If you use this code in your research, please also cite their work!

Download and extract Google landmarkv2 train and val images with annotations from https://github.com/cvdfoundation/google-landmark.

Download ROxf and RPar datastes with annotations. We expect the directory structure to be the following:

/data/
  ├─ Google-landmark-v2 # train images
  │   ├─ train.csv
  │   ├─ train_clean.csv
  │   ├─ GLDv2-clean-train-split.pkl
  │   ├─ GLDv2-clean-val-split.pkl
  |   └─ train
  └─test # test images
      ├─ roxford5k
      |   ├─ jpg
      |   └─ gnd_roxford5k.pkl
      └─ rparis6k
          ├─ jpg
          └─ gnd_rparis6k.pkl

Training

To train Token on a single node with 4 gpus for 30 epochs run:

sh experiment.sh

A single epoch takes 2.5 hours, so 30 epoch training takes around 3 days on a single machine with 4 3090Ti cards.

We train Token with SGD setting learning rate to 0.01. The refinement block is trained with dropout of 0.1, and linearly decaying scheduler is adopted to gradually decay the learning rate to 0 when the desired number of steps is reached.

Evaluation

To evaluate on Roxf and Rparis with a single GPU run:

python test.py

and get results as below

>> Test Dataset: roxford5k *** local aggregation >>
>> mAP Medium: 82.28, Hard: 66.57

>> Test Dataset: rparis6k *** local aggregation >>
>> mAP Medium: 89.34, Hard: 78.56

We found that there is a change in performance when the test environment is different, for example, when the environment is GeForce RTX 2080Ti with cuda 10.2, pytorch 1.7.1 and torchvision 0.8.2, the test performance is

>> Test Dataset: roxford5k *** local aggregation >>
>> mAP Medium: 81.36, Hard: 62.09

>> Test Dataset: rparis6k *** local aggregation >>
>> mAP Medium: 90.19, Hard: 80.16

Qualitative examples

Selected qualitative examples of different methods. Top-11 results are shown in the figure. The image with green denotes the true positives and the red bounding boxes are false positives.

Token

Owner
Hui Wu
Department of Electronic Engineering and Information Science University of Science and Technology of China
Hui Wu
Simplifying third-party authentication for web applications.

Velruse is a set of authentication routines that provide a unified way to have a website user authenticate to a variety of different identity provider

Ben Bangert 253 Nov 14, 2022
Luca Security Concept

Luca Security Concept This is the document source of luca's security concept. Please go here for the HTML version: https://luca-app.de/securityconcept

luca 43 Oct 22, 2022
A JOSE implementation in Python

python-jose A JOSE implementation in Python Docs are available on ReadTheDocs. The JavaScript Object Signing and Encryption (JOSE) technologies - JSON

Michael Davis 1.2k Dec 28, 2022
This script will pull and analyze syscalls in given application(s) allowing for easier security research purposes

SyscallExtractorAnalyzer This script will pull and analyze syscalls in given application(s) allowing for easier security research purposes Goals Teach

Truvis Thornton 18 Jul 09, 2022
Auth for use with FastAPI

FastAPI Auth Pluggable auth for use with FastAPI Supports OAuth2 Password Flow Uses JWT access and refresh tokens 100% mypy and test coverage Supports

David Montague 95 Jan 02, 2023
An open source Flask extension that provides JWT support (with batteries included)!

Flask-JWT-Extended Features Flask-JWT-Extended not only adds support for using JSON Web Tokens (JWT) to Flask for protecting views, but also many help

Landon Gilbert-Bland 1.4k Jan 04, 2023
Corsair_scan is a security tool to test Cross-Origin Resource Sharing (CORS).

Welcome to Corsair_scan Corsair_scan is a security tool to test Cross-Origin Resource Sharing (CORS) misconfigurations. CORS is a mechanism that allow

Santander Security Research 116 Nov 09, 2022
Automatic login utility of free Wi-Fi captive portals

wicafe Automatic login utility of free Wi-Fi captive portals Disclaimer: read and grant the Terms of Service of Wi-Fi services before using it! This u

Takumi Sueda 8 May 31, 2022
JSON Web Token implementation in Python

PyJWT A Python implementation of RFC 7519. Original implementation was written by @progrium. Sponsor If you want to quickly add secure token-based aut

José Padilla 4.5k Jan 09, 2023
Authentication with fastapi and jwt cd realistic

Authentication with fastapi and jwt cd realistic Dependencies bcrypt==3.1.7 data

Fredh Macau 1 Jan 04, 2022
Generate payloads that force authentication against an attacker machine

Hashgrab Generates scf, url & lnk payloads to put onto a smb share. These force authentication to an attacker machine in order to grab hashes (for exa

xct 35 Dec 20, 2022
User-related REST API based on the awesome Django REST Framework

Django REST Registration User registration REST API, based on Django REST Framework. Documentation Full documentation for the project is available at

Andrzej Pragacz 399 Jan 03, 2023
A Python package, that allows you to acquire your RecNet authorization bearer token with your account credentials!

RecNet-Login This is a Python package, that allows you to acquire your RecNet bearer token with your account credentials! Installation Done via git: p

Jesse 6 Aug 18, 2022
This python package provides a simple password reset strategy for django rest framework

Django Rest Password Reset This python package provides a simple password reset strategy for django rest framework, where users can request password r

Anexia 363 Dec 24, 2022
Simple two factor authemtication system, made by me.

Simple two factor authemtication system, made by me. Honestly, i don't even know How 2FAs work I just used my knowledge and did whatever i could. Send

Refined 5 Jan 04, 2022
CheckList-Api - Created with django rest framework and JWT(Json Web Tokens for Authentication)

CheckList Api created with django rest framework and JWT(Json Web Tokens for Aut

shantanu nimkar 1 Jan 24, 2022
OAuth2 goodies for the Djangonauts!

Django OAuth Toolkit OAuth2 goodies for the Djangonauts! If you are facing one or more of the following: Your Django app exposes a web API you want to

Jazzband 2.7k Dec 31, 2022
Beihang University Network Authentication Login

北航自动网络认证使用说明 主文件 gw_buaa.py # @file gw_buaa.py # @author Dong # @date 2022-01-25 # @email windcicada 0 Jul 22, 2022

Multi-user accounts for Django projects

django-organizations Summary Groups and multi-user account management Author Ben Lopatin (http://benlopatin.com) Status Separate individual user ident

Ben Lopatin 1.1k Jan 02, 2023
Flask App With Login

Flask App With Login by FranciscoCharles Este projeto basico é o resultado do estudos de algumas funcionalidades do micro framework Flask do Python. O

Charles 3 Nov 14, 2021