Final project in KAIST AI class

Overview

mmodal_mixer

MLP-Mixer based Multi-modal image-text retrieval

Image:

Original image is cropped with 16 x 16 patch size without overlap. Then, it is reshaped to (batch, (hxw), (patch x patch x channel)).

Text:

Also, original text is tokenized and embedded with BERT-based approach (BERT-base-uncased).

Data processing:

When we train our model, we randomly samples(50 %) reports to make the matched- and un-matched image-text set. Basically, matched and un-matched set is classified with label information using chexpert labeler, we consider unmatched set when randomly sampled report is not exactly same with original one.

Mixer based approach is trained efficiently with xxxx throuput with xxx accuracy.

Exp settings.

batch: 256 batch epoch: 50 epoch

Chest X-ray Image-reports retrieval

Model spec: patch size:16, embedding dim: 768

Input spec: img size: 224x224x3 -> pathch size: (224/16) x (224/16) text max len: 128 legth

input embedding: cls, txt, sep, img output: matched or unmatched

Results

Max memory: 15.903G

Model forward time: Model backward time:

Owner
SuperSuperMoon
PhD student at Graduate School of AI, KAIST. Previously AI research engineer at Samsung Medical Center. Medical AI. Computer Vision & NLP.
SuperSuperMoon
This is an implementation of NeuronJ work with python.

NeuronJ This is an implementation of NeuronJ work with python. NeuronJ is a plug-in for ImageJ that allows you to create and edit neurons masks. Image

Mohammad Mahdi Samei 3 Aug 28, 2022
An example using debezium and mysql with docker-compose

debezium-mysql An example using debezium and mysql with docker-compose The docker compose starts the Zookeeper, Kafka, Mysql and Debezium Connect. Aft

HorĂ¡cio Dias Baptista Neto 4 May 21, 2022
SimilarWeb for Team ACT v.0.0.1

SimilarWeb for Team ACT v.0.0.1 This module has been built to provide a better environment specifically for Similarweb in Team ACT. This module itself

Sunkyeong Lee 0 Dec 29, 2021
A shim for the typeshed changes in mypy 0.900

types-all A shim for the typeshed changes in mypy 0.900 installation pip install types-all why --install-types is annoying, this installs all the thin

Anthony Sottile 28 Oct 20, 2022
Helps compare between New and Old Tax Regime.

Income-Tax-Calculator Helps compare between New and Old Tax Regime. Sample Console Input/Output

2 Jan 10, 2022
Camera track the tip of a pen to use as a drawing tablet

cablet Camera track the tip of a pen to use as a drawing tablet Setup You will need: Writing utensil with a colored tip (preferably blue or green) Bac

14 Feb 20, 2022
A pypi package details search python module

A pypi package details search python module

Fayas Noushad 5 Nov 30, 2021
Python Multilingual Ucrel Semantic Analysis System

PymUSAS Python Multilingual Ucrel Semantic Analysis System, it currently is a rule based token level semantic tagger which can be added to any spaCy p

UCREL 13 Nov 18, 2022
Purge all transformation orientations addon for Blender 2.8 and newer versions

CTO Purge This add-on adds a new button to Blender's Transformation Orientation panel which empowers the user to purge all of his/her custom transform

MMMrqs 10 Dec 29, 2022
AndroidEnv is a Python library that exposes an Android device as a Reinforcement Learning (RL) environment.

AndroidEnv is a Python library that exposes an Android device as a Reinforcement Learning (RL) environment.

DeepMind 814 Dec 26, 2022
Python library for generating CycloneDX SBOMs

Python Library for generating CycloneDX This CycloneDX module for Python can generate valid CycloneDX bill-of-material document containing an aggregat

CycloneDX SBOM Standard 31 Dec 16, 2022
A set of simple functions to upload and fetch pastes on paste.uploadgram.me

pastegram-py A set of simple functions to upload and fetch pastes on paste.uploadgram.me. API Documentation Methods upload_paste(contents: bytes, file

Uploadgram 3 Sep 13, 2022
A test repository to build a python package and publish the package to Artifact Registry using GCB

A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.

1 Feb 09, 2022
Script to calculate delegator epoch returns for all pillars

znn_delegator_calculator Script to calculate estimated delegator epoch returns for all Pillars, so you can delegate to the best one. You can find me o

2 Dec 03, 2021
đŸ“½ Streamlit application powered by a PyScaffold project setup

streamlit-demo Streamlit application powered by a PyScaffold project setup. Work in progress: The idea of this repo is to demonstrate how to package a

PyScaffold 2 Oct 10, 2022
WGGCommute - Adding Commute Times to WG-Gesucht Listings

WGGCommute - Adding Commute Times to WG-Gesucht Listings This is a barebones implementation of a chrome extension that can be used to add commute time

Jannis 2 Jul 20, 2022
Source code for Learn Programming: Python

This repository contains the source code of the game engine behind Learn Programming: Python. The two key files are game.py (the main source of the ga

Niema Moshiri 25 Apr 24, 2022
Prometheus exporter for Spigot accounts

SpigotExporter Prometheus exporter for Spigot accounts What it provides SpigotExporter will output metrics for each of your plugins and a cumulative d

Jacob Bashista 5 Dec 20, 2021
Github dorking tool

gh-dork Supply a list of dorks and, optionally, one of the following: a user (-u) a file with a list of users (-uf) an organization (-org) a file with

Molly White 119 Dec 21, 2022
Simple tooling for marking deprecated functions or classes and re-routing to the new successors' instance.

pyDeprecate Simple tooling for marking deprecated functions or classes and re-routing to the new successors' instance

Jirka Borovec 45 Nov 24, 2022