Javascript image annotation tool based on image segmentation.

Overview

JS Segment Annotator

Javascript image annotation tool based on image segmentation.

  • Label image regions with mouse.
  • Written in vanilla Javascript, with require.js dependency (packaged).
  • Pure client-side implementation of image segmentation.

A browser must support HTML canvas to use this tool.

There is an online demo.

Importing data

Prepare a JSON file that looks like the following. The required fields are labels and imageURLs. The annotationURLs are for existing data and can be omitted. Place the JSON file inside the data/ directory.

{
  "labels": [
    "background",
    "skin",
    "hair",
    "dress",
    "glasses",
    "jacket",
    "skirt"
  ],
  "imageURLs": [
    "data/images/1.jpg",
    "data/images/2.jpg"
  ],
  "annotationURLs": [
    "data/annotations/1.png",
    "data/annotations/2.png"
  ]
}

Then edit main.js to point to this JSON file. Open a Web browser and visit index.html.

Known issues

Browser incompatibility

A segmentation result can greatly differ due to the difference in Javascript implementation across Web browsers. The difference stems from numerical precision of floating point numbers, and there is no easy way to produce the exact same result across browsers.

Python tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

import numpy as np
from PIL import Image

# Decode
encoded = np.array(Image.open('data/annotations/1.png'))
annotation = np.bitwise_or(np.bitwise_or(
    encoded[:, :, 0].astype(np.uint32),
    encoded[:, :, 1].astype(np.uint32) << 8),
    encoded[:, :, 2].astype(np.uint32) << 16)

print(np.unique(annotation))

# Encode
Image.fromarray(np.stack([
    np.bitwise_and(annotation, 255),
    np.bitwise_and(annotation >> 8, 255),
    np.bitwise_and(annotation >> 16, 255),
    ], axis=2).astype(np.uint8)).save('encoded.png')

JSON

Use JSON module.

import json

with open('data/example.json', 'r') as f:
    dataset = json.load(f)

Using dataURL

Do the following to convert between dataURL and NumPy format.

from PIL import Image
import base64
import io

# Encode
with io.BytesIO() as buffer:
    encoded.save(buffer, format='png')
    data_url = b'data:image/png;base64,' + base64.b64encode(buffer.getvalue())

# Decode
binary = base64.b64decode(data_url.replace(b'data:image/png;base64,', b''))
encoded = Image.open(io.BytesIO(binary))

Matlab tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

% Decode

X = imread('data/annotations/0.png');
annotation = X(:, :, 1);
annotation = bitor(annotation, bitshift(X(:, :, 2), 8));
annotation = bitor(annotation, bitshift(X(:, :, 3), 16));

% Encode

X = cat(3, bitand(annotation, 255), ...
           bitand(bitshift(annotation, -8), 255), ...
           bitand(bitshift(annotation, -16)), 255));
imwrite(uint8(X), 'data/annotations/0.png');

JSON

Use the matlab-json package.

Using dataURL

Get the byte encoding tools.

Do the following to convert between dataURL and Matlab format.

% Decode

dataURL = 'data:image/png;base64,...';
png_data = base64decode(strrep(dataURL, 'data:image/png;base64,', ''));
annotation = imdecode(png_data, 'png');

% Encode

png_data = imencode(annotation, 'png');
dataURL = ['data:image/png;base64,', base64encode(png_data)];

Citation

We appreciate if you cite the following article in an academic paper. The tool was originally developed for this work.

@article{tangseng2017looking,
Author        = {Pongsate Tangseng and Zhipeng Wu and Kota Yamaguchi},
Title         = {Looking at Outfit to Parse Clothing},
Eprint        = {1703.01386v1},
ArchivePrefix = {arXiv},
PrimaryClass  = {cs.CV},
Year          = {2017},
Month         = {Mar},
Url           = {http://arxiv.org/abs/1703.01386v1}
}
Comments
  • Tool crashes on loading Pre-Annotated Images

    Tool crashes on loading Pre-Annotated Images

    Hi @kyamagu ,

    Thank you for sharing this project.

    I'm trying to use custom annotation images and I've created the JSON file accordingly. The tool loads up the relevant classes color-coded over the image. But when I try to correct some segmentation the tool stops working and the following error is printed in the console.

    Browser Console Dump localhost-1505715475195.txt

    SampleJSON untitled )

    Sample Browser Screenshot untitled

    opened by anandcu3 15
  • cant compile paperdoll

    cant compile paperdoll

    Hi, sorry to write it here, i was not able to find a repo for paper doll. I can not compile the paper doll. it seems I have some problem with gco, if I comment it out everything else will work, then when I type in

    result = feature_calculator.apply(config, input_sample) I will get Invalid MEX-file '/BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64': /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64)

    Error in detect_fast>passmsg (line 128) [score0(:,:,k),Ix0(:,:,k),Iy0(:,:,k)] = shiftdt(child.score(:,:,k), child.w(1,k), child.w(2,k), child.w(3,k), child.w(4,k),child.startx(k),child.starty(k),Nx,Ny,child.step);

    Error in detect_fast (line 43) [msg,parts(k).Ix,parts(k).Iy,parts(k).Ik] = passmsg(parts(k),parts(par));

    Error in pose.estimate>process (line 38) box = detect_fast(im, model, detector_threshold);

    Error in pose.estimate (line 21) boxes{i} = process(model, samples{i}, scale, nms_threshold);

    Error in pose_calculator.apply (line 24) samples(i).(config.output) = pose.estimate(config.model, ...

    Error in feature_calculator.apply (line 25) sample = calculators{j}(config{j}, sample, varargin{:}, 'Encode', false);

    result =

     []
    

    can you help me with it please?

    opened by hosnasattar 11
  • Erase functionality

    Erase functionality

    I want to add an Erase button which erases any selection previously done. I used an button which when clicked calls a function very similar to the function responsible for Brush, which is:

    for (var y = -1 * brushSize; y <= brushSize; y++) {
        for (var x = -1 * brushSize; x <= brushSize; x++) {
          if (x * x + y * y > brushLimit) continue;
          var offset =
            4 *
            ((pos[1] + y) * this.layers.visualization.canvas.width + (pos[0] + x));
          offsets.push(offset);
          labels.push(label);
        }
      }
      annotator._updateAnnotation(offsets, labels);
    

    with one difference:

     labels.push(label);
    

    The last line inside the for loop I've hardcoded 0 being pushed into labels.

     labels.push(0);
    

    0 represents the background white color so on clicking the Erase button and selecting any area within a selection, it looks like stuff is being erased since the color of the label gets replaced by the background. But in truth this isn't actually erasing anything, just continues to add label with white background. So when I export it, the erased area expectedly still shows up.

    Any pointers on how to delete previously selected area?

    opened by Abdul-Ghani 10
  • How to code RGB and index information (8b)

    How to code RGB and index information (8b)

    Hi @kyamagu

    thank you for such a nice tool to annotate pictures. I'd like to ask if it is possible to do following:

    I already customized your tool and add 3 classes (Dress, Trousers, Jacket) as can be seen on this picture: 33 On the next picture below is annotated image from Pascal VOC 2012 for segmentation, containing RGB information + index for object and also the same for background. 11

    However, output from JS segmenter looks like this: 22

    If it is possible, could you please tell us on this example how to convert image from 32b to 8b and how to code the demanded information into image as it is in attached annotated VOC Pascal above. E.g. for background --> index 0 and RGB{0 0 0} for jacket --> index 3 and RGB{192 128 0}.

    Thank you in advance, much appreciated!

    opened by marticsvk 3
  • can't export annotation image with more than 2 images

    can't export annotation image with more than 2 images

    when I put more than 2 images in the imageURLs field in the json file, I can only view the first two images when I visit index.html. I can view all the other images by clicking "Next" though.

    The problem is: I can annotate the first two images and export the annotation pictures. But as for the other images, I am able to annotate the image, but not able to export it: There is no response when I click the "export" button.

    How can I solve this issue?

    p.s. The browser I use is Firefox(v42.0 Mac). index.html appeared to be blank with Chrome(v47.0 Mac) and Safari(v9.0).

    Thanks, john

    opened by johnzw 3
  • Some code using js-segment-annotator

    Some code using js-segment-annotator

    I am working on a code that uses this project, thought you might be interested.

    Repo: https://github.com/AKSHAYUBHAT/Vision Demo: http://vision.akshaybhat.com/ 3

    opened by ghost 3
  • Anyway way to debug when finding boundaries goes wrong?

    Anyway way to debug when finding boundaries goes wrong?

    I love your work here and greatly appreciate you sharing it. I'm not at all overly familiar with some of the concepts but wanted to use this in a proof of concept.

    I was using most of the default values and simply porting the code over to not be AMD modules. I have everything working except for the most important part - I've introduced a bug in the segmentation logic. What I end up with is almost perfect squares -

    image

    I've been combing over settings and options and trying to see what is different but nothing stands out. I know this isn't specific to your work on this library, but wanted to see if you had any tips for debugging this part of the code or if you had any tips for what could be going wrong?

    question 
    opened by plwalters 2
  • Annotated image are mostly zero pixel value

    Annotated image are mostly zero pixel value

    @kyamagu i used this tool for annotating my images for caffe-segnet,i first saw that all images are darker(you can hardly notice edges of annotated objects) but i thought on pixel level it might be different,but after checking in matlab,the whole image is almolt zero pixel.i would be happy if you could point out what i am doing wrong.thx

    opened by eliethesaiyan 2
  • Add a polygon-tool.

    Add a polygon-tool.

    • polygon tool
      • left click: span line between points
      • right click: erase drawn lines
    • keyboard shortcuts:
      • ctrl-key: switch between SLIC and polygon-tool
    • new buttons
    opened by NicolaiHarich 2
  • Image export

    Image export

    I cannot export the labeled image. I guess it was the incompatibility of the browser. Could you please suggest the right browser to use? Thanks in advance.

    SL

    question 
    opened by lsymuyu 1
  • Possible to update the annotation image after editing?

    Possible to update the annotation image after editing?

    I wanted to know if it is possible to have a save option which will update the annotation image in the data/annotations/ folder.

    If this is possible, where should I add this functionality?

    wontfix 
    opened by ghost 1
  • Polygon is difficult to close

    Polygon is difficult to close

    I find a bit difficult to close and complete a polygon. I checked the code but I don't understand where the relevant part is. How is the polygon closure checked?

    opened by FSet89 0
  • Brush tool used in the lateral border paints pixels on the other side of the image

    Brush tool used in the lateral border paints pixels on the other side of the image

    When using brush tool in the lateral border of one image, the area of the brush tool that stand out paints the pixels on the oposite border of the image.

    opened by polalbacar 0
  • The annotation masks disappear when clicked on Next button

    The annotation masks disappear when clicked on Next button

    I am able to export the annotated masks but as soon as I click on the "next" so that I can annotate the other images, the masks previously made are gone. We have to annotate the images again.

    It seems as if as soon as we hit "next", the app is removing the overlaid mask that has just been made and I am left with nothing. I have just cloned the library and started working.

    question 
    opened by nikhil-occipitaltech 5
  • Better annotation display

    Better annotation display

    Hi, Thanks for this awesome tool. May I propose an improvement in order to speed up the annotation process. I saw this in other tools and it was really efficient : the possibility to display raw images only in already annoted zones. I put below two examples of vegetation / soil annotation.

    Only vegetation toto

    Only soil: tata

    I checked your code but Im not so familiar with images overlay with javascript and I failed to display annotations with alpha = 1. Could you give me an advice ?

    enhancement question 
    opened by jerem-lab 3
  • The Upload image size is limited

    The Upload image size is limited

    HI, I just meet a problem when trying to upload images which are more than 50kb,these images showing in Firefox all are grey with nothing to see. Maybe the image compression is ok, beside that, any solutions ? Thank you for advance..

    wontfix 
    opened by wzx918 2
Releases(v2.0.1)
Owner
Kota Yamaguchi
Research scientist
Kota Yamaguchi
QR Code Generator

In this project, we'll be using some libraries to instantly generate authentic QR Codes and export them in various formats

Hassan Shahzad 3 Jun 02, 2022
Nutrify - take a photo of food and learn about it

Nutrify - take a photo of food and learn about it Work in progress. To make this a thing, we're going to need lots of food images... Start uploading y

Daniel Bourke 93 Dec 30, 2022
Depix is a tool for recovering passwords from pixelized screenshots.

This implementation works on pixelized images that were created with a linear box filter. In this article I cover background information on pixelization and similar research.

23.1k Jan 04, 2023
QR-code Generator with a basic GUI.

Qr_generator_python Qr code generator with a basic GUI. ❔ About the QR-Code-Generator This project Generates QR codes to sites, e-mails and plain text

Tecixck 2 Oct 11, 2021
DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics

DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics.

Frederik Berlaen 344 Jan 06, 2023
Validate arbitrary image uploads from incoming data urls while preserving file integrity but removing EXIF and unwanted artifacts and RCE exploit potential

Validate arbitrary base64-encoded image uploads as incoming data urls while preserving image integrity but removing EXIF and unwanted artifacts and mitigating RCE-exploit potential.

A3R0 1 Jan 10, 2022
Png2Jpg tool will help you convert from png image format to jpg images format.

PNG 2 JPG All codes assume running from root directory. Please update the sys path at the beginning of the codes before running. Over View Png2Jpg too

Nguyễn Trường Lâu 2 Dec 27, 2021
A functional and efficient python implementation of the 3D version of Maxwell's equations

py-maxwell-fdfd Solving Maxwell's equations via A python implementation of the 3D curl-curl E-field equations. This code contains additional work to e

Nathan Zhao 12 Dec 11, 2022
An esoteric visual language that takes image files as input based on a multi-tape turing machine, designed for compatibility with C.

vizh An esoteric visual language that takes image files as input based on a multi-tape turing machine, designed for compatibility with C. Overview Her

Sy Brand 228 Dec 17, 2022
Raven is a tool written in Python3 allowing you to generate an unique image with some text.

🐦 Raven is a tool written in Python3 allowing you to generate an unique image with some text. It does it by searching the text on Google, do

Billy 39 Dec 20, 2022
A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for quickly creating new images from the one assigned to the field.

django-versatileimagefield A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for creat

Jonathan Ellenberger 490 Dec 13, 2022
OctoPrint is the snappy web interface for your 3D printer!

OctoPrint OctoPrint provides a snappy web interface for controlling consumer 3D printers. It is Free Software and released under the GNU Affero Genera

OctoPrint 7.1k Jan 03, 2023
NFT collection generator. Generates layered images

NFT collection generator Generates layered images, whole collections. Provides additional functionality. Repository includes three scripts generate.py

Gleb Gonchar 10 Nov 15, 2022
A simple image to text converter with GUI!

TEXTEMAGE! Textemage is a quick tool that extracts text from images, it is a Python based GUI program(also available in executable version). This is a

Akascape 5 Oct 26, 2022
QR fixer part is standalone but for image to FQR conversion

f-qr-fixer QR fixer part is standalone but for image to FQR conversion it requires Pillow (can be installed with easy_install), qrtools (on ubuntu the

2 Nov 22, 2022
LGVL helper script to batch and convert with lvgl offline image converter

script to batch and convert with lvgl offline image converter

Yohann 1 Oct 05, 2022
Art directed cropping, useful for responsive images

Art direction sets a focal point and can be used when you need multiple copies of the same Image but also in in different proportions.

Daniel 1 Aug 16, 2022
🖼️ Draw Images or GIFs in your terminal

Drawitor Draw Images/GIFs in your terminal. Install pip install drawitor CLI Tool drawitor cat_dancing.gif Library The library is written in a simple

Eliaz Bobadilla 7 Dec 15, 2022
Deep Illuminator is a data augmentation tool designed for image relighting.

Deep Illuminator Deep Illuminator is a data augmentation tool designed for image relighting. It can be used to easily and efficiently genera

George Chogovadze 52 Nov 29, 2022
Wand is a ctypes-based simple ImageMagick binding for Python

Wand Wand is a ctypes-based simple ImageMagick binding for Python, supporting 2.7, 3.3+, and PyPy. All functionalities of MagickWand API are implement

Eric McConville 1.2k Jan 03, 2023