Software to help automate collecting crowdsourced annotations using Mechanical Turk.

Overview

Video Crowdsourcing

Software to help automate collecting crowdsourced annotations using Mechanical Turk.

The goal of this project is to enable crowdsourced collection of annotations on video data. This was built to collect skill annotations on medium length snippets of video (1-2 minutes), but was built with flexibility in mind so researchers can adapt the code to fit their needs.


How it Works

Videos from a YouTube playlist are used to programatically build surveys, including a "qualification" survey to verify responses. These surveys are sent to Mechanical Turk to create HITs for crowd workers. Once on Mechanical Turk, this software includes tools to manage payments to workers who do and do not pass the qualification questions. Finally, all responses from the workers can be collected in one place.


Instructions

1) Install requirements

You will need:

  • Access to a command line (terminal)
  • Download of this respository
    • git clone https://github.com/mpeven/Video_Crowdsourcing.git
  • Python
    • Note: this can be done easily using Conda to install Python and required libraries
  • Installation of required libraries
    • If using conda: conda install -c conda-forge --file requirements.txt
    • If using pip: pip install -r requirements.txt

2) Run Command Line Interface (CLI)

The CLI can be run with python main.py and should guide you through the rest of steps outlined below. Refer to this README if more details are needed.

3) Upload videos

  1. Upload videos to YouTube
    • Go to https://studio.youtube.com/ and click 'Create' to upload videos
    • Make sure videos are published and do not have 'Draft' status
    • IMPORTANT: Make sure videos are listed as Unlisted or Public (Private YouTube videos can't be seen in the survey)
  2. Create YouTube playlists for qualification videos and survey (un-annotated) videos
    • Once the videos are uploaded, create these two playlists and move them into the correct playlist
  3. Put title of the YouTube playlists in the SURVEY section of the config file

4) Create surveys

  1. Get access to YouTube Data API
    • Instructions here: link
    • IMPORTANT: Make sure you set "Application type" as Desktop app when you are on the page "Create OAuth client ID"
    • Download the JSON file of the OAuth client secrets and remember the path for the next step
  2. Fill out the needed sections of the config file
    • YOUTUBE section: oauth client secrets json file location
    • SURVEY section: number of videos per survey
  3. Create surveys using the option in the CLI
  4. Verify the survey is correct by opening the sample survey in a web browser

5) MTURK steps

  1. Create an AWS account
    • Instructions here: link
    • Put the access keys in the config file
  2. Create a Mechanical Turk Account
  3. Create a Mechanical Turk "Sandbox" Account for testing
  4. Upload sandbox-mode HITs using CLI
  5. Upload live HITs using CLI
  6. Periodically check on status and manage payments

Authors

  • Michael Peven (main contact - mpeven@jhu.edu)
  • Tingwen Guo

This work builds upon previous work done by Anand Malpani and Colin Lea


Acknowledgements

We would like to thank the following for support and funding:

  • Swaroop Vedula
  • Gregory Hager
  • Science of Learning Institute
Owner
Mike Peven
Mike Peven
Utility to extract Fantasy Grounds Unity Line-of-sight and lighting files from a Univeral VTT file exported from Dungeondraft

uvtt2fgu Utility to extract Fantasy Grounds Unity Line-of-sight and lighting files from a Univeral VTT file exported from Dungeondraft This program wo

Andre Kostur 29 Dec 05, 2022
An awesome tool to save articles from RSS feed to Pocket automatically.

RSS2Pocket An awesome tool to save articles from RSS feed to Pocket automatically. About the Project I used to use IFTTT to save articles from RSS fee

Hank Liao 10 Nov 12, 2022
Macro recording and metaprogramming in Python

macro-kit is a package for efficient macro recording and metaprogramming in Python using abstract syntax tree (AST).

8 Aug 31, 2022
.bvh to .mcfunction file converter.

bvh-to-mcf .bvh file to .mcfunction converter

Hanmin Kim 28 Nov 21, 2022
A BlackJack simulator in Python to simulate thousands or millions of hands using different strategies.

BlackJack Simulator (in Python) A BlackJack simulator to play any number of hands using different strategies The Rules To keep the code relatively sim

Hamid 4 Jun 24, 2022
✨ Un générateur d'adresse IP aléatoire totalement fait en Python par moi, et en français.

IP Generateur ❗ Un générateur d'adresse IP aléatoire totalement fait en Python par moi, et en français. 🔮 Avec l'utilisation du module "random", j'ai

MrGabin 3 Jun 06, 2021
A python module to validate input.

A python module to validate input.

Matthias 6 Sep 13, 2022
Generate random german words

Generate random german words / Generiere zufällige deutsche Wörter Getting Started Pip install with pip install zufallsworte Install the library with

Maximilian Freitag 5 Mar 24, 2022
A python module to update the console without flashing.

A python module to update the console without flashing.

Matthias 112 Dec 19, 2022
Tools to connect to and interact with the Mila cluster

milatools The milatools package provides the mila command, which is meant to help with connecting to and interacting with the Mila cluster. Install Re

Mila 32 Dec 01, 2022
Tools for binary data on cassette

Micro Manchester Tape Storage Tools for storing binary data on cassette Includes: Python script for encoding Arduino sketch for decoding Eagle CAD fil

Zack Nelson 28 Dec 25, 2022
Simple web index to use bloom filter for Pwned Passwords

pwbloom Simple web index to use bloom filter for Pwned Passwords The index.py runs a simple CGI web service checking passwords with a bloom filter for

Hanno Böck 4 Nov 23, 2021
Check username

Checker-Oukee Check username It checks the available usernames and creates a new account for them Doesn't need proxies Create a file with usernames an

4 Jun 05, 2022
one_click_kag_server is a program which tries to fully automate the creation of a King Arthur's Gold server.

one_click_kag_server is a program which tries to fully automate the creation of a King Arthur's Gold server.

Benjamin Gorman 4 Jan 05, 2022
A clock app, which helps you with routine tasks.

Clock This app helps you with routine tasks. Alarm Clock Timer Stop Watch World Time (Which city you want) About me Full name: Matin Ardestani Age: 14

Matin Ardestani 13 Jul 30, 2022
Generates a random prnt.sc link and display image.

Generates a random prnt.sc link and display image.

Emirhan 3 Oct 08, 2021
Python Libraries with functions and constants related to electrical engineering.

ElectricPy Electrical-Engineering-for-Python Python Libraries with functions and constants related to electrical engineering. The functions and consta

Joe Stanley 39 Dec 23, 2022
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.

Boltons boltons should be builtins. Boltons is a set of over 230 BSD-licensed, pure-Python utilities in the same spirit as — and yet conspicuously mis

Mahmoud Hashemi 6k Jan 04, 2023
Search, generate & deliver Msfvenom payloads in an quick and easy way

Goal Search, generate & deliver payloads in an quick and easy way Be as simple as possible BUT with all msfvenom payloads. Ever lost time searching th

2 Mar 03, 2022
password generator

Password generator technologies used What is? It is Password generator How to Download? Download on releases Clone repo git clone https://github.com/m

1 Dec 16, 2021