Software to help automate collecting crowdsourced annotations using Mechanical Turk.

Overview

Video Crowdsourcing

Software to help automate collecting crowdsourced annotations using Mechanical Turk.

The goal of this project is to enable crowdsourced collection of annotations on video data. This was built to collect skill annotations on medium length snippets of video (1-2 minutes), but was built with flexibility in mind so researchers can adapt the code to fit their needs.


How it Works

Videos from a YouTube playlist are used to programatically build surveys, including a "qualification" survey to verify responses. These surveys are sent to Mechanical Turk to create HITs for crowd workers. Once on Mechanical Turk, this software includes tools to manage payments to workers who do and do not pass the qualification questions. Finally, all responses from the workers can be collected in one place.


Instructions

1) Install requirements

You will need:

  • Access to a command line (terminal)
  • Download of this respository
    • git clone https://github.com/mpeven/Video_Crowdsourcing.git
  • Python
    • Note: this can be done easily using Conda to install Python and required libraries
  • Installation of required libraries
    • If using conda: conda install -c conda-forge --file requirements.txt
    • If using pip: pip install -r requirements.txt

2) Run Command Line Interface (CLI)

The CLI can be run with python main.py and should guide you through the rest of steps outlined below. Refer to this README if more details are needed.

3) Upload videos

  1. Upload videos to YouTube
    • Go to https://studio.youtube.com/ and click 'Create' to upload videos
    • Make sure videos are published and do not have 'Draft' status
    • IMPORTANT: Make sure videos are listed as Unlisted or Public (Private YouTube videos can't be seen in the survey)
  2. Create YouTube playlists for qualification videos and survey (un-annotated) videos
    • Once the videos are uploaded, create these two playlists and move them into the correct playlist
  3. Put title of the YouTube playlists in the SURVEY section of the config file

4) Create surveys

  1. Get access to YouTube Data API
    • Instructions here: link
    • IMPORTANT: Make sure you set "Application type" as Desktop app when you are on the page "Create OAuth client ID"
    • Download the JSON file of the OAuth client secrets and remember the path for the next step
  2. Fill out the needed sections of the config file
    • YOUTUBE section: oauth client secrets json file location
    • SURVEY section: number of videos per survey
  3. Create surveys using the option in the CLI
  4. Verify the survey is correct by opening the sample survey in a web browser

5) MTURK steps

  1. Create an AWS account
    • Instructions here: link
    • Put the access keys in the config file
  2. Create a Mechanical Turk Account
  3. Create a Mechanical Turk "Sandbox" Account for testing
  4. Upload sandbox-mode HITs using CLI
  5. Upload live HITs using CLI
  6. Periodically check on status and manage payments

Authors

  • Michael Peven (main contact - mpeven@jhu.edu)
  • Tingwen Guo

This work builds upon previous work done by Anand Malpani and Colin Lea


Acknowledgements

We would like to thank the following for support and funding:

  • Swaroop Vedula
  • Gregory Hager
  • Science of Learning Institute
Owner
Mike Peven
Mike Peven
Kanye West Lyrics Generator

aikanye Kanye West Lyrics Generator Python script for generating Kanye West lyrics Put kanye.txt in the same folder as the python script and run "pyth

4 Jan 21, 2022
Link-tree - Script that iterate over the links found in each page

link-tree Script that iterate over the links found in each page, recursively fin

Rodrigo Stramantinoli 2 Jan 05, 2022
DUQ is a python package for working with physical Dimensions, Units, and Quantities.

DUQ is a python package for working with physical Dimensions, Units, and Quantities.

2 Nov 02, 2022
Regression Metrics Calculation Made easy

Regression Metrics Mean Absolute Error Mean Square Error Root Mean Square Error Root Mean Square Logarithmic Error Root Mean Square Logarithmic Error

Ashish Patel 12 Jan 02, 2023
isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type.

isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections and by type. It provides a command line utility, Python library and plugins for various e

Python Code Quality Authority 5.5k Jan 08, 2023
A python app which aggregates and splits costs from multiple public cloud providers into a csv

Cloud Billing This project aggregates the costs public cloud resources by accounts, services and tags by importing the invoices from public cloud prov

1 Oct 04, 2022
This python program will display all SSID usernames and SSID passwords you once connected to your laptop

Windows-Wifi-password-extractor This python program will display all SSID usernames and SSID passwords you once connected to your laptop How to run th

Bhaskar Pal 3 Apr 26, 2022
This repository contains scripts that help you validate QR codes.

Validation tools This repository contains scripts that help you validate QR codes. It's hacky, and a warning for Apple Silicon users: the dependencies

Ryan Barrett 8 Mar 01, 2022
Standard implementations of FedLab and its provided benchmarks.

FedLab-benchmarks This repo contains standard implementations of FedLab and its provided benchmarks. Currently, following algorithms or benchrmarks ar

SMILELab-FL 104 Dec 05, 2022
a demo show how to dump lldb info to ida.

用一个demo来聊聊动态trace 这个仓库能做什么? 帮助理解动态trace的思想。仓库内的demo,可操作,可实践。 动态trace核心思想: 动态记录一个函数内每一条指令的执行中产生的信息,并导入IDA,用来弥补IDA等静态分析工具的不足。 反编译看一下 先clone仓库,把hellolldb

25 Nov 28, 2022
This utility synchronises spelling dictionaries from various tools with each other.

This utility synchronises spelling dictionaries from various tools with each other. This way the words that have been trained on MS Office are also correctly checked in vim or Firefox. And vice versa

Patrice Neff 2 Feb 11, 2022
A collection of resources/tools and analyses for the angr binary analysis framework.

Awesome angr A collection of resources/tools and analyses for the angr binary analysis framework. This page does not only collect links and external r

105 Jan 02, 2023
UUID version 7, which are time-sortable (following the Peabody RFC4122 draft)

uuid7 - time-sortable UUIDs This module implements the version 7 UUIDs, proposed by Peabody and Davis in https://www.ietf.org/id/draft-peabody-dispatc

Steve Simmons 22 Dec 20, 2022
Patch the pclntable from Go binaries

Pretrain and Fine-tune a T5 model with Flax on GCP This tutorial details how pretrain and fine-tune a FlaxT5 model from HuggingFace using a TPU VM ava

6 Oct 05, 2022
A multipurpose python module

pysherlock pysherlock is a Python library for dealing with web scraping using images, it's a Python application of the rendertron headless browser API

Sachit 2 Nov 11, 2021
JavaScript to Python Translator & JavaScript interpreter written in 100% pure Python🚀

Pure Python JavaScript Translator/Interpreter Everything is done in 100% pure Python so it's extremely easy to install and use. Supports Python 2 & 3.

Piotr Dabkowski 2.1k Dec 30, 2022
A collection of custom scripts for working with Quake assets.

Custom Quake Tools A collection of custom scripts for working with Quake assets. Features Script to list all BSP files in a Quake mod

Jason Brownlee 3 Jul 05, 2022
python package for generating typescript grpc-web stubs from protobuf files.

grpc-web-proto-compile NOTE: This package has been superseded by romnn/proto-compile, which provides the same functionality but offers a lot more flex

Roman Dahm 0 Sep 05, 2021
A functional standard library for Python.

Toolz A set of utility functions for iterators, functions, and dictionaries. See the PyToolz documentation at https://toolz.readthedocs.io LICENSE New

4.1k Dec 30, 2022
Python script to get some stats on nodes in a Blender material nodetree

Python script to get some stats on nodes in a Blender material nodetree. It counts the nodes, the node types and the max deep level for group nodes.

Alek Mugnozzo 2 Sep 03, 2022