Python MapReduce library written in Cython.

Related tags

Miscellaneoushadoopy
Overview
Brandyn White 
  
Andrew Miller 
  
   

Source  
   https://github.com/bwhite/hadoopy/
Issues  
   https://github.com/bwhite/hadoopy/issues
Docs    
   http://bwhite.github.com/hadoopy/

IRC: #hadoopy @ freenode.net

Requirements
python development headers (python-dev), build tools (build-essential)

Optional
cython (>=.13) (without this it falls back to the pregenerated .c files)

Features
- oozie support
- Automated job parallelization 'auto-oozie' available in the hadoopy_flow project (maintained out of branch)
- typedbytes support (very fast)
- Local execution of unmodified MapReduce job with launch_local
- Read/write sequence files of TypedBytes directly to HDFS from python (readtb, writetb)
- Works on OS X
- Allows printing to stdout and stderr in Hadoop tasks without causing problems (uses the 'pipe hopping' technique, both are available in the task's stderr)
- critical path is in Cython
- works on clusters without any extra installation, Python, or any Python libraries (uses Pyinstaller that is included in this source tree)
- Simple HDFS access (readtb and ls) inside Python, even inside running jobs
- Unit test interface
- Reporting using status and counters (and print statements! no need to be scared of them in Hadoopy)
- Supports design patterns in the Lin/Dyer book (
   http://www.umiacs.umd.edu/~jimmylin/book.html)

Limitations
- Hadoop Local currently unsupported due to a bug in Hadoop's handling of the distributed cache in this mode.  Use psuedo-distributed instead for now.  (
   https://github.com/bwhite/hadoopy/issues/40)

Used in
- A Case for Query by Image and Text Content: Searching Computer Help using Screenshots and Keywords (to appear in WWW'11)
- Web-Scale Computer Vision using MapReduce for Multimedia Data Mining (at KDD'10)
- Vitrieve: Visual Search engine
- Picarus: Hadoop computer vision toolbox

Ubuntu Install (others are similar)
sudo apt-get install python-dev build-essential
sudo python setup.py install
  
 
Owner
Brandyn White
Brandyn White
🤖🤖 Jarvis is an virtual assistant which can some tasks easy for you like surfing on web opening an app and much more... 🤖🤖

Jarvis 🤖 🤖 Jarvis is an virtual assistant which can some tasks easy for you like surfing on web opening an app and much more... 🤖 🤖 Developer : su

1 Nov 08, 2021
Safe temperature monitor for baby's room. Made for Raspberry Pi Pico.

Baby Safe Temperature Monitor This project is meant to build a temperature safety monitor for a baby or small child's room. Studies have shown the ris

Jeff Geerling 72 Oct 09, 2022
Uma versão em Python/Ursina do aplicativo Real Drum (android).

Real Drum Descrição Esta é uma versão alternativa feita em Python com a engine Ursina do aplicatio Real Drum (presente no Google Play Store). Como exe

hayukimori 5 Aug 20, 2022
Larvamatch - Find your larva or punk match.

LarvaMatch Find your larva or punk match. UI TBD API (not started) The API will allow you to specify a punk by token id to find a larva match, and vic

1 Jan 02, 2022
synchronize projects via yaml/json manifest. built on libvcs

vcspull - synchronize your repos. built on libvcs Manage your commonly used repos from YAML / JSON manifest(s). Compare to myrepos. Great if you use t

python utilities for version control 200 Dec 20, 2022
API to summarize input text

summaries API to summarize input text normal run $ docker-compose exec web python -m pytest disable warnings $ docker-compose exec web python -m pytes

Brad 1 Sep 08, 2021
The fetch of the delegator list and the input of the epoch nonce need to be done independently

raffle The fetch of the delegator list and the input of the epoch nonce need to be done independently. Get the list of delegators at the epoch change.

1 Dec 15, 2021
A gamey, snakey esoteric programming language

Snak Snak is an esolang based on the classic snake game. Installation You will need python3. To use the visualizer, you will need the curses module. T

David Rutter 3 Oct 10, 2022
Demo of a WAM Prolog implementation in Python

Prol: WAM demo This is a simplified Warren Abstract Machine (WAM) implementation for Prolog, that showcases the main instructions, compiling, register

Bruno Kim Medeiros Cesar 62 Dec 26, 2022
A normal phoneNumber tracker made with python.

A normal phoneNumber tracker made with python.

CLAYZANE 2 Dec 30, 2021
Timetable scripts for python

Timetable Scripts timetable_to_json: https://beta.elektronplus.pl/timetable classes_taught_by_teacher: a.adam (aa) ['1Tc', '1Td', '3Te', '3Ti', '4Tf',

Elektron++ 2 Jan 02, 2022
Telegram bot to remove the forwarded tag from messages.

Anonymous Sender Bot @AnonySendBot Telegram bot to remove the forwarded tag from messages. Table of Contents Usage Deploy To Heroku Local Deploying En

Stark Bots 26 Nov 24, 2022
peace-performance (Rust) binding for python. To calculate star ratings and performance points for all osu! gamemodes

peace-performance-python Fast, To calculate star ratings and performance points for all osu! gamemodes peace-performance (Rust) binding for python bas

9 Sep 19, 2022
A simple bot that will help you in your learning and make it more fun.

hyperskill-SimpleChattyBot-python A simple bot that will help you in your learning and make it more fun. Syntax bot.py Stages Stage #1: Zuhura Bot we

1 Nov 09, 2021
4Geeks Academy Full-Stack Developer program final project.

Final Project Chavi, Clara y Pablo 4Geeks Academy Full-Stack Developer program final project. Authors Javier Manteca - Coding - chavisam Clara Rojano

1 Feb 05, 2022
BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset.

BloodCheck BloodCheck enables Red and Blue Teams to manage multiple Neo4j databases and run Cypher queries against a BloodHound dataset. Installation

Mr B0b 16 Nov 05, 2021
List Less Than Ten with python

List Less Than Ten with python

PyLaboratory 0 Feb 07, 2022
You can change your mac address with this program.

1 - Warning! You can use this program with Kali Linux. Therefore if you don't install the Kali Linux. Firstly you need to install Kali Linux. 2 - Star

Mustafa Bahadır Doğrusöz 1 Jun 10, 2022
adbsync - An ADB syncing helper

adbsync - An ADB syncing helper What's this? Everytime I wanted to make a backup of my phone, or restore those files onto it, I had to use everytime t

Giovanni Gualtieri 3 Aug 05, 2022
Grammar of Scalable Linked Interactive Nucleotide Graphics

Gosling.js Gosling.js is a declarative grammar for interactive (epi)genomics visualization on the Web. ⚠️ Please be aware that the grammar of Gosling.

Gosling 126 Nov 29, 2022