Python MapReduce library written in Cython.

Related tags

Miscellaneoushadoopy
Overview
Brandyn White 
  
Andrew Miller 
  
   

Source  
   https://github.com/bwhite/hadoopy/
Issues  
   https://github.com/bwhite/hadoopy/issues
Docs    
   http://bwhite.github.com/hadoopy/

IRC: #hadoopy @ freenode.net

Requirements
python development headers (python-dev), build tools (build-essential)

Optional
cython (>=.13) (without this it falls back to the pregenerated .c files)

Features
- oozie support
- Automated job parallelization 'auto-oozie' available in the hadoopy_flow project (maintained out of branch)
- typedbytes support (very fast)
- Local execution of unmodified MapReduce job with launch_local
- Read/write sequence files of TypedBytes directly to HDFS from python (readtb, writetb)
- Works on OS X
- Allows printing to stdout and stderr in Hadoop tasks without causing problems (uses the 'pipe hopping' technique, both are available in the task's stderr)
- critical path is in Cython
- works on clusters without any extra installation, Python, or any Python libraries (uses Pyinstaller that is included in this source tree)
- Simple HDFS access (readtb and ls) inside Python, even inside running jobs
- Unit test interface
- Reporting using status and counters (and print statements! no need to be scared of them in Hadoopy)
- Supports design patterns in the Lin/Dyer book (
   http://www.umiacs.umd.edu/~jimmylin/book.html)

Limitations
- Hadoop Local currently unsupported due to a bug in Hadoop's handling of the distributed cache in this mode.  Use psuedo-distributed instead for now.  (
   https://github.com/bwhite/hadoopy/issues/40)

Used in
- A Case for Query by Image and Text Content: Searching Computer Help using Screenshots and Keywords (to appear in WWW'11)
- Web-Scale Computer Vision using MapReduce for Multimedia Data Mining (at KDD'10)
- Vitrieve: Visual Search engine
- Picarus: Hadoop computer vision toolbox

Ubuntu Install (others are similar)
sudo apt-get install python-dev build-essential
sudo python setup.py install
  
 
Owner
Brandyn White
Brandyn White
Petuhlang is a joke-like language, based on Python.

Petuhlang is a joke-like language, based on Python. It updates builtins to make a new syntax based on operators rewrite.

DenyS 9 Jun 19, 2022
Sardana integration into the Jupyter ecosystem.

sardana-jupyter Sardana integration into the Jupyter ecosystem.

Marc Espín 1 Dec 23, 2021
Algorand Python API examples

Algorand-Py Algorand Python API examples This repo will hold example scripts to monitor activities on Algorand main net. You can: Monitor your assets

Karthik Dutt 2 Jan 23, 2022
Medical appointments No-Show classifier

Medical Appointments No-shows Why do 20% of patients miss their scheduled appointments? A person makes a doctor appointment, receives all the instruct

4 Apr 20, 2022
A few of my adventures with Devito.

Devito-playbox A few of my adventures with Devito. This repository contains a few notebooks and scripts that will lead me in the road of learning this

Átila Saraiva Quintela Soares 1 Feb 08, 2022
This repo is related to Google Coding Challenge, given to Bright Network Internship Experience 2021.

BrightNetworkUK-GCC-2021 This repo is related to Google Coding Challenge, given to Bright Network Internship Experience 2021. Language used here is py

Dareer Ahmad Mufti 28 May 23, 2022
Iris-client - Python client for DFIR-IRIS

Python client dfir_iris_client offers a Python interface to communicate with IRI

DFIR-IRIS 11 Dec 22, 2022
This is a working model for which I have used python.

Jarvis_voiceAssistance This is a working model for which I have used python. This model can: 1)Play a video or song on youtube. 2)Tell us time. 3)Tell

Hardik Jain 1 Jan 30, 2022
Track testrail productivity in automated reporting to multiple teams

django_web_app_for_testrail testrail is a test case management tool which helps any organization to track all consumption and testing of manual and au

Vignesh 2 Nov 21, 2021
Simplified web browser made in python for a college project

Python browser Simplified web browser made in python for a college project. Web browser has bookmarks, history, multiple tabs, toolbar. It was made on

AmirHossein Mohammadi 9 Jul 25, 2022
Secret santa is a fun and easy way to get together with your friends and/or family with a gift for them.

Vaccine Validator Tool to validate domestic New Zealand vaccine passes Create a new virtual environment: python3 -m venv ./venv Activate virtual envi

2 Dec 06, 2021
This repository can help you made a PocketMine-MP Server with Termux apps!

Hello This GitHub repository can made you a Server PocketMine-MP On development! How to Install Open Termux Type "pkg install git && python" If python

1 Mar 04, 2022
tagls is a language server based on gtags.

tagls tagls is a language server based on gtags. Why I wrote it? Almost all modern editors have great support to LSP, but language servers based on se

daquexian 31 Dec 01, 2022
Waydroid is a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu.

Waydroid is a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu.

WayDroid 4.7k Jan 08, 2023
🦕 Compile Deno executables and compress them for all platforms easily

Denoc Compile Deno executables and compress them for all platforms easily. Install You can install denoc from PyPI like any other package: pip install

Eliaz Bobadilla 8 Apr 04, 2022
Module 2's katas from Launch X's python introduction course.

Module2Katas Module 2's katas from Launch X's python introduction course. Virtual environment creation process (on Windows): Create a folder in any de

Javier Méndez 1 Feb 10, 2022
NCAR/UCAR virtual Python Tutorial Seminar Series lesson on MetPy.

The Project Pythia Python Tutorial Seminar Series continues with a lesson on MetPy on Wednesday, 2 February 2022 at 1 PM Mountain Standard Time.

Project Pythia Tutorials 6 Oct 09, 2022
Desafio Final do Mod1 do Bootcamp EDC - v2 usando a RAIS

IGTI - Bootcamp Engenheiro de Dados Cloud Módulo 1 - Desafio Final - RAIS 2020 Código do Desafio Final V2 do Bootcamp Engenheiro de Dados Cloud do IGT

Neylson Crepalde 17 Nov 02, 2022
Logging-monitoring-instrumentation - A brief repository on logging monitoring and instrumentation in Python

logging-monitoring-instrumentation A brief repository on logging monitoring and

Noah Gift 6 Feb 17, 2022
Projeto de análise de dados com SQL

Project-Analizyng-International-Debt-Statistics- Projeto de análise de dados com SQL - Plataforma Data Camp Descrição do Projeto : Não é que nós human

Lorrayne Silva 1 Feb 01, 2022