Evaluate the performance of BigDL (Distributed Deep Learning on Apache Spark) in big data analysis problems.

Introduction

BigDL is a distributed deep learning library for Apache Spark; with BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

Installation

• Please download BigDL Packages or pip install BigDL (conda)

How to run Program on Spark

Usage: spark-submit-with-bigdl.sh + [options] + file.py

Options:

• master MASTER URL: spark, yarn, k8s, local.
• local[k]: Run Spark locally with k worker threads as logical cores on your machine.
• File.py: File for executing program.

System configuration

Program run on system includes:

• System/Host Processor: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
• CPU(s): 48
• Core(s) per socket: 12
• Socket(s): 2
• Memory: 183 G (free)

Data Description and Run Model

It is a dataset of 60,000 small square 28×28 pixel grayscale images of handwritten single digits between 0 and 9. The MNIST data is split into three parts: 60,000 data points of training data, 10,000 points of test data.

With this BigDL Problem, We use LSTM model for MNIST digit classification problem.

BigDL Performance Evaluation

Execution running time

Computation Evaluation (SPEED UP)