Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.

Photo by Denny Müller on Unsplash AWS Automated Inventory ( aws-auto-inventory ) Automates creation of detailed inventories from AWS resources. Table

AWS Samples 123 Dec 26, 2022
A Open source Discord Token Grabber with several very useful features coded in python 3.9

Kiwee-Grabber A Open source Discord Token Grabber with several very useful features coded in python 3.9 This only works on any python 3.9 versions. re

Vesper 40 Jan 01, 2023
A Discord Server Cloner Which Can Clone Any Discord Server In Just Few Minutes

A Discord Server Cloner Which Can Clone Any Discord Server In Just Few Minutes.

samet 4 Jul 23, 2022
Twitter for Python!

Tweepy: Twitter for Python! Installation The easiest way to install the latest version from PyPI is by using pip: pip install tweepy You can also use

9.4k Jan 07, 2023
Easy Google Translate: Unofficial Google Translate API

easygoogletranslate Unofficial Google Translate API. This library does not need an api key or something else to use, it's free and simple. You can eit

Ahmet Eren Odacı 9 Nov 06, 2022
unofficial source of the discord bot, “haunting.” created by: vorqz, vert, & Veltz

hauntingSRC unofficial source of the discord bot, “haunting.” created by: vorqz, vert, & Veltz reasoning: creators skidded the most of this bot and do

Vast 11 Nov 04, 2022
Python client for Vektonn

Python client for Vektonn Installation Install the latest version: $ pip install vektonn Install specific version: $ pip install vektonn==1.2.3 Upgrad

Vektonn 16 Dec 09, 2022
Python Twitter API

Python Twitter Tools The Minimalist Twitter API for Python is a Python API for Twitter, everyone's favorite Web 2.0 Facebook-style status updater for

Mike Verdone 2.9k Jan 03, 2023
CRUD database for python discord bot developers that stores data on discord text channels

Discord Database A CRUD (Create Read Update Delete) database for python Discord bot developers. All data is stored in key-value pairs directly on disc

Ankush Singh 7 Oct 22, 2022
Automatically pulls specified repository whenever a specified file is pushed. Great for working collaboratively when you need to run something locally.

autopull Simple python tool that allows you to automatically pull from a github repository whenever a file with a specified name is uploaded installat

carreb 0 Sep 27, 2022
The Social-Engineer Toolkit (SET) is specifically designed to perform advanced attacks against the human element.

The Social-Engineer Toolkit (SET) The Social-Engineer Toolkit (SET) is specifically designed to perform advanced attacks against the human element. SE

Professor 6 Nov 28, 2022
Client to allow skytrack to be used with GSPro Golf simulator application

Skytrack Interface for GSPro A Basic Interface connection from Skytrack Launch Monitors to be able to play simulator golf via GSPro About The Project

James Peruggia 2 Oct 24, 2021
A replacement for Reddit /r/copypasta CummyBot2000 with extra measures to avoid it being banned.

CummyBot1984 A replacement for Reddit /r/copypasta's CummyBot2000 with extra measures to respect Reddit's API rules. Features Copies and replies to ev

2 Feb 21, 2022
Python library for RetroMMO related stuff, including API wrapper

python library for RetroMMO related stuff, including API wrapper.

1 Nov 25, 2021
A Multi-Tool with 30+Options.

A Multi-Tool with 30+Options.

Mervin404 15 Apr 12, 2022
Yes, it's true :revolving_hearts: This repository has 301 stars.

Yes, it's true! Inspired by a similar repository from @RealPeha, but implemented using a webhook on AWS Lambda and API Gateway, so it's serv

510 Dec 28, 2022
This is RequestTrackerBot and it used for tracking request made by user in a group

This is a Request Tracker Bot repo, It is for those who upload content like movies, anime, etc. It can be used for tracking request of content that your members asked for.

Abhijeet 27 Dec 29, 2022
🚀🔥使用Python连接阿里云盘, 实现了官方大部分功能 👍👍

aligo 🚀 🔥 使用Python连接阿里云盘, 实现了官方大部分功能 👍 👍 为了完善代码提示, 方便大家代码书写, aligo 引入了一些 python 3.8 的新特性, 所以要求 python = 3.8.* pip install aligo 或 pip install ali

455 Jan 08, 2023
Integrating the Daraja-Api with Python language

Mpesa-Daraja-Api Integrating the Daraja-Api with Python language. Credentials.py file This file contains the consumer key and the consumer secrete key

Morvin Ian 3 Nov 09, 2022
A Python API to retrieve and read MLB GameDay data

mlbgame mlbgame is a Python API to retrieve and read MLB GameDay data. mlbgame works with real time data, getting information as games are being playe

Zach Panzarino 493 Dec 13, 2022