Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Google Sheets Python API

Google Spreadsheets Python API v4 Simple interface for working with Google Sheets. Features: Open a spreadsheet by title, key or url. Read, write, and

Anton Burnashev 6.2k Dec 30, 2022
TM1py is a Python package that wraps the TM1 REST API in a simple to use library.

By wrapping the IBM Planning Analytics (TM1) REST API in a concise Python framework, TM1py facilitates Python developments for TM1. Interacting with T

Cubewise CODE 147 Dec 15, 2022
1.本项目采用Python Flask框架开发提供(应用管理,实例管理,Ansible管理,LDAP管理等相关功能)

op-devops-api 1.本项目采用Python Flask框架开发提供(应用管理,实例管理,Ansible管理,LDAP管理等相关功能) 后端项目配套前端项目为:op-devops-ui jenkinsManager 一.插件python-jenkins bug修复 (1).插件版本 pyt

3 Nov 12, 2021
🔍 Google Search unofficial API for Python with no external dependencies

Python Google Search API Unofficial Google Search API for Python. It uses web scraping in the background and is compatible with both Python 2 and 3. W

Avi Aryan 204 Dec 28, 2022
Ghostbuster - Eliminate dangling elastic IPs by performing analysis on your resources within all your AWS accounts

Table of Contents Table of Contents Ghostbuster The problem Project Features Ins

Assetnote 182 Dec 24, 2022
REPO USERBOT YANG DIBUAT DARI BERBAGAI REPO USERBOT GITHUB.

Lord Userbot Userbot Yang Digunakan Untuk Bersenang-Senang Di Telegram Repo Lord Userbot Repo Yang Dibuat Alvin Dari Berbagai Repo Userbot Github CARA

Alvin 70 Jan 02, 2023
Turns any script into a telegram bot

pytobot Turns any script into a telegram bot Install pip install --upgrade pytobot Usage Script: while True: message = input() if message == "

Dmitry Kotov 17 Jan 06, 2023
Simple debugger and tester for dico-command.

dp Simple debugger and tester for dico-command. Installation pip install -U dico-dp Usage bot = dico_command.Bot(...) ... bot.load_module("dp") Comma

3 Nov 19, 2022
RedFish API Toolkit

RedFish API Toolkit RedFish API Toolkit Build Status GitHub Actions Travis CI Requirements requirements.txt requirements-dev.txt Dependencies Document

Larry Smith Jr. 1 Nov 20, 2021
pokemon-colorscripts compatible for mac

Pokemon colorscripts some scripts to print out images of pokemons to terminal. Inspired by DT's colorscripts compilation Description Prints out colore

43 Jan 06, 2023
A Telegram bot for Download songs in mp3 format from YouTube and Extract lyrics from Genius.com ❤️

MeudsaMusic A Telegram bot for Download songs in mp3 format from YouTube and Extract lyrics from Genius.com ❤️ Commands Reach @MedusaMusic on Telegram

Bibee 14 Oct 06, 2022
Termux Pkg

PKG Install Termux All Basic Pkg. Installation : pkg update && pkg upgrade && pkg install python && pkg install python2 && pkg install git && git clon

ɴᴏʙɪᴛᴀシ︎ 1 Oct 28, 2021
Using Streamlit to build a simple UI on top of the OpenSea API

OpenSea API Explorer Using Streamlit to build a simple UI on top of the OpenSea API. 🤝 Contributing Contributions, issues and feature requests are we

Gavin Capriola 1 Jan 04, 2022
ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

ShadowClone allows you to distribute your long running tasks dynamically across thousands of serverless functions and gives you the results within seconds where it would have taken hours to complete

240 Jan 06, 2023
PlexAutoSkip - Automatically skip content in Plex

PlexAutoSkip Automatically skip tagged content in Plex A background python scrip

Michael Higgins 97 Dec 21, 2022
veez music bot is a telegram music bot project, allow you to play music on voice chat group telegram.

🎶 VEEZ MUSIC BOT Veez Music is a telegram bot project that's allow you to play music on telegram voice chat group. Requirements 📝 FFmpeg NodeJS node

levina 143 Jun 19, 2022
A Discord token stealer app written in Python 3.

Discord Token Stealer A Discord token stealer app written in Python 3. This version of the grabber only supports Windows. Features No local caching Tr

cankat 45 Jan 03, 2023
Tracker to check the covid shot slot availability in India and send mobile alerts via Twilio Messaging Service.

Cowin-Slot-Tracker Tracker to check the covid vaccine slot availability in India and send mobile notifications through Twilio Messaging Service. Requi

invalid username 27 Nov 12, 2022
It was increasingly cumbersome to eye-grep CF output in the AWS console.

cfplot Overview It was increasingly cumbersome to eye-grep CF output in the AWS console. I couldn't find another tool out there to provide individual

46 Dec 26, 2022
A tool for transferring server variable values from one intersect gamedata.db to another

Server Variable Transfer Tool Purpose This tool exists for use with the Intersect Engine (Ascension Game Dev GitHub). Its purpose is to UPDATE one sql

AVild 2 Oct 27, 2021