xpdt: eXPeditious Data Transfer
About
xpdt is (yet another) language for defining data-types and generating code for serializing and deserializing them. It aims to produce code with little or no overhead and is based on fixed-length representations which allows for zero-copy deserialization and (at-most-)one-copy writes (source to buffer).
The generated C code, in particular, is highly optimized and often permits the elimination of data-copying for writes and enables optimizations such as loop-unrolling for fixed-length objects. This can lead to read speeds in excess of 500 million objects per second (~1.8 nsec per object).
Examples
The xpdt source language looks similar to C struct definitions:
struct timestamp {
u32 tv_sec;
u32 tv_nsec;
};
struct point {
i32 x;
i32 y;
i32 z;
};
struct line {
timestamp time;
point line_start;
point line_end;
bytes comment;
};
Fixed width integer types from 8 to 128 bit are supported, along with the bytes type, which is a variable-length sequence of bytes.
Target Languages
The following target languages are currently supported:
- C
- Python
The C code is very highly optimized.
The Python code is about as well optimized for CPython as I can make it. It uses typed NamedTuple for objects, which has some small overhead over regular tuples, and it uses struct.Struct to do the packing/unpacking. I have also code-golfed the generated bytecodes down to what I think is minimal given the design constraints. As a result, performance of the pure Python code is comparable to a JSON library implemented in C or Rust.
For better performance in Python, it may be desirable to develop a Cython target. In some instances CFFI structs may be more performant since they can avoid the creation/destruction of an object for each record.
Target languages are implemented purely as jinja2 templates.
Serialization format
The serialization format for fixed-length objects is simply a packed C struct.
For any object which contains bytes type fields:
- a 32bit unsigned record length is prepended to the struct
- all
bytestype fields are converted tou32and contain the length of the bytes - all bytes contents are appended after the struct in the order in which they appear
For example, following the example above, the serialization would be:
u32 tot_len # = 41
u32 time.tv_sec
u32 time.tv_usec
i32 line_start.x
i32 line_start.y
i32 line_start.z
i32 line_end.x
i32 line_end.y
i32 line_end.z
u32 comment # = 5
u8 'H'
u8 'e'
u8 'l'
u8 'l'
u8 'o'
Features
The feature-set is, as of now, pretty slim.
There are no array / sequence / map types, and no keyed unions.
Support for such things may be added in future provided that suitable implementations exist. An implementation is suitable if:
- It admits a zero (or close to zero) overhead implementation
- it causes no overhead when the feature isn't being used
License
The compiler is released under the GPLv3.
The C support code/headers are released under the MIT license.
The generated code is yours.
157 Dec 16, 2022
40 Dec 22, 2022
31 Jun 27, 2022
47 Jun 30, 2022
25 Dec 8, 2022
370 Dec 27, 2022
38 Oct 14, 2022
131 Dec 19, 2022
33 Jan 5, 2023
1.3k Jan 07, 2023
179 Jan 02, 2023
394 Dec 30, 2022
2 Dec 19, 2022
32 Dec 23, 2022
1 Jan 23, 2022
25 Nov 24, 2022
75 Dec 22, 2022
267 Dec 23, 2022
31 Oct 07, 2022
258 Jan 02, 2023
29 Dec 05, 2022
132 Dec 24, 2022
1 Nov 17, 2021
1 Dec 18, 2021
1.2k Jan 04, 2023
2 Oct 12, 2022
50 Dec 29, 2022
17 Dec 12, 2022
1 Jul 14, 2022