XBin Format

The XBin (XINA Binary) format provides a XINA standard binary format for time based data files. It uses the file extension xbin.
The xbin format organizes key-value data by time. The data content is a series of rows in ascending time order, with each row having a single microsecond precision Unix time, unique within the file.
Segment Format
XBin data is often encoded in segments, which are defined by an initial 1, 2, or 4 byte unsigned integer length, then that number of bytes. These are referred to in this document as:

seg1 (up to 255 bytes)

seg2 (up to 65,535 bytes)

seg4 (up to 2,147,483,647 bytes)

If the length value of a segment is zero there is no following data and the value is considered empty.
Examples
The string "foo" has a 3 byte UTF-8 encoding: 0x66, 0x6f, 0x6f.
As a seg1, this is encoded with a total of 4 bytes (the initial byte containing the length, 3):
0x03 0x66 0x6f 0x6f
As a seg2, 5 bytes:
0x00 0x03 0x66 0x6f 0x6f
And as a seg4, 7 bytes:
0x00 0x00 0x00 0x03 0x66 0x6f 0x6f
Value Format
Each value starts with a 1 byte unsigned integer indicating the value type, followed by additional byte(s) containing the value itself, as applicable.
Value Type Definition

Code
Value
Length (bytes)
Description

0
null
0
literal null / empty string

1
ref dict index
1
index 0 to 255 (see below)

2
ref dict index
2
index 256 to 65,535

3
ref dict index
4
index 65,536 to 2,147,483,647

4
true
0
boolean literal

5
false
0
boolean literal

6
int1
1
1 byte signed integer

7
int2
2
2 byte signed integer

8
int4
4
4 byte signed integer

9
int8
8
8 byte signed integer

10
float4
4
4 byte floating point

11
float8
8
8 byte floating point

12
string1
variable
seg1 UTF-8 encoded string

13
string2
variable
seg2 UTF-8 encoded string

14
string4
variable
seg4 UTF-8 encoded string

15
json1
variable
seg1 UTF-8 encoded JSON

16
json2
variable
seg2 UTF-8 encoded JSON

17
json4
variable
seg4 UTF-8 encoded JSON

18
jsonarray1
variable
seg1 UTF-8 encoded JSON array

19
jsonarray2
variable
seg2 UTF-8 encoded JSON array

20
jsonarray4
variable
seg4 UTF-8 encoded JSON array

21
jsonobject1
variable
seg1 UTF-8 encoded JSON object

22
jsonobject2
variable
seg2 UTF-8 encoded JSON object

23
jsonobject4
variable
seg4 UTF-8 encoded JSON object

24
bytes1
variable
seg1 raw byte array

25
bytes2
variable
seg2 raw byte array

26
bytes4
variable
seg4 raw byte array

27
xstring1
variable
seg1 xstring

28
xstring2
variable
seg2 xstring

29
xstring4
variable
seg4 xstring

30
xjsonarray1
variable
seg1 xjson array

31
xjsonarray2
variable
seg2 xjson array

32
xjsonarray4
variable
seg4 xjson array

33
xjsonobject1
variable
seg1 xjson object

34
xjsonobject2
variable
seg2 xjson object

35
xjsonobject4
variable
seg4 xjson object

36 - 255
unusued, reserved

XString Format
The xstring value type allows chaining mutliple encoded values to be interpretted as a string. The xstring segment length must be the total number of bytes of all encoded values in the string.
Note that although any data type may be included in an xstring, the exact string representation of certain values may vary depending on the decoding environment (specifically, the formatting of floating point values) and thus it is not recommended to include them in xstring values. JSON values will be converted to their minimal string representation. Byte arrays will be converted to a hex string. Null values will be treated as an empty string.
XJSON Array Format
The xjsonarray value type allows chaining mutliple encoded values to be interpretted as a JSON array. The xjsonarray segment length must be the total number of bytes of all encoded values in the array.
XJSON Object Format
The xjsonobject value type allows chaining mutliple encoded values to be interpretted as a JSON object. Each pair of values in the list is interpretted as a key-value pair. The xjsonobject segment length must be the total number of bytes of all encoded key-value pairs in the object. Note that key values must resolve to a string, xstring, number, boolean, or null (which will be interpretted as an empty string key).
Examples
Null Value:

Code
Content (0 bytes)

0x00

300 (as 2 byte integer):

Code
Content (2 bytes)

0x07
0x01 0x2c

0.24 (as 8 byte float):

Code
Content (8 bytes)

0x0b
0x3f 0xce 0xb8 0x51 0xEB 0x85 0x1E 0xb8

"foo" (as string1):

Code
Content (4 bytes)

0x0c
0x03 0x66 0x6f 0x6f

{"foo":"bar"} (as json1):

Code
Content (14 bytes)

0x0f
0x0d 0x7b 0x22 0x66 0x6f 0x6f 0x22 0x3a 0x22 0x62 0x61 0x72 0x22 0x7d

"foo123" (as xstring1, split as string1 "foo" and int1 123):

Code
Content (7 bytes)

0x1b
[ 0x06 ](total length) [ 0x03 0x66 0x6f 0x6f ]("foo") [ 0x04 0x7b ](123)

Reference Dictionary
The xbin format provides user-managed compression through the reference dictionary. It can contain up to the 4 byte signed integer index space (2,147,483,647). The order of values affects the compression ratio; index 0-255 can be represented with a single byte, 256-65,535 with 2 bytes, and above requires 4 bytes.
Binary File Format
UUID
The file starts with a 16 byte binary encoded UUID. This is intended to uniquely identify the file, but the exact implementation and usage beyond this is not explicitly defined as part of the format definition. For XINA purposes two xbin files with the same UUID would be expected to be identical.
Header
A value which must either be null or a jsonobject1, jsonobject2, or jsonobject4. This is currently a placeholder with no defined parameters.
Reference Dict
A seg4 containing 0 to 2,147,483,647 encoded values, which may be referenced by zero based index with the reference dict index value types.
Rows
Each row contains:

8 byte signed integer containing Unix time with microsecond precision
seg4 of row data, containing

header, single value which must either be null or a jsonobject1, jsonobject2, or jsonobject4

one or more key,value pairs

The row header is currently a placeholder with no defined parameters.
Example File
Given a data set with UUID 9462ef87-f232-4694-922c-12b93c95e27c:

t
voltage
current
label

0
5
10
"foo"

1

"bar"

2
5
null

A corresponding xbin file containing the same data would be:
UUID (16 bytes)
0x94 0x62 0xef 0x87 0xf2 0x32 0x46 0x94 0x92 0x2c 0x12 0xb9 0x3c 0x95 0xe2 0x7c
Header (1 byte)
0x00 (null, 1 byte)
Reference Dict, three values, "voltage", "current", "label" (29 bytes)
0x00 0x00 0x00 0x19 (seg4 length, 25)
0x0a 0x07 0x76 0x6f 0x6c 0x74 0x61 0x67 0x65 ("voltage", 9 bytes)
0x0a 0x07 0x63 0x75 0x72 0x72 0x65 0x6e 0x74 ("current", 9 bytes)
0x0a 0x05 0x6c 0x61 0x62 0x65 0x6c ("label", 7 bytes)
Row t0 (22 bytes)
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 (time, 0, 8 bytes)
0x00 0x00 0x00 0x0e (row length, 15, 4 bytes)
0x00 (header, null, 1 byte)
0x01 0x00 (reference to index 0, "voltage", 2 bytes)
0xff (type code reference to index 0, 5, 1 byte)
0x01 0x01 (reference to index 1, "current", 2 bytes)
0x04 0x0a (integer value 10, 2 bytes)
0x01 0x02 (reference to index 2, "label", 2 bytes)
0x0a 0x03 0x66 0x6f 0x6f (string "foo", 5 bytes)
Row t1 (20 bytes)
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x01 (time, 1, 8 bytes)
0x00 0x00 0x00 0x08 (row length, 8, 4 bytes)
0x00 (header, null, 1 byte)
0x01 0x02 (reference to index 2, "label", 2 bytes)
0x0a 0x03 0x62 0x61 0x72 (string "bar", 5 bytes)
Row t2 (19 bytes)
0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x02 (time, 2, 8 bytes)
0x00 0x00 0x00 0x0e (row length, 15, 4 bytes)
0x00 (header, null, 1 byte)
0x01 0x00 (reference to index 0, "voltage", 2 bytes)
0x00 (type code reference to index 0, 5, 1 byte)
0x01 0x01 (reference to index 1, "current", 2 bytes)
0x00 (null, 1 byte)