# Supplementary Files and Telemetry File Preprocessing

The 699 Python tools create metadata to speed up and simplify its
operations:

### .index.tm.mom Files 

To verify that packets have not been corrupted, the Python tools verify
that each packet\'s checksum matches that packet\'s data. To avoid
needlessly repeating this expensive operation, the Python tools create a
.index.tm.mom file in the same directory as the tm.mom file which
contains `<packet type>`:`<packet length>` entries. For
example:

`8:179`\
`8:22`\
`17:484`

The first entry represents a packet of type 8 that is 179 bytes long
(this length includes the packet header and checksum).

If the Python tools detect a .index.tm.mom file, the tools will use
these packet entries to quickly extract packets from the file without
recomputing the checksum. Corrupted packets (such as packets with
checksums that don\'t match their data) will have an entry with the
packet type set to zero. Upon encountering such an entry, the Python
tools will issue a warning, telling the user the indices of the skipped
bytes.

If the tm.mom file has a newer timestamp than the .index.tm.mom file,
the .index file will be deleted, and a new one will be computed.
Creating a new .index file includes verifying all packet checksums. In
this way, packet checksums are verified only once per tm.mom file
change.

(This description uses the term \".index.tm.mom\" for the sake of
clarity, but .index files are not specific to MOMA.)

### GSE Packet Metadata 

The Python tools place additional metadata in the second packet of each
tm.mom file. (The first two packets in every tm.mom file are
GSE-generated packets, so adding this metadata is not changing \"real\"
data.) Similar to .index files, this metadata allows expensive
computations to happen only once per TID. This metadata is a single JSON
object. For example:

```json
{
 "total_pkts": 6285,
 "earliest_pkt_unix_timestamp": 1418415862.637699,
 "earliest_pkt_sclk": 74300701,
 "earliest_pkt_unix_timestamp_no_rsim_correction": 1418408433.567599,
 "earliest_pkt_sclk_no_rsim_correction": 1,
 "last_pkt_unix_timestamp": 1418420434.78211,
 "last_pkt_sclk": 4970713,
 "first_time_pkt_rover_seconds": 471680511,
 "first_time_pkt_rover_subseconds": 56741,
 "first_time_pkt_moma_ticks": 151142,
 "timestamp_override_table": []
}
```

These fields are defined as follows:

-   \"total_pkts\": the total number of packets in the tm.mom file
-   \"earliest_pkt_unix_timestamp\": the Unix timestamp (in seconds) of
    the earliest packet in the tm.mom file, **AFTER RSIM packet
    timestamp correction is taken into account.**
-   \"earliest_pkt_sclk\": the raw timestamp in the packet header of the
    earliest packet in the tm.mom file, **AFTER RSIM packet timestamp
    correction is taken into account.**
-   \"earliest_pkt_unix_timestamp_no_rsim_correction\": the Unix
    timestamp (in seconds) of the earliest packet in the tm.mom file,
    **BEFORE RSIM packet timestamp correction is taken into account.**
-   \"earliest_pkt_sclk_no_rsim_correction\": the raw timestamp in the
    packet header of the earliest packet in the tm.mom file, **BEFORE
    RSIM packet timestamp correction is taken into account.**
-   \"last_pkt_unix_timestamp\": the Unix timestamp (in seconds) of the
    last (i.e., youngest) packet in the tm.mom file. This is not the
    very last packet in the file; this is the packet with the largest
    Unix timestamp.
-   \"last_pkt_sclk\": the raw timestamp in the packet header of the
    last (i.e., youngest) packet in the tm.mom file. This is not the
    very last packet in the file; this is the packet with the largest
    Unix timestamp.
-   \"first_time_pkt_rover_seconds\": the rover seconds field (HKID 526
    or HKID 550) of the first packet that can be used to resolve packet
    timestamps. This field will not be populated by a digital status
    packet with null time fields.
-   \"first_time_pkt_rover_subseconds\": the rover subseconds field
    (HKID 527 or HKID 551) of the first packet that can be used to
    resolve packet timestamps.
-   \"first_time_pkt_moma_ticks\": the FSW 10 MHz ticks field (HKID 528
    or HKID 552) of the first packet that can be used to resolve packet
    timestamps.
-   \"timestamp_override_table\": Placeholder for an unimplemented
    feature.

**With the exception of total_pkts, any of these fields may be null.**

As of this writing, MOMA telemetry files are the only telemetry files
with metadata stored in their second GSE packets.

The script preprocess.py will create a copy of a tm.mom file with this
JSON data in the second GSE packet\'s message log field.

### .metadata Files 

(11/2/15 update: Soon, the 699 Python tools will start using the new
tm.meta files for metadata. Some (or all) metadata described in this
article will be moved to the tm.meta file.)

It\'s possible that the Python tools will run on a tm.mom file that does
not have metadata in its second GSE packet. If this occurs, the Python
tools will create a .metadata.tm.mom file in the same directory as the
tm.mom file. This .metadata file contains the same JSON object as
described above. Then, future processing of the tm.mom file will be
faster, because the Python tools can extract information from the
.metadata file instead of examining all packets to determine such
information.

.metadata files are intended to avoid the risks involved with altering a
tm.mom file on a user\'s computer. In the unlikely event that a tm.mom
file has metadata in its second GSE packet AND a .metadata file, the
.metadata file will be ignored.