Skip to main content

Supplementary Files and Telemetry File Preprocessing

The 699 Python tools create metadata to speed up and simplify its operations:

.index.tm.mom Files

To verify that packets have not been corrupted, the Python tools verify that each packet's checksum matches that packet's data. To avoid needlessly repeating this expensive operation, the Python tools create a .index.tm.mom file in the same directory as the tm.mom file which contains <packet type>:<packet length> entries. For example:

8:179
8:22
17:484

The first entry represents a packet of type 8 that is 179 bytes long (this length includes the packet header and checksum).

If the Python tools detect a .index.tm.mom file, the tools will use these packet entries to quickly extract packets from the file without recomputing the checksum. Corrupted packets (such as packets with checksums that don't match their data) will have an entry with the packet type set to zero. Upon encountering such an entry, the Python tools will issue a warning, telling the user the indices of the skipped bytes.

If the tm.mom file has a newer timestamp than the .index.tm.mom file, the .index file will be deleted, and a new one will be computed. Creating a new .index file includes verifying all packet checksums. In this way, packet checksums are verified only once per tm.mom file change.

(This description uses the term ".index.tm.mom" for the sake of clarity, but .index files are not specific to MOMA.)

GSE Packet Metadata

The Python tools place additional metadata in the second packet of each tm.mom file. (The first two packets in every tm.mom file are GSE-generated packets, so adding this metadata is not changing "real" data.) Similar to .index files, this metadata allows expensive computations to happen only once per TID. This metadata is a single JSON object. For example:

{
 "total_pkts": 6285,
 "earliest_pkt_unix_timestamp": 1418415862.637699,
 "earliest_pkt_sclk": 74300701,
 "earliest_pkt_unix_timestamp_no_rsim_correction": 1418408433.567599,
 "earliest_pkt_sclk_no_rsim_correction": 1,
 "last_pkt_unix_timestamp": 1418420434.78211,
 "last_pkt_sclk": 4970713,
 "first_time_pkt_rover_seconds": 471680511,
 "first_time_pkt_rover_subseconds": 56741,
 "first_time_pkt_moma_ticks": 151142,
 "timestamp_override_table": []
}

These fields are defined as follows:

  • "total_pkts": the total number of packets in the tm.mom file
  • "earliest_pkt_unix_timestamp": the Unix timestamp (in seconds) of the earliest packet in the tm.mom file, AFTER RSIM packet timestamp correction is taken into account.
  • "earliest_pkt_sclk": the raw timestamp in the packet header of the earliest packet in the tm.mom file, AFTER RSIM packet timestamp correction is taken into account.
  • "earliest_pkt_unix_timestamp_no_rsim_correction": the Unix timestamp (in seconds) of the earliest packet in the tm.mom file, BEFORE RSIM packet timestamp correction is taken into account.
  • "earliest_pkt_sclk_no_rsim_correction": the raw timestamp in the packet header of the earliest packet in the tm.mom file, BEFORE RSIM packet timestamp correction is taken into account.
  • "last_pkt_unix_timestamp": the Unix timestamp (in seconds) of the last (i.e., youngest) packet in the tm.mom file. This is not the very last packet in the file; this is the packet with the largest Unix timestamp.
  • "last_pkt_sclk": the raw timestamp in the packet header of the last (i.e., youngest) packet in the tm.mom file. This is not the very last packet in the file; this is the packet with the largest Unix timestamp.
  • "first_time_pkt_rover_seconds": the rover seconds field (HKID 526 or HKID 550) of the first packet that can be used to resolve packet timestamps. This field will not be populated by a digital status packet with null time fields.
  • "first_time_pkt_rover_subseconds": the rover subseconds field (HKID 527 or HKID 551) of the first packet that can be used to resolve packet timestamps.
  • "first_time_pkt_moma_ticks": the FSW 10 MHz ticks field (HKID 528 or HKID 552) of the first packet that can be used to resolve packet timestamps.
  • "timestamp_override_table": Placeholder for an unimplemented feature.

With the exception of total_pkts, any of these fields may be null.

As of this writing, MOMA telemetry files are the only telemetry files with metadata stored in their second GSE packets.

The script preprocess.py will create a copy of a tm.mom file with this JSON data in the second GSE packet's message log field.

.metadata Files

(11/2/15 update: Soon, the 699 Python tools will start using the new tm.meta files for metadata. Some (or all) metadata described in this article will be moved to the tm.meta file.)

It's possible that the Python tools will run on a tm.mom file that does not have metadata in its second GSE packet. If this occurs, the Python tools will create a .metadata.tm.mom file in the same directory as the tm.mom file. This .metadata file contains the same JSON object as described above. Then, future processing of the tm.mom file will be faster, because the Python tools can extract information from the .metadata file instead of examining all packets to determine such information.

.metadata files are intended to avoid the risks involved with altering a tm.mom file on a user's computer. In the unlikely event that a tm.mom file has metadata in its second GSE packet AND a .metadata file, the .metadata file will be ignored.

No Comments
Back to top