699util

This page presents a general overview of working with Python for SAM data processing tasks.

Note for SAM users: Distribution through .dmg file is no longer recommended. See PDL_Setup

Note for MOMA users: MOMA Python packages can be found at MOMA_Python_Packages.

Installation Instructions

For installation instructions, go to the Python Tool Installation page.

Bug Reports

Please do not report bugs on this wiki. Instead, use the Bugzilla server we have set up for bug tracking and reporting. Once you have set up an account, you may file a bug for the python software at the page for Command-line Utils.

List of Scripts

Script Name	Description
checksum.py	Calculate the Fletcher's checksum of input or a list of existing files.
datavol.py	Estimate the data volumen of a telemetry file.
faradaycup.py	Correlate faraday cup scans to faraday cup currents.
feiget.py	Download data from FEI.
fei2tm.py	Convert EDRs or DPs to a telemetry file.
gcms.py	Print a tab-delimited table of GCMS data organized by both mass and time.
gcsci.py	Print GC TCD data in a tab-delimited table.
gcsciplot.py	Plot GC TCD data, with or without TCD offset correction.
getcsvreports	Script for PDL to download MSL engineering reports and sort their contents by sol.
hkdump.py	Print the values in a SAM type-10 housekeeping packet.
hot.py	Create a tab-delimited CSV file showing the cumulative number of tics the heaters have been on.
htrduty.py	Plot the duty cycle and temperature of a given SAM heater.
listfei.py	List and sort the DPs or EDRs in the current working directory.
massrange.py	Plot the masses and bands being scanned at a given time.
movefei.py	Move files downloaded from FEI into the appropriate TID folder.
ontime.py	Calculate how long a given component was "on" (e.g., how long a valve open, or how long an HK value was above a certain threshold)
phd.py	Plot the pulse-height distribution at a given marker ID.
powercut.py	Cut a power profile into two independent power profiles.
ql_ega.py	Generate the plots for a quicklook.
runall.py	Run a command or python code in every telemetry directory.
sclk2scet.py	Convert an SCLK timestamp to SCET time.
t0.py	Display the start time of a telemetry file in various formats.
timestamp.py	Print the start time of the file in a human-readable format.
tm2rdr.py	Create a Reduced Data Record from a telemetry file.
tmdb.py	A method used in backtics to get the axis name for a database entry.
tmdiff.py	A diff-like program for telemetry files.
tmdump.py	Produce a hex dump of telemetry file, packet by packet.
tmexcerpt.py	Create a telemetry file containing only a certain marker or packet type.
tmfields.py	Print a tab-delimited table of housekeeping data.
tmglue.py	Concatenate two telemetry files together.
tmkeys.py	Search for and display the telemetry database.
tmmarker.py	Print a list of the markers and marker IDs in the telemetry file.
tmmsg.py	Print the contents of the message log packets.
tmplot.py	Plot housekeeping data.
tmsanitize.py	Remove malformed data from a packet.
tmscan.py	Plot a DAC scan.
tmsequence.py	Print a report detailing whether packet sequence numbers increase normally.
tmstat.py	Print the statistics of a given HK value (min, median, maximum, mean).
tmsummary.py	Print a summary of each packet in a telemetry file.
triplets.py	Print a tab-delimited table of GCMS data organized by time.
valvestat.py	Print the valve operations (like the tmvalves viewer in SAM Data View).

Version History

1.00.00

Source: tmutil-1.00.00.tar.gz

Release Date: 2011-11-18

• Python modules and scripts for processing telemetry data.

1.00.01

Source: tmutil-1.00.01.tar.gz

Release Date: 2011-11-22

• Updated tmenv to look for $HOME/SAM/gse/ in addition to the original $HOME/gse/ setup.
• Fixed a bug which caused the GC-specific sequence number of GC HK Packets to overwrite the packet sequence number.

1.00.02

Source: tmutil-1.00.02.tar.gz

Release Date: 2011-11-30

• Fixed bug which caused tmplot.py to crash when data was missing.

1.00.03

Source: tmutil-1.00.03.tar.gz

Release Date: 2012-12-02

• Added checksum.py script for calculating checksums of files.
• Fixed a bug which calculated the CWD for find_tmfile at the start of execution, not dynamically as the function was called.
• Made relative time default for triplets.py.
• Added regular expression-type searching for tmmsg.py, and grep-like -v (invert) option.
• Added plotref and limref scripts for LADEE EBT processing.
• Numerous bugfixes for problems observed in first round of regression testing.

1.00.04

Source: tmutil-1.00.04.tar.gz

Release Date: 2011-12-15

• Overhauled dead time correction.
• Fixed bug with showing marker IDs in tmplot.
• Fixed boot time correction for LADEE/MAVEN to avoid problems caused by race condition with time sync.

1.00.05

Source: tmutil-1.00.05.tar.gz

Release Date: 2012-01-06

• Simplified options in tmfields.py.
• Added limref
• Added plotref
• Added rf_subpacket_cal.py
• Added subpacket_timing.py
• Added on-line help for almost every script (i.e., -h option)

1.00.06

Source: tmutil-1.00.06.tar.gz

Release Date: 2012-01-09

• Added background subtraction functions.
• Fixed infinite loop bug in tmread.util.step_pair.

1.00.07

Source: (missing)

Release Date: 2012-01-15

• Added newtid.py
• Added godata command for the shell.
• Changed sorting behavior in tmdir to use date rather than TID.

1.01.00

Source: tmutil-1.01.00.tar.gz

Release Date: 2012-03-13

• Added support for MAVEN NGMS data.
• Added lib699 package for functions not specific to telemetry.
• Added checksum.py to default distribution.
• Added massrange.py.
• Added ontime.py.
• Added t0.py.
• Added tmstat.py.
• Added tmsummary.py.

1.01.01

Source: tmutil-1.01.01.tar.gz

Release Date: 2012-03-14

• Fixed grievous errors that prevented installation.

1.02.00

Source: 699util-1.02.00.tar.gz

Revision: 66

Release Date: 2012-04-05

• Bundled TMUtil and SamUtil into 699util.
• Added support for SVN operations scripts.
• Improvements to lib699.
• Workaround to total_ordering bug in Python 2.7.1.

1.02.01

Source: 699util-1.02.01.tar.gz

Revision: 69

Release Date: 2012-05-10

• Fixed bug with setting dac mode in triplets.py.

1.02.02

Source: 699util-1.02.02.tar.gz

Revision: 88

Release Date: 2012-05-14

• Backend changes to Mission and QMS Science Packet classes.
• Incorporation of regression testing-framework (not distributed).

1.03.00

Source: 699util-1.03.00.tar.gz

Revision: 118

Release Date: 2012-05-24

• Incorporated support for time correction as a configurable file. Abandoned old concept of "offset correction."
• Added scripts for generating time configuration files from MSL-generated SCLK-SCET files.

1.03.01

Source: 699util-1.03.01.tar.gz

Revision: 150

Release Date: 2012-06-19

• Added gcsciplot.py
• Added override of Gnuplot.py's insistence on using "aqua" as the default terminal for Macs.
• Backend restructuring of feiutil.
• Added highlighting support to lib699.ttyutil and tmmsg.
• Added preliminary support for SAM Testbed.

1.03.02

Source: 699util-1.03.02.tar.gz

Revision: 167

Release Date: 2012-07-05

• Now tolerates missing SCLK/SCET file (issues a warning).
• Updates to tm2rdr.py and pds3 package.

1.04.00

Source: 699util-1.04.00.tar.gz

Revision: 326

Release Date: 2012-10-06

• Added lots of scripts, including lingering scripts that had not yet been ported to 699util and new SAM ops software
• Added lib699.collections
• Fixed svnutil performance bugs
• A little bit of Python 3 support (not fully implemented)
• tmenv moved into tmread, now uses .699config.INI file
• Added XINA SVN-post-commit hooks

1.05.00

Warning! This version (1.05.00) has a bug causing incorrect outputs for dead-time corrected QMS data. Do not draw conclusions from any data extracted by this version of the software. The bug was fixed in 1.05.01.

Source: 699util-1.05.00.tar.gz (zip version: 699util-1.05.00.zip)

Windows Installer: 699util-1.05.00.macosx-10.6-intel.exe

Revision: 702

Release Date: 2013-04-17

• Too many changes to list. Incorporated every change to software since last October.

1.05.01

Source: 699util-1.05.01.tar.gz (zip version: 699util-1.05.01.zip)

Windows Installer: 699util-1.05.01.exe

Revision: 797

Release Date: 2013-05-31

• Fixed a bug with dead-time correction that causes incorrect data to be printed.
• Packet resequencing now based on packet "creation time" rather than packet timestamp.
• Fixed bug in tmfields which caused program to crash when a telemetry file did not have a defined TID.
• Many minor backend changes and bug fixes.

2.00.00

DMG (Mac) Installation Package: https://s3.amazonaws.com/699_bin/699util/699util-2.00.dmg

TAR Archive (Mac/Linux) Installation Package: https://s3.amazonaws.com/699_bin/699util/699util-2.00-installer.tar.gz

Revision: 1242

Release Date: 2013-09-05

• Now supports Python 3 instead of Python 2
• Improvements to packet parsing behavior
• Representation of time as discrete units
• Reorganization of scripts
• Streamlining of gnuplot interface

2.00.01

DMG (Mac) Installation Package: https://s3.amazonaws.com/699_bin/699util/699util-2.00.01.dmg

TAR Archive (Mac/Linux) Installation Package: https://s3.amazonaws.com/699_bin/699util/699util-2.00.01-installer.tar.gz

Revision: 1249

Release Date: 2013-09-12

• Added tm2igor.pl and igormeta.py.
• Fixed bug with ontime.py; now correctly tracks the last stretch for which a value is considered "on"
• Added samvtool alias to shell configuration script.

2.00.02

DMG (Mac) Installation Package: https://s3.amazonaws.com/699_bin/699util/699util-2.00.02.dmg

TAR Archive (Mac/Linux) Installation Package: https://s3.amazonaws.com/699_bin/699util/699util-2.00.02-installer.tar.gz

Revision: 1256

Release Date: 2013-09-13

• Added smscuplog.py.
• Full working version of nms2pds.py.
• Added nms_assemble.py.

Workaround (From Mike Wong) If installation fails with an error of AttributeError: \_convert while building the enum34 package, try: Delete the tarfile enum34-1.0.tar.gz that came in the 699util*dmg disk image. Replace it with a newer tarfile from a link such as pypi.python.org/pypi/enum34/1.1.6. Edit the file requirements.txt, replacing "enum34==1.0" with "enum34>=1.1.6". Then try the install script again.