# 699util

This page presents a general overview of working with Python for SAM
data processing tasks.

**Note for SAM users:** Distribution through .dmg file is no longer
recommended. See [PDL_Setup](PDL_Setup "wikilink")

**Note for MOMA users:** MOMA Python packages can be found at
[MOMA_Python_Packages](https://wiki.xina.io/books/moma/page/moma-python-packages).

## Installation Instructions

For installation instructions, go to the [Python Tool
Installation](https://wiki.xina.io/link/143#bkmrk-page-title "wikilink") page.

## Bug Reports

Please do not report bugs on this wiki. Instead, use the [Bugzilla
server](https://wush.net/bugzilla/sw699/) we have set up for bug
tracking and reporting. Once you have set up an account, you may file a
bug for the python software at the page for [Command-line
Utils](https://wush.net/bugzilla/sw699/enter_bug.cgi?product=Command-line%20Utils).

## List of Scripts

|Script Name    | Description
|---------------|--------------------------------------------------------------------------------------------------------------------------------------
| checksum.py  |   Calculate the Fletcher\'s checksum of input or a list of existing files.
 | datavol.py     | Estimate the data volumen of a telemetry file.
  faradaycup.py  | Correlate faraday cup scans to faraday cup currents.
  feiget.py      | Download data from FEI.
  fei2tm.py     |  Convert EDRs or DPs to a telemetry file.
  gcms.py     |    Print a tab-delimited table of GCMS data organized by both mass and time.
  gcsci.py     |   Print GC TCD data in a tab-delimited table.
  gcsciplot.py  |  Plot GC TCD data, with or without TCD offset correction.
  getcsvreports  | Script for PDL to download MSL engineering reports and sort their contents by sol.
  hkdump.py      | Print the values in a SAM type-10 housekeeping packet.
  hot.py         | Create a tab-delimited CSV file showing the cumulative number of tics the heaters have been on.
  htrduty.py    |  Plot the duty cycle and temperature of a given SAM heater.
  listfei.py    | List and sort the DPs or EDRs in the current working directory.
  massrange.py  |  Plot the masses and bands being scanned at a given time.
  movefei.py    |  Move files downloaded from FEI into the appropriate TID folder.
  ontime.py     |  Calculate how long a given component was \"on\" (e.g., how long a valve open, or how long an HK value was above a certain threshold)
  phd.py        |  Plot the pulse-height distribution at a given marker ID.
  powercut.py   |  Cut a power profile into two independent power profiles.
  ql_ega.py     |  Generate the plots for a quicklook.
  runall.py     |  Run a command or python code in every telemetry directory.
  sclk2scet.py  |  Convert an SCLK timestamp to SCET time.
  t0.py         |  Display the start time of a telemetry file in various formats.
  timestamp.py   | Print the start time of the file in a human-readable format.
  tm2rdr.py      | Create a Reduced Data Record from a telemetry file.
  tmdb.py        | A method used in backtics to get the axis name for a database entry.
  tmdiff.py      | A diff-like program for telemetry files.
  tmdump.py      | Produce a hex dump of telemetry file, packet by packet.
  tmexcerpt.py   | Create a telemetry file containing only a certain marker or packet type.
  tmfields.py    | Print a tab-delimited table of housekeeping data.
  tmglue.py      | Concatenate two telemetry files together.
  tmkeys.py      | Search for and display the telemetry database.
  tmmarker.py    | Print a list of the markers and marker IDs in the telemetry file.
  tmmsg.py       | Print the contents of the message log packets.
  tmplot.py      | Plot housekeeping data.
  tmsanitize.py  | Remove malformed data from a packet.
  tmscan.py      | Plot a DAC scan.
  tmsequence.py  | Print a report detailing whether packet sequence numbers increase normally.
  tmstat.py      | Print the statistics of a given HK value (min, median, maximum, mean).
  tmsummary.py   | Print a summary of each packet in a telemetry file.
  triplets.py    | Print a tab-delimited table of GCMS data organized by time.
  valvestat.py   | Print the valve operations (like the tmvalves viewer in SAM Data View).

## Version History

### 1.00.00

**Source:**
[tmutil-1.00.00.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.00.tar.gz)

**Release Date:** 2011-11-18

-   Python modules and scripts for processing telemetry data.

### 1.00.01

**Source:**
[tmutil-1.00.01.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.01.tar.gz)

**Release Date:** 2011-11-22

-   Updated tmenv to look for \$HOME/SAM/gse/ in addition to the
    original \$HOME/gse/ setup.
-   Fixed a bug which caused the GC-specific sequence number of GC HK
    Packets to overwrite the packet sequence number.

### 1.00.02

**Source:**
[tmutil-1.00.02.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.02.tar.gz)

**Release Date:** 2011-11-30

-   Fixed bug which caused tmplot.py to crash when data was missing.

### 1.00.03

**Source:**
[tmutil-1.00.03.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.03.tar.gz)

**Release Date:** 2012-12-02

-   Added checksum.py script for calculating checksums of files.
-   Fixed a bug which calculated the CWD for find_tmfile at the start of
    execution, not dynamically as the function was called.
-   Made relative time default for triplets.py.
-   Added regular expression-type searching for tmmsg.py, and grep-like
    -v (invert) option.
-   Added plotref and limref scripts for LADEE EBT processing.
-   Numerous bugfixes for problems observed in first round of regression
    testing.

### 1.00.04

**Source:**
[tmutil-1.00.04.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.04.tar.gz)

**Release Date:** 2011-12-15

-   Overhauled dead time correction.
-   Fixed bug with showing marker IDs in tmplot.
-   Fixed boot time correction for LADEE/MAVEN to avoid problems caused
    by race condition with time sync.

### 1.00.05

**Source:**
[tmutil-1.00.05.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.05.tar.gz)

**Release Date:** 2012-01-06

-   Simplified options in tmfields.py.
-   Added limref
-   Added plotref
-   Added rf_subpacket_cal.py
-   Added subpacket_timing.py
-   Added on-line help for almost every script (i.e., `-h` option)

### 1.00.06

**Source:**
[tmutil-1.00.06.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.00.06.tar.gz)

**Release Date:** 2012-01-09

-   Added background subtraction functions.
-   Fixed infinite loop bug in tmread.util.step_pair.

### 1.00.07

**Source:** (missing)

**Release Date:** 2012-01-15

-   Added newtid.py
-   Added `godata` command for the shell.
-   Changed sorting behavior in tmdir to use date rather than TID.

### 1.01.00

**Source:**
[tmutil-1.01.00.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.01.00.tar.gz)

**Release Date:** 2012-03-13

-   Added support for MAVEN NGMS data.
-   Added lib699 package for functions not specific to telemetry.
-   Added checksum.py to default distribution.
-   Added massrange.py.
-   Added ontime.py.
-   Added t0.py.
-   Added tmstat.py.
-   Added tmsummary.py.

### 1.01.01

**Source:**
[tmutil-1.01.01.tar.gz](https://s3.amazonaws.com/699_bin/tmutil/tmutil-1.01.01.tar.gz)

**Release Date:** 2012-03-14

-   Fixed grievous errors that prevented installation.

### 1.02.00

**Source:**
[699util-1.02.00.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.02.00.tar.gz)

**Revision:** 66

**Release Date:** 2012-04-05

-   Bundled TMUtil and SamUtil into 699util.
-   Added support for SVN operations scripts.
-   Improvements to lib699.
-   Workaround to total_ordering bug in Python 2.7.1.

### 1.02.01

**Source:**
[699util-1.02.01.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.02.01.tar.gz)

**Revision:** 69

**Release Date:** 2012-05-10

-   Fixed bug with setting dac mode in triplets.py.

### 1.02.02

**Source:**
[699util-1.02.02.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.02.02.tar.gz)

**Revision:** 88

**Release Date:** 2012-05-14

-   Backend changes to Mission and QMS Science Packet classes.
-   Incorporation of regression testing-framework (not distributed).

### 1.03.00

**Source:**
[699util-1.03.00.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.03.00.tar.gz)

**Revision:** 118

**Release Date:** 2012-05-24

-   Incorporated support for time correction as a configurable file.
    Abandoned old concept of \"offset correction.\"
-   Added scripts for generating time configuration files from
    MSL-generated SCLK-SCET files.

### 1.03.01

**Source:**
[699util-1.03.01.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.03.01.tar.gz)

**Revision:** 150

**Release Date:** 2012-06-19

-   Added gcsciplot.py
-   Added override of Gnuplot.py\'s insistence on using \"aqua\" as the
    default terminal for Macs.
-   Backend restructuring of feiutil.
-   Added highlighting support to lib699.ttyutil and tmmsg.
-   Added preliminary support for SAM Testbed.

### 1.03.02 

**Source:**
[699util-1.03.02.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.03.02.tar.gz)

**Revision:** 167

**Release Date:** 2012-07-05

-   Now tolerates missing SCLK/SCET file (issues a warning).
-   Updates to tm2rdr.py and pds3 package.

### 1.04.00 

**Source:**
[699util-1.04.00.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.04.00.tar.gz)

**Revision:** 326

**Release Date:** 2012-10-06

-   Added lots of scripts, including lingering scripts that had not yet
    been ported to 699util and new SAM ops software
-   Added lib699.collections
-   Fixed svnutil performance bugs
-   A little bit of Python 3 support (not fully implemented)
-   tmenv moved into tmread, now uses .699config.INI file
-   Added XINA SVN-post-commit hooks

### 1.05.00

**[Warning! This version (1.05.00) has a bug causing incorrect outputs
for dead-time corrected QMS data. Do not draw conclusions from any data
extracted by this version of the software. The bug was fixed in
1.05.01.](Warning!_This_version_(1.05.00)_has_a_bug_causing_incorrect_outputs_for_dead-time_corrected_QMS_data._Do_not_draw_conclusions_from_any_data_extracted_by_this_version_of_the_software._The_bug_was_fixed_in_1.05.01. "wikilink")**

**Source:**
[699util-1.05.00.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.05.00.tar.gz)
(zip version:
[699util-1.05.00.zip](https://s3.amazonaws.com/699_bin/699util/699util-1.05.00.zip))

**Windows Installer:**
[699util-1.05.00.macosx-10.6-intel.exe](https://s3.amazonaws.com/699_bin/699util/699util-1.05.00.macosx-10.6-intel.exe)

**Revision:** 702

**Release Date:** 2013-04-17

-   Too many changes to list. Incorporated every change to software
    since last October.

### 1.05.01 

**Source:**
[699util-1.05.01.tar.gz](https://s3.amazonaws.com/699_bin/699util/699util-1.05.01.tar.gz)
(zip version:
[699util-1.05.01.zip](https://s3.amazonaws.com/699_bin/699util/699util-1.05.01.zip))

**Windows Installer:**
[699util-1.05.01.exe](https://s3.amazonaws.com/699_bin/699util/699util-1.05.01.exe)

**Revision:** 797

**Release Date:** 2013-05-31

-   Fixed a bug with dead-time correction that causes incorrect data to
    be printed.
-   Packet resequencing now based on packet \"creation time\" rather
    than packet timestamp.
-   Fixed bug in tmfields which caused program to crash when a telemetry
    file did not have a defined TID.
-   Many minor backend changes and bug fixes.

### 2.00.00

**DMG (Mac) Installation Package:**
<https://s3.amazonaws.com/699_bin/699util/699util-2.00.dmg>

**TAR Archive (Mac/Linux) Installation Package:**
<https://s3.amazonaws.com/699_bin/699util/699util-2.00-installer.tar.gz>

**Revision:** 1242

**Release Date:** 2013-09-05

-   Now supports Python 3 instead of Python 2
-   Improvements to packet parsing behavior
-   Representation of time as discrete units
-   Reorganization of scripts
-   Streamlining of gnuplot interface

### 2.00.01 

**DMG (Mac) Installation Package:**
<https://s3.amazonaws.com/699_bin/699util/699util-2.00.01.dmg>

**TAR Archive (Mac/Linux) Installation Package:**
<https://s3.amazonaws.com/699_bin/699util/699util-2.00.01-installer.tar.gz>

**Revision:** 1249

**Release Date:** 2013-09-12

-   Added tm2igor.pl and igormeta.py.
-   Fixed bug with ontime.py; now correctly tracks the last stretch for
    which a value is considered \"on\"
-   Added samvtool alias to shell configuration script.

### 2.00.02 

**DMG (Mac) Installation Package:**
<https://s3.amazonaws.com/699_bin/699util/699util-2.00.02.dmg>

**TAR Archive (Mac/Linux) Installation Package:**
<https://s3.amazonaws.com/699_bin/699util/699util-2.00.02-installer.tar.gz>

**Revision:** 1256

**Release Date:** 2013-09-13

-   Added smscuplog.py.
-   Full working version of nms2pds.py.
-   Added nms_assemble.py.

**Workaround** (From Mike Wong) If
installation fails with an error of
`AttributeError: \_convert` while building the enum34 package, try:
Delete the tarfile enum34-1.0.tar.gz that came in the 699util\*dmg disk image.
Replace it with a newer tarfile from a link such as [pypi.python.org/pypi/enum34/1.1.6](https://pypi.python.org/pypi/enum34/1.1.6).
Edit the file requirements.txt, replacing `"enum34==1.0"` with `"enum34>=1.1.6"`. Then try the install script again.