XINA Data Mining

Overview

Telemetry archives (also called "TID's") are imported into XINA by the process described here. "TID" is short-hand for "Telemetry Identifier", a text string that uniquely describes the experiment contained in the telemetry archive.

The SVN repository is typically laid out using the following heirarchy:

where {model} matches the XINA database name and {tid} is a unique description of the telemetry archive.

The XINA Data Mining process uses a shared network drive from the mining server (mine699.gsfc.nasa.gov) to the IOC server (e.g., MOMAIOC/DRAMSIOC/etc). Each mission uses a mission-specific folder on this shared drive to perform the data mining processes described below (e.g., /mine699/missions/moma/).

This flow chart shows the major steps taken during the data mining process:

Importing files to XINA

  1. The user commits a TID to Subversion (SVN).
  2. On MOMAIOC/DRAMSIOC/etc):
  1. The XINA Commit Watch utility running on MINE699 detects new rev file. Each mission has a separate running instance of this utility.
  1. The XINA Commit utility is run by XINA Commit Watch (above)
  1. The XINA Import utility is run by XINA Commit Watch (above)

Handling Import Errors

XINA Commit Watch will not move on to the next rev file until the current file has been completed successfully

Debugging

Log output from XINA Commit Watch, XINA Commit, and XINA Import is stored in ```/mine699/mission/*/log```. - Errors generated during the "commit phase" are generally easier to fix, and my indicate a problem with a configuration file or script. After a commit phase bug is fixed, the rev.lock file can simply be deleted. XINA Commit Watch will perform a clean attempt to re-process the entire revision. - Errors generated during the "import phase" can be more complicated to fix because some operations may have already been imported before the failure, and running them again may cause new errors. It is necessary to check the archive directory for completed operations before attempting to re-process the revision. It may be preferable to perform corrections on the mined files depending on the error (for example, a typo in a field name), import the files manually, transfer the rev file to the rev archive manually, and then remove the rev.lock file.

Preload

For mining previously committed data, each mission has directory ```/mine699/mission/*/preload```

Revision #11
Created 9 May 2024 16:47:44 by Michael Burkhart
Updated 13 May 2024 13:47:32 by Michael Burkhart