# c699util Developer's Guide

*If you have not already done so, please read the [c699util User's Guide](https://wiki.xina.io/books/699/page/c699util-users-guide) to familiarize yourself with c699util and its capabilities.*

This article demonstrates how to add features to c699util and describes
parts of the c699util build, test, and release process. I found it
challenging to create c699util due to the Qt4, 699 C++ code, SWIG, and
qmake learning curves, and so I hope to ease that learning process for
others. I assume that readers are familiar with Qt4 and the 699 C++
code.

## SWIG

SWIG (Simplified Wrapper Interface Generator) is the program used to add
a Python interface (i.e., c699util) to the 699 C++ code. As of this
writing, c699util is created using SWIG 3.0.8.

Let\'s assume that we have code in files called example.cpp and
example.hpp, and that we want to use SWIG to make this code available in
Python. Our next step would be to create an *interface file* called
example.i, and then put the declarations of the code that we want to
expose to Python in that file; this is mostly just copying and pasting
the declarations in example.hpp. We could then run SWIG as follows:

`swig -Wall -c++ -python example.i`

and SWIG would generate two files: example_wrap.cxx and example.py.
example_wrap.cxx now needs to be compiled into a shared object library,
and example.py will look for that shared object library when it is
imported. This means that example.py is useless until example_wrap.cxx
is built.

We would then build example_wrap.cxx as a dynamic library and name the
resulting file \_example.so (SWIG expects this naming convention \-- an
underscore followed by the module name. Also, there\'s nothing wrong
with renaming a dynamic library file. These file types have no strictly
enforced file extension.).

At this point, we could open up our Python interpreter and type \"import
example\". example.py would be imported, and it would then load
\_example.so. example.\*pp now has a Python interface.

A key takeaway from this description is that \"running SWIG\"
successfully meant generating two files \-- example.py and \_example.so.
Together, these two files are our new Python module.

Of course, the devil is in the details, and it\'s easy to make mistakes
in this process. So, developers who modify c699util (or create something
similar to it) are encouraged to start by making sure that they can
follow [this tutorial](http://www.expobrain.net/2011/01/23/swig-tutorial-for-mac-os-x/)
(I had to get rid of the -arch i386 flag when I went through the
article), which can be thought of as the \"Hello, World!\" of SWIG. As
with any complicated software development, a smart approach is to start
with something simple that works well (such as the example in this
article) and then to make small changes/enhancements to it, ensuring
that the small changes work as expected.

I found the SWIG documentation and mailing list to be very helpful,
particularly:
 
-   [SWIG Documentation Prefix](http://www.expobrain.net/2011/01/23/swig-tutorial-for-mac-os-x/)
	-   [SWIG Documentation Introduction](http://www.swig.org/Doc3.0/Introduction.html#Introduction)
		-   [SWIG Basics](http://www.swig.org/Doc3.0/SWIG.html#SWIG)
-   [SWIG and C++](http://www.swig.org/Doc3.0/SWIGPlus.html#SWIGPlus)
-   [SWIG and Python](http://swig.org/Doc3.0/Python.html)

While those last two links lead to very long articles, I found it
worthwhile to grit my teeth and read them in their entireties. If you
find yourself maintaining c699util, you should probably do the same.
However, if you\'re just going to make a small change to c699util, you
can probably get away with just \"cookbooking\" off of the simpler
examples in the first two links as well as existing c699util code. (If
you do so, be sure to update test_c699util.py! This will be described
below.)

Additionally, my personal notes on SWIG and c699util have been shared
with the MOMA software team. Contact a team member if you need access.

## How c699util and the 699 C++ code work together 

c699util is defined in three files: c699util.cpp, c699util.hpp, and
c699util.i. c699util.cpp and c699util.hpp (occasionally referred to as
c699util.\*pp) can be thought of as the \"bridge\" between the 699 C++
code and the Python scripts that want to use the features defined in the
699 C++ code. As such, c699util.\*pp are responsible for hiding 699 C++
implementation details from those Python scripts. c699util.i contains
the routines from c699util.\*pp that we want exposed to Python; again,
this is mainly done by copying declarations from c699util.hpp into
c699util.i.

The general approach taken by c699util.\*pp is to define a class that
Python scripts expect, such as TMFile, and within that class, use 699
C++ objects to implement useful methods. For example:

```c++
class TMFile 
{
public: 
    // todo - I'm forgetting to use const in these methods 
    TMFile(std::string filename); 
    std::string filename(); 
    std::string absolute_directory(); 
    std::string directory_string();
    std::string get_mcal_filename(); 
    long long int file_size(); 
    int tid(); 
    int length();     
    bool exists(); 
    TMPacket get_pkt(int index);

    double t0_in_unix_time() const; 
     
private: 
    QString filename_;
    TmMeta tmMeta_;
    MomGse::MomTelemetryFilePtr momaTlmFilePtr_; 
    MomGse::MarkerCache markerCache_;
    MomGse::MomaTimestampResolver& timestampResolver_; 
};
```

As you might expect, TMFile generates packet objects via
momaTlmFilePtr\_, and determines their marker and timestamp properties
by using markerCache\_ and timestampResolver\_. TMPacket,
ScienceDataCacheWrapper, and PythonStyleMomaScan follow the same design
pattern.

## How SWIG and qmake work together 

I don\'t want to describe our entire build system, but there are a few
c699util.pro details worth understanding:

```
macx {
    INCLUDEPATH += /Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m 
    LIBS +=        /Library/Frameworks/Python.framework/Versions/3.4/lib/libpython3.4.dylib
    INCLUDEPATH += /Library/Frameworks/QtCore.framework/Versions/4/Headers
}
linux-g++ {
    # When Python3 is installed on wherever, it should be installed with
    # "./configure --enable-shared". This is necessary so that the library files
    # generated contain position independent (-fPIC) code. 
    INCLUDEPATH += /usr/local/include/python3.4m
    LIBS += /usr/local/lib/python3.4/config-3.4m/libpython3.4m.a
}
```

This section is the most likely to cause issues. Python can be installed
in a few different ways, and in some of those ways these locations may
not be valid. However, if developers [install Python 3.4.3 from the
Python website](https://www.python.org/downloads/release/python-343/) ,
then there shouldn\'t be any problems. Don\'t worry about the linux-g++
section for now. It\'s only for the CentOS virtual machine which is used
to build c699util for mine699.

`system(swig -Wall -c++ -python c699util.i)`

Running qmake on the c699util.pro file invokes SWIG.

```
macx {
    QMAKE_LFLAGS_PLUGIN -= -dynamiclib
    QMAKE_LFLAGS_PLUGIN += -bundle
}
linux-g++ {
    QMAKE_LFLAGS_PLUGIN += -fPIC
}
```

When linking on OSX, replace the -dynamiclib flag with the -bundle flag,
ensuring that c699util is in the [Mach-O file
format](https://en.wikipedia.org/wiki/Mach-O). I don\'t remember why
this is necessary. When linking on the CentOS VM, produce
[position-independent code](http://stackoverflow.com/a/5311538/5373846).

```
# Bundles have no strictly enforced file extension, but swig expects the 
# libraries it uses to be of the form _<module name>{=html}.so , so rename the output
# file. (Don't forget the leading underscore. It's a common pitfall.)
macx {
    QMAKE_POST_LINK = "mv libc699util.dylib _c699util.so"
}
linux-g++ {
    QMAKE_POST_LINK = "mv libc699util.so _c699util.so"
}
```

That comment should sufficiently explain this part of c699util.pro.

## How to add a feature to c699util 

### Getting c699util Source Code 

c699util is located in
<svn://repos699.gsfc.nasa.gov/cppcode/qt4/trunk/c699util>.

### Setting up test_c699util.py 

c699util/test_c699util.py is a fully automated test of c699util. I\'ve
done my best to ensure that every c699util feature is thoroughly tested
inside of test_c699util.py.

So, if you find yourself modifying c699util, ***ensure that
test_c699util.py remains a thorough test of c699util by adding good
tests to it!*** This test suite runs on both OSX and CentOS and has
saved us a lot of trouble by finding subtle bugs before our users could
be affected by them. It should be run after every new feature is added
to ensure that nothing else has broken (it runs in less than 60
seconds). Now, to set up your environment to run test_c699util.py, do
the following:

If you have not done so, [follow the instructions here to get MOMA Data
View and check out MOMA
data](http://699wiki.com/wiki/index.php/MOMA_Data_View).

Now, if you have not already done so, [tunnel into
MOMAIOC](http://699wiki.com/wiki/index.php/MOMA_Data_View#Tunneling_into_momaioc)
and then open up a new tab. In this new tab, type

`cd ~`

`svn co svn://localhost:6994/momadata/test_data ~/momadata/test_data`

Then, open up any TID inside of test_data using MOMA Data View. This
should add a MOMA_TEST_DATA group to your \~/.699config.INI file. Once
you\'ve done this, close MOMA Data View and open up \~/.699config.INI in
a text editor. Modify this line:

```
[MOMA_TEST_DATA]
<other lines>
tm_database=/Users/mpallone/momagse/TMDef/MOM_TM_Database.txt
```
by putting "Test_\" in front of MOM_TM_Database.txt. It should now
look like:

```
[MOMA_TEST_DATA]
<other lines>
tm_database=/Users/mpallone/momagse/TMDef/Test_MOM_TM_Database.txt
```

This file is a copy of MOM_TM_Database.txt which has been frozen just
for the sake of test_c699util.py. This way, future conversion changes to
the real database won\'t break test_c699util.py.

Next, ensure that
[tmread](http://699wiki.com/wiki/index.php/Python_Tool_Installation) is
installed on your system.

At this point, you should have the necessary test_c699util.py
dependencies. Building and running test_c699util.py is described in the
next section.

### Building c699util 

Whenever I release a new version of c699util, I run the following
commands:

```
make clean
qmake -spec macx-g++ c699util.pro # For CentOS, I can just run "qmake"
make release
cat top_of_c699util_dot_py.py c699util.py > temp_c699util.py && mv temp_c699util.py c699util.py
./test_c699util.py
./copy_into_699util.py # Only do this if test_c699util.py passes all tests! 
```


(Actually, these commands are inside of the files osx_release and
centos_release, and I just run one of those scripts depending on the OS
I\'m using.)

copy_into_699util.py is a special script which writes c699util metadata
to a file called VersionInfo.txt, and then moves \_c699util.so,
c699util.py, and VersionInfo into their \"release homes\" in the 699util
repository. (How c699util fits into the 699util release scheme is
described later in this article; do not concern yourself with such
details at this moment.) VersionInfo.txt is important because it tells
software engineers who are debugging c699util (a) what version of
c699util is running on a user\'s computer, and (b) if any uncommitted
changes in the working copy made it into the user\'s version of
c699util. Please only release c699util by using copy_into_699util.py.

### Example Feature 

Let\'s pretend that we want to add a description() method to c699util\'s
TMPacket class. While we\'re at it, we\'d like to make it so that
calling str(pkt_object) in a Python script calls this description()
method.

In c699util.hpp, we\'d add the following:

```c++
/*****************************************************************************
 * TMPacket definition 
 *****************************************************************************/
class TMPacket 
{
public:
    TMPacket(GseLib::pbytes packetBytesPtr, int pktIndex,
             const MomGse::MarkerCache* markerCachePtr,
             const MomGse::MomaTimestampResolver& timestampResolver); 

    std::string description();  // this method was just added
```

Next, in c699util.cpp, we would define the description() method:

```
/*****************************************************************************
 * TMPacket Implementation 
 *****************************************************************************/

TMPacket::TMPacket(GseLib::pbytes packetBytesPtr, int pktIndex,
                   const MomGse::MarkerCache* markerCachePtr,
                   const MomGse::MomaTimestampResolver& timestampResolver) 
    : packetBytesPtr_(packetBytesPtr),
      pktIndex_(pktIndex),
      markerData_(markerCachePtr->getMarker(pktIndex, &hasMarkerData_)),
      unixTimestamp_(timestampResolver.getPktUnixTime(pktIndex)),
      relativeTimestamp_(timestampResolver.getPktRelativeTime(pktIndex))
{
    // no further initialization needed 
}    

std::string description()
{
    return std::string("this is a description of a packet); 
}
```

Then, in c699util.i, we would copy this declaration:

```c++
/*****************************************************************************
 * TMPacket definition 
 *****************************************************************************/
class TMPacket 
{
public:
    TMPacket(GseLib::pbytes packetBytesPtr, int pktIndex,
             const MomGse::MarkerCache* markerCachePtr,
             const MomGse::MomaTimestampResolver& timestampResolver); 

    std::string description(); // this method was just added
    %pythoncode %{
        def __str__(self): 
            return self.description()
    %}
```

Notice that the change we made to c699util.i is exactly the same as the
change we just made to c699util.hpp, except for the `%pythoncode` stuff.
(If we didn\'t want to add a \_\_str\_\_() method to TMPacket, then
c699util.hpp and c699util.i would change in exactly the same way.)

`%pythoncode` is a SWIG feature that allows us to define Python code that
appears in the Python file that the \'swig\' command outputs (which, in
our case, is c699util.py). The \_\_str\_\_() method defined above will
now be a method of the TMPacket class, just as description() is a method
of the TMPacket class. Once c699util is built, calling str(pkt_object)
will invoke the \_\_str\_\_() method defined above, which will then call
description(). %pythoncode is an extremely useful SWIG feature because
it (a) allows us to use magical Python methods (such as generators and
iterators) that are difficult or unrealistic to implement in C++, (b)
allows us to use concise, powerful Python syntax when defining our new
classes rather than writing needlessly verbose C++ code, and (c) allows
us to import and use Python modules at runtime \-- even tmread modules!
For example, consider the following uses of %pythoncode in the TMFile
class:

```c++
class TMFile 
    
    <other TMFile methods go here>{=html} 

    %pythoncode %{
        def test_name(self): 
            return self.directory_string().split('-', maxsplit=5)[5]
    %}

    %pythoncode %{
        def message_log(self): 
            from tmread import MessageLog
            return MessageLog(self)
    %}
```

Defining `test_name()` using `%pythoncode` lets take advantage of Python\'s
convenient string manipulation features. Defining `message_log()` using
`%pythoncode` makes it possible to use tmread\'s well-tested and powerful
MessageLog class, because we can import and use it at runtime.

### Release Process 

This process should be followed every time c699util is released. It
requires a previously mentioned CentOS virtual machine, which exists
solely to build and test c699util for mine699. Passwords for this VM
have been distributed to the MOMA team. Copies of the VM exist on Mark
Pallone\'s external hard drive and on the MOMA Data iMac located (as of
this writing) in GSFC Building 33 F109A, ECN #2377998, inside of
/Users/lab/Documents/c699util-virtual-machines.

#### c699util Release Process 

1.  **Update test_c699util.py so that it thoroughly tests your new
    feature.**
2.  Commit the code being released.
3.  On your Mac, run ./osx_release. This will:
    1.  do a clean build of c699util
    2.  run test_c699util.py
    3.  copy the relevant files (including working copy differences)
        into \~/labcode/699util/c699util, but only if the test
        succeeded.
4.  cd into \~/labcode/699util/c699util/osx, and 'svn commit' the newly
    built files
5.  On the CentOS build VM, "svn up" the momagse and qt4 directories.
6.  Run ./centos_release. This will:
    1.  do a clean build of c699util
    2.  run test_c699util.py
    3.  copy the relevant files (including working copy differences)
        into \~/labcode/699util/c699util, but only if the test
        succeeded.
7.  cd into \~/labcode/699util/c699util/centos, and \'svn commit\' the
    newly built files
8.  Log into mine699. On your own account (e.g., \'mpallone\' instead of
    the moma user):
    1.  svn update the Python tools
    2.  svn update \~/labcode/qt4
    3.  cd into \~/labcode/qt4/c699util, and run test_c699util.py
    4.  If this works, switch to the \'moma\' user and update the Python
        tools
9.  Build a new Python tools package, being sure to test that the
    package works on our OSX VMs.
10. Add a new entry to the [release page](http://699wiki.com/wiki/index.php/MOMA_Python_Packages).
11. Commit the new DMG to `momagse/Apps/py699`.
12. Document any new c699util features in the [c699util user\'s guide](http://699wiki.com/wiki/index.php/C699util_User%27s_Guide).
13. If necessary, update and release `extract_hk_data_main` on mine699
    as well. (Strictly speaking this has nothing to do with c699util,
    but we shouldn\'t forget about this script when we release updates
    to c699util.)

It\'s annoying to run the test in three places, but sometimes bugs exist
in one environment but not others.

## c699util dependencies for both OS X and CentOS 

It\'s useful to be aware of c699util dependencies when debugging. On
OSX, \_c699util.so contains the following dependencies:

```
(py699) gs699-mpallone:c699util mpallone$ otool -L _c699util.so
_c699util.so:
   /Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0)
   QtGui.framework/Versions/4/QtGui (compatibility version 4.8.0, current version 4.8.4)
   QtCore.framework/Versions/4/QtCore (compatibility version 4.8.0, current version 4.8.4)
   QtNetwork.framework/Versions/4/QtNetwork (compatibility version 4.8.0, current version 4.8.4)
   /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 104.1.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
   /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 915.0.0)
```

The expected location of these dependencies can be tweaked by using the
install_name_tool program. Python 3.4 should be in the specified
location if it is installed through the DMG file available on the Python
website. The use of Homebrew in this particular case is discouraged. The
Qt dependencies should be discoverable once Qt4 is installed. The
remaining dependencies were already available on my Mac; I suspect that
they\'re standard on all modern Macs.

Unlike the OSX distribution of c699util, the CentOS distribution has the
Python and Qt dependencies statically compiled into the \_c699util.so
file. The .so file\'s remaining dependencies are all standard (i.e.,
they\'re available by default on CentOS machines such as
mine699.gsfc.nasa.gov).

## c699util, the 699 developer setup, and the 699util package release scheme 

Releasing c699util means committing new versions of

-   c699util.py
-   \_c699util.so
-   VersionInfo.txt

into the repos699 SVN server, in the locations
`labcode/699util/c699util/osx` and `labcode/699util/c699util/centos`.
The rest of this section describes how c699util fits into the the
699util developer\'s setup and the 699util package release system.

### Review of the 699util Environment 

Even for those who are familiar with [the 699util installation
article](http://699wiki.com/wiki/index.php/Python_Tool_Installation),
how all the different parts of 699util fit together can still be hard to
understand. So, reviewing this should make the rest of this section much
easier to understand.

The 699util environment has four major parts:

1.  699 Data
2.  699 tool configuration/settings
3.  699 Code
4.  Local environment configuration

**1. 699 Data.** Setting up 699 data simply means checking out folders
such as `momadata/etu`, `momadata/fm`, etc. from
[momaioc.gsfc.nasa.gov\'s SVN
repository](http://699wiki.com/wiki/index.php/MOMA_Data_View#Getting_the_data).
Typically, momadata is checked out into the home directory, but this is
not required.

**2. 699 tool configuration/settings.** Items in this category mainly
tell 699 software where data is, and how to interpret that data. This
category includes:

-   [checking out
    momagse](http://699wiki.com/wiki/index.php/MOMA_Data_View#Downloading_momagse)
-   using MOMA Data View to initialize the \~/.699config.INI file.
    Simply open up a TID for each model that you have checked out in
    your `momadata` working copy.
-   (If you\'re doing a developer install, you will also have to
    [checkout
    config699](http://699wiki.com/wiki/index.php/Python_Tool_Installation#Installing_config699).)

**3. 699 code.** Developers can [check out 699util from
SVN](http://699wiki.com/wiki/index.php/Python_Tool_Installation#Installation_for_Developers).
Users can find MOMA script DMGs
[here.](http://699wiki.com/wiki/index.php/MOMA_Python_Packages)

**4. Local environment configuration.** This tends to be the most
opaque. It consists of the following:

-   [Installing Python 3.4.3 from the Python
    website](https://www.python.org/ftp/python/3.4.3/python-3.4.3-macosx10.6.pkg),
    and [updating your .bashrc, .bash_profile, or .profile
    file](http://699wiki.com/wiki/index.php/Python_Tool_Installation#Macintosh_Installation)
    so that the newly installed version of Python can be found via the
    PATH environment variable.
-   [setting up a virtual environment using the program
    virtualenv](http://699wiki.com/wiki/index.php/Python_Tool_Installation#Installing_699util).
    A virtual environment can be thought of as a folder containing
    libraries and executable programs, as well as the environment
    variables that reference those libraries and executables. In the
    699util world, virtualenv is used to create either `~/py699` or
    `~/py699-moma`. Python 3.4.3 is copied into this directory and, in
    the case of the package install, 699util scripts are also copied.
    Typically, the last line of a user\'s `~/.bashrc` file \"activates\"
    the virtual environment, which essentially just puts the virtual
    environment\'s binary folder (e.g., `~/py699/bin`) at the front of
    the PATH variable. Then, when a scripts starts with
    `#!/usr/bin/env python3`, the first instance of the python3
    executable found by the `/usr/bin/env` program will be the one
    inside of the `~/py699` directory.
-   The location of the 699util scripts, such as tmplot.py. In the
    package install meant for users (i.e., non-developers), these
    scripts are simply copied into the virtual environment folder
    `~/py699/bin`. For developers who already have these scripts checked
    out into a working copy, the location of those scripts is added to
    the PATH variable as follows:
    -   the developer opens up a terminal
    -   the terminal sources the developer\'s `~/.bashrc` file
    -   the `~/.bashrc` file sources the `~/config699/bash_config.sh`
        file, which adds the script directories to the PATH variable.
-   The location of 699util libraries. This is very similar to how
    scripts are handled. For non-developers, the 699util Python library
    is in the virtual environment\'s library folder. For developers, the
    `~/config699/bash_config.sh` directory adds the libraries to the
    PYTHONPATH environment variable, which Python uses when looking for
    modules that need to be imported.

### Developer Setup 

As mentioned above, the developer install sources
`~/config699/bash_config.sh`, which sets PATH and PYTHONPATH environment
variables. `bash_config.sh` determines the version of c699util that\'s
correct for the user\'s operating system, and adds that directory to the
PYTHONPATH variable. More explicitly, if c699util exists in this
location:

`~/labcode/699util/c699util/osx/c699util.py`

`~/labcode/699util/c699util/osx/_c699util.so `

then the directory `${HOME}/labcode/699util/c699util/osx` will be added
to the user\'s PYTHONPATH, so that the `import c699util` code inside of
a Python script can know to look in this directory.

As mentioned in a previous section, `_c699util.so` has some Qt4
dependencies. When \`import c699util\` is executed, some code at the top
of c699util.py will determine if \_c699util.so\'s dependencies are
unresolved, and if so, change the dependencies to point to the correct
location on the user\'s computer.

### 699util Package Release 

Non-developers (i.e., users) install 699util by mounting a 699util .dmg
file and running the `install699util.sh` script. The package release
scheme does not use the aforementioned `~/config699/bash_config.sh`
script. The .dmg file contains the following in its root directory:

-   \_c699util.so
-   699util.py
-   Qt4 dependencies

`install699util.sh` simply copies these files into the virtual
environment folder `~/py699/lib`, and then tweaks the user\'s .bashrc
file so that `~/py699/lib` is always in the users\' PYTHONPATH whenever
a new terminal is opened.