Pipeline Preliminaries

Operating system and Python dependencies

  • The whole pipeline was developed and runs in Linux.  I use Ubuntu 18.04 LTS.
  • Python 3 is assumed.  I use version 3.6.7
  • There are many dependencies, most of which are installed if you create the virtual environment as I describe below.  The following instructions assume you are using this environment.
  • This virtual environment does not install the astrometry.net engine (plate solving).  Installation of this as well as all index files appropriate to your telescope’s field of view and depth is detailed below.  I suggest you separately test this installation on your own images prior to letting PESTphot loose on it.  If you have installed it correctly the command solve-field will work.  Read the docs for further info.
  • You will also need to manually enter the path to the Montage (for making star field images) commands.  Instructions are below.
  • A live internet connection is needed for Gaia database interrogation.  The script does not check for this, but will fail.  If you are running without being on-line then turn off Gaia checking in config.

An easy way of making sure all the right dependencies (including numpy, pandas, matplotlib, xml, astropy, sklearn, tkinter, APLPy and montage-wrapper for Python) are installed is to create a virtual environment using Anaconda and the environment.yml file included in the zip download:
conda env create -f environment.yml

This will create a virtual environment called pestapps.


Installing astrometry.net and index files

Here are some more detailed instructions.  You could try to compile the source code from the astrometry.net site, but in Ubuntu at least, it has been packaged, so the following in a terminal should work:

sudo apt install astrometry.net

It will not give you many messages during installation, and just finishes silently.  After installation you will need to download and install the index files.  First find out where to put the index files by searching for the file astrometry.cfg (in my system this file was in usr/local/astrometry/etc/).

In this file will be something like this:

# In which directories should we search for indices?
add_path /usr/local/astrometry/data

You can download the index file manually, but this is a pain because there are 48 files in each series, and I use three series of files.  Instead I used a bash script with the following lines (change the application data directory as necessary):

cd /usr/local/astrometry/data
sudo wget http://broiler.astrometry.net/~dstn/4200/index-4204-{00..47}.fits
sudo wget http://data.astrometry.net/4200/index-4203-{00..47}.fits
sudo wget http://data.astrometry.net/4200/index-4202-{00..47}.fits

Just check before running that the data server is still correct.  It has changed in the past.  This set of indices was enough for my setup.  Confirm that it works for yours too by running the solve-field command with some of your images.

Installing Montage

Montage should have been installed during the virtual environment creation.  If for some reason you want to build it from scratch:

To make sure the system can find the Montage commands $PATH will need to include their directory.  Do the following after creating the virtual environment;

  • Locate the Montage commands. In the files window, search for mProject or mAdd
  • This will give the location of the executable file, e.g. /usr/bin
  • Type $PATH
  • If the Montage path is not on the list, add this line to ~/.profile: export PATH=”$PATH:/usr/bin”
  • You may have to re-login to get the $PATH properly set


Pipeline configuration

Key settings for the pipeline are in the file pest_pipeline_config.txt.  Hereafter I will refer to this as ‘config‘.

Once you have downloaded the files (below) the first thing to do is edit config.  It’s a text file so use any text editor to change the following parameters:

  • observatory
  • telescope
  • camera
  • scale
  • proj_dir
  • catalog_dir
  • magpy_param_path
  • signature

If you are a TESS observer, or wish to upload to AAVSO, also edit:

  • tess_user
  • aavso_obs_code

These options are covered in more detail in the Pipeline Config page.


 Directory structure and file naming conventions:

The pipeline is pretty rigid in expecting things to be in the right place, and being named right.

  • Working directory must have the path ‘anything/target_name/obs_date’.  target_name and obs_date are used in multiple places in the pipeline e.g. reports, charts and identification of the catalog file.   For example: ~/Documents/Astro_obs/Astro_active/KELT/KEBMMS18C001339_s/2017-11-03
  • The format of obs_date can be anything you like, but if you wish to comply with TESS SG1 standards, use YYYY-MM-DD.  This exact string is used in plots and reports, but the dashes (‘-’) are stripped out in filenames of those products meant to be uploaded to ExoFOP.
  • Stick with a fixed location for the C-Munipack catalog files.  Its path is set in config.
  • The same with C-Munipack project files.  Its path is set in config.
  • config must be placed in ~/bin.  The bash scripts that call Python can be placed anywhere on your $PATH.  Don’t forget to make them executable!
  • Python scripts can be placed anywhere convenient.  But edit ~/bin/pestphot, and ~/bin/magpy to point to them.
  • Other expected filenames are noted later in this document – follow them!


Pipeline files

File nameVersionFunctionTypeLocation
PESTphot.py2.4Reads C-Munipack photometry files, converts JD to BJD, de-trends potential ensemble stars. Checks around the target for variable stars. Plots outliers and LC for all stars close to the target. NEB clearance report. Plate solve and makes Finder images.Python scriptpython directory
MagPy.py5.4Reads instr mags, does diff photometry, ensemble optimisation, reporting and LC plotting.Python scriptpython directory
pestphot-calls PESTphot.pybash script~/bin or on $PATH
magpy-calls MagPy.pybash script~/bin or on $PATH
pest_pipeline_config.txt-Read by all parts of the pipeline for options and parameters such as 'simple mode', directories, observatory name, plate scale etc.text file~/bin
environment.yml-Used to set up a virtual environment with the required dependencies for the pipeline.conda environment file-

Download the gzip file using the link below, unzip and copy them to the specified location.

Issue date: 15 Sept 2019; PESTphot = 2.4; MagPy = 5.4; First issue of pipeline.

Next:  C-Munipack

Back to PEST pipeline page index.