NAV file format - backplanes, metadata keywords

[giardiaspace]

Topic: Format of navigated FITS image files. I am new to Pylanetary. I assume the navigation step is like what I do in my IDL Simnav code: add metadata and backplanes. There may also be some manipulation of the image data itself, such as rotating north up or resampling to a finer equivalent plate scale.

Motivation: I have tons of processed image data and I would love for the files to be compatible with Pylanetary. I also think my format has some nice features that help the usability of the data. I would be happy if the Pylanetary format had a superset of my IDL Simnav keywords, and I would still be happy if the keywords sometimes had different names. This is not just for HST/WFC3 data obviously. As long as there was some kind of keyword like PYLANVER that gave the version of the Pylanetary format, I could include translator code to handle any differences (until I migrate 100% to Pylanetary).

Backplanes: I copied HST image data. Different instruments and levels of data processing/products from the HST pipeline have different FITS structures, but the one I adapted for NAV files resembles HST/WFC3 "FLT" files: it has extension 0 (primary HDU) with no data and only metadata, followed by as many extensions as needed, with minimal metadata in those subsequent extensions. For example, a navigated Uranus image looks like this:

> fitsinfo 221110_845_1959_nav.fits                                                                           

No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     447   ()      
  1  SCI           1 ImageHDU        11   (530, 527)   float32   
  2  LAT           1 ImageHDU        12   (530, 527)   float32   
  3  LON           1 ImageHDU        12   (530, 527)   float32   
  4  EMI           1 ImageHDU        12   (530, 527)   float32   
  5  INC           1 ImageHDU        12   (530, 527)   float32   

Metadata: My principle has been to keep all the existing telescope/pipeline metadata, and add some keywords related to navigation. These keywords then can help with further science analysis, reprojection of the image data (cylindrical or polar maps), calibration to I/F reflectance units, etc.

Here are some of the most important keywords in my opion, but this is a draft list and hopefully there is a chance later on to refine the list. These are only Simnav-added keywords. Many telescope keywords (e.g., DATE-OBS and TIME-OBS) are equally important but assumed to not be Pylanetary's responsibility:

              / JPL EPHEMERIS DATA

TRG_ROT =        267.544900000 / Target N.Pole angle (degrees E of N)
TRG_RA  =        44.1721242500 / Target RA (degrees, J2000)
TRG_DEC =        16.4194880833 / Target DEC (degrees, J2000)
TRG_R_A =        1.88576990417 / Target equatorial radius (arcsec)
TRG_R_B =        1.84253420778 / Target polar radius (arcsec)
TRG_LON =        1.69132734167 / Sub-observer longitude (degrees, SysIII)
TRG_LAT =        59.3026484417 / Sub-observer latitude (degrees, planetographic)
SUN_LON =        1.76138321667 / Sub-solar longitude (degrees, SysIII)
SUN_LAT =        59.3716203417 / Sub-solar latitude (degrees, planetographic)
TRG_PHAS=      0.0739191666667 / Sun-object-observer angle (degrees)
TRG_R   =        19.6775273874 / Sun-Target distance (AU)
TRG_D   =        18.6876591479 / Observer-target distance (AU)

These always come in handy, and it's nice to be able to make tables of various things (usually longitude) without having to keep querying Horizons.

              / HUMAN OPERATOR LAT-LON FITTING

LLMETHOD= 'LIMB (simnav_fast 1.045)' / Lat-lon grid fitting method (LIMB/FEATURE
SIG_XPOS=     0.00142966361451 / Uncertainty in lat-lon grid offset (arcsec)
SIG_YPOS=     0.00142966361451 / Uncertainty in lat-lon grid offset (arcsec)
SIG_LAT =      0.0434378016642 / Equivalent lat. uncert. at sub-earth (degrees)
SIG_LON =      0.0434378016642 / Equivalent lon. uncert. at sub-earth (degrees)
XSHIFT  =     0.00699885162728 / Image center to ephemeris disk center (arcsec)
YSHIFT  =     0.00515903111723 / Image center to ephemeris disk center (arcsec)
NAV_X0  =        265.206597824 / X-pixel coord of sub-observer point (0-indexed)
NAV_Y0  =        263.520851185 / Y-pixel coord of sub-observer point (0-indexed)
LON_SYS = 'III     '           / IAU Longitude system used
LAT_TYPE= 'Planetographic'     / Latitude system for map
LAT_SHAP= 'Oblate spheroid'    / Shape model used
R_A_KM  =        25559.0000000 / Target equatorial radius (km)
R_B_KM  =        24973.0000000 / Target polar radius (km)
CROP_X0 =                  774 / pix trimmed from input, left (post-mag., 0-inde
CROP_Y0 =                  759 / pix trimmed from input, bottom (post-mag., 0-in

These are the main navigation keywords. Most of the first ones are related to the planet center relative to the image, or relative to the RA/DEC it's supposed to be at (which can also be found in telescope header keywords). The section mentions "human operator" because in my experience, automated methods are good but end up being biased by artifacts in individual frames. In Simnav, the human operator sets the final planet center, but the uncertainty estimate in the navigation is the rms offset between the human operator solution and the solutions from two automated algorithms.

In some cases, we made NAV files that are cropped with respect to the original image files, to remove blank space around the planet. This is fine, but it can cause issues trying to reconstruct what happened if you only have the final image. So the CROP_X0 and CROP_Y0 keywords are used to relate properties in the final image to properties in the input image to Simnav. With these keywords (not present in earlier versions of Simnav), recalibrated data can be renavigated without additional human interaction.

              / FRAME SCALE AND LIMIT DATA

MISSVAL =                  NAN / Missing data flag value
PPA     =        100.959109012 / Pixels per arcsec
PIXSCAL =           0.00990500 / Arcsec per pixel
NAVMAG  =                    4 / Magnification with respect to original data

I like NaN for a bad data flag, but this keyword always annoys FITS format checkers. So maybe the value should be 'NaN' (ascii). In any case, NaN lets you take averages and such only over good data using Numpy or IDL. Another bad data flag could be used, and listed here, and then codes could use that instead of NaN to do smarter averaging.

The other keywords are for magnification. I find it useful for some smaller targets to enlarge the image. These keywords track that process and ensure that the plate scale in the final product is known to the user (in case they try to reconstruct it from the telescope pipeline metadata and find inconsistencies).

BUNIT   = 'ELECTRONS/S'        / Units of science product
PHOTFLAM=        4.6016378E-19 / Inverse sensitivity, ergs/cm2/A/e-
PHOTIF  =    0.000148261705616 / Convert counts/sec to I/F
SIG_PHOT=    7.56008235524E-06 / Uncertainty estimate for PHOTIF

These photometry keywords are not well organized in the header. They apply to the SCI (science data) extension, which for drizzled WFC3 data has units of e-/sec. the PHOTFLAM is from the STScI pipeline, and multiplying this factor by the image array data in e-/sec converts the image to FLAM units of erg/s/cm^2/Angstrom.

For planetary work in the UV/vis/near-IR regime, we want reflectivity in I/F units, which we can get by PHOTIF*SCI. Similarly the SIG_PHOT gives the uncertainty, which includes uncertainty in the solar spectrum.

I made a Google Drive folder PYLAN_FITS_format with example Uranus NAV and REG format FITS files. REG is a cylindrical projection map format, but we can discuss that in a separate issue. Also included is a Python script from Troy that converts NAV to REG, although it is not fully debugged, so the REG file here was generated in IDL.

[emolter]

Thanks so much, Mike! Let's talk about this at our meeting next week. I was planning to do the majority of this work myself, and I already have the bare-bones implemented. This detailed description of the format gives me a ton of help understanding what else is needed, not only with writing the code but also for documentation.

I know you're busy, but if you'd like to contribute something that isn't Python, you might consider taking the lead on the documentation of this file format as the "industry standard" for solar system data with a backplane.