Example pipeline execution¶

Here is an example of what it takes to configure and run a pipeline with flat-fielding and background subtraction,

Setup CRDS¶

Make sure you have a local checkout of the CRDS cache as explained in the Getting started page. Run the setup_local_crds.sh to setup the enviroment variables needed to point the crds software to the CRDS cache. Optionally source this in your shell configuration to automatically set this up.

Get input simulations¶

Download simulated input FITS files for the IRIS imager from Figshare. It contains a raw science frame, a raw flat frame and a raw background frame.

Preprocess the flat frame¶

First we need to remove the dark frame from the flat frame and normalize it. A dark frame is already available in the CRDS and the pipeline knows how to retrieve it based on the metadata in the FITS file headers.

We can check in the package documentation what are the available pipelines and check the configuration options of the pipeline.PreprocessFlatfield class.

We do not need to customize it so we can directly call it from tmtrun and pass the input FITS file:

tmtrun iris_pipeline.pipeline.PreprocessFlatfield raw_flat_frame_cal.fits

This will pickup the relevant dark frame from the CRDS and process the file:

2019-10-04 17:59:48,057 - stpipe.PreprocessFlatfield - INFO - PreprocessFlatfield instance created.
2019-10-04 17:59:48,059 - stpipe.PreprocessFlatfield.dark_current - INFO - DarkCurrentStep instance created.
2019-10-04 17:59:48,060 - stpipe.PreprocessFlatfield.normalize - INFO - NormalizeStep instance created.
2019-10-04 17:59:48,099 - stpipe.PreprocessFlatfield - INFO - Step PreprocessFlatfield running with args ('raw_flat_frame_cal.fits',).
2019-10-04 17:59:48,554 - stpipe.PreprocessFlatfield - INFO - Prefetching reference files for dataset: 'raw_flat_frame_cal.fits' reftypes = ['dark']
2019-10-04 17:59:49,306 - stpipe.PreprocessFlatfield - INFO - Prefetch for DARK reference file is '/home/azonca/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits'.
2019-10-04 17:59:49,307 - stpipe.PreprocessFlatfield - INFO - Starting preprocess flatfield ...
2019-10-04 17:59:53,490 - stpipe.PreprocessFlatfield - INFO - Processing product raw_flat_frame
2019-10-04 17:59:53,490 - stpipe.PreprocessFlatfield - INFO - Working on input raw_flat_frame_cal.fits ...
2019-10-04 17:59:53,641 - stpipe.PreprocessFlatfield.dark_current - INFO - Step dark_current running with args (<IRISImageModel(4096, 4096) from raw_flat_frame_cal.fits>,).
2019-10-04 17:59:53,658 - stpipe.PreprocessFlatfield.dark_current - INFO - Using DARK reference file /home/azonca/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits
2019-10-04 17:59:54,058 - stpipe.PreprocessFlatfield.dark_current - INFO - Step dark_current done
2019-10-04 17:59:54,101 - stpipe.PreprocessFlatfield.normalize - INFO - Step normalize running with args (<IRISImageModel(4096, 4096) from raw_flat_frame_cal.fits>,).
2019-10-04 17:59:54,472 - stpipe.PreprocessFlatfield.normalize - INFO - Step normalize done
2019-10-04 17:59:54,472 - stpipe.PreprocessFlatfield - INFO - Finished processing product raw_flat_frame
2019-10-04 17:59:54,473 - stpipe.PreprocessFlatfield - INFO - ... ending preprocess flatfield
2019-10-04 17:59:54,811 - stpipe.PreprocessFlatfield - INFO - Saved model in raw_flat_frame_flat.fits
2019-10-04 17:59:54,811 - stpipe.PreprocessFlatfield - INFO - Step PreprocessFlatfield done

We have an output file raw_flat_frame_flat.fits and we can rename it:

mv raw_flat_frame_flat.fits flat.fits

Configure the image processing pipeline¶

The Image2Pipeline can be configured using a INI-style configuration file, image2_iris.cfg:

name = "Image2Pipeline"
class = "iris_pipeline.pipeline.Image2Pipeline"
save_results = True
    [steps]
      [[bkg_subtract]]
      [[assign_wcs]]
        skip = True
      [[flat_field]]
        config_file = flat_field.cfg
      [[photom]]
        skip = True
      [[resample]]
        skip = True

first we specify that we want to execute the pipeline defined in iris_pipeline.pipeline.Image2Pipeline, then we can configure each of the steps, for example skip some of them. Also we can import the configuration of a step from another file, in this case flat_field.cfg:

name = "flat_field"
class = "jwst.flatfield.FlatFieldStep"
# Optional filename suffix for output flats (only for MOS data).
flat_suffix = None
override_flat = 'flat.fits'

If we do not define override_flat, the pipeline will look up a suitable flat from the CRDS, in this case instead we specify a local flat.fits file.

Define the input data¶

JWST created a specification for defining how input files should be used by a pipeline, it is a JSON file named an association, see the JWST documentation.

In our example we need to specify a input raw science frame ad a background to be subtracted, see asn_subtract_bg_flat.json:

{
    "asn_rule": "Asn_Lv2Image",
    "asn_pool": "pool",
    "asn_type": "image2",
    "products": [
        {
            "name": "test_iris_subtract_bg_flat",
            "members": [
                {
                    "expname": "raw_science_frame_sci.fits",
                    "exptype": "science"
                },
                {
                    "expname": "raw_background_frame_cal.fits",
                    "exptype": "background"
                }
            ]
        }
    ]
}

Execute the pipeline from the command line¶

We can use tmtrun from a terminal to execute the pipeline:

tmtrun image2_iris.cfg asn_subtract_bg_flat.json

here is the output log:

2019-10-04 18:13:46,453 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2019-10-04 18:13:46,454 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2019-10-04 18:13:46,456 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2019-10-04 18:13:46,458 - stpipe.Image2Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2019-10-04 18:13:46,460 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2019-10-04 18:13:46,461 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2019-10-04 18:13:46,463 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2019-10-04 18:13:46,500 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args ('asn_subtract_bg_flat.json',).
2019-10-04 18:13:47,130 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'raw_science_frame_sci.fits' reftypes = ['dark']
2019-10-04 18:13:47,645 - stpipe.Image2Pipeline - INFO - Prefetch for DARK reference file is '/home/azonca/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits'.
2019-10-04 18:13:47,645 - stpipe.Image2Pipeline - INFO - Override for FLAT reference file is '/home/azonca/p/software/iris_pipeline/iris_pipeline/tests/data/flat.fits'.
2019-10-04 18:13:47,645 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'raw_background_frame_cal.fits' reftypes = ['dark']
2019-10-04 18:13:47,651 - stpipe.Image2Pipeline - INFO - Prefetch for DARK reference file is '/home/azonca/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits'.
2019-10-04 18:13:47,651 - stpipe.Image2Pipeline - INFO - Override for FLAT reference file is '/home/azonca/p/software/iris_pipeline/iris_pipeline/tests/data/flat.fits'.
2019-10-04 18:13:47,651 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2019-10-04 18:13:47,659 - stpipe.Image2Pipeline - INFO - Processing product test_iris_subtract_bg_flat
2019-10-04 18:13:47,659 - stpipe.Image2Pipeline - INFO - Working on input raw_science_frame_sci.fits ...
2019-10-04 18:13:47,918 - stpipe.Image2Pipeline.bkg_subtract - INFO - Step bkg_subtract running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>, ['raw_background_frame_cal.fits']).
2019-10-04 18:13:53,796 - stpipe.Image2Pipeline.bkg_subtract - INFO - Step bkg_subtract done
2019-10-04 18:13:53,854 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2019-10-04 18:13:53,855 - stpipe.Image2Pipeline.assign_wcs - INFO - Step skipped.
2019-10-04 18:13:53,856 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2019-10-04 18:13:53,898 - stpipe.Image2Pipeline.dark_current - INFO - Step dark_current running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2019-10-04 18:13:53,945 - stpipe.Image2Pipeline.dark_current - INFO - Using DARK reference file /home/azonca/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits
2019-10-04 18:13:54,503 - stpipe.Image2Pipeline.dark_current - INFO - Step dark_current done
2019-10-04 18:13:54,566 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2019-10-04 18:13:55,328 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2019-10-04 18:13:55,369 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2019-10-04 18:13:55,369 - stpipe.Image2Pipeline.photom - INFO - Step skipped.
2019-10-04 18:13:55,370 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2019-10-04 18:13:55,370 - stpipe.Image2Pipeline - INFO - Finished processing product test_iris_subtract_bg_flat
2019-10-04 18:13:55,370 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2019-10-04 18:13:55,606 - stpipe.Image2Pipeline - INFO - Saved model in test_iris_subtract_bg_flat_cal.fits
2019-10-04 18:13:55,606 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done

After completion, the reduced science frame test_iris_subtract_bg_flat_cal.fits is written to disk, it includes all the metadata it had initially and additional details about the processing steps that were executed.

Navigation