Example pipeline execution¶
Here is an example of what it takes to configure and run a pipeline with flat-fielding and background subtraction.
Setup CRDS¶
Make sure you have a local checkout of the CRDS cache as explained in the Getting started page. Either run source setup_local_crds.sh to setup the enviroment variables in the shell or execute setup_local_crds.py for Python. This is needed to point the CRDS software to the CRDS cache. Optionally source this in your shell configuration to automatically set this up.
[1]:
%run ../setup_local_crds.py
Get input simulations¶
Download simulated input FITS files for the IRIS imager from Figshare. It contains a raw science frame, a raw flat frame and a raw background frame.
[2]:
import urllib.request
import zipfile
import os.path
if not os.path.exists("iris_example_data/raw_flat_frame_cal.fits"):
urllib.request.urlretrieve(
"https://ndownloader.figshare.com/articles/9941939/versions/1",
"iris_example_data.zip"
)
with zipfile.ZipFile("iris_example_data.zip", 'r') as zip_ref:
zip_ref.extractall("iris_example_data")
# this could take a few minutes
[3]:
%ls iris_example_data/
raw_background_frame_cal.fits raw_science_frame_sci.fits
raw_flat_frame_cal.fits
Preprocess the flat frame¶
First we need to remove the dark frame from the flat frame and normalize it. A dark frame is already available in the CRDS and the pipeline knows how to retrieve it based on the metadata in the FITS file headers.
We can check in the package documentation what are the available pipelines and check the configuration options of the pipeline.ProcessFlatfieldL2 class.
We do not need to customize it so we can directly call it from tmtrun and pass the input FITS file:
This will pickup the relevant dark frame from the CRDS and process the file:
[4]:
!tmtrun iris_pipeline.pipeline.ProcessFlatfieldL2 iris_example_data/raw_flat_frame_cal.fits
2021-06-29 10:00:29,767 - CRDS - ERROR - Error determining best reference for 'pars-darkcurrentstep' = Unknown reference type 'pars-darkcurrentstep'
2021-06-29 10:00:29,779 - CRDS - ERROR - Error determining best reference for 'pars-normalizestep' = Unknown reference type 'pars-normalizestep'
2021-06-29 10:00:29,788 - CRDS - ERROR - Error determining best reference for 'pars-processflatfieldl2' = Unknown reference type 'pars-processflatfieldl2'
2021-06-29 10:00:29,790 - stpipe.ProcessFlatfieldL2 - INFO - ProcessFlatfieldL2 instance created.
2021-06-29 10:00:29,791 - stpipe.ProcessFlatfieldL2.dark_current - INFO - DarkCurrentStep instance created.
2021-06-29 10:00:29,792 - stpipe.ProcessFlatfieldL2.normalize - INFO - NormalizeStep instance created.
2021-06-29 10:00:29,792 - stpipe - INFO - Hostname: fv-az95-499
2021-06-29 10:00:29,792 - stpipe - INFO - OS: Linux
2021-06-29 10:00:29,839 - stpipe.ProcessFlatfieldL2 - INFO - Step ProcessFlatfieldL2 running with args ('iris_example_data/raw_flat_frame_cal.fits',).
2021-06-29 10:00:29,842 - stpipe.ProcessFlatfieldL2 - INFO - Step ProcessFlatfieldL2 parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'iris_example_data', 'steps': {'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'iris_example_data', 'dark_output': None}, 'normalize': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'iris_example_data', 'method': 'median'}}}
2021-06-29 10:00:30,070 - stpipe.ProcessFlatfieldL2 - INFO - Prefetching reference files for dataset: 'raw_flat_frame_cal.fits' reftypes = ['dark']
2021-06-29 10:00:30,079 - stpipe.ProcessFlatfieldL2 - INFO - Prefetch for DARK reference file is '/home/runner/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits'.
2021-06-29 10:00:30,079 - stpipe.ProcessFlatfieldL2 - INFO - Starting preprocess flatfield ...
2021-06-29 10:00:30,079 - stpipe.ProcessFlatfieldL2 - INFO - Processing product iris_example_data/raw_flat_frame
2021-06-29 10:00:30,080 - stpipe.ProcessFlatfieldL2 - INFO - Working on input iris_example_data/raw_flat_frame_cal.fits ...
2021-06-29 10:00:30,348 - stpipe.ProcessFlatfieldL2.dark_current - INFO - Step dark_current running with args (<IRISImageModel(4096, 4096) from raw_flat_frame_cal.fits>,).
2021-06-29 10:00:30,349 - stpipe.ProcessFlatfieldL2.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'iris_example_data', 'dark_output': None}
2021-06-29 10:00:30,373 - stpipe.ProcessFlatfieldL2.dark_current - INFO - Using DARK reference file /home/runner/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits
2021-06-29 10:00:30,814 - stpipe.ProcessFlatfieldL2.dark_current - INFO - Step dark_current done
2021-06-29 10:00:30,867 - stpipe.ProcessFlatfieldL2.normalize - INFO - Step normalize running with args (<IRISImageModel(4096, 4096) from raw_flat_frame_cal.fits>,).
2021-06-29 10:00:30,868 - stpipe.ProcessFlatfieldL2.normalize - INFO - Step normalize parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': 'iris_example_data', 'method': 'median'}
2021-06-29 10:00:31,449 - stpipe.ProcessFlatfieldL2.normalize - INFO - running normalize with method median
2021-06-29 10:00:31,484 - stpipe.ProcessFlatfieldL2.normalize - INFO - Step normalize done
2021-06-29 10:00:31,485 - stpipe.ProcessFlatfieldL2 - INFO - Finished processing product iris_example_data/raw_flat_frame
2021-06-29 10:00:31,485 - stpipe.ProcessFlatfieldL2 - INFO - ... ending preprocess flatfield
2021-06-29 10:00:31,820 - stpipe.ProcessFlatfieldL2 - INFO - Saved model in raw_flat_frame_flat.fits
2021-06-29 10:00:31,820 - stpipe.ProcessFlatfieldL2 - INFO - Step ProcessFlatfieldL2 done
Configure the image processing pipeline¶
The `ProcessImagerL2Pipeline <api/iris_pipeline.pipeline.ProcessImagerL2Pipeline.rst#iris_pipeline.ProcessImagerL2Pipeline>`__ can be configured using a INI-style configuration file:
[5]:
%%file image2_iris.cfg
name = "ProcessImagerL2Pipeline"
class = "iris_pipeline.pipeline.ProcessImagerL2Pipeline"
save_results = True
[steps]
[[bkg_subtract]]
[[assign_wcs]]
skip = True
[[flat_field]]
config_file = flat_field.cfg
[[photom]]
skip = True
[[resample]]
skip = True
Writing image2_iris.cfg
first we specify that we want to execute the pipeline defined in the pipeline, then we can configure each of the steps, for example skip some of them. Also we can import the configuration of a step from another file, in this case flat_field.cfg:
[6]:
%%file flat_field.cfg
name = "flat_field"
class = "jwst.flatfield.FlatFieldStep"
override_flat = 'raw_flat_frame_flat.fits'
Writing flat_field.cfg
Define the input data¶
JWST created a specification for defining how input files should be used by a pipeline, it is a JSON file named an association, see the JWST documentation.
In our example we need to specify a input raw science frame ad a background to be subtracted:
[7]:
%%file association.json
{
"asn_rule": "Asn_Lv2Image",
"asn_pool": "pool",
"asn_type": "image2",
"products": [
{
"name": "test_iris_subtract_bg_flat",
"members": [
{
"expname": "iris_example_data/raw_science_frame_sci.fits",
"exptype": "science"
},
{
"expname": "iris_example_data/raw_background_frame_cal.fits",
"exptype": "background"
}
]
}
]
}
Writing association.json
Execute the pipeline from the command line¶
Execute the pipeline from the command line We can use tmtrun from a terminal to execute the pipeline:
[8]:
!tmtrun image2_iris.cfg association.json
2021-06-29 10:00:35,010 - CRDS - ERROR - Error determining best reference for 'pars-backgroundstep' = Unknown reference type 'pars-backgroundstep'
2021-06-29 10:00:35,278 - CRDS - ERROR - Error determining best reference for 'pars-assignwcsstep' = Unknown reference type 'pars-assignwcsstep'
2021-06-29 10:00:35,547 - CRDS - ERROR - Error determining best reference for 'pars-parsesubarraymapstep' = Unknown reference type 'pars-parsesubarraymapstep'
2021-06-29 10:00:35,858 - CRDS - ERROR - Error determining best reference for 'pars-darkcurrentstep' = Unknown reference type 'pars-darkcurrentstep'
2021-06-29 10:00:36,361 - CRDS - ERROR - Error determining best reference for 'pars-flatfieldstep' = Unknown reference type 'pars-flatfieldstep'
2021-06-29 10:00:36,692 - CRDS - ERROR - Error determining best reference for 'pars-photomstep' = Unknown reference type 'pars-photomstep'
2021-06-29 10:00:37,192 - CRDS - ERROR - Error determining best reference for 'pars-resamplestep' = Unknown reference type 'pars-resamplestep'
2021-06-29 10:00:37,672 - CRDS - ERROR - Error determining best reference for 'pars-processimagerl2pipeline' = Unknown reference type 'pars-processimagerl2pipeline'
2021-06-29 10:00:37,679 - stpipe.ProcessImagerL2Pipeline - INFO - ProcessImagerL2Pipeline instance created.
2021-06-29 10:00:37,680 - stpipe.ProcessImagerL2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2021-06-29 10:00:37,681 - stpipe.ProcessImagerL2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2021-06-29 10:00:37,682 - stpipe.ProcessImagerL2Pipeline.parse_subarray_map - INFO - ParseSubarrayMapStep instance created.
2021-06-29 10:00:37,683 - stpipe.ProcessImagerL2Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2021-06-29 10:00:37,684 - stpipe.ProcessImagerL2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2021-06-29 10:00:37,685 - stpipe.ProcessImagerL2Pipeline.photom - INFO - PhotomStep instance created.
2021-06-29 10:00:37,686 - stpipe.ProcessImagerL2Pipeline.resample - INFO - ResampleStep instance created.
2021-06-29 10:00:37,686 - stpipe - INFO - Hostname: fv-az95-499
2021-06-29 10:00:37,686 - stpipe - INFO - OS: Linux
2021-06-29 10:00:37,787 - stpipe.ProcessImagerL2Pipeline - INFO - Step ProcessImagerL2Pipeline running with args ('association.json',).
2021-06-29 10:00:37,793 - stpipe.ProcessImagerL2Pipeline - INFO - Step ProcessImagerL2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sigma': 3.0, 'maxiters': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'parse_subarray_map': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'exptime', 'single': False, 'blendheaders': True}}}
2021-06-29 10:00:38,070 - stpipe.ProcessImagerL2Pipeline - INFO - Prefetching reference files for dataset: 'raw_science_frame_sci.fits' reftypes = ['dark']
2021-06-29 10:00:38,079 - stpipe.ProcessImagerL2Pipeline - INFO - Prefetch for DARK reference file is '/home/runner/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits'.
2021-06-29 10:00:38,079 - stpipe.ProcessImagerL2Pipeline - INFO - Override for FLAT reference file is '/home/runner/work/iris_pipeline/iris_pipeline/docs/raw_flat_frame_flat.fits'.
2021-06-29 10:00:38,080 - stpipe.ProcessImagerL2Pipeline - INFO - Starting calwebb_image2 ...
2021-06-29 10:00:38,092 - stpipe.ProcessImagerL2Pipeline - INFO - Processing product test_iris_subtract_bg_flat
2021-06-29 10:00:38,092 - stpipe.ProcessImagerL2Pipeline - INFO - Working on input iris_example_data/raw_science_frame_sci.fits ...
2021-06-29 10:00:38,401 - stpipe.ProcessImagerL2Pipeline.bkg_subtract - INFO - Step bkg_subtract running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>, ['iris_example_data/raw_background_frame_cal.fits']).
2021-06-29 10:00:38,402 - stpipe.ProcessImagerL2Pipeline.bkg_subtract - INFO - Step bkg_subtract parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'bsub', 'search_output_file': True, 'input_dir': '', 'sigma': 3.0, 'maxiters': None}
2021-06-29 10:00:44,273 - stpipe.ProcessImagerL2Pipeline.bkg_subtract - INFO - Step bkg_subtract done
2021-06-29 10:00:44,329 - stpipe.ProcessImagerL2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2021-06-29 10:00:44,330 - stpipe.ProcessImagerL2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2021-06-29 10:00:44,330 - stpipe.ProcessImagerL2Pipeline.assign_wcs - INFO - Step skipped.
2021-06-29 10:00:44,332 - stpipe.ProcessImagerL2Pipeline.assign_wcs - INFO - Step assign_wcs done
2021-06-29 10:00:44,369 - stpipe.ProcessImagerL2Pipeline.parse_subarray_map - INFO - Step parse_subarray_map running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2021-06-29 10:00:44,369 - stpipe.ProcessImagerL2Pipeline.parse_subarray_map - INFO - Step parse_subarray_map parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2021-06-29 10:00:44,385 - stpipe.ProcessImagerL2Pipeline.parse_subarray_map - INFO - No SUBARR_MAP extension found
2021-06-29 10:00:44,386 - stpipe.ProcessImagerL2Pipeline.parse_subarray_map - INFO - Step parse_subarray_map done
2021-06-29 10:00:44,423 - stpipe.ProcessImagerL2Pipeline.dark_current - INFO - Step dark_current running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2021-06-29 10:00:44,424 - stpipe.ProcessImagerL2Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}
2021-06-29 10:00:44,447 - stpipe.ProcessImagerL2Pipeline.dark_current - INFO - Using DARK reference file /home/runner/crds_cache/references/tmt/iris/tmt_iris_dark_0001.fits
2021-06-29 10:00:44,859 - stpipe.ProcessImagerL2Pipeline.dark_current - INFO - Step dark_current done
2021-06-29 10:00:44,913 - stpipe.ProcessImagerL2Pipeline.flat_field - INFO - Step flat_field running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2021-06-29 10:00:44,914 - stpipe.ProcessImagerL2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2021-06-29 10:00:45,615 - stpipe.ProcessImagerL2Pipeline.flat_field - INFO - Step flat_field done
2021-06-29 10:00:45,671 - stpipe.ProcessImagerL2Pipeline.photom - INFO - Step photom running with args (<IRISImageModel(4096, 4096) from raw_science_frame_sci.fits>,).
2021-06-29 10:00:45,672 - stpipe.ProcessImagerL2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2021-06-29 10:00:45,672 - stpipe.ProcessImagerL2Pipeline.photom - INFO - Step skipped.
2021-06-29 10:00:45,673 - stpipe.ProcessImagerL2Pipeline.photom - INFO - Step photom done
2021-06-29 10:00:45,673 - stpipe.ProcessImagerL2Pipeline - INFO - Finished processing product test_iris_subtract_bg_flat
2021-06-29 10:00:45,674 - stpipe.ProcessImagerL2Pipeline - INFO - ... ending calwebb_image2
2021-06-29 10:00:46,056 - stpipe.ProcessImagerL2Pipeline - INFO - Saved model in test_iris_subtract_bg_flat_cal.fits
2021-06-29 10:00:46,056 - stpipe.ProcessImagerL2Pipeline - INFO - Step ProcessImagerL2Pipeline done
Plot the reduced frame¶
Finally we can use iris_pipeline to load the data and plot them:
[9]:
import iris_pipeline
reduced_science_frame = iris_pipeline.datamodels.IRISImageModel("test_iris_subtract_bg_flat_cal.fits")
try:
%matplotlib inline
import matplotlib.pyplot as plt
plt.figure()
plt.imshow(reduced_science_frame.data, vmin=0, vmax=1500)
plt.title("Processed frame")
except ImportError:
pass
[10]:
!rm image2_iris.cfg association.json flat_field.cfg