Nobeyama Pipeline Users Guide

2021-04 Nobeyama Pipeline Team

Overview

The Nobeyama Pipeline is an automated end-to-end data reduction software for the data taken by Nobeyama 45m Radio Telescope (NRO45m). This document describes what is Nobeyama Pipeline, how to use it, and how to examine its products.

Current version of the Nobeyama Pipeline supports On-the-Fly (OTF) observing mode. In particular, the Pipeline is optimized to the OTF observation with FOREST. Download uncalibrated MeasurementSets (MS) from the Nobeyama Science Archive if you want to process data by Nobeyama Pipeline by yourself. The Nobeyama Science Archive also provides users with the pipeline products, which is a set of output files of the pipeline execution. Pipeline products stored in the archive is the one obtained by running Nobeyama Pipeline with the default parameter setting. To obtain the pipeline product, users may download it from the archive or process uncalibrated MS with Nobeyama Pipeline. Users might want to process the data with Nobeyama Pipeline when result obtained from the archive is not satisfactory so that parameters need to be tweaked, or when there are multiple data that should be processed at once.

The Nobeyama Pipeline has been developed based on the ALMA Pipeline, which is built on top of Common Astronomy Software Applications (CASA). Therefore, ALMA Pipeline Users Guide should be the best material to understand the Nobeyama Pipeline. ALMA Pipeline Users Guide can be downloaded from ALMA Science Portal.

The Nobeyama Pipeline shares most of the code with ALMA Pipeline. Major difference is in their "recipes" which are the sequence of pipeline processing steps ("stages"). Nobeyama Pipeline has its own "importdata" and "exportdata" stages because their implementations are strongly dependent on the observatory. Also, some stages that are not suitable to NRO45m are skipped in the Nobeyama Pipeline. The hsd_k2jycal stage is skipped because it is ALMA-specific while the hsd_flagdata stage is skipped as flagging metrics implemented in this stage are not appropriate to Nobeyama Pipeline.

Pipeline Products

Products are the set of files generated by the pipeline processing. Standard products are listed as follows:

There are additional products in the Nobeyama Pipeline.

Additional template files listed above are necessary for restoredata process. The rebase_and_image scripts are the template for manual data reduction subsequent to restoredata process. Detail about restoredata process is described in Restoring Calibrated MS. Weblogs are slightly different between the ALMA Pipeline and the Nobeyama Pipeline. The differences are described in the next section.

!!IMPORTANT NOTE!! Unlike ALMA, quality assurance was not performed for the products. Therefore, users may need to run the Nobeyama Pipeline with tweaked parameters by themselves when quality of the products is not satisfactory.

Examining Weblog

WebLog is a collection of html files and plots that summarizes data processing by pipeline. It is always a good practice to examine WebLog after downloading pipeline products and running pipeline to inspect quality of data reduction. You can open and browse WebLog using a web browser, e.g., Firefox. See appendix for solutions of known WebLog display problem. You should read sections 9 and 11 of ALMA Pipeline Users Guide for the latest version to understand contents and structure of pipeline weblog in general.

Here, major differences between ALMA Pipeline and Nobeyama Pipeline are described.

MeasurementSet Overview Pages

In the weblog generated by Nobeyama Pipeline, PWV plots and "Antenna Positions" plots are omitted. The former is due to the lack of necessary data to produce the plot while the latter is dropped because it is meaningless for single dish antenna.

!!IMPORTANT NOTE!! Beams of the multi-beam receivers such as FOREST are treated as antennas in the Nobeyama Pipeline as well as in CASA. That is because that beams are translated into antennas when the data are converted into MS.

By Task Summary Page

According to the difference of recipe, hsd_flagdata and hsd_k2jycal stages are not listed in the weblog generated by the Nobeyama Pipeline. QA scores listed in this page is optimized to ALMA Pipeline. Therefore, some scores may be misleading for Nobeyama Pipeline. For example, QA score for hsd_imaging stage is quite sensitive to the number of masked pixels whose imaging weights are too low (say, less than 10% of median weight). This is because that grid spacings of the OTF observation for ALMA are strictly fixed depending on observing frequency. On the other hand, observing setup for the NRO45m is so flexible that there may be the observing projects that intentionally set sparse spacings for the OTF observation. Such observation could produce coarse images that lead to imaging QA score to be almost zero.

hsd_imaging

In the Nobeyama Pipeline, the "Detailed Profile Map" is not created to shorten execution duration. Therefore, links to these plots located in the task execution summary page for hsd_imaging stage are not available.

Processing Data with Nobeyama Pipeline

In this section, several ways to execute the Nobeyama Pipeline are described.

Preparation

CASA

First of all, you must have the correct version of CASA to run the Nobeyama Pipeline. The CASA releases for ALMA "Cycle 7 reprise" or later, i.e. CASA 6.1.1-15 or higher, are compatible with Nobeyama Pipeline. Please see ALMA Science Portal to get the latest version of CASA for Nobeyama Pipeline. You can download CASA from either of the following sites:

Choose appropriate distribution according to your data reduction environment, and follow installation instruction for tar-ball distribution. You might want to set PATH environment variable to launch CASA by simply typing casa.

# for bash, please update directory name depending on your CASA verison
$ export PATH=/path/to/casa-6.1.1-15-pipeline-2020.1.0.40/bin:$PATH

Directory Structure

The instruction described in the following sections assumes fixed directory structure shown below. You should prepare two directories named rawdata and working in your working directory. The rawdata directory is intended to place uncalibrated data in MS format while the working directory is the place to run the Nobeyama Pipeline. You shoud have the script nropipe.py, which can be downloaded from here, at the working directory or somewhere else you can access. Note that, due to the security restriction, filename of the script is nropipe.txt on the web server. Please rename the file to nropipe.py after you download it. Initial state of your working directory should look like below:

$ tree -L 2 .
.
├── rawdata
│   └── nro-20180305JST102524-v200811.ms
└── working
    └── nropipe.py

After the execution of the Nobeyama Pipeline, you will find the new directory named products in your working directory. This is the directory that stores all the products of the Nobeyama Pipeline.

Running nropipe.py

The most convenient way is to run nropipe.py.

$ cd working
$ casa --nologger --nogui --agg -c nropipe.py

If you are working on a remote server, it is safe to combine pipeline execution with xvfb-run and/or nohup.

# login to remote server without X11 forwarding
$ ssh -x yourname@remoteserver
$ cd /path/to/working
$ nohup xvfb-run -d casa --nologger --nogui --agg -c nropipe.py &
# you can safely logout from the server without interrupting pipeline execution
$ exit

Editing and Running casa_pipescript.py

The casa_pipescript.py is one of the products of the Nobeyama Pipeline. This is a script that can be executed by -c option of CASA. You must set --pipeline option when you run the script. You may need to change the path to MS file to run the script (adding ../rawdata/ before MS name). With casa_pipescript.py, you may optionally modify parameters and customize behavior of each pipeline stage, which is defined as a CASA task. Full list of options for each task can be found in the Pipeline Reference Manual which is available on ALMA Science Portal. After updating data path and modifying the script, it can be executed by, e.g.,

$ casa --nologger --nogui —pipeline -c casa_pipescript.py

Contents of casa_pipescript.py look like below.

__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Nobeyama Radio Observatory')
context.set_state('ProjectSummary', 'telescope', 'NRO')
context.set_state('ProjectStructure', 'recipe_name', 'hsdn_calimage')
try:
    hsdn_importdata(vis=['nro-20180305JST102524-v200811.ms'], session=['default'])
    h_tsyscal(pipelinemode="automatic")
    hsd_tsysflag(pipelinemode="automatic")
    hsd_skycal(pipelinemode="automatic")
    hsd_applycal(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_imaging(pipelinemode="automatic")
finally:
    h_save()

Restoring Calibrated MS (NOT SUPPORTED YET)

WARNING
The restore procedures described below does not work for the CASA 6.1.1 release. They will be supported in the future releases.

In addition to casa_pipescript.py, you will also have another script casa_piperestorescript.py in pipeline products. It is a template file to restore calibrated MS from uncalibrated MS. In order to run restore script, both uncalibrated MeasurementSet and all pipeline products should be located at rawdata directory. Pipeline products can be generated by a previous pipeline execution or downloaded from Nobeyama Science Archive. An example of casa_piperestorescript.py is shown below.

__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Nobeyama Radio Observatory')
context.set_state('ProjectSummary', 'telescope', 'NRO')
context.set_state('ProjectStructure', 'recipe_name', 'hsdn_calimage')
try:
    hsdn_restoredata(vis=['nro-20180305JST102524-v200811.ms'], reffile='./nroscalefile.csv')
finally:
    h_save()

Again, the script can be run with -c option. You need to set --pipeline option to run the script. Also, path to the MS may need to be updated.

One advantage over the pipeline processing with casa_pipescript.py is a capability to apply scaling factors for multi-beam receivers such as FOREST to compensate for the variation of receiver gain among beams. You can update nroscalefile.csv according to the measurement of relative gain correction factor. As shown in the example contents of casa_piperestorescript.py above, the hsdn_restoredata has the parameter reffile to accept the file name for the scaling. By default, the script assumes that nroscalefile.csv is placed in the working directory. You should locate nroscalefile.csv at the working directory or update reffile according to the name or the location of the scaling file. Example contents of the nroscalefile.csv is shown below. See description in header of nroscalefile.csv about the file format and how to modify scaling factors.

#MS,Beam,Spwid,Polarization,Factor
mg2-20170525142607-180419.ms,NRO-BEAM0,0,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM0,0,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM0,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM0,1,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,0,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,0,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,1,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM2,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM2,1,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,0,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,0,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,1,YY,1.0

After restoration, you can proceed subsequent data reduction steps by either (1) running pipeline tasks by combining restore script with casa_pipescript.py or (2) running casa tasks with the help of template reduction script rebase_and_image, which is also part of pipeline products. For the case (1), since hsdn_restoredata task does the processing corresponding to the stages up to hsd_applycal, combined script will look like below.

__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Nobeyama Radio Observatory')
context.set_state('ProjectSummary', 'telescope', 'NRO')
context.set_state('ProjectStructure', 'recipe_name', 'hsdn_calimage')
try:
    hsdn_restoredata(vis=['nro-20180305JST102524-v200811.ms'], reffile='./nroscalefile.csv')
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_baseline(pipelinemode="automatic")
    hsd_blflag(pipelinemode="automatic")
    hsd_imaging(pipelinemode="automatic")
finally:
    h_save()

In the case (2), the rebase_and_image script performs baseline subtraction and imaging by using sdbaseline and sdimaging directly rather than executing pipeline tasks. It means that the processing is simpler and faster than the case (1) because the rebase_and_image is a deterministic script and it skipps heuristics to optimize the processing parameter at runtime. The rebase_and_image script can be run with -c option as well. Below is the example to restore calibrated data with casa_piperestorescript.py and then further process the data with rebase_and_image script.

# you are now in "restore" directory
$ pwd
/path/to/restore
# copy nroscalefile.csv from "products" directory
# and edit the contents accordingly
$ cp ../products/nroscalefile.csv .
# copy casa_piperestorescript.py from "products" directory
$ cp ../products/casa_piperestorescript.py .
# run casa_piperestorescript.py
$ casa --pipeline -c casa_piprestorescript.py
# run rebase_and_image script
$ casa -c ../products/rebase_and_image.py

Appendix: Solutions of Known Weblog Display Problem

When you encounter a problem when opening the weblog, you should consider launching http server at localhost. If you have python or CASA the following command launches http server.

python3 -m http.server --bind localhost 8000

In the above example, the working directory will become the root so that you can access the weblog as a relative path from there, e.g., http://localhost:8000/path/to/weblog/html/index.html.

Alternatively, Pipeline task, h_weblog, does above for you. Invoke casa with --pipeline option at working directory and execute the task as shown below.

h_weblog(relpath=‘pipeline-*****/html’)

Otherwise, you need to change the browser setting as described below. Note that these settings lower the browser security so you may need to revert the setting when you access external website.

Firefox

Navigate to about:config and search for the privacy.file_unique_origin preference. Change the preference value to false.

Safari

Open Safari preferences, navigate to Advanced tab and check the Show Develop in menu bar option. From the new Develop menu option now visible at the top of the screen, select Disable Local File Restriction.

Chrome

The --disable-web-security and --user-data-dir command line arguments must passed to Chrome via the command line. For example, on MacOS start Chrome like this:

/Applications/Chrome.app/Contents/MacOS/Chrome --disable-web-security --user-data-dir=~/tmp