The Nobeyama Pipeline is an automated end-to-end data reduction software for the data taken by Nobeyama 45m Radio Telescope (NRO45m). This document describes what is Nobeyama Pipeline, how to use it, and how to examine its products.
Current version of the Nobeyama Pipeline supports On-the-Fly (OTF) observing mode. In particular, the Pipeline is optimized to the OTF observation with FOREST. Download uncalibrated MeasurementSets (MS) from the Nobeyama Science Data Archive if you want to process data by Nobeyama Pipeline by yourself. The Nobeyama Science Data Archive also provides users with the pipeline products, which is a set of output files of the pipeline execution. Pipeline product stored in the archive is the one obtained by running Nobeyama Pipeline with the default parameter setting. To obtain the pipeline product, users may download it from the archive or process uncalibrated MS with Nobeyama Pipeline. Users might want to process the data with Nobeyama Pipeline when result obtained from the archive is not satisfactory so that parameters need to be tweaked, or when there are multiple data that should be processed at once.
The Nobeyama Pipeline has been developed based on the ALMA Pipeline, which is built on top of Common Astronomy Software Applications (CASA). Therefore, ALMA Pipeline Users Guide should be the best material to understand the Nobeyama Pipeline. ALMA Pipeline Users Guide can be downloaded from ALMA Science Portal.
The Nobeyama Pipeline shares most of the code with ALMA Pipeline. Major difference is in their “recipes” which are the sequence of pipeline processing steps (“stages”). Nobeyama Pipeline has its own “importdata” and “exportdata” stages because their implementations are strongly dependent on the observatory. Also, some stages that are not suitable to NRO45m are skipped in the Nobeyama Pipeline. The hsd_k2jycal stage is skipped because it is ALMA-specific while the hsd_flagdata stage is skipped as flagging metrics implemented in this stage are not appropriate to Nobeyama Pipeline.
Products are a set of files generated by the pipeline processing. Standard products are listed as follows:
weblog.tgz)casa_pipescript.py)casa_piperestorescript.py)casa_commands.log)pipeline_aquareport.xml)pipeline_manifest.xml)There are additional products in the Nobeyama Pipeline.
nroscalefile.csv)rebase_and_image.py)rebase_and_image_config.py)Additional template files listed above are necessary for restoredata process. The rebase_and_image scripts are the template for manual data reduction subsequent to restoredata process. Details of restoredata process are described in Restoring Calibrated MS. Weblogs are slightly different between the ALMA Pipeline and the Nobeyama Pipeline. The differences are described in the next section.
!!IMPORTANT NOTE!! Unlike ALMA, quality assurance was not performed for the products. Therefore, users may need to run the Nobeyama Pipeline with tweaked parameters by themselves when quality of the products is not satisfactory.
WebLog is a collection of html files and plots that summarizes data processing by pipeline. It is always a good practice to examine WebLog after downloading pipeline products and running pipeline to inspect the quality of data reduction. You can open and browse WebLog using a web browser, e.g., Firefox. See appendix for solutions of known WebLog display problem. You should read sections 9 and 11 of ALMA Pipeline Users Guide for the latest version to understand contents and structure of pipeline weblog in general.
Here, major differences between ALMA Pipeline and Nobeyama Pipeline are described.
In the weblog generated by Nobeyama Pipeline, PWV plots and “Antenna Positions” plots are omitted. The former is due to the lack of necessary data to produce the plot while the latter is dropped because it is meaningless for single dish antenna.
!!IMPORTANT NOTE!! Beams of the multi-beam receivers such as FOREST are treated as antennas in the Nobeyama Pipeline as well as in CASA. That is because “beams” are translated into “antennas” when the data are converted into MS.
According to the difference of recipe, hsd_flagdata and hsd_k2jycal stages are not listed in the weblog generated by the Nobeyama Pipeline. QA scores listed in this page is optimized to ALMA Pipeline. Therefore, some scores may be misleading for Nobeyama Pipeline. For example, QA score for hsd_imaging stage is quite sensitive to the number of masked pixels whose imaging weights are too low (say, less than 10% of median weight). This is because the grid spacings of the OTF observation for ALMA are strictly fixed depending on observing frequency. On the other hand, observing setup for the NRO45m is so flexible that there may be the observing projects that intentionally set sparse spacings for the OTF observation. Such observation could produce coarse images that lead to imaging QA score to be almost zero.
hsd_imagingIn the Nobeyama Pipeline, the “Detailed Profile Map” is not created to shorten execution duration. Therefore, links to these plots located in the task execution summary page for hsd_imaging stage are not available.
In this section, several ways to execute the Nobeyama Pipeline are described.
First of all, you must have the correct version of CASA to run the Nobeyama Pipeline. The CASA releases for ALMA “Cycle 7 reprise” or later, i.e. CASA 6.1.1-15 or higher, are compatible with Nobeyama Pipeline. The latest version is recommended. Please see ALMA Science Portal to obtain the latest version of CASA for Nobeyama Pipeline. You can download CASA from either of the following sites:
Choose appropriate distribution according to your data reduction environment, and follow installation instruction for tar-ball distribution. You might want to set PATH environment variable to launch CASA by simply typing casa.
# for bash, please update directory name depending on your CASA verison
$ export PATH=/path/to/casa-6.1.1-15-pipeline-2020.1.0.40/bin:$PATHThe instruction described in the following sections assumes fixed directory structure shown below. You should prepare two directories named rawdata and working in your working directory. The rawdata directory is intended to place uncalibrated data in MS format while the working directory is the place to run the Nobeyama Pipeline. You should have the script nropipe.py, which can be downloaded from here, at the working directory or somewhere else you can access. Note that, due to the security restriction, filename of the script is nropipe.txt on the web server. Please rename the file to nropipe.py after you download it. Initial state of your working directory should look like below:
$ tree -L 2 .
.
├── rawdata
│ └── nro-20180305JST102524-v200811.ms
└── working
└── nropipe.pyThe archived data, which has extra suffix .tgz, is also acceptable as input data. After the execution of the Nobeyama Pipeline, you will find the new directory named products in your working directory. This is the directory that stores all the products of the Nobeyama Pipeline.
nropipe.pyThe most convenient way is to run nropipe.py.
$ cd working
$ casa --nologger --nogui --agg -c nropipe.pyIf you are working on a remote server, it is safe to combine pipeline execution with xvfb-run and/or nohup.
# login to remote server without X11 forwarding
$ ssh -x yourname@remoteserver
$ cd /path/to/working
$ nohup xvfb-run -d casa --nologger --nogui --agg -c nropipe.py &
# you can safely logout from the server without interrupting pipeline execution
$ exitcasa_pipescript.pyThe casa_pipescript.py is one of the products of the Nobeyama Pipeline. This is a script that can be executed by -c option of CASA. You must set --pipeline option when you run the script. You may need to change the path to MS file to run the script (adding ../rawdata/ before MS name). With casa_pipescript.py, you may optionally modify parameters and customize behavior of each pipeline stage, which is defined as a CASA task. Full list of options for each task can be found in the Pipeline Reference Manual which is available on ALMA Science Portal. After updating data path and modifying the script, it can be executed by, e.g.,
$ casa --nologger --nogui —pipeline -c casa_pipescript.pyContents of casa_pipescript.py look like below.
__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Nobeyama Radio Observatory')
context.set_state('ProjectSummary', 'telescope', 'NRO')
context.set_state('ProjectStructure', 'recipe_name', 'hsdn_calimage')
try:
hsdn_importdata(vis=['nro-20180305JST102524-v200811.ms'], session=['default'])
h_tsyscal(pipelinemode="automatic")
hsd_tsysflag(pipelinemode="automatic")
hsd_skycal(pipelinemode="automatic")
hsd_applycal(pipelinemode="automatic")
hsd_baseline(pipelinemode="automatic")
hsd_blflag(pipelinemode="automatic")
hsd_baseline(pipelinemode="automatic")
hsd_blflag(pipelinemode="automatic")
hsd_imaging(pipelinemode="automatic")
finally:
h_save()WARNING Please make sure that your products are based on CASA 6.2.1 or higher when you try the restore procedures described below. The restore procedures does not work for the CASA 6.1.1 release.
In addition to casa_pipescript.py, you will also have another script casa_piperestorescript.py in pipeline products. It is a template file to restore calibrated MS from uncalibrated MS. In order to run restore script, both uncalibrated MS and all pipeline products should be obtained. Pipeline products can be obtained from Nobeyama Science Data Archive. Alternatively, it can be generated from uncalibrated MS by running Nobeyama Pipeline. Please make sure that the products were created by CASA version 6.2.1 or higher. You can check the version in the WebLog overview page. Please look at the “Pipeline Summary” section (see figure below), and check “CASA Version” row to find out the CASA version that the products are based on. The overview page is included in the WebLog digest PDF appeared the search result of Nobeyama Science Data Archive. So, you should be able to check the CASA version before you download the products.
An example of casa_piperestorescript.py is shown below.
__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Nobeyama Radio Observatory')
context.set_state('ProjectSummary', 'telescope', 'NRO')
context.set_state('ProjectStructure', 'recipe_name', 'hsdn_calimage')
try:
hsdn_restoredata(vis=['nro-20180305JST102524-v200811.ms'], reffile='./nroscalefile.csv')
finally:
h_save()Again, the script can be run with -c option, with the --pipeline option to import pipeline during startup. By default, the script searches specified MS for ../rawdata. If you locate the MS in another directory, path to the MS needs to be updated. The script accepts raw content of the Archive, which is compressed MS with suffix .tgz as well. You need to append the suffix if you wish to process compressed MS directly. Do not locate the MS in the working directory. This will cause unexpected error. Pipeline products need to be downloaded from the Archive. You can add a parameter, products_dir, in hsdn_restoredata to specify the location of the Pipeline products.
One advantage of the restore procedure with casa_piperestoresctipt.py over the pipeline processing using casa_pipescript.py is a capability to apply scaling factors for multi-beam receivers such as FOREST to compensate for the variation of receiver gain among beams. You can update nroscalefile.csv according to the measurement of relative gain correction factor. As shown in the example contents of casa_piperestorescript.py above, the hsdn_restoredata has the parameter reffile to accept the file name for the scaling. By default, the script assumes that nroscalefile.csv is placed in the working directory. You should locate nroscalefile.csv at the working directory or update reffile according to the name or the location of the scaling file. Example contents of the nroscalefile.csv is shown below. See description in header of nroscalefile.csv about the file format and how to modify scaling factors.
#MS,Beam,Spwid,Polarization,Factor
mg2-20170525142607-180419.ms,NRO-BEAM0,0,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM0,0,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM0,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM0,1,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,0,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,0,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM1,1,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM2,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM2,1,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,0,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,0,YY,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,1,XX,1.0
mg2-20170525142607-180419.ms,NRO-BEAM3,1,YY,1.0After restoration, you can proceed subsequent data reduction steps by either (1) running pipeline tasks by combining restore script with casa_pipescript.py or (2) running casa tasks with the help of template reduction script rebase_and_image, which is also part of pipeline products. For the case (1), since hsdn_restoredata task does the processing corresponding to the stages up to hsd_applycal, combined script will look like below.
__rethrow_casa_exceptions = True
context = h_init()
context.set_state('ProjectSummary', 'observatory', 'Nobeyama Radio Observatory')
context.set_state('ProjectSummary', 'telescope', 'NRO')
context.set_state('ProjectStructure', 'recipe_name', 'hsdn_calimage')
try:
hsdn_restoredata(vis=['nro-20180305JST102524-v200811.ms'], reffile='./nroscalefile.csv')
hsd_baseline(pipelinemode="automatic")
hsd_blflag(pipelinemode="automatic")
hsd_baseline(pipelinemode="automatic")
hsd_blflag(pipelinemode="automatic")
hsd_imaging(pipelinemode="automatic")
finally:
h_save()In the case (2), the rebase_and_image script performs baseline subtraction and imaging by using sdbaseline and sdimaging directly rather than executing pipeline tasks. It means that the processing is simpler and faster than the case (1) because the rebase_and_image is a deterministic script and it skips heuristics to optimize the processing parameter at runtime. The rebase_and_image script can be run with -c option as well. Below is the example to restore calibrated data with casa_piperestorescript.py and then further process the data with rebase_and_image script. The rebase_and_image script will output all the files to ../rebase. Note that rebase_and_image.py is a template script that implements minimal processing steps. You can edit the script according to your needs.
# you are now in "restore" directory
$ pwd
/path/to/restore
# copy nroscalefile.csv from "products" directory
# and edit the contents accordingly
$ cp ../products/nroscalefile.csv .
# copy casa_piperestorescript.py from "products" directory
$ cp ../products/casa_piperestorescript.py .
# run casa_piperestorescript.py
$ casa --pipeline -c casa_piprestorescript.py
# run rebase_and_image script
# output files will be found in ../rebase
$ casa -c ../products/rebase_and_image.pyWhen you encounter a problem when opening the weblog, you should consider launching http server at localhost. If you have python or CASA the following command launches http server.
python3 -m http.server --bind localhost 8000By default, the current directory, where the above command is executed, will become the root path so that, if the weblog is located at ./relative/path/to/weblog/html/index.html, you can access the weblog with http://localhost:8000/relative/path/to/weblog/html/index.html. If current directory is not preferable, you can specify the root path via --directory option.
Alternatively, Pipeline task, h_weblog, does above for you. Invoke casa with --pipeline option at working directory and execute the task as shown below.
h_weblog(relpath=‘pipeline-*****/html’)Otherwise, you need to change the browser setting as described below. Note that these settings lower the browser security so you may need to revert the setting when you access external website.
Navigate to about:config and search for the privacy.file_unique_origin preference. Change the preference value to false.
Open Safari preferences, navigate to Advanced tab and check the Show Develop in menu bar option. From the new Develop menu option now visible at the top of the screen, select Disable Local File Restriction.
The --disable-web-security and --user-data-dir command line arguments must passed to Chrome via the command line. For example, on MacOS start Chrome like this:
/Applications/Chrome.app/Contents/MacOS/Chrome --disable-web-security --user-data-dir=~/tmp