Mislabelled scans in higher dump rate observations

Description and impact

Observations taken at higher dump rates (2 and 4 s periods) often have a one or two dumps labelled as belonging to the next source to be observed, before the antennas start slewing to a target.

This will effectively result in power from the previous source being included in calibration and/or imaging. This could lead to spurious detection of a faint source at the phase centre of your target field, or incorrect flux density measurements.

Unfortunately we do not currently have any procedures that will flag this, so the SDP pipeline images may be affected.

How to identify the problem

We strongly advise users to always check the scan listings in the archive, as well as the script progress log, to ensure that the scan sequence has been executed as expected, as follows.

Click on the details dropdown arrow to the right below the observation listing.

Open

Scroll down to the scan listing and examine the column named ScanState. The CompScanLabel is a historical artefact and may be ignored in this discussion.

Open

The expected sequence of observations is shown in the first green block above. The antennas slew to the target (e.g. scan 0). Once the antenna is on target, its status changes to ‘locked’, and the scan is then labelled as a track (scan 1). The antennas then slew to the next target in the observation sequence.

Note that scan 4 (marked in red) changes target label from J2147-7536 to J2147-8132, and ScanState is still ‘track'. This happens because there is a small delay between the target being set, and the antenna controller being given the command to slew to the next target. However, the antennas have not started to slew, and are still fully locked on the previous target (the gain calibrator), so there are 4s of data that is mislabelled. This flux will be included as target visibilities when imaging.

The antennas start slewing to J2147-8132 in scan 5, and are locked and tracking in scan 6. Note that the problem recurs in scan 7.

For your interest, this is the accompanying script log.

How to deal with it

You will have to treat any SDP image as suspect, and recalibrate and image the dataset after removing the incorrectly labelled scans.

We plan to fix this issue in the future in katdal, or in the logic of the observation script itself. In the meantime manual selection or flagging will have to be done.

Using CASA

If you have already downloaded a measurement set - you can examine the scan listing from listobs and flag incorrect tracks. Note that the scan numbering will be different, since any scans labelled as slews will not be converted to MS. Look for very short duration scans, while referring to the observation details as described above.

Using mvftoms.py

If you have not yet downloaded the data, you can use mvf2ms.py to select the scans to convert to MS.

under development Note that this is not yet supported by the archive interface - users will need to download and install katdal and use a security token to access the datastore directly from the archive.

1. Use the first download icon 'copy rdb link with token to clipboard' to obtain a valid url to the dataset. Note that the example above has been moved to tape - if this is the case with your data you will need to request for it to be loaded back onto disk.

2. Identify the scans that you wish to keep. Note that slews are automatically discarded. Scans can be identified by a comma-separated list, or a CASA-style range.

3. mvftoms.py -o new_dataset.ms --scans 1,3,6,9,12,14 <rdblink with security token>