This is a walkthrough of the items listed under your observation, and what to look out for.

Introduction

The SARAO archive can be found on https://archive.sarao.ac.za/.

Once you have logged in, you will be taken to a screen that lists all observations that you have access to. If you are not on any active proposals, you will only see non-proprietary data consisting of observations past their proprietary periods.

Open

Screenshot of archive listing of openly available data'

Note that the tape icon indicates that the data is no longer readily available (on spinning disk) but has been moved to long term tape storage. This will be true for any data older than six months, unless otherwise requested by the PI. A request to restage the data may be sent (by clicking on the tape icon) but since this is a manual process it may take up to a few weeks before the data is made available for download.

Alternatively, once you receive a notification from the Astronomer on Duty via the OPT that your observation is complete, you can follow the link to the archive to view the observation.

Getting access to datasets as a co-investigator

It is up to the PI of the proposal to grant access to their team members. PIs of accepted projects will be given administrator rights for their project and may enable access for Co-Is by following these instructions.

Archive GUI components

Note that older observations may not have all of the features or attributes discussed here.

Open

Screenshot of archive listing with active projects.

Calibration reports and additional information

Clicking on the expander button will give you much more information about your observation.

Details

This section will give you further observation information such as the antennas used and scan listings.

Check that the scan sequence is as expected. Please refer to our article on Mislabelled scans in higher dump rate observations if your observations have been run at dump periods of 2 or 4 seconds.

Script log

This is the log of the actual observation script used to control the telescope. It is worth scrolling through this to check whether any antennas were lagging, or if windstows occurred.

The observation script currently does not have any capability to detect when antennas are in windstow, but antennas with no target lock will be flagged by the Control and Monitoring system (CAM). It is advised to always apply CAM flags. For more on flags please see this section.

Delay calibration observation

This is a short track on one of the primary calibrators, run immediately after a new array is configured and built (any change in band or correlator mode requires a rebuild). The derived delay calibrations are then applied to the data subsequently recorded data. In addition, the noise diode is fired and and H-V phase delays are calculated (but not applied). This calibration block and be used for polarisation calibration, or T_sys calibration.

Calibration reports

A detailed description of the calibration pipeline is available here.

Continuum image quality report

The image quality report is under active development. An initial presentation on the report is available here.

Spectral imager report

A brief description of the spectral pipeline is available here. Note that the spectral imager pipeline may be disabled at times due to operational reasons.

Continuum image and spectral cubes

The continuum imager will run for on-source integration times greater than 15 minutes. This limit is imposed in order to keep up in real time with the telescope schedule.

The spectral line imager will run for on-source integration times greater than 45 minutes. However, the spectral imager may occasionally be disabled due to operational reasons. Users relying on the spectral imager products should indicate so clearly in their proposals.

Download options

Measurement sets

Most users would want their data in the CASA Measurement Set Format (MS). MeerKAT data is stored in a native format known as MeerKAT Visibility Format (MVF) which is a distributed structure. The in-house package katdal (KAT data access layer) is used to retrieve the data and the script mvftoms.py is used to convert to MS format. Note that this process can take several days, depending on the size of the dataset and the load on our transfer servers. Once the file is ready, you will receive an email with a download link.

Since the last archive update of 9 March 2022, users will be taken directly to the configuration options for measurement set configuration. The estimated dataset size is indicated in the top left, next to the ‘Create Direct Download’ button.

Note that the MS file link automatically generated has a validity period of 60 days that is send directly to your registered credentials. Once the link-validity period has expired, then only one is able to re-request another set of MS parameters as part of the SARAO data life-cycle policy. So choose wisely before you can generate a dataset with new parameters.

Transferring to processing partners

SARAO has dedicated high speed links to IDIA and the CHPC. Should you have an account with one of these facilities, you can trigger a push of your data to them. Note that the dataset will first be converted to measurement set, so the same caveats as above apply.

Arrangements to reduce data at IDIA (Inter-university Institute for Data Intensive Astronomy) will have to be made independently. South African based PIs can apply to use IDIA. To find out more, go to the IDIA site . Information with regards to datasets that end up in IDIA, will have to be followed up with the IDIA Help Desk. For further IDIA access, related support and support for the ilifu Research Facility, please contact support@ilifu.ac.za.

Direct access using katdal

If you wish to work directly with a subset of the visibility data, you can use katdal directly. Clicking on the first icon in the listing will copy a link to the index file (rdb file) and a security token, which you can use in a python environment to manipulate the data in any way you wish.

Documentation on katdal can be found here.

LTO 7 tape

Large projects can arrange with SARAO to have their data shipped on tape, at their own expense.