Accessing and examining your data
This is a walkthrough of the items listed under your observation, and what to look out for.
Introduction
Please note that after six months, data is moved from spinning disk to tape.
Users are requested to download their data as soon as possible after observations have been done.
At the moment, the only data products that are restaged from tape are the flags and visibilities. No image products are restored. Users should be prepared to image the data themselves.
The SARAO archive can be found on https://archive.sarao.ac.za/.
Once you have logged in, you will be taken to a screen that lists all observations that you have access to. If you are not on any active proposals, you will only see non-proprietary data consisting of observations past their proprietary periods.
Note that the tape icon indicates that the data is no longer readily available (on spinning disk) but has been moved to long term tape storage. This will be true for any data older than six months, unless otherwise requested by the PI. A request to restage the data may be sent (by clicking on the tape icon) but since this is a manual process it may take up to a few weeks before the data is made available for download.
Alternatively, once you receive a notification from the Astronomer on Duty via the OPT that your observation is complete, you can follow the link to the archive to view the observation.
Getting access to datasets as a co-investigator
It is up to the PI of the proposal to grant access to their team members. PIs of accepted projects will be given administrator rights for their project and may enable access for Co-Is by following these instructions.
Archive GUI components
Note that older observations may not have all of the features or attributes discussed here.
Calibration reports and additional information
Clicking on the expander button will give you much more information about your observation.
Details
This section will give you further observation information such as the antennas used and scan listings.
Check that the scan sequence is as expected. Please refer to our article on https://skaafrica.atlassian.net/wiki/spaces/ESDKB/pages/1451950122 if your observations have been run at dump periods of 2 or 4 seconds.
Script log
This is the log of the actual observation script used to control the telescope. It is worth scrolling through this to check whether any antennas were lagging, or if windstows occurred.
The observation script currently does not have any capability to detect when antennas are in windstow, but antennas with no target lock will be flagged by the Control and Monitoring system (CAM). It is advised to always apply CAM flags. For more on flags please see this section.
Delay calibration observation
This is a short track on one of the primary calibrators, run immediately after a new array is configured and built (any change in band or correlator mode requires a rebuild). The derived delay calibrations are then applied to the data subsequently recorded data. In addition, the noise diode is fired and and H-V phase delays are calculated (but not applied). This calibration block and be used for polarisation calibration, or Tsys calibration.
Calibration reports
A detailed description of the calibration pipeline is available here.
Continuum image quality report
The image quality report is under active development. An initial presentation on the report is available here.
Spectral imager report
A brief description of the spectral pipeline is available here. Note that the spectral imager pipeline may be disabled at times due to operational reasons.
Continuum image and spectral cubes
The continuum imager will run for on-source integration times greater than 15 minutes. This limit is imposed in order to keep up in real time with the telescope schedule.
The spectral line imager will run for on-source integration times greater than 45 minutes. However, the spectral imager may occasionally be disabled due to operational reasons. Users relying on the spectral imager products should indicate so clearly in their proposals.
Download options
Measurement sets
Most users would want their data in the CASA Measurement Set Format (MS). MeerKAT data is stored in a native format known as MeerKAT Visibility Format (MVF) which is a distributed structure. The in-house package katdal (KAT data access layer) is used to retrieve the data and the script mvftoms.py is used to convert to MS format. Note that this process can take several days, depending on the size of the dataset and the load on our transfer servers. Once the file is ready, you will receive an email with a download link.
The download link for your measurement set is valid for a month. Please contact the SARAO helpdesk in advance should you need this period extended.
Since the last archive update of 9 March 2022, users will be taken directly to the configuration options for measurement set configuration. The estimated dataset size is indicated in the top left, next to the ‘Create Direct Download’ button.
Note that currently the old MS file will need to be cleared manually by the archive support team before you can generate a dataset with new parameters, so choose wisely. If you run into problems, please contact the helpdesk for assistance.
A note to observers of strong spectral lines: The SDP flagger is very good at finding narrow lines and flagging them, particularly Galactic HI and maser lines. In this case you should not apply the ingest_rfi flags. It is strongly advised that you keep cam and data_lost flags since these are due to system issues such as antennas losing target lock or correlator faults.
Transferring to processing partners
SARAO has dedicated high speed links to IDIA and the CHPC. Should you have an account with one of these facilities, you can trigger a push of your data to them. Note that the dataset will first be converted to measurement set, so the same caveats as above apply.
Arrangements to reduce data at IDIA (Inter-university Institute for Data Intensive Astronomy) will have to be made independently. South African based PIs can apply to use IDIA. To find out more, go to the IDIA site . Information with regards to datasets that end up in IDIA, will have to be followed up with the IDIA Help Desk. For further IDIA access, related support and support for the ilifu Research Facility, please contact support@ilifu.ac.za.
Direct access using katdal
If you wish to work directly with a subset of the visibility data, you can use katdal directly. Clicking on the first icon in the listing will copy a link to the index file (rdb file) and a security token, which you can use in a python environment to manipulate the data in any way you wish.
Documentation on katdal can be found here.
LTO 7 tape
Large projects can arrange with SARAO to have their data shipped on tape, at their own expense.