Introduction

MeerKAT supports a number of guest instruments/backends, which can be made available to general MeerKAT users subject to conditions which are outlined in relevant Calls for proposals.

In addition, some USEs operate commensally with all observations, and are enabled by default. These enable detections of rapid transients, and the Breakthrough Listen SETI program.

Open

Signal flow for the MeerKAT correlator beamformer and backend systems

PTUSE

The Pulsar Timing User-Supplied Equipment (PTUSE) allows observers to use beamformed data from the 64-dish MeerKAT telescope to acquire data in three different modes for radio pulsar studies. Users can obtain data in fold-mode, filterbank (or search) mode and baseband mode, and some combinations thereof. Observations with sub-arrays are also supported, as are observations from different frequency bands, and from up to four coherent beams within the primary beam. In addition to pulsars, any time-variable source can in principle be observed in one of the pulsar search modes, such as RRATs, magnetars, giant pulse emitters or repeating FRBs.

Sub-arrays

The 64-dish MeerKAT telescope correlator can form tied array beams from different numbers of dishes, with the most common used for pulsar applications being the maximum 64 or 2x32. Due to limited dish availability, typical numbers are between 59-64 dishes for the 64-dish mode and 29-32 dishes for the two-sub-array modes. In the current call for proposals, observers are encouraged to restrict their sub-arrays to either 2x32 dishes or 1x64 dishes. Users should keep in mind that if the source position is uncertain, it might be worth reducing the diameter of the array so that the source will appear well inside the width of the tied beam. MeerKAT observations of globular clusters often drop (zero-weight) the outlying antennas (> 1000m from the core) so that most of the pulsars appear in the tied beam.

Frequency Bands

The most common frequency bands for pulsar observation are L-band (856-1712 MHz) and UHF (544-1088 MHz). At L-band the top and bottom 48 channels (of 1024) are usually omitted from analysis due to band roll-off giving an effective bandwidth of 775.75 MHz. To avoid aliasing and inter-channel bleeding, the usual 1024 channel mode at L-band downweights the channels at their edges as described in Bailes et al. (2020).

Recently S-band has been commissioned, but routine calibration is not yet established. S-band has five potential bands from 1750-3500 MHz, each of which is 875 MHz wide.

Sensitivity

If the full array is available, the system temperature at L-band is approximately 18K and the gain near 2.8 K/Jy although this varies across the band. At UHF it is approximately 10% worse, but the sky background temperature is much higher, scaling as typically . Since pulsars are steep spectrum objects with typical spectral indices of -1 to -3, the UHF band provides excellent opportunities if the pulsar is not scattered.

Modes

Fold mode

This is the most often used and best tested mode. Every 8s pulsars are coherently dedispersed with a nominal DM, folded at the apparent topocentric period of the pulsar and the data binned with 1024 bins, 1024 frequency channels and four Stokes parameters. If a calibrator has been observed (strongly recommended), the data are polarisation calibrated automatically and to a high degree of accuracy at L-band and UHF frequencies. Absolute flux calibration is not possible without assuming the radiometer equation is applicable, and the radio frequency interference has been excised. Observers specify an integration time, and a single psrfits PSRCHIVE-compliant archive is produced in psrfits format. The first and last integration are unreliable and normally discarded as the time lag between PTUSE start and stop times is unreliable at the few second level. 

Observers need to supply the coordinates of the source and a tempo2-compliant ephemeris by email to SARAO, as well as the desired integration time and frequency resolution. These are currently held in a single repository, and coordination with MeerTime will be required if you want to change a folding ephemeris. 

Observations appropriately labelled will be automatically transferred to the OzSTAR supercomputer at Swinburne and cleaned for RFI by the Meerpipe pipeline if desired. Obtaining an account on OzSTAR will enable raw data transfer to your own computer - email Matthew Bailes to obtain an account. Clock correction files are maintained by Dr Ryan Shannon, and he can be emailed for the latest files that take typically 1 month to be updated with the clock correction files. MeerTime is routinely achieving sub-100 ns timing on PSR J1909-3744.

If there is more than 1 source in the primary beam, a tied beam can be placed on up to four pulsars that can be observed simultaneously. Due to hardware memory limitations, the maximum spacing between pulsars is only ~15 arc minutes at 20cm.

Some observers may wish to use more than 1024 frequency channels. Internally the PTUSE machines use dspsr to coherently dedisperse the data, and higher resolutions can be obtained. 8192 channels is reliable, 16384 is not recommended except at UHF.

Rarely, observers use the 4096 channel mode of the beamformer, in which case PTUSE defaults to 4096 channels at the expense of time resolution. There is interchannel leakage of power which can lead to artefacts in the pulse profiles.

Filterbank or Search Mode

The 4 PTUSE machines can each produce coherently dedispersed filterbank data that has 1024 frequency channels, an integer number of time samples added together and up to four Stokes parameters. It is routine to take filterbank data on one PTUSE machine and folded data on another. It is also possible to, in real time, coherently dedisperse the data and do “on-the-fly-fscrunching” to a smaller number of frequency channels to avoid data overload. This is useful when employing the 4096-channel mode from the beamformer. Extremely high DM/frequency^3 ratios may fail and should be tested before use. 

The search mode data all record 8 bits per Stokes parameter, and the data are integrated typically for an integer number of time samples at the native time resolution of the filterbank. At UHF the fastest time resolution is therefore in multiples of 1024/(544e6) ~ 1.8823 microseconds and at L-band multiples of 1024/(856e6) ~ 1.196 microseconds. However, attempts to record data at this rate will fail, because the disk drives can’t sustain the output data rate. The maximum data rate we recommend is 420 MB/s.

Open

The data rates for 2-bit data. The L-band 8-bit data rates are four times this.

Observers specify a desired time sampling rate, and this is rounded to the nearest integer number of samples.  It is not recommended to attempt sampling rates less than 9 s. Search mode at S-band is untested at present but would probably work if the time sampling is at least 10 s. Slower sampling rates should be routine.

Baseband Mode

The PTUSE machines can, for a limited time, record the tied beam baseband data directly to disk. Unlike the other modes, observers will quickly fill the disks if their observations last for too long. Once on disk the data can be read by tools like dspsr and used for a multitude of studies but the PTUSE machines are required for other experiments and the data will need to be transferred off soon after the observation ceases. The raw data rate at L-band is 856x4=3.424 GB/s and the absolute maximum time the observation can last is the capacity of the disk divided by this rate, or about 7300/3.424~2000 seconds. The top and bottom 48 channels can be omitted to extend the integration time.

At UHF the experiment could last about 3300s or almost 1 hour per PTUSE machine. Anyone wishing to use this mode should be prepared to liaise closely with SARAO and Swinburne staff to ensure there is a pathway to both record and move the data from the machine once it is recorded. It may be necessary to enlist the support of MPIfR and the University of Manchester who have LTO-9 tape drives on site. It would be necessary for the observers to supply LTO-9 tapes for data transfer. Unfortunately the same switch that connects the MPIfR tape robot is used by MeerKAT and transfers are limited to 50 MB/s. For a 7 TB observation this means almost two days to transfer the data. It may be possible to enlist the help of Swinburne to process the data on the PTUSE machines when observations are not making use of them, but the PTUSE machines are meant to remain robust data acquisition machines, not general user machines and their use for baseband processing is discouraged.

In principle an observer could use all 4 PTUSE machines in a round robin/sequential fashion to record baseband observations up to a few hours in length. The baseband data are divided into 8 second files that can be read by dspsr. Observers should be aware that the baseband data have artefacts imprinted upon them by the polyphase filterbank that produces the channelised data.

Radio Frequency Interference

The UHF band is usually remarkably clean, with the exception of a small region a few 10s of MHz wide. The 20cm band is punctuated by satellite transmissions, but often ~80% of it is good for science. Observers are encouraged to read Bailes et al. (2020) PASA Vol 37, 28B for a complete description of the instrument and its performance.

Acknowledgement statements

FBFUSE/APSUSE

The Filterbanking Beamformer User Supplied Equipment (FBFUSE) and Accelerated Pulsar Search User Supplied Equipment (APSUSE), developed by the Max-Planck-Institut für Radioastronomie (MPIfR), provide MeerKAT users the means to perform wide-field-of-view pulsar searches on coherently beamformed total power observations. The instruments consist of two GPU-accelerated computing clusters, with FBFUSE providing total power beamforming and APSUSE providing data recording and analysis.

Beamforming capabilities

FBFUSE supports up to 768 independently steerable beams. Users typically configure the beams as tilings of a given shape and size. Here a hexpack algorithm will attempt to fill a user-defined bounding box at a given overlap between beams (determined by the fractional gain response of the beam at some distance from its boresight). It is possible to configure both independently steered beams and multiple tilings together in a single observation. For detail on the beamforming and beam positioning methods used in FBFUSE we direct the reader to Chen et al. (2021).

Recording capabilities

When configured, the APSUSE instrument will attempt to record all beams produced by FBFUSE up to a maximum ingest rate of ~12 GB/s. Users may trade between beams, bandwidth and time resolution while remaining within this cap. The following configurations are currently supported:

Further configurations that trade between time and frequency resolution may be considered upon request. Data recorded by APSUSE are written to file in SigProc Filterbank format and are compatible with common pulsar processing software.

Processing capabilities

The APSUSE cluster consists of 60 processing nodes with 120 Nvidia GTX 1080Ti GPUs. A GPU-accelerated pulsar searching pipeline is deployed on the cluster consisting of:

• RFI mitigation

• Constant acceleration FFT-based pulsar search

• Spatial candidate filtering

• Candidate folding

• Candidate classification

Candidates are written as PSRCHIVE archives and tools to support offline candidate viewing are available from the TRAPUM and MMGPS collaborations. Details of the pipeline implementation may be found in Padmanabh et al (2023).

We currently do not support the deployment of custom searching pipelines on the APSUSE hardware. In any case, technical discussions with the MPIfR team should be undertaken prior to proposal submission.

Transferring data off-site

In the case of very small data volumes (several GBs) it is possible to perform network transfer of data products off site. This covers use cases such as the transfer of folded candidates. However, typical APSUSE data sets are 10-100s TB in size and as such network transfer is not possible. For proposers wishing to move large volumes of data off site, there is an LTO-9 tape archive local to the cluster that is used for data transfer. Proposers must ship their own tapes to SARAO should they wish to use this capability. For targets with known dispersion measures, a subbanding pipeline may be used to incoherently dedisperse the data and integrate it in frequency to a smaller (e.g. 32 - 256) number of channels before transfer.

Acknowledgment statements

TUSE

The Transient User Supplied Equipment (TUSE) is developed by the MeerTRAP team and the University of Manchester and is funded by the ERC. It provides MeerKAT users the means to perform wide-field-of-view fast transient searches on incoherently and coherently beamformed total power observations. TUSE is a GPU-accelerated computing cluster and it takes as input the FBFUSE beams. Operating TUSE requires the support of an experienced user. Proposers interested in using the instrument are required to consult the MeerTRAP lead Ben Stappers for a technical evaluation and must include at least one experienced user as a Co-I on their proposal.

Beamforming capabilities

TUSE receives beams as formed by FBFUSE. It supports processing of 1 incoherent beam and up to 768 independently steerable coherent beams. The default TUSE configuration (as given to FBFUSE) in commensal observations is to tile out a circular region centred on the pointing centre of the primary (incoherent) beam using a hexpack algorithm and the coherent beams are chosen to overlap at 25% of their maximum sensitivity. The incoherent beams are combined using all available telescopes and the coherent beams use the maximum number of available dishes from the innermost 44 dishes in the core of MeerKAT. We note that changing the overlap percentage and the number of dishes used to form the coherent beams is possible, relatively easily, however it requires that the set up be changed by hand at the time of the observation.

It is possible to configure both independently steered beams and multiple tilings together in a single observation. However, that will require that the specification of the beams be done by FBFUSE (called TRAPUM CA) and not TUSE and would need to be tested in advance. For detail on the beamforming and beam positioning methods used in FBFUSE we direct the reader to Chen et al. (2021).

Input capabilities

When configured, the TUSE instrument will process all beams produced by FBFUSE up to a maximum ingest rate of less than 3 Gbit/s/node. In the table below we show the standard setups for TUSE when it is operating in its full commensal mode.

Processing capabilities

The TUSE cluster consists of 65 processing nodes each with 2 Nvidia GTX 1080Ti GPUs. When operating with 768 coherent beams, these are distributed across 64 nodes and the incoherent beam is sent to a 65th node. Candidates are searched for in real-time using the AstroAccelerate GPU based software wrapped using the CHEETAH software and controlled by custom developed control software. The settings for the real-time processing are given in the Table below.

A lower DM cut-off is applied during candidate sifting and is set to exclude leaking zero-DM candidates.

RFI mitigation is done before passing the data to AstroAccelerate, this uses the IQRM approach combined with the so-called z-dot filter to remove zero-DM RFI. No fixed mask is used, however 32 channels are masked at the edge of each frequency band.

The real-time processing selects events that are above a S/N of 8 as measured by AstroAccelerate in the DM-t plane. These candidates are then sifted and clustered on a per beam basis in the usual way to minimise duplicates. For candidates that survive this process the total-power beam-formed data with the time resolution and number of channels of the data received from FBFUSE are saved to disk. These are saved in SIGPROC filterbank format with associated meta-data. The full-span of the dispersion measure sweep is saved plus 0.5s either side. The filterbank file is the original data, i.e. with no RFI cleaning applied.

These data are also passed to our Machine Learning code and given a label. If deemed as likely to be a transient then the candidates and meta-data and plots are also saved to an HDF5 file, with the filterbank and DM-T and Frequency-T planes used for the identification saved as well. If the candidate is not thought to be a transient then everything but the filterbank file are saved to the HDF5 file. This is our long term storage data product.

We note that diagnostic plots are formed which can be viewed and used to decide whether any of the candidate data should be kept.

Transient Buffer

FBFUSE has a ~53 second transient buffer which is capable of capturing ~0.3 ms of data around a transient event when triggered by a positive identification of a transient by the candidate classification algorithm on TUSE. At the time of writing this mode is working, but there are still a number of edge cases and sometimes triggers do not arrive within the 53 second buffer window. It can also be that there are too many triggers due to RFI. As this mode also has an enormous data rate and requires specialist post-processing, which is not automated, if you want this mode we require working with a MeerTRAP co-I.

The transient buffer data products would be correlated visibilities and a beamformed data set for a single beam can be prepared with the assistance of the MeerTRAP co-I.

Transferring data off-site

TUSE has limited storage for candidate data and so inspection of candidates should be made rapidly so that any data that is no longer needed can be deleted. This inspection can be done using the available diagnostic plots.

In the case of very small data volumes (several GBs, i.e. few hundred candidates) it is possible to perform network transfer of data products off site. For the HDF5 files we also archive those long term on an s3 Bucket maintained by SARAO, but which has limited access tools.

We note that moving the raw transient data off site is not possible as the data rate is too high and we do not have other storage options.

Acknowledgement and Citation

Please cite Rajwade et al. (2022) for TUSE/MeerTRAP and also cite the appropriate papers for FBFUSE as given in the FBFUSE documentation.

Known Issues

The TUSE processing pipeline can be badly affected by lightning. MeerKAT can see lightning out to large distances from the site, and this can lead to a large number of candidates that overwhelm the system and we may need to stop observing.

There is a gain variation issue with the digitisers of MeerKAT which mean that strong RFI (e.g. in the mobile phone bands) can lead to broadband features. The nature of these, and the spacing between them, mean that despite the quality of the zero-DM filtering, they can leak through as candidates.

We have cut-offs for the total number of candidates we can process in real time; this is quite a large number, but when RFI/lightning/bright known pulsars are present they can sometimes swamp the system.