ASP User Guide
(1) Introduction
This document describes how to use the ASP (Arecibo Signal
Processor) for observations with the Arecibo telescope. The
current system provides capability for real-time signal processing
of up to 64-MHz bandwidth that is hardware divided into 16 channels using
the ABPP (Arecibo-Berkeley Pulsar Processor).
Currently (July 2003) pulsar coherent dispersion removal and
synchronous averaging of Stokes parameters
is the most developed of the possible applications. The structure
of processing core is such that other applications can be readily
programmed and managed.
The basic architecture consists of the ABPP and the ASP PC cluster
that are linked via a small interface board for level conversion
and 1 PPS synchronization. The ABPP selects an IF source (Gregorian,
CH, Internal Noise), sets analog los, filters and levels, digitizes
and channelizes each analog signal to provide ASP with 16 channels
of 4bit complex, dual polarization data. The ASP consists of two
"Data Server" PCs that read in the data from the ABPP hardware and
pass it to "Slave" PCs. ASP has a master PC for interface with the
outside world. The monitor & control gui, ANUBIS, is run on the master
PC and is coordinated with telescope & receiver activity via scramnet.
Data products from the Slaves appears on the master's disks.
Telescope/Receiver--ABPP--ASP Configuration
ASP Layout
ABPP-ASP Installed 03may10
For GASP (Green Bank Astronomy Signal Processor) we will have
twice the bandwidth capability, 128 MHz, provided by analog-digital
"Midend" electronics (documentation coming). This will feed four
data servers with their EDT digital I/O cards. The four data
servers will feed a 48-port, 96-Gbs switch (
HP2848)
via channel bonded Gbs network interfaces. Further description
of slaves, data storage TBD.
(2) Before the observing run
The ASP can be run in standalone mode to familiarize yourself with
the gui control, the signal processing architecture, and the data
products. Both computer noise generation and disk data sets can
be used as inputs in place of telescope signals.
(3) ANUBIS - the ASP gui
Picture here. Then describe some of the buttons. Actual
use is in sections following. Need cookbook of manual
steps. Then cookbook of "telescope control" steps.
ANUBIS Screens
(4) IF source, IF level & LO issues
The ABPP can select one of three signals for its IF source: the
Gregorian receivers, the Carriage House 430-MHz receiver, or
an internal noise source. Select one of these paths on ANUBIS
gui; the ABPP PC must be available for actual application of
this selection via TTL control of IF Switch.
Once the ABPP IF is selected and the channel bandwidth selection
is made, commands must be sent to the ABPP PC to:
set the analog LOs, set the analog low pass filters, and
set up the digital filter boards that generate
the 8 channels for each Data Server with appropriate (digital) mixer
frequencies and low pass filter coefficients.
Once the ABPP analog modules and digital boards are setup AND
the telescope is on source and front end receiver is setup, THEN
the levels into the digital filter board A/D converters can
be set. See signal path in diagram
above.
(5) Taking data
Once the ABPP and ASP are setup, then click on "start observation".
At this point one can allow observing to continue, click on "stop
observation" which will allow current integration to finish and
then return ASP to standby state with data flowing into data
servers, OR click on "abort observation" which will terminate
the integration immediately and return ASP to standby state.
Is there any planned communication of "taking data" state from
ASP/ANUBIS back to telescope/receiver gui?
(6) Data Monitoring & Calibration
Follow the signal:
The attenuator settings from the ABPP Mixer/Filter module
provide an inverse measure of signal strength at the point where
Arecibo IF is passed to ABPP. Good readings are 15+/-5 dB; bad
readings are less than ~6 dB and near 30 dB.
"soft_scope" can be used to explore properties of the IF
signals as they pass through the Data Servers. Currently this
is invoked via ANUBIS gui. A text menu window provides options for
channel/polarization selection and histogram/spectrum/time series
display. An rms is provided and a good value is ~2.2; the 4b
data can range from -7 to +7. The spectrum passband can be corrected
by use of FIR filter coefficients; TBD. "soft_scope" has a vector
voltmeter to measure phase coherence/stability of injected tone; TBD.
The spectra of all channels is displayed at bottom of screen; TBD.
Use of standard calibration data; TBD. A daemon will inspect folded
profile data directory. When calibration Scans are available, this data
will be reduced in first order fashion and calibration to K and Jy
provided to realtime data displays (spectra and pulse profile).
Profile monitoring; TBD.
(7) After the observing run
(8) Routines for reading ASP data
(9) Software backend architecture
General
Shared Memory
MPI
Master
ANUBIS
Link to Telescope/Receiver Monitor & Control
Full Spectrum Display
Softscope
Monitor Window (Time/Date,Telescope,GPS,ABPP gain)
Data Server
edt2mem
startedt2mem,stopedt2mem,statedt,killedt
noise2mem
disk2mem
mem2disk
Slave Cores
spectrum
coherence?
coherent dispersion removal
detection
folding
(10) Glossary
SYSTEM
ASP: Arecibo signal processor
ANUBIS: Graphical User Interface for controlling the ASP
data collection software.
MASTER: (1) The Master node of the ASP cluster
(2) The software process running on the master node during
an observation, which controls other processes. It is always
given MPI rank 0.
DATA SERVER: (1) The nodes of the ASP cluster which
have EDT cards for reading ABPP data (2) Any node which
supplies raw data read from a file [typically we expect
these to be the same nodes as #2]. (3) The software running
on the Data Server nodes which supplies data.
SLAVE: Nodes on the ASP cluster dedicated to computational tasks.
COLLECTOR: Software (MPI process) which collects data from
Slave nodes, collates it, and writes output data files.
Runs on the master node.
SUBCOLLECTOR: Software running on Slave nodes which holds data
for assembly by the collector.
xxxx [what is this called?]: Software running on Slave nodes which
SOFT SCOPE: Software for real-time asynchronous monitoring of
passband and power levels. Called by ANUBIS button to collect
data from shared memory buffers in Data Servers.
Independent of the main MPI software analysis package.
XXXX[ need a name for this!]: the MPI-based software package
for control, distribution, and analysis of telescope voltage
measurements.
EDT2MEM: software for reading data from an EDT card and
storing in shared memory on a Data Server.
MEM2NET: software for reading data from shared memory
on a Data Server and transmitting in to a Slave.
HEADER
DATA AND FILE NOMENCLATURE
1. Session
-- one Session consists of many
observations (e.g., several hrs, many PSRs and/or RFs)
and is associated with the observatory assigned program code
and associated title, lead PI and actual observer.
Each Session has a unique ASP number starting with useage
in 2003. We need to experiment with this. It is useful
to have an identifier for a logical block of data that
might automate reduction.
2. Scan
-- a time-continuous observation of a given pulsar
at fixed receiver settings (apart from calibration).
The Scan has a unique number starting each MJD. For folded data
the results of a Scan are contained typically in a single disk file with
name of MJD.Scan.FrequencyFileNumber.TimeFileNumber; in fixed field
lengths the filename is mmmmm.ssss.f.ttt.
3. Dump
-- a Scan is divided into many Dumps. Each Dump can mean
a folded profile, or a single pulse, or an average spectrum, etc.
If we have a standard sequence of pulsed cal measurements and then source
measurements the cal Dump will be included with the source Dump in
the same Scan file. Implicitly all Dumps have same RF & Gain.
The Dump has a sequential number starting from 1 each observation.
A hierarchical data base is started as an ascii log of:
Session # -> info
Scan # -> info
Dump # -> info
ANUBIS displays Session #, Scan #, Dump #
3a. FrequencyFileNumber
-- this filename extension is at least
used to distinguish between data saved to disk from separate Data Servers.
One digit/character should suffice. When we save raw voltages in the
Data Servers, this index distinguishes otherwise identical filenames.
When we save raw voltages on the Slave nodes, then the FrequencyFileNumber
needs to go to two digits, or be in Hex.
3b. TimeFileNumber
-- this filename extension is useful for
large volume data saving to disk. The numbering is sequential starting
from 0 within each Scan. For example, a 3h Scan might be saved in 4
separate files. The proposed field extent is 3 digits.
4. Buffer
-- (what is code usage?) The EDT DIO card in the data
servers brings the raw voltage data from the hardware filter bank into the Ring
Buffers in memory which are defined by the EDT-supplied driver.
We currently use 1 MB which is 64kV for our 16B dual
polarization, complex, 4b voltage Vectors (V) and has
duration of 16 ms for 4-MHz bandwidth.
5. Chunk
-- the data block sent from a data
server to the Slaves is currently twice the length of
the FFT required for coherent dispersion removal.
There's slight question about optimization of data
transfers here. If we need to worry, then "Buffer" length
will be optimized for EDT/Ring Buffer DMA operation. And
"Chunk" length might need to be optimized for ethernet
transfer independent of its choice for FFT length.
Is there also an overlap "Chunklet" that might not be
half a Chunk in some applications. A later bridge...
Time is counted past start second via Buffer count which is
then transferred to Chunk count (the Chunk contains a SubChunk
worth of new data). Practically, the Chunk time is computed
and passed with each Chunk.
6. Sample (Rate)
-- a raw data Sample is a complex voltage vector of
(currently) length 16. This data is comes into the EDT via the ABPP-EDT
Interface at the Sample Rate.
(11) ASP Photo Guide
ABPP-ASP Interfaces
(03may17 - Power/Clock cable and EDT cable to Data Server 1 not installed)
Function: Grey cables from ABPP DFB boards provide 4 channels of data
with two samples each slow clock. There
are two cables per DFB and two DFBs are used for the two polarizations.
Total input bits per sample per Interface: 2 polarizations * 8 channels *
2 complex data at 4b, or 32B. The EDT interface is 16b, 2B. The fast clock
is used to walk through all channels each slow (sample) clock.
(12) ABPP Schematics for ASP
In the following figure the IF strip and the Mixer/Filter module
of the ABPP are shown. IF signals from: [position 1] the Carriage House
(430 MHz via upconverter to ABPP IF); [position 2] the Gregorian;
and [position 3] an internal noise source are brought to a
Mini-Circuits switch set by the ABPP PC via the Analog Interface
Board (AIB). The output is passed through a crude high/low pass
filter pair to limit any signals outside 250-500 MHz. The IF is
then divided four ways for the four Mixer/Filter modules.
The Mixer/Filter (M/F) modules begin with an programmable attenuator
(0-31.5 dB in 0.5 dB steps and moderate accuracy). These values are reported
in the ASP Monitor window. The goal is to produce 0 to +1 dBm signal into
DFB A/D converter. A power detector is read and compared to tabulated
data by the ABPP PC (setlevel command). The low pass filter in the
M/F module is set depending in the DFB channel bandwidth (*4 owing to
need for 8 channels and DSB operation of DFB).
ABPP Analog Section
Room here for further description of DFB. A schematic that shows
A/D, SRAM mixer, Harris decimating FIR filer, EPROM translator/encoder.
Also sketch of FIR coefficients and location of files that one might
use for a priori bandpass correction.
The ABPP resides in a two-rack unit that also houses the Caltech Baseband
Recorder (CBR). Formerly this also housed BACSPIN. Location of various crates
of electronics and their interconnections is shown below.
ABPP/CBR Rack Layout
Put in some here regarding ABPP PC and monitor & control description & files.
(13) Links:
FFTW
EDT
- select ->technical->documentation
and "Installation Instructions (hardware and software, Solaris, Linux and Windows" or
manual "PCI CD (High Speed DMA Interface) User's Guide" and addenda.
PGPLOT
3. tcl/tk [tclX etc]
4. Numerical Recipes
7. TEMPO