Data Acquisition System for the CALICE AHCAL Calorimeter

The AHCAL DAQ architecture adopted the aggregation hierarchy from the DAQ concept [3], but the hardware was developed from scratch. The aggregation factors are shown in figure 3. The following list describes the individual components in the DAQ chain:

SPIROC

is an ASIC developed by the Omega group [6], which has 36 input channels, an 0SCA (Switched Capacitor Array) for storing up to 16 events in analog form and an integrated 12-bit ADC. Charge (ADC) and Time (TDC) information is stored for each channel independently. Only signals passing a programmable threshold are stored.

HBU (AHCAL Base Unit)

is a $36\text{\times}36\text{\,}\mathrm{c}\mathrm{m}$ large board developed at DESY [7], which has 4 SPIROCs organized into 2 readout chains. HBUs can be connected to each other via flat connectors, forming a slab of 6 HBUs up to $2.2\text{\,}\mathrm{m}$ long as shown in figure 1 (left).

DIF (Detector InterFace)

is the first aggregation component, which controls and reads out up to 3 slabs, which corresponds to 18 HBUs or 72 ASICs. The data is sent out via HDMI connectors with a custom serial protocol. The latest version contains a Zynq-7020 SoC (System-on-Chip), an FPGA with an embedded dual-core ARM processor [8].

LDA (Link Data Aggregator)

reads out all DIFs from 2 wedges of the AHCAL barrel (figure 1 (left)) from 96 HDMI inputs and sends the data directly to a computer via GbE (Gigabit Ethernet). Details are shown in section 2.2.

CCC (Clock_and_Control_Card)

delivers a beam clock to the LDA and starts the acquisition cycle with the presence of a spill. It synchronizes the clock in common beam tests of different detectors. During beam tests it also delivers a delayed trigger validation signal.

The LDA hardware exists in two form factors, that have the same functionality and are interchangeable. The Mini-LDA (figure 4 (left)) is a custom FMC-LPC (FPGA Mezzanine Card - Low Pin Count) mezzanine with 10 HDMI connectors to connect to DIFs and 1 HDMI connector to connect to CCC. The mezzanine is attached to a Zedboard, a commercial development board [9] hosting a Zynq-7020 SoC [8]. The Wing-LDA (figure 1 (right)) has in total 96 micro-HDMI ports. The connector pitch matches the AHCAL layer spacing in the barrel and the overall length of the board is $110\text{\,}\mathrm{cm}$, as shown in figure 1 (left). It consists of 3 passive fan-out boards with 32 micro-HDMI connectors each, that connect to an active daughter-board. The daughter-board hosts 4 Kintex-7 FPGAs (xc7k160), each serving 24 ports from passive boards. Those FPGAs are connected to a central commercial Mars ZX3 SoC module [10], which hosts the same Zynq SoC as the Mini-LDA.

The firmware of the both LDAs, Mini-LDA and Wing-LDA, share similar source code, except for the Zynq $\leftrightarrow$ Kintex FPGA link, which splits the design into 2 FPGA partitions in the Wing-LDA form factor. The Wing-LDA FPGA firmware block diagram is shown in figure 5. The DIF sends 100-byte packet fragments to the “DIF PORT” module, where the ASIC packets are built for each SPIROC in the block memory in $40\text{\,}\mathrm{MHz}$ clock domain. Once a full ASIC packet is completed, the pointer is sent to the memory manager, which moves the data from the memory of all “DIF PORTS” to a $256\text{\,}\mathrm{KiB}$ large buffer, from where it is transported over the AXI-stream [11] infrastructure.

The “FPGA link” module has a circular buffer for up to 8 packets. A CRC-16 check-sum is added for each packet. Two bytes are encoded using the 8b10b code and serialized in $200\text{\,}\mathrm{MHz}$ clock domain in DDR mode, providing a dual $400\text{\,}\mathrm{Mbit}\text{\,}{\mathrm{s}}^{-1}$ link in each direction. The check-sum of received packets is checked and in case of error the packet is re-transmitted up to 3 times. Control words (transmit succeed or fail, buffer full) are interleaved in the stream via dedicated 8b10b control symbols. Fast commands from the CCC (start and stop acquisition, synchronization) are also transmitted as dedicated control symbols with highest priority, maintaining a consistent command propagation delay.

Packets from the $256\text{\,}\mathrm{KiB}$ AXI-stream FIFO are moved to the DDR memory using an AXI DMA (Direct Memory Access) core in a simple transaction mode (scatter-gather engine not used). Packet boundaries are omitted in the DMA transfer in order to minimize the overhead of initiating the transfer. The DMA is initiated using a custom written Linux kernel module, which makes received data accessible via a Linux character device. The processor part of the Zynq SoC is running a Petalinux Linux distribution [12] on both cores and hosts a simple TCP server, that streams all data from the character device.

The data volumes are shown at figure 3. The SPIROC produces $02.4\text{\,}\mathrm{kB}$ packets depending on how many of up to 16 events were stored in the acquisition phase. The LED light calibration fills all memory cells of all SPIROCs and uses the maximum of the data volume per readout cycle and defines the minimum required bandwidth for the DAQ, which is $17\text{\,}\mathrm{MB}$ per LDA per readout cycle. This corresponds to $85\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$ for the ILD geometry and accelerator repetition frequency of $5\text{\,}\mathrm{Hz}$.

Data from all SPIROCs has to be readout within $\sim 198\text{\,}\mathrm{ms}$ before the next acquisition cycle starts. For 6 HBUs in a slab, which corresponds to 24 SPIROCs in two shared readout chain buses, the minimum transfer speed is $1.2\text{\,}\mathrm{Mbit}\text{\,}{\mathrm{s}}^{-1}$. A $1.67\text{\,}\mathrm{MHz}$ readout speed is currently implemented. The SPIROC itself specifies a $5\text{\,}\mathrm{MHz}$ readout clock frequency limit. Similarly, a DIF has to send up to $180\text{\,}\mathrm{kB}$ data chunks with a speed at least $7.2\text{\,}\mathrm{Mbit}\text{\,}{\mathrm{s}}^{-1}$. The currently used serial protocol uses a $10\text{\,}\mathrm{MHz}$ clock, providing $8.42\text{\,}\mathrm{Mbit}\text{\,}{\mathrm{s}}^{-1}$ bandwidth.

The LDA needs to be able to supply continuously $85\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$ over a GbE port. This requirement is close to the Ethernet bandwidth limit during the LED calibration and might get replaced for the final detector in order to provide bandwidth overhead.

Since the LDA will be the device with the largest data traffic, it has been subjected to bandwidth benchmarking. The FPGAs internally process the data from the DIFs with $320\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$. The link between the FPGAs of a Wing-LDA is capable of transferring nearly $80\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$ for each slave FPGA and the further processing in the central FPGA has a $320\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$ limit. The speed of the DMA transfer to the processor memory depends on the size of the packets being moved by the DMA IP core and the size of the buffer used for reading from the character device, as shown in figure 6. The maximum transfer speed of $242\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$ is achieved when a $256\text{\,}\mathrm{kB}$ read buffer is used and small packets are merged into $256\text{\,}\mathrm{kB}$ chunks in the FPGA.

The software part in the SoC, which only sends the data from the memory to the TCP socket, does not yet reach the requested bandwidth. The maximum data transfer rate over the TCP was measured to be only $6\text{\,}\mathrm{MB}\text{\,}{\mathrm{s}}^{-1}$, one order of magnitude below the requirements.

It is not foreseen, that the current Linux TCP socket implementation will be used in the final detector installation. It will be replaced for the final detector. The overall TCP performance is enough for serving the beam tests with the currently available layers without hitting the bandwidth limit, but will need to be optimized for larger prototypes, especially when the spill rate will be higher than the $5\text{\,}\mathrm{Hz}$ expected at the ILC.