Design, construction and operation of the ProtoDUNE-SP Liquid Argon TPC

The cryostat is a free-standing, electrically isolated structure assembled in the EHN1 hall at the CERN Prévessin site. The room-temperature outer structure provides the structural integrity of the entire setup, being capable of withstanding the hydrostatic pressure of the LAr, the pressure of the argon vapor, the detector weight, and gravitational and seismic forces. The overall nominal outer dimensions are: $11.4~\mathrm{m}\,\times\,11.4~\mathrm{m}\,\times\,10.8~\mathrm{m}$ (W$\times$L$\times$H) and the nominal inner cryostat dimensions are: $8.5~\mathrm{m}\,\times\,8.5~\mathrm{m}\,\times\,7.9~\mathrm{m}$.

These dimensions are dictated by several constraints: the required active volumes of LAr and ullage, the layout of the penetrations, installation needs, distances from the active volume to the cryostat inner walls, and insulation thicknesses.

The outer structure consists of an assembly of prefabricated S460ML [8] steel modules of three configurations: standard structural modules used to construct the walls and floor, corner elements and a web of metallic frames to interconnect the modules. The complete structure and the modules used to assemble each face of the cryostat are shown in Figure 3. The modules are assembled using standard hot-rolled profiles welded together with bolted connections.

Three of the four cryostat outer walls are identical. The front wall, through which the beam enters, has a beam window (Section 4.1.3) and a separate wall section that is sealed in place over a Temporary Construction Opening (TCO), a 7.3 m high and 1.2 m wide entrance used for installing detector components.

The inner vessel design, installed inside the warm support structure, is based on the LNG transport membrane technology developed by the firm GTT (Gaztransport & Technigaz) [9]. The overall thickness of the inner vessel is 800 mm. The design fulfills the requirement that thermal fluxes not exceed 5-6 W/m^{2 22}2This value also accounts for the aging properties of the insulation material (20 years of operation).. The outermost layer of the inner vessel is a 10 mm thick stainless steel membrane  [10] (the tertiary membrane) that provides an effective gas enclosure and allows for control of the nitrogen atmosphere inside the insulation volume, which prevents condensation in the insulation layer.

Moving inward, just inside that membrane is the insulation layer that consists of two 390 mm thick layers of a foam specially developed for this purpose, supported by plywood plates of 9 mm and 12 mm thickness. The foam material is expanded polyurethane of density 90 kg/m³. The outer insulation layer is attached to the tertiary membrane by a set of rods and special mastic. Between the two insulation layers is a secondary LAr containment layer made of GTT-proprietary Triplex, a composite material consisting of a thin sheet of aluminum laminated on both sides by a layer of glass cloth and resin. In contact with the innermost side of the foam insulation is the 304L stainless steel 1.2 mm thick primary membrane that holds the LAr. The primary membrane can expand or shrink in two dimensions as a function of the temperature, thanks to a special corrugation configuration that allows it to respond like a bi-dimensional spring. A cross sectional view of the insulation and membrane layers is shown in Figure 4(a). Figure 4(b) shows a view of the interior of the ProtoDUNE-SP cryostat before the insertion of the detector modules and the closure of the TCO. The corrugations of the primary membrane are visible.

Once all the large detector components have been installed inside the cryostat, the TCO is closed. The sealing is done in two steps since the entire length of the TCO is divided along the horizontal axis, into two parts. First the top part is bolted and sealed with the standard outer structure section. The layers of insulation are installed from inside the cryostat, following the standard sequence presented above. Once the primary membrane of the top part of the TCO is also installed, all the material needed for the bottom part is brought inside the cryostat and the sequence done for the top part is repeated. Since the sealing of the TCO involves welding activities inside the inner volume, a temporary “dirty room” was built to protect the detector components during those operations.

Figure 5 shows the TCO. The circular opening for the beam entrance is also visible.

The penetrations into the cryostat are installed on the roof of the warm structure, with a couple of exceptions. To keep the high level of purity required, LAr is extracted from a point as low as possible in the cryostat and pushed by cryo-pumps to the external filtering system through the liquid recirculation circuit (see Section 3.3). A special penetration is thus required low on one of the cryostat’s side walls to connect to the LAr pumps. This penetration has a dedicated system of safety valves and involves local modifications to the insulation panels and the primary membrane.

Since ProtoDUNE-SP was designed for exposure to a charged particle beam from CERN’s SPS, a second modification to the cryostat was made to accommodate a beam window. The window minimises energy loss and scattering, which would be much higher if the beam were to pass through all the cryostat layers (see Section 4.1.3). The installation of the beam entrance window does not require any particular modification of the warm support structure. The tertiary membrane has been perforated with a hole where the beam entrance window is installed. The beam entrance window is composed of 175 $\mu m$-thick Mylar foil and a gate valve that opens when the beam is present and closes for safety reasons when the beam is off. The insulation between the secondary and the tertiary membrane is removed in the region around the beam window and the foam between the secondary and the primary membrane is replaced with a lower density foam (9 kg/m³). Finally the plywood supporting the primary membrane in the vicinity of the beam window penetration is replaced with a Nomex honeycomb plate sandwiched between thin G10 layers. Nomex is a polymer material with high thermal resistance and Nomex sandwiches are well known for their structural resistance . Thermal and stress analyses to validate the design were conducted in collaboration with GTT. The total amount of material, including the (unaffected) primary membrane, given a 0.3 mm G10 thickness and a 0.3-mm-thick steel beam window, is equivalent to 10% of a radiation length.

The other penetrations, all on the cryostat roof, are grouped by position, diameter, and function. They provide feedthroughs for the TPC detector power and signals, support for the argon instrumentation devices, and feedthroughs for the instrumentation signals. Of the 55 penetrations, 43 go through to the liquid of which seven are dedicated to the cryogenics system, nine to the detector support system, six to detector charge and light readout, and one to the cathode HV feedthrough. One is for the beamplug services, one for the diffuser fibre, 11 for the monitoring system (cameras, temperature and pressure sensors, purity monitors). There are also two manholes and five spares. The remaining 12 penetrations are limited to the insulation volume and are dedicated to either input/output of nitrogen gas or temperature and pressure sensors.

Figure 6 shows steps in the assembly sequence of the outer support structure. Installation starts with a planarity survey of the floor and the positioning of the elastomer pads and G10 strips (for seismic protection and electrical isolation) on which the cryostat will be placed. Three pre-assembled large wall pieces are positioned horizontally on temporary concrete supports, then three web interlink modules are positioned between them and connected together as shown in Figure 3 (right). The stainless-steel plates of the outermost (tertiary) membrane are then welded together and a helium leak test is performed to qualify the welds (see Section 3.2.1). To ensure the planarity of each completed wall, a detailed survey is carried out before it is lifted and installed in its final position on the elastometer pads. Wherever needed, temporary stabilisers are added to the experimental pit concrete walls to ensure safety during this process. Once the four walls are in place, the four pre-assembled cryostat corners are installed and the stabilisers are removed. Finally, the roof modules are assembled and the roof is installed. A final helium check of the SS plate welding is performed, and a global three-dimensional scan is done to verify the internal dimensions.

Once the outer structure and the tertiary membrane are installed, the pre-cut inner layer sections that come pre-assembled with mastic and fasteners are installed. After each layer is completed, all voids are filled to stop any circulation paths. At the end, helium leak tests are performed once again (see Section 3.2.1). The installation proceeds by layers (see Figure 4(a)) with the inner membrane installed last. Installation of the ProtoDUNE-SP warm structure took about 12 weeks.

Leak testing is performed on the external warm structure, the inner membrane, and all the penetrations. The tests are generally done using helium and a leak detector in sniffing mode. A deviation from the environmental background ($\sim 2-3\times 10^{-6}$ mbar l/s) is considered a possible leak.

The walls of the cryostat outer structure (see Section 3.1.1), despite their separation from the LAr volume, are required to be leak-tight for two main reasons: $(i)$ they are the last barrier between the LAr volume and the external world; and $(ii)$ any leaks on the outer surface would be sources of localised heat and humidity injection. The latter would produce water that would freeze inside the insulation volume, weakening it and making it less effective. The checks were performed during the assembly of ProtoDUNE-SP and no leaks were found.

The cryostat inner membrane (see Section 3.1.2) was tested twice for leaks. The first test, upon completion of the membrane installation, was the official qualification for leak-tightness by the vendor, and was performed on all welding seams. In this test, the insulation volume, normally flushed with N₂ gas, undergoes a few cycles of evacuation (down to roughly 550 mbar), then is filled with helium to match the external pressure. This allows a uniform fill of the insulation with helium. Two test holes in the lower inner membrane are intentionally not welded at this stage, but simply closed with mastic to allow occasional checks for the continued presence of helium in the volume under test. The few small leaks found were swiftly repaired. These test conditions also allowed for continuous monitoring of the insulation space internal pressure stability (Primary Barrier Global Test).

The second leak test involved pulling vacuum around localised sections of welding lines and testing them one by one. The test measures the leak rate from the helium-rich atmosphere, through the welds of the primary membrane, to a local volume evacuated by means of a vacuum box (Figure 7). The vacuum boxes come in a variety of shapes to fit corrugations, corners, and flat welded regions. Overall, about 80% of the total length of welding lines on the inner membrane was tested and no leaks were found. It is worth mentioning that a similar double-test procedure was performed on the ProtoDUNE-DP cryostat’s inner membrane and yielded the same result. This gives confidence that the helium leak-check performed by the vendor can be fully trusted when it comes time to verify the cryostat inner membranes for the far detector, and that performing the second test will be unnecessary. Retesting single welded lines on a DUNE-scale cryostat would require months.

The leak-tightness of all feedthrough flanges was verified. This was done to avoid the potential formation of argon gas (GAr) pockets on the cryostat roof, a working area during operation, and also to prevent air leaks that could pollute the LAr volume. Each penetration was tested after closing the corresponding flanged connections.

The test procedure changed slightly depending on the stage of the detector preparation: the helium was injected into the penetration under test either from the inside (cryostat volume still accessible) or using the argon gas exhaust lines present in every penetration (detector installation completed).

All flanges except one were certified to be leak-tight; the chimney of one temperature profiler (see Section 4.5.2) was found to be faulty. None of the actions taken fixed the leak entirely, therefore an enclosure was constructed and installed around the leaky flange, defining a buffer volume in which argon gas circulated continuously.

The cryostat inner and outer structures must satisfy European regulations and standards, as the ProtoDUNE-SP cryostat was installed and operated in Europe. In addition, since the design and the construction methods and technologies are prototypes for the larger DUNE cryostats, compliance with U.S. standards and regulations is also required.

Rigorous quality assurance and control procedures were carried out on all materials and techniques used for the construction of the ProtoDUNE-SP cryostat. The cryostat outer structure is instrumented with strain gauges and displacement sensors for making the required measurements. To fully validate the mechanical performance of the cryostat, a careful stress analysis of each structural component was done, together with pressure tests before and after filling with LAr.

Two Finite Element Analysis (FEA) models were developed to evaluate the cryostat design against the required codes and regulations. Predictions from the models were compared to the experimental data collected during the commissioning phase and operation. A relatively simple beam model includes simulations only of the cryostat outer structure and is used to study the moments and forces acting on the cryostat. This model was developed for the relative ease of obtaining reliable verification studies. The shell model is a more complex simulation that includes the entire inner vessel, which contributes to the overall stiffness of the structure. This model was developed to provide detailed predictions of structural deformations as well as stresses on the frame and outer membrane. Its results are compared against the measured stresses to validate the cryostat behaviour for the sizing case (i.e., the cryostat full of LAr and at 350 mbarg over-pressure). It can predict strain and deformation during the three main phases: commissioning, filling with LAr, and operation. Figure 8 illustrates a possible deformation during standard operation. Note that the image vastly exaggerates the actual deformation.

Strain gauges and displacement sensors are installed at the positions where the simulation predicts the highest stresses. All sides of the cryostat are instrumented in similar locations to check the symmetrical behaviour of the structure.

Four displacement sensors, commonly known as Linear Variable Differential Transformers (LVDTs), are installed to monitor the symmetry and magnitude of the cryostat’s global deformation. A total of 55 strain gauges (SGs) are used to measure local deformation of the structure under stress. A detailed description of the instrumentation can be found in [11].

The sensors are read out continuously during cryogenics commissioning, in order to $(i)$ check whether the maximum stresses and strains are exceeded in the most loaded areas and whether the structure demonstrates elastic behaviour; $(ii)$ monitor the maximum deformations and their symmetry; $(iii)$ serve as a secondary safety indicator of abnormal cryostat behaviour during commissioning.

The ProtoDUNE-SP monitoring system is complemented by a pressure gauge to measure the pressure difference between the cryostat and the local atmospheric pressure, and four temperature sensors to monitor variations in the pit where the cryostat is installed and on the surface of its outer structure.

Pressure tests validate the safety of the cryostat under pressure so that work activities can proceed in and around it, and they provide a final leak check on the cryostat roof penetrations. The success of these tests is a fundamental prerequisite for the cryogenics safety permit approval, which is required for the start of the cryogenics commissioning.

In a first pressure test, with the cryostat still empty, the over-pressure is increased to 200 mbarg, then decreased in steps of 50 mbarg. The maximum value is that which allows testing of the overall structure while keeping the most stressed areas within the elastic range. This value must also be far below safety valve maximum pressure, which is set at 350 mbarg. A second pressure test is performed at the end of the cryogenics commissioning, before starting operations, with the cryostat 96% full of LAr. The cryostat pressure is increased up to an over-pressure of 280 mbarg, again in steps of 50 mbarg. When performed on ProtoDUNE-SP each test lasted a few hours and the maximal over-pressure remained stable for about an hour.

All tests performed during the validation campaign were successful and the FEA model reproduced the cryostat behaviour in all cases. The pressure tests demonstrated the elastic behaviour of the cryostat structure: strain gauges showed a clear linear relation with the pressure and returned to zero at the end of the cycle (see Figure 9). Identically instrumented locations on the different walls showed symmetric readings, with the exception of the wall with the closed TCO, where differences are expected.

The values of the displacement and strain gauges measured during the fill were compared to the FEA predictions and found to fulfill the safety requirements. This comparison also validated the FEA model itself, which was then used to predict the cryostat behaviour at the sizing case. The outcome of the simulation was compatible with both EU and U.S. standards, thus providing the final qualification of the inner and outer cryostat mechanical structures.

The Process Flow Diagram for the ProtoDUNE cryogenics system is shown in Figure 10. The cryogenics infrastructure is divided into three main parts: the external cryogenics located outside the building housing the detector, the proximity cryogenics located next to the cryostat, and the internal cryogenics located inside the cryostat. The external cryogenics facility includes the systems used for the receipt and storage of the cryogens used in the cryogenics system. Designed to serve both ProtoDUNE detectors, separate lines take LAr, GAr, and LN₂ to both installations. A 50 m³ (70 t LAr capacity) vertical dewar was used to receive the LAr deliveries for the ProtoDUNE-SP initial filling period. The specifications for the oxygen, water, and nitrogen contamination in the delivered argon are 2, 1, and 2 ppm, respectively. An analyzer rack near the dewar was used to check the levels of these impurities in the delivered LAr batches and a 55 kW vaporizer was used to deliver the gaseous argon to the cryostat. LN₂ deliveries were received and stored in a second 50 m³ vertical dewar (40 t LN₂ capacity); the LN₂ is used in cool-down and normal operations, as explained below. A $0.5\,\mu$m mechanical filter is located on the LAr feed line to prevent any impurities in the LAr supply from entering the purification system and the cryostat. Since a GAr/H₂ mixture (2% H₂) is used to regenerate the LAr and GAr purification filters, a separate 10 m³ storage dewar dedicated to this function connects to a cylinder of hydrogen (H₂) and a GAr line.

To fulfill the LAr purity requirements, the LAr and the liquid boil-off needs to be collected, purified, and reintroduced into the system. The proximity cryogenics takes care of all actions during the recirculation of both in liquid and gas phase. The system comprises the argon condenser, the purification system for the LAr and GAr, the LAr circulation pumps, and the LAr/LN₂ phase separators.

The recirculation done at the liquid phase represents the major flow. During normal operations, the continuous re-circulation rate was 7.0 t/h, giving a full volume turnover time of 4.6 days, which kept the impurity concentration below $20$ ppt oxygen equivalent. The LAr is transferred to the liquid purification system via external pumps located on one side of the cryostat more than 5 m below the liquid level to avoid cavitation or vapor-entrapment. This purification system further reduces the initial impurity levels. The system has two pumps available to ensure continuous operation during maintenance. Only one pump is in service at a time. The purification system consists of three filter vessels; the first contains molecular sieve (4 Å) to remove water, and the others contain alumina porous granules covered by highly active metallic copper for catalytic removal of O₂ by Cu oxidization. In addition, $15\,\mu$m mechanical filters are installed at the exit of the chemical filters. Saturation of the chemical filter occurs when the trapped/reacted impurity budget exceeds the removal capacity of the filter material. At this point the LAr flow is brought directly to the mechanical filter and the regeneration of the saturated chemical filter can start. Filter regeneration typically takes two days, after which the system returns to nominal operation.

The second main flow is given by the recirculation of the gas boil-off. The LAr in the cryostat continuously evaporates at a slow rate due to unavoidable heat ingress. Therefore the boil-off is collected, condensed, purified, and reintroduced into the system. The main part of the argon vapour, roughly corresponding to 75% of the total boil-off, is collected via a dedicated penetration in the cryostat roof. The vapor is then directed towards the argon condenser, which is a heat exchanger that uses the vaporization of LN₂ to provide the cooling power to condense the GAr. The newly condensed argon is then injected into the main liquid flow where it follows the liquid purification path carried out by means of the cold filters. About 15% of the argon boil-off is removed through purge pipes connected to each penetration on the roof. This portion of gas is driven towards a purification system at RT that is composed of chemical filters similar to those used for the liquid. The design has a diaphragm pump to raise the pressure of the gas and a pressure-control valve to continuously monitor the gas flow to the condenser and maintain the pressure within the cryostat. The standard operating pressure is 1050 mbar (absolute); on occasion the pressure was regulated on gauge from a minimum of 10 mbar above ambient to a maximum of 250 mbar. Under standard conditions the pressure regulation remains better than $\pm 1$ mbar.

The diaphragm pump failed at some point during operations. While this pump was bypassed, the flow of the argon gas was instead controlled by adjusting the pressure. This demonstrated that the diaphragm pump was not essential and highlights the importance of designing flexibilty into the cryogenics system.

The internal cryogenics encompasses all the cryogenics equipment located inside the cryostat, including manifolds for the cool-down operation and for the distribution of the liquid and gaseous argon. The pipes for LAr distribution are positioned on the bottom of the cryostat and the outlets are at the end of the pipes, roughly opposite to the side penetration from which the purification system extracts the LAr (see Figure 11). The LAr entering the tank is warmer than the ambient by 0.4 K. This is a consequence of the isentropic efficiency of the LAr pump and the total heat load going to the transfer lines and valve boxes involved in the LAr circulation and purification processes. This temperature difference is crucial in that it induces an upward flow of the newly purified argon from its entry point at the bottom of the cryostat.

Once the cryostat construction and the installation of all scientific equipment is complete, the cryostat is cleaned (dust removal) in preparation for cool-down and fill. The first step is the purge in open loop (also called piston-purge) in which the atmosphere in the cryostat is replaced with argon gas. To ensure the purity of the input argon, the argon pipings are isolated, evacuated to less than 0.1 mbar absolute pressure, and back-filled with high-purity argon gas. This cycle is repeated several times to reduce contamination levels in the piping to the ppm level. The argon gas is then injected into the cryostat through a set of pipes at the bottom of the cryostat. The flow nozzles are directed downward to spread the gas across the bottom of the tank and produce a stable, upwardly advancing argon wave front. The exhaust is removed from the top using the main GAr outlet and vented outside of the building. A control valve regulates the pressure in the cryostat during this entire operation. The side purge lines located on each roof penetration are also used to evacuate the exhaust gas, ensuring that all volumes (especially trapped volumes) are properly purged. The vertical flow velocity of the advancing GAr is set to 1.2 m/h, which is twice the diffusion rate of the air downward. This causes the advancing pure argon-gas wave front to displace the air rather than just dilute it. After 44 hours (seven volume changes), the purge process is complete with residual air reduced to a few ppm.

The second stage of the purge process is done in “closed loop" for one week. During this stage the GAr is recirculated through the GAr purifier and sent back to the bottom of the cryostat. The gas purification system further removed the water-vapor outgassing from the FR4 circuit-board material and the plastic-jacketed power and signal cables present inside the cryostat. This closed-loop purge process further reduced the oxygen and water contamination inside the cryostat to sub-ppm and ppm-levels, respectively, at which point the cool-down could start. It also allowed assessment of the residual leak rate to atmosphere and the nitrogen content before cool-down.

The cool-down of the cryostat and detector is performed by flowing LAr and GAr into the cryostat through a manifold located near the top. Sprayers deliver a mix of LAr and GAr in atomised form that is distributed inside the cryostat by another set of GAr-only sprayers. During this operation the gas is exhausted using the pressure control valve, keeping the cryostat pressure at 70 mbarg. The sprayers guarantee a flat profile of the fluid (LAr and GAr) coming out, so as to meet the cool-down requirements of the cryostat and in particular those of the detector, which are more stringent. The maximum cool-down rate for the TPC is 40 K/h with a maximum temperature difference between any two points in the detector volume of $\sim$50 K.

Once the cryostat and the TPC are cold, the fill starts. LAr is transferred through the cryostat-filling pipework from the 70 t LAr storage tank via the filtration system for purification and the LAr phase separator, which allow for the injection in the cryostat of argon in both liquid and gas phases. The ProtoDUNE-SP filling process took about six weeks. Figure 12 shows the liquid level, pressure, and temperature trends observed during the filling of the cryostat.

Once the cryostat is filled, the system can enter steady-state operation. During operation, as shown in Figure 10, the boil-off that passes through the purge pipes on the signal feedthroughs is filtered directly at room temperature and then condensed with the main boil off. Before being reintroduced in the cryostat as liquid, it is purified and mixed with the bulk of the LAr coming from the cryostat.

Table 1 lists the final parameters of the ProtoDUNE-SP cryogenics, Figure 13 shows the stability of the cryostat pressure against ambient pressure variations during the beam run period (October - November 2018). Table 2 shows the heat leak balance of the cryogenics system as measured during steady-state operations after the cryogenics cold commissioning.

At the end of operations the tank will be emptied and the LAr will be returned to the storage tank outside the building, and from there unloaded back to LAr trucks.

The pressure inside the cryostat is maintained within a very narrow range by a set of active controls. Pressure-control valves can increase or decrease the cooling power in the condenser by controlling the amount of LN₂ flowing to the heat exchanger and the amount of GN₂ that is vented. Other pressure-control valves can be used to vent GAr to atmosphere and/or introduce clean GAr from the storage, as needed.

During normal operation the pressure-control valves are set to control the internal cryostat pressure to $1050\pm 1$ mbar absolute. Excursions of a few percent from this value set off warnings to alert the operator to intervene, but more severe deviations (i.e., pressure exceeding 200 mbarg or going below 30 mbarg) trigger automatic actions: the system isolates the cryostat from any potential source of pressure, first closing the valves connecting the cryostat to the circuits used for gas make-up, purge, or cool-down, and the valve to the phase separator. Then the LAr circulation pump is stopped and the side penetration valves are closed.

If the pressure is too high, the system increases the LN₂ flow through the heat exchanger inside the condenser and powers down any heat sources within the cryostat (e.g., detector electronics). At this point some of the GAr is vented to reduce the pressure in a controlled way. On the other hand if the pressure is too low, fresh GAr can be introduced into the cryostat through the GAr make-up line, a line dedicated to taking in argon directly from the outside supply.

The ability of the control system to maintain a set pressure is dependent on the size of pressure fluctuations (due to changes in flow, heat load, temperature, atmospheric pressure, etc.) and the volume of gas in the system. During normal operation ProtoDUNE-SP has 0.45 m of gas ullage at the top of the cryostat. This is 5% of the total argon volume and it is the typical vapour fraction used for cryogenic storage vessels. Reaction times to changes in the heat load are slow, typically on the order of one hour.

The cryostat is equipped with an additional high-integrity, mechanical, fail-safe over-pressure and under-pressure protection system capable of preventing catastrophic structural failure of the cryostat in any condition. The system is composed of a diverter valve connected to a set of two devices, combination pressure and vacuum safety valves (PSV/VSVs), located on the cryostat roof. Functions for both over- and under-pressure are combined in a single device for efficiency. One of the two identical devices in the system is installed in service mode and the other in stand-by, to guarantee that one is always active in case, for example, maintenance is required.

The device in service monitors the differential pressure between the inside and the outside of the cryostat and opens rapidly when the differential pressure is outside a preset range. In case of excessive pressure, the PSV/VSV opens and argon is released. The pressure within the cryostat falls and argon gas discharges into the argon vent riser. The valve is designed to close when the pressure returns below the preset level. In case the pressure detected is too low, the PSV/VSV opens and allows air to enter the cryostat to restore a safe positive pressure.

The cryogenics control system is a fully automated system that enables continuous cryogenic operation through the various modes of operation described in Section 3.3.2, while allowing manual actions by the operator as needed. It is a modular system, both in terms of the electrical and control hardware and the process control programming.

This system’s combination of automated and manual operation allowed its implementation at an early stage of the project, i.e., before the project inputs were fully defined, and enabled adjustments as the process and instrumentation evolved. It also made it possible to commission the cryogenics system in batches, i.e., to operate in one mode while implementing the control programming for the next mode.

The electrical and control architecture is based on a functional analysis of the cryogenics system and the subsequent Product Breakdown Structure (PBS), Piping and Instrumentation Diagram (P&ID), parts list (detailing the characteristics of the instrumentation), and process logics specification. The hardware architecture consists of three different hardware installations, each controlled by a dedicated Programmable Logic Controller (PLC) Siemens s317, remote input/output modules (I/O), and control cabinets. Each hardware installation corresponds to one of the three main cryogenics subsystems, namely, the external infrastructure for the storage and the proximity system for the two ProtoDUNE cryostats. Each control cabinet is dedicated to a specific function, e.g., LAr/nitrogen management, purification, phase separation, condensing, argon circulation. Each cabinet includes industrial cryogenics instrumentation: actuators (valves, pumps) and sensors. The infrastructure for the ProtoDUNE-SP hardware installation includes a total of 10 control cabinets and 630 signals.

The control system software was developed using the UNIfied Industrial COntrol System (UNICOS) [14]. The software architecture relies on a Process Control Object (PCO) breakdown structure. The process logic specification used as basis for the programming was designed with a modular structure and relies on associated option modes, sequencers, and interlocks. The option modes of the master PCO correspond to the sequence of operations of the cryogenics system, namely, default (which covers the fall-back situation with the cryostat pressure protection always active), open-loop purge, closed-loop purge, cool-down, fill, steady-state, and emptying. These option modes are used either to switch on a set of dependant units fulfilling the function for a given stage of operation or to set a collection of states of the controlled objects, e.g., fixed position or regulation mode for a valve, or a set-point.

In case of abnormal behaviour, software interlocks prevent any further automatic actions. The specifications and programming were completed and implemented sequentially, in batches corresponding to the operation modes. This allowed starting safely with an incomplete system, continuing to program while the system was already in operation, and adjusting the program as needed during commissioning.

The system is equipped with several features to ensure safe and continuous cryogenics operation, maintainability, and to provide flexibility for system evolution. For example, all equipment essential to the functioning of the system is powered from redundant electrical power supplies, including Uninterruptable Power Supplies (UPS) and diesel generators. Vacuum enclosures placed in the cryostat pit are constantly monitored. A degraded vacuum could signal a potential argon leak, therefore an alarm would be raised to indicate this condition.

Quality assurance and quality control were performed during the design, construction, installation and commissioning phases. During the construction and installation phases, non-destructive tests (X-rays, He leak tests and pressure tests) were successfully executed. During cold commissioning another series of tests was completed successfully, the main two of which were a check of the I/O signals and a functional test of all valves and equipment. The I/O check consisted of verifying all connections and the synchronisation of the 630 signals, testing the response to actions like opening/closing valves and to sensor readings, and verifying the sensor calibration or run calibration sequence for each control valve. In addition, tests were run on the Ethernet or hardwired signal lines dedicated to the exchange of information between the cryogenics system and the detector system, the safety system, and the CERN Central Control room.

A ‘mirror’ station (not connected to the actual field equipment) provided a functional test environment for the process control logic. The tests were subsequently carried out on the real system to verify first alarms, interlocks, and the sequential function charts. Operation modes were tested afterwards, during the system commissioning.

The CPA is located in the middle of the TPC, dividing the detector into two equal-distance drift volumes. The CPA’s 7 m $\times$ 6 m area is made up of six panels, each of which is constructed of three vertically stacked modules. The same modular structure and materials will be used in the far detector design. The scope of the ProtoDUNE-SP CPA includes:

• •

  18 CPA modules, each with a frame and resistive sheet,

• •

  HV bus connecting the resistive sheets and modules, and

• •

  HV cup for receiving input from the power supply.

Several requirements are placed on the HV system. Electrically, the CPA must provide an equipotential surface at $-$180 kV nominal bias voltage that remains stable for data taking. The resistive sheet must provide slow discharge of accumulated charge in the event of an unexpected HV breakdown, to prevent damage to the readout electronics. To ensure a uniform drift field, the flatness must remain to within 1 cm while submerged in LAr. Mechanically, the CPA needs to support the full weight of the four connected top and bottom FC modules, as well as that of a person during the installation. It must allow for cryostat roof contraction during the cool-down to LAr temperature. Furthermore, it must be constructable within the cryostat, and its materials must inhibit trapped LAr volumes.

The cathode plane surface is made of a resistive material, which, in the event of HV breakdown at a point on a CPA module, will restrict the sudden change in voltage to a relatively localised area, thus preventing discharges from potentially damaging either the cold electronics (CE), the cryostat, or the capacitively coupled anode planes. The rest of the CPA maintains its original bias voltage, and gradually discharges to ground through the high resistivity of the cathode material.

The 18 CPA modules are constructed of strong 6 cm thick FR4 (the fire-retardant version of G10) frames. The frames hold 3 mm thick FR4 sheets laminated on both sides with a commercial resistive Kapton film of type D11261075 [15]. Each CPA module is 1.16 m wide and 2 m high, and they stack three high to form a CPA panel of height 6 m. The CPA plane consists of six panels placed side-by-side and has the same dimensions as each of the two APA planes.

The surface of the frame facing the APAs is covered by a set of resistive FR4 strips with a bias voltage different from that of the CPA resistive sheets, chosen such that the frame itself causes no distortion in the drift field.

The outer edges of the cathode plane facing the cryostat wall are covered with the same metal profile assemblies as used in the field cage (FC), described in Section 4.1.2. This limits electric field strength to below 30 kV/cm, as required, in the areas around the CPA frame and eliminates the need for a special design of these most crucial regions of the cathode plane. The edges of the CPA are effectively a continuation of the FC, as shown in Figure 15. Since the FC profiles are the only objects facing grounded surfaces, they are the most likely candidates for HV discharges to ground. To limit peak current flow, these edge profiles are connected to the field-shaping strips by means of a wire jumper.

The CPA is connected to the HV feedthrough through a receptacle, called the HV cup as shown in Figure 15, at the downstream side of the cryostat (with respect to the beam entrance) and biased at $-$180 kV. It provides the voltage and the required current to all the FC modules (top, bottom, and end walls) through electrical interconnects (Section 4.1.2).

The design takes into account deformation and stress due to the pressure from the circulating LAr as well as shrinkage due to the temperature change. For example, to ensure contact between the CPA modules after cool-down, a gap of 0.7 mm, corresponding to the calculated amount of separation that contraction will remove as they cool down, has been introduced. The joints between the FC and the CPA are also designed to accommodate an estimated shrinkage of 5.2 mm of the steel supporting beam between the CPA and APAs.

Three modules are stacked vertically to form the 6 m height of a CPA panel as shown in Figure 16. The frames of these modules are bolted together using tongue-and-groove connections at the ends. The resistive cathode sheets and the field-shaping strips are connected using metallic tabs to ensure redundant electrical contact between the CPAs.

Each CPA panel is suspended from the cathode rail using a central lifting bar. Due to the roof contraction as the cryostat is cooled, it was calculated that each CPA would move $\sim$2 mm relative to its neighbours. Several pin-and-slot connections are implemented at the long edges of the CPA panels to ensure the co-planarity of the modules while allowing for a small vertical displacement.

The electrical connectivity of the resistive sheets within a CPA panel is maintained by several tabs through the edge frames. The voltage is passed from one CPA panel to another through embedded cables in the panels, referred to collectively as the HV bus, as shown in Figure 15. Redundant connections in the HV bus between CPA panels are used to ensure reliability. The HV bus also provides a low-resistance path for the voltage needed to feed the FC resistive divider chains. The required connections to the FC modules are made at the edges of the CPA. Along its perimeter, the HV bus cables are hidden between the field-shaping strip overhang and the main cathode resistive sheet. The cables are capable of withstanding the full cathode bias voltage to prevent direct arcing to (and as a result, the recharging of) a CPA that discharges to ground. Connections are flexible in order to allow for FC deployment, thermal contraction, and motion between separately supported CPA components.

The FC covers the top, bottom, and endwalls of all the drift volumes, thus providing the necessary boundary conditions to ensure a uniform electric field and shielding it from the cryostat walls. The FC is made of adjacent extruded aluminum profiles running perpendicular to the drift field and set at increasing potentials along the 3.6 m drift distance from the CPA HV (-180kV) to ground potential at the APA planes. Other elements of the FC are the ground planes (GP) sitting above the top and below the bottom FC modules. They confine the electric field in the liquid phase, avoiding high field both in the gas phase at the top of the cryostat and close to the piping at the bottom. The structures holding the profiles are made of insulating fibreglass-reinforced plastic (FRP). FRP has good mechanical strength at cryogenic temperatures and low coefficient of thermal expansion. The FC modules come in two distinct types: the identical top and bottom modules, which run the full length of the detector, and the endwall FC modules, which are installed vertically to close the detector drift volume at either end.

The FC is divided into mechanically and electrically independent modules, which comes with several advantages. As a consequence of electrically subdividing the modules, the stored energy and therefore the risk of detector damage is limited. In addition, the division acts as a protection from transient surges. FC modules have their own, independent voltage divider network providing the necessary linear voltage gradient. In case of a resistor failure in a divider chain field, distortions would be restricted to the FC module concerned. The mechanical subdivision simplifies construction and assembly of the FC.

There are six top and six bottom FC assemblies, all like the one shown in Figure 17. The assemblies are constructed from pultruded FRP I-beams and box beams that support the aforementioned aluminum profiles. The length and width are 2.3 m and 3.5 m, respectively, each assembly comprising 57 aluminum profiles. A GP consisting of modular perforated stainless steel sheets runs along the outside surface of each top and bottom FC with a 20 cm clearance. The gas region at the top of the volume, also referred to as the ullage, which is necessary for safe and stable operation of the LAr cryogenics system, contains many grounded conducting components with sharp features near which the electric field could easily exceed the breakdown strength of gaseous argon if directly exposed to the energised FC. The GP protects against this. The bottom FCs are equipped with GPs to shield from cryogenic piping and other sensors with sharp features on the cryostat floor.

The connections between the top and bottom FC modules and the CPAs are made with aluminum hinges 2.54 cm in thickness that allow the modules to be folded in on the CPA during installation. The hinges are electrically connected to the second profile from the CPA. The connections to the APAs are made with stainless steel latches that are engaged once the top and bottom FC modules are unfolded and fully extended towards the APA. A top FC module being lifted for installation on the CPA panel is shown in Figure 18.

Each of the two drift volumes has two endwall FCs one on each end. Each endwall FC is in turn composed of four stacked endwall FC modules, the topmost equipped with hanger plates. Each endwall FC module is constructed of two FRP box beams each 3.5 m long as shown in Figure 19 (dark grey) and Figure 20. The endwalls are not equipped with GPs as there is enough side clearance between the cryostat wall and FC, thus avoiding high E-fields. In ProtoDUNE-SP the endwall at the beam entry point is customized to hold the beam plug, described in Section 4.1.3.

The FC modules consist of extruded aluminum field-shaping-profiles. The profile shape minimises the electric field strength between a given profile and its neighbours and between a profile and other surrounding parts. The profile ends have a higher surface electric field, especially those at the corners of the FC (boundary with APA or CPA). To prevent HV breakdowns in the LAr, the ends of the profiles are encapsulated by custom UHMWPE (Ultra-High-Molecular-Weight Polyethylene) caps. The caps are designed and experimentally verified to withstand the full voltage across their thickness.

A resistive divider chain interconnects all the metal profiles of each FC module to provide a linear voltage gradient between the cathode and anode planes.

The resistive divider chain is a chain of resistor divider boards each with eight resistive stages in series. Each stage (corresponding to a 6 cm distance between FC profile centers) consists of two 1 G$\Omega$ resistors in parallel yielding a parallel resistance of 0.5 G$\Omega$ per stage to hold a nominal voltage difference of 3 kV. In the event of a HV breakdown, each stage is protected against HV discharge by varistors. Three varistors (with 1.8 kV clamping voltage each) are wired in series and placed in parallel with the associated resistors. A photo and schematic of the resistor divider board are shown in Figure 21. Each FC divider chain connects to an FC termination board in parallel with a grounded fail-safe circuit at the APA end. The FC termination boards are mounted on the top of the upper APAs and the bottom of the lower APAs. Each termination board provides a default termination resistance, and an SHV cable connection to the outside of the cryostat, via the CE signal feedthrough flange, through which it is possible to either supply a different termination voltage to the FC or monitor the current flowing through the divider chain, or both.

To minimise the material interactions of the particle beam in the cryostat upstream of the TPC, a volume of LAr along the beam path (between the cryostat inner membrane and the FC) is displaced, and replaced by a less dense volume of dry nitrogen gas. The gas is contained within the beam plug, a cylindrical glass-fiber composite pressure vessel, about 50 cm in length and 22 cm in diameter. It is illustrated in Figure 22. A pressure relief valve and a burst disk are installed on the nitrogen fill line on the top of the cryostat (externally) to ensure the pressure inside the beam plug does not exceed the safety level of about 22 psi. The nitrogen system schematic is shown in Figure 23.

The beam plug is secured to the endwall FC support structure as illustrated in Figure 22. Beam is fed into one of the two available drift volumes of the TPC, the drift volume not receiving beam is used for cosmic ray studies. The front portion of the beam plug extends 5 cm into the active region of the TPC through an opening in the FC. The FC support is designed with sufficient strength and stiffness to support its weight. The total internal volume of the beam plug is about 16 liters.

The requirements on the acceptable leak rate is between $7.8\times 10^{-5}$ scc/s and $15.6\times 10^{-5}$ scc/s. This is very conservative and is roughly equivalent to leaking 15% of the nitrogen in the beam plug over the course of a year. In the worst-case scenario in which all the nitrogen in the beam plug leaks into the LAr cryostat, the increase in concentration is about 0.1 ppm, which is still a factor of 10 below the maximum acceptable level, as specified by light-detection requirements. Over the course of about 1.5 years of beam plug operations in LAr at ProtoDUNE-SP, no detectable leak was observed.

At nominal operation, the voltage difference across the beam plug (between the first and the last grading ring) is 165 kV. To minimise risk of electrical discharges, the beam plug is divided into sections, each of which is bonded to stainless steel conductive grading rings. The seven grading rings are connected in series with three parallel paths of resistor chains. The ring closest to the FC is electrically connected to one of the FC profiles. The electrode ring nearest the cryostat wall is grounded to the cryostat (detector) ground. The maximum total power dissipated by the resistor chain is about 0.6 W.

The TPC high voltage (HV) components include the HV power supply, cables, filter circuit, HV feedthrough, and monitoring instrumentation for currents and voltages (both steady state and transient).

The $-$180 kV voltage necessary to produce the required electric field of 500 V/cm is delivered by the power supply through an RC filter and a HV feedthrough to the CPA. The design of the HV feedthrough is based on the reliable construction technique adopted for the ICARUS HV feedthrough [16] and shown in Figure 24. Before installation, the feedthrough was successfully operated for several days in a test stand at voltages up to 300 kV.

The feedthrough design has a coaxial geometry, with an inner conductor (HV) and an outer conductor (ground) insulated by UHMWPE, as illustrated in Figure 24. The outer conductor, a stainless-steel tube, surrounds the insulator, extending down through the cryostat into the LAr. In this geometry, the E field is confined within regions occupied by high-dielectric-strength media (UHMW PE and LAr). The inner conductor is made of a thin-walled stainless steel tube to minimise the heat input and to avoid the creation of argon gas bubbles around the lower end of the feedthrough. A contact, welded at the upper end for the connection to the HV cable, and a round-shaped elastic contact for the connection to the cathode, screwed at the lower end, completes the inner electrode. Special care has been taken in the assembly to ensure complete filling of the space between the inner and outer conductors with the (polyethylene) PE dielectric, and to guarantee leak-tightness at ultra-high vacuum levels.

Filter resistors are placed between the power supply and the feedthrough. Along with the cables, these resistors reduce the discharge impact by partitioning the stored energy in the system. The resistors and cables together also serve as a low-pass filter reducing the 30 kHz voltage ripple on the output of the power supply.

The filter resistors are of a cylindrical design. Each end of an HV resistor is electrically connected to a cable receptacle. A cylindrical insulator is placed around the resistor, and a grounded stainless steel tube surrounds the insulator.

The instrumentation in ProtoDUNE-SP provided useful information on HV stability. Outside the cryostat, the HV power supply and cable-mounted toroids monitor the HV. The power supply has capabilities down to tens of nA in current read-back and is able to sample the current and voltage every 300 ms. The cable-mounted toroid is sensitive to fast changes in current; the polarity of a toroid’s signal indicates the location of the current-drawing feature as either upstream or downstream of it.

Inside the cryostat, pick-off points near the anode monitor the current in each resistor chain. Additionally, the voltage of the ground planes (GPs) above and below each drift region can diagnose problems via a high-value resistor connecting the GP to the cryostat.

During cool-down and LAr filling, a power supply was used to supply $-$1 kV to the cathode and monitor the current draw of the system. As the system cooled from room temperature to LAr temperature, the resistance increased by $\sim$10%, consistent with expectations. Once the LAr level had exceeded the height of the top GP, the voltage was ramped up to the nominal voltage.

Two types of instabilities emerged in the cold side of the HV system. The first type was a so-called current blip, during which the system drew a small excessive current that persisted for no more than a few seconds. The magnitude of the excess current during current blips increased over the subsequent three weeks from 1% to 20%. The second type of instability, called a “current streamer,” exhibited persistent excessive current draw from the HV power supply with accompanying excessive current detected on a GP and on the beam plug. These two types of instabilities occurred periodically throughout the beam run. The frequency of both types increased over time after the system was powered on, until a steady state of about ten current blips/day and one current streamer every four hours was reached. These effects are consistent with a slow charging-up process of the insulating components of the FC supports, which then experience partial discharges that are recorded as HV instabilities. This process restarts after every long HV-off period.

In addition, these processes seemed to be enhanced by the LAr bulk high purity, which allowed the electric current to develop. At low purity electronegative impurities acted as quenchers, blocking the development of the leakage current. During the 2018 beam run periods, priority was given to operating the ProtoDUNE-SP detector with maximal uptime in order to collect as much beam data as possible at the nominal HV conditions [17]. In some cases, mostly outside of the beam run period, the HV system was turned off momentarily to allow the HV system components to discharge. This is reflected as larger dips in the uptime plot shown in Figure 25 [18]. During periods when the rest of the subsystems (including the beam) were stable, the moving 12-hour HV uptime fluctuated between 96% and 98%. Automated controls to quench the current streamers were then successfully implemented in an auto-recovery mode. These helped to increase the uptime significantly, by optimising the ramping down and up of the HV power supply voltage, which was performed in less than four minutes. The process of the auto-recovery mode is shown in Figure 26.

Investigating the long-term behaviour of the HV instabilities and understanding their origin became goals of the long-term operation of ProtoDUNE-SP in 2019. As mentioned above, it appeared that the current-streamer effect is a charging-up process with its frequency increasing with time after a long HV-off period. This behaviour was observed repeatedly and this cause was confirmed in 2019. The current-streamer rate stabilised at 4-6 events per day, and the location remained on the same single ground plane (GP#6). The rate and location were approximately independent of the HV applied on the CPA in the 90 kV to 180 kV range.

More recently, after changing the LAr re-circulation pump in April 2019, the detector was operated for several months in very stable cryogenic conditions and with very high and stable LAr purity (as measured by purity monitors and cosmic rays). During this period, a significant evolution was observed. The HV system was set and operated at the nominal value of 180 kV at the CPA for several weeks without interruption.

To better understand the current-streamer phenomenon, the HV system was operated for about fifty days without the auto-recovery script, and the current streamers were left to evolve naturally. They typically lasted six to 12 hours, exhibiting steady current and voltage drawn from the HV power supply, and they eventually self-quenched without any intervention. During this period, the repetition rate was significantly reduced to about one current streamer every 10-14 days; this rate can be compared to the 4-6 per day in the previous periods with auto-recovery on.

The auto-recovery script was then re-enabled and the current-streamer rate stabilised at about one in every 20 hours; in addition, the intensity of the current streamer on the GP was reduced with respect to the previous periods. As in the previous runs, the current streamers always occurred on the same GP (GP#6) with a small leakage current on the beam plug hose, which is close to GP#6.

This behaviour is a further indication that the current streamers were, in fact, a slow discharge process of charged-up insulating materials present in the high-field region outside of the FC. The auto-recovery mode did not allow a full discharge, so the charging up was faster, and the streamer repetition rate was shorter.

The LAr purity loss experienced at the end of July 2019 was accompanied by the complete disappearance of any HV instabilities. They gradually reappeared when the electron lifetime exceeded 200 microseconds, and their intensity constantly increased as purity improved. This behaviour replicated that observed after the initial filling, and is consistent with what has been observed on other similar systems [19], thus supporting the hypothesis that the HV instabilities are enhanced by the absence of electronegative impurities in high-purity LAr.

The effects of the current streamers on the front-end electronic noise and the PD background rate were investigated. No effect of the current streamers on the FE electronics was observed. On the other hand, recent analysis of the data collected by the PDS during active current streamers has indicated a high single photon rate on the upper upstream part of the TPC. This is consistent with the activities recorded on GP#6, which is located exactly at this upper upstream area.

Anode Plane Assemblies (APAs) are the detector elements used to detect ionization electrons created by charged particles traversing the LAr volume inside the ProtoDUNE-SP TPC. There are two APA arrays, one on the outer side of each of the two drift volumes. Each array comprises three APAs (6.3 m tall, 2.3 m wide, 0.12 m thick) hung vertically and adjacent to each other. Each APA has layers of sense and shielding wires wrapped around a framework of lightweight, rectangular stainless steel tubing, as shown in Figure 27. The sense wire readout is performed by cold electronics (CE) attached at the upper end (head) of each APA.

The APAs are designed and built to address the physics performance specifications listed in Table 3, which are defined to ensure high-efficiency event reconstruction throughout the entire active volume of the LArTPC.

Identifying minimum-ionizing particles (MIPs) is a function of several detector parameters, including argon purity, drift distance, diffusion, APA sense-wire pitch, and equivalent noise charge (ENC). It is required that MIPs originating anywhere inside the active volume of the detector be reconstructed with 100$\%$ efficiency. The choice of wire pitch combined with the other high-level parameter values, listed in Table 4, is expected to enable this efficiency and provide good tracking resolution and good granularity for particle identification.

The specified vertex resolution of 1.5 cm along each coordinate direction follows from a requirement that it be possible to determine the fiducial volume to 1%. The chosen wire pitch achieves this for the $y$ and $z$ coordinates. The resolution on $x$, the drift coordinate, will be higher than in the $y$-$z$ plane due to the combination of drift velocity and electronics sampling rate.

In the construction of an APA, the frame is first covered on both sides with a fine mesh that defines a uniform ground across the frame. Along the length of the frame and around it, over the mesh layer, layers of sense and shielding wires are strung at carefully selected angles relative to each other. The wires are terminated on boards that both anchor them and provide the connections to the CE.

The grid plane wires (G) are not connected to the electronic readout. Separate sets of grid wires run along each side of the APA, parallel to its long edge. The two planes of induction wires (U and V) wrap continuously around both sides of the APA in a helical fashion. The collection plane wires (X) run vertically, parallel to the grid wires. The ordering of the layers, from the outside in, is G-U-V-X, with the mesh layer at the centre. All wire layers span the entire face on both sides of the APA frame.

The angle of the induction planes in the APA ($\pm$35.7^∘) was chosen such that each induction wire only crosses a given collection wire one time, reducing the ambiguities that the reconstruction must address. Coupled with the wire pitch, this angle makes it possible for an integer multiple of electronics boards to read out one APA.

The operating voltages of the APA layers are listed in Table 5. When operated at these voltages, the drifting ionization follows trajectories around the grid and induction wires, ultimately terminating on a collection plane wire; i.e., the grid and induction layers are completely transparent to drifting ionization, and the collection plane is completely opaque. The grid layer is present for pulse-shaping purposes, effectively shielding the first induction plane from the drifting charge and removing the long leading edge from the signals on that layer. The mesh layer serves to shield the sense planes from pickup from the PDS.

The wrapped configuration allows the APA array to fully cover the active area of the LArTPC, minimizing the amount of dead space between the APAs that would otherwise be occupied by electronics and associated cabling.

The current design of the DUNE-SP far detector module implements three APA arrays that run the 60 m length of the TPC. One runs down the centre of the detector, the other two are installed along the outer walls, and two CPA arrays are interleaved between them, creating a set of four drift fields. The central APA array, flanked by drift fields, requires sensitivity on both sides, which the wrapped induction-plane wire design enables. Whereas this double-sided feature is not strictly necessary for the ProtoDUNE-SP arrangement, in which only the inner side of each APA faces a drift field, it is compatible with this setup, as the grid layer on the wall side effectively blocks any ionization generated outside the TPC from drifting in to the sense wires.

The APAs are wound with 150$\mu$m (.006 in) diameter beryllium copper (CuBe) wire ($\sim$98% copper, $\sim$1.9% beryllium), used for its high durability and yield strength. The X- and G-plane wires extend the full 6 m length of the APA and are not wrapped. The diagonal, wrapped wires (U and V planes) extend 3.9 m across each face of the APA. To prevent deflection from gravity, electrostatic forces, and liquid drag from any moving LAr, a set of combs supports these wires at regular intervals along the length of the APA, keeping the longest unsupported wire length under 1.6 m. The combs are slotted pieces of 0.5 mm thick G10, mounted on each of the frame’s four cross braces. The wires are held at a tension of 5 N.

The wire tension and wire placement accuracy specifications ensure that the wires are held taut in place with no sag. Wire sag can impact both the precision of reconstruction and the transparency of the TPC. The tension of 5 N is low enough that as the wires contract during cool-down, they stay safely below the vendor-tested 25.6 N breaking tension value.

All APA wires are terminated on stacked wire boards, installed at the head end of each APA to provide the connections to the CE, as shown in Figure 28. The APA has ten adjacent sets of these board stacks on each side rather than one long stack spanning the entire width. Within each stack, one board corresponds to one wire plane. Attachment of the wire boards begins with the (innermost) X wire plane to the lowest board in the stack. These wires are strung top to bottom along each side of the APA frame, soldered, epoxied to connections on the board, and trimmed. As each subsequent layer is applied, its wires are attached to the next highest wire board layer. Mill-Max pins and sockets provide the electrical connections between the circuit boards within a stack.

The wire boards for the X and G planes accommodate 48 wires each, and those for the U and V planes accommodate 40 each. Each board stack, therefore, has 176 wires, but only 128 signal channels since the G wires are not read out. The total of 20 stacks per APA results in 2,560 signal channels and a total of 3,520 wires starting at the top of the APA and ending at the bottom. Each APA holds a total of $\sim$ 23.4 km of wire. Figure 29 depicts the connections between the different elements of the APA electrical circuit.

At the head end of the APA, the 4.75 mm wire-plane spacing is set by the thickness of these wire boards. The first plane’s wires solder to the surface of the first board, the second plane’s wires to the surface of the second board, and so on. For installation, temporary toothed-edge boards beyond these wire boards are used to align and hold the wires until they are soldered to pads on the wire boards. After soldering, the extra wire is snipped off.

Attached to the wire board stack, capacitive-resistive (CR) boards provide DC bias and AC coupling to the wires. The CR boards carry a bias resistor (51 M$\Omega$) and a DC-blocking capacitor (3900 pF) for each wire in the X and U planes. Connections from the CR boards to the CE are made through a pair of 96-pin Samtec connectors. In the case of the outermost G plane, wire boards connect adjacent groups of four wires together and bias each group through an RC filter whose components are placed on special G plane CR boards. All CR boards are attached to the board stacks after fabrication of all wire planes.

Pins extending outward from the CR boards provide connections from the APA to the modularly designed CE. Each board stack has one CE module connected to it. Each CE module is housed in a small metallic enclosure called a CE box that provides electrical shielding and mechanical support, simplifies installation and replacement, and helps with the dissipation of argon gas generated by the warm electronic components. The CE modules are mounted in such a way that any of them can be removed from the inner side of the APA after APA installation. Figure 30 illustrates the CE boxes and their installation in ProtoDUNE-SP.

The ProtoDUNE-SP TPC read-out electronics are referred to as “cold electronics (CE)” because the system resides in the LAr, mounted directly on the APA, thus reducing channel capacitance and noise by minimising the length of the connection between an anode wire and its corresponding electronics input.

The CE signal processing is implemented in ASIC chips using CMOS technology, which has been demonstrated to perform well at cryogenic temperatures, and includes amplification, shaping, digitisation, buffering, and multiplexing (MUX) of the signals. The CE is continuously read out, resulting in a digitised ADC sampling from each APA channel (wire) at a rate of up to every 500 ns (2 MHz maximum sampling rate).

The 2,560 channels from each APA are read out by 20 front-end motherboards (FEMBs), each providing digitised wire readout from 128 channels. One cable bundle connects each FEMB to the outside of the cryostat via a CE feedthrough in the signal cable flange at the top of the cryostat, where a single flange services each APA. Each cable bundle contains wires for low-voltage (LV) power, high-speed data readout, and clock/digital-control signal distribution. In addition to the CE cables, eight separate cables carry the TPC wire-bias and FC termination voltages from the signal flange to the APA wire-bias boards as shown schematically in Figure 31.

The main component of the CE architecture, illustrated in Figure 32, is the 128-channel FEMB, which itself consists of an analogue motherboard and an attached FPGA mezzanine card for processing the digital outputs. Each APA is instrumented with 20 FEMBs, for a total of 2,560 channels per APA. The FEMBs plug directly into the APA CR boards, making the connections from the U, V, and X plane wires to the charge amplifier circuits as short as possible.

The analogue mother board is instrumented with eight 16-channel FE ASICs, eight 16-channel ADC ASICs, LV power regulators, and input-signal protection circuits. The 16-channel FE ASIC provides amplification and pulse shaping. The 16-channel ADC ASIC comprises 12-bit digitisers performing at speeds up to 2 MS/s, local buffering, and an 8:1 MUX stage with two pairs of serial readout lines in parallel. Figure 33 shows the analog motherboard, the FPGA mezzanine, and the complete FEMB assembly.

Each FE ASIC channel has a charge amplifier circuit with a programmable gain, selectable from 4.7, 7.8, 14, and 25 mV/fC (full-scale charge of 55, 100, 180 and 300 fC), a high-order anti-aliasing filter with programmable time constant (peaking time 0.5, 1, 2, and 3 $\mathrm{\mu}$s), an option to enable AC coupling, and a baseline adjustment for operation with either the collecting (200 mV) or the non-collecting (900 mV) wires. Shared among the 16 channels in the FE ASIC are the bias circuits, programming registers, a temperature monitor, an analog buffer for signal monitoring, and the digital interface. The FE ASIC’s estimated power dissipation is about 6 mW per channel at 1.8 V supply.

The FE ASIC was implemented using the commercial CMOS process (0.18 $\mu$m and 1.8 V), which is expected to be available for at least another ten years. The charge amplifier input MOSFET is a p-channel biased at 2 mA with a L/W (channel length/width) ratio of 0.27 $\mu$m / 10 $\mu$m, followed by dual cascade stages.

Each FE ASIC channel is equipped with an injection capacitor that can be used for testing and calibration and is enabled or disabled through a dedicated register. The injection capacitance has been measured using a calibrated external capacitor. The measurements show that the calibration capacitance is extremely stable, changing from 184 fF at room temperature to 183 fF at 77 K. This result and the measured stability of the peaking time demonstrate the high stability of the passive components as a function of temperature. Channel-to-channel and chip-to-chip variation in the calibration capacitor is typically less than 1%.

The ADC ASIC design is also implemented using the CMOS process (0.18 $\mu$m and 1.8 V). The ADC ASIC is a complex design with 320,000 transistors, while the FE ASIC has 16,000. The transistor design work has been done following the rules for long cryogenic lifetime. Shared among the 16 channels in the ADC ASIC are the bias circuits, programming registers, an 8:1 MUX, and the digital interface. The estimated power dissipation of ADC ASIC is below 5 mW per channel at 1.8 V supply.

The ADC ASIC data are passed to the FPGA mezzanine board for transmission to the warm electronics located on the outside of the signal flange. The FPGA has four 4:1 MUX circuits that combine the 16 serial lines from the eight ADC channels into four serial lines of 32 channels each, and four $\sim$ 1.2 Gigabit-per-second (Gbps) serial drivers that drive the data in each line over cold cables to the warm interface electronics (WIBs). The FPGA on the mezzanine card is also responsible for communicating with the control and timing systems from the WIB and providing the clock and control signals required by the FE and ADC ASICs.

The FPGA and all other electrical components on the FEMB assembly has been evaluated and characterised at RT (300 K) and LN2 (77 K) temperature. During these tests the FEMB has been temperature-cycled multiple times. In addition, power-cycle tests at cryogenic temperature have been performed. Figure 34 shows the measured ENC as a function of filter-time constant (peaking time) for two different gains as measured on a prototype FEMB. ENC is the value of charge (in electrons) injected across the detector capacitance that would produce at the output of the shaping amplifier a signal whose amplitude equals the output RMS noise. These measurements were made with the prototype FEMB at both RT and submerged in LN2 with a wire-simulating input capacitance of $C_{f}~=~150$ pF (equivalent to an approximately 7 m sense wire load). In LN2, for peaking times $>$1 $\mu$s, less than 600 e^- was measured. For comparison, a MIP travelling perpendicularly to the wire plane in the direction of wire spacing is expected to deposit at least $\sim$ 10,000 e^- on the collection wires, for a worst-case SNR is about 16 to 1³³3For the minimum performance requirements of 500 V/cm and a 3 ms electron lifetime. One of the key ingredients to achieving a low-level noise performance is the grounding of the experimental setup. The details of the ProtoDUNE-SP grounding scheme and methods to ensure a low noise level are discussed in Section 5.4

As mentioned in Section 4.2.1, each FEMB is enclosed in a Faraday box, called a CE box, that provides shielding from noise. As shown in Figure 35, it provides the electrical connection between the FEMB and the APA frame. Mounting hardware inside the CE box connects the ground plane of the FEMB to the box casing, which is electrically connected to the APA frame via twisted conducting wire. This is the only point of contact between the FEMB and APA, except for the input amplifier circuits connected to the CR board, which also terminate to ground at the APA frame.

The warm interface electronics are housed in warm interface electronics crates (WIECs) attached directly to the signal flange. The WIEC shown in Figure 36 contains one power and timing card (PTC), up to five warm interface boards (WIBs) and a passive power and timing backplane (PTB), which fans out signals and LV power from the PTC to the WIBs.

The WIB is the interface between the DAQ system and up to four FEMBs. It receives the system clock and control signals from the timing system and provides for processing and fan-out of those signals to the four FEMBs. The WIB also receives the high-speed data signals from the four FEMBs and transmits them to the DAQ system over optical fibres. The WIBs are attached directly to the TPC CE feedthrough on the signal flange. The feedthrough board is a PCB with connectors to the cold signal and LV power cables fitted between the compression plate on the cold side, and sockets for the WIB on the warm side. Strain relief for the cold cables is supported from the back end of the feedthrough.

The PTC provides a bidirectional fibre interface to the timing system. The clock and data streams are separately fanned-out to the five WIBs. The PTC fans the clocks out to the WIB over the PTB, which is a passive backplane attached directly to the PTC and WIBs. The received clock on the WIB is separated into clock and data using a clock/data separator.

The PTC also receives LV power for all CE connected through the signal flange, approximately 250 W at 48 V for a fully-loaded flange (one PTC, five WIB, and 20 FEMB). The LV power is then stepped down to 12 V via a DC/DC converter on-board the PTC and fanned out on the PTB to each WIB, which provides the necessary 12 V DC/DC conversions and fans the LV power out to each of the cold FEMBs supplied by that WIB, as shown in Figure 37. The majority of the 250 W drawn by a full flange is dissipated in the LAr by the cold FEMB.

Each WIB contains a unique IP address for its UDP (User Datagram Protocol) slow control interface. In addition, the WIB is capable of receiving the encoded system-timing signals over bi-directional optical fibres on the front panel, and processing these using either the on-board FPGA or clock synthesizer chip to provide the 50 MHz clock required by the CE.

The FPGA on the WIB is an Altera Arria V GT variant, which requires a 125 MHz clock for its state machine that is provided by an on-board crystal oscillator. It can drive the high-speed data to the DAQ system up to 10.3125 Gbps per link, implying that all data from two FEMB (2$\times$5 Gbps) could be transmitted on a single link. On top of that, the FPGA has an additional Gbps Ethernet transceiver I/O based on the 125 MHz clock, which provides real-time digital data readout to the slow control system as well.

The dip-coated collector design uses diamond-polished acrylic, cut from cast LUCITE UTRAN manufactured by Palskolite, Inc., and supplied by EMCO Industrial Plastics as the light-guide “bar”. The bar has an index of refraction of 1.49 and is dipped in a solution of tetraphenyl-butadiene (1,1,4,4-Tetraphenyl-1,3-Butadiene, TPB) and other solvents (including a surfactant) to produce a wavelength-shifting (WLS) layer on the outside surface. When a particle passes through the active volume, VUV light produced from scintillation at 128 nm interacts with the surface coating on the bar, where it is shifted to blue light at 425 nm, and isotropically emitted. Once the blue light enters the light guide, it travels via total internal reflection and is absorbed by a collection of photosensors at one end. A conceptual example of the dip-coated WLS light guide is shown in Figure 39.

Bar construction began with the industrial manufacture of the acrylic, which was shipped pre-polished. The first quality control step was a visual inspection of the acrylic which ensured that bars used for production were relatively free from scratches or chips and matched pre-production bars in terms of polish quality. Production bars were then measured at five points of contact using calipers to verify their required dimensions and tolerances: ($209.270\pm 0.127\text{\,cm})\,\times\,(8.471\,\pm 0.127\text{\,cm})\,\times\,(0.599\,\pm 0.060\text{\,cm})$. Once a bar was approved for use, a unique serial number was etched on it. The next steps were to anneal (bake and then slowly cool) the bars then dip them in the WLS solution. Annealing was done to strengthen the bar material and as a form of quality control. Bar annealing took place in a large oven in Lab 3 at Fermilab that was set at about $76.7^{\circ}$C. Baking lasted approximately four hours. Annealed bars were required to bear no visual signs of crazing (mechanical stress breaks that make the bar appear cloudy). Accepted bars were cleaned with ethanol and dipped in a coating solution, consisting of 50 mL toluene, 12 ml ethanol, 0.1 g acrylic pellets, and 0.1 g TPB for five minutes, where the acrylic pellets ensure that the coating index of refraction matches that of the bar [23]. A mechanised dipping vessel was prepared to automate the process and standardise bar quality. Finally, successfully prepared, annealed, and dipped bars were hung in air under a fume hood to dry for at least 30 minutes. The dipping and drying processes were sensitive to moisture and required a low-humidity environment (<15% relative humidity). The dipping vessel at Fermilab is shown in Figure 40.

Quality control for fully dipped and dried bars consisted of observing the attenuation of 200 nm light in the bars in both warm and LAr temperature regimes. The warm measurements used full-size bars whereas measurements at LAr temperatures took place on shorter bars that were cut to fit within the TallBo facility at the Proton Assembly Building (PAB) at Fermilab. Both measurement campaigns yielded an attenuation length in excess of 2 m.

The double-shift light-guide collector [24, 25] combines the use of WLS TPB-coated radiator plates with a WLS light-guide bar. In this design, LAr scintillated light at 128 nm undergoes two wavelength-shifting steps, once to 425 nm blue light in the plate and then to 490 nm green light in the bar. A conceptual example of the double-shift WLS light guide is shown in Figure 39. The benefits of this design are in the increased efficiency of photosensor detection for light at 490 nm and in the production and quality control of the technology. During and after construction, bars are separable and components can be independently studied and optimised.

The outer WLS material is a set of twelve separate radiator plates that completely cover the light guide, six on each side, placed end-to-end. The plates are made from 1/16 inch acrylic (purchased in one order from McMaster Carr) that were laser-cut in pairs at Fort Collins Plastics in Fort Collins, CO. The plates are coated by hand in a WLS solution consisting of 5 gm of scintillation grade (>99%) TPB to 1,000 gm of dichloromethane, using a Binks high-volume lower pressure sprayer system. Two plates, cut from a single piece of acrylic, along with three breakout tabs for quality control, are coated with the WLS at the same time (tab-plate-tab-plate-tab).

Once the plates are coated, they are annealed in a vacuum oven at 80^∘C, a temperature just below the melting point of the acrylic. The annealing incorporates the TPB into the acrylic, ensuring that it will not separate or precipitate due to aging or cryocycling in LAr. Metrology tests are carried out to ensure that no changes in bar dimension occurred during annealing. The plates are then separated and the breakout tabs are tested for brightness using a McPherson VUV monochromator with an H₂ lamp source, and calibrated with a NIST-calibrated photodiode. Each plate is evaluated with respect to a standard tab, averaging the response from the two tabs on either side of it [26]. The evaluation is relative; plates are accepted only if they are brighter than baseline plates that meet efficiency goals derived from DUNE physics requirements. The measurement of baseline plate efficiency was conducted on a LAr test stand at Fermilab, on two prototype paddles using a cosmic ray trigger [24].

The second WLS material and primary light guide for the double-shift collectors is made from special commercial Eljen EJ-280PSM crosslinked polystyrene. The polysterene was cut into bars that were required to meet the following dimensions within tolerances: ($209.15\pm 0.05\text{~cm})\times(8.6\pm 0.05\text{~cm})\times(0.6\pm 0.05\text{~cm}$). Bars that passed were next accepted or rejected on the basis of internal attenuation length, which was measured in a dark box with a 435 nm LED at room temperature. The correlation between attenuation length at room temperature and at LAr temperature was established at Indiana University on a set of shorter Eljen bars. The results of these combined studies showed that a room temperature attenuation length measurement greater than 6.4 m in the dark box ensured an attenuation length in LAr greater than 2 m. An attenuation length greater than 6.80 m as measured in the dark box at Indiana University was therefore required for acceptance [26].

The operating concept of the ARAPUCA is the trapping of incident photons inside a highly reflective chamber until they reach a sensor, yielding high detection efficiency despite a limited active coverage [27]. A single ARAPUCA cell consists of a highly reflective chamber with a single-direction acceptance window. The window consists of a dichroic optical filter sandwiched between thin layers of wavelength shifting materials. A conceptual demonstration of the ARAPUCA concept is shown in Figure 41 and its construction is shown in Figure 42.

For the collector to act as a photon trap, the external face of the dichroic filter has a WLS coating with an emission wavelength less than the cutoff wavelength of the filter. The transmitted photons pass through the filter where they encounter a second WLS-coated surface. This second coating has emission spectra that exceed the cutoff wavelength, thus trapping the photon inside the box. Trapped photons reflect off the inner walls and the filter surface(s) (of reflectivity typically greater than 98%) and have a very high probability of impinging on a photosensor before being lost to absorption. Dichroic filters with cutoff at 400 nm were acquired from Omega Optical. The internal surfaces of the filter are lined with VIKUITI™3M specular reflector coated with TPB, and P-Terphenyl (1,4-Diphenylbenzene, PTP) is applied to the external side of the filter. PTP and TPB coatings were applied using a resistive evaporation technique under a clean vacuum of at least $2.0\times 10^{-6}$ torr. A quartz microbalance was used to monitor rate and thickness during coating and was consistently placed in the same position relative to the substrates and target material. PTP and TPB surface densities are about 200 and 300$\mu g/\mathrm{cm}^{2}$, respectively.

Quality control for each batch of evaporated coating consisted of the random testing of one of four pieces in liquid argon. The test pieces were cooled in cold argon vapor for two hours before being fully immersed in LAr for up to 20 hours. When a piece did not pass this test, the entire batch was thoroughly cleaned of the coating, recoated, and a piece retested. There were some early failures that required recoating, but after establishing a pre-shipping cleaning and packaging protocol from the vendor and a cleaning protocol before coating, no batches were rejected. Before installation in ProtoDUNE-SP, several ARAPUCA prototypes were tested in LAr with $\alpha$ sources and cosmic ray muons [28].

The ProtoDUNE-SP PDS uses three different silicon photon sensor (SiPM) models commercially manufactured by two different companies, SensL Technologies, Ltd. (now, ON Technologies) and Hamamatsu Photonics. All sensors have the same active area, $6\times 6$ mm². Samples of the different sensors types were tested and characterised at LN₂ temperature before use.

SensL Technologies, Ltd provided (MicroFC-60035-SMT) sensors with pixel size 35 $\mu$m. Two types of series models were used, A-Series and C-Series, with the switch due to a change in availabilty from the manufacturer. SensL devices equip most of the WLS bar-type modules, i.e., 21 dip-coated modules and 22 double-shift modules. Each module array contains 12 SiPMs passively ganged in groups of three and read out as four independent channels. A total of 172 channels are equipped with 516 SensL SiPMs.

Hamamatsu Photonics provided two versions of its Multi-Pixel Photon Counter (MPPC) of model number S13360-6050, a CQ “Quartz Window” type and a VE “Through Silicon Via” type for use in the WLS bar detectors and the ARAPUCAs. Both have a pixel of 50 $\mu$m. The VE MPPC is coated with epoxy resin and uses through-hole electrodes called TSV (through-silicon via). In VE MPPCs, the space around the active area is reduced with respect to standard wire-bonding techniques, allowing for better packaging. The Quartz Window MPPC is an uncoated MPPC in a ceramic package with a quartz window, aimed to better resist thermal stresses. All ARAPUCA modules are equipped with Hamamatsu CQ MPPCs, while eight dip-coated and seven double-shift modules are equipped with VE MPPCs. All photosensors were tested at warm temperature before mounting them on the boards, to verify compliance with producer specifications. The equipped boards were thermal-cycled and tested again before being coupled to the lightguides.

For each photon detector, signals from one of the three different ganging schemes are summed together and connected, in the cold volume, to a 20 m long multi-conductor Cat6 cable that feeds through the detector flange. No front-end electronics are necessary in the cold volume for the operation of the ProtoDUNE-SP PDS. The readout system transmits unamplified signals from the photosensors in the LAr volume to the outside of the cryostat and performs processing and digitisation using a SiPM Signal Processor (SSP), a readout developed and manufactured by Argonne National Laboratory. Signal cables from each of the six detector flanges connect directly to four of 24 SSP modules. An SSP module consists of 12 individual readout channels packaged in a self-contained 1U module. Each channel contains a fully-differential voltage amplifier and a 14 bit, 150 MHz analogue-to-digital converter (ADC) that digitises the waveforms received from the SiPM arrays. The front-end amplifier is configured as fully-differential with a high common-mode rejection, and receives the SiPM signals into a termination resistor that matches the characteristic impedance of the signal cable. The SSP module can operate three separate WLS light guide modules or one ARAPUCA module.

The digitised data is stored in pipelines in the SSP, for up to 13 $\mu$s for a single output, per channel. The processing is pipelined, and performed by a Xilinx Artix-7 Field-Programmable Gate Array (FPGA). The FPGA implements an independent Data Processor (DP) for each channel. The processing incorporates a leading edge discriminator for detecting events and a constant fraction discriminator (CFD) for sub-clock timing resolution. An operational schematic of the SSP is shown in Figure 43.

Each channel can be individually triggered by any of the following: a periodic timestamp trigger, a SSP-internal trigger based on a leading-edge discriminator local to the individual channel, or a SSP-external trigger from the timing system. If a SSP trigger is present, the channel will produce a data packet consisting of a header and a waveform of predefined length, which comprises a series of ADC values. The header contains bookkeeping information (e.g., module and channel numbers, timestamps, trigger type) and some calculated integral values for the waveform. Because the SSP trigger system is distinct from the global trigger, the unit of data produced when an SSP channel triggers is called a “packet,” and the term “trigger” refers to the global triggers only. The SSP boardreader generates a fragment when a trigger produced by the timing system is observed, and this fragment will contain all packets received from the SSP with timestamps in a window $\pm$ 2.5 ms from the timestamp of the trigger.

In general, an SSP fragment will contain a fixed 12 packets with identical timestamps corresponding to the trigger time, one for each channel, and also an arbitrary number of additional packets generated when a channel’s discriminator fires. It should be expected that a different number of packets will be observed for each channel within a given fragment. In the case that only the SSP-external trigger is enabled, exactly 12 packets should be present in a single fragment. In such a case the packets received by the SSP that do not fall within the window around a timing system trigger will be dropped and never included in a fragment. All 24 SSPs are individually synchronised to the dedicated ProtoDUNE-SP timing system time-stamp.

All PD modules for ProtoDUNE-SP arrived at CERN ready for APA installation. PD modules were shipped from Colorado State University (CSU), via a special PD-Crate, which contained 12 PD modules, individually packaged in an anti-static bag and placed between anti-static foam. The final quality control steps before installation were a visual inspection of each PD module and an assessment of each module using a PD darkbox scanner, shown in Figure 44.

In the standard procedure for the darkbox scanning measurements, both the module photosensors (SiPMs/MPPCs) and the darkbox-LED were allowed to warm up for 30 minutes, using nominal voltage. This assured stability within each scan and reliability for module-to-module comparisons. The PDS group used LBNEWare, custom made software which interfaces with the SSPs directly, as the DAQ for these tests. For each module scan, the LED was set to pulse and read out 5000 times at multiple locations along each side of the PD module. This allowed for sufficient signal observation and determination of the mean response of the module along each position. The mean response was determined by fitting a Gaussian to the 5000 integrated waveforms. Prior to arrival at CERN an attenuation length had been calculated for each PD module using the relative brightness (mean signal) along each position. At CERN, a cross-check in pseudo-attenuation length was made and the module with the longest pseudo-attenuation length was selected for installation into the APA. Modules were placed facing the TPC for the best photon yield response. Spare modules were also sent to CERN and were sorted for use in the detector through the same module-to-module comparisons using the same pseudo-attenuation measurement. Figure 44 shows the result of a sample scan [26].

Each APA frame held ten PDS modules, inserted on rails, equally spaced along the full length of the APA frame. The spacing between modules along the $y$ direction was approximately 60 cm. PDS modules were inserted into the frames of completed APAs, between the sets of wire layers. Two ARAPUCA devices were mounted in ProtoDUNE-SP, one in APA 3 directly in front of the beam plug for the observation of photons from beam interactions, and the other in the middle of APA 6 to observe photons from cosmic particles. The two light-guide designs filled the remaining modules in alternating positions in the APAs. Once the PD modules were installed into an APA, the entire APA and its components were tested in the cold box. The full PDS chain, including photosensors, cold cables, warm cables, SSP, DAQ, and the connection to the slow control system was then tested. The full test included tests for functionality, stability, trigger rates, threshold calibrations, and PD module response and comparison, as well as thermal-cycling at $\sim 150$K. With respect to the slow control and DAQ, nominal configuration settings were implemented in GUI format, with additional configurations that could be uploaded at run time.

Data rates were also tested using the DUNE Light Calibration Modules (LCMs) and the Detector Control System data limits were established and optimised for data taking. The cold box helped determine if live data were sufficient by using the data to prepare for online monitoring diagnostics, such as persistent traces, leading-edge amplitude histograms, integral waveforms, average waveforms, and internal/external triggering for every PDS channel. The functionality of each PDS channel was verified by inspecting waveforms using persistent traces in the online monitoring system, as shown in Figure 45. The cold box tests and online monitoring allowed the observation of overall clean waveforms in the data. They also provided an opportunity to observe faults with the grounding or disconnects in the SSPs, channels, and cables. The cold box tests also allowed preliminary module-to-module comparisons.

The cryogenic PD cables were installed at the same time as the PD modules. Each APA has ten Cat6 PD cables, one per module, with alternating installation sides. During installation of the APAs in the cryostat, cables were routed through feedthroughs in the cryostat roof. From there, the cables were connected to the SSP modules (see Figure 46). During the cold box tests and after installation, the PDS was fully interfaced with both the Detector Control System (DCS) and the monitoring system. Photosensors can be biased and unbiased from the DCS, their status can be displayed, and the bias voltage can be modified manually or according to different preset conditions, as shown in Figure 47.

A UV-light-based calibration and monitoring system, designed and fabricated by Argonne National Laboratory, is used to calibrate SiPM gain and cross-talk, and to monitor linearity, time resolution, and long-term stability of the system. The system hardware consists of both warm and cold components. The system has no active components within the cryostat and in no way alters the operation of the PDS or the HV system. The active system component consists of an external 1U rack-mount Light Calibration Module (LCM). The LCM generates 275 nm UV-LED light pulses that propagate through a quartz fiber-optic cable to diffusers located at the CPA. The calibration module consists of an FPGA-based control logic unit coupled to an internal LED Pulser Module (LPM) and an additional bulk power supply. Light diffusers located on the CPA surface uniformly illuminate the APA surface and hence the PDS light collection modules. Five light diffusers on each face of the CPA plane are used: one in the center and four in each of the four CPA corners, as shown in Figure 48.

The calibration system produces UV light flashes with a predefined pulse amplitude, pulse width, repetition rate, and pulse duration. It also provided an external trigger for the light collection system. Pulse multiplicity control offers the option to produce two pulses at a fixed time difference to study timing properties of the photon system as well as trigger delays. The UV light system was used as a complement to cosmic-ray muons for calibration.

The performance of the full PDS as measured during ProtoDUNE-SP operations is reported in detail in the ProtoDUNE-SP performance paper [3]. A summary is included here for completeness. All channels exhibit a linearity of gain response to varying bias voltage ($\mathrm{V_{B}}$). The actual breakdown voltage ($\mathrm{V_{bd}}$) of the multi-sensor channel at LAr temperature, however, shows a relatively large spread, particularly for the 12-H-MPPC channels of the ARAPUCA modules. For these 12-H-MPPC channels, the signal-to-noise (SNR) values are around 6, while for the 3-S-SiPM channels of the double-shift and dip-coated bar modules the SNR is in the range 10 to 12. Within the uncertainties of the calibration measurements taken during operations, neither the gain nor the other parameters were found to drift significantly over time for any of the sensors used in the ProtoDUNE-SP PDS. The photon detection efficiency was evaluated through eight independent measurements using muon and electron data at four different beam momenta, supplemented by simulation. Photon detection efficiencies average 2% for the single area ARAPUCA cell, 0.21% for the double-shift module, and 0.08% for the dip-coated modules (see Table 4 in [3]). An extrapolation of the performance to a PDS system consisting entirely of ARAPUCA modules indicates that it can perform calorimetric energy reconstruction with an expected light yield of 1.9 photons/MeV at 3.3 m from the anode. This performance exceeds the specifications of the DUNE far detector by almost a factor of four.

Achieving an electron lifetime that is long enough so that electrons can drift sufficient distances is a challenging aspect of LArTPCs. The electron (e-) lifetime in a LArTPC is inversely proportional to, and determined by, the electronegative impurity concentration in the LAr, making the LAr purity an essential concern for successful operation and physics reach of the detector. Electron loss due to electronegative impurities in a LArTPC can be parameterised as $N(t)=N(0)e^{-t/\tau}$, where $N(0)$ is the number of electrons generated by ionization and not recombined with argon ions, $N(t)$ is the number of electrons after drift time $t$, and $\tau$ is the electron lifetime. ProtoDUNE-SP was designed to have the same 3.6 m drift distance and 500 V/cm electric field as planned for the DUNE-SP far detector modules. Given the drift velocity of approximately 1.5 mm/$\mathrm{\mu s}$ in this field, the drift time from cathode to anode is roughly 2.3 ms. Thus, if the electron lifetime is $\mathrm{2.3ms}/[-\ln(0.8)]\simeq\mathrm{10~ms}$, then the LArTPC signal attenuation, $[N(0)-N(t)]/N(0)$, remains less than 20% over the entire drift distance.

Purity monitors are used to independently infer the effective free electron lifetime in a LArTPC. It works by illuminating a photocathode with deep UV light to generate an electron current, then collects the drifted current at an anode a known distance away. Attenuation of the current is related to the electron lifetime. In the ProtoDUNE-SP purity monitors the UV light is generated by an external xenon light source and delivered by quartz fibres to the inside of the cryostat. The fraction of photoelectrons generated at the purity monitor cathode that arrives at the anode ($Q_{A}/Q_{C}$) after the drift time $t$ is a measure of the electron lifetime $\tau$: $Q_{A}/Q_{C}=e^{-t/\tau}$, where $Q_{C}$ is the combined charge of the electrons generated at the photocathode, $Q_{A}$ is the combined charge of the electrons collected by the anode after drift time $t$, and $\tau$ is the electron lifetime.

The purity monitors are placed inside the cryostat but outside of the TPC volume due to their size.

Although ProtoDUNE-SP receives ample cosmic ray data to perform electron lifetime measurements, the purity monitor system was found to be essential for providing quick, reliable, real-time information. It enabled operators to catch purity-related changes caused by LAr recirculation issues in time to correct them.

In addition, since the purity monitors have much smaller volumes than the LArTPC, the measurements from this system are affected to a much smaller degree by the space charge caused by cosmic rays. Since purity monitors provide run-by-run electron lifetime measurement in liquid argon, they have unique importance for DUNE’s deep-underground far detector charge calibration, where the cosmic-ray-based calibration is very challenging due to the low cosmic statistics.

The purity monitor design follows that of the monitors used in the ICARUS experiment [16] (Figure 55). It consists of a double-grid ion chamber with four parallel, circular electrodes: a disk holding a photocathode, two grid rings (one each in front of the anode and the cathode), and an anode disk. The cathode grid (labelled “ground-grid” in the figure) is held at ground potential. The cathode, anode grid, and anode can each be independently biased via modified vacuum-grade HV feedthroughs. The anode grid and the field-shaping rings are connected to the cathode grid by an internal chain of 50 $\mathrm{M\Omega}$ resistors to ensure the uniformity of the electric field in the drift regions. A stainless steel mesh cylinder is used as a Faraday cage to isolate the purity monitor from external electrostatic background.

Low signal strength has limited the precision and measuring ranges of purity monitors in previous LArTPC experiments [16, 12]. To boost the signal strength for use in ProtoDUNE-SP, UV light was delivered to each purity monitor by eight fibres and an 8-channel feedthrough, resulting in signal magnitudes six times larger than with a single fibre. This increase came without penalty in timing resolution.

The electron drift time is a key indicator of the measurement range and precision of the purity monitor. The electron drift time depends on the cathode–anode distance and the applied voltage within the monitor. During the commissioning and beam phases of ProtoDUNE-SP the electron drift time within the detector varied from 150$\mathrm{\mu s}$ to 3 ms. The purity monitors were operated at different high voltage settings that covered an electron drift time measurement range of 35$\mathrm{\mu s}$ to more than 10 ms. The electron drift lifetime exceeded 3 ms during the beam phase.

Three purity monitors were installed at heights of 1.8 m, 3.7 m, and 5.6 m from the bottom of the ProtoDUNE-SP cryostat to continuously monitor the LAr purity during all phases: commissioning, beam test, and operation. Figures 56 and 57 show the anode-to-cathode signal ratios $Q_{A}/Q_{C}$ and the electron lifetime measured by each purity monitor from the commissioning phase, which started in September 2018, through the entire beam test, which ended in November 2018, and continued through the non-beam operations, which ended in February 2020. The shaded bands represent uncertainties in the measurements. All $Q_{A}/Q_{C}$ are normalised to a drift time of 2.3 ms.

Each electron-lifetime measurement by the purity monitors is based on the signal ratios $Q_{A}/Q_{C}$ from 200 UV flashes at the same location occurring within a 40 second window. These measurements, taken regularly, were able to indicate incipient circulation-related issues on a quick timescale, mitigating potentially serious consequences for the detector. They caught a filter saturation during LAr filling and recirculation pump outages; these incidents show as sudden dips in Figures 56 and 57. The measurement uncertainties in Figure 56 include statistical and time-dependent fluctuations as well as uncertainties in the baseline of the purity monitor signal waveform, the cathode and anode RC constants, the inefficiency of grid shielding, and the electrical transparency of the grids. The “inefficiency of grid shielding” refers to the inefficiency of the cathode and anode grids to shield against induced currents on the anode and cathode while electrons are drifting between the anode and cathode grids. The electrical transparency of a grid is the proportion of the electrons that pass the grid. Other uncertainties in such quantities as signal rise time and electron drift time were found to be small. The overall uncertainties in purity-monitor charge ratio measurement $\frac{\Delta(Q_{A}/Q_{C})}{(Q_{A}/Q_{C})}$ at 2.3 ms drift time for the purity monitors, from highest position to lowest, are $1.9\%$, $2.2\%$, and $3.9\%$, respectively.

Note that given $Q_{A}/Q_{C}=e^{-t/\tau}$, when $Q_{A}/Q_{C}=1$, the measured lifetime $\tau$ is infinity. As shown in 57, when the LAr purity is stabilised after filtering, $Q_{A}/Q_{C}$ is close to one, which means that the electron loss (due to impurities) after drifting for 2.3 ms is very small. The top and middle purity monitors measured the electron lifetime to be greater than 70 ms, and the bottom monitor measured it as greater than 30 ms. Considering the inverse relationship between the drift electron lifetime and the amount of oxygen equivalent impurity, the estimate predicts the impurity never went above 40 ppt equivalent of oxygen in the week of data-taking. At the end of beam data-taking on November 11th, 2018, the impurity in the detector can be estimated to be approximately 3.4$\pm$0.7 ppt oxygen equivalent [30, 3]. These measurements indicate that ProtoDUNE-SP exceeded the high LAr purity required by the DUNE far detector. They also corroborate the high electron lifetime measurements ProtoDUNE-SP has previously reported using TPC tracks matched to the CRT [3], [31].

Again given $Q_{A}/Q_{C}=e^{-t/\tau}$, the relative uncertainty in lifetime $\Delta\tau/\tau$ is propagated as $\Delta\tau/\tau=(\tau/t)\frac{\Delta(Q_{A}/Q_{C})}{(Q_{A}/Q_{C})}$. It follows then that for a given $Q_{A}/Q_{C}$ uncertainty of $\frac{\Delta(Q_{A}/Q_{C})}{(Q_{A}/Q_{C})}$, the relative uncertainty in the measured electron lifetime is proportional to the electron lifetime $\tau$ and inversely proportional to the electron drift time $t$. For a long electron lifetime, the relative uncertainty in this lifetime is large, as shown in Figure 57. However, since the main sources of uncertainties in $Q_{A}/Q_{C}$ are not correlated with the lifetime, $\frac{\Delta(Q_{A}/Q_{C})}{(Q_{A}/Q_{C})}$ is not significantly correlated with the electron lifetime or the drift time, and therefore the large relative lifetime uncertainty at high purity does not indicate lower precision in the purity monitor’s measurement of the charge loss during the electron drift time. The run-by-run purity monitor electron lifetime has been included in the ProtoDUNE-SP calibration database for TPC charge and energy calibration. The performance of the purity monitor based electron lifetime calibration tested with ProtoDUNE-SP cosmic rays is shown in Figure 58 [32]. After the purity-monitor-based lifetime calibration, the charge loss on the TPC signal due to LAr impurity is mostly corrected.

Since the purity monitors cannot be placed within the TPC volume, to monitor the cryogenics system and the LAr for homogeneous mixing, temperature monitors are used as a substitute. An extensive set of temperature measurements is recorded to create a detailed 3D temperature map of the detector volume that is used as input for the computational fluid dynamic (CFD) model validation. Results from these CFD simulations can predict the LAr purity across the entire cryostat volume. The vertical coordinate is particularly important since it corresponds closely to the LAr recirculation and uniformity.

In ProtoDUNE-SP 92 high-precision temperature sensors are distributed near the TPC walls in two configurations (see Figure 59): i) a formation of two high-density vertical arrays (called T-gradient monitors), and ii) coarser 2D horizontal arrays at the top and bottom of the TPC. The bottom sensors are mounted on the LAr return pipes and the top sensors on the ground planes.

Three elements are common to all systems: sensors, cables and readout. Lake Shore PT102 platinum sensors with 100 $\Omega$ resistance (at 0°C) were chosen based on previous experience from the 35 t prototype[33]. For the inner readout cables, a custom cable made by Axon’ was used. This 3.7 mm diameter cable consists of four AWG 28 teflon-jacketed copper wires, forming two twisted pairs, with a metallic external shield and an outer teflon jacket . The four-wire configuration eliminates any influence of wire resistance on the measurements.

Finally, the readout system consists of an electronic circuit that includes: i) a precise 1 mA current source for the excitation of the sensors, ii) a multiplexer reading out the different temperature sensors and forwarding the selected one to a single channel, and iii) a readout system based on the National Instrument Compact RIO Device with a high-accuracy voltage signal readout NI9238 module that provides 24-bit resolution over a 1 V range. In addition, 12 standard temperature sensors are in contact with the bottom of the cryostat to detect the presence of LAr when filling starts, and five standard sensors are lined up vertically and epoxied onto one of the lateral walls to measure the temperature of the cryostat membrane at different heights during cool-down and filling.

In addition to the distributed temperature sensors described above, a vertical array of 48 sensors was installed 20 cm away from the lateral field cage (see Figure 59-left). Vertical spacing is 11.8 cm for both the top and bottom 16 sensors and 23.6 cm for the 16 in the middle, with the bottom (top-most) sensor 30.2 (738.5) cm from the bottom surface of the cryostat. Mechanical rigidity is provided by an 80 $\times$ 25 $\times$ 3 mm³ U-shaped fibre glass profile (FGP) that holds sensors and cables. Given the proximity to the field cage, the entire system is surrounded by a Faraday cage made up of 19 vertical 6-mm-diameter stainless steel rods that are arranged to form a cylinder 12.5 cm in diameter. The rods and FGP are hung from the top of the cryostat and are mechanically decoupled from each other to allow for their different contraction rates in the LAr. The rods are passed through FR4 rings attached to the FGP to maintain the cylindrical shape.

Sensors are mounted on a 52 $\times$ 14 mm² PCB with an IDC-4 connector, such that they can be plugged in at any time. Cables are housed in the inner part of the FGP and run vertically from the sensor to the flange, which has eight Sub-D 25-pin connectors.

The temperature sensors are cross-calibrated in a controlled laboratory environment. During the calibration procedure, a set of four sensors (one of which is kept as a reference for all sets) are placed next to each other and submerged in LAr several times. The calibration procedure relies on the assumption of equal temperature for all sensors. Convection is minimised by placing the four sensors inside a 50 mm diameter, 1 mm thick cylindrical aluminum capsule (allowing fast cool-down in gas before immersion), which is placed in the centre of a 3D printed box with two independent concentric LAr volumes, and surrounded by 10 cm thick polystyrene walls. The stability of the calibration system, and the performance of the sensors and the readout system are shown in Figure 59-right. The accuracy of the calibration was estimated to be 2.6 mK [34]. This takes into account the uncertainty on the offset between any two sensors in the static T-Gradient monitor as well as differences between the four immersions in LAr.

Finally, a movable system of temperature sensors, called the Dynamic T-gradient monitor, was also installed in ProtoDUNE-SP. The Dynamic T-gradient monitor is equipped with a stepper motor connected to a carrier rod on which 24 temperature sensors are mounted. The carrier rod with sensors is contained inside the gas-tight enclosure shown in Figure 60-left. The stepper motor is mounted on the outside of the enclosure. Vertical spacing is 10 cm for the five sensors at the top and bottom, and 50 cm for the 14 sensors in the middle, where the bottom (top-most) sensor is 10 (750) cm from the bottom of the cryostat. Each sensor is soldered to a PCB. Cables are routed along the carrier rod and connected to a flange with Sub-D connectors on both sides.

The stepper motor engages with the carrier rod via a ferrofluidic dynamic seal and a pinion and gear that converts the motor’s rotational motion into linear motion of the rod without jeopardising the integrity of the argon atmosphere inside the enclosure.

The stepper motor can move the rod vertically, enabling temperature measurements at various heights in the TPC, and at the same heights by different sensors. Assuming a stable temperature profile over the calibration period of a couple of minutes, any offset in the temperature readings at a given height by different sensors can be attributed to the intrinsic offset between the two sensors, providing constants for their cross-calibration. By linking all adjacent sensors, it is possible to calibrate out all offsets with respect to a single one and achieve precise measurement of the relative vertical T-gradient. See Figure 60-right for the comparison of sensor readout before and after calibration. As will be shown in Section 4.5, final confirmation of the robustness of the system was achieved during the “pump-off” detector period when the temperature became uniform over the detector height.

The systems described above have been collecting data for more than two years, resulting in a deeper understanding of LAr dynamics inside a cryostat. Some important outcomes are discussed below.

Stability studies show that except near the LAr surface the difference in temperature between any two sensors in the same T-Gradient monitor remains constant to within 3 mK, demonstrating the stability of the LAr system under standard operating conditions, and importantly, the reliability and longevity of the sensors.

The slow and continuous process of progressively moving each sensor up (or down) to the location previously occupied by its neighbour is expected to yield a height profile that is effectively flat. Deviations from flatness would indicate a problem with one or more sensors.

Vertical and horizontal cross-calibration profiles of the Dynamic T-Gradient monitor have been compiled under various conditions (e.g., stable, with recirculation system on and off, at different pressures). The pumps-off (i.e., no recirculation) measurements are particularly interesting since small temperature gradients are expected and were in fact observed by the Dynamic T-Gradient monitor (see Figure 62-left). The flatness of this profile to within a few mK illustrates the homogeneity of LAr temperature. This measured homogeneity was used to re-calibrate the Static T-Gradient monitor (this is the so-called “pumps-off” calibration method) and the result shows good agreement with the laboratory calibration (see Figure 62-right).

These studies have demonstrated significant differences in temperature based on whether the recirculation system is on or off. These differences appear in both vertical and horizontal planes. The grid of sensors placed 40 cm apart over the cryostat bottom shows a temperate peak (20 mK) below the cathode plane with a decrease towards the pump on the bottom beam left when the recirculation system is on (see Figure 61-left), and a more homogeneous distribution when it is off (see Figure 61-right).

Finally, Figure 63 shows a comparison of the vertical profiles (under stable conditions) to CFD simulations for a variety of boundary conditions (e.g., varying flow rate and temperature of the incoming LAr). The CFD model reasonably predicts the main features of the data, but some details still need to be understood, e.g., the bump at 6.2 m and the lower measured temperature at the bottom of the cryostat. More studies on CFD boundary conditions are needed.

Cameras provide direct visual information about the state of the detector during critical operations and when damage or unusual conditions are suspected. Cameras can be used to verify stability, straightness, and alignment of the hanging TPC structures during cool-down and filling; ensure that no bubbling occurs near the ground planes; and inspect the state of movable parts in the detector (i.e., the dynamic temperature sensor). Eleven cameras were deployed in ProtoDUNE-SP at the locations shown in Figure 64. They successfully provided views of the detector during filling and throughout its operations. Two types of cameras, were deployed, “cold” for fixed, long-term use and “warm” for short-term inspections.

The cold fixed cameras monitor the following items during filling:

• •

  positions of the corners of each APA, CPA, FC, and GP (1 mm resolution);

• •

  relative straightness and alignment of the APAs, CPA, and FC ($\lesssim 1\,mm$)

• •

  relative positions of profiles and endcaps (0.5 mm resolution); and

• •

  the LAr surface, specifically, the presence of bubbling or debris.

One design for the ProtoDUNE-SP fixed cameras uses an enclosure similar to the successful EXO-100 design [35], see Figure 65. Cameras 101, 102, 104, and 105, shown in Figure 64, were installed in this type of enclosure. A thermocouple in the enclosure allows temperature monitoring, and a heating element provides temperature control. SUB-D connectors are used at the cryostat flanges and the camera enclosure for signal, power, and control connections. An alternative successful design uses an acrylic enclosure, see Figure 65, bottom left. Cameras 001, 002, 004, and 005, shown in Figure 64, are placed in acrylic enclosures. All have operated successfully.

The inspection cameras are selected to be as versatile as possible to cover the range of intended uses. The following inspections have been done with the warm cameras:

• •

  status of HV-feedthrough and cup,

• •

  status of FC profiles, endcaps (0.5 mm resolution),

• •

  vertical deployment of calibration sources,

• •

  status of thermometers, especially dynamic thermometers,

• •

  HV discharge, corona, or streamers on HV feedthrough, cup, or FC,

• •

  relative positions of profiles and endcaps (0.5 mm resolution), and

• •

  sense wires at the top of the outer APA wire planes (0.5 mm resolution).

Unlike the fixed cameras, the inspection cameras operate only during an inspection. It is more practical to use commercial cameras for this purpose, which requires keeping the cameras warmer than -150^∘C during deployment. Cameras of the same commercial model were used successfully to observe discharges in LAr from a distance of 120 cm [36].

In this design, the warm camera is contained inside a gas-tight acrylic tube inserted into the feedthrough, and can be removed for servicing, upgrade, or replacement at any time. Figure 66 shows an acrylic tube enclosure and camera immediately before deployment. These acrylic tube enclosures were deployed at the positions marked 201, 202, and 203 in Figure 64, and equipped with cameras with fish-eye lenses during initial operation. One camera was then successfully removed without any evidence of having contaminated the LAr. The other cameras were left in and used during post-beam operations.

The light-emitting system uses LEDs to illuminate the parts of the detector in the cameras’ fields of view with selected wavelengths (IR and visible) that the cameras can detect. Performance criteria for the light-emission system include the light-detection efficiency of the cameras and the constraint on heat generation inside the cryostat. Very high-efficiency LEDs help reduce heat generation; a 750 nm LED [37] with a specification equivalent to 33% conversion of electrical input power to light was used.

One set of IR-LEDs is mounted directly under camera 002 near the beam-left manhole, shown in Figure 67. It is constructed from the IR LED modules that were removed after the initial operation. Each LED module has 30 LEDs. This set of LEDs draws about 2.4 A of current at 12 V at room temperature, and about 1.8 A in LAr. The assembly is installed vertically, with the LEDs aimed upstream of the beam entry behind the beam left APAs. When powered, the centres of the LEDs have a slight red glow.

Additionally, chains of LEDs connected in series and driven by a constant current are used for broad illumination (see Figure 67 right), with each LED paired in parallel with an opposite polarity LED and a resistor (see Figure 68). This provides two different wavelengths of illumination by simply changing the direction of the drive current, and allows continued use of the chain even if an individual LED fails.

As each arriving APA container is opened inside EHN1, special lifting fixtures are attached to each end of the APA, then attached to two conveyances that lift the APA out of the container, and rotate it 90^∘ to vertical. The sequence is shown in Figure 70.

Once an APA is properly oriented, the lifting strap and fixtures are removed from its lower edge, the roof hatch on the material sas is opened, and the APA is lowered through the hatch. It is then transferred to a rolling trolley attached to a series of rails, and moved into the clean room. The APA then goes through a series of acceptance tests for both electrical integrity and wire tension, as well as an inspection for broken wires or any other damage that could have resulted from shipment and handling. At this stage, survey of the APA geometry was also performed.

Next, ten photon detectors (PDs) are inserted into alternating sides of the APA frame, five from each direction. After insertion, a PD is attached mechanically to the APA frame with fasteners, and a single electronics cable is attached and strain-relieved. Each PD is tested immediately after installation to ensure proper operation and to verify the cable readout. The design model and a photograph of an APA in the clean room during PD installation are shown in Figure 71.

After PD installation, CR boards are mounted on the geometry boards then 20 cold electronics (CE) boxes are installed at the top of the APA frame. Figure 72 indicates the location of the CE boxes on the APA and shows them being installed. These boxes are connected via matching electrical connectors on the FEMB. Mechanical fasteners affix the CE enclosure to brackets supported by the APA frame. Prior to their installation, the CE boxes undergo testing several times at room temperature for noise and channel response. Only those that are fully functional and have ENC levels typical for room temperature operations, 1000 – 1500 e^-, are installed on the APA.

The details of the warm tests and their results can be found in [38]. In addition to the TPC read-out electronics, each CE box includes a bundle of data and power cables that connect the electronics to WIEC (warm interface electronics crate). Cable bundles were grouped together at appropriate lengths for the WIEC connections, and secured on the cable trays.

At this point, the APA is moved via the rails to the integrated cold test stand where the warm and cold tests are performed (see Section 5.3).

After completing the cold test procedure the APA is slowly warmed up back to room temperature. Then the cold box is opened, the cables are disconnected and secured, and the APA is extracted from the box on the rail system, ready to be moved into the cryostat and positioned for the final installation. Installation inside the cryostat is described later in this section.

Upon arrival at EHN1, each individual CPA module, weighing roughly 24 kg, is lifted out of its shipping crate by hand.

Three 1.16 m wide, 2 m tall CPA modules are placed on a flat table surface and screwed/pinned together end-to-end to form a 6 m tall CPA column. Then a crane attaches at the top end of the CPA column with appropriate lifting straps and shackles, and rotates it to vertical.

As each successive CPA column is ready, it is affixed adjacent to the previous one lengthwise with 1 mm separation. Alignment is provided by two pins located on one side of each CPA that fit into a vertical slot on the adjoining side of the next CPA.

Three basic elements comprise the FC: the top, bottom and end-wall FC assemblies. The top and bottom FC assemblies are effectively mirror assemblies that are hinged from the top and bottom of the CPAs, respectively. Figure 73 (upper left) is a schematic of a top/bottom FC assembly in which the ground plane covers one side of the field-shaping profiles. The upper right image in the figure illustrates a CPA pair with top and bottom FCs attached. The lower two images show units constructed inside the NP04 clean room.

An end-wall FC assembly includes four stacked end-wall FC modules, stacked via the overhead hoist near the TCO. Figure 74 shows two images of an end-wall module, the design model and an assembled unit hanging on the rail system inside the clean room. Each of these assemblies was moved into the cryostat via the rails and positioned on the appropriate beam in the detector support system (DSS). The end-wall FC is supported by a spreader bar (hung from the same beam) that can swivel about the support point in order to allow proper positioning with respect to the APA and CPA.

The tight ProtoDUNE-SP schedule imposed a requirement that the DCS implement a proven and reliable solution. The commercial Supervisory Control And Data Acquisition (SCADA) WinCC-OA [49] from Siemens, widely used at CERN, was selected as the basis. The ProtoDUNE-SP DCS implements two CERN frameworks on top of WinCC-OA: the Joint Controls Project (JCOP) [50] and the Unified Industrial Control System (UNICOS) [51]. These frameworks provide guidelines, components, and tools designed to facilitate the implementation of a homogeneous control system. Their most salient features are:

• •

  automatic Programmable Logic Control (PLC) project generation;

• •

  a number of sets of predefined widget tools, faceplates, a Finite State Machine (FSM) toolkit, and a generic configurable Human Machine Interface (HMI);

• •

  an alarm and alert configuration system with automatic mail and SMS notifications, derived from the system used by the LHC experiments;

• •

  an access control component providing detailed authorisation schemes to assign specific privileges to different user groups;

• •

  a run-time database, accessible for processing, visualisation, etc., in which to store the data coming from the devices;

• •

  an archiver for long-term storage of the runtime database and offline use of that data;

• •

  a C-compatible scripting language with a very large library of functions that can be used to implement all WinCC-OA functionality;

• •

  standard libraries for commonly used hardware (e.g., CAEN and Wiener power supplies) that allow for centralised configuration of the implemented functions and parameters such as alarms, archiving parameters, operational values, and default panels, all at the crate, board, or channel level;

• •

  support of a wide range of drivers and communication protocols such as Siemens S7, MODBUS, OPC DA, OPC UA, the CERN Distributed Information (DIM)[52], and Data Interchange Protocol (DIP)[53].

To minimise the risk of unintended operator actions and maximise the stability of the system, the DCS interface presents panels that are designed to be as simple and straightforward as possible. The interface relies on standard color coding (see Figure 84) and is based on two main concepts:

• •

  dynamic graphical objects that allow the operator to navigate to dedicated panels for each detector subsystem, and

• •

  data widgets, objects displayed as data values that allow operations to be performed on them, such as plotting historical values or checking status.

For basic control operations, the DCS uses the Finite State Machine (FSM) component from JCOP [54]. The FSM acts as an abstraction layer that simplifies control of the detector. Based on a well defined set of states and transitions, each part of the detector can be controlled hierarchically. Once the topology of the detector is defined, the FSM component is responsible for propagating the actions sent to the different parts of the detector and obtaining, as a result of these actions or asynchronous incidents, their corresponding states. An example of the FSM used for the detector power control is shown in Figure 85. Access Control, implemented in the FSM, ensures the safe operation of the detector.

For advanced detector operations, specific and more detailed panels have been designed. Rather than using an FSM for putting the detector – or any of its components – in a preset state, the advanced panels allow users with appropriate authorisation full control over all parts of the detector. The advanced panels control the lowest-level architecture of the detector, allowing the authorised experts to modify operational parameters, load pre-set configurations from files, set limits for alerts, and/or directly control critical devices beyond the scope of the FSM.

The DCS subsystems are connected to a dedicated ProtoDUNE-SP detector network, known as the NP04 network. The DCS can communicate with the individual subsystems using a variety of communication protocols, indicated by the white arrows in Figure 86. Furthermore, the DCS is accessible over the Internet using a remote connection protocol, and the most relevant data are published to a web server for real-time off-site monitoring. A brief description of the different subsystems, and how the DCS manages them, is given below.

• •

  The Detector Power Control (DPC): The DPC is a set of power supplies that provide power to the different parts of the detector, including the cold electronics, the PDS, the HV for the field cage, and the CRT. These power supplies are connected to the main DCS application by means of the OPC DA driver [55]. The JCOP mass configuration tool is used to set all the power supply parameters, e.g., operating values, addresses, and archiving parameters, and to populate the default control panels.

• •

  TPC cold electronics (CE): To assess their status, the DCS continuously monitors the main parameters of the CE crates, WIBs, and FEMBs through the detector network by means of the DIM protocol. A hardware interlock between the CE cooling fans and the CE low-voltage power is in place to prevent powering on of the CE in case of fan failure.

• •

  Photon Detector system (PDS): The DCS provides an interface with the PDS for monitoring its working conditions and tuning its configuration for accurate operation. These configurations can be stored in a file and imported by the DCS. The interface is based on the DIM protocol. Critical interlocks have been implemented in the DCS to prevent any potential damage to the PDS when the purity monitors’ xenon lamp flashes or the cryostat LEDs are turned on.

• •

  HV system: The DCS for the HV system is composed of two tightly correlated components: the controls of the Heinzinger power supply [56] that provides HV to the TPC cathode, and the current monitors on the ground planes and beam plug. A dedicated National Instruments cRIO FPGA [57] performs the HV control process through a high-resolution analogue interface. A novel feature, called Recovery Mode, was developed to resolve streamer events (extended discharges, see Section 4.1.4), which are characterised by high current draw. Recovery mode automatically lowers the HV setpoint while continuously monitoring the current draw at the power supply until the current draw is below the current limit. It then restores the HV setpoint to its nominal value.

  An example of an automatic streamer recovery is given in Figure 26. The current signals from the 12 ground planes and the beam plug are connected to the same HV control system. In the case of a ‘stream’ or discharge event, the DCS records the main HV parameters at 20 kHz for 5 seconds with a pre-trigger of one second into a fast-acquisition file that it sends to the DAQ system for off-line analysis.

• •

  Purity monitors: The purity monitor system is controlled locally by a PCIe Alazartech ATS310 digitiser installed in a computer dedicated to this task. The local user interface was developed in LabVIEW software, which includes an OPC UA server, communicating with the main DCS as a client. The DCS provides a dedicated user panel for monitoring its status and setting the configuration for operation. The DCS can select different internal digitiser working modes and choose the appropriate HV configuration. Once configured, the DCS supports automatically running the purity monitor DAQ with one click, after which it displays all the relevant information. As previously mentioned, to protect the PDS, the DCS implements an interlock system to prevent the purity monitors from operating during data-taking runs.

• •

  Gradient Temperature sensors: Temperature sensors (PT100) are installed in the ProtoDUNE-SP cryostat to monitor the temperature gradient of the liquid with high precision ($\pm$ 2mK) (see Section 4.5.2). A customised electronic interface developed by the collaboration is connected to a National Instruments cRIO FPGA that publishes all temperature values to the main DCS database through an OPC UA server. The same interface controls the position of the dynamic gradient sensor. The data stored in the database is used for off-line calculations and simulations. The DCS also publishes the maximum and minimum temperatures of the static gradient via DIP; these are used by the cryogenics system for alarm reporting. Figure 87 shows an example of a liquid argon temperature measurement taken during standard operations. A comparison with the predicted values from the Computational Fluid Dynamics (CFD) simulation is also shown.

  

• •

  Ground Impedance Monitor: As described in Section 5.4 the detector ground is isolated from the building ground, and the DCS monitors the ground decoupling through a Ground Impedance Monitor (Zmon). This device is connected to the main DCS by means of a DIM protocol. If a grounding problem is detected, the ground impedance monitor sounds an alarm. The DCS alerts operators and records incoming data.

• •

  Multi-purpose PLC: A S7-1500 Siemens PLC connected to the main DCS application by means of the S7 driver[58] performs several tasks using the CERN UNICOS framework:

  • –

    controls and monitors the electrical network and ambient parameters (temperature, atmospheric pressure, humidity);

  • –

    performs the interlock logic of the CE LV power supply according to the cooling crate status;

  • –

    reads the relative and absolute pressures inside the cryostat;

  • –

    performs the control and interlock logic of the LED light installed inside the cryostat;

  • –

    operates the cameras and the associated heaters;

  • –

    controls heaters on the CE and PDS flanges to dry out water condensation; and

  • –

    ensures the connection to the HBM system that reads the cryostat instrumentation gauges (see Section 3.2.2) and publishes the values for each sensor to the DCS.

• •

  DAQ environment control: Monitoring the racks where the DAQ system is installed is also critical for ensuring the correct recording of the ProtoDUNE-SP data. One Siemens S7-1500 PLC controls and supervises the environmental conditions and the main electrical and cooling parameters. In case of a fault, this PLC turns off all DAQ rack power to avoid overheating. A second Siemens S7-1500 PLC is connected to all internal DAQ rack door controls by means of a MODBUS network that acquires the individual rack door statuses and configuration parameters. Both PLCs are integrated in the main DCS through a Siemens S7 driver. They were developed using the CERN UNICOS framework.

• •

  Cryogenics system: The DCS and cryogenics systems continuously exchange information; any incorrect cryogenic condition may affect the detector’s operation. As the cryogenics control system is installed in a different network, the data exchange is performed by means of a DIP protocol that only allows data subscription by an external system.