ELT-scale Adaptive Optics real-time control with the
Intel Xeon Phi Many Integrated Core Architecture

The dependence of AO system requirements on telescope diameter is an extremely important consideration for the next generation of Extremely Large Telescopes (ELTs), including the Giant Magellan Telescope (GMT, Johns et al., 2004), the Thirty Meter Telescope (TMT, Stepp & Strom, 2004) and the European Southern Observatory (ELT, Spyromilio et al., 2008), with primary mirror diameters greater than $20\text{\,}\mathrm{m}$. The computational complexity of the conventional wavefront reconstruction algorithm scales with the fourth power of telescope diameter; this is due to the dimensions of the control matrix which are governed by the number of correcting elements in the DM and the number elements of the WFS which both scale to the second power of telescope diameter. This presents a huge challenge in the process of designing an RTC suitable for ELT scale AO, both in the choice of hardware suitable to process the computational demands and with producing software capable of delivering performance that meets the requirements of the AO system.

Many typical AO reconstruction techniques involve computing one or more large matrix-vector-multiplications (MVMs) where the dimensions of the matrix are defined by the number of degrees of freedom (DoF) in the system. A control matrix, which contains information relating to how certain wavefront measurements correspond to the appropriate actuator commands, is multiplied by a vector containing the wavefront slope measurements, the results of which can yield the wavefront shape required for correction. A control matrix of dimensions (NxM) is made up of N DoF of a wavefront slope measurement and M actuators in the correcting element. This makes the MVM very large for ELTs requiring on the order of $N^{2}$ calculations where $N\approx M$.

The large MVM calculation is a memory-bandwidth bound problem, since the processing time is dependent on the rate that the processing unit can read the large matrix from memory (typically several GB), each AO RTC cycle. This therefore favours computational architectures with large memory bandwidth. The Intel Xeon Phi (Intel, 2017) is one such architecture, containing a $16\text{\,}\mathrm{GB}$ MCDRAM package, with a measured bandwidth as high as $480\text{\,}\mathrm{GB}\text{\,}{\mathrm{s}}^{-1}$. Modern graphics processing units (GPUs) also have large memory bandwidths. However, a standard x86 CPU will typically have an achievable memory bandwidth of <$90\text{\,}\mathrm{GB}\text{\,}{\mathrm{s}}^{-1}$ and is therefore at a massive disadvantage for rapid processing of large MVMs. They are therefore not well suited for ELT-scale AO RTC. The Xeon Phi contains a large number of cores, has a large memory bandwidth, and is also programmable as a conventional processor.

Knights Landing (KNL) is the third generation Xeon Phi processor and is the first to be released in the self-booting socketed form factor, with a number of variants, as given in table 1. Previous generation Xeon Phi chips were available as accelerators only. The KNL processor can therefore be used just like a conventional server processor and can run the Linux operating system and standard software environment. Existing applications can be ported to the Xeon Phi quickly: recompilation is usually not even required, though code will not be well optimised in this case.

The KNL introduces additional instruction sets, which can be utilised for improved performance, including $512\text{\,}\mathrm{bit}$ CPU registers, which allow Single Instruction Multiple Data (SIMD) operation on 16 single-precision floating point numbers simultaneously per core, per instruction cycle. This improves the parallelisation advantage of the KNL architecture over previous processors, which included a maximum of $256\text{\,}\mathrm{bit}$ wide vector registers. This $512\text{\,}\mathrm{bit}$ register is also included in forthcoming (and most recent) standard Intel CPUs, so any code optimisations made for this feature will also be applicable to future non-Xeon-Phi CPUs. However, it should be noted that for the KNL system, high utilisation of 512 bit instructions reduces the base core clock speed by 200Hz, which is taken into consideration in the peak SP TFLOPS calculated in Table 1.

The wavefront sensor cameras can be directly attached to the KNL via the PCIe bus. This is an advantage over accelerator cards such as previous the generation Xeon Phi cards and GPUs, where, unless specific effort is taken (often requiring specific network cards, and low level device programming), image data must be transferred first to the CPU from the camera, and then out to the accelerator (essentially 2 PCIe transfers, with increased latency and jitter). The previous generation of the Xeon Phi (Knights Corner) has also been investigated for AO RTC (Barr et al., 2015), showing promise for future processor generations (i.e. the KNL).

The operating system (OS) installed on the Xeon Phi used in this paper is CentOS Linux 7.3 (The-CentOS-Project, 2001). To obtain the best low latency and low jitter performance various changes have been made to the default settings of the BIOS, the operating system and the kernel. The main changes to the BIOS settings involve turning off Intel Hyper-threading, which allows more logical threads to execute concurrently on hardware cores. Removing Hyper-threading allows each software thread to be pinned to a single hardware core and removes scheduling inefficiencies caused when cores switch between different Hyper-Threads.

Other BIOS settings include Xeon Phi specific settings which relate to how the CPU handles memory addressing, with information available online (Intel, 2015), and different modes which determine how the fast Multi-channel DRAM (MCDRAM) is allocated, either accessible like standard RAM, reserved for the OS as a large last level cache (LLC), or a mixture of the two; these modes are termed ’flat’, ’cache’, and ’hybrid’ respectively.

OS and kernel setup refers to options such as isolating certain CPU cores so that the OS doesn’t schedule any program to run on these cores without specific instruction, and other options relating to CPU interrupts and different power and performance modes.

During our testing, we have identified that best performance is achieved with the CPU set to Quadrant memory addressing mode, and the MCDRAM was set to ’flat’ mode. In ’flat’ mode, the MCDRAM is visible to the CPU on a separate NUMA node from the standard RAM and so this must be addressed either by explicitly allocating the memory in the program (using a NUMA library), or by executing the program on the specific NUMA node to make use of the fast MCDRAM. In this report the MCDRAM was allocated by running software with the ‘numactl’ command with the ‘–membind=nodes’ option, ensuring that the entire RTC is allocated on this NUMA node. On the Xeon Phi, the MCDRAM is 16 GB in size, which is sufficient to fit a whole ELT-scale RTC.

Multi-core CPU systems have become the norm in recent years leading to DARC being developed using a multi-threaded real-time core with the POSIX (’The-Open-Group’, 2016) pthread library. The main method of ensuring thread synchronisation has been by the use of pthread mutexes and condition variables. A mutex is a mutual-exclusion variable which allows threads to ‘lock’ a certain section of code, preventing other threads from accessing these protected regions. If a thread calls the lock function on an unlocked mutex variable, then that thread will be allowed to proceed and then the mutex becomes locked. Any other threads which attempt to lock this mutex will have to wait at the lock function until the mutex is unlocked.

If multiple threads are waiting at a mutex then they will proceed one by one as the mutex is repeatedly locked and unlocked by the preceding thread. A thread waiting at a mutex will generally be descheduled by the operating system scheduler and put to sleep, reducing power consumption and freeing up the hardware for other threads to be scheduled. This works well for low order multi-core systems with 2-16 CPU cores, as it allows for more threads than physical CPU cores and the simultaneous descheduling and rescheduling of these few threads when they are waiting at the same mutex has little overhead.

However, for the Xeon Phi MIC architecture with $\geq 64$ low power cores, DARC needs to be configured to execute a single thread per core with > 48 threads to achieve maximum performance for ELT-scale SCAO (Figure 4). This can cause problems when using mutexes and condition variables as the constant sleeping and waking of this large number of threads significantly increases latency. The solution that we have developed is to use a structure similar to mutexes called spinlocks, which also protect critical sections of parallel code but instead of sleeping and descheduling, threads simply wait until they can proceed. This waiting process constantly consumes CPU cycles but increases the system’s responsiveness which massively reduces the latency when using many threads as it is then much faster to resume operation.

Unfortunately, condition variables do not work with spinlocks and so we replace these where possible by simple volatile flag variables, taking care to ensure that thread safety is maintained.

The $512\text{\,}\mathrm{bit}$ wide vector registers present on the Xeon Phi allow up to 16 single precision (SP) operations to be performed per cycle per CPU core. An operation in this case can be a fused-multiply add (FMA) operation which combines an addition and a multiplication, allowing up to 16 SP additions and 16 SP multiplications per instruction cycle. This is double the previous specification of $256\text{\,}\mathrm{bit}$ vector registers allowing a theoretical 2X speed up for vectorisable computations. Vectorisation is generally handled by the compiler: depending on the level of optimisation chosen at compile time, a certain amount of auto-vectorisation will occur. However, steps can be taken to aid the compiler and investigate where vectorisation occurs or does not occur. Essentially, if the compiler is able to detect that vector or matrix operations include 16-float boundaries at the same points, then these operations can be vectorised. This therefore usually means that by aligning memory to the nearest 64 bytes, vectorisation will be aided.

The allocation of subapertures to specific threads within DARC can be optimised such that each thread processes a multiple of 16 slope measurements when calculating its own section of the wavefront reconstruction MVM. As each subaperture has 2 slope measurements, one for each of the x and y directions, we therefore ensure that the subapertures are allocated to threads such that each thread processes a multiple of 8 sub-apertures as a block.

Alignment of array memory to page cache boundaries is important so that the data required for the vectorised instructions can be loaded into the registers efficiently and with the right ordering. This can be done when allocating memory for the arrays using the posix _memalign (’The-Open-Group’, 2016) function call which aligns the amount of memory required at the specified boundary. The next step is to then ensure that sections which can be vectorised are written in such a way that the compiler can apply auto-vectorisation; Intel provides a guide which details the necessary steps (Intel, 2012).

Because the wavefront reconstruction MVM is memory bandwidth bound, due to the relatively simple mathematical operations but large data size, investigating ways to reduce the memory bandwidth dependence is an important consideration for ELT-scale AO RTC. A potential solution is to store the control matrix using $16\text{\,}\mathrm{bit}$ floating point format, rather than the conventional $32\text{\,}\mathrm{bit}$ format. The format used for the $16\text{\,}\mathrm{bit}$ floats is the IEEE 754 specification for binary16 (IEEE, 2008) which reduces the exponent from $85\text{\,}\mathrm{bit}\mathrm{s}$ and the mantissa from $2310\text{\,}\mathrm{bit}\mathrm{s}$.

This change does result in some loss of precision in the control matrix, however the available precision is still greater than that of the wavefront slope measurements (which are based on integer-valued detector measurements) and is therefore still considered sufficient for the reconstruction (Basden et al., 2010a). Every AO loop iteration, this control matrix is then loaded into CPU registers, converted to $32\text{\,}\mathrm{bit}$ format for operations (necessary since the Xeon Phi cannot perform $16\text{\,}\mathrm{bit}$ floating point mathematical operations), and the DM vector computed. The reduction in memory bandwidth required can therefore reduce AO system latency.

As each DARC thread processes a set of subapertures from beginning to end, the result of each thread’s execution is a partial DM vector for those subapertures. To combine these results into a final DM command vector they must be all be summed together. Previously DARC has achieved this by using a mutex to lock the final DM command vector whilst each thread adds its vector into it in turn. This works well for small numbers of threads on fast cores, however for the KNL case where there are more threads on comparatively slower cores, this serial addition can be a source of a large amount of extra latency.

A solution that we have developed is a branching tree-like algorithm which allows groups of threads to add their partial DM vectors and so each group can work in parallel. A group will be defined by a spin-lock and a thread-barrier. Within the group a thread will get the lock whilst it copies its partial DM vector into a temporary output array and once each thread has copied it’s partial vector it waits at the barrier for the other threads in the group to finish. At the completion of a groups work, one thread from each group will take ownership of the temporary output array and move on to the next group. This can be seen in Figure 5 where each group adds its partial DM vector into the red box before that moves down to the next group where the process is repeated until the final DM command vector results.

The latency of an RTC is defined as the time from last pixel to readying the final DM command and so reducing this time interval is paramount to improving the performance of the RTC. The different options for handling and processing the pixel stream are shown in Figure 6, with each subsequent option reducing the time taken from last pixel to end of thread computation.

As each DARC thread processes its subapertures from beginning to end, they must be allocated a specific set of subapertures to process, the most simple and naive way of assigning subapertures would be to divide them equally amongst the threads, Figure 6(c). However as can be seen the threads which process the earlier arriving subapertures finish their work before the later threads, the subapertures are assigned equally among threads in an ascending order, as the thread number increases so does the time waiting for pixels, yet the processing time is constant.

Seen in Figure 6(d) is an option whereby the subapertures are not allocated equally among the threads, the threads that process earlier subapertures are given more work to do and the later ones are given less. This ensures that the threads finish their work at roughly the same time and so helps to reduce the time between the last pixel arriving and the final DM command being sent out. However, the time waiting for pixels also changes as processing more subapertures requires waiting for more pixels, which can be seen in the different sized blocks for pixel waiting. This is a lot more complex in practice as there will not be a one-to-one relation between the number of sub-apertures and the number of pixels that need to be waited for.

Figure 8 shows the frametime results of DARC configured for ELT-scale SCAO (in a similar configuration as above however with 5316 actuators, to mimic the ELTs M4 Adaptive mirror, and 4632 subapertures for a total of 9264 slope measurements) though without a physical camera connected, measured over ${10}^{6}$ frames. It can be seen in Figure 8(a) that for the system using an Intel motherboard (model S72000, 7250 processor), there are a small number of regular single frame outliers which add about $200250\text{\,}\mathrm{\SIUnitSymbolMicro s}$ to the frame time, roughly every $63.75\text{\,}\mathrm{s}$. We have determined that these events are due to the Intel system management interface on the motherboard, which periodically polls the processor for information. There appears to be no way in which this can be turned off. The presence of these interrupts can be verified using this code:

1. 1.

   for SEC in ‘seq 0 200‘; do echo -n "$SEC "; rdmsr -p 0 -d 0x34; sleep 1; done,

which has been used to confirm their presence on the Intel S72000 motherboard used in this report.

Figure 8(b) shows results taken using a Ninja Development platform Xeon Phi using a Supermicro motherboard (model K1SPE with 7210 processor). Here it can be seen that these $64\text{\,}\mathrm{s}$ period events are not present. It is therefore important to take care when evaluating motherboards suitable for AO RTC. Histograms of both measurements are shown in Figure 8(c), the difference in mean frametime between the two distributions is due to the specification of the processors used in each motherboard; $1.41.3\text{\,}\mathrm{GHz}$ clock speed, $480450\text{\,}\mathrm{GB}\text{\,}{\mathrm{s}}^{-1}$ memory bandwidth (Table 1). From this figure, it can be seen that the distribution of latency measurements is approximately Gaussian, except for the outliers.

Therefore, DARC is able to operate ELT-scale SCAO with a $0.93\pm 0.01\text{\,}\mathrm{ms}$ frame time, corresponding to a $1070\pm 10\text{\,}\mathrm{Hz}$ maximum frame rate. When the $64\text{\,}\mathrm{s}$ events are included, the instantaneous peak-to-peak jitter over a million frames is $263\text{\,}\mathrm{\SIUnitSymbolMicro s}$, while ignoring these events reduces the peak-to-peak jitter to $92.7\text{\,}\mathrm{\SIUnitSymbolMicro s}$ and the RMS jitter is only $11.4\text{\,}\mathrm{\SIUnitSymbolMicro s}$. The Ninja development platform can operate ELT-scale SCAO with a $1.01\pm 0.01\text{\,}\mathrm{ms}$ frame time, corresponding to a $990\pm 10\text{\,}\mathrm{Hz}$ maximum frame rate. This is a lower maximum performance than the Intel motherboard system due to the difference in processor specification which is as expected.

For investigating the effect on the latency of the RTC on the Xeon Phi when storing the control matrix as $16\text{\,}\mathrm{bit}$ floating point values we had to use our own implementation of an MVM algorithm. This is because the Intel MKL library that is used to calculate the reconstruction MVM in previous results is unable to operate directly on $16\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}$ floating point values. To be able to use it, the values would need to be converted to $32\text{\,}\mathrm{bit}$ before each call to the MKL library. This would not be ideal as MKL works best on larger MVM problem sizes and converting a large amount of the control matrix to $32\text{\,}\mathrm{bit}$ would defeat the purpose of storing it as $16\text{\,}\mathrm{bit}$ and making too many calls to the library would vastly increase the latency.

Our custom MVM implementation uses Intel intrinsic instructions to convert the $16\text{\,}\mathrm{bit}$ values of the control matrix immediately before they are operated upon. Up to 16 conversions can be done in a single instruction with the results stored in one of the $512\text{\,}\mathrm{bit}$ vector registers ready for the FMA MVM operations. This ensures that there is a minimum transfer of $32\text{\,}\mathrm{bit}$ values to conserve memory bandwidth.

A similar custom $32\text{\,}\mathrm{bit}$ MVM implementation, simply loading the data instead of converting it, shows that this algorithm isn’t as optimised as MKL. It gives an average frametime of $0.995\pm 0.006\text{\,}\mathrm{ms}$ with an RMS jitter of $5.96\text{\,}\mathrm{\SIUnitSymbolMicro s}$ which can be compared to results that use MKL on the same processor/motherboard of $0.93\pm 0.01\text{\,}\mathrm{ms}$ from Figure 8.

The algorithm that uses a $16\text{\,}\mathrm{bit}$ control matrix decreases this average frametime to $0.902\pm 0.006\text{\,}\mathrm{ms}$ with an RMS jitter of $5.60\text{\,}\mathrm{\SIUnitSymbolMicro s}$. The need to use a less optimised custom MVM and the need to convert to $32\text{\,}\mathrm{bit}$ floats introduces extra overhead which greatly reduces the potential gain. These results show that this implementation of storing and converting the 16-bit control matrix increases performance by only $3\text{\,}\mathrm{\char 37\relax}$ over the best case MKL results.

The next iteration of Xeon Phi after KNL, Knights Mill (KNM) which is available now, includes support for Intel variable precision operations (vector neural network instruction, VNNI) which include intrinsic instructions that can directly operate on $16\text{\,}\mathrm{bit}$ integer values by addition and multiplication to produce an accumulated $32\text{\,}\mathrm{bit}$ integer sum. This would require some conversion of the control matrix and slope values to fixed-point $16\text{\,}\mathrm{bit}$ precision integers but could reduce the memory bandwidth requirement without introducing a costly $1632\text{\,}\mathrm{bit}$ conversion for each control matrix value at execution time. Simulations have shown that $16\text{\,}\mathrm{bit}$ fixed-point values would just be sufficient to provide the required precision (Basden et al., 2010a), however when taking into account a real system with misalignments it may not be adequate.

This functionality comes via a redesigned vector processing unit (VPU) which also doubles the number of SP FMA operations possible per instruction cycle, increasing the theoretical peak SP-FLOPS 2X over KNL. This is achieved with quad FMA (QFMA) instructions which allow for sequential FMA to accumulate over four sets of calculations with a single instruction. There are caveats to this however due the instruction pipeline of the VPUs, so 2X speed up is unlikely, but the QFMA operations should definitely benefit the highly vectorisable MVM without needing to use the $16\text{\,}\mathrm{bit}$ VNNI; investigation into KNM VNNI and QFMA instructions could be useful future work.

This paper has only discussed the SCAO regime of AO for ELT-scale but the next generation telescopes all have more complex multi-conjugate or multi-object AO systems planned for which suitable RTCs will be required. One of these systems is the ELTs Multi-conjugate Adaptive Optics Relay (MAORY) which will employ 6 Laser Guide Stars (LGS) along with three Natural Guide Stars (NGS) using 3 DMs for the correction, increasing the computation requirements by more than 6 times that of ELT SCAO. Our initial investigation (Jenkins et al., 2018) for this involves multiple Xeon Phi systems working in parallel to process the images from each WFS independently before combining the partial DM vectors and delivering to the DMs. This presents significant challenges in addition to those for preparing DARC for ELT-scale SCAO.