We have detected your current browser version is not the latest one. Xilinx.com uses the latest web technologies to bring you the best online experience possible. Please upgrade to a Xilinx.com supported browser:Chrome, Firefox, Internet Explorer 11, Safari. Thank you!

FPGAs Aid Search for Dark Energy with CHIME Telescope

by Xilinx Employee on ‎11-10-2014 02:43 PM (43,764 Views)


By Steve Leibson, Editor in Chief, Xcell Daily


A mysterious force is causing the universe to expand at an ever-increasing rate. Physicists don’t really know what it is but they have named it: dark energy. There’s no known way to directly detect dark energy experimentally. Evidence is strictly indirect. Observing and measuring periodic fluctuations in the distribution of baryonic matter, specifically neutral hydrogen, across hundreds of millions of light years may be among the best methods we have for studying dark energy. These variations, called baryonic acoustic oscillations (BAO), serve as a “standard ruler” for cosmological distance. A group of researchers based at universities in Canada are developing a radio telescope purpose-built for measuring BAO at the Dominion Radio Astrophysical Observatory in the heart of Canada’s wine country on the south end of Okanagan Lake in British Columbia.


This telescope, called the Canadian Hydrogen Intensity Mapping Experiment (CHIME), reconstructs an image of the overhead sky without having to point at a specific direction. The telescope processes the signals received across a large array of radio antennas using a technique called interferometry. Historically, most interferometry telescopes have reconstructed images across a small region of the sky by mechanically steering several antennas, each in its own telescope dish, to point at that region. CHIME has no moving parts, so it may be thought of as a digital telescope. It reconstructs an image of the entire overhead sky by digitally processing the information its antennas have received as the Earth rotates through a 24-hour period. A Xilinx Kintex-7 FPGA is a key component.


For CHIME to succeed, the team had to overcome sizable technological challenges similar to those encountered in commercial applications. For example, CHIME samples the voltage at each of its 2,560 antennas at 800MHz and divides the samples into different frequency bands. Information from all antennas moves over a gigantic Ethernet network to one processing location so that it can be used to reconstruct the sky image at that frequency. CHIME uses a custom-made network consisting primarily of high-speed serial transceivers connecting FPGAs point to point through low-loss, full-mesh custom backplanes. In total, CHIME combines 4,480 high-speed interconnects, each capable of operating at 10Gbps, into what could be one of the world’s largest purpose-built networks. CHIME’s total information bandwidth is 8Tbps.


Once the data for a particular observing frequency is brought together in one place, the information has to be correlated. Essentially, the data stream from each antenna needs to be multiplied with the data stream from every other antenna. This amounts to about 25,602 multiplications every 1.25 nanoseconds. Indeed, the “size” of a radio telescope correlator can be quantified as the square of the number of antennas times the frequency bandwidth, NANT2 x BW. Needless to say, the CHIME correlator is a monster. These challenges are not unlike those encountered in advanced telecommunications hubs, where fast networking is needed to assemble data sent across fiber-optic trunk lines. Large numbers of rapid correlations are also used in financial market analysis, where correlating information from various indicators, with a range of time lags, provides valuable insight for fund managers.


Before examining the technology that has been developed to overcome these challenges, let’s first take a moment to explore the science objectives of the project.





According to NASA: “In the early 1990s, one thing was fairly certain about the expansion of the universe. It might have enough energy density to stop its expansion and recollapse, it might have so little energy density that it would never stop expanding, but gravity was certain to slow the expansion as time went on. Granted, the slowing had not been observed, but, theoretically, the universe had to slow. The universe is full of matter and the attractive force of gravity pulls all matter together. Then came 1998 and the Hubble Space Telescope (HST) observations of very distant supernovae that showed that, a long time ago, the universe was actually expanding more slowly than it is today. So the expansion of the universe has not been slowing due to gravity, as everyone thought, it has been accelerating. No one expected this, no one knew how to explain it. But something was causing it.


“Eventually theorists came up with three sorts of explanations. Maybe it was a result of a long-discarded version of Einstein’s theory of gravity, one that contained what was called a ‘cosmological constant.’ Maybe there was some strange kind of energy-fluid that filled space. Maybe there is something wrong with Einstein’s theory of gravity and a new theory could include some kind of field that creates this cosmic acceleration. Theorists still don’t know what the correct explanation is, but they have given the solution a name. It is called dark energy.


“More is unknown than is known. We know how much dark energy there is because we know how it affects the universe’s expansion. Other than that, it is a complete mystery. But it is an important mystery. It turns out that roughly 68% of the universe is dark energy.”


Mapping matter density across the universe requires measuring the large-scale power spectrum of the 21cm emission line from neutral hydrogen at a cosmological scale. Measuring the age of the universe when this light was emitted provides information on the cosmic expansion rate, which in turn is driven by dark energy.


Here’s how the Dark Energy Survey—a collaborative effort involving two dozen research institutions—describes BAO:


“Imagine dropping a pebble into a pond on a windless day. A circular wave travels outward on the surface. Now imagine the pond suddenly freezing, fixing these small ripples in the surface of the ice. In an analogous fashion, approximately 370,000 years after the Big Bang, electrons and protons combined to form neutral hydrogen, ‘freezing’ in place acoustic pressure waves that had been created when the universe first began to form structure. These pressure waves are called baryon acoustic oscillations and the distance they have traveled is known as the sound horizon.


This distance is just the speed of sound times the age of the universe when they froze. Just as there is an increased air density in a normal sound wave, there is a slight increase in the chance of finding lumps of matter, and therefore galaxies, separated by the sound horizon distance. Today, this sound horizon distance is about 450 million light years, and it provides a standard ruler for cosmological distance measurements.”


(The Dark Energy Survey involves more than 120 collaborating scientists from 23 organizations who have banded together to survey a large portion of the sky visible from the southern hemisphere using the new Dark Energy Camera mounted on the Blanco 4-meter telescope at the Cerro Tololo Inter-American Observatory, perched high in the Chilean Andes.)


Measuring BAO across intergalactic distances coupled with red-shift data provides cosmologists with additional data about dark energy by mapping out the Hubble parameter—formerly known as the Hubble constant but no longer believed to be a constant over time. This information will help to explain how the universe’s expansion rate has changed over time, giving us more information about dark energy’s effects. By peering back to this key time when dark energy first began to overcome the pull of gravity, astronomers hope to obtain new insight that will provide hints about the theoretical nature of dark energy.





CHIME is a stationary radio telescope array consisting of multiple adjacent parabolic cylinders; it depends upon the Earth’s rotation to scan the sky. CHIME Pathfinder is a smaller version of CHIME with 1/7 the collecting area—a pilot project—with 128 dual-polarization feed antennas and low-noise amplifiers (LNAs) spaced 30 cm apart along the focal lines of the telescope’s two parabolic cylinders. Briefly, signals from the feeds are bandpass-filtered to the 400- to 800-MHz band; the filtered analog signals are sent to 1.25-Gsample/sec ADCs that operate at 800 Msamples/sec. The sampled data is then fed to FPGAs that channelize the signals into 1,024 frequency bins. Each frequency bin is 390kHz wide. The FPGAs then shuffle the channelized bins over a 10Gbps, full-mesh network to group like-frequency bins together. Grouped frequency bins are shipped off to a large bank of GPUs for real-time N2 correlation and averaging.


CHIME Pathfinder has already been built and tested. The full CHIME telescope is now under construction. The total number of multiplications the CHIME telescope will perform will be NANT2 x BW = 25602 x 800 MHz = 5242 teraMACs per second. CHIME Pathfinder, with just NANT2 x BW = 2562 x 800 MHz= 52 teraMACs/s, will already be one of the world’s most powerful correlators. The full CHIME telescope correlator will be immense. If the cosmological objectives of the CHIME experiment seem complex, the practical implementation of the experimental apparatus is significantly more straightforward, although the engineering of the equipment is state of the art.





Both the CHIME and CHIME Pathfinder arrays consist of reflector antennas built from multiple parabolic cylinders. The reflective part of the antenna is made from galvanized steel mesh attached to an underlying structure that provides shape and support. Each CHIME Pathfinder parabola is 20m across and 37m long with a north-south orientation. The structure is not very different from that of any warehouse roof, except that the east-west cross-section is parabolic, the “roof” is made of wire mesh instead of sheet metal and it’s upside down, and there’s no floor.


The researchers chose wire mesh as a compromise between radio wave reflectivity and the ability to shed snow. (It’s in Canada, remember?) CHIME has no moving parts. Unlike a conventional radio telescope, it cannot be tipped over to shed accumulated snowfall. A coarse wire mesh allows snow to fall through, which avoids loss of observation time due to perturbed reflectivity. The CHIME Pathfinder has two such parabolic cylinders covering 1,500m2. The full CHIME telescope has five such cylinders but they’re 100m long instead of 37m. The full CHIME reflector covers 10,000m2.


A walkway suspended above each reflector and located at each parabolic cylinder’s focus line holds a linear series of polarized feed antennas and LNAs, with two LNAs per feed for the two polarized channels in each feed. Received signals are sent over coaxial cables to a shielded RF enclosure for filtering, further amplification, and processing. The CHIME Pathfinder has 64 dual-polarization feeds per parabolic cylinder—128 in total. The full CHIME telescope will have 1,280 dual-polarization feeds. The cylinders have an instantaneous field of view of approximately 100 degrees from north to south and 1 to 2 degrees from east to west. This narrow field of view sweeps the sky as the Earth turns.


Figure 1 shows the two CHIME Pathfinder reflectors and the focus-line walkway suspended above. Each dual-polarization feed is made from printed-circuit boards in the shape of a cloverleaf. Tuned baluns combine differential signals from pairs of adjacent cloverleaf petals to form one single-ended output. The feed petals, balun stem, and support base are all made from PCBs. Conventional circuit board materials are unacceptably lossy for astronomical instrumentation, so the group used Teflon-based PCB material for both the balun stem and support base.



Fig 1 Anderlinder_PathfinderAndStaryNight.jpeg



Figure 1 – Walkways suspended above each reflector in the CHIME telescope carry the dual-polarization feeds and low-noise amplifiers.





The PCB feeds route the 400- to 800-MHz signal to the co-located LNAs, which have two gain stages. The first gain stage employs an Avago ATF-54143 high-electron-mobility transistor (HEMT) followed by an Avago MGA-62563 RFIC. Each LNA drives 60m of coaxial cable, which terminates in a central shielded enclosure made from a sea container that holds the rest of the analog electronics including a custom Minicircuits 400- to 800-MHz bandpass filter and three more analog gain stages. Another 2m of coaxial cable delivers the filtered signals to ADC daughtercards plugged into digital back-end processing cards known as ICE motherboards. Figure 2 is a block diagram of the CHIME’s analog front end.



Fig 2 CHIME Analog Path.jpg


Figure 2 – Analog signals from the low-noise amplifiers travel over 60 meters of coax to an RF room inside a shielded, metal sea container.



Each daughtercard holds two e2v EV8AQ160 quad 8-bit, 1.25Gsample/sec ADCs operating at 800Msamples/sec, making a total of 16 ADCs per ICE motherboard. The ICE motherboards are designed to be generic and they will be repurposed in the upgrade camera for the South Pole Telescope and for the Square Kilometer Array prototype. Figure 3 is a photo of the blue ICE motherboard with two red ADC daughtercards mounted on top.



Fig 3 MotherboardWithFMCs.jpg



Figure 3 – The blue ICE motherboard with an onboard Kintex-7 FPGA supports 16 ADCs mounted on the two red daughtercards.



The ADC daughtercards communicate with the ICE motherboard using LVDS signals and FMC-compliant high-pin-count connectors. The LVDS signals from the ADCs connect to a Xilinx Kintex-7 XC7K420T FPGA, which handles the data from all 16 ADCs on the two daughtercards. The ICE motherboard’s FPGA configuration was developed in VHDL using the Xilinx Vivado Design Suite. The configuration is subdivided into modules that interconnect using the AXI4-Streaming bus protocol.





The Kintex-7 FPGA on the ICE motherboard channelizes the input data from the ADCs into frequency bands using a custom polyphase filter bank followed by FFTs. The result is scaled to (4+4) bit complex values with saturation. Data streams from the 16 FPGA channelizers are then reordered by a crossbar module (also in the FPGA) that aligns the data stream and then routes selected frequency bins to one of 16 output streams.


An FPGA-based data-shuffling module sends the reordered streams to each of 15 other ICE boards in a 9U crate called an ICEBOX. All 16 boards communicate over a mesh backplane using 19 of the Kintex-7 FPGA’s high-speed GTX serdes transceivers operating at 15.5Gbps. The FPGA can also receive trigger, synchronization and GPS-based time-stamp signals from the backplane. The ICE motherboard has two QSFP+ connectors that link to a GPU node and one SFP+ connector that connects to a control computer. Figure 4 shows a block diagram of the ICEBOX crate.



Fig 4 CHIME Pathfinder Crate Block Diagram.jpg


Figure 4 – An ICEBOX crate holds as many as 19 ICE motherboards.



The FPGA’s multigigabit transceivers communicate directly over the full mesh backplane creating a passive, low-cost, high-speed, data-shuffling network. The backplane is constructed of a low-loss material (Panasonic Megtron 6). Data sent over this network is encoded in 64B/66B format, scrambled to balance the DC content of the data, and encapsulated in simple packets with a cyclic redundancy check (CRC) code to detect transmission errors. After the full data-shuffling transaction, each FPGA possesses a subset of 64 frequency bins from all 256 channelizers in the crate. Consider the amount of traffic described in that last sentence. For the full-scale CHIME telescope: 2,560 channels x 8 bits/channel x 800 Msamples/sec = 2 Tbytes/sec. That’s a lot of data movement.


According to Matt Dobbs, associate professor of physics and associate member of the Department of Electrical and Computer Engineering at McGill University and part of the CHIME collaboration, the large number of serial transceivers in each FPGA and their very low cost drove the design decision to implement the data-shuffle/corner-turn portion of the DSP algorithms using the Kintex-7 FPGA’s GTX transceivers and a full-mesh backplane. By contrast, he said, a commercial 10Gbps Ethernet network solution would have been much more expensive and power hungry. Each ICE board requires approximately 75W, so the entire ICEBOX needs about 1.2kW plus some overhead for power-conversion efficiency. Figure 5 is a photo of a partially populated ICEBOX.



 Fig 5 IceBox.JPG



Figure 5 – This partially populated ICEBOX crate is part of the CHIME Pathfinder experiment.



The 16 ICE-board data streams pass through another crossbar that rearranges the data into eight output streams, each containing eight frequency bins from all of the ICE board channelizers. A custom array of eight 10Gbps Ethernet UDP packet transmitters (designed to minimize FPGA resources by using a subset of the full UDP protocol) then sends these eight data streams to a GPU node through two QSFP+ optical connectors.





CHIME Pathfinder’s correlator consists of 16 identical GPU processing nodes. Each node processes 1/16 (25MHz) of the full CHIME bandwidth. Each node is housed in a 4U rack-mount chassis and built primarily of high-end consumer-level components. Processing takes place in two AMD r9 280x GPUs and one r9 270x GPU. A pair of enterprise network interface cards (NICs) receives a total of eight 10-Gbps network connections, streaming a total of 51.2Gbps of radiometric data, along with associated headers and flags.


CHIME’s designers chose GPUs for this correlation task for ease of programming. The correlation operation takes place in a custom processing kernel written in the OpenCL language at the University of Toronto. However, energy efficiency and GPU cooling are significant concerns. Each GPU card requires 400W. The CHIME Pathfinder needs 20 GPU boards with a power requirement of 8kW. The full-scale CHIME needs 2,000 GPU cards and 200kW of electricity, which could cost about $150,000 per year. In addition, the CHIME design team tested several brands and types of GPUs for thermal performance and is presently evaluating direct-to-chip water cooling as a solution to the thermal issues associated with GPUs.


According to Dobbs, the team chose to use GPUs for spatial correlation to get maximum flexibility and faster implementation time at the expense of much higher power consumption. “Once we work all of this out, we might very well move the spatial correlation onto FPGAs,” he said. “We’re in new territory in terms of the image reconstruction algorithms, so flexibility is the key for now.”



For further information on the CHIME program, see “Canadian Hydrogen Intensity Mapping Experiment (CHIME) Pathfinder.” A NASA site offers further insight into dark energy: “Dark Energy, Dark Matter.



Note: This article is reproduced from Xcell Daily, issue 89. It originally appeared in shorter form as a blog post in Xcell Daily, “Looking for Dark Energy using FPGAs, GPUs, and CHIME in Canada’s wine country.”



Click here for the PDF download of Xcell Journal Issue 89.



Click here to read Xcell Journal Issue 89 online.


About the Author
  • Be sure to join the Xilinx LinkedIn group to get an update for every new Xcell Daily post! ******************** Steve Leibson is the Director of Strategic Marketing and Business Planning at Xilinx. He started as a system design engineer at HP in the early days of desktop computing, then switched to EDA at Cadnetix, and subsequently became a technical editor for EDN Magazine. He's served as Editor in Chief of EDN Magazine, Embedded Developers Journal, and Microprocessor Report. He has extensive experience in computing, microprocessors, microcontrollers, embedded systems design, design IP, EDA, and programmable logic.