Direct data transfer between FPGAs Virtex-7 via PCI Express bus
Abstract
This article describes two types of data transfers via PCI Express bus involving several FPGA. The first one is a simultaneous DMA data transfer between the system memory and different FPGA chips. The second one is a simultaneous direct data transfer between different FPGA.
The data transfer speed was measured for both cases with results being about 99% from maximum speed for PCIe x4 Gen 2.0 link for the direct transfer between FPGAs (1603 MB/s for 128 bytes payload and 1740 MB/s for 256 bytes payload). The direct data transfer latency was also measured to be 0,7 us for one intermediate PCIe switch and 1 us for three intermediate switches.
Also the effect of simultaneous transfers on data transfer speed was studied with the result that, as long as the aggregate transfer speed does not overcome the shared link bandwidth, each transfer is performed on its maximum speed; after that the shared link utilization reaches 100% with its bandwidth being distributed equally between individual transfers.
The data transfer speed was measured for both cases with results being about 99% from maximum speed for PCIe x4 Gen 2.0 link for the direct transfer between FPGAs (1603 MB/s for 128 bytes payload and 1740 MB/s for 256 bytes payload). The direct data transfer latency was also measured to be 0,7 us for one intermediate PCIe switch and 1 us for three intermediate switches.
Also the effect of simultaneous transfers on data transfer speed was studied with the result that, as long as the aggregate transfer speed does not overcome the shared link bandwidth, each transfer is performed on its maximum speed; after that the shared link utilization reaches 100% with its bandwidth being distributed equally between individual transfers.
References
1. Rosta LTD, 2013. http://www.rosta.ru/.
2. nVidiaCorporation. GPUDirect. https://developer.nvidia.com/gpudirect.
3. Ra Inta, David J. Bowman, Susan M. Scott, "The “Chimera”: An Off-The-Shelf CPU GPGPU FPGA Hybrid Computing Platform", International Journal of Reconfigurable Computing, 2012.
4. Bruno da Silva, An Braeken, Erik H. D'Hollander, Abdellah Touhafi, Jan G. Cornelis, Jan Lemeire, "Performance and toolchain of a combined GPU/FPGA desktop", Proceedings of the ACM/SIGDA international symposium on Field programmable gate arrays, 2013.
5. Ray Bittner, Erik Ruf, Alessandro Forin, "Direct GPU/FPGA Communication Via PCI Express", Cluster Computing, 2013.
6. Sunita Jain, Guru Prasanna, "Point-to-Point Connectivity Using Integrated Endpoint Block for PCI Express Designs", Xilinx Corporation, XAPP869, 2007.
7. 7 Series FPGAs Integrated Block for PCI Express v1.7 Product Guide, Xilinx Corporation, 2012.
8. Alex Goldhammer, John Ayer, "Understanding Performance of PCI Express Systems", Xilinx Corporation, WP350, 2008.
Review
For citations:
Rumyantsev Yu.A. Direct data transfer between FPGAs Virtex-7 via PCI Express bus. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2013;24. (In Russ.)
Email this article 