InfiniBand is a switched fabric communications link primarily used in high-performance computing. Switched fabric, switching fabric, or just fabric, is a Network topology where network nodes connect with each other via one or more Network switches High-performance computing (HPC uses Supercomputers and Computer clusters to solve advanced computing problems Its features include quality of service and failover, and it is designed to be scalable. Failover is the capability to switch over automatically to a redundant or standby Computer server, System, or network upon the failure In Telecommunications and Software engineering, scalability is a desirable property of a system a network or a process which indicates its ability to either The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. In Computing, input/output, or I/O, refers to the communication between an Information processing system (such as a Computer) and the outside It is a superset of the Virtual Interface Architecture. The Virtual Interface Architecture (VIA is an abstract model of a user-level zero-copy network, and is the basis for InfiniBand and IWARP.
Contents |
Description
| Single (SDR) | Double (DDR) | Quad (QDR) | |
|---|---|---|---|
| 1X | 2 Gbit/s | 4 Gbit/s | 8 Gbit/s |
| 4X | 8 Gbit/s | 16 Gbit/s | 32 Gbit/s |
| 12X | 24 Gbit/s | 48 Gbit/s | 96 Gbit/s |
Like Fibre Channel, PCI Express, Serial ATA, and many other modern interconnects, InfiniBand is a point-to-point bidirectional serial link intended for the connection of processors with high speed peripherals such as disks. Quad data rate (or quad pumping) is a communication signaling technique wherein data is transmitted at both the rising and falling edges of the clock signal much the same way Fibre Channel, or FC, is a Gigabit -speed network technology primarily used for Storage networking. Not to be confused with PCI-X, a different bus architecture Peripheral Component Interconnect Express, officially abbreviated as PCI-E In Telecommunication and Computer science, serial communication is the process of sending data one Bit at one time sequentially over a Communication It supports several signalling rates and, as with PCI Express, links can be bonded together for additional bandwidth. Not to be confused with PCI-X, a different bus architecture Peripheral Component Interconnect Express, officially abbreviated as PCI-E Channel bonding (also known as "Ethernet bonding" is a Computer networking arrangement in which two or more Network interfaces on a host computer are combined
Signalling rate
The serial connection's signalling rate is 2. 5 gigabit per second (Gbit/s) in each direction per connection. In telecommunications Bit rate or Data transfer rate is the average number of Bits characters or blocks per unit time passing between equipment in a data transmission InfiniBand supports double (DDR) and quad data (QDR) speeds, for 5 Gbit/s or 10 Gbit/s respectively, at the same data-clock rate. In Computing, a Computer bus operating with double data rate transfers data on both the rising and falling edges of the Clock signal. Quad data rate (or quad pumping) is a communication signaling technique wherein data is transmitted at both the rising and falling edges of the clock signal much the same way
Links use 8B/10B encoding — every 10 bits sent carry 8bits of data — so that the useful data transmission rate is four-fifths the raw rate. In Telecommunications 8b/10b is a Line code that maps 8-bit symbols to 10-bit symbols to achieve DC-balance (see Thus single, double, and quad data rates carry 2, 4, or 8 Gbit/s respectively.
Links can be aggregated in units of 4 or 12, called 4X or 12X. A quad-rate 12X link therefore carries 120 Gbit/s raw, or 96 Gbit/s of useful data. Most systems today use either a 4X 2. 5Gb/s (SDR) or 5Gb/s (DDR) connection. InfiniBand QDR was already demonstrated during 2007, with expectations of productions systems during 2008. Larger systems with 12x links are typically used for cluster and supercomputer interconnects and for inter-switch connections. A supercomputer is a Computer that is at the frontline of processing capacity particularly speed of calculation (at the time of its introduction A
Latency
The single data rate switch chips have a latency of 200 nanoseconds, and DDR switch chips have a latency of 140 nanoseconds. Latency is a time delay between the moment something is initiated and the moment one of its effects begins or becomes detectable The second ( SI symbol s) sometimes abbreviated sec, is the name of a unit of Time, and is the International System of Units The second ( SI symbol s) sometimes abbreviated sec, is the name of a unit of Time, and is the International System of Units The end-to end latency range is from 1. 07 microseconds MPI latency (Mellanox ConnectX HCAs) to 1. Message Passing Interface ( MPI) is a specification for an API that allows many computers to communicate with one another 29 microseconds MPI latency (Qlogic InfiniPath HTX HCAs) to 2. 6 microseconds (Mellanox InfiniHost III HCAs). Various InfiniBand host channel adapters (HCA) exist in the market today, each with different latency and bandwidth characteristics. InfiniBand also provides RDMA capabilities for low CPU overhead. Remote Direct Memory Access ( RDMA) allows data to move directly from the memory of one Computer into that of another without involving either The latency for RDMA operations is <1us microseconds (Mellanox ConnectX HCAs).
Topology
InfiniBand uses a switched fabric topology, as opposed to a hierarchical switched network like Ethernet. Switched fabric, switching fabric, or just fabric, is a Network topology where network nodes connect with each other via one or more Network switches Ethernet is a family of frame -based Computer networking technologies for Local area networks (LANs
Like the channel model used in most mainframe computers, all transmissions begin or end at a channel adapter. Mainframes (often colloquially referred to as Big Iron) are Computers used mainly by large organizations for critical applications typically bulk data Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service.
Messages
Data is transmitted in packets of up to 4 kB that are taken together to form a message. A message can be:
- a direct memory access read from or, write to, a remote node (RDMA)
- a channel send or receive
- a transaction-based operation (that can be reversed)
- a multicast transmission. Direct memory access ( DMA) is a feature of modern Computers and Microprocessors that allows certain hardware subsystems within the computer to access system Remote Direct Memory Access ( RDMA) allows data to move directly from the memory of one Computer into that of another without involving either Multicast is a network addressing method for the delivery of Information to a group of destinations simultaneously using the most efficient strategy
- an atomic operation
Programming
One caveat is that InfiniBand has no standard programming interface. An atomic operation in computer science refers to a set of operations that can be combined so that they appear to the rest of the system to be a single operation with only The standard only lists a set of "verbs"; functions that must exist. The syntax of these functions is left to the vendors. The most common to date has been the syntax developed by OpenFabrics Alliance, which was adopted by most of the InfiniBand vendors, both for Linux and Windows. The OpenFabrics Alliance (http//wwwopenfabricsorg a not-for-profit organization evangelizes and facilitates the adoption of RDMA (Remote Direct Memory Access
History
InfiniBand is the result of merging two competing designs, Future I/O, developed by Compaq, IBM, and Hewlett-Packard, with Next Generation I/O (ngio), developed by Intel, Microsoft, and Sun. Compaq Computer Corporation was an American Personal computer company founded in 1982 and is now a brand name of Hewlett-Packard. International Business Machines Corporation abbreviated IBM and nicknamed "Big Blue", is a multinational Computer Technology Microsoft Corporation is an American multinational Computer technology Corporation, which rose to dominate the Home computer Sun Microsystems Inc ( is a multinational vendor of Computers computer components Computer software, and Information technology services From the Compaq side, the roots were derived from Tandem's ServerNet. Tandem Computers was an early manufacturer of Fault-tolerant computer systems, marketed to the growing number of Transaction processing customers who used them for Servernet was developed by Epic Games to provide a method of server to server link between List Server Applications hosted For a short time before the group came up with a new name, InfiniBand was called System I/O.
InfiniBand was originally envisioned as a comprehensive "system area network" that would connect CPUs and provide all high speed I/O for "back-office" applications. In this role it would potentially replace just about every datacenter I/O standard including PCI, Fibre Channel, and various networks like Ethernet. The Peripheral Component Interconnect, or PCI Standard (commonly PCI) specifies a Computer bus for attaching peripheral devices to a Computer Fibre Channel, or FC, is a Gigabit -speed network technology primarily used for Storage networking. Ethernet is a family of frame -based Computer networking technologies for Local area networks (LANs Instead, all of the CPUs and peripherals would be connected into a single pan-datacenter switched InfiniBand fabric. This vision offered a number of advantages in addition to greater speed, not the least of which is that I/O workload would be largely lifted from computer and storage. In theory, this should make the construction of clusters much easier, and potentially less expensive, because more devices could be shared and they could be easily moved around as workloads shifted. Proponents of a less comprehensive vision saw InfiniBand as a pervasive, low latency, high bandwidth, low overhead interconnect for commercial datacenters, albeit one that might perhaps only connect servers and storage to each other, while leaving more local connections to other protocols and standards such as PCI.
So far InfiniBand has become the de-facto interconnect of choice for high performance computing, and its adoption as seen in the TOP500 supercomputers list is faster than Ethernet. The TOP500 project ranks and details the 500 most powerful known Computer systems in the world A supercomputer is a Computer that is at the frontline of processing capacity particularly speed of calculation (at the time of its introduction (However, one should note that Top500 uses Linpack for benchmark, which as a neatly parallel computing task tends to be fairly easy on the interconnect; InfiniBand shouldn't be confused with the custom-built interconnects of vector supercomputers. LINPACK is a software library for performing numerical Linear algebra on digital computers For example, the NEC SX-9 provides 128 GB/s of low-latency interconnect bandwidth between each computing node, compared to the 96 Gbit/s of an InfiniBand 12X Quad Data Rate link. is a Japanese multinational IT company headquartered in Minato Tokyo, Japan. The SX-9 is a Supercomputer built by NEC Corporation. The SX-9 Series implements an SMP system in a compact node module and uses an enhanced version of ) Enterprise datacenters have seen more limited use. It is used today mostly for performance focused computer cluster applications, and there are some efforts to adapt InfiniBand as a "standard" interconnect between low-cost machines as well. A number of the TOP500 supercomputers have used InfiniBand including the low-cost System X built by Virginia Tech. The TOP500 project ranks and details the 500 most powerful known Computer systems in the world A supercomputer is a Computer that is at the frontline of processing capacity particularly speed of calculation (at the time of its introduction System X is a Supercomputer assembled by Virginia Tech in the summer of 2003, that was originally composed of 1100 Apple Power Mac G5 In another example of InfiniBand use within high performance computing, the Cray XD1 uses built-in Mellanox InfiniBand switches to create a fabric between HyperTransport-connected Opteron-based compute nodes. The Cray XD1 is an entry-level Supercomputer range made by Cray Inc The Opteron is AMD 's X86 server processor line and was the first processor to implement the AMD64 Instruction set architecture (known
SGI, among others, has also released storage utilizing LSI products with InfiniBand "target adapters". Silicon Graphics Inc (commonly initialised to SGI, historically sometimes referred to as Silicon Graphics Computer Systems or SGCS) is a company This product essentially competes with architectures such as Fibre Channel, iSCSI, and other traditional storage area networks. In computing the iSCSI (for " Internet SCSI " protocol allows clients (called initiators) to send SCSI commands ( ''CDBs'' In Information technology, a storage area network ( SAN) is an architecture to attach remote computer storage devices (such as Disk arrays tape libraries Such target adapter-based discs would become a part of the fabric of a given network, in a fashion similar to DEC VMS clustering. Digital Equipment Corporation was a pioneering American company in the Computer industry Open Virtual Memory System ( OpenVMS) initially known just as Virtual Memory System ( VMS) is the name of a High-end Computer server The advantage to this configuration would be lower latency and higher availability to nodes on the network (because of the fabric nature of the network).
The cable InfiniBand uses (CX4) is also commonly used to connect SAS Serial Attached SCSI HBAs to external (SAS) disk arrays. Introduction A typical Serial Attached SCSI system consists of the following basic components An Initiator is a device that originates device service
See also
External links
- An Introduction to the InfiniBand Architecture
- The InfiniBand Trade Association homepage
- An InfiniBand Technology Overview
- Is InfiniBand poised for a comeback?
- InfiniBand edging into storage market
- OpenFabrics Alliance
- InfiniBand simulation model
Dapyx Software network: MP3 Explorer | Ebook Manager | Zenithic