LWN.net Logo

Open-Sourcing Fibre Channel over Ethernet (eWeek)

eWeek covers Intel's release of GPLv2-licensed Fibre Channel over Ethernet (FCoE) code for Linux. "FCoE's purpose is to enable data centers to consolidate LAN and SAN (storage area network) traffic over 10GB Ethernet. FC, which comes in speeds from 2 to the just arriving 8G bps, is commonly used in data center SANs. In recent years it's been challenged by iSCSI. Fibre, which, despite the name can run both on copper and fiber-optic cables, is seen as faster and more reliable, while iSCSI is commonly thought of as less expensive. Intel, along with FCoE's founder Cisco Systems, is hoping to combine the virtues of both Fibre and iSCSI with this new high-speed, dual-purpose network fabric." Further: "Unlike iSCSI, FCoE does not run on the TCP/IP stack. This is Fibre Channel on Ethernet without the overhead or the management and analysis tools of TCP/IP."
(Log in to post comments)

Relation to existing network block device/ATA-over-ethernet projects?

Posted Dec 20, 2007 21:21 UTC (Thu) by tarvin (subscriber, #4412) [Link]

It seems there's already a number of other raw-storage-via-network projects, such as the ATA
over Ethernet project. Does anyone know how Open-FCoE compares to those?

Relation to existing network block device/ATA-over-ethernet projects?

Posted Dec 20, 2007 23:31 UTC (Thu) by magila (subscriber, #49627) [Link]

AoE, being based on ATA, isn't really suited for enterprise use. FCoE seems to be intended as
the FC camp's response to iSCSI. Besides running over raw ethernet versus TCP/IP, I'm guessing
the main difference would be in how they integrate with the rest of a data center. If you're
already using FC I imagine it makes a lot of sense to use the same protocol for tunneling SAN
traffic over your LAN.

Relation to existing network block device/ATA-over-ethernet projects?

Posted Dec 21, 2007 3:44 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

iSCSI is SCSI over TCP/IP over Ethernet

FC is SCSI over Fiberchannel

FCoE sounds like it's TCP/IP over Ethernet

AoE is ATA (a subset of SCSI) over TCP/IP implemented by a single vendor

FCoE should be significantly more efficient then iSCSI, but it probably has significant
limitations as well. Since it doesn't use TCP/IP it can't route between segments using
standard routers, and I wonder how the various traffic optimizations in switches will interact
(for example, switches detect broadcast traffic and send it to all ports while limiting
non-broadcast traffic to the source and destination ports of the switch)

what this looks like to me is an acknowledgment that the FC people are falling behind in the
performance field (while 8Gb FC may actually be faster then 10Gb Ethernet, it's also FAR more
expensive and bleeding edge), and Ethernet equipment tends to work togeather between different
vendors far better then a less popular media like FC.

I wouldn't want to mix FCoE with my normal network traffic on the same switch, but I can
defiantly see the attraction of using a set of dedicated commodity Ethernet switches for a SAN
instead of the far more expensive fiber switches and cards


Relation to existing network block device/ATA-over-ethernet projects?

Posted Dec 21, 2007 5:59 UTC (Fri) by paravoid (subscriber, #32869) [Link]

You got it all wrong :-)
iSCSI is SCSI over TCP/IP over *anything*
AoE is ATA (which has nothing to do with SCSI) over raw Ethernet.

FC is a protocol, usually used with fiber-optic cables but working with copper as well, that
usually carries SCSI commands.
FCoE is a protocol to pass FC over raw Ethernet. No TCP/IP stack in the middle.

Relation to existing network block device/ATA-over-ethernet projects?

Posted Dec 21, 2007 7:21 UTC (Fri) by Ross (subscriber, #4065) [Link]

Switches determine unicast vs. broadcast and which nodes are on which port via Ethernet
headers (there's an Ethernet broadcast address), so that should "just work".

Relation to existing network block device/ATA-over-ethernet projects?

Posted Dec 21, 2007 9:50 UTC (Fri) by drag (subscriber, #31333) [Link]

It's all about being cheaper. 

Gigabit ethernet is the standard for storage/clustering network fabric. Why? Because it's
cheap. You only use expensive stuff like FC or Miranet or whatnot if you have to.

ISCSI is ideal for gigabit ethernet in a lot of ways. It's cross-platform for one, and has
widespread support in both effective software AND hardware oriented solutions.

But if your running a storage array network setup then it's not like you realy need a whole
lot of routing capability. For one the security on those networks is very weak, so you don't
want to mix up your regular network into it if you can help it. TCP/IP on gigabit ethernet can
nail your CPU to the wall with all of it's interrupts. 1Gb/s ethernet still uses the tiny 1500
mtu ethernet frames.. when your maxing out your network that is a LOT Of interrupts.  You can
use 'Jumbo Frames' to lower that by quite a bit, but still it's a large amount of overhead. 

I don't know the maximum MTU on a 10Gb ethernet (or 10GB?), but I hope the limit is huge.. but
why do you want all that TCP/IP overhead on something you don't want to realy route around?
After all that sort of thing is what NFS or Lustre is for. So iSCSI plus (relative) cheap
switches and 10GB ethernet is just the logical solution. 

Relation to existing network block device/ATA-over-ethernet projects?

Posted Jan 3, 2008 5:48 UTC (Thu) by gdt (subscriber, #6284) [Link]

The MTU on any ethernet -- including 10GE -- is 1500 bytes. IEEE will not standardise any
other value as they wish to retain compatibility with other 802.3 devices.

Manufacturers of 10GE equipment all allow operation in a non-standard mode. All support 9000B
and most support 64KB. These frames still use CRC32 for error detection, so if you use 64KB
you are accepting a Hamming Distance of 2 rather than 3.

Open-Sourcing Fibre Channel over Ethernet (eWeek)

Posted Dec 21, 2007 6:21 UTC (Fri) by jd (guest, #26381) [Link]

Instead of having drivers for each and every x-over-y protocol, maybe it would make more sense
to have a generic encapsulation driver and some sort of definition of how to perform the
specific encapsulation. Yes, code-driven is faster than data-driven, but modern systems (and
modern local busses) are an entire order of magnitude or two faster than most networks, so you
can afford a performance hit.
<p>
(Even Infiniband, which is about the fastest network out there, is only 12 lanes in a given
direction, tops, which both PCI Express 2.x and HyperTransport 3.x can exceed. That's before
you consider the fact that any network of more than 2 nodes can - and probably will - totally
saturate the network.)
<p>
The questions then are whether using data, rather than code, would reduce the potential for
bugs and exploits, and whether data would take less space than loadable modules. Even if the
overhead issue was considered unimportant, safer code or significantly more compact code might
be considered enough of an advantage to be worth it.

Open-Sourcing Fibre Channel over Ethernet (eWeek)

Posted Dec 21, 2007 10:55 UTC (Fri) by muwlgr (guest, #35359) [Link]

There was nice protocol, HyperSCSI, i.e. SCSI over raw Ethernet. It was developed by some
smart SE-Asian guys (.hk, or .sg, don't remember now). Unfortunately, no more news is heard
about it. It seemed to have a good chance to become the optimal tradeoff for simple SANs. May
be someone decided to commercialize it deeply :>

Open-Sourcing Fibre Channel over Ethernet (eWeek)

Posted Dec 21, 2007 11:12 UTC (Fri) by muwlgr (guest, #35359) [Link]

Ah yes. Reading the article. It's just Intel wanting to sell us "this new high-speed,
dual-purpose network fabric". Better give me real up-to-date HyperSCSI. But that would not
earn more money to Intel, you know.

Copyright © 2007, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds