Oracle ZFS Appliance 40GBE arrived


Turns out, Vivaldi goes to the previous page by pressing Ctrl+left arrow. Along with Alt+left arrow. I learned this the hard way, after almost completing this blogpost. I could kill. And also hugs to Tumblr for not having an auto-save feature in 2017. Neither a ‘Save’ button that won’t make me open the draft for editing AGAIN. So let’s give this another go.

Last week we finally received the 40GBE cards for our shiny new Oracle ZS5-2 appliance. To my knowledge, this is the first one to have such connectivity. Pretty cool, huh?

Here’s a few pics. The Intel logo suggests that this might be an XL710 variant under the hood:

Note that you need OS8.7 for these cards to work. Other than this, no other tricks involved, I reconfigured the datalink to use these instead of the 10G devices and that’s it, everything works perfectly ever since.

We plug these cards into a Mellanox SX1012. DAC cables suck and getting proper fiber equipment is mostly trial-n-error so here’s what worked for us:

  • Fiberstore 12FMTPOM4 MTP cables (make sure you get polarity B cables, A cables won’t work)
  • Fiberstore QSFP-SR4-40G transceivers

These work just fine with the Oracle card, but for whatever reason I couldn’t get them to work neither with the Intel XL710 nor the Mellanox MCX314A-BCCT. So far I’ve only contacted Mellanox support but they sound like a broken record. I’ve already talked to 3 agents but they all link me to a document listing cable assemblies. In the meanwhile we’ve ordered some “Mellanox compatible” transceivers, will see if they work or not, I’ll update this post once they arrive.

One last note about Fiberstore: they have this habit of sticking the same PN on several different products, so use direct links instead of just PNs. Still, they’re the cheapest vendor by far, other transceivers start around $400 or so. But if you find something even cheaper, let me know!

Update: the Mellanox-compatible MC2210411-SR4 transceiver works with the MCX314A-BCCT but not with the Intel XL710.