Inspur Hotswap Cages

Taobao has many 2U hot-swap cages for sale with SAS expanders and daisy-chain capabilities. These are disassembled from 2U Inspur servers disposed by hyperscalers' reverse supply chains.

A cottage industry of accessories and enclosures for these hot-swap cages has also sprung up around them.

TODO

• Bays: 12 × 3.5″ SAS3/SATA3
• Ports: 3 × SAS3 - Mini-SAS HD (SFF-8643) connectors
• Tray P/N: RM13919-01D (OEM: Chenbro)
• Expander chip: PMC Sierra PM8388 SXP 24x3G (24 ports - 12 connected to rear ports, 12 connected to drives)

• Daisy chain mode
  • Use any 1 × Mini-SAS HD connector as 4 × SAS wide link to HBA
  • Use any 1 × Mini-SAS HD connector as 4 × SAS wide link to downstream cages' expanders
• Passthrough mode
  • Connect all 3 × Mini-SAS HD connectors to HBA(s), all disks are passed through 1:1
  • If only (any) 2 × Mini-SAS HD connectors are connected to HBA(s), 4 × drive trays will be non-functional i.e. there is no support for 8:12 HBA:drive SAS links

Front view of hotswap cage:

phy0-phy11 are mapped to the mini-SAS HD connectors.

LEDs can be controlled by ledctl, e.g.:

• ledctl normal=/dev/disk/by-path/pci-0000\:01\:00.0-sas-exp0xdeadbeefdeadbeef-phy12-lun-0
• ledctl normal=/dev/sda

Supported patterns:

• normal/off: Status LED off
• locate: Blue status LED
• rebuild: Purple status LED
• failure: Red status LED
• All other patterns cause LED color(s) to get stuck until valid patterns are used to set and clear stuck LED color(s)

• sasutils
  • sas_discover -vv - SAS devices, topology (tree) view
  • sas_devices [-v…] - SAS devices, list view
• ledctl
  • OpenZFS' zed automatically indicates vdev failures on supported enclosures' drive failure LED(s), if failed drives are replaced into other enclosures/bays, the failure LED may get stuck; ledctl can be used to clear the failure indicator
• udisks2
  • find /dev/disk/by-path -name "*sas-exp0x56c92bf*lun-0" -exec sudo udisksctl power-off -b {} ';' - spindown all disks in all expander backplanes