Serial Attached SCSI (SAS) is a circuit switched computer network

(You may wish to read A brief introduction to SCSI before reading this article.)

Background. Serial Attached SCSI (SAS), first released in 2004, is the serial replacement for Parallel SCSI; what Serial ATA is to Parallel ATA, SAS is to Parallel SCSI. Hard drives today are manufactured with either a SATA or, for “enterprise” drives, a SAS interface.

SAS Architecture. SAS is defined in two specifications: firstly in the SAS standard itself, and secondly in the SAS Protocol Layer (SPL) standard.

The former specification is comprised mainly of a) electrical and analog concerns, and b) pictures of the absurdly large number of different physical connector types supported by SAS, whereas the latter specification describes the actual SAS protocol itself which runs over the physical interconnect.

This standard is an utter brick; SAS can be in reality be basically viewed as a computer networking protocol, and the standard thus has to define a whole network stack with multiple layers, complete with routing protocols (think SAS expanders).

SAS is also pretty bananas in a lot of ways. SAS packets reuse the Fibre Channel frame header, except you can't actually route the packets based on this. You see, the Fibre Channel header uses 24-bit source and destination address fields, and SAS addresses are 64 bits... therefore the SAS addresses are hashed to form the values which go in the fields in the Fibre Channel header. This, in turn, means that the hashed addresses obviously aren't guaranteed to be unique, so you couldn't safely route SAS packets based on the header even if you wanted to, raising the question of what the point was of using the Fibre Channel header in the first place.

So, how is SAS actually routed? It's circuit switched. Yes, really. In order for an initiator to communicate with a target through, say, a SAS expander, it must first send a link-level Open message, which is specially relayed by expanders through the network to the target. In a manner reminiscent of ringback tones, expanders continuously transmit Arbitration in Progress primitives to the connection originator while the connection is being made. The target must respond to the connection request with an Open Accept (or Open Reject) primitive. Once this connection is established, all packets sent by the initiator are routed by any intermediary expanders to that target, no matter how they are addressed in their headers. Link-level messages control expanders by opening and closing circuits, and an expander simply blindly forwards any packets received through the current circuit.

This means that it is impossible for an initiator to talk to multiple targets simultaneously. Instead, to change the target it is communicating with, it must disconnect and reconnect to another target. Since SCSI is designed to allow many commands to be executing concurrently, in practice an initiator will connect to a target, submit a command, then immediately disconnect so it can talk to other targets. Which means when a target has completed a command and wants to talk to the initiator, it has to “reconnect” to the original initiator using the same Open/Open Accept link-layer protocol in a circuit-switched fashion, and send the results of the command.

It is not an exaggeration to say that an accurate analogy for this protocol would be if you had a desk phone which could only make one call at a time, and wanted to submit commands to a variety of people simultaneously, and so spent all your time calling people up, waiting for them to answer the phone, telling them what to do, and then instead of waiting the milliseconds it would take for them to finish the task, immediately hanging up again and waiting to be called back, so that you can call other people in the meantime.

Those familiar with how parallel SCSI worked worked will note that this circuit-switched nature bears an awful lot of resemblance to it. Parallel SCSI used exactly the same concepts, with initiators connecting to targets and targets reconnecting to initiators as necessary — a design which of course made much more sense for SPI, being that it was an electrically shared medium requiring arbitration. It has been previously observed that the design of SAS resembles what you would get if you took the designers of SPI and told them to “make it serial”. Even the term arbitration is still used to describe the connection process.

Note that this connection process is still required even in the degenerate case where no expanders are used and a SAS HBA is connected directly to a SAS hard drive; the initiator must still open a circuit to the drive via the Open/Open Accept process. Amusingly connections are automatically closed after a configurable period of inactivity, and as far as I can tell this is still (rather pointlessly) the case in the direct-attach case.

Further reading. For those who want a detailed introduction to the internals of SAS — which, to be clear, most people setting up SAS arrays will never need to know even slightly — this book is a good read, which explores the internals of SAS in depth.