The PowerPC AS Tagged Memory Extensions

Background. In 1988, IBM released its AS/400 minicomputer based on a CISC ISA whose instruction set was never published. The AS/400 line, which was subsequently renamed to “eServer iSeries” and then to IBM i, was subsequently migrated to PowerPC RISC CPUs. The hardware was unified into the IBM Power Servers line, so that the IBM i operating system now runs exclusively on vanilla IBM Power Servers along with Linux, AIX, etc.

AS/400 was unusual for its use of tagged memory, in which a single bit of metadata may be associated with every word of RAM. This was implemented on AS/400 using extra-wide memory modules with additional storage to hold the tag bits. In order to maintain compatibility when transitioning to PowerPC, a means to support this tagged memory had to be developed.

PowerPC AS refers to the architectural extensions to the Power ISA which were made to support PowerPC-based AS/400 and which are still used today to support IBM i. These architectural extensions were never documented by IBM and are implemented only by IBM POWER CPUs (POWER5, POWER6, POWER7, POWER8, POWER9, etc.).¹

IBM has never published information about these extensions, so they remain largely mysterious to most people. In reality though, these extensions aren't nearly as secret as IBM thinks; if you add together the fragments of publically available information, a complete picture of the extensions is available. Since, as far as I can tell, nobody has done it before, this document describes what is publically known about these ISA extensions in one place.

Hardware requirements. Instead of using the extra-wide bespoke memory modules that CISC AS/400 used, the PowerPC era moved to using industry-standard DIMMs without extra storage for the tag bits. The tag bits are instead coded into the ECC syndrome; this works since all IBM Power servers use ECC memory exclusively. In other words, information theory shenanigans enable these metadata bits to be stored with no additional overhead.²

The tags. The architecture provides for one tag bit for every 16 bytes of (aligned) memory.

Strictly speaking, a hardware implementation is free to implement either one tag bit for every 16 bytes, one tag bit for every 8 bytes, or even one tag bit for every 4, 2 or 1 bytes, so long as it has some way of storing this information (ECC syndrome permitting). However, the ISA extensions only allow the logical AND of all the tag bits on a 128-bit quadword to be retrieved, which means it is opaque to the architecture whether the hardware is actually storing one tag bit per 16, 8, 4, 2 or 1 bytes.

This means, for example, that one generation of hardware implementing PowerPC AS could store one tag bit for every 4 bytes, and a subsequent generation could store one tag bit for every 16 bytes, and a generation after that could store one tag bit for every 8 bytes, with no software-visible changes; the only information about tag bits that can be retrieved via the PowerPC AS extensions is whether all of the bits on an aligned 16-byte quadword are set.

Because of this, I don't actually know whether current or previous IBM hardware stores a tag bit per 16, 8, or 4 bytes, since it is opaque to software. However, there is some evidence to suggest early implementations of PowerPC AS used one tag bit per 8 bytes. It's also known that very early CISC machines used one tag bit per 4 bytes.

In general, performing an arbitrary store to an aligned 128-bit quadword or any part of it (that is, a doubleword, word, halfword or byte comprising any part of it) will clear the tag bits. Only specially performed stores set the tag bits.

The tagged memory extensions are used to anoint pointers as being legitimate, allowing pointer forgery to be detected, forming the basis of a primitive capability-based architecture. (Pointers in IBM i are 16 bytes. Since PowerPC is a 64-bit architecture, only half of this space is used for actual memory addressing; the other half is reserved for metadata and goes mostly unused, aside from a few bits used to specify a type field. You might consider this some extreme futureproofing.)

Since writing to any part of a 16-byte pointer clears its tag bits (for example, changing the third byte with a “store byte” instruction), changing the pointer in any way clears its tag bits, de-anointing it as a legitimate pointer.

Registers. A processor implementing PowerPC AS runs in either Tags Inactive (TI) or Tags Active (TA) mode. In Tags Inactive, the processor functions like a completely normal Power ISA implementation.

Tags Active mode is enabled by setting bit MSR.TA (bit 1 of the MSR in IBM bit numbering; i.e., 1<<62).

The other architectural register which is extended is XER. Bit 43 of XER (again, in IBM bit numbering), called TAG, holds the tag bit for a 128-bit quadword. When a 128-bit quadword is loaded, the tag bit for that quadword is placed in XER[43]. When a 128-bit quadword is stored, the tag bit(s) for that memory are set from XER[43].

Behaviour in Tags Active mode. Setting TA mode changes the behaviour of the processor in two fundamental ways:

Firstly, it allows you to use the PowerPC AS instructions below (which are otherwise considered illegal instructions).

Secondly, it changes how the processor calculates displacements. For example, consider the “Load Word” instruction:

lw  rX, disp(rY)

which loads a 32-bit word from [rY]+disp and places the result in rX. When TA mode is enabled, if the addition of disp and the contents of rY would cause a different aligned 16 MiB region of memory to be addressed from that addressed by rY alone, an exception occurs (unless the 16 most significant bits of rY are zero, in which case this overflow check is bypassed). The purpose of this is discussed subsequently.

The MMU also works a bit differently in TA mode, so that SLS addresses can be treated directly as VAs rather than EAs, bypassing the SLB. (TBD)

Instructions. The known instructions which become available in TA mode are as follows:

LQ: Actually a part of the normal ISA nowadays. However, in TA mode it is extended to load the tag bit for the quadword into XER[43]. The instruction is also extended with a third operand (in bits 28:31 inclusive), an immediate, which appears to mask the loaded value in some way.

STQ: Also part of the normal ISA nowadays. In TA mode it stores the tag bit from the current value of XER[43].

SETTAG: This sets XER[43] to 1. It is generally used immediately before a store when it is desired to anoint a quadword as being a valid pointer. Encoding: No operands, encodes as 0x7C0103E6.

TXER: “Trap on XER”. This is actually a general purpose instruction which appears to be usable to trap on any XER bit, however it is only available in TA mode. It is generally used to ensure the tag bit is set immediately after an LQ instruction. Encoding: Major opcode 31, minor opcode 36 (placed in bits 25:30). Three immediate operands in 6:10, 11:20 and 21:24.

SELII, SELIR, SELRI, SELRR: These are like a ternary operator predicated on a specified bit in XER. II means both operands are immediates, IR means one is an immediate and one is a register, etc. One operand is selected if the specified bit in XER is set, otherwise the other operand is selected. Variants ending in a . update the condition register. Encoding: Major opcode 30, minor opcode 24 (selii), 25 (selii.), 26 (selir), 27 (selir.), 28 (selri), 29 (selri.), 30 (selrr), 31 (selrr.) in bits 27:31. Destination register in 11:15, first and second operands (register or immediate depending on mnemonic) in 6:10 and 16:20, final immediate in 21:24.

LTPTR: This wasn't part of the original PowerPC AS extensions and was a late addition to the family added some generations later to improve performance. It functions like LQ, except the loaded value is zero if the tag bits weren't set. See this 2009 patent, which includes an encoding diagram in the style of the Power ISA manual (but note that the bit offsets in the diagram are very patently wrong; this seems to be a typo. You can easily figure out the correct ones by looking at the encoding conventions of other Power ISA instructions, however).

SCV: System call vectored. I won't bother to describe this because curiously, this was declassified in version 3.0 of the Power ISA specification and now appears in the publically available ISA manual. This is somewhat strange because this instruction has traditionally only been used internally by IBM i, and the way that it works is very specific to how IBM i is laid out in memory. See the ISA specification to see what I mean.

Extensions provide no security. It is vital to note that nothing about these ISA extensions provides any kind of security invariant against a party which can generate arbitrary machine code, even if only in unprivileged mode. The tagged memory extensions don't stop you from doing anything. As such, they can principally be viewed as providing a performance enhancement for the IBM i operating system, which uses these instructions to keep track of pointer validity. It is the IBM i OS which enforces security invariants, for example by always following every pointer LQ with a TXER.

This works because in IBM i, all machine code executed by the system is generated by the kernel translating from a virtual intermediate language. Compare with JITing JavaScript engines, the Java VM, Singularity, Nebulet, etc. In other words, the PowerPC AS extensions can be viewed as a hardware accelerator for a trusted translator.

Extensions are extremely specific to IBM i. I noted above that when enabling TA mode, the register-displacement addition will trap if it causes overflow into a different 16 MiB aligned segment of memory. This is because IBM i likes to allocate memory in 16 MiB “segments”, and this enables the trusted translator to generate code dereferencing segment pointers offset by an (untrusted) user-provided offset without having to check on every single displacement addition whether a segment overflow would occur. In other words, the hardware is hardcoded to the precise design of IBM i. The hardware cannot, to my knowledge, be reconfigured to trap on a different carry bit/segment size. As a special exception which is even more specific to IBM i, the 16 MiB overflow check is disabled for addresses below 256 TiB (i.e., having the 16 most significant bits as zero); this is used to support Teraspaces.

Likewise, you need only consider the calling convention used by the SVC instruction now publically documented in v3.0 of the ISA manual to see how specific some of these instructions are to the precise nature of IBM i. It is unlikely any other OS could make constructive use of these ISA extensions, aside from maybe the simple ability to store one tag bit per 128 bit quadword, which could be useful.

Further reading:

• AS/400 Capability Addressing
• AS/400 Machine-Level Programming
• as400guide.pdf, some background information on IBM i (work in progress)

Note: All bit numbering on this page is in IBM bit notation i.e., bit 0 is the most significant bit. Bit ranges (X:Y) are inclusive of bits X and Y.

See also: The Talos II, Blackbird POWER9 systems support tagged memory