On Nexuses

Nexuses are an underrecognised utility in computing, despite the fact that they are commonly used and absolutely essential to usable computer systems. A nexus, broadly, is some sort of infrastructural system which is consumed by many different applications written by many different people. Many of these might be called “platforms”, but there are plenty of nexuses which would not be called platforms.

The filesystem, for example, is one of the most fundamental nexuses, because it is something that every application on a computer system makes use of. This makes the filesystem a powerful medium for interchange and communication between programs, even notwithstanding the persistent storage it provides. Take a look at the contents of /run (a tmpfs) on a modern Linux system to see that the filesystem is useful as a nexus quite separately from the persistence it may optionally provide.

These are what I call integration nexuses. They are nexuses which assist in the integration of many separate programs. By the mutual use of nexuses such as the filesystem, programs can be integrated into one whole. These programs might compose to form one application. The distribution of this application might be accomplished using a virtual machine or container strategy, allowing the filesystem to be used as an integration nexus while concealing it from the outer system, which does not need to have any access to it. This is essentially the namespacing of an integration nexus.

There are other integration nexuses. The operating system provides a common idea of a ‘process’, which allows programs to manipulate and supervise those processes as they please. Desktop systems provide countless others, such as the clipboard. Windows provides common interfaces such as COM and OLE.

Integration nexuses can also be found in programming languages. For example, in Go, the flag and expvars packages and net/http's DefaultServeMux are some of the most commonly used integration nexuses provided by the standard library. These facilities are integration nexuses because they represent a way for unrelated code which is not specifically aware of other code to interoperate in some way. For example, the expvars package allows code to export performance counters and gauges. Each metric has a name and a corresponding value. Metrics could be things like “how many times has this function been called: 42” or “how many requests are currently being handled”.

The important detail is that these metrics are registered in the expvars package, and can be consumed by arbitrary code (e.g. to ship them to some sort of monitoring system) without regard to the specific library that created them. Thus the expvars package constitutes an integration nexus, because it allows unrelated code, mutually ignorant of the others' existence, to interoperate in some well-defined and mutually beneficial manner.

Libraries can use the net/http package's default serve mux to register HTTP handlers on any path they like. The expvars package, for example, automatically registers a handler at /debug/vars which provides the value of all expvars. This is an example of consuming one integration nexus and providing access to it using another!

The important point is that inclusion of a handler in the default serve mux can be performed automatically by any package. It doesn't require updating a central list of HTTP handlers, a central source of truth. Any library can show up and join in.

Similarly, the flag package allows any library to register flags. These flags become available for use on the command line automatically so long as the program uses the flag package to handle command line argument parsing. Thus the flag package constitutes an integration nexus.

It should be noted that Go's default serve mux is wholly unsuitable for public use. It provides access to potentially sensitive information such as expvars (which by default, includes the command line with which the program was run — a command line shouldn't contain sensitive credentials, but easily could), or even the remote profiling tool. When providing a public service, a separate serve mux should be created. Yet the default serve mux retains its utility as an integration nexus, and could have internal access to it retained, for example by having it listen on a separate, firewalled port. For example, the default serve mux could allow a service health check library to automatically export a way for a load balancer to check service health. The load balancer retrieves health check information from the port running the default serve mux, but sends requests to the port running the separate, public serve mux.

Nexus Rejection

Imagine, for a second, that a program did not use the filesystem as a nexus. This might not be possible, as executing the program from a file may be the only way to launch a process on the system; but suppose that the program forms its own 'enclave'. For example, maybe it allocates a large, multi-gigabyte file on disk, then implements its own internal filesystem on top of it. It has essentially eschewed the nexus provided by the OS in favour of its own; but because it is the only consumer of this new filesystem, it can hardly be said to be a nexus, except to the extent that it is provided to multiple internal subcomponents of the program.

This particular example is an example of the “inner system effect”. For the purposes of discussing nexuses, though, I call this ‘nexus rejection’. Nexus rejection can also occur for processes/threads: A program might implement its own runtime on which it runs its own ‘green’ threads, and schedule these all on a single OS thread. The OS thus sees only one thread, which has many consequences, such as the fact that multi-core processors cannot be taken advantage of, and the program becomes more opaque to outside observation, as well as countless others.

There can be plenty of good reasons for nexus rejection. Video games commonly aggregate their very large number of data files into large, monolithic archive files, demonstrating filesystem nexus rejection. This can be necessary when filesystems are too slow to provide acceptable load performance. Programming languages such as Go or Erlang, which have their own ideas of how to do I/O, may implement their own scheduling and process system and thus exhibit process nexus rejection. This allow such systems to exhibit massive concurrency and allow programmers the pleasure of writing ‘synchronous’ code which is actually scheduled asynchronously.

But nexus rejection is never without consequence. Integration nexuses provide a common language or commons for programs to interoperate; by eschewing a nexus, one eschews a whole ecosystem and shoulders the burdens solved by that nexus oneself. It makes the operation of the program more opaque and peculiar. The program may come to be disliked or considered an ‘oddball’ by administrators, because it doesn't play by the same rules. The program's replacement for the nexuses it rejects may not be as high quality, because the nexuses of the operating system may be subject to continuous improvement and optimization, whereas the proprietary nexuses of the rejecting program may be left to rot once they work well enough.

But integration is not the only potential role of a nexus, or the only type of nexus. Maintenance nexuses are just as important.

Maintenance Nexuses

A maintenance nexus does for humans what an integration nexus does for programs; it provides a way for a human to get into the internals of a system and inspect and modify it as they need. A filesystem thus counts as both an integration and maintenance nexus, for it is accessible to both programs and humans. Processes too can be manipulated by humans as necessary.

Maintenance nexuses may take many forms, but the most likely and generally most powerful form is the command line interface. A shell provides access to the various tools provided by the system. These tools may be placed in a common location such as /usr/bin. Even something as simple as the very fact that programs are conventionally placed in such a directory constitutes a nexus of sorts, because it allows programs to be executed easily by searching $PATH. Even environment variables themselves constitute another nexus.

In my view, the presence of integration and maintenance nexuses is a key part of an operating system. Although I am unfamiliar with Erlang, I gather that the Erlang runtime provides some sort of maintenance nexus, in which one can connect to an Erlang program and obtain some manner of command line, by which the current state of the system can be manipulated from the inside out. Since Erlang rejects the process nexus of the operating system in favour of its own nexus, and provides a maintenance nexus, I would contest that Erlang is, essentially, an operating system. The fact that Erlang can now be run directly on Xen supports this idea.

Other programs are turning into operating systems, too. Emacs Lisp probably qualifies; it provides a common runtime and a maintenance nexus in the form of the Lisp REPL. The most obvious and runaway example is, of course, the web browser. Web browsers don't provide much in the way of maintenance nexuses, but JavaScript REPLs for developer use and other developer tools may count. The distinguishing feature of the architecture of the Mozilla platform, XUL/XPCOM, offers an integration nexus of extraordinary power, which is what gives Mozilla such a powerful range of addons. (Mozilla has, disastrously, announced an intention to “deprecate” XUL/XPCOM, thus discarding its one great distinguishing feature and differentiator from Chrome).

The rejection of maintenance nexuses is much rarer than the rejection of integration nexuses; humans don't want to learn a whole new “inner system” command line to deal with a program's complicated innards (SQL databases may be an example, though; many administrative commands must be performed from an SQL command line, not an OS command line).

But there is an interesting fad of eschewing maintenance nexuses entirely, at least at the system level. I say “eschew” rather than “reject” here because when I used “reject” above, I mainly used it in the context of replacing the functionality of a nexus with one's own implementation; but here the idea is to do away with the need for maintenance nexuses entirely.

This was proposed in a blog post titled “No SSH” (moronically, that page is blank if you disable JavaScript, and I can find no better article; it is additionally completely unnecessary and senseless, as the page is eminently readable if you merely disable styles). The idea is that by eschewing maintenance nexuses, individual (virtual) hosts become unmaintainable, and thus are maintained only by replacing them. This is essentially a “rederivation only” model with regard to my previous article on normativity.

To conclude, nexuses are important. They allow systems to be effectively composed from parts, and provide vital maintainability. Nexus rejection is sometimes necessary but has severe costs. Thus, it's useful to notice when you're designing something which is, essentially, a nexus, and consider how to make it as uncontentious and universal as possible, to reduce the occurrence of rejection.

Additionally, by viewing nexuses as perhaps the core value proposition of traditional operating systems (in comparison to new unikernel-type offerings such as Mirage OS), operating systems in disguise can be recognised for what they are; where you have integration nexuses and maintenance nexuses in union, what you have is, for many intents and purposes, an operating system.

Addendum: What makes a language a language?

Nexuses are important. If a language provides nexuses, those nexuses are in some sense, part of the language. I use “language” here not in the usual technical sense, but in the looser sense that UML is a “language”; it's a common schema, a common set of lines by which software is drawn.

Suppose I have a library, and I tell you it's written in C. Think for a moment about what that implies. What can you infer from the fact that it's written in C? What you can't infer is any of the following:

What platforms or architectures it runs on
Whether it's garbage-collected
Whether it uses synchronous IO, asynchronous IO on top of some reactor, threads, fibres, select, poll, epoll, etc.
If it doesn't use synchronous IO, what IO library it uses
Whether it uses return codes for error handling, or errno, or some sort of setjmp/longjmp based solution.
Whether it uses the “object oriented C” style, or some sort of C-based object system like glib, or something else entirely.
In what format it expects command line arguments
What coding style it uses

This means that in effect, C is not a language but a family of languages. There may be no difference to a C compiler whether you're using synchronous or asynchronous IO, but it makes a world of difference to interoperability between different libraries. If you use asynchronous IO and there's a library that you might want to use, you can't easily use that library if it does synchronous IO, because it would block the thread. If you replace the question of language as in what compiler you use, with the much more useful question of language as in what libraries can I use?, C is not a language but a family of languages.

With C++ it's even worse, because nobody programs in C++ — everyone programs in their own unique subset of the language. If you are told a program is written in C++, that leaves the following questions:

Does it use, or eschew, exceptions?
Does it use, or eschew, C++ RTTI?
Does it use operator overloading?
Does it use, or eschew, the STL?
Does it use, or eschew, C++11/C++14/etc. features?
Does it use templates?
Does it use virtual functions?
Does it use multiple or virtual inheritance?
Does it use static objects with constructors?

Every C++ programmer has a different answer to these questions. If you're developing for a platform which doesn't support C++ exceptions, and you want to use a library which requires them, you're screwed. If you're developing for a platform which does support C++ exceptions, but your code eschews exceptions for certain reasons, you're also screwed: your codebase wasn't written to be exception safe. Even if you can use a library which chooses a different subset, the cooperation between the two feels somewhat grudging and mismatched.

The fact that a library is written in a given language is meaningless if everyone uses a different subset of that language. Thus, in language design, it is desirable to minimize features which are likely to be rejected. Languages should be restricted to the set of features which will be universally adopted by the user base. Can you imagine a Python library which eschews Python dictionaries and strings and invents its own from lists? Nobody does this — so at least when someone says something's written in Python we have that common ground, whereas when someone says something's written in C++ I have no idea how dictionaries, strings and lists will be represented (aside from a probably vain hope that it will use the STL).

This leads to the general concept of “cross-cutting considerations”. A cross-cutting consideration (CCC) is a consideration that affects essentially every line of code. A cross-cutting consideration is something that, if you change it, you basically have to throw away all of the code you've already written — or at least reread it all to ensure it's all still applicable, rewriting it where it isn't.

Changing the feature subset of a language you use is, of course, a cross-cutting consideration. See the lists for C and C++ above. Thus, the fewer cross-cutting considerations are necessary for programs written in a language, the more reusable code in that language is and the more likely you will be able to use any given library written in that language.

Take Python, for example. While everyone writing Python uses Python dictionaries, lists and strings, the I/O situation is much more bleak. If you want to do I/O in Python, you have so many options: normal synchronous I/O, Twisted, Stackless Python, the various Stackless Python-esque tasklet systems which can run on CPython, etc. This is a very big stain on Python and introduces massive cross-cutting consideration variation.

Go, on the other hand, doesn't have this problem. There is one way to do I/O in Go. Just like Python, nobody eschews Go slices (variable-length arrays) or maps (dictionaries); but in addition, nobody eschews the Go way of doing I/O either. This makes the statement “this library is written in Go” extremely meaningful, especially if you're doing a lot of network programming. The set of cross-cutting considerations is very small. Go's standard coding standard means that even coding style doesn't constitute a cross-cutting consideration, albeit by essentially fascist means. (Though Go code can be reformatted automatically. If you really hate the style, you can reformat it to what you like before working on it, then format it back before committing.)

Another thing that helps Go is that, unlike C++, there is a great lack of tacked-on features which only some people choose to use. No strange, forgotten language features like method pointers or virtual inheritance.

But another cross-cutting consideration is integration nexuses. I already explained some of Go's integration nexuses: the expvars, flag and net/http packages. I think that rejection of these nexuses poses a threat to Go because it increases the number of cross-cutting considerations and thus decreases library compatibility and increases the potential reasons for library rejection, damaging and fracturing the ecosystem and making it not so much of a language as a family of languages. The sheer improbability of having to reject a library in favour of doing it yourself in Go is one of the things that makes Go so great — along with the sheer ease of bringing in libraries, thanks to the wholly-implicit, convention-based build and packaging system.

In fact, it's quite remarkable just how un-rejected net/http (the standard HTTP client and server package) is. Essentially nobody uses a different HTTP server. In Python, interfaces like WSGI accommodate the variety of different Python HTTP servers or gateway interfaces out there. In Go, everyone uses net/http. This lack of rejection of a common library (which has compatibility implications, with regard to HTTP handlers written against it provided by other libraries) further decreases the set of cross-cutting considerations and schism within the language.

The rejection of the expvars and flag libraries by other libraries constitutes a form of nexus rejection which in some regard actually threatens Go as a language, by threatening to turn it more into a set of languages, no matter in how small a way. These are fractional issues I bring up, but they serve well as examples for a general trend, of fractured languages, which creates a tragic wastage of effort.

Which is why it is unfortunate that the expvars and flag packages, to put it simply, aren't very good. I myself am guilty as charged of rejecting these nexuses where on occasion, they simply don't meet my needs. But no matter how poor these packages may be at actually providing functionality, they excel in providing an integration nexus which acts as matchmaker from metric producers to consumers, and from command line argument consumers to parsers. Bizarrely, then, it may actually be better for these packages to provide no functionality at all, but the mere registration of objects implementing minimal interfaces, which consuming code can figure out how to usefully use when it comes to it. Go's support for interface upgrades could make this a viable strategy.

(If you aren't familiar with Go, interface upgrades essentially mean that you can cast an object of a given interface type to another type at runtime and find out whether it succeeded. In other words, it allows you to dynamically determine whether an object implements a method. In Go, support for interfaces is implicit; you don't need to explicitly declare that you implement an interface; if you implement the necessary methods, you're considered to implement that interface. Sometimes, code which receives an object of a given interface type will test whether it supports a superior interface and if so, use enhanced functionality. Otherwise, a standard level of functionality is used. These are called interface upgrades. Here's an article explaining the technique, which is used in some places in the Go standard library.)

Essentially, imagine a thread safe version of this:

type Expvar interface{}
var Expvars map[string]Expvar

This is, apparently, completely useless. What use is a map of objects you can't do anything with? The idea is that expvars implement methods that are applicable to them, and interface upgrades are used by consumers to figure out how to deal with them.

We can of course impose minimum requirements. Let's revise the Expvar type:

type Expvar interface {
  Name() string
  String() string
}

Now we are at least guaranteed that we can get the name, and some sort of string representation of every expvar. But the rest is for consuming code to figure out as it goes. The important thing is that this enables permissionless innovation. There's no need for the author of the nexus package to be consulted whenever improvements are desired; the author of a package providing expvars, and the author of a package consuming expvars, can simply decide to expose a new method as they see fit.

Sure, there's a lack of guarantees with this ‘wing-it’ attitude. But look at the stuff people store in DNS TXT records; there have been protests by standard writers that overloading TXT to have semantic meaning shouldn't be done, and a proper RRtype should be allocated. This led to the allocation of the SPF RRtype, which just stores the same data that a TXT SPF record would. But these protests — a call, essentially, for “get-permission-first” innovation — don't stand a chance in the face of the sheer pragmatism of the TXT record and the permissionless innovation it enables. The ‘wing-it’ attitude works.

Ultimately then, I believe that when writing a package which is essentially a nexus, one should consider making that package not actually do anything at all. It's hard to find a reason to reject a package that doesn't do anything — and the less nexus rejection there is, the more commonality there is between code. The more commonality there is, the more meaningful the language is as a category; the more reusable the code written in it.