Death to the Win32 console subsystem

The Win32 console subsystem is an unmitigated piece of crap, as probably anyone who has actually had to use, or program for it, will testify.

But I'm not writing this to address the failings of conhost.exe itself. (For those unaware, conhost.exe is the actual console program; instances of it are created automatically as necessary for Win32 “console” programs such as cmd.exe.)

No, rather I'm writing to hopefully make things easier for people who want to write Windows applications which are GUI applications but which can also usefully function as command line programs.

The centrepiece of the ridiculousness necessitating the hacks I deliberate upon below is, of course, the fact that you must compile a program as being either a “Windows” or “console” program. This is indicated to the OS in a flag set in the program image headers.

If you specify the subsystem as “Windows”, you get no console when launching the program. But at the same time, the process will immediately detach from the console if you execute it from, say, cmd.exe, leaving the program with no opportunity to print output to the console or take input from it. In this circumstance the best one can do is allocate a new console (AllocConsole), creating a wholly new window to process input or deliver output. This is a terrible user experience for a console program to offer, almost unusable.

So if you want to make a program usable from the command line you have to specify the subsystem as “Console”. Then the program works fine from the console, but you get a superflous popup console if the program is executed other than from the console. If the program can operate both as a GUI and a CLI program, this is probably not what you want.

There are four tricks I have discovered at various points which are useful for managing these issues:

You can detect whether a console window was created solely to service the program, or whether the console window already existed (e.g. because the program is being executed from cmd.exe).

bool win32con_is_exclusive(void) {
  DWORD pids[2];
  DWORD num_pids = GetConsoleProcessList(pids, ARRAYLEN(pids));
  return num_pids <= 1;
}

bool win32con_exists(void) {
 return !!GetConsoleWindow();
}

You can hide a console window to which you are attached.

By combining this with trick no. 1, you can hide a console if you detect that the console is exclusive to you (meaning that the program was not executed from the command line). This will leave a momentary flicker of the console before it is hidden, however; possibly longer if it takes time for Windows to load the program image and its dependencies. Supposedly this flicker can be somewhat mitigated using a shortcut set to start minimized.
```
void win32con_hide(void) {
  HWND wnd = GetConsoleWindow();
  if (wnd)
    ShowWindow(wnd, SW_HIDE);
}
```
You can launch a console process in such a way that its console window is never shown, not even briefly. To do this, set CREATE_NO_WINDOW in dwCreationFlags when calling CreateProcess.

The disadvantage of this is that it requires you to control the invocation of the process.
Even Windows 7 still supports executables with the extension .com (though of course they must be PE files), and will prefer these files when executing from the command line.

This means that you can have two copies of your program executable, one linked with the “Windows” subsystem named foo.exe and one with the “Console” subsystem named foo.com. Typing foo from the command line will launch foo.com, and you can point shortcuts to foo.exe.

Moreover, you can avoid the need to compile your program twice and combine this with trick no. 3 by creating a small stub program which you name foo.exe and which simply executes foo.com, passing its arguments but also setting CREATE_NO_WINDOW in the call to CreateProcess.

This stub program can either have the name of the .com program to execute hardcoded, or automatically determine the correct name by analyzing its own path and changing .exe to .com, or use both strategies with one as a fallback.

Death to the Win32 console subsystem

See also