Building A Rich Command Shell

What do I Want?

First up, what should we consider to be characteristics of a "rich command shell" for the purposes of this post?

For my interests, I want to see:

• Works in a regular terminal.
• Also works in terminals with more advanced features.
• Adapts to the size and capabilities of the terminal, although I recognize that feature negotiation doesn't work for everything, so some things will be controlled by settings.
• Works when not run in a terminal (like having output piped to a file or another process).
• Can be run in conjunction with a more specialized program that might present an even richer interface or embed the functionality somehow.

The shorter version of this is:

• It should work like things do today, in the existing environment.
• It should adaptively support richer means of interaction as well without violating typical assumptions of today.
• The means of adaptation should be flexible and allow progressively richer output and interaction models.

This might be boring to some people. They may want to replace the "everything is a stream of bytes" model of Unix. They may want to replace all existing terminals with something that supports full HTML and related technologies or be able to assume that all terminals support some sort of inline media display.

On the other hand, this might be reassuring to some people who would otherwise be afraid that "rich command shell" was going to mean the tools have a GUI, require a mouse, and are no longer scriptable or whatever.

The world evolves slowly and moves in fits and starts, and I'm fine with that.

Use Cases

What are some specific examples of things that I want to be sure are possible? What are some details that I'm less concerned about?

I want to be able to:

• Auto-detect or set a content type and see a different quality of output. The changes should be able to handle at least these levels of output:
  • Plain text with no style.
  • Text with ANSI coloring and standard VT codes.
  • Text with some form of inline media display, be that inline images such as PNGs or the support for things like Sixel or Regis graphics.
• Adapt the content to the width of the window. While this doesn't sound terribly exciting at first, this becomes more complicated when you're looking at structured output

I am less concerned that:

• Higher end display outputs may not be accessible from the same program, but may require compiling a plugin or shared library form of the program so that it can be loaded into the display shell that is providing the advanced interface.
• Not all functionality is present everywhere.

Pretty Printing as a Foundation

While thinking about this, I re-found a comment in the Open Dylan source code which seemed pretty relevant:

Program notes need to be stored and later displayed in browsers. This presents us with two problems. At the point of creation we have no way of knowing the column width that will be used when the note is displayed. There may even be more than one width if we want to be smart when a browser window is resized. A second problem arises if we store a program note in the form of a condition string + arguments and the arguments are context sensitive. We could just save everything as a string, but then the logical structure of the message is lost. An alternative is to store the text in a more structured form. This is the purpose of the <ppml> class and its derivatives. The interface is based on Oppen's 1980 TOPLAS paper.

To clarify a few things:

• Program notes are any output generated by the compiler to be consumed by the user.
• A browser window in the Open Dylan IDE is anything to browse some data, like a list of errors, not a "web browser".
• A condition is like an exception, especially insofar as the term is used here.
• The TOPLAS paper by Oppen is Prettyprinting by Derek C. Oppen, TOPLAS volume 2 number 4, ACM, 1980. I don't know of a freely available copy of this paper, but it has been used as the basis for incredibly large number of pretty printers over the years.

The basic idea is that you have a pipeline for output that goes from the program's data structures and messages to a list of (nested) pretty print control nodes, which is then rendered to the output device. To do this, you have 2 sets of functions:

• Program Data -> Pretty Print Control Node
• Pretty Print Control Node -> Output Device

So, you take the program data and messages, construct the pretty print control nodes and then format that to the output:

define compiler-sideways method print-object
    (condition :: <simple-warning>, stream :: <stream>) => ()
  let body = apply(format-to-ppml, condition.condition-format-string,
                                   condition.condition-format-arguments);
  let ppml-condition = ppml-block(vector(ppml-string("Warning: "),
                                         ppml-break(offset: 2, space: 0),
                                         body),
                                  offset: 0);
  ppml-print(ppml-condition,
             make(<ppml-printer>, margin: 100,
                  output-function:
                    method (s :: <string>) write(stream, s) end,
                  newline-function: method () write(stream, "\n") end));
end method print-object;

Moving On From Pretty Printing

The concepts and techniques of pretty printing appear to give us a good foundation to build upon:

• We can build a tree of nodes from our output.
• We can customize the rendering of these nodes based upon characteristics of the output device.

While the overall technique is a good approach, pretty printing control nodes are solving the single problem of how to flexibly layout output in the face of changing output widths. We'll need something more flexible to solve our problems and build a solid solution.

In my previous post on Rich Command Shells, I mentioned towards the end some examples from text-based games where a markup language was used to handle output so that it could be displayed in different ways depending on the output device.

This is the same shape of solution as the pretty printing: A pipeline for output that goes from the program's data structures and messages to a document, which is rendered to the output device.

In this case, we don't necessarily need an actual markup language and associated parser as we can just construct the documents in memory.

We do, however, need a rich set of node types that let us provide the structure and control over our document and that provide enough context to each rendering back-end for it to make a good decision about how to represent the data.

The Docutils Document Tree provides a fairly solid basis and is a useful starting point.

We'll look at this approach in a new post soon.