January 4, 2018

Harfbuzz Adventures

Lately, I've been thinking about text layout in Rust. We'll get back to that at some point. This led to me to looking at the Rust bindings for Harfbuzz. Since then, I've started some submitting some fixes to the Rust bindings, including working towards updating to the current version of Harfbuzz. (This was a couple of weeks ago. Then I went on an extended holiday.)

Then, I decided to look at making some improvements upstream in Harfbuzz. I've now had a few minor patch submissions (#669, #670) accepted there as well and am working on more, including improving the support for the cmake build system.

One annoying thing is that since I use ninja as a build tool (instead of make), I have to modify many projects to include this bit of configuration (#674):

if (UNIX AND CMAKE_GENERATOR STREQUAL "Ninja")
  if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
    set (CMAKE_CXX_FLAGS "-fcolor-diagnostics ${CMAKE_CXX_FLAGS}")
    set (CMAKE_C_FLAGS "-fcolor-diagnostics ${CMAKE_C_FLAGS}")
  endif ()
  if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
    set (CMAKE_CXX_FLAGS "-fdiagnostics-color ${CMAKE_CXX_FLAGS}")
    set (CMAKE_C_FLAGS "-fdiagnostics-color ${CMAKE_C_FLAGS}")
  endif ()
endif ()

This gets old! I wish this could be fixed upstream somewhere, but I haven't had the time / motivation myself to think about this.

I also noticed when building on macOS that …

read more »

GraphQL beyond the Web

GraphQL is (from their site):

DECLARATIVE
Query responses are decided by the client rather than the server. A GraphQL query returns exactly what a client asks for and no more.
COMPOSITIONAL
A GraphQL query itself is a hierarchical set of fields. The query is shaped just like the data it returns. It is a natural way for product engineers to describe data requirements.
STRONG-TYPED
A GraphQL query can be ensured to be valid within a GraphQL type system at development time allowing the server to make guarantees about the response. This makes it easier to build high-quality client tools.

The GraphQL website has a lot of additional information, documentation, and some initial implementations.

I'm going to assume that you're at least roughly familiar with it, but as a quick example, this query:

{
  hero {
    id
    name
    friends {
      id
      name
    }
  }
}

This query could return something like:

{
  "data": {
    "hero": {
      "id": "2001",
      "name": "R2-D2",
      "friends": [
        {
          "id": "1000",
          "name": "Luke Skywalker"
        },
        {
          "id": "1002",
          "name": "Han Solo"
        },
        {
          "id": "1003",
          "name": "Leia Organa"
        }
      ]
    }
  }
}

A simpler query over the same data that is parameterized might look like:

query FetchSomeIDQuery($someId: String!) {
  human(id: $someId) {
    name
  }
}

And a response from that, given a value of 1000 for $someId would …

read more »

Finding a User Interface library

I'm building something for building developer tools. That's pretty vague, but I'd like to keep it that way for now. To do this, I wanted to use an existing user interface library. Unfortunately, it seems like it is always impossible to find what I want. I apparently want too much.

My Requirements

So, what am I looking for? These are just my own personal desires and I certainly don't expect that everyone will agree with them.

React
I'm planning to use React in my rendering layer, so it would be nice if the user interface framework that I use has solid support for React. This seems like a no-brainer. That said, there is another layer in my architecture which describes user interfaces and that is what gets lowered or converted to the actual widget definitions. In theory, this means that multiple view layers could exist (perhaps even using something like React Native). But for the start, I need solid React support.
TypeScript
Much like my desire for React support, I am writing my code in TypeScript. That doesn't mean that the UI has to be in TypeScript, but it would make it more convenient. Otherwise, typing definition files (.d.ts …
read more »

Querying OS Information

For a project that I'm working on, I would like to be able to retrieve information from the OS about a variety of things:

  • File system
  • Current processes running
  • Process memory maps
  • Open file handles in a process
  • File attributes
  • File type, mime type
  • ... and a lot more!

Some of these are very easy and are just a set of calls to POSIX functions. Others are highly dependent on the underlying platform.

I want to be able to access this data from an application that is either running natively or from within node.js when using Electron. I want it on a minimum of Mac OS X, Linux, FreeBSD and Windows. Support for other platforms such as NetBSD would be nice.

I don't want this to be very large. The overall code required is likely to be fairly small. Hopefully, a large number of dependencies would not be brought in by using a library that provides this information.

I want the data in JSON-LD format. This is JSON, but with some additional fields, like @type, that help my application present the data correctly. It would be nice if there were a way to subset the data such that some of …

read more »

Building A Rich Command Shell

Previously, I wrote about some existing examples of Rich Command Shells, some from the past, some from the present.

Now, I'd like to look at this from the practical side of things. I have some software that I would like to see have a rich command shell.

What do I Want?

First up, what should we consider to be characteristics of a "rich command shell" for the purposes of this post?

For my interests, I want to see:

  • Works in a regular terminal.
  • Also works in terminals with more advanced features.
  • Adapts to the size and capabilities of the terminal, although I recognize that feature negotiation doesn't work for everything, so some things will be controlled by settings.
  • Works when not run in a terminal (like having output piped to a file or another process).
  • Can be run in conjunction with a more specialized program that might present an even richer interface or embed the functionality somehow.

The shorter version of this is:

  • It should work like things do today, in the existing environment.
  • It should adaptively support richer means of interaction as well without violating typical assumptions of today.
  • The means of adaptation should be flexible and allow progressively …
read more »

Compiling to Javascript or not?

JavaScript is commonly seen as a flawed language. It has plenty of flaws, some of which are more painful than others. One common way that people deal with these flaws is to use another language that compiles to JavaScript and to then write their code in that other language.

At first, this sounds pretty appealing. You get to choose which of a few languages to use to address your complaints with JavaScript. Unhappy with the type system? Try out Dart, TypeScript, AtScript, Elm, or even PureScript. Want a more concise syntax? Perhaps CoffeeScript is your thing. Do you wish you were using a Lisp? Well, there are lots of options there as well, starting with ClojureScript or Parenscript. There are many choices.

My Project

My project involves creating a platform for building tools that is based on Atom Shell and uses a plug-in model to allow users to extend the core platform.

I also have some requirements:

  • I need access to things that will be coming in future versions of JavaScript (and fairly soon). An example of this is support for working with 64 bit integers (a firm requirement).
  • Given that we have a single deployment platform (Atom Shell, with …
read more »

Looking at Packet Capture and Dissection

For the project that I am working on (not yet disclosed), I was thinking that it would be interesting to be able to integrate a view of what is happening on the network, much like Wireshark and other tools can provide. The view would be more targeted towards what the user was doing, but the overall idea would be the same: capture network traffic and perform some basic analysis on it to display it visually.

This led to some interesting research, which I've decided to discuss here! (For those who feel this is a long post, there's a summary of sorts at the end.)

Capturing Traffic

node_pcap

Given that I'm using Atom Shell to build this application and that uses Node.JS, one of my first thoughts was to take a look at node_pcap. After all, node_pcap was used successfully in tools like htracr, so perhaps it would work well here.

Unfortunately, I ran into a series of issues with node_pcap. For one thing, it wants to run the capture in the same process, and since performing a capture requires putting the network interface into promiscuous mode, it requires elevated privileges. I am not comfortable with the idea that my …

read more »

Atom Shell and ES6 Harmony

When building an application using the Atom Shell, you may want to take advantage of newer Javascript features from ES6 (Harmony) like arrow functions, generators, for ... of loops and many other things which are not enabled in the V8 Javascript engine by default.

Fortunately, enabling this is pretty easy!

app.on('ready', function() {
  app.commandLine.appendSwitch('js-flags', '--harmony');
  ...
}

When starting up your Atom Shell application, you can append switches to the command line that will be given to the embedded Chromium browser engine.

Note that if you have an incorrect flag being passed, the other flags that you set may be ignored, so keep an eye on this as you update to newer versions of Atom Shell as the valid flags may have changed.

Unfortunately, it can be confusing figuring out which version of V8 is being used and what ES6 features are available. As of the time of this writing, Atom Shell 0.18.1 is using a Chromium 38 and so it falls in between the Chromium 37 and 39 columns on this compatibility chart. (But don't look at the Node Harmony column for figuring out what Atom Shell supports!)

read more »

Rich Command Shells

Many applications benefit from a mix of a graphical and textual interface. This has ranged from entire environments, like the Lisp Machine and the follow-on Common Lisp Interface Manager (CLIM), to specific applications like Mathematica and others which employ a "notebook interface". There have also been various attempts at producing richer terminal applications which can work with either standard or specialized applications to produce richer, more interactive output. Graphically enhanced textual interfaces have also found their place in various genres of games over the years.

In our case, we're interested in enhancing the interaction between a user and a specific tool. We'll take a look here at some of the ideas that have come before to get an idea of what people have already done to inform what we do for ourselves.

Some of the things discussed here are old, some are new. Some are long lost and forgotten (one link below is to archive.org) while others are actively being worked on today.

Common Lisp Interface Manager

Like many things, early research into graphically enhanced and interactive textual interfaces was carried out within the Lisp world. An early paper Presentation Based User Interfaces by Eugene C. Ciccarelli IV at …

read more »

Initial Thoughts on Polymer

I've been doing some web development off and on for many years (since 1996). I started out in an interpreted language on top of an object database that was serving up a web-based discussion forum that I was writing and shipping. I moved on from there to various other tools over the years including various home-grown templating languages, Django templates, Jinja2 and others. I helped produce web applications using ExtJS, others using jQuery / jQuery UI, and experimented with various other frameworks over the years. I've followed along with what people are doing in ReactJS and related tools like Om. It is all interesting.

A Recent Project

I recently built a web-based UI for a memory / heap profiler for emscripten. This is just some Jinja2 templates and Flask on the server side. The UI is using Bootstrap and went through several iterations of various JS tools and frameworks for handling graphs, grids, and charts. This was something of a nightmare (and honestly, the code still is terrible). Many tools didn't play well together. Some had issues due to CSS or JS clashing. Others just wanted to control more than I wanted to let them control. Getting jQuery UI, Bootstrap, Slick Grid …

read more »