Notes from a Wayward Monkeyhttp://waywardmonkeys.org/2018-01-04T00:00:00+00:00January 4, 20182018-01-04T00:00:00+00:002018-01-04T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2018-01-04:/2018/01/04/notebook/<div class="section" id="harfbuzz-adventures">
<h2>Harfbuzz Adventures</h2>
<p>Lately, I've been thinking about text layout in Rust. We'll get back
to that at some point. This led to me to looking at the <a class="reference external" href="https://github.com/servo/rust-harfbuzz">Rust bindings</a>
for <a class="reference external" href="http://www.harfbuzz.org/">Harfbuzz</a>. Since then, I've started some submitting some fixes to
the Rust bindings, including working towards updating to the current
version of Harfbuzz. (This was a couple of weeks ago. Then I went on an
extended holiday.)</p>
<p>Then, I decided to look at making some improvements upstream in Harfbuzz.
I've now had a few minor patch submissions (<a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/669">#669</a>, <a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/670">#670</a>) accepted
there as well and am working on more, including improving the support
for the <a class="reference external" href="https://www.cmake.org/">cmake</a> build system.</p>
<p>One annoying thing is that since I use <a class="reference external" href="https://ninja-build.org/">ninja</a> as a build tool (instead
of <tt class="docutils literal">make</tt>), I have to modify many projects to include this bit of
configuration (<a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/674">#674</a>):</p>
<div class="highlight"><pre><span></span><span class="nb">if</span> <span class="p">(</span><span class="s">UNIX</span> <span class="s">AND</span> <span class="s">CMAKE_GENERATOR</span> <span class="s">STREQUAL</span> <span class="s2">"Ninja"</span><span class="p">)</span>
<span class="nb">if</span> <span class="p">(</span><span class="s">CMAKE_CXX_COMPILER_ID</span> <span class="s">STREQUAL</span> <span class="s2">"Clang"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_CXX_FLAGS</span> <span class="s2">"-fcolor-diagnostics ${CMAKE_CXX_FLAGS}"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_C_FLAGS</span> <span class="s2">"-fcolor-diagnostics ${CMAKE_C_FLAGS}"</span><span class="p">)</span>
<span class="nb">endif</span> <span class="p">()</span>
<span class="nb">if</span> <span class="p">(</span><span class="s">CMAKE_CXX_COMPILER_ID</span> <span class="s">STREQUAL</span> <span class="s2">"GNU"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_CXX_FLAGS</span> <span class="s2">"-fdiagnostics-color ${CMAKE_CXX_FLAGS}"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_C_FLAGS</span> <span class="s2">"-fdiagnostics-color ${CMAKE_C_FLAGS}"</span><span class="p">)</span>
<span class="nb">endif</span> <span class="p">()</span>
<span class="nb">endif</span> <span class="p">()</span>
</pre></div>
<p>This gets old! I wish this could be fixed upstream somewhere, but I
haven't had the time / motivation myself to think about this.</p>
<p>I also noticed when building on macOS that …</p></div><div class="section" id="harfbuzz-adventures">
<h2>Harfbuzz Adventures</h2>
<p>Lately, I've been thinking about text layout in Rust. We'll get back
to that at some point. This led to me to looking at the <a class="reference external" href="https://github.com/servo/rust-harfbuzz">Rust bindings</a>
for <a class="reference external" href="http://www.harfbuzz.org/">Harfbuzz</a>. Since then, I've started some submitting some fixes to
the Rust bindings, including working towards updating to the current
version of Harfbuzz. (This was a couple of weeks ago. Then I went on an
extended holiday.)</p>
<p>Then, I decided to look at making some improvements upstream in Harfbuzz.
I've now had a few minor patch submissions (<a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/669">#669</a>, <a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/670">#670</a>) accepted
there as well and am working on more, including improving the support
for the <a class="reference external" href="https://www.cmake.org/">cmake</a> build system.</p>
<p>One annoying thing is that since I use <a class="reference external" href="https://ninja-build.org/">ninja</a> as a build tool (instead
of <tt class="docutils literal">make</tt>), I have to modify many projects to include this bit of
configuration (<a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/674">#674</a>):</p>
<div class="highlight"><pre><span></span><span class="nb">if</span> <span class="p">(</span><span class="s">UNIX</span> <span class="s">AND</span> <span class="s">CMAKE_GENERATOR</span> <span class="s">STREQUAL</span> <span class="s2">"Ninja"</span><span class="p">)</span>
<span class="nb">if</span> <span class="p">(</span><span class="s">CMAKE_CXX_COMPILER_ID</span> <span class="s">STREQUAL</span> <span class="s2">"Clang"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_CXX_FLAGS</span> <span class="s2">"-fcolor-diagnostics ${CMAKE_CXX_FLAGS}"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_C_FLAGS</span> <span class="s2">"-fcolor-diagnostics ${CMAKE_C_FLAGS}"</span><span class="p">)</span>
<span class="nb">endif</span> <span class="p">()</span>
<span class="nb">if</span> <span class="p">(</span><span class="s">CMAKE_CXX_COMPILER_ID</span> <span class="s">STREQUAL</span> <span class="s2">"GNU"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_CXX_FLAGS</span> <span class="s2">"-fdiagnostics-color ${CMAKE_CXX_FLAGS}"</span><span class="p">)</span>
<span class="nb">set</span> <span class="p">(</span><span class="s">CMAKE_C_FLAGS</span> <span class="s2">"-fdiagnostics-color ${CMAKE_C_FLAGS}"</span><span class="p">)</span>
<span class="nb">endif</span> <span class="p">()</span>
<span class="nb">endif</span> <span class="p">()</span>
</pre></div>
<p>This gets old! I wish this could be fixed upstream somewhere, but I
haven't had the time / motivation myself to think about this.</p>
<p>I also noticed when building on macOS that there were a large number of
warnings due to usage of some APIs that had been deprecated with macOS 10.12.
Since this was over a year old, I figured that I would take a look. While
I consider working on making it use C++11 <tt class="docutils literal"><span class="pre">std::atomic</span></tt>, I submitted an
easy fix (<a class="reference external" href="https://github.com/harfbuzz/harfbuzz/pull/676">#676</a>) for now that makes it use the compiler primitives that
are already used on Linux.</p>
<p>It looks like there's a good bit of stuff in Harfbuzz that could be cleaned
up some. I started looking at it to improve the docs so that the Rust
bindings would have better docs, but now I'm working on the build system
and looking at some minor issues in the code.</p>
</div>
<div class="section" id="gtk-doc">
<h2>GTK-Doc</h2>
<p>I started working on getting the docs for Harfbuzz to build with <tt class="docutils literal">cmake</tt>
as well. This led me to finding that Harfbuzz uses <a class="reference external" href="https://www.gtk.org/gtk-doc/">GTK-Doc</a> for its
documentation. Unfortunately, the integration files that are shipped with
GTK-Doc are broken. I've submitted <a class="reference external" href="https://bugzilla.gnome.org/show_bug.cgi?id=792148">a fix upstream</a>. I noticed that others
have run into this as well, but no fix was done.</p>
<p>There are still things to do here. I suspect that the docs for using cmake
with GTK-Doc are not entirely correct. And now that someone is using it,
there may need to be a way to pass different flags or otherwise improve
the process.</p>
</div>
<div class="section" id="z3">
<h2>Z3</h2>
<p>This week, I also submitted some fixes for minor issues in the <a class="reference external" href="https://github.com/Z3Prover/z3">Z3</a> theorem
prover that I'd been working on over the holidays. These weren't terribly
exciting, but are a start on helping to improve code quality. I hope to
continue doing this at random points in time in the future.</p>
</div>
<div class="section" id="spectre-and-meltdown">
<h2>Spectre and Meltdown</h2>
<p>The big news today has been the release of the information about the <a class="reference external" href="https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html">attacks
on CPUs</a>.</p>
<p>I wonder how RISC-V will respond and how vulnerable (or not) it is to these
sorts of things. Does it benefit by being less advanced? Is this a point at
which having the full source for your CPU could be an asset, and you aren't
necessarily at the mercy of a single vendor, like Intel? (Not that this would
be an easy problem to fix for anyone...)</p>
<p>Also, how will this impact all of the hobbyist and other small scale OSes
out there? SeL4 has <a class="reference external" href="http://sel4.systems/pipermail/devel/2018-January/001809.html">indicated</a> that they're working om this. How about Fuchsia
and others?</p>
<p>What impact might this have on the design of a kernel? If syscalls (or really
any transition between security contexts) must become more expensive, is there
room to think about changing how we think about syscalls? In the past, I've
thought about how many programs are effectively querying the kernel for data
piece by piece rather than being able to write a single query that would
return the data that they want, much like GraphQL or SPARQL. I'm not suggesting
using an actual query language with a textual representation, but treating
things within the OS as items in a graph and being able to operate at
that level might be interesting.</p>
<p>Similarly, it seems like some of the mitigations being discussed and pushed
out could be done in a better way in a language like Rust than in C / C++.
An example of this might be <a class="reference external" href="https://developer.arm.com/support/security-update/compiler-support-for-mitigations">ARM's __builtin_load_no_speculate</a> as a new
compiler primitive.</p>
</div>
GraphQL beyond the Web2016-03-15T00:00:00+00:002016-03-15T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2016-03-15:/2016/03/15/graphql-beyond-the-web/<p><a class="reference external" href="http://graphql.org/">GraphQL</a> is (from their site):</p>
<blockquote>
<dl class="docutils">
<dt>DECLARATIVE</dt>
<dd>Query responses are decided by the client rather than the
server. A GraphQL query returns exactly what a client asks
for and no more.</dd>
<dt>COMPOSITIONAL</dt>
<dd>A GraphQL query itself is a hierarchical set of fields. The
query is shaped just like the data it returns. It is a natural
way for product engineers to describe data requirements.</dd>
<dt>STRONG-TYPED</dt>
<dd>A GraphQL query can be ensured to be valid within a GraphQL
type system at development time allowing the server to make
guarantees about the response. This makes it easier to build
high-quality client tools.</dd>
</dl>
</blockquote>
<p>The <a class="reference external" href="http://graphql.org/">GraphQL</a> website has a lot of additional information, documentation,
and some initial implementations.</p>
<p>I'm going to assume that you're at least roughly familiar with it, but
as a quick example, this query:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="nx">hero</span> <span class="p">{</span>
<span class="nx">id</span>
<span class="nx">name</span>
<span class="nx">friends</span> <span class="p">{</span>
<span class="nx">id</span>
<span class="nx">name</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>This query could return something like:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="s2">"data"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"hero"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"2001"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"R2-D2"</span><span class="p">,</span>
<span class="s2">"friends"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"1000"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Luke Skywalker"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"1002"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Han Solo"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"1003"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Leia Organa"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>A simpler query over the same data that is parameterized might look like:</p>
<pre class="code javascript literal-block">
<span class="nx">query</span> <span class="nx">FetchSomeIDQuery</span><span class="p">(</span><span class="nx">$someId</span><span class="o">:</span> <span class="nb">String</span><span class="o">!</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">human</span><span class="p">(</span><span class="nx">id</span><span class="o">:</span> <span class="nx">$someId</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">name</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>And a response from that, given a value of <tt class="docutils literal">1000</tt> for <tt class="docutils literal">$someId</tt> would …</p><p><a class="reference external" href="http://graphql.org/">GraphQL</a> is (from their site):</p>
<blockquote>
<dl class="docutils">
<dt>DECLARATIVE</dt>
<dd>Query responses are decided by the client rather than the
server. A GraphQL query returns exactly what a client asks
for and no more.</dd>
<dt>COMPOSITIONAL</dt>
<dd>A GraphQL query itself is a hierarchical set of fields. The
query is shaped just like the data it returns. It is a natural
way for product engineers to describe data requirements.</dd>
<dt>STRONG-TYPED</dt>
<dd>A GraphQL query can be ensured to be valid within a GraphQL
type system at development time allowing the server to make
guarantees about the response. This makes it easier to build
high-quality client tools.</dd>
</dl>
</blockquote>
<p>The <a class="reference external" href="http://graphql.org/">GraphQL</a> website has a lot of additional information, documentation,
and some initial implementations.</p>
<p>I'm going to assume that you're at least roughly familiar with it, but
as a quick example, this query:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="nx">hero</span> <span class="p">{</span>
<span class="nx">id</span>
<span class="nx">name</span>
<span class="nx">friends</span> <span class="p">{</span>
<span class="nx">id</span>
<span class="nx">name</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>This query could return something like:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="s2">"data"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"hero"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"2001"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"R2-D2"</span><span class="p">,</span>
<span class="s2">"friends"</span><span class="o">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"1000"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Luke Skywalker"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"1002"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Han Solo"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="s2">"id"</span><span class="o">:</span> <span class="s2">"1003"</span><span class="p">,</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Leia Organa"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>A simpler query over the same data that is parameterized might look like:</p>
<pre class="code javascript literal-block">
<span class="nx">query</span> <span class="nx">FetchSomeIDQuery</span><span class="p">(</span><span class="nx">$someId</span><span class="o">:</span> <span class="nb">String</span><span class="o">!</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">human</span><span class="p">(</span><span class="nx">id</span><span class="o">:</span> <span class="nx">$someId</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">name</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>And a response from that, given a value of <tt class="docutils literal">1000</tt> for <tt class="docutils literal">$someId</tt> would be:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="s2">"data"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"human"</span><span class="o">:</span> <span class="p">{</span>
<span class="s2">"name"</span><span class="o">:</span> <span class="s2">"Luke Skywalker"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>This gives a very rough idea of what GraphQL can look like and a brief
demonstration of how the query can determine the "shape" of the data
that is returned.</p>
<div class="section" id="beyond-the-web">
<h2>Beyond the Web</h2>
<p>When I saw GraphQL, I started thinking about how this could improve some
other systems that I work with.</p>
<div class="section" id="lldb">
<h3>LLDB</h3>
<p>For example, I have been working with <a class="reference external" href="http://lldb.llvm.org/">LLDB</a> off and on over the last
year and a half. The code involved in getting some data about the
current threads and their stacks is somewhat tedious:</p>
<pre class="code python literal-block">
<span class="k">for</span> <span class="n">thread_idx</span><span class="p">,</span> <span class="n">thread</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">process</span><span class="p">):</span>
<span class="k">for</span> <span class="n">frame_idx</span><span class="p">,</span> <span class="n">frame</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">thread</span><span class="p">):</span>
<span class="n">function</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">GetFunctionName</span><span class="p">()</span>
<span class="n">function_name</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">GetFunctionName</span><span class="p">()</span> <span class="ow">or</span> <span class="s1">''</span>
<span class="n">address</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">GetPCAddress</span><span class="p">()</span><span class="o">.</span><span class="n">GetLoadAddress</span><span class="p">(</span><span class="n">target</span><span class="p">)</span>
<span class="n">module</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">GetModule</span><span class="p">()</span><span class="o">.</span><span class="n">GetFileSpec</span><span class="p">()</span><span class="o">.</span><span class="n">GetFilename</span><span class="p">()</span>
<span class="n">file_name</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">GetLineEntry</span><span class="p">()</span><span class="o">.</span><span class="n">GetFileSpec</span><span class="p">()</span><span class="o">.</span><span class="n">GetFilename</span><span class="p">()</span>
<span class="n">line_number</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">GetLineEntry</span><span class="p">()</span><span class="o">.</span><span class="n">GetLine</span><span class="p">()</span>
</pre>
<p>Each of these calls crosses from Python into LLDB. This creates
a pretty big surface area for the API, that while flexible, is
pretty substantial. Additionally, some times a request to LLDB
for data might involve a call over IPC (or even RPC) to the
<tt class="docutils literal"><span class="pre">lldb-server</span></tt> or the actual process that is being debugged.</p>
<p>This makes managing responsiveness and delays in updating a
user interface more cumbersome (chained promises, etc).</p>
</div>
<div class="section" id="querying-os-information">
<h3>Querying OS Information</h3>
<p>For another project, I discussed in a <a class="reference external" href="http://waywardmonkeys.org/2016/03/07/querying-os-information/">recent post</a>, how I'd like
to be able to make queries against the underlying OS for things like
process lists, open file handles, process memory maps and a lot more.</p>
<p>Rather than a traditional API, I would find it useful to be able
to get back a bunch of JSON.</p>
<p>Some of the information that my application needs is very specific
to a particular view, so just getting <em>everything</em> back in a JSON
blob isn't ideal. Some information is more expensive to gather and
maintain, again making getting everything at once less than ideal.</p>
<p>But I also don't want a traditional API where I have to get data
piece by piece.</p>
</div>
<div class="section" id="graphql">
<h3>GraphQL?</h3>
<p>GraphQL seems to be an interesting option to improve the developer
experience here.</p>
<p>A query against LLDB to fetch a bunch of information about current
threads and stack frames might look like:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="nx">thread</span> <span class="p">{</span>
<span class="nx">id</span>
<span class="nx">name</span>
<span class="nx">frames</span> <span class="p">{</span>
<span class="nx">id</span>
<span class="nx">name</span>
<span class="nx">pc</span><span class="o">-</span><span class="nx">address</span>
<span class="kd">function</span> <span class="p">{</span>
<span class="nx">address</span>
<span class="nx">name</span>
<span class="nx">module</span> <span class="p">{</span>
<span class="nx">name</span>
<span class="p">}</span>
<span class="nx">file_name</span>
<span class="nx">line_number</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>This would allow LLDB to optimize this however it likes to avoid
round trips. From the perspective of the developer working with
LLDB, it would be a single request that returns with all of the
relevant data. LLDB could even send this query over the wire to
the <tt class="docutils literal"><span class="pre">lldb-server</span></tt> to avoid even more overhead.</p>
<p>It also feels like it would be an API simplification, for the
parts that are associated with querying for data.</p>
<p>On the OS introspection side of things, a GraphQL query for
the current process list might just be:</p>
<pre class="code javascript literal-block">
<span class="p">{</span>
<span class="nx">process</span> <span class="p">{</span>
<span class="nx">pid</span>
<span class="nx">ppid</span>
<span class="nx">command</span>
<span class="nx">time</span>
<span class="nx">mem</span>
<span class="p">}</span>
<span class="p">}</span>
</pre>
<p>A more detailed view of the process list or a highly detailed
view of a single process would modify that query to fetch the
data that they need.</p>
</div>
</div>
<div class="section" id="only-json">
<h2>Only JSON?</h2>
<p>One concern that is commonly voiced is "What if I don't want JSON?"</p>
<p>Perhaps you want data in your own efficient encoding like <a class="reference external" href="http://msgpack.org/">msgpack</a>
or <a class="reference external" href="http://cbor.io/">CBOR</a> or directly into your application structures. Perhaps you
are running with a JS engine in the same process and want to directly
create the JS engine's objects rather than first constructing some
JSON and then parsing it.</p>
<p>For this, I think it would be a good idea to have a <tt class="docutils literal">value_builder</tt>
interface that can be implemented by the calling application and
passed along with a query. The interface should probably be event-based
like SAX parsers of old, so that it has functions like <tt class="docutils literal"><span class="pre">start-object</span></tt>,
<tt class="docutils literal"><span class="pre">start-array</span></tt>, <tt class="docutils literal"><span class="pre">start-object</span></tt>, <tt class="docutils literal"><span class="pre">start-key</span></tt>, <tt class="docutils literal"><span class="pre">end-object</span></tt>, and
so on. This API could expose a richer type system than JSON natively
offers.</p>
</div>
<div class="section" id="an-update">
<h2>An Update</h2>
<p><a class="reference external" href="https://github.com/krytarowski">Kamil Rytarowski</a>, a NetBSD hacker, has decided to write a library
called <a class="reference external" href="https://github.com/krytarowski/netquery">netquery</a> which will hopefully allow the sort of OS queries
that I want to see. This should be a great opportunity to experiment
with <a class="reference external" href="http://graphql.org/">GraphQL</a> beyond the web.</p>
<p>I look forward to contributing to <a class="reference external" href="https://github.com/krytarowski/netquery">netquery</a>.</p>
</div>
Finding a User Interface library2016-03-14T00:00:00+00:002016-03-14T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2016-03-14:/2016/03/14/finding-a-user-interface-library/<p>I'm building something for building developer tools. That's pretty
vague, but I'd like to keep it that way for now. To do this, I
wanted to use an existing user interface library. Unfortunately,
it seems like it is always impossible to find what I want. I apparently
want too much.</p>
<div class="section" id="my-requirements">
<h2>My Requirements</h2>
<p>So, what am I looking for? These are just my own personal desires and
I certainly don't expect that everyone will agree with them.</p>
<dl class="docutils">
<dt><strong>React</strong></dt>
<dd>I'm planning to use <a class="reference external" href="http://facebook.github.io/react/">React</a> in my rendering layer, so it would be nice if
the user interface framework that I use has solid support for React.
This seems like a no-brainer. That said, there is another layer in my
architecture which describes user interfaces and that is what gets
lowered or converted to the actual widget definitions. In theory, this
means that multiple view layers could exist (perhaps even using something
like React Native). But for the start, I need solid React support.</dd>
<dt><strong>TypeScript</strong></dt>
<dd>Much like my desire for React support, I am writing my code in <a class="reference external" href="http://www.typescriptlang.org/">TypeScript</a>.
That doesn't mean that the UI <em>has</em> to be in TypeScript, but it would
make it more convenient. Otherwise, typing definition files (<tt class="docutils literal">.d.ts …</tt></dd></dl></div><p>I'm building something for building developer tools. That's pretty
vague, but I'd like to keep it that way for now. To do this, I
wanted to use an existing user interface library. Unfortunately,
it seems like it is always impossible to find what I want. I apparently
want too much.</p>
<div class="section" id="my-requirements">
<h2>My Requirements</h2>
<p>So, what am I looking for? These are just my own personal desires and
I certainly don't expect that everyone will agree with them.</p>
<dl class="docutils">
<dt><strong>React</strong></dt>
<dd>I'm planning to use <a class="reference external" href="http://facebook.github.io/react/">React</a> in my rendering layer, so it would be nice if
the user interface framework that I use has solid support for React.
This seems like a no-brainer. That said, there is another layer in my
architecture which describes user interfaces and that is what gets
lowered or converted to the actual widget definitions. In theory, this
means that multiple view layers could exist (perhaps even using something
like React Native). But for the start, I need solid React support.</dd>
<dt><strong>TypeScript</strong></dt>
<dd>Much like my desire for React support, I am writing my code in <a class="reference external" href="http://www.typescriptlang.org/">TypeScript</a>.
That doesn't mean that the UI <em>has</em> to be in TypeScript, but it would
make it more convenient. Otherwise, typing definition files (<tt class="docutils literal">.d.ts</tt>)
will need to be maintained and kept up to date.</dd>
<dt><strong>Documentation</strong></dt>
<dd>Fortunately, with the widespread availability of component library
pages that are full of examples, documentation is much more readily
available today than in the past. But some libraries still try
to get away without having good documentation or documentation that
is patchy and inconsistent in quality. World class tools need
world class documentation.</dd>
<dt><strong>Accessibility</strong></dt>
<dd>Having an application be accessible is important. It is important
to serve disabled users well, but since we're building development
tools, it is important as many developers enjoy being able to use
tools without a mouse. In addition to providing <a class="reference external" href="https://www.w3.org/WAI/intro/aria">ARIA</a> attributes,
we will also need to provide rich keyboard navigation for elements
such as tab widgets, sliders and select boxes.</dd>
<dt><strong>Theming</strong></dt>
<dd>Everyone wants theming these days. Even better is theming that lets
someone customize the look and feel of a framework fairly easily
without having to modify each and every widget. There are a lot
of approaches that are being taken with this in the React world
including css-modules and inline styles. I don't know how I want to
see this solved in my UI framework of choice, just that I do
want to see it have integrated theming support throughout.</dd>
<dt><strong>Look and Feel</strong></dt>
<dd>Related to theming, I'm hoping that the default look and feel isn't
trying to mimic a given platform as that never ends up working out
very well.</dd>
<dt><strong>Internationalization / localization</strong></dt>
<dd>In addition to supporting localized display of numbers, dates,
and other information, all text messages should be localized.
Like theming, this is something that needs to be built into
the framework in a consistent and comprehensive way.</dd>
<dt><strong>Actively Maintained</strong></dt>
<dd>I don't want to use a library which isn't being actively
maintained. I'd like to see recent activity, both in terms
of improvements as well as bug fixes.</dd>
<dt><strong>Licensing</strong></dt>
<dd>I prefer fairly open licensing.</dd>
</dl>
</div>
<div class="section" id="ant-design">
<h2>Ant Design</h2>
<p>As is often the case, my apparent idealism has led me to a place
that I didn't really expect to find myself.</p>
<p>I've looked at several libraries, including, but not limited to:</p>
<ul class="simple">
<li>Material UI</li>
<li>React-Photonkit</li>
<li>Ant Design</li>
</ul>
<p>In the end, I think that I'm going to go with <a class="reference external" href="http://ant.design/docs/react/introduce">Ant Design</a> and
help to evolve it in directions that match my own goals. If that
fails, I can always fork it and build on it.</p>
<p>Ant Design is the work of a team at Alibaba in China.</p>
<p>How well does Ant Design meet my requirements?</p>
<p><strong>React</strong></p>
<p>Ant Design is written for React, so there is no issue here.</p>
<p>Ant Design is a re-packaging of a <a class="reference external" href="http://react-component.github.io/badgeboard/">number of components</a> written by the
same people. These components are available separately (and individually).
They have also been used by other UI frameworks like <a class="reference external" href="http://uxco.re/">UXCore</a>. (Interestingly,
UXCore is the work of another team at Alibaba.)</p>
<p><strong>TypeScript</strong></p>
<p>While it is not written in TypeScript, it does have a set of typings
available in <a class="reference external" href="https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/antd">DefinitelyTyped</a>. This is a new development from the
last couple of weeks.</p>
<p><strong>Documentation</strong></p>
<p>Ant Design has documentation for each widget and what properties that
widget has. There are working examples of many of the features provided.
The one issue here is that these are in Chinese, but that hasn't really
been an issue for me yet between Google Translate and viewing the source
of the examples.</p>
<p><strong>Accessibility</strong></p>
<p>Ant Design doesn't currently do much with <a class="reference external" href="https://www.w3.org/WAI/intro/aria">ARIA</a> attributes. I am
hopeful that this can be improved via pull requests and discussion
with the core team.</p>
<p>It does have some support additional keyboard navigation but further
investigation will be needed in this area.</p>
<p><strong>Theming</strong></p>
<p>Ant Design is built with theming in mind, but I haven't investigated
how much theming is supported or how easy it is.</p>
<p><strong>Look and Feel</strong></p>
<p>Ant Design has a look of its own. It doesn't try to mimic an
existing platform or vendor. I find it visually appealing.</p>
<p><strong>Internationalization / localization</strong></p>
<p>Ant Design has some support for loading messages from locale files
for Chinese and English text. The locale support within Ant Design
supports dynamically changing the language and updating the rendered UI.
This has been improved within the last couple of weeks.</p>
<p>For further localization concerns, there is <a class="reference external" href="https://github.com/ant-design/intl-example">an example</a> of it being
integrated with <a class="reference external" href="https://github.com/yahoo/react-intl">react-intl</a>.</p>
<p><strong>Actively Maintained</strong></p>
<p>It is being actively maintained by a commercial entity and is under
active development. To feel out how the maintainers treated the project,
I decided to do an initial pull request.</p>
<p>I filed a bug with a question about moving from usage of <tt class="docutils literal">React.createClass</tt>
to ES2015-style classes that extend <tt class="docutils literal">React.Component</tt>. After an initial
confirmation from them that they were interested, I submitted a partial
patch as a pull request for feedback.</p>
<p>All responses have been prompt, even on weekends. They have been positive
and encouraging. Requests to improve my work have been clear and they
have provided an example of what they mean.</p>
<p>I chose my original issue as it would be an excuse to go through a lot
of the code, but also it was a good signal for how receptive they were
to changes that touched many places within the code from an 'outsider'
as well as how much they value consistency and uniformity within their
code.</p>
<p><strong>Licensing</strong></p>
<p>Happily, Ant Design and the underlying components are all MIT licensed.</p>
</div>
<div class="section" id="what-next">
<h2>What Next?</h2>
<p>I'm going to continue trying out <a class="reference external" href="http://ant.design/docs/react/introduce">Ant Design</a> within my prototypes. I'll
also continue to talk with the upstream about making further improvements
and helping out with improving the code. I'd really like to see the
documentation and some other materials available with an English translation
and I'm interested in helping out with that.</p>
<p>I think that, with some effort to make it more accessible to people who
don't speak Chinese, Ant Design and the underlying React components could
be a pretty interesting framework for many more people than it currently
serves.</p>
</div>
Querying OS Information2016-03-07T00:00:00+00:002016-03-07T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2016-03-07:/2016/03/07/querying-os-information/<p>For a project that I'm working on, I would like to be able to retrieve
information from the OS about a variety of things:</p>
<ul class="simple">
<li>File system</li>
<li>Current processes running</li>
<li>Process memory maps</li>
<li>Open file handles in a process</li>
<li>File attributes</li>
<li>File type, mime type</li>
<li>... and a lot more!</li>
</ul>
<p>Some of these are very easy and are just a set of calls to POSIX functions.
Others are highly dependent on the underlying platform.</p>
<p>I want to be able to access this data from an application that is either
running natively or from within node.js when using Electron. I want it on
a minimum of Mac OS X, Linux, FreeBSD and Windows. Support for other
platforms such as NetBSD would be nice.</p>
<p>I don't want this to be very large. The overall code required is likely to
be fairly small. Hopefully, a large number of dependencies would not be
brought in by using a library that provides this information.</p>
<p>I want the data in <a class="reference external" href="http://json-ld.org/">JSON-LD</a> format. This is JSON, but with some additional
fields, like <tt class="docutils literal">@type</tt>, that help my application present the data correctly.
It would be nice if there were a way to subset the data such that some of …</p><p>For a project that I'm working on, I would like to be able to retrieve
information from the OS about a variety of things:</p>
<ul class="simple">
<li>File system</li>
<li>Current processes running</li>
<li>Process memory maps</li>
<li>Open file handles in a process</li>
<li>File attributes</li>
<li>File type, mime type</li>
<li>... and a lot more!</li>
</ul>
<p>Some of these are very easy and are just a set of calls to POSIX functions.
Others are highly dependent on the underlying platform.</p>
<p>I want to be able to access this data from an application that is either
running natively or from within node.js when using Electron. I want it on
a minimum of Mac OS X, Linux, FreeBSD and Windows. Support for other
platforms such as NetBSD would be nice.</p>
<p>I don't want this to be very large. The overall code required is likely to
be fairly small. Hopefully, a large number of dependencies would not be
brought in by using a library that provides this information.</p>
<p>I want the data in <a class="reference external" href="http://json-ld.org/">JSON-LD</a> format. This is JSON, but with some additional
fields, like <tt class="docutils literal">@type</tt>, that help my application present the data correctly.
It would be nice if there were a way to subset the data such that some of it
wouldn't even be computed. This could be done via a query language ala SQL
or <a class="reference external" href="http://graphql.org/">GraphQL</a>.</p>
<div class="section" id="osquery">
<h2>osquery</h2>
<p>Someone suggested that I look into Facebook's <a class="reference external" href="https://osquery.io/">osquery</a> project. On the
surface, it looked very interesting. It supports using SQL to query a
lot of data from the operating system and can encode the results as JSON.</p>
<p>Unfortunately, I don't think that I can use it.</p>
<p>The largest problem is that isn't a library that I can simply use from within
another application, it is intended to run as a separate daemon process
and communicates with client applications over Thrift. This is ideal for the
use cases that <em>osquery</em> is designed for, but less ideal for me.</p>
<p>Also, <em>osquery</em> is quite large and includes a lot of additional libraries
and functionality: Google's flag and log libraries, some Boost libraries,
Apache Thrift and RocksDB.</p>
<p>Another complication is that the platform support in <em>osquery</em> is limited
to Linux, Mac OS X and unofficial support for FreeBSD. There is no support
yet for Windows. The wide range of functionality makes a complete port to
Windows more difficult.</p>
<p>It would be nice if the JSON output from <em>osquery</em> could include JSON-LD
metadata, but that also means deciding and standardizing on that metadata.</p>
</div>
<div class="section" id="what-path-forward">
<h2>What path forward?</h2>
<p>I am not sure if it is worth contacting the <em>osquery</em> upstream about my
concerns. They have a product that they're happy with and are continuing
to develop. My concerns and requirements don't exactly match up against
theirs. And maybe that's fine!</p>
<p>For other reasons, which I'll detail in a different post and as noted
briefly above, it is really tempting to work out a way to query data like
this with GraphQL.</p>
</div>
Building A Rich Command Shell2015-04-27T00:00:00+00:002015-04-27T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2015-04-27:/2015/04/27/building-a-rich-command-shell/<p>Previously, I wrote about some existing examples of <a class="reference external" href="http://waywardmonkeys.org/2014/10/10/rich-command-shells/">Rich Command Shells</a>,
some from the past, some from the present.</p>
<p>Now, I'd like to look at this from the practical side of things. I have some
software that I would like to see have a rich command shell.</p>
<div class="section" id="what-do-i-want">
<h2>What do I Want?</h2>
<p>First up, what should we consider to be characteristics of a "rich command
shell" for the purposes of this post?</p>
<p>For my interests, I want to see:</p>
<ul class="simple">
<li>Works in a regular terminal.</li>
<li>Also works in terminals with more advanced features.</li>
<li>Adapts to the size and capabilities of the terminal, although I recognize
that feature negotiation doesn't work for everything, so some things
will be controlled by settings.</li>
<li>Works when not run in a terminal (like having output piped to a file or
another process).</li>
<li>Can be run in conjunction with a more specialized program that might
present an even richer interface or embed the functionality somehow.</li>
</ul>
<p>The shorter version of this is:</p>
<ul class="simple">
<li>It should work like things do today, in the existing environment.</li>
<li>It should adaptively support richer means of interaction as well without
violating typical assumptions of today.</li>
<li>The means of adaptation should be flexible and allow progressively …</li></ul></div><p>Previously, I wrote about some existing examples of <a class="reference external" href="http://waywardmonkeys.org/2014/10/10/rich-command-shells/">Rich Command Shells</a>,
some from the past, some from the present.</p>
<p>Now, I'd like to look at this from the practical side of things. I have some
software that I would like to see have a rich command shell.</p>
<div class="section" id="what-do-i-want">
<h2>What do I Want?</h2>
<p>First up, what should we consider to be characteristics of a "rich command
shell" for the purposes of this post?</p>
<p>For my interests, I want to see:</p>
<ul class="simple">
<li>Works in a regular terminal.</li>
<li>Also works in terminals with more advanced features.</li>
<li>Adapts to the size and capabilities of the terminal, although I recognize
that feature negotiation doesn't work for everything, so some things
will be controlled by settings.</li>
<li>Works when not run in a terminal (like having output piped to a file or
another process).</li>
<li>Can be run in conjunction with a more specialized program that might
present an even richer interface or embed the functionality somehow.</li>
</ul>
<p>The shorter version of this is:</p>
<ul class="simple">
<li>It should work like things do today, in the existing environment.</li>
<li>It should adaptively support richer means of interaction as well without
violating typical assumptions of today.</li>
<li>The means of adaptation should be flexible and allow progressively
richer output and interaction models.</li>
</ul>
<p>This might be boring to some people. They may want to replace the "everything
is a stream of bytes" model of Unix. They may want to replace all existing
terminals with something that supports full HTML and related technologies
or be able to assume that all terminals support some sort of inline media
display.</p>
<p>On the other hand, this might be reassuring to some people who would
otherwise be afraid that "rich command shell" was going to mean the tools
have a GUI, require a mouse, and are no longer scriptable or whatever.</p>
<p>The world evolves slowly and moves in fits and starts, and I'm fine with
that.</p>
</div>
<div class="section" id="use-cases">
<h2>Use Cases</h2>
<p>What are some specific examples of things that I want to be sure are
possible? What are some details that I'm less concerned about?</p>
<p>I want to be able to:</p>
<ul class="simple">
<li>Auto-detect or set a content type and see a different quality
of output. The changes should be able to handle at least these
levels of output:<ul>
<li>Plain text with no style.</li>
<li>Text with ANSI coloring and standard VT codes.</li>
<li>Text with some form of inline media display, be that inline
images such as PNGs or the support for things like Sixel or
Regis graphics.</li>
</ul>
</li>
<li>Adapt the content to the width of the window. While this doesn't
sound terribly exciting at first, this becomes more complicated
when you're looking at structured output</li>
</ul>
<p>I am less concerned that:</p>
<ul class="simple">
<li>Higher end display outputs may not be accessible from the same
program, but may require compiling a plugin or shared library
form of the program so that it can be loaded into the display
shell that is providing the advanced interface.</li>
<li>Not all functionality is present everywhere.</li>
</ul>
</div>
<div class="section" id="pretty-printing-as-a-foundation">
<h2>Pretty Printing as a Foundation</h2>
<p>While thinking about this, I re-found a comment in the <a class="reference external" href="http://opendylan.org/">Open Dylan</a>
source code which seemed pretty relevant:</p>
<blockquote>
Program notes need to be stored and later displayed in browsers. This
presents us with two problems. At the point of creation we have no
way of knowing the column width that will be used when the note is
displayed. There may even be more than one width if we want to be
smart when a browser window is resized. A second problem arises if
we store a program note in the form of a condition string + arguments
and the arguments are context sensitive. We could just save everything
as a string, but then the logical structure of the message is lost.
An alternative is to store the text in a more structured form. This
is the purpose of the <tt class="docutils literal"><ppml></tt> class and its derivatives. The
interface is based on Oppen's 1980 TOPLAS paper.</blockquote>
<p>To clarify a few things:</p>
<ul class="simple">
<li>Program notes are any output generated by the compiler to be consumed
by the user.</li>
<li>A browser window in the Open Dylan IDE is anything to browse some
data, like a list of errors, not a "web browser".</li>
<li>A condition is like an exception, especially insofar as the term
is used here.</li>
<li>The TOPLAS paper by Oppen is <a class="reference external" href="http://dl.acm.org/citation.cfm?id=357114.357115">Prettyprinting</a> by Derek C. Oppen,
TOPLAS volume 2 number 4, ACM, 1980. I don't know of a freely
available copy of this paper, but it has been used as the basis
for incredibly large number of pretty printers over the years.</li>
</ul>
<p>The basic idea is that you have a pipeline for output that goes from
the program's data structures and messages to a list of (nested) pretty
print control nodes, which is then rendered to the output device. To do
this, you have 2 sets of functions:</p>
<ul class="simple">
<li>Program Data -> Pretty Print Control Node</li>
<li>Pretty Print Control Node -> Output Device</li>
</ul>
<p>So, you take the program data and messages, construct the pretty
print control nodes and then format that to the output:</p>
<div class="highlight"><pre><span></span><span class="k">define</span> <span class="nb">compiler-sideways</span> <span class="nb">method</span> <span class="n">print-object</span>
<span class="p">(</span><span class="n">condition</span> <span class="p">::</span> <span class="nc"><simple-warning></span><span class="p">,</span> <span class="n">stream</span> <span class="p">::</span> <span class="nc"><stream></span><span class="p">)</span> <span class="p">=></span> <span class="p">()</span>
<span class="k">let</span> <span class="n">body</span> <span class="o">=</span> <span class="nb">apply</span><span class="p">(</span><span class="n">format-to-ppml</span><span class="p">,</span> <span class="n">condition</span><span class="p">.</span><span class="nb">condition-format-string</span><span class="p">,</span>
<span class="n">condition</span><span class="p">.</span><span class="nb">condition-format-arguments</span><span class="p">);</span>
<span class="k">let</span> <span class="n">ppml-condition</span> <span class="o">=</span> <span class="n">ppml-block</span><span class="p">(</span><span class="nb">vector</span><span class="p">(</span><span class="n">ppml-string</span><span class="p">(</span><span class="s">"Warning: "</span><span class="p">),</span>
<span class="n">ppml-break</span><span class="p">(</span><span class="k">offset:</span> <span class="mi">2</span><span class="p">,</span> <span class="k">space:</span> <span class="mi">0</span><span class="p">),</span>
<span class="n">body</span><span class="p">),</span>
<span class="k">offset:</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">ppml-print</span><span class="p">(</span><span class="n">ppml-condition</span><span class="p">,</span>
<span class="nb">make</span><span class="p">(</span><span class="nc"><ppml-printer></span><span class="p">,</span> <span class="k">margin:</span> <span class="mi">100</span><span class="p">,</span>
<span class="k">output-function:</span>
<span class="nb">method</span> <span class="p">(</span><span class="n">s</span> <span class="p">::</span> <span class="nc"><string></span><span class="p">)</span> <span class="n">write</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="k">end</span><span class="p">,</span>
<span class="k">newline-function:</span> <span class="nb">method</span> <span class="p">()</span> <span class="n">write</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span> <span class="k">end</span><span class="p">));</span>
<span class="k">end</span> <span class="nb">method</span> <span class="n">print-object</span><span class="p">;</span>
</pre></div>
</div>
<div class="section" id="moving-on-from-pretty-printing">
<h2>Moving On From Pretty Printing</h2>
<p>The concepts and techniques of pretty printing appear to give us a good
foundation to build upon:</p>
<ul class="simple">
<li>We can build a tree of nodes from our output.</li>
<li>We can customize the rendering of these nodes based upon characteristics
of the output device.</li>
</ul>
<p>While the overall technique is a good approach, pretty printing control
nodes are solving the single problem of how to flexibly layout output in
the face of changing output widths. We'll need something more flexible
to solve our problems and build a solid solution.</p>
<p>In my previous post on <a class="reference external" href="http://waywardmonkeys.org/2014/10/10/rich-command-shells/">Rich Command Shells</a>, I mentioned towards the end
some examples from text-based games where a markup language was used
to handle output so that it could be displayed in different ways depending
on the output device.</p>
<p>This is the same shape of solution as the pretty printing: A pipeline
for output that goes from the program's data structures and messages to
a document, which is rendered to the output device.</p>
<p>In this case, we don't necessarily need an actual markup language and
associated parser as we can just construct the documents in memory.</p>
<p>We do, however, need a rich set of node types that let us provide the
structure and control over our document and that provide enough context
to each rendering back-end for it to make a good decision about how
to represent the data.</p>
<p>The <a class="reference external" href="http://docutils.sourceforge.net/docs/ref/doctree.html">Docutils Document Tree</a> provides a fairly solid basis and is a
useful starting point.</p>
<p>We'll look at this approach in a new post soon.</p>
</div>
Compiling to Javascript or not?2014-11-16T00:00:00+00:002014-11-16T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2014-11-16:/2014/11/16/compiling-to-javascript-or-not/<p>JavaScript is commonly seen as a flawed language. It has plenty
of flaws, some of which are more painful than others. One common
way that people deal with these flaws is to use another language
that compiles to JavaScript and to then write their code in that
other language.</p>
<p>At first, this sounds pretty appealing. You get to choose which
of a few languages to use to address your complaints with JavaScript.
Unhappy with the type system? Try out Dart, TypeScript, AtScript, Elm,
or even PureScript. Want a more concise syntax? Perhaps CoffeeScript
is your thing. Do you wish you were using a Lisp? Well, there are lots of
options there as well, starting with ClojureScript or Parenscript.
There are <a class="reference external" href="https://github.com/jashkenas/coffeescript/wiki/list-of-languages-that-compile-to-js">many choices</a>.</p>
<div class="section" id="my-project">
<h2>My Project</h2>
<p>My project involves creating a platform for building tools that is
based on <a class="reference external" href="https://github.com/atom/atom-shell/">Atom Shell</a> and uses a plug-in model to allow users to
extend the core platform.</p>
<p>I also have some requirements:</p>
<ul class="simple">
<li>I need access to things that will be coming in future versions
of JavaScript (and fairly soon). An example of this is support
for working with 64 bit integers (a firm requirement).</li>
<li>Given that we have a single deployment platform (Atom Shell,
with …</li></ul></div><p>JavaScript is commonly seen as a flawed language. It has plenty
of flaws, some of which are more painful than others. One common
way that people deal with these flaws is to use another language
that compiles to JavaScript and to then write their code in that
other language.</p>
<p>At first, this sounds pretty appealing. You get to choose which
of a few languages to use to address your complaints with JavaScript.
Unhappy with the type system? Try out Dart, TypeScript, AtScript, Elm,
or even PureScript. Want a more concise syntax? Perhaps CoffeeScript
is your thing. Do you wish you were using a Lisp? Well, there are lots of
options there as well, starting with ClojureScript or Parenscript.
There are <a class="reference external" href="https://github.com/jashkenas/coffeescript/wiki/list-of-languages-that-compile-to-js">many choices</a>.</p>
<div class="section" id="my-project">
<h2>My Project</h2>
<p>My project involves creating a platform for building tools that is
based on <a class="reference external" href="https://github.com/atom/atom-shell/">Atom Shell</a> and uses a plug-in model to allow users to
extend the core platform.</p>
<p>I also have some requirements:</p>
<ul class="simple">
<li>I need access to things that will be coming in future versions
of JavaScript (and fairly soon). An example of this is support
for working with 64 bit integers (a firm requirement).</li>
<li>Given that we have a single deployment platform (Atom Shell,
with a known version of the Chromium browser and the V8 JavaScript
engine), it would be nice to be able to take advantage of
developments as they occur.</li>
<li>Users will be able to enter code within the application and
have it run. This means that the compiler needs to be able
to be distributed (easily) with the application and invoked
on arbitrary user code.</li>
<li>I don't want to limit people's options too much in how they
produce their plug-ins. If someone wants to develop their
plug-in in a language that compiles to JavaScript, it would
be ideal if they could do so, as long as that language
provides for sufficient interoperability.</li>
</ul>
</div>
<div class="section" id="downsides">
<h2>Downsides?</h2>
<p>I think that the upsides to compiling from another language
to JavaScript are fairly well known and understood.</p>
<p>However, when evaluating whether or not to use an alternative
language to compile to JavaScript or rather to write directly
in JavaScript now, it seems there are some potential downsides.</p>
<div class="section" id="keeping-up-with-javascript">
<h3>Keeping up with JavaScript</h3>
<p>JavaScript is constantly evolving. In fact, I am counting on and
relying upon that evolution, given that I need some features
in the future that aren't even specified yet (like decent support
for 64 bit integers).</p>
<p>Using ES6 features isn't possible yet in many languages that
compile to JavaScript. Some support ES3, some support ES5. Adding
to this difficulty is that the decisions for what to support and
when may not be something that is done in the open or on a short
time frame.</p>
<p>The extent to which this matters in a language depends upon the
extent to which the language differs from JavaScript versus being
some form of sugar for JavaScript. For example, PureScript would
probably care less about a lot of this, while this is more of an
issue for TypeScript.</p>
</div>
<div class="section" id="bi-directional-interoperability">
<h3>Bi-Directional Interoperability?</h3>
<p>It is important to me that I be able to invoke JavaScript libraries
from whatever language I am using. However, it is also important
that I be able to write libraries in this language and be able to
invoke them from JavaScript.</p>
<p>This isn't an issue for CoffeeScript or TypeScript, but it is an
issue for PureScript and others where the interoperability is not
bi-directional.</p>
</div>
<div class="section" id="loss-of-control">
<h3>Loss of Control</h3>
<p>Any time you start to use something that you don't write and
manage yourself, you potentially lose some control. You need to
convince another community that something is desirable, or useful,
or even doable.</p>
<p>When working with JavaScript, you have already ceded a lot of
control over your platform. By using Atom Shell, I've conceded
even more. What happens if I need something from the language
that I'm using that doesn't exist yet? How friendly to contributions
is the language community? How set in stone are they?</p>
</div>
<div class="section" id="toolchain-support">
<h3>Toolchain Support</h3>
<p>Can the compiler and associated tools be shipped with my application?
Does the compiler produce decent feedback on errors of various sorts?</p>
<p>This is clearly much easier when the compiler is written in or compiles
to JavaScript.</p>
</div>
</div>
<div class="section" id="what-am-i-going-to-do">
<h2>What am I going to do?</h2>
<p>When I started writing this post, I had thought that I would decide
to just use JavaScript directly.</p>
<p>However, for now at least, I am going to try to use TypeScript. I
have a lot to learn, but I think TypeScript comes pretty close to
meeting my requirements and the type system and community adoption
are both pretty nice things.</p>
</div>
Looking at Packet Capture and Dissection2014-10-26T00:00:00+00:002014-10-26T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2014-10-26:/2014/10/26/looking-at-packet-dissection/<p>For the project that I am working on (not yet disclosed), I was
thinking that it would be interesting to be able to integrate a
view of what is happening on the network, much like <a class="reference external" href="http://www.wireshark.org/">Wireshark</a>
and other tools can provide. The view would be more targeted
towards what the user was doing, but the overall idea would be
the same: capture network traffic and perform some basic analysis
on it to display it visually.</p>
<p>This led to some interesting research, which I've decided to discuss
here! (For those who feel this is a long post, there's a summary
of sorts at the end.)</p>
<div class="section" id="capturing-traffic">
<h2>Capturing Traffic</h2>
<div class="section" id="node-pcap">
<h3>node_pcap</h3>
<p>Given that I'm using <a class="reference external" href="https://github.com/atom/atom-shell/">Atom Shell</a> to build this application and that
uses <a class="reference external" href="http://nodejs.org/">Node.JS</a>, one of my first thoughts was to take a look at
<a class="reference external" href="https://github.com/mranney/node_pcap">node_pcap</a>. After all, <tt class="docutils literal">node_pcap</tt> was used successfully in
tools like <a class="reference external" href="https://github.com/mnot/htracr">htracr</a>, so perhaps it would work well here.</p>
<p>Unfortunately, I ran into a series of issues with <tt class="docutils literal">node_pcap</tt>. For
one thing, it wants to run the capture in the same process, and since
performing a capture requires putting the network interface into
promiscuous mode, it requires elevated privileges. I am not comfortable
with the idea that my …</p></div></div><p>For the project that I am working on (not yet disclosed), I was
thinking that it would be interesting to be able to integrate a
view of what is happening on the network, much like <a class="reference external" href="http://www.wireshark.org/">Wireshark</a>
and other tools can provide. The view would be more targeted
towards what the user was doing, but the overall idea would be
the same: capture network traffic and perform some basic analysis
on it to display it visually.</p>
<p>This led to some interesting research, which I've decided to discuss
here! (For those who feel this is a long post, there's a summary
of sorts at the end.)</p>
<div class="section" id="capturing-traffic">
<h2>Capturing Traffic</h2>
<div class="section" id="node-pcap">
<h3>node_pcap</h3>
<p>Given that I'm using <a class="reference external" href="https://github.com/atom/atom-shell/">Atom Shell</a> to build this application and that
uses <a class="reference external" href="http://nodejs.org/">Node.JS</a>, one of my first thoughts was to take a look at
<a class="reference external" href="https://github.com/mranney/node_pcap">node_pcap</a>. After all, <tt class="docutils literal">node_pcap</tt> was used successfully in
tools like <a class="reference external" href="https://github.com/mnot/htracr">htracr</a>, so perhaps it would work well here.</p>
<p>Unfortunately, I ran into a series of issues with <tt class="docutils literal">node_pcap</tt>. For
one thing, it wants to run the capture in the same process, and since
performing a capture requires putting the network interface into
promiscuous mode, it requires elevated privileges. I am not comfortable
with the idea that my application would require elevated / Administrator
privileges to run and the security risk seemed high.</p>
<p>Additionally, <tt class="docutils literal">node_pcap</tt> is native code given that it is interfacing
with a native library. Atom Shell is using the unstable development
version of Node.js, 0.11.x rather than the current stable version of
Node.js, 0.10.x (for good reasons). Unfortunately, the APIs used by
native code to write extensions to Node.js have changed substantially
in the development versions of Node.js (also for good reasons), so
compiling <tt class="docutils literal">node_pcap</tt> for use in Atom Shell would be problematic
and require work. While there is a project, <a class="reference external" href="https://github.com/rvagg/nan">nan</a> or "Native Abstractions
for Node.js", which helps simplify this process, it doesn't really
look like <tt class="docutils literal">node_pcap</tt> is heavily maintained, so it isn't clear
how useful this effort would be in the long run, especially given
the security concerns that I already raised.</p>
<p>That said, there is some interesting code in JavaScript in <tt class="docutils literal">node_pcap</tt>
for tracking TCP states and doing some HTTP processing. We'll come
back to that later.</p>
</div>
<div class="section" id="scapy">
<h3>scapy</h3>
<p>Another interesting tool is <a class="reference external" href="http://www.secdev.org/projects/scapy/">scapy</a> which is written in Python. Sadly,
it looks like they forgot how to maintain their own website. In fact,
almost all of the links on this site to anything related to the code
are currently broken. With some digging, you'll find the <a class="reference external" href="http://bb.secdev.org/scapy">development
site for scapy</a>.</p>
<p>Anyway, <tt class="docutils literal">scapy</tt> is interesting and looks pretty powerful. It would
definitely have to run as a separate process, so some of the security
issues are sorted out already. However, it would mean including enough
of a Python distribution for <tt class="docutils literal">scapy</tt> to run.</p>
<p>Finally, <tt class="docutils literal">scapy</tt> is licensed under the GPLv2 license. While this isn't
an issue directly since we'd be running it as a sub-process and not
linked into our codebase, it could be an issue for users down the road
who want to extend part of the system and it complicates licensing which
is otherwise a combination of MIT, BSD and Apache 2 licenses so far
for the most part.</p>
<p>The issue of having to bundle a Python distribution (and having to deal
with that on each platform) is enough of an issue for me to not want
to consider this for now.</p>
</div>
<div class="section" id="wireshark">
<h3>Wireshark</h3>
<p><a class="reference external" href="http://www.wireshark.org/">Wireshark</a> has an impressive ecosystem and a lot of support. But it is
less clear to me how to integrate this cleanly. Issues include how to
package it up, the undocumented nature of working with Wireshark as
a library (<tt class="docutils literal">libwireshark</tt>), the GPLv2 licensing, and so on.</p>
</div>
<div class="section" id="tcpdump">
<h3>tcpdump</h3>
<p>While very simple, <tt class="docutils literal">tcpdump</tt> is an appealing option. It is already
present on many hosts for collecting network traffic. For collecting
traffic, I should be able to just run it via a subprocess, using
sudo, and have it output PCAP-formatted binary output and read
that via a pipe.</p>
<p>Parsing PCAP data using JavaScript looks to be pretty straight forward
and in fact, there's already a tool for doing so that integrates
well with Node's streams: <cite>node-pcap-parser</cite>.</p>
<p>This leaves me with no licensing issues, something that works across
Mac OS X, Linux and FreeBSD, and without the security issues of
running my main process with elevated privileges.</p>
</div>
</div>
<div class="section" id="packet-dissection">
<h2>Packet Dissection</h2>
<p>We're still left with the issue of now needing to dissect the packets
that we're getting via PCAP.</p>
<div class="section" id="existing-options">
<h3>Existing Options</h3>
<p>Again, there are strong options available for inspecting and dissecting
packets like Wireshark and Scapy. And again, many of the same complaints
about them are true. One new complaint can be lodged against Wireshark:
it has a long <a class="reference external" href="http://www.cvedetails.com/vendor/4861/Wireshark.html">history of vulnerabilities</a> due to bugs in parsing and
dissecting packets.</p>
</div>
<div class="section" id="a-new-option">
<h3>A New Option?</h3>
<p>As we noted before, <a class="reference external" href="https://github.com/mranney/node_pcap">node_pcap</a> actually contains some support for
packet inspection. I don't want to use <tt class="docutils literal">node_pcap</tt> for capturing
traffic for the reasons described before, but perhaps the code
in <tt class="docutils literal">node_pcap</tt> can be adapted for use in a new project?</p>
<p>Along similar lines, there is the start of a network analyzer in
the <a class="reference external" href="http://opendylan.org/">Dylan</a> language called <a class="reference external" href="https://github.com/dylan-hackers/network-night-vision">Network Night Vision</a> which contains
<a class="reference external" href="https://github.com/dylan-hackers/network-night-vision/tree/master/protocols">descriptions of many networking protocols</a>. One thing to note about
Network Night Vision is that it uses <a class="reference external" href="http://opendylan.org/documentation/binary-data/">binary-data</a> which is a declarative
way to describe how a packet should be parsed from and assembled to a
buffer. This was described in a paper <a class="reference external" href="http://www.itu.dk/people/hame/secure-networking.pdf">Secure Networking</a> and
<a class="reference external" href="http://www.itu.dk/people/hame/ilc07-final.pdf">A domain-specific language for manipulation of binary data in Dylan</a>
(presented at ILC 2007).</p>
<p>Interestingly, there are similar binary parser libraries available in
the Node.JS ecosystem. To me, the most interesting one was <a class="reference external" href="https://github.com/keichi/binary-parser">binary-parser</a>,
in part due to how it generates code to have good performance. Most of
the available options in the Node.JS ecosystem (like <a class="reference external" href="https://github.com/deoxxa/dissolve">dissolve</a> and
<a class="reference external" href="https://github.com/substack/node-binary">node-binary</a>) are pretty similar in terms of their features and flexibility.</p>
<p>Is there room in the world for a new family of packet dissectors and
a corresponding framework, based on <a class="reference external" href="https://github.com/keichi/binary-parser">binary-parser</a>? Good question!
Are we reinventing the world? Perhaps. But with some clear goals:</p>
<ul class="simple">
<li>Cleanly separating packet dissection from capturing traffic so that
not everything needs to be running with elevated privileges.</li>
<li>Using a declarative and safe approach to dissection to avoid
security vulnerabilities due to parsing / dissecting bugs.</li>
<li>Openly and liberally licensed under an MIT, BSD or Apache 2
license.</li>
</ul>
</div>
</div>
<div class="section" id="digging-in">
<h2>Digging In!</h2>
<p>So, let's dig in and see what happens when we actually experiment
with some of the above!</p>
<p>Starting with just launching <tt class="docutils literal">tcpdump</tt> and getting some packets
seems like a good start:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="nx">sudo</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'sudo'</span><span class="p">);</span>
<span class="nx">tcpdump</span> <span class="o">=</span> <span class="nx">sudo</span><span class="p">([</span><span class="s1">'tcpdump'</span><span class="p">,</span> <span class="s1">'-c'</span><span class="p">,</span> <span class="s1">'3'</span><span class="p">,</span> <span class="s1">'-w'</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">,</span> <span class="s1">'-U'</span><span class="p">,</span> <span class="s1">'tcp port 80'</span><span class="p">]);</span>
<span class="nx">tcpdump</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'close'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'tcpdump complete.'</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">tcpdump</span><span class="p">.</span><span class="nx">stdout</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'data'</span><span class="p">),</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Got some pcap data.'</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">tcpdump</span><span class="p">.</span><span class="nx">stderr</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'data'</span><span class="p">),</span> <span class="kd">function</span><span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">toString</span><span class="p">());</span>
<span class="p">});</span>
</pre></div>
<p>This is pretty straight forward. Now, we're going to wire it up to
<a class="reference external" href="https://github.com/kunklejr/node-pcap-parser">node-pcap-parser</a>:</p>
<div class="highlight"><pre><span></span><span class="kd">var</span> <span class="nx">sudo</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'sudo'</span><span class="p">),</span>
<span class="nx">pcap_parser</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'pcap-parser'</span><span class="p">);</span>
<span class="nx">tcpdump</span> <span class="o">=</span> <span class="nx">sudo</span><span class="p">([</span><span class="s1">'tcpdump'</span><span class="p">,</span> <span class="s1">'-c'</span><span class="p">,</span> <span class="s1">'3'</span><span class="p">,</span> <span class="s1">'-w'</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">,</span> <span class="s1">'-U'</span><span class="p">,</span> <span class="s1">'-i'</span><span class="p">,</span> <span class="s1">'en0'</span><span class="p">,</span> <span class="s1">'tcp port 80'</span><span class="p">]);</span>
<span class="nx">tcpdump</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'close'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'tcpdump complete.'</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">tcpdump</span><span class="p">.</span><span class="nx">stderr</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'data'</span><span class="p">,</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">data</span><span class="p">.</span><span class="nx">toString</span><span class="p">());</span>
<span class="p">});</span>
<span class="nx">parser</span> <span class="o">=</span> <span class="nx">pcap_parser</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">tcpdump</span><span class="p">.</span><span class="nx">stdout</span><span class="p">);</span>
<span class="nx">parser</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'packet'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">packet</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">packet</span><span class="p">.</span><span class="nx">header</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">packet</span><span class="p">.</span><span class="nx">data</span><span class="p">);</span>
<span class="p">});</span>
</pre></div>
<p>The differences here are:</p>
<ul class="simple">
<li>We wire up the <tt class="docutils literal">tcpdump.stdout</tt> to a PCAP parser and display the
result of the <tt class="docutils literal">packet</tt> event rather than a <tt class="docutils literal">data</tt> event.</li>
<li>We specify <tt class="docutils literal"><span class="pre">-i</span> en0</tt> in the <tt class="docutils literal">tcpdump</tt> command line. This is because
<tt class="docutils literal">tcpdump</tt> on Mac OS X dumps in pcap-ng format but the parser being
used here doesn't support that. So, by specifying an interface,
<tt class="docutils literal">tcpdump</tt> drops back to the old format which this parser can
understand.</li>
</ul>
<p>Now, we can move on and just do quick and dirty IPv4 and TCPv4 parsers
using <a class="reference external" href="https://github.com/keichi/binary-parser">binary-parser</a> (it has them as examples), and call them on the
packet data:</p>
<div class="highlight"><pre><span></span><span class="nx">parser</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'packet'</span><span class="p">,</span> <span class="kd">function</span><span class="p">(</span><span class="nx">packet</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">ip</span> <span class="o">=</span> <span class="nx">ipv4Parser</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">packet</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">14</span><span class="p">));</span>
<span class="kd">var</span> <span class="nx">tcp</span> <span class="o">=</span> <span class="nx">tcpv4Parser</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">ip</span><span class="p">.</span><span class="nx">payload</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">ip</span><span class="p">.</span><span class="nx">payload</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="nx">tcp</span><span class="p">.</span><span class="nx">dataOffset</span> <span class="o">*</span> <span class="mi">4</span><span class="p">).</span><span class="nx">toString</span><span class="p">());</span>
<span class="p">});</span>
</pre></div>
<p>Since we're just doing this quick and dirty, we carved off the first
14 bytes as they're the Ethernet framing (2 6 byte MAC addresses and
a 2 byte type code).</p>
<p>Notably, we aren't:</p>
<ul class="simple">
<li>Parsing pcap-ng data like tcpdump emits by default.</li>
<li>Correctly dealing with anything and are just assuming everything
is a TCP packet.</li>
<li>Emitting events so that other things can respond to the traffic.</li>
<li>Using anything like a nice library structure or dissector
framework.</li>
</ul>
<p>But this is a good enough proof of concept. From here, we can address
the above and do something like the TCP and HTTP trackers in
<a class="reference external" href="https://github.com/mranney/node_pcap">node_pcap</a>.</p>
</div>
<div class="section" id="a-brief-summary">
<h2>A Brief Summary</h2>
<p>I wasn't happy with some existing solutions for capturing or
dissecting packets for a variety of reasons:</p>
<ul class="simple">
<li>I didn't want to run the capture in the same process due to the
security issues involved and needing to run the capture process with
special or elevated privileges.</li>
<li>I wanted open and liberal licensing to not complicate the licensing
of my own product.</li>
<li>I wanted to minimize the amount of native code involved and the
complexities of supporting both Node 0.10.x and 0.11.x (required
for usage in <a class="reference external" href="https://github.com/atom/atom-shell/">Atom Shell</a>).</li>
<li>I didn't want to have to bundle / package a distribution of Python
(on some platforms).</li>
<li>I wasn't excited by something with a long <a class="reference external" href="http://www.cvedetails.com/vendor/4861/Wireshark.html">history of vulnerabilities</a>.</li>
<li>I want a declarative approach to performing packet dissection.</li>
<li>I want something that integrates cleanly with JavaScript and the
model of emitting events.</li>
</ul>
<p>And it looks like I'll end up creating a framework that meets my
needs, so stay tuned!</p>
</div>
Atom Shell and ES6 Harmony2014-10-19T00:00:00+00:002014-10-19T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2014-10-19:/2014/10/19/atom-shell-and-es6-harmony/<p>When building an application using the <a class="reference external" href="https://github.com/atom/atom-shell">Atom Shell</a>, you may want to
take advantage of newer Javascript features from <a class="reference external" href="http://en.wikipedia.org/wiki/ECMAScript#ECMAScript_Harmony_.286th_Edition.29">ES6 (Harmony)</a> like
arrow functions, generators, <tt class="docutils literal">for ... of</tt> loops and many other things
which are not enabled in the V8 Javascript engine by default.</p>
<p>Fortunately, enabling this is pretty easy!</p>
<div class="highlight"><pre><span></span><span class="nx">app</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'ready'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">commandLine</span><span class="p">.</span><span class="nx">appendSwitch</span><span class="p">(</span><span class="s1">'js-flags'</span><span class="p">,</span> <span class="s1">'--harmony'</span><span class="p">);</span>
<span class="p">...</span>
<span class="p">}</span>
</pre></div>
<p>When starting up your Atom Shell application, you can append switches
to the command line that will be given to the embedded Chromium browser
engine.</p>
<p>Note that if you have an incorrect flag being passed, the other flags
that you set may be ignored, so keep an eye on this as you update to
newer versions of Atom Shell as the valid flags may have changed.</p>
<p>Unfortunately, it can be confusing figuring out which version of V8
is being used and what ES6 features are available. As of the time
of this writing, Atom Shell 0.18.1 is using a Chromium 38 and so
it falls in between the Chromium 37 and 39 columns on this
<a class="reference external" href="http://kangax.github.io/compat-table/es6/">compatibility chart</a>. (But don't look at the Node Harmony column
for figuring out what Atom Shell supports!)</p>
<p>When building an application using the <a class="reference external" href="https://github.com/atom/atom-shell">Atom Shell</a>, you may want to
take advantage of newer Javascript features from <a class="reference external" href="http://en.wikipedia.org/wiki/ECMAScript#ECMAScript_Harmony_.286th_Edition.29">ES6 (Harmony)</a> like
arrow functions, generators, <tt class="docutils literal">for ... of</tt> loops and many other things
which are not enabled in the V8 Javascript engine by default.</p>
<p>Fortunately, enabling this is pretty easy!</p>
<div class="highlight"><pre><span></span><span class="nx">app</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'ready'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">commandLine</span><span class="p">.</span><span class="nx">appendSwitch</span><span class="p">(</span><span class="s1">'js-flags'</span><span class="p">,</span> <span class="s1">'--harmony'</span><span class="p">);</span>
<span class="p">...</span>
<span class="p">}</span>
</pre></div>
<p>When starting up your Atom Shell application, you can append switches
to the command line that will be given to the embedded Chromium browser
engine.</p>
<p>Note that if you have an incorrect flag being passed, the other flags
that you set may be ignored, so keep an eye on this as you update to
newer versions of Atom Shell as the valid flags may have changed.</p>
<p>Unfortunately, it can be confusing figuring out which version of V8
is being used and what ES6 features are available. As of the time
of this writing, Atom Shell 0.18.1 is using a Chromium 38 and so
it falls in between the Chromium 37 and 39 columns on this
<a class="reference external" href="http://kangax.github.io/compat-table/es6/">compatibility chart</a>. (But don't look at the Node Harmony column
for figuring out what Atom Shell supports!)</p>
Rich Command Shells2014-10-10T00:00:00+00:002014-10-10T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2014-10-10:/2014/10/10/rich-command-shells/<p>Many applications benefit from a mix of a graphical and textual interface.
This has ranged from entire environments, like the Lisp Machine and the
follow-on Common Lisp Interface Manager (CLIM), to specific applications
like Mathematica and others which employ a "notebook interface". There
have also been various attempts at producing richer terminal applications
which can work with either standard or specialized applications to produce
richer, more interactive output. Graphically enhanced textual interfaces
have also found their place in various genres of games over the years.</p>
<p>In our case, we're interested in enhancing the interaction between a
user and a specific tool. We'll take a look here at some of the ideas
that have come before to get an idea of what people have already done
to inform what we do for ourselves.</p>
<p>Some of the things discussed here are old, some are new. Some are long
lost and forgotten (one link below is to archive.org) while others are
actively being worked on today.</p>
<div class="section" id="common-lisp-interface-manager">
<h2>Common Lisp Interface Manager</h2>
<p>Like many things, early research into graphically enhanced and interactive
textual interfaces was carried out within the Lisp world. An early paper
<a class="reference external" href="ftp://publications.ai.mit.edu/ai-publications/pdf/AITR-794.pdf">Presentation Based User Interfaces</a> by Eugene C. Ciccarelli IV at …</p></div><p>Many applications benefit from a mix of a graphical and textual interface.
This has ranged from entire environments, like the Lisp Machine and the
follow-on Common Lisp Interface Manager (CLIM), to specific applications
like Mathematica and others which employ a "notebook interface". There
have also been various attempts at producing richer terminal applications
which can work with either standard or specialized applications to produce
richer, more interactive output. Graphically enhanced textual interfaces
have also found their place in various genres of games over the years.</p>
<p>In our case, we're interested in enhancing the interaction between a
user and a specific tool. We'll take a look here at some of the ideas
that have come before to get an idea of what people have already done
to inform what we do for ourselves.</p>
<p>Some of the things discussed here are old, some are new. Some are long
lost and forgotten (one link below is to archive.org) while others are
actively being worked on today.</p>
<div class="section" id="common-lisp-interface-manager">
<h2>Common Lisp Interface Manager</h2>
<p>Like many things, early research into graphically enhanced and interactive
textual interfaces was carried out within the Lisp world. An early paper
<a class="reference external" href="ftp://publications.ai.mit.edu/ai-publications/pdf/AITR-794.pdf">Presentation Based User Interfaces</a> by Eugene C. Ciccarelli IV at MIT
from 1984 laid the groundwork for something called "presentations". This
was further evolved as part of Dynamic Windows on the Symbolics Lisp
Machine, and then carried over into the <a class="reference external" href="http://en.wikipedia.org/wiki/Common_Lisp_Interface_Manager">Common Lisp Interface Manager</a>.</p>
<p>From the <a class="reference external" href="http://bauhh.dyndns.org:8000/clim-spec/23-1.html">CLIM specification</a>:</p>
<blockquote>
<p>The core around which the CLIM application user interface model is
built is the concept of the application-defined user interface data
type. Each application has its own set of semantically significant
user interface entities; a CAD program for designing circuits has
kits various kinds of components (gates, resistors, and so on), while
a database manager has its relations and field types. These entities
have to be displayed to the user (possibly in more than one displayed
representation) and the user has to be able to interact with and
specify the entities via pointer gestures and keyboard input. Frequently
each user interface entity has a corresponding Lisp data type (such as
an application-specific structure or CLOS class definition), but this
is not always the case. The data representation for an interaction entity
may be a primitive Lisp data type. In fact, it is possible for several
different user interface entities to use the same Lisp data type for
their internal representation, for example, building floor numbers
and employee vacation day totals could both be represented internally
as integers.</p>
<p>CLIM provides a framework for defining the appearance and behavior of
these user interface entities via the presentation type mechanism. A
presentation type can be thought of as a CLOS class that has some
additional functionality pertaining to its roles in the user interface
of an application. By defining a presentation type the application
programmer defines all of the user interface components of the entity:</p>
<ul class="simple">
<li>Its displayed representation, textual or graphical</li>
<li>Textual representation, for user input via the keyboard</li>
<li>Pointer sensitivity, for user input via the pointer</li>
</ul>
<p>In other words, by defining a presentation type, the application
programmer describes in one place all the information about an
object necessary to display it to the user and interact with the
user for object input.</p>
</blockquote>
<p>A shorter, more <a class="reference external" href="http://www.kantz.com/clim-primer/presentation-types.htm">concise description</a>, might be:</p>
<blockquote>
<p>A presentation is an association between</p>
<ul class="simple">
<li>a type of object,</li>
<li>an instance of that type of object,</li>
<li>and that object's visual representation.</li>
</ul>
<p>Or more specifically, when output is done as a presentation, CLIM
creates an output record associated with an object and a presentation
type, and saves that record in the window's output history.</p>
</blockquote>
<p>The capabilities of CLIM can be, perhaps, be illustrated more clearly:</p>
<div class="img-polaroid figure align-center">
<img alt="CLIM listener screenshot" src="/images/command-shell-lisp-listener.png" />
<p class="caption">A screenshot of the <a class="reference external" href="http://en.wikipedia.org/wiki/File:Listener.png">CLIM Listener from Wikipedia</a>.</p>
</div>
<p>You can see in this screenshot that commands such as <tt class="docutils literal">Show Directory</tt>
and <tt class="docutils literal">Show Class Subclasses</tt> can have rich output. The extent of the
interaction possible isn't on display in the screenshot. (It also
shows the presence of inline completion helpers like <tt class="docutils literal">(pathname)</tt>
and <tt class="docutils literal">(class)</tt>, but that's just part of the nice command parser system
present in CLIM.</p>
<p>What did CLIM get right? CLIM provided:</p>
<ul class="simple">
<li>The ability for a presentation to vary itself according
to the capabilities of the output device.</li>
<li>Rich output types, not just text.</li>
<li>Interactive output via the mouse. Output could be used to fill
requirements for subsequent commands. (Like selecting a file from
a previous <tt class="docutils literal">Show Directory</tt> when prompted for a file.)</li>
</ul>
<p>CLIM did a lot of things very well and provided a pretty enjoyable
environment within which to work.</p>
</div>
<div class="section" id="notebook-interfaces">
<h2>Notebook Interfaces</h2>
<p>Notebook interfaces are interactive documents that represent the
history of commands that have been run within a session. They can
usually be saved, replayed, and shared with other people. Notebook
interfaces typically support rich multimedia, including code with
syntax highlighting, mathematical formulas, graphs, charts, and
formatted text.</p>
<p><a class="reference external" href="https://reference.wolfram.com/language/tutorial/UsingANotebookInterface.html">Mathematica</a> was one of the early pioneers in this area. A simple
screenshot doesn't do Mathematica justice. Try to find the
time to watch Mathematica <a class="reference external" href="http://www.youtube.com/watch?v=EjCWdsrVcBM">in action</a> during the Strange Loop 2014
keynote by Stephen Wolfram. It really is something to be seen.</p>
<div class="img-polaroid figure align-center">
<img alt="Mathematic screenshot" src="/images/command-shell-mathematica.gif" />
<p class="caption">Mathematica's Notebook interface</p>
</div>
<p>The <a class="reference external" href="http://ipython.org/">iPython</a> implementation is open source, widely used and now
supports multiple languages, not just Python.</p>
<p>The iPython notebook interface can be used with multiple front-ends:</p>
<div class="img-polaroid figure align-center">
<img alt="iPython in terminal screenshot" src="/images/command-shell-ipython-terminal.png" />
<p class="caption">iPython running in a terminal</p>
</div>
<div class="img-polaroid figure align-center">
<img alt="iPython in web browser screenshot" src="/images/command-shell-ipython-notebook.png" />
<p class="caption">iPython using a web browser to display the notebook</p>
</div>
<p>iPython 2.0 adds support for <a class="reference external" href="http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Interactive%20Widgets/Index.ipynb">interactive widgets</a></p>
</div>
<div class="section" id="terminals">
<h2>Terminals</h2>
<p>Long ago, terminals could do much more exciting things than they do
today. They supported various graphics protocols for rendering bitmap
and vector graphics.</p>
<p>Check out <a class="reference external" href="https://github.com/saitoha/libsixel">libsixel</a> and <a class="reference external" href="https://github.com/saitoha/PySixel">PySixel</a> for some examples of what can be done
with <a class="reference external" href="http://en.wikipedia.org/wiki/Sixel">Sixel</a> graphics. The terminal emulators <a class="reference external" href="https://bitbucket.org/arakiken/mlterm">mlterm</a> and <a class="reference external" href="http://zuse.jp/tanasinn/">Tanasinn</a>
support this (among others). gnuplot and netpbm support Sixel output:</p>
<div class="img-polaroid figure align-center">
<img alt="libsixel screenshot" src="/images/command-shell-sixel-gnuplot.png" />
<p class="caption">Image from <a class="reference external" href="https://github.com/saitoha/libsixel">libsixel</a></p>
</div>
<p>As an aside, it looks like <a class="reference external" href="http://saitoha.github.io/">Saitoha</a> is on a personal mission to spread support
for Sixel graphics and has done a lot of work in this area. That's awesome
dedication!</p>
<p><a class="reference external" href="https://github.com/gnachman/iTerm2">iTerm2</a> supports embedding images as can be <a class="reference external" href="http://www.iterm2.com/images.html">seen here</a>. <a class="reference external" href="https://www.enlightenment.org/p.php?p=about/terminology">Terminology</a>
also supports embedding images and other media.</p>
</div>
<div class="section" id="rich-terminal-applications">
<h2>Rich Terminal Applications</h2>
<p>There have been many interesting attempts to provide a rich terminal
application for working with the Unix shell, especially with the
advent of the web browser. An early example of a browser-backed shell
was <a class="reference external" href="http://www.xml.com/pub/a/2000/06/07/xmlterm/">XMLterm</a>. Since then, there has also been <a class="reference external" href="https://github.com/unconed/TermKit">TermKit</a>, which has
since passed away. The author of <a class="reference external" href="http://www.xml.com/pub/a/2000/06/07/xmlterm/">XMLterm</a> now works on <a class="reference external" href="https://github.com/mitotic/graphterm">GraphTerm</a>.</p>
<p>While these are all interesting in their own ways, they are not terribly
useful for the type of application that we're looking to build. These
tend to assume that you're creating a new ecosystem surrounding replacing
the Unix shell experience.</p>
<div class="img-polaroid figure align-center">
<img alt="GraphTerm screenshot" src="/images/command-shell-graphterm-ssh-plot.png" />
<p class="caption">An example GraphTerm screenshot</p>
</div>
</div>
<div class="section" id="textual-game-interfaces">
<h2>Textual Game Interfaces</h2>
<p>For now, this is mainly worth mentioning as a curiosity. I have no good
links to point to examples of this. Many games in the 1980s and early
1990s had text input interfaces while the game itself would display
graphics.</p>
<p>Some of the richest textual interfaces that I saw however were in some of
the programmable MUDs, like <a class="reference external" href="http://en.wikipedia.org/wiki/LambdaMOO">LambdaMOO</a> and similar systems. In these,
the entire interface was both programmable and text-based. However,
some systems supported early hypertext-capable clients such as <a class="reference external" href="http://pueblo.sourceforge.net/pueblo/">Pueblo</a>,
<a class="reference external" href="http://www.awns.com/tkMOO-light/">TkMOO-light</a> (with a plug-in) and research systems like <a class="reference external" href="http://ftp.lambda.moo.mud.org/pub/MOO/papers/JupiterAV.ps">Jupiter</a>,
the <a class="reference external" href="http://ftp.lambda.moo.mud.org/pub/MOO/papers/JupiterWin.ps">Jupiter Windowing system</a> and <a class="reference external" href="http://web.archive.org/web/*/http://tchat.research.att.net/">TWin</a> (which ran on top of out
an out-of-band client/server communication protocol known as <a class="reference external" href="http://www.moo.mud.org/mcp2/">MCP</a>).</p>
<p>There was a lot of interesting work in this area, much of it largely
lost to the sands of time.</p>
<p>I found an old screenshot of a game client that we did with IE, a
custom control for talking to the game server, and a whole lot of JavaScript
and CSS back in 2002 or 2003:</p>
<div class="img-polaroid figure align-center">
<img alt="Grendel's Revene screenshot" src="/images/command-shell-grendels-revenge.png" />
</div>
<p>Another interesting aspect of textual interfaces in games is that they
often required some formatting of the text itself. In some MOO and the
Cold system that I mentioned above, we had a markup language for text
that let us control how it was output for various terminal types. (In
Cold, we supported plain telnet, ANSI text, HTML, Pueblo, and a couple
of other custom outputs, like the screenshot from Grendel's Revenge
above.) In Cold, we could provide links, various layout options
(definition lists, bullet lists, tables, etc.) and the system would
handle making sure everything looked good, that ANSI codes were
used where appropriate, or that the right HTML tags were rendered.</p>
<p>Nowadays, one might use Markdown, ReStructuredText or other things
to achieve some of the same effects, but not everything is possible
with those.</p>
<p>For a simple example from the Cold help system, this markup:</p>
<pre class="literal-block">
{p}Nodes from this point down are for core subsystem
documentation and documentation on specific objects.
{p}
{dl columned:
{dt: {b:{link node=$help_index_subsystem:Subsystem}}}
{dd: Subsystems Index}
{dt: {b:{link node=$help_index_objects:Object}}}
{dd: Core Objects Index}}
</pre>
<p>Would render in plain text as:</p>
<pre class="literal-block">
Nodes from this point down are for core subsystem documentation and
documentation on specific objects.
[Subsystem] Subsystems Index
[Object] Core Objects Index
</pre>
<p>This sort of thing is very useful in a command shell for things
like help text, formatted paragraphs, proper line wrapping,
optionally displaying colored text, etc.</p>
</div>
<div class="section" id="others">
<h2>Others</h2>
<p>We've just taken a brief look at a few different enriched textual
user interfaces. There are surely many others or many details not covered
above. I encourage you to write about them in your own blog posts!</p>
<p>I'll write soon about some applications that I'm working on and how
we might be able to build upon some of the ideas that have come before
as well as how we can take advantage of some of what exists today.</p>
</div>
Initial Thoughts on Polymer2014-10-02T00:00:00+00:002014-10-02T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2014-10-02:/2014/10/02/initial-thoughts-on-polymer/<p>I've been doing some web development off and on for many years (since 1996).
I started out in an interpreted language on top of an object database that
was serving up a web-based discussion forum that I was writing and shipping.
I moved on from there to various other tools over the years including various
home-grown templating languages, Django templates, Jinja2 and others. I
helped produce web applications using ExtJS, others using jQuery / jQuery UI,
and experimented with various other frameworks over the years. I've followed
along with what people are doing in ReactJS and related tools like Om.
It is all interesting.</p>
<div class="section" id="a-recent-project">
<h2>A Recent Project</h2>
<p>I recently built a web-based UI for a memory / heap profiler for emscripten.
This is just some Jinja2 templates and Flask on the server side. The UI is
using Bootstrap and went through several iterations of various JS tools and
frameworks for handling graphs, grids, and charts. This was something of
a nightmare (and honestly, the code still is terrible). Many tools didn't
play well together. Some had issues due to CSS or JS clashing. Others just
wanted to control more than I wanted to let them control. Getting jQuery UI,
Bootstrap, Slick Grid …</p></div><p>I've been doing some web development off and on for many years (since 1996).
I started out in an interpreted language on top of an object database that
was serving up a web-based discussion forum that I was writing and shipping.
I moved on from there to various other tools over the years including various
home-grown templating languages, Django templates, Jinja2 and others. I
helped produce web applications using ExtJS, others using jQuery / jQuery UI,
and experimented with various other frameworks over the years. I've followed
along with what people are doing in ReactJS and related tools like Om.
It is all interesting.</p>
<div class="section" id="a-recent-project">
<h2>A Recent Project</h2>
<p>I recently built a web-based UI for a memory / heap profiler for emscripten.
This is just some Jinja2 templates and Flask on the server side. The UI is
using Bootstrap and went through several iterations of various JS tools and
frameworks for handling graphs, grids, and charts. This was something of
a nightmare (and honestly, the code still is terrible). Many tools didn't
play well together. Some had issues due to CSS or JS clashing. Others just
wanted to control more than I wanted to let them control. Getting jQuery UI,
Bootstrap, Slick Grid, and other things all playing nicely together in a
single page wasn't entirely fun. And how should I write my new controls /
visualizations / widgets? There wasn't any clearly right answer and given
the lack of an over-arching framework, anything would've involved even more
work, and it wasn't going to be of direct benefit to my client, so I didn't
have the time to do it.</p>
<p>Things are a bit of a mish-mash. It works, but it definitely isn't
pretty on the inside.</p>
</div>
<div class="section" id="an-upcoming-project">
<h2>An Upcoming Project</h2>
<p>For an upcoming project, which I'm just calling ProjectX for now,
I need to build a really solid user interface running inside <a class="reference external" href="https://github.com/atom/atom-shell">Atom
Shell</a> talking to a C++ library via Node.js bindings. This project
needs a strong extensibility mechanism with plugins, including
UI. This is interesting and brings up some questions and possible
requirements.</p>
<ul class="simple">
<li>Should ProjectX mandate that everyone extending it use a particular
framework? Some things require that people use ReactJS or Angular
or Ember or ExtJS or even an entirely different language like
ClojureScript.</li>
<li>What should a plugin export? How should others use it? How
should events and other things be coordinated?</li>
<li>What is a good widget set to use for building the interface?
How complete is it? How many other people are working with
and extending it and perhaps even publishing their own
widgets?</li>
<li>I'm almost certainly going to have a number of custom widgets
that I'll have to make. Should I publish them for others to
use?</li>
<li>What options are available to me because I'm using Atom Shell
and don't have to worry about random browsers out in the field?
What can I reap from being able to assume a relatively current
version of the Chromium browser?</li>
</ul>
</div>
<div class="section" id="web-components">
<h2>Web Components</h2>
<p>Having been looking around for the last couple of years to keep
up with the world of web application development, I already
knew that I wouldn't be too happy with many of the solutions
out there. I didn't want to bet everything on Ember or Angular
or frameworks in ClojureScript.</p>
<p><a class="reference external" href="http://webcomponents.org/">Web Components</a> though ... Now that sounds interesting!</p>
<p>So, I went and watched <a class="reference external" href="http://www.youtube.com/watch?v=8OJ7ih8EE7s">Google I/O 2014 - Polymer and Web Components
change everything you know about Web development</a> and <a class="reference external" href="http://www.youtube.com/watch?v=HKrYfrAzqFA">Google I/O
2014 - Unlock the next era of UI development with Polymer</a> to catch
up with where things were at today.</p>
<p>A bit over a year ago, I had heard about Google's <a class="reference external" href="https://www.polymer-project.org/">Polymer Project</a>
after Google I/O 2013. This sounded really exciting and interesting,
but it wasn't clear how good the browser support was, whether or
not other browsers would be coming along, whether or not it would
see much uptake in the community, or even whether or not it was
practical or ready for production usage. I decided against using
it as the time on a project due to the massive amount of uncertainty
that I felt.</p>
<p>This project though is going to run in Atom Shell. I don't have to
worry about supporting IE or Safari or even FireFox. Admittedly,
it is still early days for Web Components and the Polymer Project,
but the promise is already visible and it is usable today.</p>
<p>How does this match up with some of the questions raised above?</p>
<ul class="simple">
<li>Web Components allow people to use what they like within
a component (within reason). It isn't going to force everyone
to learn Angular or ClojureScript or ReactJS or some other
framework. They will have to learn some new technologies,
but these are technologies designed to assist in inter-operation.</li>
<li>A plugin can be provided as a set of web components. They
can use events with the DOM like anything else. They can
almost feel like native extensions to the browser environment.</li>
<li>Finding a good widget set still remains to be done. There are
a couple of options here, from Polymer's "Core" elements to
Polymer's "Paper" elements to things like Mozilla's "Brick"
elements. Even Microsoft has publicly experimented with providing
web components for WinJS. It seems like more and more components
are being published on a regular basis, so the options available
are only growing.</li>
<li>Can I publish my own web components? Sure! (In fact, I'm
excited enough about being able to do so that I went and
registered componentfoundry.org and a GitHub organization
to do so.)</li>
<li>Atom Shell really helps make this an easier choice. Since
Atom Shell is using Chromium 37.x (currently), it already
has full support for the Web Component technologies, so
I won't need much or any of the polyfill from Polymer. I
will still be using their "sugaring" layer though.</li>
</ul>
<p>Admittedly, there will be many things to sort out and figure
out. Can we find a set of elements that look good together
and that integrate well together? Will the promise of component
interoperability come to fruition here? Will highly specialized
components be easy to find or build, like grids that can
hold half a million rows or that integrate charts that use
D3 to render large datasets?</p>
<p>These questions seem to be open issues for pretty much
every framework though, so hopefully the advantages of
Web Components will work out. Perhaps I'll be able to help
push things along by providing some new components.</p>
<p>For the first time in a long while, I'm excited about the
idea of building an application using web technologies
rather than resigned to having to do so!</p>
</div>
SWIG and JavaScript - Part One2014-10-01T00:00:00+00:002014-10-01T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2014-10-01:/2014/10/01/swig-and-javascript-part-one/<p>I experiment with a lot of ideas for projects and see which ideas stick,
which seem interesting, which require something that I can't provide, etc.</p>
<p>A project that I'm working on now may well be interesting and within
my capabilities and the resources that I can muster, but I don't want to
identify it specifically yet.</p>
<p>Anyway, this project requires using Node.js to talk to a C++ library with
an extensive API. An interesting detail is that this library already has
a solid Python API, built via SWIG.</p>
<div class="section" id="javascript-and-swig">
<h2>JavaScript and SWIG</h2>
<p>Version 3.0 of <a class="reference external" href="http://www.swig.org/">SWIG</a> began to support JavaScript. It supports using
JavaScriptCore or V8 as the JS engine, and Node.js as a specialization
of the V8 support.</p>
<p>For now, I have hacked the SWIG interface files from the C++ library
to wrap Python-specific portions in <tt class="docutils literal">#ifdef SWIG_PYTHON</tt> so that
they can be shared with JavaScript. In the longer term, should this
project work out and I decide to upstream the changes to support
having JavaScript bindings, I will clean this up and move some things
into separate files for each language in a tidier fashion.</p>
<p>In fairly short order, I was able to get a Node …</p></div><p>I experiment with a lot of ideas for projects and see which ideas stick,
which seem interesting, which require something that I can't provide, etc.</p>
<p>A project that I'm working on now may well be interesting and within
my capabilities and the resources that I can muster, but I don't want to
identify it specifically yet.</p>
<p>Anyway, this project requires using Node.js to talk to a C++ library with
an extensive API. An interesting detail is that this library already has
a solid Python API, built via SWIG.</p>
<div class="section" id="javascript-and-swig">
<h2>JavaScript and SWIG</h2>
<p>Version 3.0 of <a class="reference external" href="http://www.swig.org/">SWIG</a> began to support JavaScript. It supports using
JavaScriptCore or V8 as the JS engine, and Node.js as a specialization
of the V8 support.</p>
<p>For now, I have hacked the SWIG interface files from the C++ library
to wrap Python-specific portions in <tt class="docutils literal">#ifdef SWIG_PYTHON</tt> so that
they can be shared with JavaScript. In the longer term, should this
project work out and I decide to upstream the changes to support
having JavaScript bindings, I will clean this up and move some things
into separate files for each language in a tidier fashion.</p>
<p>In fairly short order, I was able to get a Node.js module built
that wrapped the library:</p>
<pre class="literal-block">
swig -c++ -javascript -node -I../../include -I. -o X_wrap.cxx ../X.swig
</pre>
<p>Next up was building this for Mac OS X and correctly linking to the
framework for the C++ project. This involved changing the configuration
for <tt class="docutils literal"><span class="pre">node-gyp</span></tt> in my <tt class="docutils literal">binding.gyp</tt>:</p>
<pre class="literal-block">
"conditions": [
["OS=='mac'", {
"xcode_settings": {
'INSTALL_PATH': '@rpath',
'LD_DYLIB_INSTALL_NAME': '',
"OTHER_CPLUSPLUSFLAGS" : ["-std=c++11", "-stdlib=libc++"],
"MACOSX_DEPLOYMENT_TARGET" : "10.8",
'OTHER_LDFLAGS': [
'-Wl,-rpath,@loader_path/../../deps',
'-F<(module_root_dir)/deps'
]
},
"link_settings" : { "libraries" : ["X.framework"] }
}],
...
</pre>
<p>This shows a couple of things:</p>
<ul class="simple">
<li>I had to set the <tt class="docutils literal">MACOSX_DEPLOYMENT_TARGET</tt> to <tt class="docutils literal">10.8</tt> so that
I would have modern C++ features available as the C++ code relied
upon things like <tt class="docutils literal"><atomic></tt>. I may be able to specify <tt class="docutils literal">10.7</tt>,
but I didn't try.</li>
<li>I enabled C++11 support and using the <tt class="docutils literal">libc++</tt> library. There's
nothing unusual about this when doing work with a C++11 codebase.</li>
<li>I had to set a <tt class="docutils literal">rpath</tt> on the binary being built and linked here
(using <tt class="docutils literal"><span class="pre">-Wl,-rpath</span></tt>) so that it would be able to find the
framework at run-time. The <tt class="docutils literal"><span class="pre">-F</span></tt> is to find the framework at
compile time.</li>
</ul>
</div>
<div class="section" id="mac-os-x-and-run-paths">
<h2>Mac OS X and Run-Paths</h2>
<p>On Mac OS X, you'll usually end up wanting to have a good understanding
of rpaths when building frameworks and executables that use them.</p>
<p>Plenty of things have been written about this already:</p>
<ul class="simple">
<li><a class="reference external" href="https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html">Run-Path Dependent Libraries</a></li>
<li><a class="reference external" href="https://mikeash.com/pyblog/friday-qa-2009-11-06-linking-and-install-names.html">Friday Q&A 2009-11-06: Linking and Install Names</a></li>
<li><a class="reference external" href="http://www.dribin.org/dave/blog/archives/2009/11/15/rpath/">Using @rpath: Why and How</a></li>
</ul>
<p>The important thing to know here is that our library needs to know
how to find the framework, so it needs to have a run-path set, and so
we do that.</p>
<p>I'm hoping that with some thought, I might be able to simplify this
in the future.</p>
</div>
<div class="section" id="node-js-and-v8-versions">
<h2>Node.js and V8 versions</h2>
<p>At this point, I could write a test script that exercised the C++
library and run it via Node.js. Yay!</p>
<p>Unfortunately, my real goal here introduced a new complication.</p>
<p>I don't want to just use this C++ library from a Node.js application.
I actually want to write a GUI for it using <a class="reference external" href="https://github.com/atom/atom-shell">Atom Shell</a>. Atom
Shell is an embedded version of the Chrome (Chromium) browser linked
up with Node.js to let you produce standalone applications rather
than a traditional browser application. Unfortunately, this introduces
some complexities into the process.</p>
<p>Atom Shell uses the unstable, development version of Node.js, 0.11.x
rather than the stable 0.10.x. Partly, this is due to the features
that were added to Node.js to support <a class="reference external" href="http://strongloop.com/strongblog/whats-new-node-js-v0-12-multiple-context-execution/">multiple contexts</a> that Atom
Shell requires. Atom Shell also uses a newer version of the V8
JavaScript engine as that is what Chromium uses. Both the newer
version of Node.js and the updated V8 engine used by Atom Shell are
not API / ABI compatible with Node.js 0.10.x. This now means that
our Swig-wrapped Node.js module as described above doesn't work!</p>
<p>Fortunately, there was a pull request available for SWIG that added
support for newer versions of V8. Unfortunately, V8 doesn't expose
its version information in the header files, so there's no easy
way to figure this out at compile time automatically! What SWIG
has done is to allow you to specify your V8 version number on
the SWIG command line. So, I installed a version of SWIG using that
pull request (which has since been merged), dug up a V8 version
number, and did a new build:</p>
<pre class="literal-block">
/opt/swig/bin/swig -c++ -javascript -node -DV8_VERSION=0x032435 \
-I../../include -I. -o X_wrap.cxx ../X.swig
</pre>
<p>Next up, we had to make some changes to how we run <tt class="docutils literal"><span class="pre">node-gyp</span></tt> to
do our actual build, but those were pretty straight forward and are
described in the <a class="reference external" href="https://github.com/atom/atom-shell/blob/master/docs/tutorial/using-native-node-modules.md#the-node-gyp-way">Atom docs</a>.</p>
<p>Now, we once again had a build of the C++ library, a SWIG-generated
JavaScript / Node.js binding for it, and the ability to load it within
the Atom Shell.</p>
<p>What problems will we run into next? Stay tuned!</p>
</div>
Hello, World!2013-02-07T00:00:00+00:002013-02-07T00:00:00+00:00Bruce Mitchener, Jr.tag:waywardmonkeys.org,2013-02-07:/2013/02/07/hello-world/<p>We're just getting started here and we'll have some more content
coming on to the site in the coming days.</p>
<p>We're just getting started here and we'll have some more content
coming on to the site in the coming days.</p>