tag:blogger.com,1999:blog-56383722017-12-10T21:25:26.895-06:00binkley's BLOGI'm a Principal at ThoughtWorks, classical composer, and physics amateur.Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.comBlogger800125tag:blogger.com,1999:blog-5638372.post-28739701333288247352017-11-01T17:41:00.002-05:002017-11-02T06:45:39.604-05:00Old scripting<h3>The landscape</h3>
<p>I'm helping someone with a scripting problem on an old system and an old
shell. How old? Try IBM AIX 6.1, first released in 2007, and ksh93
"e" released in ... 1993. At least the AIX is a version of 6.1 from 2014!
(Kudos to IBM for treating long-term support seriously.)</p>
<p>A second point to ponder. The goal is to improve remote
scripting&mdash;running scripts on a remote machine. In this
environment, <code>ssh</code> exists but is not used. The remote execution
tool chosen is <code>rexec</code>, <a
href="https://books.google.com/books?id=_-1jwRwNaEoC&lpg=PA198&ots=KxAUu0sVYq&dq=rsh%20rexec%20dangerous&pg=PA198#v=onepage&q=rsh%20rexec%20dangerous&f=false">considered
one of the most
dangerous tools possible</a>. But my remit is not to address the
insecurity, just to improve the scripting. (They know this is a bad,
and are actively working to eventually resolve.)</p>
<p>So, given these constraints, what problem am I solving?</p>
<h3>Example problem</h3>
<p>This environment makes extensive use of remotely executed scripts to
wire together a distributed, locally-hosted system. Current scripts
duplicate the same approach, each implemented as a one-off: Copy a
script to a remote machine with <code>rcp</code>; use
<code>rexec</code> to invoke the script, capturing the output to a
file on the remote host; copy the captured file back to the local host;
process the output file; sometimes clean up the remote host afterwards.
</p>
<p>Some gotchas to watch out for with <code>ksh93e</code> or
<code>rexec</code>:</p>
<ul>
<li>Function tracing - Using the standard <code>xtrace</code> setting
to trace script execution in <code>ksh93</code> has <a
href="https://stackoverflow.com/a/2274727/250924">problems
with tracing functions</a>, and requires using <em>old</em>-style
function syntax
</li>
<li>Variable scope - To keep variables local to a function in
<code>ksh93</code>, you must <a
href="https://stackoverflow.com/a/12004099/250924">use the
<em>new</em>-style function syntax</a> (note the conflict with
tracing)
</li>
<li>Exit broken with trap - When calling <code>exit</code> to quit a
remote script, <code>trap</code> does not get a correct
<code>$?</code> variable (it is always 0, as <code>exit</code>
succeeded in returning a non-0 exit status). Instead one must "set"
<code>$?</code> with the code of a failing command, and then leave
with a plain call to <code>exit</code></li>
<li>No pipefail - Release "e" of ksh93 just does not know anything
about <code>set -o pipefail</code>, and there is no uninstrusive
workaround. This now common feature showed up <a
href="http://www.kornshell.com/doc/faq.html">in release
"g"</a></li>
<li>No exit code - Would you believe <code>rexec</code> does not
itself <a
href="https://www.unix.com/unix-for-dummies-questions-and-answers/20824-exit-codes-rexec.html">exit
with the exit code of the remote command</a>, never has, and
never will? It always exits 0 if the remote command could be started.
</li>
<li>Buffered <code>stderr</code> - Empirically, <code>rexec</code> (at
least the version with this AIX) <em>buffers</em> the <code>stderr
</code> stream of remote commands, and only flushes when <code>
rexec</code> exits, so the sense of ordering between <code>
stdout</code>, <code>stderr</code> and the command-line prompt is
even worse than usual (the actual handling <a
href="http://www.manpagez.com/man/8/rexecd/">is
unspecified</a>)
</li>
</ul>
<p>This problem and environment triggers a memory: The last time I worked
on AIX was in 1994, and it was almost the same problem! I really
thought I had escaped those days.</p>
<h3>A solution</h3>
<p>So I refactored. I couldn't change the use of <code>rexec</code>&mdash;this
environment is not ready for SSH key management&mdash;, I couldn't
replace KSH93 with BASH or replace AIX with Linux, but I could do
something about the imperfect duplication and random detritus files.</p>
<h4>The solution</h4>
<p>Note the need to call a <code>fail</code> function instead of
<code>exit</code> directly because of poor interaction with
<code>trap</code>.</p>
<p>Assuming some help, such as a global <code>progname</code> variable
(which could simply be <code>$0</code>), and avoiding remote
temporary files:</p>
<pre class="code">_transfer_exit_code() {
while read line
do
case $line in
^[0-9] | ^[1-9][0-9] | ^11[0-9] | ^12[0-7] ) return ${line#^} ;;
* ) printf '%s\n' "$line" ;;
esac
done
return 1 # ksh93e lacks pipefail; we get here when 'rscript' failed
}
rscript() {
case $# in
0 | 1 )
echo "$progname: BUG: Usage: rexec SCRIPT-NAME HOSTNAME [ARGS]..." &gt;&amp;2 ;;
* ) script_name=$1 ; shift
hostname=$1 ; shift ;;
esac
# Trace callers script if we ourselves are being traced
case $- in
*x* ) _set_x='set -x' ;;
esac
rexec $hostname /usr/bin/ksh93 -s "$@" &lt;&lt;EOS | _transfer_exit_code
set - "$@" # Only reasonable way to pass through function arguments
# Work around AIX ksh93 return code of exit ignored by trap
fail() {
return \$1
}
# Our hook to capture the exit code for rexec who dumbly swallows it
trap 'rc=\$?; echo ^\$rc; exit \$rc' EXIT
PS4='+$script_name:\$(( LINENO - 14 )) (\$SECONDS) '
$_set_x
# The callers script
$(cat)
EOS
}</pre>
<h4>Example use</h4>
<pre class="">#!/usr/bin/ksh93
progname=${0##*/}
PS4='+$progname:$LINENO ($SECONDS) '
usage() {
echo "Usage: $0 [-d] HOSTNAME"
}
. rexec.ksh
debug=false
while getopts :d opt
do
case $opt in
d ) debug=true ;;
* ) usage &gt;&amp;2 ; exit 2 ;;
esac
done
shift $(( OPTIND - 1 ))
case $# in
1 ) hostname=$1 ;;
* ) usage &gt;&amp;2 ; exit 2 ;;
esac
$debug &amp;&amp; set -x
script_name=My-Remote-Script
tmp=${TMPDIR-/tmp}/$progname.$RANDOM
trap 'rm -f $tmp' EXIT
rscript $script_name $hostname Katy &lt;&lt;'EOS' &gt;$tmp
echo $#: $1
fail 3
EOS
case $? in
3 ) ;;
* ) echo "$0: Did not pass through exit code" &gt;&amp;2 ; exit 1 ;;
esac
case "$(&lt;$tmp)" in
'1: Katy' ) ;;
* ) echo "$0: Did not pass through arguments" &gt;&amp;2 ; exit 1 ;;
esac</pre>
<h4>Source</h4>
<p>The code is <a
href="https://github.com/binkley/shell/tree/master/ksh">in GitHub</a>.
</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-25648830536707406592017-09-19T07:23:00.002-05:002017-09-19T07:34:59.090-05:00Help for JDBC with Java streams<p>We wanted to use JDBC with Java Streams, but encountered several
difficulties. Fortunately we found solutions with rather small bits of
code.</p>
<h3 id="checked-exceptions">Checked exceptions</h3>
<p>The main obstacle was the JDBC API throwing <code>SQLException</code> for
all API methods used in our code. <code>SQLException</code> is a <em>checked
exception</em>, so must be declared in our method signatures, or
caught otherwise. However the Streams API only accepts methods which
declare to throw <em>no</em> checked exceptions, so something simple like
this will not compile:</p>
<pre class="code">stream(results).
map(row -&gt; row.getString("label")). // checked exception
forEach(this::processLabel);</pre>
<p>The call to <code>ResultSet.getString(String)</code> throws a checked
exception. The usual approach is to wrap the call, and handle the
exception in the wrapping method:</p>
<pre class="code">method String streamGetLabel(final ResultSet results) {
try {
return results.getString("label");
} catch (final SQLException e) {
throw new UncheckedIOException(e);
}
}</pre>
<p>(Here <code>UncheckedIOException</code> is an unchecked exception wrapper
we wrote for <code>SQLException</code>, similar to <code>UncheckedIOException</code>
in the JDK for <code>IOException</code>.) </p>
<p>Then the stream becomes:</p>
<pre class="code">stream(results).
map(this::streamGetLabel).
forEach(this::processLabel);</pre>
<p>This is OK, however needing to write a wrapper method for each time we
wanted to use JDBC in a stream became tedious.</p>
<h4 id="checked-exceptions-solution">Solution</h4>
<p>First we wrote a SAM (more on <a
href="https://zeroturnaround.com/rebellabs/java-8-the-first-taste-of-lambdas/"
title="Java 8: The First Taste of Lambdas">SAM interfaces/classes</a>)
interface as a lookalike for the JDK <code>Function</code> interface: this
is what <code>Stream.map(Function)</code> wants.</code>The lookalike is
different in that it throws <code>SQLException</code>: </p>
<pre class="code">@FunctionalInterface
public interface SQLFunction&lt;T, R&gt; {
R apply(final T t) throws SQLException;
// Other default methods - no more abstract methods
}</pre>
<p>Then we used this in a closed <code>Function</code> implementation to wrap
and delegate to the lookalike, and throw
<code>UncheckedSQLException</code> if the lookalike throws <code>SQLException</code>:
</p>
<pre class="code">@RequiredArgsConstructor(staticName = "applyUnchecked")
public final class UncheckedSQLFunction&lt;T, R&gt;
implements Function&lt;T, R&gt; {
private final SQLFunction&lt;T, R&gt; wrapped;
@Override
public R apply(final T t) {
try {
return wrapped.apply(t);
} catch (final SQLException e) {
throw new UncheckedSQLException(e);
}
}
}</pre>
<p>(Here we use the excellent <a
href="https://projectlombok.org/features/constructor"
title="@NoArgsConstructor, @RequiredArgsConstructor, @AllArgsConstructor">Lombok
library</a> to generate our constructor, and give us a static convenience
method, "applyUnchecked".)</p>
<p>Finally some static importing, and our example streams use becomes:</p>
<pre class="code">stream(results).
map(applyUnchecked(row -> row.getString("label"))).
forEach(this::processLabel);</pre>
<p>Or with more help:</p>
<pre class="code">stream(results).
map(getString("label")).
forEach(this::processLabel);</pre>
<p>We wrote similar lookalikes and wrappers for <code>Predicate</code> and
<code>Consumer</code>. It would be easy enough to write them for other
Java functional interfaces, such as <code>BiFunction</code>.</p>
<h3 id="streaming-result-sets">Streaming result sets</h3>
<p>The next difficulty we tackled was how to loop over <code>ResultSet</code>,
and use them with Streams.</p>
<p>(A side note: <code>ResultSet</code> is not a <em>set</em> but a
<em>list</em>: rows are ordered, and they can duplicate each other in
their column data. However, they were named after the SQL concept of sets,
not the Java one.)</p>
<p>Fundamentally, a <code>ResultSet</code> is not an <a
href="https://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html"><em>iterator</em></a>,
but is close:</p>
<table>
<tr>
<th>Iterator</th>
<th>ResultSet</th>
<th>Returns</th>
<th>&nbsp;</th>
</tr>
<tr>
<td><code>hasNext()</code></td>
<td><code>next()</code></td>
<td><code>boolean</code></td>
<td>&nbsp;</td>
</tr>
<tr>
<td><code>next()</code></td>
<td><code>this</code></td>
<td><code>ResultSet</code></td>
<td>(Yes, the <code>ResultSet</code> itself is the equivalent)</td>
</tr>
</table>
<h4 id="streaming-result-sets-solution">Solution</h4>
<p>To provide a <code>ResultSet</code> as an <em>iterator</em>:</p>
<pre class="code">final List&lt;String&gt; values = new ArrayList<>();
for (final ResultSet row : iterable(results)) {
values.add(row.getString("value"));
}</pre>
<p>Moreso, to provide one as a <em>stream</em>:</p>
<pre class="code">final List&lt;String&gt; values = stream(results).
map(getString("value")).
collect(toList());</pre>
<p>Any SQL failures are thrown as <em>unchecked exceptions</em>. The stream
has
<a href="https://docs.oracle.com/javase/8/docs/api/java/util/Spliterator.html#characteristics--">the
characteristics</a>: immutable, nonnull, and ordered.</p>
<h3 id="transactions">Transactions</h3>
<p>We found JDBC transactions to be tricky. Fundamentally they are tied to a
<em>connection</em>; there is no proper nesting. (To simulate nesting, use
separate connections. Even then, there is no guarantee of ordering from
the database engine.) And they have a baroque API, relying on diddling of
the "auto-commit" setting with care needed to restore its setting after
the transaction concludes. Several bugs ensued before we switched to using
small helper interfaces and methods.</p>
<p>Further, some programming languages do not distinguish <code>void</code>
from other return types (e.g., <code>Unit</code> type): Java is not one of
them. Likewise for user vs primitive types (<code>Boolean</code> vs <code>boolean</code>).
Hence, there are separate transaction blocks for consumers, functions, and
predicates.</p>
<h4 id="transactions-solution">Solution</h4>
<p>One example explains them all. Consider functions and the
<code>SQLFunction</code> lookalike interface:</p>
<pre class="code">@RequiredArgsConstructor(staticName = "applyTransacted")
public final class TransactedFunction&lt;T, R&gt;
implements SQLFunction&lt;T, R&gt; {
private final Connection connection;
private final SQLFunction&lt;T, R&gt; wrapped;
@Override
public R apply(final T in)
throws SQLException {
connection.setAutoCommit(false);
try {
final R out = wrapped.apply(in);
connection.commit();
return out;
} catch (final SQLException e) {
connection.rollback();
throw e;
} finally {
connection.setAutoCommit(true);
}
}
}</pre>
<p>(The pattern is the same for other collection operations.)</p>
<p>With a helper and static importing:</p>
<pre class="code">final Integer value = applyTransacted(connection, in -> 0).apply("string");</pre>
<p>Or when the transaction fails:</p>
<pre class="code">applyTransacted(connection, in -> {
throw new SQLException("Something went wrong");
}).apply("string");</pre>
<h3 id="some-convenience">Some convenience</h3>
<p>Many places in these examples are improved with helper functions, or for
transactions, with <a href="https://en.wikipedia.org/wiki/Currying">currying</a>
(similar to <a href="https://en.wikipedia.org/wiki/Builder_pattern">the
builder pattern</a>). Hence, the wide use of Lombok static
constructors. Transactions are another example as they need a <code>Connection</code>
for begin/commit/rollback.</p>
<h4 id="some-convenience-solution">Solution</h4>
<p>A simple helper curries connection for transactions:</p>
<pre class="code">@RequiredArgsConstructor(staticName = "with")
public final class WithConnection {
private final Connection connection;
public &lt;T&gt; Predicate&lt;T&gt; testTransacted(final SQLPredicate&lt;T&gt; wrapped) {
return UncheckedSQLPredicate.testUnchecked(
TransactedPredicate.&lt;T&gt;testTransacted(connection, wrapped));
}
public &lt;T, R&gt; Function&lt;T, R&gt; applyTransacted(
final SQLFunction&lt;T, R&gt; wrapped) {
return UncheckedSQLFunction.applyUnchecked(
TransactedFunction.&lt;T, R&gt;applyTransacted(connection,
wrapped));
}
public &lt;T&gt; Consumer&lt;T&gt; acceptTransacted(final SQLConsumer&lt;T&gt; wrapped) {
return UncheckedSQLConsumer.acceptUnchecked(
TransactedConsumer.&lt;T&gt;acceptTransacted(connection, wrapped));
}
}</pre>
<p>Example use:</p>
<pre class="code">final Optional&lt;Integer&gt; value = Stream.of(0).
filter(with(connection).testTransacted(in -&gt; true)).
findFirst();</pre>
<p>(Yes, we might also describe the code as <a
href="https://en.wikipedia.org/wiki/Partial_application"><em>partial
application</em></a>. The "object-oriented" implementation confuses
matters with the hidden <code>this</code> reference.)</p>
<h3>Conclusion</h3>
<p>There is nothing we did that was difficult or complex: simple one-liner
interfaces, simple wrapper implementations of Java funcional interfaces,
some rote JDBC best practices. The main difficulty was conceptual: seeing
the duplication of many, small wrapper methods, and pulling out their
commonality. This is a good pattern to keep in mind throughout your
code.</p>
<h3><strong>UPDATE:</strong></h3>
<p>And the source: <a href="https://github.com/binkley/java-sql-stream">Java
helpers for JDBC and Streams</a>.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-14151874964276610352017-08-11T12:45:00.002-05:002017-08-12T07:29:25.437-05:00How to write clean Java<p>I cannot tell you how to write <em>good</em> Java, but I can help you write
<strong>clean</strong> Java. As usual, automation is key. It's all about
the tooling.</p>
<p>The best thing about good tooling is that they work together: each covers a
different area, does not impeded another tool, and fixing one complaint a
tool reveals often fixes complaints from other tools.</p>
<p>This advice applies to any programming language, not just Java. Java,
having the most mature ecosystem, is instructive.</p>
<h3 id="the-tools">The tools</h3>
<dl>
<dt><em>Use good source control</em></dt>
<dd><a href="https://git-scm.com/" title="Git">Git</a> is your best
choice. Use either <a href="https://trunkbaseddevelopment.com/"
title="Trunk Based Development">trunk-based
development</a> with <a
href="https://trunkbaseddevelopment.com/feature-flags/"
title="Feature flags - Trunk Based Development">feature
toggles (feature flags)</a>, or <a
href="https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow">gitflow</a>
patterns, depending on your needs and organization. Require full
builds and testing before pushing to a shared repository; <a
href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks"
title="Git - Git Hooks">hooks</a> can help together with your
build tool.
</dd>
<dt><em>Use good build automation</em></dt>
<dd><a href="https://maven.apache.org/" title="Welcome to Apache Maven">Maven</a>
or <a href="https://gradle.org/" title="Gradle Build Tool">Gradle</a>
are your best choices; either is excellent. Teach the tool to be
strict and fail builds if anything is not just right. Treat your build
configuration as code, part of the same repository and held to the
same hygiene standards. Keep your local build fast.
</dd>
<dt><em>Use a good editor</em></dt>
<dd><a href="https://www.jetbrains.com/idea/"
title="IntelliJ IDEA: The Java IDE for Professional Developers by JetBrains">IntelliJ</a>
is the best choice. Use <a
href="https://www.jetbrains.com/help/idea/code-inspection.html"
title="Code Inspection">inspections</a>, <a
href="https://www.jetbrains.com/help/idea/intention-actions.html"
title="Intention Actions">intentions</a>, and <a
href="https://www.jetbrains.com/help/idea/reformatting-source-code.html"
title="Reformatting Source Code">reformatting</a> obsessively:
code diffs should only show real changes, not whitespace or
formatting. Use <a href="https://plugins.jetbrains.com/"
title="JetBrains Plugin Repository">plugins</a> to
ease integrating IntelliJ with other tooling.
</dd>
<dt><em>Keep your style consistent</em></dt>
<dd><a href="http://checkstyle.sourceforge.net/" title="Checkstyle">Checkstyle</a>
is your best choice; other choices are needed for non-Java JVM
languages. Fail your build if checkstyle complains. Keep your IntelliJ
formatting and Checkstyle rules consistent. Generally, choose either
<a href="http://checkstyle.sourceforge.net/sun_style.html"
title="checkstyle &#x2013; Sun's Java Style">Sun</a> or <a
href="http://checkstyle.sourceforge.net/google_style.html"
title="checkstyle &#x2013; Google's Style">Google</a> coding
standards, and be leary of deviating from them; if unsure, pick Sun.
</dd>
<dt><em>Fully test your code</em></dt>
<dd><a href="http://www.eclemma.org/jacoco/"
title="EclEmma - JaCoCo Java Code Coverage Library">JaCoCo</a>
works well. Fail your build if coverage drops below a threshhold;
rachet up the threshhold as coverage improves. Start low and aim for a
95%+ threshhold. Follow <a
href="https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html"
title="Google Testing Blog: Just Say No to More End-to-End Tests">the
<em>Test Pyramid</em></a>; save your end-to-end tests for CI, or
run locally only once before pushing commits.
</dd>
<dt><em>Be zealous in looking for problems</em></dt>
<dd><a href="http://findbugs.sourceforge.net/"
title="FindBugs&trade; - Find Bugs in Java Programs">FindBugs</a>,
<a href="https://sourceforge.net/projects/pmd/" title="PMD">PMD</a>,
or <a href="http://errorprone.info/" title="Error Prone">Error
Prone</a> are your best choices (pick one); other choices may work
better for non-Java JVM languages. Fail your build if your choice
complains. Be judicious in disabling complaints (for example, FindBugs
"experimental" checks, should likely be disabled).
</dd>
<dt><em>Use code generation</em></dt>
<dd><a href="https://projectlombok.org/" title="Project Lombok">Lombok</a>
is your first choice. <a
href="https://medium.com/@iammert/annotation-processing-dont-repeat-yourself-generate-your-code-8425e60c6657"
title="Annotation Processing : Don’t Repeat Yourself, Generate Your Code.">Add
others as needed</a> (or build domain-specific code generators).
Generated code does not need test coverage, style checks, or bug
detection if the generator is clean and well-tested: trust it.
</dd>
</dl>
<h3>Update</h3>
<p><a href="https://plus.google.com/+DanWallach">Dan Wallach</a> reminded me
of Error Prone, added above.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-82959700327579119072017-08-11T08:20:00.003-05:002017-08-11T09:07:05.073-05:00Cygwin terminal in IntelliJ<p>IntelliJ sports an excellent terminal emulator (the "Terminal" tab at
bottom of the editor). By default it brings up a terminal native to your
Operating System: <code>CMD.EXE</code> on Windows, <code>$SHELL</code> on
Linux and Mac.</p>
<p>However I prefer Cygwin when I work on Windows. WSL is incredible, but
there are still interoperability issues between its filesystem and
Windows-native programs, and IntelliJ (which relies on
<code>java.exe</code>, a Windows-native program) <a
href="https://youtrack.jetbrains.com/issue/IDEA-171457"
title="Support for WSL project ">is still working on it</a>.</p>
<p>So, how to open a Cygwin terminal in IntelliJ? Setting the program to start
in <code>Settings|Tools|Terminal|Shell path</code>, the most obvious thing
to do, does not quite work:</p>
<pre class="code">C:\cygwin64\bin\bash.exe</pre>
<p>This is a non-interactive shell, and does not source your profile. The next
try is:</p>
<pre class="code">C:\cygwin64\bin\bash.exe --login -i</pre>
<p>This produces an error from IntelliJ that it cannot start the program
correctly. A little checking says the leading command needs to be quoted,
else IntelliJ treats the entire line as the name of the command, not as a
command followed by flags. OK:</p>
<pre class="code">"C:\cygwin64\bin\bash.exe" --login -i</pre>
<p>Hey, I have a shell! Unfortunately, it starts in my home directory, not in
my project root. Starting in the project root is one of the nice features
of the terminal in IntelliJ. Finally, <a
title="Configuring IntelliJ's terminal to run Cygwin"
href="https://gist.github.com/mpicker0/a6a3f10e6b9278074f93">two
changes</a>. First the IntelliJ setting:</p>
<pre class="code">"C:\cygwin64\bin\bash" -c "exec /usr/bin/env INTELLIJ=true $SHELL --login -i"</pre>
<p>And an addition to my <code>~/.bashrc</code>:</p>
<pre class="code">${INTELLIJ-false} && cd ${OLDPWD-.}</pre>
<p>Ipso presto!</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-1984558311695645782017-06-12T11:53:00.001-05:002017-06-13T07:53:03.037-05:00Example JVM agent in Kotlin<p>Oleg Shelajev wrote an <a
href="https://zeroturnaround.com/rebellabs/how-to-inspect-classes-in-your-jvm/"
title="How-to guide to writing a javaagent">excellent tutorial
post</a> on writing JVM agents. These are bits of code which run <em>before</em>
your <code>main()</code> method. Why do this? It permits <a
href="https://docs.oracle.com/javase/8/docs/api/java/lang/instrument/Instrumentation.html">some
interesting tricks</a>, chiefly modifying classes as they are loaded,
but also estimating the actual memory used by Java objects.</p>
<p>I gave this a try myself, but rather than writing my agent in Java, I wrote
it in Kotlin. It was straight-forward, with only one gotcha.</p>
<h3>AgentX.kt</h3>
<pre class="code">@file:JvmName("AgentX")
package hm.binkley.labs.skratch.jvmagent
import java.lang.instrument.Instrumentation
fun premain(arguments: String?, instrumentation: Instrumentation) {
println("Hello from AgentX 'premain'!")
}
fun main(args: Array&lt;String&gt;) {
println("Hello from AgentX 'main'!")
}</pre>
<p>OK, the non-gotcha. You can declare functions at the <em>package</em>
level. This acts just like static methods in Java, with simpler syntax (no
potentially artificial wrapper class to hold the static method). The two
obvious examples in the above code are <code>main()</code> and <code>premain()</code>.
</p>
<p>But <a
href="https://kotlinlang.org/docs/reference/java-to-kotlin-interop.html#package-level-functions"
title="Calling Kotlin from Java">when calling Kotlin from Java</a>,
you use a wrapping class name in the the fully-qualified method name. My
Kotlin file is named "AgentX.kt", so the default class name for Java is
"AgentXKt". I'm lazy, wanted to save some typing, so I used a Kotlin
package-level annotation to name the wrapping class just "AgentX".</p>
<h3>Output</h3>
<p>The JVM requires an absolute path to any agent jar, and I'm running Cygwin,
so a little help to get a full path. Similarly, I used <a
href="https://maven.apache.org/plugins/maven-shade-plugin/"
title="Apache Maven Shade Plugin">the Maven shade plugin</a> to
build a single uber-jar holding my own classes, and those of my
dependencies (the Kotlin standard library).</p>
<pre class="code" style="white-space: pre-wrap">$ java -javaagent:$(cygpath -m $PWD/target/skratch-0-SNAPSHOT.jar) -jar target/skratch-0-SNAPSHOT.jar
Hello from AgentX 'premain'!
Hello from AgentX 'main'!</pre>
<p>Project is here: <a href="https://github.com/binkley/skratch">https://github.com/binkley/skratch</a>.
</p>
<h3>Gotcha</h3>
<p>Enough preamble, now the gotcha. Unlike Java, Kotlin helps you <a
href="https://kotlinlang.org/docs/reference/null-safety.html"
title="Null Safety">protect yourself from <code>null</code>s</a>
without boilerplate code. So in <code>premain()</code> for the "arguments"
parameter, you need to use <strong><code>String?</code></strong> rather
than <code>String</code> as the parameter type as the JVM may pass you a
<code>null</code>. The first time I tried the code, I didn't realize this
and it blew up:</p>
<pre class="code" style="white-space: pre-wrap">$ java -javaagent:$(cygpath -m $PWD/target/skratch-0-SNAPSHOT.jar) -cp target/skratch-0-SNAPSHOT.jar hm.binkley.labs.skratch.jvmagent.AgentX
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.IllegalArgumentException: Parameter specified as non-null is null: method hm.binkley.labs.skratch.jvmagent.AgentX.premain, parameter arguments
at hm.binkley.labs.skratch.jvmagent.AgentX.premain(AgentX.kt)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed</pre>
<p>Interesting! Kotlin found the issue at runtime. It can't find it at compile
time as the JVM API for "premain" is pure convention without an interface
or class to inspect.</p>
<p>Let's try running the agent a different way. The command-line lets us pass
options, and these become the "arguments" parameter:</p>
<pre class="code" style="white-space: pre-wrap">$ java -javaagent:$(cygpath -m $PWD/target/skratch-0-SNAPSHOT.jar)= -cp target/skratch-0-SNAPSHOT.jar hm.binkley.labs.skratch.jvmagent.AgentX
Hello from AgentX 'premain'!
Hello from AgentX 'main'!</pre>
<p>Sneaky. The mere presence of the "=" on the command line turns the
"arguments" parameter from <code>null</code> to an empty string.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-28511278763681532992017-05-17T10:59:00.003-05:002017-05-17T10:59:54.383-05:00SVG takeaways<p>I started playing with SVG for another blog post I'm working on. I wanted
to learn more about SVG, an XML application. Some of my takeaways:</p>
<h3>Z ordering</h3>
<p>Elements are drawn in the order they appear in the XML. So if you want
element <var>B</var> to overlay element <var>A</var>, then write
<var>A</var> earlier in XML, and write <var>B</var> later. Then
<var>B</var> will be drawn after <var>A</var>, and lay on top of it (if
they overlap), hiding bits of <var>A</var> if <var>B</var> isn't
transparent.</p>
<h3>Scaling</h3>
<p>You define your own coordinate system, and X/Y coordinates are all relative
to this drawing area, called the "view box". So if you declare the view
box to have an <var>X</var> ranging 0-20 "units" and a <var>Y</var>
ranging 0-100 "units", then your elements will use an <var>X</var> between
0 and 20 and a <var>Y</var> between 0 and 100.</p>
<p>Recall the "S" in "SVG" stands for <em>scalable</em>. So when I say
"units", these are relative units in context of the view box, and so in
the example, "50%" of the X coordinate is 10 "units". You don't write down
"units" in the XML, just provide the <var>X</var> and <var>Y</var> ranges.
</p>
<p>What size is this on screen? Your browser or device or PDF will render the
SVG view box to fit, and scale your elements accordingly. This is great
for diagrams and drawings. If you do nothing special to stretch or squeeze
the view box, your elements will retain perspective and always fit the
page.</p>
<p>Embedding SVG in HTML5, I found this produced nice standalone diagrams
within the vertical flow of text (using the X/Y from the example). The
"style" bit is important:</p>
<pre class="code">
&lt;svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="0 0 100 20"
width="80%" style="margin: auto; display: block"&gt;
&lt;!-- Write elements here --&gt;
&lt;/svg&gt;
</pre>
<h3>Styling</h3>
<p>Element attributes are consistent. So when I figured out that <code>stroke="black"</code>
drew black lines, I could use the same attribute on any element to assign
color. This was nice for learning the SVG language piecewise, picking up
bits as I went along.</p>
<p>Fractional values are OK in several contexts. So when I wanted skinnier
lines, I could write <code>stroke-width="0.5"</code>.</p>
<h3>A full example</h3>
<p>Here's the SVG for that post I mentioned at start. I provide it without
explanation; take your best guess what it's for!</p>
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" viewBox="0 0 100 20"
width="80%" style="margin: auto; display: block">
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="1" refY="5" orient="auto"
stroke="green" fill="green">
<path d="M 0 0 L 10 5 L 0 10 z"></path>
</marker>
</defs>
<line x1="30" y1="0" x2="30" y2="30" stroke="lightgrey"
stroke-dasharray="1,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="30" y1="0" x2="30" y2="30" transform="rotate(7 30 10)"
stroke="grey" stroke-dasharray="1,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="50" y1="0" x2="50" y2="30" stroke="lightgrey"
stroke-dasharray="1,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="50" y1="0" x2="50" y2="30" transform="rotate(7 50 10)"
stroke="grey" stroke-dasharray="1,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="70" y1="0" x2="70" y2="30" stroke="lightgrey"
stroke-dasharray="1,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="70" y1="0" x2="70" y2="30" transform="rotate(7 70 10)"
stroke="grey" stroke-dasharray="1,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="10" y1="10" x2="90" y2="10" stroke="lightblue"
stroke-dasharray="2,1" stroke-linecap="round"
stroke-width="0.5"></line>
<line x1="10" y1="10" x2="90" y2="10" transform="rotate(-7 50 10)"
stroke="blue"></line>
<!-- TODO: How to calculate x/y for text? -->
<text x="13" y="9" font-size="7" font-style="italic">Work</text>
<line x1="15" y1="3" x2="80" y2="3" stroke="green"
marker-end="url(#arrow)"></line>
<text x="74" y="16" font-size="7" font-style="italic">Env</text>
<line x1="85" y1="17" x2="20" y2="17" stroke="green"
marker-end="url(#arrow)"></line>
<rect x="10" y="0" width="80" height="20" stroke="navy"
fill="transparent"></rect>
<text x="10" y="9" transform="rotate(-90 10 10)" font-size="10"
style="text-anchor: middle">Dev
</text>
<text x="90" y="9" transform="rotate(90 90 10)" font-size="10"
style="text-anchor: middle">Prod
</text>
</svg>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-77469253939851113032017-04-29T13:36:00.002-05:002017-06-23T05:24:16.409-05:00Maven color logging on Cygwin<p><strong>WORKAROUND:</strong> Thanks to comment by <a
href="https://plus.google.com/u/0/111012722058314768782">Zart
Colwing</a>, there is a workaround. Add
<code>-Djansi.passthrough=true</code> to <code>MAVEN_OPTS</code>. It is
not enough to add the flag to the command line; it needs to be seen by
JANSI before maven begins parsing the command line. See <a
href="https://issues.apache.org/jira/browse/MNG-6242">No color for
maven on Cygwin</a> to track the issue.</p>
<h3>Post</h3>
<p>I'm quite happy Maven 3.5.0 has added color logging!</p>
<p>With earlier versions of maven, I used Jean-Christophe Guy's <a
href="https://github.com/jcgay/maven-color">excellent maven-color
extension</a>. On Windows that involved some manual hacking of my maven
installation, but on Mac, it was trivial with homebrew.</p>
<p>So now I'm getting color output from maven out of the box. Except when I
don't.</p>
<p>You see, this new feature relies on <a
href="https://github.com/fusesource/jansi">the good JAnsi library</a>.
And JAnsi presently has an issue with color on Cygwin. When I'm at home,
Cygwin is my mainstay, used on my gaming-cum-programming rig, so this is
significant to me. What is the issue? No color&mdash;JAnsi detects I'm on
Windows, and uses the native Windows console color handling, which doesn't
work in Mintty or Xterm. Those use standard ANSI escape sequences, etc.
rather than an OS-specific library.</p>
<p>Digging through the source for JAnsi, I find <a
href="https://github.com/fusesource/jansi/blob/c6830ac1e54b1f8fd3985c5fe595a835ad56046a/jansi/src/main/java/org/fusesource/jansi/AnsiConsole.java#L46">the
trouble spot</a>:</p>
<pre class="code">private static final boolean IS_WINDOWS = System.getProperty("os.name").toLowerCase(Locale.ENGLISH).contains("win");</pre>
<p>Aha! The special Windows-handling code kicks in when the
<code>os.name</code> system property contains "win" (in any case). Using
the <code>jps</code> and <code>jinfo</code> tools that come with the JDK,
I double-checked against a long-running maven build (16848 just was the
PID used by the JVM for this maven build; use <code>jps</code> to list all
current PIDs):</p>
<pre class="code">$ jinfo -sysprops 16848 | grep os.name
os.name = Windows 10</pre>
<p>Some experimenting found a way to work around that:</p>
<pre class="code">MAVEN_OPTS=-Dos.name=Cygwin mvn clean</pre>
<p>(You need to use <code>MAVEN_OPTS</code> rather than passing <code>-Dos.name=Cygwin</code>
to maven; once maven starts the value is immutable.)</p>
<p>Color is back. It turns out, any value for <code>os.name</code> will work
(for example, "Bob") as long as it doesn't contain "win". I picked
"Cygwin" for explicitness.</p>
<p><strong>UPDATE</strong>: I have to rethink this. Sure I get color now, but
at the expense of <code>maven test</code> failing as maven no longer
believes I'm on Windows, so is unhappy at the lack of a
<code>/bin/sh</code> and general UNIX filesystem. One step forward, two
steps back.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-59664535108294228782017-04-21T13:50:00.000-05:002017-04-21T13:50:15.810-05:00Quick Kotlin idiom: naming things<p>Kotlin is very much a better Java. A great example is this simple idiom for
naming things:</p>
<pre class="code">open class Named&lt;in T&gt;(val name: String, check: (T) -&gt; Boolean)
: (T) -&gt; Boolean by check {
override fun toString() = name
}</pre>
<p>So I can write something like this:</p>
<pre class="code">fun maybe_three(check: (Int) -&gt; Boolean) {
if (check(3)) do_the_three_thing()
else println("\"$check says\", Do not pass go.")
}
maybe_three(Named("Is it three?") { i -&gt; 3 == i })
maybe_three(Named("Is it four?") { i -&gt; 4 == i })</pre>
<p>The first call of <code>maybe_three</code> prints: <samp>Breakfast, lunch,
dinner</samp>. The second call prints: <samp>"Is it four?" says, Do not
pass go.</samp></p>
<p>Many variations on this are possible, not just functions! What makes this
example work nicely is <a
href="https://kotlinlang.org/docs/reference/delegation.html">delegation</a>&mdash;the
magical <code>by</code> keyword&mdash; for the general feature of naming
things by overriding <code>toString()</code>; and for the function
delegated to, the elegant <a
href="https://kotlinlang.org/docs/reference/lambdas.html">lambda</a>
(anonymous function) syntax for the last parameter. You can delegate
anything, not just functions, so you could make named maps, named business
objects, <i>et al</i>, by using delegation on existing types without
needing to change them.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-3037157744588540842017-04-13T17:12:00.001-05:002017-04-15T08:50:00.174-05:00WSL, first experiences<p>I first tried Windows Subsystem for Linux in December 2016, but was not
successful in installing, so I held off.</p>
<p>After getting the Windows 10 Creators Edition update, I saw <a
href="https://blogs.msdn.microsoft.com/commandline/2017/04/11/windows-10-creators-update-whats-new-in-bashwsl-windows-console/"
title="Windows 10 Creators Update: What’s new in Bash/WSL & Windows Console">how
much work and love</a> went into improving WSL, and decided to try again.
I was rewarded.</p>
<p>On the whole, the installation was smooth, brief, you might even say <a
href="http://catb.org/jargon/html/T/trivial.html">trivial</a>. There
were Windows reboots to enable Developer Mode, and after installing WSL,
but much solid effort has gone into making Windows reboots quick and
painless, and with a regular Linux distro I'd have rebooted anyhow after
upgrading, so no disgruntlement.</p>
<p>And what did I get for my efforts? WSL bash is bash. Just bash. Really, it
is just plain old bash, with all the command line tools I've grown
accustomed to over 30 years. The best praise I can give a tool: <em>It
just works</em>. And WSL just works. (But see <a href="#almost-there"><cite>Almost
there</cite></a>, below.)</p>
<p>Out of the box WSL runs Ubuntu 16.04 (Xenial), the official LTS
distribution (long-term support). This is a sane choice for Microsoft.
It's stable, reliable, secure, tested, trusted. For anyone wanting a
working Linux command line, this is a go-to choice. Still, I updated
it.</p>
<h3 id="things-i-changed">Things I changed</h3>
<p>Even with all the goodness, there were some things I had to change:</p>
<dl>
<dt>The terminal</dt>
<dd>I immediately installed <a href="https://github.com/mintty/wsltty"
title="Mintty as a terminal for Bash on Ubuntu on Windows / WSL">Mintty
for WSL</a>. I've grown to love Mintty on Cygwin, trusting it as a
reliable and featureful terminal emulator without going overboard.
It's a tasteful balance, well executed. And <code>CMD.EXE</code>,
though much improved, still is not there (but may head there; we'll
see if PowerShell wins out).
</dd>
<dt>DBus</dt>
<dd>Not to get into flamewars, I just accept that Ubuntu uses DBus. By
default it doesn't run on WSL, but this was easy to fix, and it made
upgrading Ubuntu smoother. Using <code>sudo</code>, edit <code>/etc/dbus-1/session.conf</code>
as
<a href="https://www.reddit.com/r/Windows10/comments/4rsmzp/bash_on_windows_getting_dbus_and_x_server_working/">others
have suggested</a> (I did it by hand, not with <code>sed</code>).
You may have to repeat after upgrading Ubuntu.
</dd>
<dt>The Ubuntu version</dt>
<dd>It seems trivial, but I was unhappy that <code>diff --color</code>
didn't work. Am I shallow&mdash;color? Some of the scripts <a
href="https://github.com/binkley/shell"
title="Helpful shell scripts and programs">I write</a> for
open source provide colorized diff output, and I'd like to work on
them in WSL without disabling this feature. Microsoft made <a
href="https://blogs.msdn.microsoft.com/commandline/2016/09/22/24-bit-color-in-the-windows-console/"
title="24-bit Color in the Windows Console!">much hay</a> over
24-bit color support in <code>CMD.EXE</code>. So <a
href="#upgrading-wsl-ubuntu" title="Upgrading WSL Ubuntu">I
updated to Ubuntu 17</a>, which includes diffutils 3.5 (the
version in which <code>--color</code> was added). Microsoft does not
official support upgrading Ubuntu, but I ran into no real problems.
</dd>
</dl>
<h3 id="upgrading-wsl-ubuntu">Upgrading WSL Ubuntu</h3>
<p><em>Caveat coder</em> &mdash; there is a <em>reason</em> this is
unsupported by Microsoft at present. I just never ran into those reasons
myself. For example, I used DBus to make upgrading happier; I am not using
any Linux desktop (graphical) programs, so maybe this could be a reason.
</p>
<p>Researching <a
href="http://sourcedigit.com/22194-upgrade-ubuntu-17-04-upgrade-ubuntu-command-line/"
title="Upgrade Ubuntu 17.04 – Upgrade Ubuntu Via Command Line">several</a>
<a href="https://www.howtoforge.com/tutorial/ubuntu-lts-update-dist-upgrade/"
title="How to Update a Ubuntu LTS release to the next LTS Version (release upgrade)">helpful</a>
Internet sources, I:</p>
<ol>
<li>Edited <tt>/etc/update-manager/release-upgrades</tt> to use "normal"
releases, not just LTS
</li>
<li>Fixed <code>/etc/dbus-1/session.conf</code></li>
<li>Ran <code>sudo do-release-upgrade</code> to move to 16.10 from 16.04
</li>
<li>Re-fixed <code>/etc/dbus-1/session.conf</code></li>
<li>Ran <code>sudo do-release-upgrade -d</code> to move to 17.04 from
16.10
</li>
</ol>
<p>(Pay attention: there are many "yN" prompts were the default is to abort:
you <strong>must</strong> enter "y" to these!)</p>
<p>When I am prompted to reboot, I quit the upgrade, close all WSL terminals,
and start a fresh one. There is <a
href="https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/"
title="Windows Subsystem for Linux Overview">no actual kernel</a>
to reboot: it remains 4.4.0-42-Microsoft throughout. The kernel is
emulated by Windows, not an actual file to boot, so upgrades only change
the packages bundled with the kernel, not the kernel itself. <a
href="https://www.microsoft.com/en-us/research/project/drawbridge/">The
underlying abstraction</a> is quite elegant.</p>
<h3 id="almost-there">Almost there</h3>
<p>Can I drop Cygwin and make WSL my daily development environment? Not quite
yet. For shell script work, WSL is excellent. But for my Kotlin, Java,
Ruby, et al, other projects, I rely on IntelliJ IDEA as my editor (though
Emacs might return into my life again). Filesystem interop between Windows
programs (such as <code>java.exe</code>) and WSL is good <a
href="https://youtrack.jetbrains.com/issue/IDEA-171457"
title="Git source control status wrong for project stored in WSL">but
not perfect</a>. </p>
<h3 id="other-options">Other options</h3>
<dl>
<dt>Cygwin on Windows</dt>
<dd>This is and has been my solution for bash on Windows for many years. I
will move to WSL when I'm ready, but I'm not ready yet. I need my
regular development cycle to work first. (See <a href="#almost-there"><cite>Almost
there</cite></a>.) There are downsides to Cygwin, for example,
coping with line endings, but it's been reliable for me.
</dd>
<dt>Homebrew on Mac</dt>
<dd>This is work. My company issues me a Mac laptop, and I use it. For the
most part, it is fine for work with colleagues and clients, though at
times the Mac is <a
href="http://appletoolbox.com/2016/07/macos-sierra-features-review/">a
little strange</a>, and much of the user experiences feels
counterintuitive. Still, the software mostly works, and the hardware
is incredibly good.
</dd>
</dl>
<p><em>But why not just use Linux?</em> Well, my daily machine at home is a
Windows box. Because it's my <em>gaming rig</em>, and <a
href="http://www.lotro.com/en"
title="The Lord of the Rings Online">games I play</a> don't run
well in Linux, and getting a Mac desktop is not currently <a
href="https://arstechnica.com/apple/2017/04/imagining-a-new-mac-pro-the-imac-pro-and-the-future-of-apples-desktops/"
title="Imagining a new Mac Pro, the “iMac Pro,” and the future of Apple’s desktops">a
pretty story</a>.</p>
<p><strong>UPDATE:</strong> More on <a
href="https://blogs.msdn.microsoft.com/wsl/2016/06/08/wsl-system-calls/">how
syscalls work</a>.</p>
<p><strong>UPDATE:</strong> Slightly dated (Microsoft is moving <em>very</em>
fast on WSL&mdash;kudos!), this is a good video presentation on <a
href="https://www.youtube.com/watch?v=_p3RtkwstNk">what happens
under the hood</a>.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-61441081744923453252017-04-12T16:12:00.002-05:002017-04-12T16:24:48.195-05:00Quick diff tip, make, et al<p>I'm using <code>make</code> for a simple shell project, to run tests before
committing. The check was trivial:</p>
<pre class="code">SHELL = bash
test:
&#9;@./run-tests t | grep 'Summary: 16 PASSED, 0 FAILED, 0 ERRORED' &gt;/dev/null</pre>
<p>This has the nice quality of <em>Silence is Golden</em>: say nothing when
all is good. However, it loses the quality of <em>Complain on Failure</em>:
it simply fails without saying why.</p>
<p>A better solution, preserving both qualities:</p>
<pre class="code">SHELL = bash
test:
&#9;@diff --color=auto \
&#9; &lt;(./run-tests t | grep 'Summary: .* PASSED, .* FAILED, .* ERRORED') \
&#9; &lt;(echo 'Summary: 16 PASSED, 0 FAILED, 0 ERRORED')</pre>
<p>It still says nothing when all is good, but now shows on failure how many
tests went awry. <em>Bonus</em>: color for programmers who like that sort
of thing.</p>
<p>Why set <code>SHELL</code> to <code>bash</code>? I'm taking advantage of <a
href="https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html#Process-Substitution"><cite>Process
Substitution</cite></a>. Essentially the command outputs inside the
subshells are turned into special kinds of files, and <code>diff</code>
likes to compare files. Ksh and Zsh also support process substitution, so
I'm going with the most widely available option.</p>
<p><strong>UPDATE:</strong></p> <p>Why are my arguments to <code>diff</code>
ordered like that? In usual testing language, I'm comparing "actual" vs
"expected", and more commonly you'll see programmers list "expected"
first.</p>
<p><code>diff</code> by default colors the <em>left-hand</em> input in <span
style="color:red">RED</span>, and the <em>right-hand</em> input in
<span style="color:green">GREEN</span>. On failure, it makes more sense to
color "actual" in red and "expected" in green. Example output on failure:
</p>
<pre class="code">$ make
<span style="color:cyan">1c1</span>
<span style="color:red">&lt; Summary: 17 PASSED, 1 FAILED, 0 ERRORED</span>
---
<span style="color:green">&gt; Summary: 19 PASSED, 0 FAILED, 0 ERRORED</span>
make: *** [Makefile:4: test] Error 1</pre>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-34339812087763103032017-04-04T10:09:00.001-05:002017-04-04T10:19:08.912-05:00Maven logging and the command line<p>I usually set up my Maven-build projects to be as quiet as possible. My
preference is "<em>Silence is Golden</em>": if the command says nothing on
the command line, it worked; if it failed, it prints to STDERR.</p>
<p>However, sometimes I want to see some output while I'm tracking down a
problem. How best to reconcile these?</p>
<p><a href="https://maven.apache.org/docs/3.3.1/release-notes.html"
title="Release Notes – Maven 3.3.1">Maven 3.3.1</a> (maybe earlier)
introduced the <code>.mvn</code> directory for your project root (note
leading DOT). In here you can keep a <code>jvm.config</code> file which
has the flags to the <code>java</code> command used when running
<code>mvn</code>. Here's my usual <code>jvm.config</code>:</p>
<pre class="code">
-Dorg.slf4j.simpleLogger.defaultLogLevel=WARN
-Dorg.slf4j.simpleLogger.log.Sisu=WARN</pre>
<p>This quiets maven down quite a bit, using the properties to control Maven's
more recent logger, SLF4J. And I normally commit that into my project code
repository.</p>
<p>And for those times I'd like more output? I could edit the file, but I
don't trust myself enough not to accidentally commit those temporary
changes. So I use the command line:</p>
<pre class="code">$ MAVEN_OPTS='-Dorg.slf4j.simpleLogger.defaultLogLevel=INFO' mvn</pre>
<p>Ultimately <a
href="https://github.com/apache/maven/blob/master/apache-maven/src/bin/mvn"><code>mvn</code>
reads <code>.mvn/jvm.config</code></a>, putting the contents into the
variable <code>MAVEN_OPTS</code>, and uses <code>MAVEN_OPTS</code> in the
invocation of <code>java</code>, and you can override the variable
yourself on the command line.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-54122370004837583142017-04-02T10:15:00.001-05:002017-04-02T10:15:54.998-05:00DDD, A Handmaid's Tale<p>(No, this is not a post about the venerable and excellent <a
href="https://www.gnu.org/software/ddd/"
title="DDD - Data Display Debugger">GNU DDD</a>.)</p>
<p><em>Documentation Driven Development</em>&mdash;DDD&mdash;is a term I just
made up (not really; read on). I was working on some code TDD-style (<a
href="http://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html"
title="The Cycles of TDD">"first, write a failing test"</a>), and
also thinking about my user documentation. My usual practice is to get my
tests and code into good shape, push-worthy, and then update the
documentation with my improvements (one hopes). Then the thought struck
me: I'm doing this wrong!</p>
<p>We write tests first as miniature specifications for the code. But my
documentation is conveying to the public my specifications. In the world
of closed-source software, this makes sense. You prepare the documentation
to ship to customers (internal or external); generally holding off until
the code is stable so your documentation is mostly accurate. After all,
with closed source, users can't see your tests or the code: the
documentation is their only view into how to use your code.</p>
<p>With open-source software, this picture is radically changed. Your users
<em>can</em> see your tests and code, in fact, you generally encourage
them to look, or fork! So now your tests are little visible public
specifications. Why documentation then?</p>
<p>Personally I still like solid documentation on open source projects. True,
I could just browse the tests. But that isn't the best way to start with
code that is new to me. I'd like to see examples, some explanation,
perhaps some architecture or high-level pictures. Hence,
documentation.</p>
<p>So, back to DDD. If I'm pushing out my tests and code to a public
repository as soon as they pass (or near enough), how is my documentation
ever to keep up? How do I encourage others to clone or fork my code, and
contribute? I still want new users to have good documentation for getting
started; I still want my tests to ultimately define my specifications. The
answer is easy: <em>First write failing documentation</em>.</p>
<p>This is not at all a new idea! See <a
href="https://collectiveidea.com/blog/archives/2014/04/21/on-documentation-driven-development"
title="On Documentation-Driven Development">Steve Richert</a>, <a
href="https://gist.github.com/zsup/9434452"
title="Documentation-Driven Development">Zach Supalla</a>, and many
others. An early form of this idea is Knuth's <a
href="http://wiki.c2.com/?LiterateProgramming">Literate
Programming</a>.</p>
<h3 id="failing-documentation">Failing documentation</h3>
<p>What is "failing documentation"?</p>
<p>Firstly, just as with "failing tests", you start with documentation of how
your code should behave, but which isn't actually the case. The ways to do
this are the usual suspects:</p>
<ul>
<li>Write examples which don't work, or possibly don't even compile</li>
<li>Write explanations which don't fit your code</li>
<li>Write step-by-step walkthroughs which can't be followed</li>
<li>Write architecture diagrams which are wrong</li>
<li>Etc, etc, etc, anything you'd put in documentation which is invalid
for your current code
</li>
</ul>
<p>Then you fix it:</p>
<ol>
<li>Write failing documentation</li>
<li>Write failing tests which correspond to the documentation</li>
<li>Fix the code to make the tests pass, and the documentation correct
</li>
</ol>
<p>Afterwards you have:</p>
<ul>
<li>Current, accurate documentation</li>
<li>Current, passing tests</li>
<li>Current, working code</li>
</ul>
<h3 id="supporting-ecosystems">Supporting ecosystems</h3>
<p>As straight-forward as DDD is to explain, some software ecosystems make it
easier to actually do than others. A standout example is Python and <a
href="https://en.wikipedia.org/wiki/Doctest"><code>doctest</code></a>.
In <code>doctest</code> you write your tests directly in the API
documentation as examples. This is a perfect marriage of documentation and
tests.</p>
<p><a href="http://swagger.io/">Swagger</a> is an interesting case. It's
generally a documentation-first approach tailored for REST API
specifications. But the documentation is "live documentation"&mdash;i.e.,
an executable web form for exploratory testing&mdash;rather than text and
code examples to read. Using DDD, you would write your REST API
specification first in Swagger, then write failing tests around that
before fixing the code to implement. Clever people <a
href="http://swagger.io/using-swagger-to-detect-breaking-api-changes/"
title="Using Swagger to detect breaking API changes">have
leveraged this</a>.</p>
<h3>About the post title</h3>
<p><cite>The Handmaid's Tale</cite> is a sly reference to Chaucer's <a
href="https://en.wikipedia.org/wiki/The_Wife_of_Bath%27s_Tale"><cite>The
Wife of Bath's Tale</cite></a> (featuring a strong protagonist balancing
among bickering companions), and <a
href="https://en.wikipedia.org/wiki/The_Merchant%27s_Tale"><cite>The
Merchants's Tale</cite></a> sequence. Documentation has often been
treated as subservient to code, an afterthought, when really it is the
first thing most new users see about a system. Give it its due.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-32165454745059270742017-04-01T15:32:00.001-05:002017-04-06T07:41:16.695-05:00Kotlinc on Cygwin<p>There may be a better way, but I found that running <code>kotlinc</code> to
bring up the Kotlin REPL, while in a Cygwin BASH shell using Mintty, did
not respond to keyboard input. A little research indicated the issue is
with JLine, which has some understandable difficulties reconciling running
under Cygwin with running under CMD.</p> <p>The workaround I used:</p>
<pre class="code">$ JAVA_OPTS='-Djline.terminal=unix' kotlinc
Welcome to Kotlin version 1.1.1 (JRE 1.8.0_121-b13)
Type :help for help, :quit for quit
>>> println("FOOBAR")
println("FOOBAR")
FOOBAR</pre> <p>Requesting JLine to use UNIX-y primitives for terminal access
solved the problem. I would like to hear about other solutions.</p> <p>
<strong>UPDATE</strong>: Edited for clarity. And some additional reading:
</p>
<ul>
<li>This is <a href="https://github.com/jline/jline2/issues/245"
title="JLine hangs reading line in GitBash">a known
issue</a></li>
<li>It impacts <a
href="http://stackoverflow.com/questions/25974993/how-to-set-system-property-jline-terminal-to-fix-no-echo-while-typing-on-cygwin"
title="How to set system property jline.terminal (to fix no echo while typing on cygwin's mintty)?">other
programs</a> than just <code>kotlinc</code></li>
<li>The issues <a href="https://github.com/jline/jline3/issues/41"
title="Cygwin pty is used when running with the default windows console"><em>may</em>
be fixed</a> in JLine3
</li>
<li>I filed <a href="https://youtrack.jetbrains.com/issue/KT-17286"
title="Kotlinc does not handline terminal input/output under Cygwin">a
report</a> with JetBrains
</li>
</ul>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-32889587021091456632017-03-18T16:25:00.001-05:002017-03-18T16:25:42.545-05:00Followup on Bash long options<p>A followup on <a
href="https://binkley.blogspot.com/2016/03/bash-long-options.html"
title="Bash long options"><cite>Bash
long options</cite>.</a></p>
<p>The top-level option parsing <code>while</code>-loop I discussed works fine
for regular options. Sometimes you need special parsing for subcommand
options. A hypothetical example might be:</p>
<pre class="code">$ my-script --toplevel-thing my-subcommand --something-wonderful option-arg</pre>
<p>Here the <code>--toplevel-thing</code> option is for <code>my-script</code>,
and <code>--something-wonderful</code> option and its
<var>option-arg</var> is for <code>my-subcommand</code>. Regular
<code>getopts</code> parsing will try to handle all options for the top
level, failing to distinguish subcommand options as separate. Further,
<code>getopts</code> in a function does not behave <a
href="http://stackoverflow.com/a/16655341"
title="Using getopts inside a Bash function">quite as expected</a>.
</p>
<p>One solution is simple and hearkens back to the pre-<code>getopts</code>
days. For the top level:</p>
<pre class="code">while (( 0 < $# ))
do
case $1 in
--toplevel-thing ) _toplevel_thing=true ; shift ;;
-* ) usage &gt;&amp;2 ; exit 2 ;;
* ) break ;;
esac
done</pre>
<p>Using a <code>while</code>-loop with explicit breaks avoids looking too far
along the command line, and wrongly consuming options meant for
subcommands. Rechecking <code>$#</code> each time through the loop breaks
gracefully. Similarly, for subcommands expressed in a function:</p>
<pre class="code">function my-subcommand {
while (( 0 < $# ))
do
case $1 in
--something-special ) local option_arg="$2" ; shift 2 ;;
* ) usage &gt;&amp;2 ; exit 2 ;;
esac
done
# Rest of my-subcommand, using `option_arg` if provided</pre>
<p>This uses the same pattern as the top level so you avoid needing to
remember to handle top level one way, and subcommand another.</p>
<p><a href="https://github.com/binkley/shell/blob/master/git/git-tdd"
title="git-tdd">An example script</a> using this pattern.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-88270005218606270992017-03-13T20:11:00.001-05:002017-03-17T09:43:00.261-05:00Frequent commits<p>Pair posting with guest <a href="https://github.com/skruegs">Sarah
Krueger</a>!</p>
<h3 id="a-source-control-pattern-for-TDD">A source control pattern for
TDD</h3>
<p>At work we recently revisited our commit practices. One issue spotted:
we didn't commit often enough. To address we adopted the source control
pattern in this post. There are lots of benefits; the one that mattered
to me most: <a
href="https://en.wikipedia.org/wiki/Don't_throw_the_baby_out_with_the_bathwater">No
more throwing the baby out with the bathwater</a>, that is, no more
two hour coding sessions only to start again and lose the good with the
bad.</p>
<p>So we worked out this command-line pattern using single unit-of-work
commits (without <code>git rebase -i</code>!):</p>
<pre class="code"># TDD cycle: edit code, run tests, rather-rinse-repeat until green
$ git pull --rebase --autostash &amp;&amp; <i>run tests</i> &amp;&amp; git commit -a --amend --no-edit</pre>
<pre class="code"># Simple unit-of-work commit, push, begin TDD cycle again
$ git commit --amend &amp;&amp; git push &amp;&amp; git commit --allow-empty -m <i>WIP</i></pre>
<h4 id="what-is-this">What is this?</h4>
<ol>
<li>Start with a pull. This ensures you are always current, and find
conflicts as soon as possible.
</li>
<li>Run your <em>full tests</em>. This depends on your project, for
example, <code>mvn verify</code> or <code>rake</code>. If some tests
are slow, split them out, and add a full test run before pushing.
</li>
<li>Amend your work to the current commit. This gives you a safe fallback
known to pass tests. Worst case you might lose some recent work, but
not hours worth. (Hint: <strong>run tests often</strong>.)
</li>
<li>When ready to push, update the commit message to the final message for
public push.
</li>
<li>Push. Share. Make the world better.</li>
<li>Restart the TDD cycle with an empty commit using a message that makes
sense to you, for example "WIP" (work in progress); the message should
be obvious not to push. Key: the TDD cycle command line only amends
commits, so you need a first, empty commit to amend against.
</li>
</ol>
<h4 id="why">Why?</h4>
<p>They key feature of this source control pattern is: Always commit after
reaching green on tests; never commit without testing. When tests
fail, the commit fails (the <code>&amp;&amp;</code> is <a
href="https://en.wikipedia.org/wiki/Short-circuit_evaluation"
title="Short-circuit evaluation">short-circuit logical and</a>).
</p>
<p>In the math sense, this pattern makes testing and committing <a
href="https://en.wikipedia.org/wiki/Bijection,_injection_and_surjection"
title="Bijection, injection and surjection"><em>one-to-one</em>
and <em>onto</em></a>. Since TDD requires frequent running of
tests, this means frequent commits when those tests pass. To avoid a
long series of tiny commits when pushing, amend to collect a unit of
work.</p>
<h4 id="bootstrapping">Bootstrapping</h4>
<p>The TDD cycle depends on an initial, empty commit. The first time using
this source control pattern:</p>
<pre class="code"># Do this after the most recent commit, before any edits
$ git commit --allow-empty -m <i>WIP</i></pre>
<h4 id="adding-files">Adding files</h4>
<p>This pattern, though very useful, does not address new files. You do
need to run <code>git add</code> with new files to include them in the
commit. Automatically adding new files can be dangerous if
<code>gitignore</code> isn't set up right.</p>
<h4 id="it-depends-on-your-style">It depends on your style</h4>
<p>The exact command line depends on your style. You could include a script
to run before tests, or before commit (though the latter might be
better done with a git pre-commit hook). You might prefer merge pulls
instead of rebase pulls. If your editor runs from the command line you
might toss <code>$EDITOR</code> at the front of the TDD cycle.</p>
<p>The command lines assume git, but this source control pattern works with
any system that supports similar functionality.</p>
<h4 id="fine-grained-commits">Fine-grained commits</h4>
<p>An example of style choice. If you prefer fine-grained commits to
unit-of-work single commits (depending on your taste or project; they're
both good practice):</p>
<pre class="code"># TDD cycle: edit code, run tests, rather-rinse-repeat until green
$ git pull --rebase --autostash &amp;&amp; <i>run tests</i> &amp;&amp; git commit -a</pre>
<pre class="code"># Fine-grained commits, push, begin TDD cycle again
$ git push</pre>
<h4 id="improving-your-life">Improving your life</h4>
<p>No matter your exact command line, it can be made friendlier for you.
Yes, shell history can story your long chain of commands. What if they
vary slightly between programmers sharing a project, or what if there
is a common standard approach? <a
href="https://medium.com/@santiaro90/write-your-own-git-subcommands-36d08f6a673e"
title="Write Your Own Git Subcommands">Extend git</a>. Let's call
our example subcommand "tdd". Save this in a file named
<code>git-tdd</code> in your <code>$PATH</code>:</p>
<pre class="code">#!/bin/sh
set -e
case $1 in
test ) git pull --rebase --autostash &amp;&amp; <i>run tests</i> &amp;&amp; git commit -a --amend --no-edit ;;
accept ) git commit --amend &amp;&amp; git push &amp;&amp; git commit --allow-empty -m <i>WIP</i> ;;
esac</pre>
<p>Now your command line becomes:</p>
<pre class="code">$ git tdd test # Repeat until unit of work is ready
$ git tdd accept</pre>
<p>The source is <a
href="https://github.com/binkley/shell/blob/master/git/git-tdd">in
GitHub</a>.</p>
<h3>Updated:</h3>
<p>An editing error left out the <a href="#why">Why?</a> section when
initially posted.</p>
<p>Remember to autostash.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-53977311929850763682017-03-11T12:08:00.001-06:002017-03-15T06:34:09.960-05:00Two BDD styles in Kotlin<p>Experimenting with BDD <em>syntax</em> in Kotlin, I tried these two styles:
</p>
<pre class="code">fun main(args: Array&lt;String&gt;) {
println(So
GIVEN "an apple"
WHEN "it falls"
THEN "Newton thinks")
}
data class BDD constructor(
val GIVEN: String, val WHEN: String, val THEN: String) {
companion object {
val So = So()
}
class So {
infix fun GIVEN(GIVEN: String) = Given(GIVEN)
data class Given(private val GIVEN: String) {
infix fun WHEN(WHEN: String) = When(GIVEN, WHEN)
data class When(private val GIVEN: String, private val WHEN: String) {
infix fun THEN(THEN: String) = BDD(GIVEN, WHEN, THEN)
}
}
}
}</pre>
<p>And:</p>
<pre class="code">fun main(args: Array&lt;String&gt;) {
println(GIVEN `an apple`
WHEN `it falls`
THEN `Newton thinks`
QED)
}
infix fun Given.`an apple`(WHEN: When) = When()
infix fun When.`it falls`(THEN: Then) = Then(GIVEN)
infix fun Then.`Newton thinks`(QED: Qed) = BDD(GIVEN, WHEN)
inline fun whoami() = Throwable().stackTrace[1].methodName
data class BDD(val GIVEN: String, val WHEN: String, val THEN: String = whoami()) {
companion object {
val GIVEN = Given()
val WHEN = When()
val THEN = Then("")
val QED = Qed()
}
class Given
class When(val GIVEN: String = whoami())
class Then(val GIVEN: String, val WHEN: String = whoami())
class Qed
}</pre>
<p>Comparing <code>main()</code> methods, which is easier to read or use? I
haven't tried implementing, just have looked at testing code style. Note
that I'm using <a
href="https://kotlinlang.org/docs/reference/functions.html#infix-notation">the
<code>infix</code> feature</a> of Kotlin to have my BDD
"GIVEN/WHEN/THEN" as punctuation free as I'm able.</p>
<p>In the one case&mdash;using strings to describe cases&mdash;, an
implementation would be more similar to Spec or Cucumber, which usually
uses pattern matching to associate text with implementation. In the other
case&mdash;using functions to describe cases&mdash;, an implementation
goes directly into the function definition. In either case, Kotlin only
supports binary infix functions, not unary (of course, you say, that's
what "infix" means!), so I need either an initial starting token
(<code>So</code> in the strings case) or an ending one (<code>QED</code>
in the functions case).</p>
<p>I'm curious how implementation sorts out.</p><p>(<a
href="https://github.com/binkley/skratch/tree/master/src/main/kotlin/hm/binkley/labs/skratch/bdd">Code
here.</a>)</p>
<h4>UPDATE:</h4>
<p>I have working code now that runs these BDD sentences but remain unclear
which of the two styles (strings vs functions) would be easier to work
with:</p>
<pre class="code">fun main(args: Array&lt;String&gt;) {
var apple: Apple? = null
upon("an apple") {
apple = Apple(Newton(thinking = false))
}
upon("it falls") {
apple?.falls()
}
upon("Newton thinks") {
assert(apple?.physicist?.thinking ?: false) {
"Newton is sleeping"
}
}
println(So
GIVEN "an apple"
WHEN "it falls"
THEN "Newton thinks")
}</pre>
<p>Vs:</p>
<pre class="code">fun main(args: Array&lt;String&gt;) {
println(GIVEN `an apple`
WHEN `it falls`
THEN `Newton thinks`
QED)
}
var apple: Apple? = null
infix fun Given.`an apple`(WHEN: When) = upon(this) {
apple = Apple(Newton(thinking = false))
}
infix fun When.`it falls`(THEN: Then) = upon(this) {
apple?.falls()
}
infix fun Then.`Newton thinks`(QED: Qed) = upon(this) {
assert(apple?.physicist?.thinking ?: false) {
"Newton is sleeping"
}
}</pre>
<p>The strings style is certainly more familiar. However, mistakes in
registering matches of "GIVEN/WHEN/THEN" clauses appear at runtime and do
not provide much help.</p>
<p>The functions style is more obtuse. However, mistakes cause compile-time
errors that are easier to understand, and your code editor can navigate
between declaration and usage.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-74467214544288295822016-09-16T19:35:00.001-05:002016-09-16T19:35:14.881-05:00Google Test, generated source, and GNU Make<p>I had trouble with this arrangement:</p> <ul><li>Using Pro*C for a client to generate "C" files from <code>.pc</code> sources</li> <li><a href="https://github.com/google/googletest/blob/master/googletest/docs/Primer.md">Google Test</a> for unit testing</li> <li>GNU Make</li></ul> <h4>What was the problem?</h4> <p>Ideally I could write a pattern rule like this:</p> <pre class="code">%-test.o: %.c %-test.cc</pre> <p>This means when I want to compile my C++ test source, it required <code>make</code> first run the Pro*C preprocessor to generate a "C" source used in the test. Why? Google tests follow this template:</p> <pre class="code">#include "source-to-test.c"
#include &lt;gtest/gtest.h&gt;
// Tests follow</pre> <p>Google test <em>includes</em> your source file (not header) so the test code has access to static variables and functions (think "private" if you're from Java or C#).</p> <p>So my problem is <code>make</code> is very clever with rules like:</p> <pre class="code">%.foo: %.bar
%.qux: %.foo</pre> <p>And knows that if you want a "bob.qux", which needs a "bob.foo", and there's no "bob.foo" but there is a file named "foo.bar", <code>make</code> follows the recipe for turning a "bar" into a "foo", and this satisfies the rule for "bob.qux".</p> <p>However the simple rule I guessed at:</p> <pre class="code">%-test.o: %.c %-test.cc</pre> <p>Doesn't work! GNU Make has a corner case when there are multiple prerequisites (dependencies), and won't make missing files even where there's another rule saying how to do so.</p> <p>There is another way to state what I want:</p> <pre class="code">%-test.o: %-test.cc: %.c</pre> <p>This is called a <em>static rule</em>. It looks promising, but again doesn't work. GNU make does not support patterns (the "%") in static rules. I would need to write each case out explicitly, e.g.:</p> <pre class="code">a-test.o: a-test.cc: a.c</pre> <p>While this <em>does</em> work, it's also a problem.</p> <hr/> <h4>What's wrong with being explicit?</h4> <p>Nothing is wrong with "explicit" per se. Usually it's a good thing. In this case, it clashes with the rule of "say it once". For each test module a programmer writes, he would need to edit the <code>Makefile</code> with a new, duplicative rule. So when new tests break, instead of thinking "my code is wrong", he needs to ask "is it my code, or my build?" Extra cognitive burden.</p> <hr/> <h4>What does work?</h4> <p>There is a way to get the benefit of a <em>static rule</em> without the duplication, but it's hackery&mdash;good hackery, to be sure, but violating the "rule of least surprise". Use <code>make</code>'s powers to rewrite the <code>Makefile</code> at run time:</p> <pre class="code">define BUILD_test
$(1:%=%.o): $(1:%-test=%.c) $(1:%=%.cc)
$$(COMPILE.cc) $$(OUTPUT_OPTION) $(1:%=%.cc)
$(1): $(1:%=%.o)
endef
$(foreach t,$(wildcard *-test.cc),$(eval $(call BUILD_test,$(t:%.cc=%))))</pre> <p>What a mouthful! If I have a "foo-test.cc" file, <code>make</code> inserts these rules into the build:</p> <pre class="code">foo-test.o: foo.c foo-test.cc
$(COMPILE.cc) $(OUTPUT_OPTION) foo-test.cc
foo-test: foo-test.o</pre> <p>I'd like something simpler, less inscrutable. Suggestions welcome!</p>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-51807057052943256442016-05-30T12:39:00.000-05:002016-05-30T12:39:11.167-05:00Gee wilickers<p>I have a common command-line pattern I grew tired of typing. An example</p> <pre class="code">$ mvn verify | tee verify.out</pre> <p>I use this pattern so often as I want to both watch the build on screen, and have a save file to grep when something goes wrong. Sometimes I also find myself telling the computer: <pre class="code">$ mvn verify | tee verify.out
$ mv verify.out verify-old.out
$ $EDITOR pom.xml
$ mvn verify | tee verify.out
$ diff verify-old.out verify.out</pre> <p>I want to see what changed in my build. But ... too much typing! So I automated with <a href="https://github.com/binkley/shell/tree/master/gee"><code>gee</code>, a mashup of <code>git</code> and <code>tee</code></a>. You can think of it as source control for &lt;STDOUT&gt;.</p> <p>Now I can type:</p> <pre class="code">$ gee -o verify.out mvn verify
$ gee -g git diff HEAD^ # Did build output change?</pre> <p>How does it work? <code>gee</code> keeps a git repository in a hidden directory (<code>.gee</code>), committing program output there using <code>tee</code>. It follows simple conventions for file name and commit message (changeable with flags), and accepts &lt;STDIN&gt;:</p> <pre class="code">$ mvn verify | gee -o verify.out -m 'After foobar edit'</pre>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-7624210844307568272016-05-22T11:53:00.001-05:002016-05-22T12:40:50.461-05:00Automating github<p>I got tired of downloading <a href="https://github.com/binkley/shell">my own scripts from Github</a> when working among multiple projects. So <a href="https://github.com/binkley/shell/blob/master/run-from-url">I automated it</a>, of course. The <em>bitsh</em> project reuses a test script from the <em>shell</em> project, and now the <a href="https://github.com/binkley/bitsh/blob/master/Makefile"><code>Makefile</code></a> for bitsh is simply: <pre class="code">SHELL=/bin/bash
test:
&#9;@[ -t 1 ] &amp;&amp; flags=-c ; \
&#9;./test-bitsh.sh -i -- $$flags t</pre> <p>When <a href="https://github.com/binkley/shell/blob/master/plain-bash-testing/run-tests"><code>run-tests</code></a> is updated in Github, bitsh automatically picks up the changes. And I learned <a href="https://en.wikipedia.org/wiki/HTTP_ETag">the value of ETag</a>.</p> <p>By the way, why "bitsh"? I hunted around for project names combining "git" and "bash" and found most of them already taken. Beggars can't be choosers.</p> <p><strong>UPDATE:</strong> I found the &lt;TAB&gt; character got munged by Blogger to a single space. This is unfortunate as you cannot copy out a valid <code>Makefile</code>. <a href="http://stackoverflow.com/a/9661018">One solution</a> is to put in an HTML tab entity.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-11569386533424872642016-05-21T14:13:00.000-05:002016-05-21T14:14:07.091-05:00Metaprogramming with Bash<p>Most programmers do not take full advantage of the languages they work in, though some languages make this a real challenge. Take <a href="https://en.wikipedia.org/wiki/Metaprogramming">metaprogramming</a>, or programs that have some self-knowledge. LISP-family languages make this easy and natural; those <a href="https://en.wikipedia.org/wiki/Macro_(computer_science)">with macros</a> even more so. Bytecode languages (think Java), and even more so object code languages (think "C"), fall back on extra-linguistic magic such as <a href="https://en.wikipedia.org/wiki/AspectJ">AOP rewriting</a>.</p> <p>Text-based languages lay in a middle ground. Best known <a href="https://en.wikipedia.org/wiki/Bash_(Unix_shell)">is Bash</a>. Rarely do programmers take full advantage of Bash features, and few would think of metaprogramming. Not as clean as LISP macros, it is still straight forward.</p> <p>As an example of function rewriting note <a href="#line_21">line 21</a>. The surrounding function body redefines an existing function, incorporating the original's body into itself. This is very similar to aspect-oriented programming with a "around" advice. Much care is taken to preserve the original function context.</p> <p>This is a fully working script&mdash;give it a try!</p> <pre class="code">#!/bin/bash
export PS4='+${BASH_SOURCE}:${LINENO}: ${FUNCNAME[0]:+${FUNCNAME[0]}(): } '
pgreen=$(printf "\e[32m")
pred=$(printf "\e[31m")
pboldred=$(printf "\e[31;1m")
preset=$(printf "\e[0m")
pcheckmark=$(printf "\xE2\x9C\x93")
pballotx=$(printf "\xE2\x9C\x97")
pinterrobang=$(printf "\xE2\x80\xBD")
function _register {
case $# in
1 ) local -r name=$1 arity=0 ;;
2 ) local -r name=$1 arity=$2 ;;
esac
read -d '' -r wrapper &lt;&lt;EOF
function $name {
# Original function
<a name="line_21">$(declare -f $name | sed '1,2d;$d')</a>
__e=\$?
shift $arity
if (( 0 &lt; __e || 0 == \$# ))
then
__tally \$__e
else
"\$@"
__e=\$?
__tally \$__e
fi
}
EOF
eval "$wrapper"
}
let __passed=0 __failed=0 __errored=0
function __tally {
local -r __e=$1
$__tallied &amp;&amp; return $__e
__tallied=true
case $__e in
0 ) let ++__passed ;;
1 ) let ++__failed ;;
* ) let ++__errored ;;
esac
_print_result $__e
return $__e
}
function _print_result {
local -r __e=$1
case $__e in
0 ) echo -e $pgreen$pcheckmark$preset $_scenario_name ;;
1 ) echo -e $pred$pballotx$preset $_scenario_name ;;
* ) echo -e $pboldred$pinterrobang$preset $_scenario_name ;;
esac
}
function check_exit {
(( $? == $1 ))
}
function make_exit {
local -r e=$1
shift
(exit $e)
"$@"
}
function check_d {
[[ $PWD == $1 ]]
}
function change_d {
cd $1
}
function variadic {
:
}
function early_return {
return $1
}
function eq {
[[ "$bob" == nancy ]]
}
function normal_return {
(exit $1)
}
function f {
local -r bob=nancy
}
function AND {
"$@"
}
function SCENARIO {
local -r _scenario_name="$1"
shift
local __tallied=false
local __e=0
pushd $PWD &gt;/dev/null
"$@"
__tally $?
popd &gt;/dev/null
}
_register f
_register normal_return 1
_register variadic 1
_register eq
_register early_return 1
_register change_d 1
_register check_exit 1
echo "Expect 10 passes, 4 failures and 4 errors:"
SCENARIO "Normal return pass direct" normal_return 0
SCENARIO "Normal return fail direct" normal_return 1
SCENARIO "Normal return error direct" normal_return 2
SCENARIO "Normal return pass indirect" f AND normal_return 0
SCENARIO "Normal return fail indirect" f AND normal_return 1
SCENARIO "Normal return error indirect" f AND normal_return 2
SCENARIO "Early return pass indirect" f AND early_return 0
SCENARIO "Early return fail indirect" f AND early_return 1
SCENARIO "Early return error indirect" f AND early_return 2
SCENARIO "Early return pass direct" early_return 0
SCENARIO "Early return fail direct" early_return 1
SCENARIO "Early return error direct" early_return 2
SCENARIO "Variadic with none" f AND variadic
SCENARIO "Variadic with one" f AND variadic apple
SCENARIO "Local vars" f AND eq
here=$PWD
SCENARIO "Change directory" change_d /tmp
SCENARIO "Check directory" check_d $here
SCENARIO "Check exit" make_exit 1 AND check_exit 1
(( 0 == __passed )) || ppassed=$pgreen
(( 0 == __failed )) || pfailed=$pred
(( 0 == __errored )) || perrored=$pboldred
cat &lt;&lt;EOS
$ppassed$__passed PASSED$preset, $pfailed$__failed FAILED$preset, $perrored$__errored ERRORED$preset
EOS</pre> <p>Practical use of this script as a test framework would pull out <code>SCENARIO</code> and <code>AND</code> to a separate source script, included with "." or <code>source</code>, put the registered functions and their helpers in another source script, and provide command-line parsing to pick out which tests to execute. <a href="https://github.com/binkley/bitsh/blob/master/test-bitsh.sh"><code>test-bitsh.sh</code></a> is an example in progress.</p>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-90965182202068281332016-05-19T16:26:00.002-05:002016-05-21T14:14:23.182-05:00Color your world<p>My coworkers use many ad hoc or single-purpose scripts, things like: checking system status, wrappers for build systems, launching services locally, etc. My UNIX background tells me, "keep it simple, avoid output; <em>Silence is Golden</em>."</p> <p>Somehow my younger colleagues aren't impressed.</p> <p>So to avoid acting my age I started sprinkling color into my scripts, and it worked. Feedback was uniformly positive. And true to my UNIX roots, I provided command line flags to disable color.</p> <p>Some lessons for budding BASHers:</p> <ol> <li>Yes, experiment and learn, but be sure to do your research. The Internet has <a href="http://www.tldp.org/LDP/abs/html/" title="Advanced Bash-Scripting Guide">outstanding</a> <a href="http://wiki.bash-hackers.org/" title="The Bash Hackers Wiki">help</a> <a href="http://stackoverflow.com/questions/tagged/bash">for BASH</a>.</li> <li>Learn standard idioms (see below).</li> <li>Don't overdo it. <a href="https://en.wikipedia.org/wiki/ANSI_escape_code#Colors">Color for summary lines and warnings</a> have more impact when the rest of the text is plain.</li> <li>Keep functions small, just as you would in other languages. BASH is a programming language with a command line, so keep your good habits when writing shell.</li> <li>Collaborate, pair! This comes naturally to my fellows. Coding is more enjoyable, goes faster and has fewer bugs.</li> </ol> <h3>Idioms</h3> <p>Most of these idioms appear in <a href="https://github.com/binkley/shell/tree/master/testing-with-bash">testing with bash</a>, a simple BASH BDD test framework I wrote for demonstration.</p> <h4>Process command-line flags</h4> <pre class="code">this=false
that=true
while getopts :htT-: opt
do
[[ - == $opt ]] && opt="${OPTARG%%=*}" OPTARG="${OPTARG#*=}"
case $opt in
h | help ) print_help ; exit 0 ;;
t | this ) this=true ;;
T | no-that ) that=false ;;
esac
done
shift $((OPTIND - 1))</pre> <h4>Keep boolean toggles simple</h4> <pre class="code">run_faster=true
if $run_faster
then
use_faster_algorithm "$@"
else
use_more_correct_algorithm "$@"
fi</pre> <h4>Simple coloring</h4> <pre class="code">pgreen="\e[32m"
pred="\e[31m"
preset="\e0m"
if $success
then
echo -e "${pgreen}PASS${preset} $test_name"
else
echo -e "${pred}FAIL${preset} $test_name - $failure_reason"
EOM
fi</pre> <h4>Consistent exit codes</h4> <pre class="code">function check_it {
local -r failed=$1
local -r syntax_error=$2
if $syntax_error
then
return 2
elif $failed
then
return 1
else
return 0
fi
}</pre> <h3>Coda</h3> <p>There are many more idioms to learn, hopefully this taste catches your interest. I was careful to include others mixed in with these (what does <code>local -r</code> do?) to whet the appetite for research. Go try <a href="http://bashdb.sourceforge.net/bashdb.html">the BASH debugger</a> sometime.</p> <p><strong>UPDATE</strong>: Fixed thinko. ANSI escape codes need to be handled by <code>echo -e</code> or <code>printf</code>, not sent directly via <code>cat</code>!</p>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-14044564663290735072016-05-02T17:28:00.001-05:002016-05-02T17:28:48.451-05:00Do not test Java getters and setters<p>An excellent project <a href="https://github.com/oshoukry/openpojo/wiki">from Osman Shoukry</a> to automate testing of Java getters and setters&mdash;that is when you have getters and setters to test. There's the rub: <em>do not write getters or setters</em>.</p> <p>For starters they violate encapsulation, exposing your objects innards to others. Ok, but there are frameworks which require them, even in 2016. What to do?</p> <p><a href="https://projectlombok.org/features/GetterSetter.html">Generate them</a>:</p> <pre class="code">@Getter
@Setter
@RequiredArgsConstructor
public final class SampleBean {
private final String left;
private String right;
}</pre> <p>With what result?</p> <pre class="code">public final class SampleBean {
private final String left;
private String right;
@java.lang.SuppressWarnings("all")
@javax.annotation.Generated("lombok")
public String getLeft() {
return this.left;
}
@java.lang.SuppressWarnings("all")
@javax.annotation.Generated("lombok")
public String getRight() {
return this.right;
}
@java.lang.SuppressWarnings("all")
@javax.annotation.Generated("lombok")
public void setRight(final String right) {
this.right = right;
}
@java.beans.ConstructorProperties({"left"})
@java.lang.SuppressWarnings("all")
@javax.annotation.Generated("lombok")
public SampleBean(final String left) {
this.left = left;
}
}</pre> <p><strong>This code need never be tested</strong>. Test the generator, not the generated code. If the generator is correct, so is the generated code. In this case, Lombok is <a href="https://github.com/rzwitserloot/lombok/tree/master/test">heavily tested</a>.</p> <p>As an alternative to the constructor taking field values, Jesse Wilson has <a href="https://publicobject.com/2016/04/10/reflection-friendly-value-objects/" title="Reflection-friendly Value Objects">interesting advice</a>.</p>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-72172252194252084962016-04-07T18:12:00.001-05:002016-05-21T14:14:37.313-05:00BDD-style fluent testing in BASH<p>I wanted to impress on my colleagues that BASH was still hip, still relevant. And that I wasn't an old hacker. So I wrote a small BDD test framework for BASH.</p> <h3>Techniques</h3> <p>Fluent coding relies on several BASH features:</p> <ul><li>Variable expansion happens before executing commands</li> <li>A shell function is indistinguishable from a program: they are called the same way</li> <li>Local function variables are dynamically scoped but only within a function, so are visible to other functions called within that scope, directly or indirectly through further function calls</li></ul> <p>Together with <a href="https://en.wikipedia.org/wiki/Builder_pattern">Builder pattern</a>, it's easy to write given/when/then tests. (Builder pattern here solves the problem not of telescoping constructors, but massive, arbitrary argument lists.)</p> <p>So when you run:</p> <pre class="code">function c {
echo "$message"
}
function b {
"$@"
}
function a {
local message="$1"
shift
"$@"
}
a "Bob's your uncle" b c</pre> <p>You see the output:</p> <pre class="code">Bob's your uncle</pre> <h4>How does this work?</h4> <p>First BASH expands variables. In function <code>a</code> this means that after the first argument is remembered and removed from the argument list, <code>"$@"</code> expands to <code>b c</code>. Then <code>b c</code> is executed.</p> <p>Then BASH calls the function <code>b</code> with argument "c". Similarly <code>b</code> expands <code>"$@"</code> to <code>c</code> and calls it.</p> <p>Finally as <code>$message</code> is visible in functions called by <code>a</code>, <code>c</code> prints the first argument passed to <code>a</code> (as it was remebered in the variable <code>$message</code>), or "Bob's your uncle" in this example.</p> <p>Running the snippet with xtrace makes this clear (assuming the snippet is saved in a file named <code>example</code>):</p> <pre class="code">bash -x example
+ a 'Bob'\''s your uncle' b c
+ local 'message=Bob'\''s your uncle'
+ shift
+ b c
+ c
+ echo 'Bob'\''s your uncle'
Bob's your uncle</pre> <p>So the test functions for <code>given_jar</code>, <code>when_run</code> and <code>then_expect</code> (along with other, similar functions) work the same way. Keep this in mind.</p> <h3>Application</h3> <p>So how does this buy me fluent BDD?</p> <p>Given these functions:</p> <pre class="code">function then_expect {
local expectation="$1"
shift
if [[ some_test "$pre_condition" "$condition" "$expectation" ]]
then
echo "PASS: $scenario"
return 0
else
echo "FAIL: $scenario"
return 1
fi
}
function when {
local condition="$1"
shift
"$@"
}
function given {
local pre_condition="$1"
shift
"$@"
}
function scenario {
local scenario="$1"
shift
"$@"
}</pre> <p>When you write:</p> <pre class="code">scenario "Some test case" \
given "Something always true" \
when "Something you want to test" \
then_expect "Some outcome"</pre> <p>Then it executes:</p> <pre class="code">some_test "Something always true" "Something you want to test" "Some outcome"</pre> <p>And prints one of:</p> <pre class="code">PASS Some test case
FAIL Some test case</pre> <p>A real example at <a href="https://github.com/binkley/shell/tree/master/testing-bash">https://github.com/binkley/shell/tree/master/testing-bash</a>.</p>
Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-90904979298816528752016-03-24T16:17:00.000-05:002017-03-18T16:27:47.703-05:00Bash long options<p><b>UPDATED</b>: Long options with arguments in the "name=value" style. The original post neglected this important case.</p> <p>For years I've never know quite the right way to handle long options in Bash without significant, ugly coding. The usual sources (<a href="http://www.tldp.org/LDP/abs/html/">Advanced Bash-Scripting Guide</a>, <a href="http://wiki.bash-hackers.org/">The Bash Hackers Wiki</a>, others) are not much help. An occasional glimpse appears on StackOverflow, but not well explained or voted.</p> <h3>Solution</h3> <p>Working with a colleague yesterday, we found this:</p> <pre class="code">name=Bob
while getopts :hn:-: opt
do
[[ - == $opt ]] &amp;&amp; opt=${OPTARG%%=*} OPTARG=${OPTARG#*=}
case $opt in
h | help ) print_help ; exit 0 ;;
n | name ) name=$OPTARG ;;
* ) print_usage &gt;&amp;2 ; exit 2 ;;
esac
done
shift $((OPTIND - 1))
echo "$0: $name"</pre> <pre>$ ./try-me -h
Usage: ./try-me [-h|--help][-n|--name &lt;name&gt]
$ ./try-me --help
Usage: ./try-me [-h|--help][-n|--name &lt;name&gt]
$ ./try-me -n Fred
./try-me: Fred
$ ./try-me --name=Fred
./try-me: Fred</pre> <p>Magic!</p> <p>I checked with bash 3.2 and 4.3. At least for these, the '-' option argument has a bit of magic when it takes an argument. When the argument to '-' starts with a dash, as in <code>--help</code> (here "-help" is the argument to the '-' option), <code>getopts</code> drops the argument's leading '-', and <code>OPTARG</code> is just the text ("help" in this example). Only '-' has this magic.</p> <p>Add a quick check for '-' at the top of the while-loop, and the case-block is simple and clear.</p> <p>Bob's your uncle.</p> <p><strong>UPDATE:</strong> <a href="https://binkley.blogspot.com/2017/03/followup-on-bash-long-options.html" title="Followup on Bash long options"><cite>Followup on Bash long options</code></a>.</p>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0tag:blogger.com,1999:blog-5638372.post-71810662772419496612016-03-01T17:09:00.001-06:002016-03-01T17:18:52.161-06:00Hand-rolling builders in Java<p>I showed off a hand-rolled example Java builder pattern today. It has some benefits over existing builder solutions, but is more work than I like:</p> <ol><li>All parts of the builder are immutable; you can pass a partially built object for another object to complete (I'm looking at you, <a href="https://projectlombok.org/features/Builder.html" title="@Builder">Lombok</a>)</li> <li>It's syntactically complete; that is, code won't compile without providing all arguments for the thing to build</li> <li>It's easy to generalize; in fact, I'm thinking about an annotation processor to generate it for you (but not there quite yet)</li></ol> <pre class="code">@EqualsAndHashCode
@ToString
public final class CartesianPoint {
public final int x;
public final int y;
public static Builder builder() {
return new Builder();
}
private CartesianPoint(final int x, final int y) {
this.x = x;
this.y = y;
}
public static final class Builder {
public WithX x(final int x) {
return new WithX(x);
}
@RequiredArgsConstructor
public static final class WithX {
private final int x;
public WithY y(final int y) {
return new WithY(y);
}
@RequiredArgsConstructor
public final class WithY {
private final int y;
public CartesianPoint build() {
return new CartesianPoint(x, y);
}
}
}
}
}</pre> <p>That was a lot to say! Which is why most times you don't hand-roll builders. Usage is obvious:</p> <pre class="code">@Test
public void shouldBuild() {
final CartesianPoint point = CartesianPoint.builder().
x(1).
y(2).
build();
assertThat(point.x).isEqualTo(1)
assertThat(point.y).isEqualTo(2);
}</pre> <p>Adding caching for equivalent values is not hard:</p> <pre class="code">public static final class Builder {
private static final ConcurrentMap<CartesianPoint, CartesianPoint>
cache = new ConcurrentHashMap<>();
public WithX x(final int x) {
return new WithX(x);
}
@RequiredArgsConstructor
public static final class WithX {
private final int x;
public WithY y(final int y) {
return new WithY(y);
}
@RequiredArgsConstructor
public final class WithY {
private final int y;
public CartesianPoint build() {
final CartesianPoint point = new CartesianPoint(x, y);
final CartesianPoint cached = cache.
putIfAbsent(point, point);
return null == cached ? point : cached;
}
}
}
}</pre>Brian Oxleyhttps://plus.google.com/118426136972205251581noreply@blogger.com0