Export

Jan242018

I created this Spark Streaming service to analyze hashtags coming out of Twitter. Why? Politics can be entertaining, or irritating, depending on your perception, but discussions about our culture and its people are worth having.

This tool is open source and provides a "top ten" report of Tweets matching any of a given (configurable) set of keywords. Enjoy!

Mar222017

Did your OpenVPN / Mikrotik certificates (both CA and client certs) expire? It turns out they were only good for one year!

The default duration for these VPN certificates, and Certificate Authority, is 365 days. My OpenVPN VPN client quit working recently (it appears stuck in a loop retrying the connection and authentication process). Oops.

The fix is to remove and re-create, then distribute, new Certificates for the OpenVPN Clients. We must first remove keys and certificates for the Mikrotik CA, Server and Clients'. If there is a simpler way, please shoot me a note below in the comments.

First, log in to your router with ssh and remove the old client certificates, server certificate and the certificate authority entries:

The certificates must be exported from the router and downloaded to your PC or laptop for dissemination. Export from the command line, then visit the admin website http://192.168.1.1/ or use sftp to retrieve the certificate files.

We can now move over to your client's .ovpn file and substitute new CA and client keys and certificates. For this procedure we refer you back to the OpenVPN Client Configuration section of last year's blog post. The .crt files can be inserted/substituted in your per-client .ovpn file as-is. The .key file must be decrypted with your chosen export-passphrase thusly:

openssl rsa -in cert_export_client1.key -text

Hope this cures your issue and my apologies for being short-sighted on my original post. I'm up to RouterOS version 6.41.3 on this end for this solution / workaround (updated 3/28/2018).

Jul222016

Lately I've been working on a couple Ubuntu open source projects, MAAS and Juju. While I have been making good use of the Ubuntu server distribution (16.04 LTS), I hadn't installed the desktop until today.

When I tried to add the Google Chrome web browser, I found a little work was required to add the Google PPA and get the install to work. There are three steps required:

Mar192016

Mar042014

Sorting in the mongo shell (JavaScript)

Sorting the results of a MongoDB query is straightforward in the mongo JavaScript shell. One provides a JSON document with associative array listing the fields to sort on and their sort order (1 for ascending, -1 for descending).

db.myCollection.find().sort( { fieldA: 1 } )

When one wants to sort on two fields, for example, fieldA first, then fieldB second, the mongo JavaScript shell obeys the left-to-right order in the associative array without complaint. An example of a two field sort in the mongo shell:

db.myCollection.find().sort( { fieldA: 1, fieldB: 1 } )

Sorting in python using pymongo

But what about a Python script using pymongo? Can one use the same associative array syntax? No.

However, one can use a List of immutable Tuples in Python to achieve the same effect. An example of the same two field sort in a Python script using pymongo:

The automated install (initiated from OpsCenter management web application) will configure the IP address for the Opscenter daemon's stomp listener for you. If you perform a tar-based installation of DSE or OpsCenter you will need to take some additional steps to configure your datastax-agent (formerly known as "opscenter-agent").

step one - configure stomp listener endpoint

After you untar the dse-x.y.z-bin.tar.gz archive, locate the dse-x.y.z/datastax-agent/conf directory relative to the install directory. Substitute the DSE version for x.y.z, e.g. dse-4.0.0/datastax-agent/conf.

Edit dse-x.y.z/datastax-agent/conf/address.yaml and set the IP address where you have installed OpsCenter (this could be a different node or host). Use your own IP address in place of the one below.

stomp_interface:192.168.111.222

step two - configure path to cassandra.yaml

This step is tricky because the DataStax documentation (URL above) is misleading. When a tar-based install is conducted, datastax-agent cannot locate DSE / Cassandra configuration files using a relative path (as of DSE 4.0.0). You will get the following error message:

datastax-agent startup error

ERROR [Initialization] 2014-02-2707:57:24,306 Exception inthread"Initialization"ERROR [StompConnection receiver] 2014-02-2707:57:24,306 failed calling listenerclojure.lang.ExceptionInfo: throw+: {:type :bad-permissions, :message "Unable to locate the cassandra.yaml configuration file. If your configuration file is not located with the Cassandra install, please set the 'conf_location' option in the Cassandra section of the OpsCenter cluster configuration file and restart opscenterd. Checked the following directories: [\"/etc/dse/cassandra/cassandra.yaml\"\"/etc/cassandra/conf/cassandra.yaml\"\"/etc/cassandra/cassandra.yaml\"\"/Users/resources/cassandra/conf/cassandra.yaml\"]"} {:object {:type :bad-permissions, :message "Unable to locate the cassandra.yaml configuration file. If your configuration file is not located with the Cassandra install, please set the 'conf_location' option in the Cassandra section of the OpsCenter cluster configuration file and restart opscenterd. Checked the following directories: [\"/etc/dse/cassandra/cassandra.yaml\"\"/etc/cassandra/conf/cassandra.yaml\"\"/etc/cassandra/cassandra.yaml\"\"/Users/resources/cassandra/conf/cassandra.yaml\"]"}, :environment {tar-location "/Users/resources/cassandra/conf/cassandra.yaml", conf nil, checked-files ["/etc/dse/cassandra/cassandra.yaml""/etc/cassandra/conf/cassandra.yaml""/etc/cassandra/cassandra.yaml""/Users/resources/cassandra/conf/cassandra.yaml"]}} at opsagent.util.cassandra_util$cassandra_conf_location.invoke(cassandra_util.clj:118) at opsagent.util.cassandra_util$get_cassandra_conf.invoke(cassandra_util.clj:130) at opsagent.opsagent$create_thrift_conf_vars.invoke(opsagent.clj:52) at opsagent.opsagent$post_interface_startup.doInvoke(opsagent.clj:95) at clojure.lang.RestFn.invoke(RestFn.java:421) at opsagent.conf$handle_new_conf.invoke(conf.clj:171) at opsagent.messaging$message_callback$fn__5192.invoke(messaging.clj:30) at opsagent.messaging.proxy$java.lang.Object$StompConnection$Listener$7f16bc72.onMessage(Unknown Source) at org.jgroups.client.StompConnection.notifyListeners(StompConnection.java:311) at org.jgroups.client.StompConnection.run(StompConnection.java:261) at java.lang.Thread.run(Thread.java:724)

The workaround for this error is to explicitly tell datastax-agent where to find cassandra.yaml. This will be in the following path relative to your DSE installation directory: resources/cassandra/conf/cassandra.yaml

Edit datastax-agent/conf/address.yaml again and add the following directive. Note that this is undocumented and the official documentation is misleading at this time. This will be fixed in an upcoming release but this workaround should help for the impatient.

Feb112014

Removal or deletion of old boot images is useful from time to time. Deleting the old, unneeded entries from the boot menu is one advantage, but the recovery of disk space is the more significant benefit.

Each boot menu item, like 'OpenIndiana-1', or 'OpenIndiana' has a corresponding Boot Environment. Each of these Boot Environments is realized with a ZFS snapshot of the root filesystem. The following utilizes OpenIndiana for the example environment, but the utility works the same way in OpenSolaris and Oracle Solaris.

To remove the ZFS snapshot, we first want to remove the boot menu entry. Management of Boot Environments, as the Solaris documentation refers to them, is not too difficult -- the beadm utility is used.

First obtain a list of existing BootEnvironments. Decide which one(s) to keep and which ones to expunge.

#dhcp-boot=pxelinux.0 # ONLY SET IF want gPXE to run script or chain-load pxelinuxdhcp-boot=tag:#OTHER,gpxe.0 # Here #OTHER means 'not OTHER': i.e. the tag is not set#dhcp-boot=tag:ECS,pxelinux.0#dhcp-boot=tag:ECS,sanboot-test.gpxe # run gpxe script

Nov072011

How to install Jetty 7 as a Windows service

Earlier versions of Jetty came with a Windows Service wrapper but Jetty 7 does not. To use Jetty in a production environment it is helpful to create a Windows service to run Jetty so that the service may run without manual intervention, like when the machine boots.

One way to accomplish this task is to make use of the (Jakarta) Commons Daemon project. In the example that follows I am using XWiki 3.2 that bundles Jetty 7. XWiki 3.2 is unzipped to C:\apps\xwiki32 and Jetty 7 resides within at C:\apps\xwiki32\jetty.

The daemon project essentially provides two Executables for the Windows operating system: prunmgr.exe and prunsrv.exe, collectively referred to as procrun. In essence, procrun provides the capability to run arbitrary Java main methods as a Windows service.

Determine Java arguments when starting Jetty

In my case I wanted to run a web application called XWiki deployed to Jetty, and have Jetty run under the Windows SYSTEM account whenever the machine boots up. I modified the start_xwiki.bat batch file to echo the Java command line before running java so I could capture all the arguments to Java. You'll need these later when configuring procrun.

In my case, the Java command line XWiki 3.2 wanted to use when running Java was:

Rename prunsrv.exe and prunmgr.exe to reflect your Jetty Service

Because Windows' Service manager (accessible from the Control Panel) identifies services by the executable filename used to start and stop the service, it is helpful to rename the two windows executables, prunsrv.exe and prunmgr.exe to reflect the fact this is a XWiki Jetty service.

Install Service

The following step configures procrun and installs the Windows Service. The JVM arguments below correspond to the arguments we captured from the original Java command line invocation from start_xwiki.bat.

Test procrun

The following will allow you to test your shiny new Windows Service by running it in the foreground so you can see any errors that may occur if the configuration isn't quite right.

XWikiJettyService.exe //TS/XWikiJettyServiceManager

Procrun logs, Stdout and Stderr logs for your service

These logs will be saved to the Jetty logs directory in the scenario described. The location of the procrun daemon logs are specified by the --LogPath=C:\apps\xwiki32\jetty\logs argument when you ran the XWikiJettyService.exe (prunsrv.exe) above.

The actual daemon stdout and stderr logs for Jetty are controlled by the -Djetty.logs=C:\apps\xwiki32\jetty\logs JVM argument.