Holger Hans Peter Freyther's bloghttp://smalltalk.gnu.org/blog/zecke
enUsing the new PackageLoader feature when creating unit tests for DBD-PostgreSQLhttp://smalltalk.gnu.org/blog/zecke/using-new-packageloader-feature-when-creating-unit-tests-dbd-postgresql
<p>In the previous post I briefly mentioned the new feature of the PackageLoader that went into the development version of GNU Smalltalk. Today I am going show a usage of it.</p>
<p>GNU Smalltalk has a database abstraction called DBI and multiple backends (MySQL, SQLite and PostgreSQL). In general I am using the SQLite backend to develop my application and depending on the users (e.g. do I need concurrent access) I will move to PostgreSQL. When moving my new application to PostgreSQL the import of data failed and this was due some issues with the conversion of Smalltalk types to PostgreSQL.</p>
<p></p>
<p>After some exploring to understand the issue I started to develop a unit test. The first thing I did was to create a test entry in the package description of DBD-PostgreSQL, fileIn a file and name a SUnit TestCase or TestSuite.</p>
<p><code><br />
<pre> &lt;test&gt;
&lt;sunit&gt;DBI.PostgreSQL.PostgresTestCase&lt;sunit&gt;
&lt;filein&gt;Tests.st&lt;/filein&gt;
&lt;/test&gt;</pre><br />
</code></p>
<p>The next thing is to start GNU Smalltalk and load the package and get the test. I cheated a bit and directly constructed a Kernel.DirPackage instance.</p>
<p><code><br />
$ gst<br />
st&gt; package := Kernel.DirPackage file: 'package.xml'<br />
st&gt; package fileIn.<br />
st&gt; test := package test.<br />
st&gt; test fileIn.<br />
</code></p>
<p>The above code has loaded the main package (and dependencies) and the test package as well. I went ahead and manually invoked the tests I have. This can be done by sending the &gt;&gt;#buildSuite message to my TestCase and then calling run on the result.</p>
<p><code><br />
st&gt; DBI.PostgreSQL.PostgresTestCase buildSuite run<br />
0 run, 0 passes<br />
</code></p>
<p>Now I can start to write the actual testcases. I decided to write one test per datatype. I created a &gt;&gt;#setUp selector that opens the database connection and creates a table that contains every built-in type of PostgreSQL as column and a &gt;&gt;#tearDown to close the database connection. Then I decided to write one test per datatype and started with the ones I knew that were broken. I could incrementally create tests and type the following commands to execute them.</p>
<p><code><br />
st&gt; test fileIn. DBI.PostgreSQL.PostgresTestCase buildSuite run<br />
</code></p>
<p>As expected my first tests were failing. I decided to use the open classes feature of Smalltalk and put the fixes to the tests into the Tests.st file and re-executed the above and my test started to pass, then I wrote another test, executed the suite, created a fix, re-loaded and ran the tests. This has worked nicely but then there was a fix that manipulated a lookup table owned by a singleton. This means that my changes to the code were not picked up as the instance was already initialized. There are multiple ways to overcome this. I could have called the initialize function of the singleton again, I could have manipulated the lookup table inside the singleton or I could have re-set the singleton. I decided to do the later using the reflection facilities of Smalltalk. I put a nil into the instanceVariable called uniqueInstance of the PGFieldConverter class.</p>
<p><code><br />
st&gt; DBI.PostgreSQL.PGFieldConverter instVarNamed: #uniqueInstance put: nil<br />
</code></p>
<p>After re-executing the testsuite the singleton was re-created and the next testcase was fixed. The only thing left to do was to move the fixes from the Tests.st to the right place in the original files and run make check on the entire codebase to check that everything works as expected.</p>http://smalltalk.gnu.org/blog/zecke/using-new-packageloader-feature-when-creating-unit-tests-dbd-postgresql#commentspackageloaderpostgressunitSat, 18 May 2013 01:21:23 -0700Holger Hans Peter Freyther761 at http://smalltalk.gnu.orgPackageLoader>>#loadPackageFromFile:http://smalltalk.gnu.org/blog/zecke/packageloader-loadpackagefromfile
<p>GNU Smalltalk has the concept of packages for a long time. By default the gst-package application will read an XML file and then create a ZIP archive. This package format is called a Star archive in GNU Smalltalk.</p>
<p>When developing it is easier to just load the package from the filesystem and this is why we now have the above PackageLoader&gt;&gt;#loadPackageFromFile:.</p>http://smalltalk.gnu.org/blog/zecke/packageloader-loadpackagefromfile#commentsquickWed, 15 May 2013 10:47:59 -0700Holger Hans Peter Freyther760 at http://smalltalk.gnu.orgUsing gst-profile to profile Iliadhttp://smalltalk.gnu.org/blog/zecke/using-gst-profile-profile-iliad
<p>I am using Iliad for the configuration of my GSM Basestation (another blog entry will follow) and the response time has not been that great when showing a lightbox (e.g. the one from todolist example). I wanted to see where Iliad is spending the time and potentially improve the performance of either GNU Smalltalk or Iliad.</p>
<p></p>
<p>The GNU Smalltalk interpreter is keeping a counter on the executed bytecodes and is exposing this profiling information to the Profiler package. This package will create a file that can be opened with the excellent <strong>kcachegrind</strong> viewer. This aligns nicely with the Unix philosophy of having one tool per task.</p>
<p>The <strong>gst-profile</strong> application will load the default image, run a script and then write the profile information to a file. This means the first thing I need to do is to load the <em>Iliad</em> package and create a snapshot of the image. Then I can use <strong>gst-profile</strong> to evaluate a function, render webpages and exit gst. This will create a file called gst-profile.PID (where PID is the value of the GST process) and this file can be opened using kcachegrind.</p>
<p><pre>
# Load iliad and save the default image
$ gst
st&gt; PackageLoader fileInPackage: 'Iliad'.
st&gt; ObjectMemory snapshot
</pre></p>
<p><pre>
# Start Iliad and run until a line feed.
$ gst-profile -e 'Iliad.SwazooIliad startOn: 8082. stdin next'
Press enter/return to exit
$ ls gst-profile.*
$ kcachegrind gst-profile.*
</pre></p>http://smalltalk.gnu.org/blog/zecke/using-gst-profile-profile-iliad#commentsarmembeddediliadSun, 05 May 2013 03:01:20 -0700Holger Hans Peter Freyther755 at http://smalltalk.gnu.orgMaking my Osmocom Smalltalk packages available for Pharohttp://smalltalk.gnu.org/blog/zecke/making-my-osmocom-smalltalk-packages-available-pharo
<p>In the last two years I have developed some Smalltalk packages that help me with (mobile) communication related tasks and I began making them available for Pharo as well. I'm using the gst-convert utility to do the conversion and the goal is to have this work as automatically as possible.</p>
<p><ul><br />
<li><strong>OsmoLogging</strong> is another logging framework modeled after the framework we developed for C. Besides having multiple backends (Transcript, syslog) and log levels there are some unique features.<br /><br />
One can attach log context to the current process. E.g. the data that is parsed, or the end-user that initiated this process. This can be used in the exception logger to provide more context, dump the raw message that might have caused the exception. But it can also be used by log filters, e.g. only show log messages that are related to a specific user.<br /><br />
The other feature is the concept of log categories. Each log message is tagged with a category and the category and backend can be decide if that category is enabled and what the default log level should be.</li><br />
<li><strong>OsmoCore</strong> is a collection of utility classes. Currently there is a class to schedule a callback in a couple of seconds. This is done to avoid to have thousands of processes that do <code>[(Delay forSeconds: 1) wait. self cancel] fork</code>. The other class allows me to dispatch a block from a central context. This is done to avoid dealing with deadlocks. I have opted for this solution as having lock hierarchies between different stacks (MGCP, SIP) is very difficult. I am considering building a mailbox like system for these stacks. That means each subsystem would have its own dispatcher.</li><br />
<li><strong>OsmoNetwork</strong> is a collection of network related utility classes. They help me working with sockets and provide some encoding/decoding classes for SCCP, ISUP, M2UA. This code is the oldest and some parts could make better use of Streams.</li><br />
<li><strong>OsmoSIP</strong> contains most of the SIP grammar described using PetitParser and code to create SIP Dialogs, deal with re-transmission.</li><br />
<li><strong>OsmoMGCP</strong> contains most of the MGCP grammar described using PetitParser and code to create MGCP Transactions, deal with the re-transmission.</li><br />
<li><strong>OsmoGSM</strong> contains classes related to the GSM specification 04.08 and 08.08. I can easily create and parse the wire messages for this specification.</li><br />
</ul></p>
<p>This code is complete enough to run mobile communication applications 24/7. It is my first work in Smalltalk though, this means there are various things that can be done better and cleaning up and re-factoring is a continuous process. The OsmoCore and OsmoLogging package is now available for Pharo and I am in the process of porting the OsmoNetwork package.</p>http://smalltalk.gnu.org/blog/zecke/making-my-osmocom-smalltalk-packages-available-pharo#commentsosmocompharoSat, 23 Feb 2013 02:30:32 -0700Holger Hans Peter Freyther704 at http://smalltalk.gnu.orgGNU Smalltalk deployment with images and image resumptionhttp://smalltalk.gnu.org/blog/zecke/gnu-smalltalk-deployment-images-and-image-resumption
<p>Once up on a time I was sitting in a cold hall at the Barcelona exhibition ground, a power outage has taken down several DVB-H platforms (racks consisting of servers, streamers, RF equipment...) and once power was restored red LEDs were blinking, systems not coming up automatically, hordes of engineers trying to boot the right kernel, trying to remember the multicast routes they had typed in by hand, chaos, hectic. It was interesting to witness that as we could lay back, continue with our work, as our platform was configured to come up automatically and it did.</p>
<p>One has to assume that stuff breaks, software, disk, RAM, CPU, power supply, anything. The only answer to that is be prepared, build your software to deal with failure (or nicely try to restart), make sure all systems start automatically, have backups, have a plan.</p>
<p>For my GSM (telephony) in Smalltalk components I didn't do that yet, mostly because my code was not very mature, I was still building up the trust relationship and so my deployment consisted of manually running GNU screen, then GNU Smalltalk, manually loading my code, starting things up. It was about time to correct that.</p>
<p>The tool of choice is the gst-remote application, it will run a GNU Smalltalk image, it can daemonize and it will listen for input from other systems. While moving to this deployment my fellow Smalltalkers were kind enough to fix some bugs on the way. There was an issue of gst-remote exiting when saving an image, the Delay class not functioning properly on resume, the wrong FileDescriptors being closed on image resumption. It is very nice to get help that fast!</p>
<p>My deployment now consists of building GNU Smalltalk packages, having a Smalltalk script to load the package and call start method of that package followed by taking a snapshot and exiting. I can then resume the image and if the software is prepared to deal with image resumption it will work.</p>
<p><pre>Eval [
| name image |
name := Smalltalk arguments first.
image := Smalltalk arguments second.
(PackageLoader packageAt: name)
fileIn;
start.
ObjectMemory
snapshot: image;
quit.
]</pre></p>http://smalltalk.gnu.org/blog/zecke/gnu-smalltalk-deployment-images-and-image-resumption#commentsSat, 24 Sep 2011 05:36:58 -0700Holger Hans Peter Freyther612 at http://smalltalk.gnu.orgOsmoST SIPhttp://smalltalk.gnu.org/blog/zecke/osmost-sip
<h3>Intro</h3>
<p>The last couple of days, to some degree weeks I was implementing a SIP stack for GNU Smalltalk to be used in my Smalltalk GSM project. The first time I encountered SIP was around 2001 when we were excited to run a SIP Phone on our Linux powered iPAQs (Linphone on Opie).</p>
<p>For Smalltalk I started with looking at SipStack by Frank Shearar (who was very responsive to questions regarding his code and SIP in general). The main problem was that his stack was incomplete and as usual it is difficult to pick things up without knowing SIP and not knowing where the code was heading.<br />
</p>
<h3>Grammar</h3>
<p>I began with mapping the BNF SIP Grammar to rules for the PetitParser parsing framework. I have shortened some rules (e.g. because the generic rule is a catch-all one and I don't need the specialized form yet).</p>
<h3>Complexity</h3>
<p>The next big task was to deal with the concepts of a SIP Dialog, a SIP Session, a SIP Transaction and getting the relationship of these entities right. E.g. to place a call one creates an unconfirmed Dialog, prepares the INVITE Transaction and passes it to the transport layer. During this transaction one can get a reply that leads to a confirmation of the dialog. In fact one can end up with multiple confirmed dialogs (called forking, the target phones start ringing in multiple places and can be picked up multiple times as well). To make matters worse some answers need to be sent sent within the same transaction, some open a new one. My code seems to work now but there are some things I don't do properly (e.g. routing). I have documented this and if I have a need for those I will address them.</p>
<h3>Processes</h3>
<p>In the last couple of years I was mostly dealing with a single select IO model, coming to Smalltalk makes me use Processes to read and write from sockets. My basic model is to have a RX Process to use Socket&gt;&gt;#next and a TX process that will do SharedQueue&gt;&gt;#next and then Socket&gt;&gt;#nextPutAll. But with having multiple processes one is required to do proper locking, e.g. have a simple locking hierarchy (first pick the lock of the CallAgent, then the lock of a Dialog/Session). This all works nicely until I reached the point that I have two sub systems talking to each other.</p>
<p>For setting up a call I might have one leg coming from GSM and use the MGCP Protocol, the other leg might be SIP. To hang-up I will need to inform the other leg of the call about it, but this means I will need to have a locking hierarchy that first takes all locks of the left leg of a call, then all locks of the right one which in turns means me I need to dispatch messages without any locks being held. Now besides having RX from maybe a couple of different Processes, I also deal with timeouts that will come from different Processes as well.</p>
<h3>Central dispatch</h3>
<p>To put an end to this (and the complications in the code) I created a simple queue that will BlockClosure&gt;&gt;#value a block from a central dispatch process, this way I have serialized everything and can mostly ignore locking. I hope that Smalltalk VMs will have a nice M:N Process model in the future and that my code will evolve into having different dispatchers running on these N native processes.</p>
<h3>Using the code</h3>
<p>st&gt; PackageLoader fileInPackage: #OsmoSIP.<br />
st&gt; transport :=SIPUdpTransport startOn: '172.16.254.55' port: 5061.<br />
st&gt; useragent := SIPUserAgent createOn: transport.<br />
st&gt; transport start. "Starts the RX/TX process"</p>
<p>st&gt; call := SIPCall fromUser: 'sip:1000@on-waves.com'<br />
host: '172.16.1.72' port: 5060 to: 'sip:9198@172.16.1.72' on: useragent.<br />
st&gt; call createCall: 'SomeSDP file'.<br />
st&gt; call cancel. "E.g. before it is connected"<br />
st&gt; call hangup. "E.g. when the call is connected."<br />
st&gt; call terminate. "End the call somehow.."</p>http://smalltalk.gnu.org/blog/zecke/osmost-sip#commentsWed, 06 Jul 2011 12:52:21 -0700Holger Hans Peter Freyther599 at http://smalltalk.gnu.orgBlock Closure and Timeoutshttp://smalltalk.gnu.org/blog/zecke/block-closure-and-timeouts
<p>For the SS7/GSM work I am doing I have to implement dialogues with other systems and by default the IO in a lightweight process in Smalltalk is blocking (built on top of SIGIO). My problem is that a remote system could make me wait for a response forever. I am not an experienced Smalltalker yet but the existing options didn't look good enough (which might be me being unexperienced). I looked into what is required to add a #timeout:do: to the BlockClosure. The idea is to run the dialogue in a very straight forward way and guard the whole operation with a Timeout.</p>
<p><code><br />
[<br />
self sendSomething.<br />
self waitForResponseOfKind: ABC.<br />
self someMoreStuff.<br />
self waitForMoe.<br />
self askOtherSystem.<br />
self conclude.<br />
] timeout: 10 do: ['The operation timed out... cleanup']<br />
</code></p>
<p>It turned out to be quite easy to do in GNU Smalltalk. The BlockClosure extension creates a new Process which uses the Delay class to wait for the delay. The new delay process holds on to the BlockClosure and has a Semaphore. If the normal evaluation exists the Semaphore signals in case the timeout occurs before the original Process queues an interrupt that sends a TimeoutNotification.</p>
<p>Is there a better way to achieve such patterns?</p>http://smalltalk.gnu.org/blog/zecke/block-closure-and-timeouts#commentsGSMussdMon, 04 Apr 2011 14:13:20 -0700Holger Hans Peter Freyther581 at http://smalltalk.gnu.orgLooking for a nice ASN.1 stackhttp://smalltalk.gnu.org/blog/zecke/looking-nice-asn-1-stack
<p>I am working on GSM/SS7 protocol implementation on and off for a bit. The current modules can be found in <a href="http://cgit.osmocom.org/cgit/smalltalk" rel="nofollow">these</a> git repositories.</p>
<p>Right now I am working on parsing and creating ISDN User Part (ISUP) messages and the general question of how to express on-wire messages composed from mandantory, variable and optional fields (class based vs. instance based). For now I am using a class based approach as this allows me to have field decoding/encoding inside these classes but I am not sure if this is the best way and will keep on exploring it.</p>
<p>After having finished the ISUP part I will move to the Transaction Capabilities Application Part (TCAP), Mobile Application Part (MAP) and Camel Application Part (CAP). All these protocols specify the messages in ASN.1 and use the Basic Encoding Rules (BER). The biggest obstacle is the lack of a ASN.1 stack. The LDAP code on SqueakSource has some basic (too basic) ASN.1 support, I have seen that VisualWorks has ASN.1 support as well but the code does not seem to be under a OpenSource license.</p>
<p>Does anyone know a freely available ASN.1 implementation that supports DER encoding and BER decoding? Any hints would be welcome.</p>http://smalltalk.gnu.org/blog/zecke/looking-nice-asn-1-stack#commentsasnGSMTue, 22 Mar 2011 22:33:00 -0700Holger Hans Peter Freyther575 at http://smalltalk.gnu.orgGSM in Smalltalk - a GSM Toolkithttp://smalltalk.gnu.org/blog/zecke/gsm-smalltalk-gsm-toolkit
<p>I started to play with smalltalk somewhere in February, more specific with the GNU Smalltalk implementation. Like it is with any new language and class library it takes a while to get productive and it took me until somewhere the last month where I finally started to do GSM handling in Smalltalk and thanks to <a href="http://laforge.gnumonks.org" rel="nofollow">laf0rge</a> the code is now in a public repository and hosted along the other Osmocom projects. You can see all the subprojects over <a href="http://cgit.osmocom.org/cgit/smalltalk/" rel="nofollow">here</a>.</p>
<p>So far I can create and parse the ipaccess IPA protocol, SCCP, BSSAP, BSSMAP and messages for Call Control and Mobility Management of GSM48. I also have a small Web-UI to connect to a MSC and then do a Location Updating Request and a Mobile Originated Call for a configurable IMSI (you will need to know your AuKey for that). It is very easy to parse and create the whole stack of messages, adding new GSM48 messages is easy as well (i still consider writing a program to get this directly from the GSM48 PDF).</p>
<p>This code is probably very well suited to do load tests of a MSC, e.g. find out how many LUs it can do a second, also to play easily with various quirks with it. Thanks to the nature of smalltalk one can easily change the creation of messages at runtime without needing to reconnect to the MSC. The next thing would probably to use it for fuzzing a MSC...</p>
<p>One note of caution, it is my first Smalltalk project and there are probably plenty of things I do wrong, I already know some of them and will work on refactoring it to a better structure. The next stop is nice ASN1 support, I will base it on the LDAP work.</p>
<p>BTW: This is a repost from my original blog.</p>http://smalltalk.gnu.org/blog/zecke/gsm-smalltalk-gsm-toolkit#commentsGSMTue, 14 Dec 2010 21:32:32 -0700Holger Hans Peter Freyther547 at http://smalltalk.gnu.orgPetitParser for GNU Smalltalkhttp://smalltalk.gnu.org/blog/zecke/petitparser-gnu-smalltalk
<p>On and off the last couple of weeks I have ported PetitParser to GNU Smalltalk. I am still a Smalltalk newbie and it was a nice learning experience and it helped me to improve the way I code on GNU Smalltalk and to learn more about GNU Smalltalk and ANSI Smalltalk.</p>
<p>To load the PetitParser.star one can type:<br />
<code>gst-package http://smalltalk.gnu.org/project/petitparser/package.xml</code></p>
<p>To load the PetitParser.star into the image do:<br />
<code>PackageLoader fileInPackage: 'PetitParser'</code></p>
<p>There are some differences between the real PetitParser and this port. GNU Smalltalk does not support binary selectors that have more than two charachters. This means that <code>==&gt;</code> and <code>&gt;=&gt;</code> had to be mapped to something else. I have picked <code>=&gt;</code> and <code>&gt;&lt;</code> for now.<br />
</p>
<p>In GNU Smalltalk we are using gst-convert to convert from Squeak syntax to the GNU Smalltalk syntax (it can do more than that as well). The first obstacle was that our Parser could not handle the PetitParser code, it was using <code>classSide</code> which STClassLoader has not seen before and then failed, Paolo was nice enough to come up with a patch for that, the second issue were binary selectors of length &gt; 2. I had the pleasure to read the ANSI Smalltalk draft, come to the conclusion that it is legal, come up with a patch to the VM anf the Parser written in Smalltalk. In the end we have only merged the code to understand these binary selectors when they come from Squeak.</p>
<p>The next step for me was to get the PetitTests passing on GNU Smalltalk. I ran into the problem that on Pharo <code>#abc = 'abc'</code> is true, but not on GNU Smalltalk, I worked on VisualGST to have a nicer to debug the tests from the SUnit GUI and that has greatly increased my productivty.</p>
<p>I have learned a lot during this excercise, and with some more work on VisualGST it can be a really nice Smalltalk IDE. I am now trying to implement the Media Gateway Control Protocol (MGCP) Grammar with PetitParser and will see how this goes.</p>http://smalltalk.gnu.org/blog/zecke/petitparser-gnu-smalltalk#commentspetitparserSun, 29 Aug 2010 21:37:35 -0700Holger Hans Peter Freyther523 at http://smalltalk.gnu.org