A quest for software excellence...

As a visually oriented developer, I remain command-line challenged. And yet there are some things that seem to be more productive in command-line form, and git seems to be one of them. I’ve not tried every GUI incarnation of a git tool, so I would be happy to be proven wrong about that. Until then, here are my notes on setting up my local machine with Git and PowerShell.

Thanks to a friend of mine, over the past few weeks the idea of zero knowledge authentication, which has been around for a long time but to which I’d never paid any attention, has been percolating in my head. Finally on Thursday (Thanksgiving holiday here in the U.S.), I dove in and added it to ServiceWire.

Now you can secure your ServiceWire RPC calls without PKI or exchanging keys or exposing a password across the wire. It validates on the server that the client knows the secret key (password) and on the client that the server knows the secret key. It creates a shared encryption key that is used for that one connection session and is never passed across the wire, allowing all subsequent communications between client and server to be encrypted with strong and fast symmetric encryption, denying any man in the middle from access to the data or even the service definition.

But this post is not about the entire zero knowledge addition to ServiceWire. Instead I want to share one part of that addition which you may use to create a very large random number. In this case, 512 bytes (that’s 4096 bits for those of you who can only count to 1). If you want a larger random number, you’ll need a larger safe prime from a Sophie Germain prime. In ServiceWire, I only include the 512 byte safe prime. There are a number of sources for larger safe primes freely available on the web.

This is not an algorithm entirely of my invention. It is based on several different algorithms I studied while investigating zero knowledge secure remote password protocols. I did take some liberties by adding a randomly selected smaller safe prime as the exponent in the algorithm.

I should note that in ServiceWire a new instance of ZkProtocol is used for each connection, so the Random object should provide a healthy random seed to the algoritm given the clock's milliseconds seed on construction. If you find this useful, I would love to hear from you.

I am pleased to see this milestone of 10,000 downloads in the short history of ServiceMq and its underlying communication library ServiceWire, a faster and simpler alternative to WCF .NET to .NET RPC. And the source code for all three can be found on GitHub here.

Over the past few weeks both libraries have been improved.

ServiceMq improvements include:

Options for the persistence of messages asynchronously to improve overall throughput when message traffic is high

ReceiveBulk and AcceptBulk methods were introduced

Message caching was refactored to improve performance and limit memory use in scenarios where large numbers of messages are sent and must wait for a destination to become available or received and must wait to be consumed

The FastFile class was refactored to support IDisposable and now dedicates a single thread each to asynchronous delete, append and write operations

Upgraded to ServiceWire 1.6.3

ServiceWire has had two minor but important bugs fixed:

Code was refactored to properly dispose of resources when a connection failure occurs.

Previously if the host was not hosting the same assembly version of the interface being used, the connection would hang. This scenario now properly throws an identifiable exception on the client and disposes of the underlying socket or named pipe stream.

Real World Use

In the last month or so, I have had the opportunity to use both of these libraries extensively at work. All of the recent improvements are a direct or indirect result of that real world use. Without disclosing work related details, I believe it is safe to say that these libraries are moving hundreds of messages per second and in some cases 30GB of data between two machines in around three minutes across perhaps 300 RPC method invocations. Some careful usage has been required given our particular use cases in order to reduce connection contention from many thousands of message writer threads across a pool of servers all talking to a single target server. I’ve no doubt that a little fine tuning on the usage side may be required, but overall I’m very happy with the results.

I hope you enjoy these libraries and please contact me if you find any problems with them or need additional functionality. Better yet, jump onto GitHub and submit a pull request of your own. I am happy to evaluate and accept well thought out requests that are in line with my vision for keeping these libraries lightweight and easy to use.

One other note

I recently published ServiceMock, a tiny experimental mocking library that has surprisingly been downloaded over 500 times. If you’re one of the crazy ones, I’d love to hear from you and what you think of it.

One of the most misunderstood and misrepresented documents in the history of software development is the Agile Manifesto. This may be due to many of its readers overlooking the phrase “there is value in the items on the right.” Most seem to focus on the items on the left only. Here’s the text that Cunningham, Fowler, Martin and other giants in the field created:

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Note that I have emphasized the items on the right. These do indeed have value but so many advocates of Agile deliberately ignore and even exclude these from their software development process and organization. Some have advocated the elimination of architecture and design entirely, leaving these open to gradual discovery through the iterative process driven by use cases and user stories and backlog tasks.

Recently I have read a number of discussions, blog posts and articles on the question of Agile and architecture. The comments and discussion around the topic have been interesting. Perhaps this is due to the notion that developing software is only about writing the code. The general theme of these sources is that architecture (and design) are at odds with Agile. This is one of the great fallacies of our time.

Architecture and Design are Software Development Artifacts

Teams and organizations who skip architecture and design will sooner or later find themselves off track and repeating work unnecessarily. Such waste is not entirely preventable, but this does not mean we should not try. Teams that incorporate these activities into their iterations, regularly revisiting architectural questions such as non-functional requirements and component design, will find that they are better able to stay on course.

Organizations that have multiple teams will find greater stability in moving forward when guided by a centralized architecture team, comprised of architects or leads who are dedicated to and work within the organizations development teams. The architecture team works in an Agile fashion, with its own backlog and its own products including working prototypes, cross cutting standardized components, documentation of the architecture and designs, and work items to be placed on the backlogs of development teams.

In this way, development teams have dependencies on the architecture team and can make requests for additional guidance or improvements or extensions to shared, standardized libraries for which the architecture team is responsible. These requests keep the backlog of the architecture team charged with work throughout the software development lifecycle.

In addition to formal activities to improve architecture and design across the organization, the architecture team should regularly interact with development teams to include (but not limited to) the following:

practice improvement activities—e.g. SOLID principles

technology deep dives—e.g. digging deeper into .NET

technical solutions brainstorming sessions—solving the hard problems

technical debt evaluation and pay-down planning

code reviews and walkthroughs—one on one and as a team

presenting and sharing solutions and ideas from other development teams

exploring and evaluating new technologies and tools

Like testing and coding, architecture and design are a part of the whole of software development. These activities are perfectly suited to Agile development practices, including SCRUM. And when all of these aspects of delivering quality software are taken into account and incorporated into your Agile process, your chances of success are greatly improved.

----

P.S. And if you add to all this a great dev ops team to support your efforts with automated build and deploy systems, your life will be that much easier and your chances of success are automatically improved.

I know. There are some really great mocking libraries. The one I’ve used the most is Moq 4. While I’ve not been a regular user of mock libraries. I am fascinated with their usefulness and I’ve recently been thinking about how I might utilize the ServiceWire dynamic proxy to create a simple and easily extended mock library. After a few hours of work this morning, the first experimental of ServiceMock comes to life.

This is not a serious attempt to replace Moq or any other mocking library. It is for the most part a way to demonstrate how to use the dynamic proxy of ServiceWire to do something more than interception or remote procedure call (RPC). It is entirely experimental, but you can get it via NuGet as well.

To create a library that takes advantage of the ServiceWire dynamic proxy, you need a factory (Mock), a channel (MockChannel) that the dynamic proxy will invoke, a channel constructor (MockDefinition) parameter class, and finally an function for invoke and exception handling should the invoke throw (MockActions). And of course, you can supply your own customized function and assign it to the MockActions instance.

The heart of the extensibility is the ability to inject your own “invoke” function via the instance of the MockActions class in the MockDefinition constructor parameter.

Of course, you might want to log the calls, aggregate counts per methodName or whatever you wish. I hope you find this useful, but I hope more that you will build your dynamic proxy wrapper for your own cool purposes.

Over the past several days, I’ve been working on a day-job project that may be using ServiceMq which uses ServiceWire under the covers. I say “may” because it depends on whether the prototype proves to be sufficiently reliable and efficient. The prototype is really more of an extended integration test with the following requirements:

Blocking Send method that throws if sending fails.

Tries the primary destination first.

Alternative destinations are tried successively until the message is successfully sent.

Control over send connection timeout failure to enable “fail fast.”

Standard caching receive.

This is because the senders are transient and may not be restarted should they fail. The sender’s also need immediate feedback because the action is part of a transaction involving other operations.

The first order of business was to add the Flash class to ServiceMq. This evolved to become the Flasher class which implements IDisposable in order to take advantage of client connection pooling using the updated PooledDictionary class in ServiceWire (more on this later). The Flasher’s Send method allows you to send a message to a primary destination with zero to many alternate destinations.

In order to support a more robust client side connection timeout, a significant improvement was made to ServiceWire. The TcpEndPoint class was introduced and an overloaded constructor was added to TcpChannel which allows you to specify a connection timeout value when creating an instance of TcpClient<T> (see below). This involved use of the Socket class’s ConnectAsync method with a SockeAsyncEventArgs object.

The final problem to solve was TCP/IP port exhaustion. My original implementation had the sending client being created and taken down with each call to the Send method. The construction overhead is minimal but the connect time and the port exhaustion problem quickly becomes a problem where there are potentially many threads sending messages from the same client machine.

To solve the problem, I used a connection pooling strategy that involved making the PooledDictionary<TKey, TValue> implement the IDisposable interface to allow for easy disposal of pooled client objects. The Send method then uses one of two client pools, depending on whether it is a local Named Pipes connection of a TCP connection.

It remains to be seen whether these changes will result in a sufficiently robust message passing and queuing system to allow it to be used on my day job project. More testing and prototyping is required. There are alternatives to which I can fall back, but none of them are trivial and all of them are less desirable. Given my personal bias, I must take extra care to scrutinize this possible solution and abandon it should it prove to be insufficient.

I have added a Broadcast method to distribute in guaranteed order a single message to multiple destinations. If one of those destinations is down, message delivery will resume when it becomes available again. If the message cannot be delivered in 24 hours, the the message will get logged to the failed log.

The new version of this library also creates send, read and failed log files by minute to assure the files are not too large when there is a very high number of messages flowing. It will also remove the sent and read files after 48 hours, a value you can change in the constructor.

Get the NuGet package here. Or check out the code on GitHub. Here’s some test code that demonstrates how these features work. (Update: the 1.2.2 package just published adds CountOutbound and CountInbound properties to allow you determine if your queues are getting backed up and more receiving or less sending should occur based on the limits you choose.)

Today I had occasion to use in a work project the PriorityQueue<T> and Heap code that I had written and blogged about recently. In doing so, I discovered a couple of bugs and fixed them and added tests to cover the issue that was uncovered.

Here’s what changed in the PriorityQueue<T>. You can follow the link above to see the change to Heap.