Blog Archive

21 Jun 2013

Enterprise level software can provide marvelous benefits with process automation; yet, how many tales can you recount of the company that has implemented Enterprise level procurement / ERP / BPM, etc and it was a death march. Pure and simple it was a nightmare, accounting couldn’t close the books for over 5 weeks, and this continued for almost a year. True story.

This post is not to be seen as refutation of automation, business process management, statistics, and economics. People, organizations, and companies need direction, require a director to set the tone, the pace and cadence for reaching goals. But there is a danger that we treat processes like science with the belief that when business behaves in a more rational fashion, things will operate like physics or mathematics. While it is true that applying scientific approach and principals to business can help, at its essence, business is not science. Here’s another analogy – while you should apply science to the field of medicine, in the end, the body and mind heal in manners that are not scientific. Ask any doctor about misdiagnosis and miracles. If he or she hasn’t been humbled by their lack of understanding of healing, change doctors!

We have to careful with model-abstraction worship. Let me tell you, the model is not the process – it’s a representation. Look at the two intrepid skiing figures above. Do you think they are thinking of the mechanics or the physics of what they are experiencing? Torque on the knees? Now put yourself in their instructors or trainers shoes and ask “how would I really go about communicating what needs to be learned to survive this type of run?”

An organization is comprised of the interactions of people, and these series of steps, while seeming out of control, create the cadence of your business. At the level where the work is completed, if you were to ask someone to describe what they do, it will start with a list items, not a diagram like the one above. At the level where the work is done,the people pushing the buttons, experience a different reality than what management experiences. If your goal is to assist those people, then your first step is not to alienate them by diagramming their busy day with pseudo scientific terms and weird charts. Remember, the director who can’t get her actors to relate to the story makes an awful film. You have to relate to others on a fundamental level, and for the most part in business, people don’t relate to this diagram. A picture is worth a thousand words, but for those whose inbox is flooded with requests, a diagram is not a picture, it a a thousand confusing questions. “Why is the box blue? Can I have a diamond instead?”

“Ok”, you ask, “what do we do instead?” Well, as luck would have, there is a good way to implement process, get approvals, and have consistency without over orchestrating business process and creating confusion. The approach is to view the decisions that people make as gateways between important steps. This is best illustrated with an example, so let’s model a bonus approval process for a sales organization.

We have a sales team who is closing deals weekly, and for each deal a manager and vice president will need to weigh in on the merits of the award. Fairly standard in most businesses, and in many cases this is managed with email. The bonus manager is the fulcrum of this process, as it he who currently reminds the sales agents that they have not supplied the correct supporting materials, as well as gently nudging the vice president of sales that she needs to get into the system and complete approvals so that the request can be released to payroll before the Wednesday paycheck run. Your role, dear reader, is to get the steps of this process that are inside the bonus managers head and create something that they will engage in. To create a fan of the new process, it’s better that the process come from the bonus manager – they have to do it, so if they create it, they own it!

Remember the diatribe against diagrams? You can’t ask the bonus manager to diagram their process, as they’re too busy. But have them list the steps – yes a simple list. It may be a very limited set of items. You get this:

Fill Out Bonus Request

Bonus Manager Reviews

Vice President Reviews

Payroll Issues payment

You can prod the bonus manager further, and get them to categorize the end states of this process. We’ll add Bonus Denied and Bonus Paid, as this will represent all the stops in the process. Now the list looks like this:

Fill Out Bonus Request

Bonus Manager Reviews

Vice President Reviews

Payroll Issues payment

Bonus Denied

Bonus Paid

The next thing to do is ask “What decisions get you from step to step? For example, what do you do when work is incomplete from the Sales Agent?” The answer – “Today? I email them and request more info”. Lets model this action as a phrase starting with a verb:

Request More Info

Now you start a helpful chart that looks like this:

Step

Decision

Next Step

Bonus Manager Review

Request More Info

Fill Out Bonus Request

Simple. It’s in English, and there is direct understanding of what happens. What’s more, these are the phrases, thoughts, and language used by the key person of the process. Their “vocab” is the context of how they think, and it’s not a stretch for understand what is occurring, no threat from the oppressive science of process modeling. You could even do this at lunch! Here is what the end product will look like:

Step

Decision

Next Step

Fill Out Bonus Request

Complete

Bonus Manager Review

Bonus Manager Review

Request More Info

Fill Out Bonus Request

Bonus Manager Review

Approve

VP Review

VP Review

Deny

Bonus Denied

VP Review

Approve

Payroll Issues Payment

Payroll Issues Payment

Complete

Bonus Paid

There’s more good news. ActiveEngine Software has a product that will take this list and create a workflow. Here is a demo video. Thanks for reading.

17 Apr 2013

Of late, Sensei needs to keep a clear head. That has meant learning to segment ideas and really, really, really focus on streamlined features. This is hard. Not because Sensei has a plethora of great ideas. That would be a nice problem to have. Many times in software development you end up this guy:

This is the state where you have things you want to accomplish, yet even when you pair things down to the “essential”, other essential, critical factors must be taken into consideration before you create a mess. This is life calling, and that string which suspends that giant sword that you noticed hovering over your head is about to snap. There is a good chance that you need more discipline, better execution tactics, far better honed chops, you name the metaphor. Sensei has been at this game for over 22 years, and still the speed that thought takes to become reality is way too slow.

With great sarcasm you can remind your self that some of the best work lays ahead, but the reality is that you still need to fight to be fluent, you have to claw your way to a Zen state of mind / no-mind. So chose, the art of bushido or the art of BS. Or maybe work smarter and enjoy life.

Before Sensei leaves you, ponder this: does “being done” mean that you’ve dropped off a product but have to get on the phone in order to make changes, and maybe now that you are struggling why couldn’t you figure out to take time when it was more critical to be fluent with your productivity?

A central theme for 2010 has been fluency, or the continual practice of certain methods to such a degree that your performance improves and you produce increasingly polished, effective solutions. For software development this has meant tools to save time and increase quality. It also means keeping an eye toward making the users of your solutions more efficient as well. In the spirit of “fluent solutions”, Sensei will end the year with a post that examines how to create a data paging solution for the jQuery data grid plug-in DataTables.Net.

DataTables can turn a HTML table into a fully functional data grid like the one offered by Telerik. This plug-in offers client side sorting, filtering/ search, as well as support for server-side processing processing of data. It is an extremely feature rich tool created by Allan Jardine, and is itself worthy of a series of posts. For this post on data paging Sensei recommends that you read through these examples to get an idea of what the data paging service needs to achieve.

Let’s get started with the goals we need to achieve when providing server-side data paging support:

Send data to client in the multiples or “chunks” that the client requests, and respond when the size of the sets requested is changed by the user.

Re-order the data set if the user clicks on a column heading. Honor the data set size when returning the data.

Filter across all columns of data based on user input. Implement this as partial matches, and again, honor the data set size.

Remember this is about flexibility, so we have the additional goals of:

Create a solution that can be reused.

Provide a mechanism to accommodate any type of .Net class using generics.

Number of columns being displayed (useful for getting individual column search info)

string

sSearch

Global search field

boolean

bEscapeRegex

Global search is regex or not

boolean

bSortable_(int)

Indicator for if a column is flagged as sortable or not on the client-side

boolean

bSearchable_(int)

Indicator for if a column is flagged as searchable or not on the client-side

string

sSearch_(int)

Individual column filter

boolean

bEscapeRegex_(int)

Individual column filter is regex or not

int

iSortingCols

Number of columns to sort on

int

iSortCol_(int)

Column being sorted on (you will need to decode this number for your database)

string

sSortDir_(int)

Direction to be sorted – “desc” or “asc”. Note that the prefix for this variable is wrong in 1.5.x where iSortDir_(int) was used)

string

sEcho

Information for DataTables to use for rendering

Reply from the server

In reply to each request for information that DataTables makes to the server, it expects to get a well formed JSON object with the following parameters.

Type

Name

Info

int

iTotalRecords

Total records, before filtering (i.e. the total number of records in the database)

int

iTotalDisplayRecords

Total records, after filtering (i.e. the total number of records after filtering has been applied – not just the number of records being returned in this result set)

string

sEcho

An unaltered copy of sEcho sent from the client side. This parameter will change with each draw (it is basically a draw count) – so it is important that this is implemented. Note that it strongly recommended for security reasons that you ‘cast’ this parameter to an integer in order to prevent Cross Site Scripting (XSS) attacks.

string

sColumns

Optional – this is a string of column names, comma separated (used in combination with sName) which will allow DataTables to reorder data on the client-side if required for display

array array mixed

aaData

The data in a 2D array

The data sent back is in the following form depicted below. Note that aaData is merely an array of strings – there is no column information. This will present a challenge in that you will not be able to simply serialize a collection and pass back the results.

As you may be aware, if you wish to use ASP.Net web services to serialize JSON you must POST to the service and instruct it to interpret your parameters as JSON. DataTables will POST variables as value pairs and this won’t work for us when POSTing to a web service. We’ll have to translate the variables to a usable format. Luckily DataTables allows us to intervene with the following code, where we create a JSON string by serializing a structure called aoData:

Our web service can now de-serialize aoData and parse the appropriate parameters. This gives us important items such as how many records to display, what columns to sort on, and what search terms should be applied in a filter.

DataTablePager Class

DataTablePager.cs is the work horse of our solution. It will sort, filter and order our data, and as an extra, serialize the results in format required by aaData. Here’s the constructor:

The parameter jsonAOData is the JSON string that contains the variables iDisplayStart, iDisplayLength, etc. These will be parsed by the method PrepAOData. The parameter queryable is the collection of records that will be filtered and parsed into JSON format required by DataTables.

The method Filter() coordinates all of the work. It’s pretty simple what we want to do: filter our data based on each column containing the search term, sort the result, then pull out the number of records we need to include in the page, and finally convert the collection into the format DataTables understands.

That said, there is some trickery that goes on in order to make this happen because we are creating a solution to is going to work with any IQueryable to we supply. This means that the filtering and the sorting will need to be dynamic.

To make the filtering dynamic we will build expression trees that will convert each property to a string, convert the string to lower case, then execute a Contains method against the value of that property. The method GenericSearchFilter() called on line 16 accomplishes this with the following lines of code:

We get an array of Expressions that when executed will tell us if the value matches our search term. What we want is to include the item if ANY of the properties is a match, so this means we have to use and OR for all of the properties. That can be accomplished with:

So with what is listed above we would be able to match all properties with against a single search term. Pretty cool. But DataTables raises the bar even higher. If you were to go to the samples page and filter using multiple partial words you would find that you could perform some very effective searches with phrases like “new chic”. This would select all records that had properties containing “new” OR “chic”. Imagine the scenario where your user wants to finds all cities “New York” or “Chicago”. We’ve all been there where we have a grid and can only search for one term, or worse, where we have to add a row to a search filter grid and constantly push a “query” button to perform our searches. DataTables does all of the with one search box – just type and the filtering begins.

GenericSearchFilter() handles that scenario. First the search term is parsed into individual terms if there is a ” ” supplied in the string. This means we will have to perform the propertyQuery for each term that we have. To return all of the records that correspond to each term we still need to perform the OR in groups, but then we need to AND these predicates together so we can get all of the groups per individual term. Here’s the source edited slightly for readability:

So GenericSearchFilter will build a humongous expression tree for all the properties in your class. To make this usable for the Where we convert it using Expression.Lambda and our Where clause just goes about its merry way. Because we have used generics, you can supply any class from your assemblies. One caveat, and Sensei is trying to find a resolution. If you have a string property to that is set to null, the expression tree fails. You’ll note that in the classes supplied in the sample, the properties that are of type string in the Tenant class are defaulted to empty in the constructor. A small price to pay for some great functionality. To sort our data we use the method ApplySort():

An extension method OrderBy will accept the name of column, the sort direction as parameters. The parameter initial will indicate if we are sorting mulitple times, so we can accomplish multi-property sort with syntax like

All good things …It’s been a long ride, this post. A lot of code discussed, a lot of ground covered. The solution is here. As always, play around and see how this can help you. If anything breaks, tell Sensei. If you have improvements, tell Sensei. DataTables is a great tool for your arsenal, hopefully the DataTablePager can help you integrate paging with large datasets as part of your solution offering.

Right now Sensei wants to sign off by toasting to you for wading through all of this, and for having the desire to build up your skills. Obtaining fluency in what you do is a hard road to travel, but it’s worth it because you get things done quicker and better with each session.