Ramblings of a techie

TL;DR Internet does still suck in most places, please make standalone apps for your products and services that work offline too.

A web page view through a bad connection

A lot of software engineers seem to agree with the popular notion that all applications will be web based one day; I am one of them, but I am someone who thinks that won’t happen soon (not before 2020 anyway) for various reasons.

Going for full web based software vs native software (on phones, tablets but also on desktop OS’s) is less performant and, when you need to do a lot of productive work, just annoying. Even if the native software is not very well designed in all aspects; going from one task to another in a productivity optimised native app will just be faster to work with. From the app (mobile/desktop) design, development and user experience side, it is possible to really push the envelop when going native which is why almost all popular desktop and mobile phone applications are native. That is not why i’m writing this post however and (slightly) worse interaction is something I can live with. Although if I have to use something a lot, it starts to really get on my nerves (try the Elance app for instance…).

Most web sites/applications are written with using mostly server side code with a frontend in HTML, CSS & Javascript. Besides a bit of AJAX, sites and applications are generally not very resilient against bad internet connections. And why would they be; everyone has a great internet connection everywhere these days right?

I travel a lot these days; UK, NL, BE, PT, ES, US, DE and others and I always have a bunch of devices with me, including laptops, tablets and phones. People who adore and keep pushing web-only applications are usually living in dense regions like cities and many of the loudest proponents are in Silicon Valley or other tech areas. Obviously the internet connection is good there. In the cities. A bit out of a lot of cities (Seattle, LA, London, NYC, Orlando, Madrid, Munchen, Faro…) the connection is (much) worse or does not work at all. Stepping a bit out of London, sometimes even the entire GSM signal is not enough to conduct phone calls on, let alone do any meaningful 3-4g work on. Around and even in Munchen I have a hard time finding a stable connection at all. A top class business hotel in the center had paid internet wifi so bad that it wouldn’t even load Gmail properly. And this is not a one-off experience; it is common, very common in 2015 still. A normal web application would not really load, an AJAX web application would throw up all kinds of errors (Jira, Hipchat) while a modern web application, once loaded (or preloaded), will work fine. It amazes me that either people don’t travel at all or maybe do not use internet a lot when travelling and just shout ‘all can be web; it is fast enough!’ from their fiber glass internet connection equipped apartments… Not having access to internet is not even the biggest problem; a very unstable internet connection is as the latter is far more common.

When referring to a modern web application I mean applications which let you use all the functionality that could possibly be used offline, actually can be used offline. For instance, writing blog posts, commenting on something, rating something, registering worked hours, creating issues etc. Content that must be updated will have a warning that it is not up to date and you click / drag to try to refresh, or it refreshes when ‘enough’ internet connectivity has been detected.

When ‘the internet is back’, the data is synced with the backend and everyone can see what happened. This is a solution to resolve the bad or no internet issue and there are enough technologies to get this to work, like Meteor or PouchDB. They are quite new and that is something to be weary off, but client-server programming is not new (and has a lot of the same issues) so it’s definitely possible to get it working well offline.

It’s not only small companies who don’t have finances to create web applications like that either; the aforementioned Jira & Hipchat web applications have it and especially Jira should work offline. But they don’t even have an official native or otherwise mobile app.

Which brings me to the next point; you have to preload the web application to work when you are offline. That’s not very convenient; you probably thought ‘there will be connectivity everywhere! It’s JFK right?’ until the plane landed and you notice you couldn’t even download your cat’s homepage properly.

A solution which is web based is for instance the Chrome Web store which allows you to download a base packet of HTML5 code which runs when you click on it, even when offline. What happens after that depends on the creator of the application (as is always the case; unless you are using Meteor or something like that, it won’t automatically work offline).

Which brings me to the conclusion; if you want me (and many people I know with me) as long term client, you need to have a mobile app, preferably native, next to your web application(s) which can work both online and offline. I happen to pick airlines based on this, I don’t work with Elance anymore because of this, I pick products for internal company use but also personal use based on the quality of the web applications and availability, at least for key productivity tasks, of a solid mobile or desktop app, preferably natively written. I wish companies would look around more and notice that the largest landmasses of this earth do not have (stable) internet and that it is quite lame to not account for the large group of (travelling) people in those parts.

You want your clients to use your products as much as possible; you can make that happen by making sure there is a top notch experience on every device, under every circumstance. Take a lesson out of the Whatsapp book.

Ever since I found this MS/Bing page about internet search engine aided code completion and read about this NSF grant given to Penn for computer aided programming, I wanted to write a post about my friend, John. He is not really called John, but his name is unique enough that he could be identified if I would use it. He is one of the people I would call StackOverflow programmers (they existed before SO but nowadays almost all results are from SO).

John is a professional coder, that is, he makes a living, a very good living, by writing code. Even more interesting, his annual income is a 6 figure number. Very impressive for a self employed coder who doesn’t do any sales himself and depends on his code to work 24/7. His productivity is on a very high level; he manages to write websites, mobile apps & games at a speed which is frightening, especially for a programmer like myself and especially considering this guy does not know what he is doing when he is writing software.

He does not understand what arrays are, does not, in fact, really understand the concept of variables. Loops are some sort of magic. Classes are convenient, but how they work exactly… Recursion? Well. Who needs that anyway right? He understands some parts of coding, because he can ask questions in enough detail to know what he has to implement and he usually understand about where to add/change code. I have not found yet where the knowledge begins / ends.

I have seen ‘his code’ and I have seen him ‘code’. The basic process is like this:

Think up what you want to create

Find some tutorial or open source (free or paid) project which does something close to what you need (pro tip: don’t care what language / framework / tech it is in; that’s easier to find)

Find a HOWTO/tutorial on how to install the software you found

Find a video how to install the development environment for the tech behind the software you found

Find the missing features in the existing software; make a list

Take a feature and search on internet, ask the software creator or ask on IRC ‘how to add the feature’

Collect enough concrete snippets to implement the feature

Change the code to match the naming of the code surrounding where you copy/pasted the snippets in (the Bing code completion does that for you)

Create a new StackOverflow feature with the snippets ‘you tried’ and ask on IRC

Go to 6 until done

I have seen him do this with Objective-C, PHP, Perl, HTML, Javascript and CSS.

Unlike most programmers who are well versed in computer science, language theory, functional programming etc, John has been making a small fortune for the past 9 years selling licences and traffic with products he wrote himself while not really understanding most of anything of what he was doing, but rather by using his business brain and a lot of copy/pasting, tinkering and working at a very hasty pace.

Is this a good or a bad thing? As someone who is very interested in formal verification of software and who is afraid of code like this I can only say, yes it mostly is, especially when someone has to maintain the code after the initial coding spree is over. There is a reason for me saying that though.

Before there was Google and most of the internet and definitely far before StackOverflow, I worked with someone who was the predecessor of a StackOverflow programmer, however it was more the ultimate copy/paste programmer. The procedure worked quite differently than searching the internet; in a Turbo Pascal book, he found the basics of what he needed and he manually copied that to the Borland TP editor under DOS. When it worked a bit like what he wanted on one screen, he proceeded to copy/paste that code literally all over the place to make 100s of programs which sold very well and the follow ups still sell well. Containing things like:

up to over a500, showing, again a lack of what an array is. The programs had to be split up in units and use Overlay because the way of programming took too much memory to compile and run.

When Windows 3.1 and later on 95 appeared on the market, the applications and the then younger me was the one who was going to port them and support them. Instead of rewriting them I thought, in my infinite wisdom, that it would be better to write a Windows wrapper around the TP sources instead of rewriting everything to something much smaller and more efficient. Big mistake. Supporting so much manual copy/paste labor which was basically puzzled together by only searching in what there already was (a very small StackOverflow the original programmer made himself).

The code bases from John and the other programmer look at lot a like, as do the gaping holes in their programming knowledge. Funnily enough, both do very well commercially; the latter has been doing very well since the 80’s like that. I know at least two other guys who are doing very well in their jobs employing the StackOverflow search procedure outlined above.

Every since I met John, I have been thinking that it should be possible to use Google to do this automatically. But even further than what MS did with Bing. Maybe Watson and Google (to search StackOverflow) as standalone AI + Watson and IRC to get people to make the details work could get somewhere in the future. The question is of course; what would be the input spec to something like that if there is no human to figure that out in the process. Maybe something to that will win me an award in an obfuscated coding contest!

Software development is very hard to grasp; it is hard for us, the people who are developing software but even harder for our clients. In the world outside software development things seem to work very differently and that gap seems larger if you have no idea what is going on when developing software. To explain the process to non developers and for non developers talking about software to their peers and back to us, there are unfortunate leaky abstractions of metaphors.

One the worst has been kind of introduced by the industry itself: components. Also building blocks and, when you talk to sales, Lego. Although components in the software sense do not mean what a lot of people think they mean, in day to day language about programming they have been piled on with plugins, portlets, webparts, gems, reusable app and to some extend plain libraries etc to mean that you can take a piece of premade software, put it in your project and strike off another feature in the list.

Sales uses this Lego metaphor during the sales process to get clients to buy products and services which are made by just clicking together a bunch of blocks we already have, preventing both high cost and bugs at the same time. No bugs as the Lego blocks were thoroughly tested at 100s of clients so they cannot contain bugs, right?

As software developers we managed to do this to ourselves with our great optimism. We were and often are the frontrunners in thinking software can actually be made completely composable. We have been making these software tools for non programmers since the 70s and keep making them in the firm believe one day software development will be like Lego. What is worse: we keep telling this to sales and marketing resulting in these great sales sessions.

But then the development starts. And even though these portlets were almost exactly what was needed or the RoR team told actually most of the functionality is actually only assembling a few 1000 gems together, the project is late, buggy, takes a lot more manhours than expected and in the timesheets the word ‘refactor’ seems to appear more than the word ‘the’. How is that possible if we were only glueing together premade Lego blocks?

Even though a lot of architects and developers drink their own koolaid in genuinly believing that this time it will be different, in reality, besides in ‘hello world’ like trivial cases, the metaphor falls flat on his face. Mostly because this abstract software science notion of components has been interpreted wrongly over time and that is getting worse still. So many clients opt for WordPress because ‘everything is already there as a plugin’; why does the project still cost $200k after esrimation and research by more conservative coders and why can a client not just click it together himself?

After the Lego metaphor has been peddled to the client and some sprint build of the project has been delivered, another favorite comes in;

The software has bugs and is not very fast, so when the client, having spent quite some cash on the project, comes in there are some discussions. Usually the agile approach, rightfully so, resulted in a large amount of changes of ‘stuff’ the client did not anticipate upfront.

The Lego didn’t exactly fit and because features are missing, it is anticipated at this point the project will take longer and thus cost more. This is the time when someone says something along the lines of: ‘when I have a house built, builders deliver in time and for the price we agreed on, how come … … ‘. This is related to the Lego and thus to the components we sold them, so this metaphor is partly our own fault again. But it also isn’t a very good metaphor for the case of the client and yet it is very persistent.

The thing is; houses are not software. Houses are often largely prefab (actual Lego!) and even if they are not, building houses is a rather clear process. And yet, I have known no-one who had a house built were the price was not higher and it did not take more time to built than agreed upfront. Why? Changes by the owner at a late time in the process. Yes, like software development, this doesn’t work well. The metaphor stands quite well if you use it from our side; say you have a house built, but there is no money for the foundation (proper architecture), you are building on swamp land (using flat files in a concurrent environment) and after the entire house was done, you want to add a nuclear bunker underneath (make the application work offline as well as online).

Another, worse, metaphor is exactly the same one as above but instead with a bridge. Now bridges are always late and over budget and that is without ‘changes’. And they collapse. So these are projects with people who know a lot about the physics of these kinds of structures and have calculated everything to a tee. And yet it is late and over budget. If they would be the same as software, with the amount of changes (I don’t like the shape of the thing, I want the pillars to be triangular!), it would never finish. Now we are getting close.

Software is very complex and the human interaction which leads up to making the software is very complex; the combination is actual rocket science and makes for this vastly complex landscape which usually cannot be estimated properly upfront as no-one knows what the outcome will be. In projects like houses and bridges, things which are more or less tried and tested repetitive processes, they cannot do it, so how is that supposed to work with software?

This brings us to, for me the worst metaphor; cars. “When I buy a car, I don’t get to pay more so why do when buying software?” “When I buy a Ferrari, I am not satisfied when I drive out with a Turismo!” Obviously this one is the worst as you are comparing a mostly custom product you had developed specially for you with a factory line mass produced one. The issue with this is Lego again; “Chris from sales told us that the process is like sticking the extras on a car with your ‘gems’, so I expect a FXX K with all the trimmings!”

When the client dictated the price and still has the car metaphor in his mind, you get classics like I recently encountered: a product made for a fixed price dictated by the client, everything else (scope, deadlines) flexible (in the contract), but when the delivery build was done, we got: “But a lot of features are missing here, we need a full featured delivery and we need it now or we’ll sue you for all the damages incurred for not having all those features!”. So I said, more as a joke than anything else; “I buy a 1983 Trabant for E250,- to drive in the F1 and will make the guy I bought the Trabant from responsible for me winning it or not.”, but he understood it now and we have a good business relationship after this.

Since that time I have been thinking more about how to turn bad metaphors into good communication aids with people who are outside of the engineering profession. Working with clients is fun and rewarding, but both sides need to understand what is at stake; software development can be a weird thing for a lot of people. Sometimes productivity falls to almost 0, sometimes it is very high; why? Sometimes a bug takes 1 minute to fix, sometimes it takes a week. The key to this is being transparent and communicating everything to the client and the team every step of the way and documenting every step so it is clear what you are talking about.

For an experiment I needed to touch the internals of MonoGame; more precisely, I required the UIView the current game was drawing on. To do that I checked out the source code of MonoGame and found that if I have a game instance;

It created an instance of GamePlatform which is an abstraction over the platform specific implementation platform, in this case iOSGamePlatform (to be found in the iOS directory of the MonoGame source tree).

Specifically, iOSGamePlatform has a ViewController method which returns a UIViewController which, as iOS developers know, has a .View property which returns the UIView.

Now the problem is that this GamePlatform (and also iOSGamePlatform) class is not available to you, the creator of games. The GamePlatform is an abstract class which you cannot use from the outside. The only way I could find the instance of this class which is used by the game is by querying game.Services.GetService(), but as one cannot reach GamePlatform, this is not an option.

As I could not find another way of getting to the required instance, I decided to use some reflection to do the trick and it worked quite well;

First, get the Dictionary which holds the instance. This is a private property:

Now controller.View will have the UIView it is using to render the game which I can now freely use however I want. If there is a nicer method to do this, I could not find it so this hack will do for now.

When I go coding on the go, I currently have a very basic text field on my mobile devices to code in. There is no code highlighting or code completion or anything like that; just a boring text field like Notepad. Now there are (apparently) hordes of people who like to write code in Notepad, but I am not one of them.

The first step to get productive on iOS or Android devices when writing code is to download a better keyboard. With the standard keyboard, it is really not possible to do much of anything on either device as far as coding is concerned. For iOS I selected ProType ; despite the bad reviews (which are mostly about the condensed style of the keyboard), I found that after a bit of getting used to, it works really well and much faster than the default keyboard for mostly anything, including chat, but definitely for coding.

For Android I use the Hacker’s keyboard which is great; on tablets it’s brilliant, but on phones it does the trick as well.

With that out of the way i’m capable of writing code relatively ok in a ‘notepad’ like TextView, but it’s of course not optimal. So then you have the options to buy some component which does highlighting and possibly code completion for you; there are a few of those but they are not very complete nor very well done. Writing something myself is an option but it will take too much time and it won’t be very robust for months or even years probably. So I opted to try integrating something open source. While it is not a native solution, ACE is a very complete code editor written in Javascript, which seems to run relatively well on mobile browsers. A native solution would have been a lot better, but I cannot find any which are advanced enough to use; Scintilla definitely is, but it does not work on Android and it would take me quite a bit of time to integrate it properly in iOS.

To properly use ACE I would need to integrate it into a WebView to integrate it into my IDE. And that is where it gets hairy. I used ACE before on the iPad on a site which uses a few tricks to make it work properly on a tablet and it was quite ok to work with. At least more than enough to like this as a solution for my IDE. In theory.

Wrapped in a WebView in Xamarin Forms (Github link) for iOS and Android, I believed I had something ok to work with. Then the issues started; obviously the scrolling in the editor is not related to the scrolling in the WebView and as such didn’t actually work at all.

Things like this should make it possible to get that to work on a touch screen, but you won’t get it to work very well unfortunately. I experimented a bit with connecting the native UIScrollView coordinates to the scrollBy() of the ACE editor but that didn’t help either.

However that was very far from perfect as well. Things that are broken and hard to fix (as far as I know):

scrolling

double cursor when you have text far to the right (CodeMirror always has this)

second delay when moving the cursor by tapping

completions like () {} [] are very annoying to work with on a touch screen

Because I lack time in experimenting further and the current results are not very positive, I think I will try to find something else to resolve this. The perfect solution, to me, would be an editor (and keyboard additions) like the one used in Codea; it works well and is robust. I believe it to be fully custom made though which, unfortunately, seems the most satisfying route at this moment…