Fix The System, Don't Blame The People In It

Menu

“Merry Christmas”? Means drinking a lot for Christmas. To the Brits, being “Merry” meant boozing it up. That’s why in the poem, “The Night Before Christmas”, Saint Nick says “Happy Christmas to all ,and too all a good night!” Merry Christmas would have ben a bit forward.

Jingle Bells? Turns out Jingle Bells was a bit of a drinking song. In fact, when it was first sung, people “jingled” their ice in their glass along with the rhythm.

Remember the lyrics?

And soon, Miss Fanny Bright
Was seated by my side,
The horse was lean and lank
Misfortune seemed his lot
He got into a drifted bank
And then we got upsot.

Upsot? What’s that? My kids and wife thought it was a weird form of “upset”, and many folks on the internet feel the same.

But it’s not. In fact, it’s a nice little pun.

“Upsot” works because it’s a rhyme with “his lot” and is kinda like “upset”
But “Upsot” also had a vernacular meaning. Ever been besot? Saw a sot on the side of the road? What would upsot mean?
Upsot works as a form of “getting drunk”, and that’s what the original people hearing this song hear, and that’s where the fun lies

Sleigh rides in the 1800s were the equivalent of drive-in movies for the 1950s crowd, a place where teens left on their own, were trusted

If you’re a teenage kid out on a date and your horse got into a snowbank? Well geesh, you’ve done got upsot.

List 10 books that have stayed with you in some way. Don’t think too hard; don’t take more than 10 minutes. It is not about the right book or great literature-just books that have affected you in some way. Then post a link to your blog here so I can see your choices.

Mine, in no particular order:

Anna Karenina. Introduced me to Tolstoy. Love the guy. His essay on Napoleon at the end of War and Peace is way too long, but important.

A Guide to the Good Life. Stoicism is not absence of feeling — that’s just the modern definition. Real stoicism is all around us, and western Christianity owes it a lot. Great read that made me rethink my attitude on things.

New Kind of Science. Boy genius grows up and tells the rest of us that we’re doing math and physics the wrong way. What’s there not to like? Tough read at times, though.

The Physics of Immortality (Omega Point). This was The Matrix before the movie, and based much more on science than the flick. Powerful ideas here that humanity is still grasping with.

The Plague/Sisyphus. “We must imagine Sisyphus happy” Once you read and absorb that? Your life isn’t the same any more.

1-2-3 Infinity. As a kid helped me understand why we had math. There were a few other books in there, but this is the one I remember.

Godel-Eschter-Bach. Snooty guy writes long, involved, puzzle of a mixed genre book. Great introduction to formal systems and a nice little adventure into his theory of what makes intelligence. It’s a deep book, but it’s also a bit of a lark. The author likes putting the ideas creatively together from music, math, and art — so much so that it’s the enjoyment of the new book that’s really more important than the rest of it. I took three runs at this before I finished it.

Code Complete. The first book that made me realize that there was this world of programming with quality as opposed to just programming to solve problems.

The Sparrow. A more recent read. It’s kinda sci-fi, but it’s also a great read about other stuff. Don’t want to spoil it. Very readable.

On Liberty. Not sure if I read this or listened to it on audio. If you don’t understand the concepts here, you shouldn’t be allowed to vote. Seriously.

Man’s Search for Meaning. Another readable book about deep stuff. The ideas here made me grow up.

How to Win Friends and Influence People. Read this as a teenager. I keep telling people about it, and I want to re-read it, so it must have had a pretty strong impact.

Beyond Bullet Points. Not a book that changed my thinking, but a book that made me realize my thinking was changing. If you want to teach people, you’re going to have to tell stories. No more slide decks with “word salad” on them!

Worst news? (I guess) I stopped losing weight. That’s good, because otherwise I would be invisible.

After losing about 105 pounds, I just reached a place where I wanted my old life back. I didn’t necessary want to eat a lot and be fat, but I wanted more energy, less constipation, and less sitting around thinking about my health. So I started adding foods back into my diet and loosened up on the 30-30 rule.

And I gained weight. About 6 pounds so far. Wouldn’t surprise me if I gain another 6, maybe a lot more. This has always been a losing battle, remember. I had no illusions coming in. The sad fact that most diet books and other professionals will not tell you is that only about 5% of people lose weight and keep it off. It’s always a losing battle. If it makes you feel any better, animals also are getting fatter — and nobody knows why.

The good news is that it’s good to be 100 pounds lighter. I enjoy it everyday. I enjoy the clothes, the reserve energy I can call on when needed, and getting more respect from people when dealing with them in social situations.

Hair falling out? Not so much. I didn’t go bald, but the top of my head is a lot shinier than it used to be. Not sure if it’s coming back or not. (I stopped cutting my hair when I had the surgery)

I had a “blockage” last night. First one in a long time. I had eaten a slice of pizza around 4, then around 6 went out for asian-fusion food: a beef stir-fry on rice. The beef was good, and I was busy talking to my business partner (remember, the goal is to get back to normal, not sit around counting food chews) so I just kind of wolfed it down. I ate maybe 7 or 8 times in a row without really thinking about what I was doing.

Big mistake. I finally realized what I had done and stopped. But it was too late. I suffered for a couple of hours, took my Papaya Enzyme, and it went away.

I guess some folks would consider this a bad thing, but I don’t look at it that way. I purposely had my digestive system altered because I eat too much. This was my new digestive system working as it should — reminding me to slow the heck down. So today I plan on doing that!

I think the surgery was one of the best health decisions I’ve ever made, but it was also a life-changing event. I used to think I was bullet-proof — my health was always pretty good. Not so much anymore. Now I realize how precarious health can be. At the same time, I’ve come to realize how important it is to get out and engage with life. I think it would have been easy to have took more time off, self-obsessed, and maybe lose another 30 pounds or so. And who knows, maybe that’s what I should have done. I remember those days of just taking the football and running with it.

At some point, however, life is about more than football. That is, there’s more to life than how much you weigh. If your digestive system is not configured for the world we live in, fix it. Then move on with your life.

Looking at the net today, I can’t help but reflect on how it’s turning out so differently than we imagined:

We wanted to exchange information, not play games.

We wanted expected open conversation and idea exchange, not constant drama, anger, and angst.

We wanted insight into the world outside ourselves, not have the world know everything there is to know about us.

We wanted freedom from oppressive governments, not the creation of a new security state.

We expected the web to either be free or paid, not monetized based on how long it could hold our attention.

Browsers were just one way of accessing the net. We expected there to be lots more.

We expected technology to empower us as individuals, not create some kind of hive supermind.

We expected human, face-to-face relationships to be augmented and thrive based on new silicon helpers. We’re finding people staying online and disconnected for most of their lives, texting somebody sitting 10 feet away instead of attempting a conversation.

Man is a social animal evolved to operate semi-autonomously as hunter-gatherers in small tribes. What we are creating is a technological system that is adapting to, emphasizing, and taking advantage all of the weaknesses of the species while not emphasizing our strengths. Another way of saying this is: we’re not getting what we expected, we’re getting what we wanted.

They say that kids growing up in a new system automatically think that system is just the way things are supposed to be. It’s really important to tell this to whoever will listen.

I was there with some of the first commercial users on the net. (Heck, I was on before that, but as a college student I had no idea of what the net was). I was there as developers and content creators started piecing together the future. I was there when Google figured out the ad model, when the average MMORPG gamer was spending 35-hours online each week, when the NSA revelations came out, when Farmville took Facebook by storm. I was there before that. I remember where we were headed.

Currently, when we want to make things happen, we move between imperative-style coding, where state is scattered all over hell and Georgia, to complicated deploy scenarios, where services talk to each other over all kinds of protocols, to even worse server architectures, where they’re set up once they just kind of grow in place. All of these pieces are moving targets with various versions and constraints. As each layer gets more complicated — the language, the services, the protocols, the architecture — we end up with more and more complex work of keeping all the pieces together and running smoothly.

We’re seeing some patterns. Pure functional programming is composable. Actually, that’s the whole idea: you compose functions into a larger transform. The Unix O/S service model, where you pipe together things like ls, grep, awk, and so forth, is also composable. Composable pieces are scriptable and easily testable; you can “play them back” to verify correctness. In addition, server architectures are also becoming scriptable, where you describe the server architecture in script and then the script builds the platform.

So why not join this all together?

Picture this: A pure FP language where you could either compile the entire code as one piece, or automatically split it up into chunks and deploy separately. You could keep the code in one place and the only thing you’d need to tweak would be the chunking. Want one big app with an input and output? Click this button. Want a dozen apps that talk to each other over a common format? Click that button. Now layer in some DevOps on top of that where certain pieces would talk to other pieces on a schedule, or across a wire, and this format, scheduling, and protocol could be specified in the code (perhaps using attributes). You could meld this into a Puppet/Ansible-style system where not only do you code the solution, but you code the deployment as well.

So you end up with one source file(s), in the same language, that covers app, inter-app-communication, scheduling, and deployment. Learn one system you learn the whole thing. Heck, roll in JS deployment and html while you’re at it. Go back to the days where you coded in one place and the things you coded just ran — you didn’t have to worry about 43 different pieces of complicated wiring spread everywhere.

I was reading a great book the other day about publishing. It was recommended by a friend. In the book, the guy recommended a certain tool. Cool! I go over and purchase the tool.

But wait, there’s more! It’s not just the tool. You can buy three different versions of the tool. Once you purchase, there’s an add-on templating pack — do I want the basic templates, the advanced ones, or the lifetime subscription? Remember, this is my business. Do I really want to risk all of that because I didn’t get the right templates?

I know what’s going on. I realize that the moment I make a purchase decision I’m at my weakest — I am primed and ready to buy most anything. After I decided on the lifetime pack of templates, which I probably do not need, I was pitched several other products — some through some nicely-placed prop web pages (I would consider a product, google it, then land on an extended funnel that consisted of a recommendation followed by a link back to the sales page). Within 5 minutes, no matter what I did to investigate the concept the tool vendor was trying to promote, I was back on his long-form sales page looking at a set of courses for 300 bucks.

Not useful.

Look guys. I know the long-form 300 bucks sales thing. Half of you guys are teaching me long form sales, and then you’re sending me over to your buddies who are trying to sell me with it.

This may come as a news flash, but even with 100 testimonials, there are some products I do not need. It’s also highly-likely that the more sales magic you put on your product, the more people you are separating from their cash who cannot afford it. Your product that is going to triple my conversion rate/sign-up rate/site traffic/penis size is probably not that useful unless I’m already delivering quality content. Clicking a few times and providing a Visa number does not make a business.

But damn, I want to believe it does. I want to believe I can buy a brand-new shiny 47-button joystick and pilot my startup like an F-16 shooting vertically off the end of the runway in Flight Simulator X. Just show me where to click.

I’ve noticed a habit in startup circles. The internet seems a fickle place, and its not unusual for a friend to come up with an idea and use it to rocket to success. They’ll have an app, or an idea for teaching, hit the right leverage spot in regards to publicity, and suddenly they’re in gravy.

For a while.

But then the numbers drop. Attrition starts taking its toll. The old methods of priming the pump don’t work. The cool kids don’t do things that way any more. And, like good startup folks, they go out and talk to their audience to find out what they want. Make the lightning strike again.

And this is where it gets interesting. Because the more my friends engage with the community, the more they look for common things people want that they can provide, they always end up somewhere on Maslow’s Hierarchy of Needs. Who doesn’t want food? Shelter? Money? And so on. It’s perfect! They’ve found the ideal niche!

So over time, each year around the world you get several hundred folks hitting the big time with DinoWidget 2.0, then the next year they’re giving out classes for using DinoWidget to self-actualize, or make more money. A year or two after that they’re doing high-pressure sales about some primitive need that we all have. Repeat that for ten years, and you’ve got an internet full of folks looking to separate you from your money. Each person may be acting in what they consider to be a moral and upstanding way, but in the aggregate? It’s a cesspool.

Just to be clear, I do not feel that most of these folks are trying to trick me in any way. But their idea of an honest description of a product and the actual impact it has on startup founders are two different things. This, also, is a natural thing. In their mind I’m the struggling business guy who just needs this one push that they can provide to hit it. This is their fantasy. We all have them.

The problem is, as a single-founder, it’s tough out there. I have limited resources and time. Most of the important stuff, like finding a product-market fit or building relationships with key thought leaders, ain’t happening with a tool. But any kind of marketing and sales system does something very subtle: it tells you a story about how awesome you are going to be once you make this purchase.

We are all very eager to buy into a narrative about ourselves where we overcome some of life’s really tough problems by making simple (and heroic) choices. Testimonials, which everybody and their brother uses today, tell us a story about folks just like us who were just where we are right now, and by making a simple yet brave choice to change their life and spend a few bucks, the clouds opened up and it was all broad sunlit uplands.

We love this. We love putting ourselves in the place of others and living their lives and making choices vicariously. Dreaming about how we could solve big problems. It’s a natural human channel for instruction and manipulation. Hell, it’s the basis of all fiction. But we can easily confuse stimulation — the absorbing of outside useful ideas — with self-stimulation — living in a world where we can solve tough problems with PayPal purchases.

Startup founders are in a particularly weak and vulnerable position. A story about making a purchase and having lots of extra income? They can buy into this. Very easily. Hell, most of the time to be a good founder you have to have an unrealistic expectation of your chances anyway. Having some long form sales page give you a hundred reasons why you’re going to make a couple decisions today and nail your sales figures is something you’re all too-willing to believe. This narrative speaks to you — and it’s a dangerous story to believe.

I don’t want to leave this on a negative note, so here are the rules I’m applying to my economic decisions:

I don’t buy things where the process is initiated by others. If it’s an ad or a tweet, I’m simply observing.

I research things people tell me about in person to see if they might be useful.

I never buy from a long-form sales page. The entire thing is manipulative. I might click there if that’s the only place, but I don’t consume content and don’t linger

As an early-stage founder, I’m concentrating on conversations with people, not metrics, not flash, not systems, nothing else. If it helps me have conversations, I’m liking it.

I am starting to discount advice given to me about how to build a startup by anybody who also sells tools to help startups. It’s not that the advice isn’t good. Heck, it’s probably the best advice around. It’s that I can’t trust the speaker to help me make the cold economic decisions necessary. I also think it’s very easy for these folks to get into a mindset of “when you’re a hammer, the world is a nail”

The startup community is eating itself. Great folks start out with awesome products, pivot a few times, then end up pitching other startup guys stuff. Why? Which audience do they know best? Startup folks, of course! It’s not a terrible thing, but it can have very evil effects. You can lose your shirt and not have a damn thing to show for it if you’re not very careful.

Just had a great time in Vegas with some other small startup guys talking about how to create something that has real value for people — how to help folks in the world. As a side-effect, the goal is to work from home, spending lots of time with your family. It’s no get-rich-quick scheme: everybody has to find their own path. But there are a lot of common things too. Most of us use the internet, most of us either used to program or program now, most of us are part of something called the Micropreneur Academy, and most us, well, are nerds.

Here are the three things I learned (or re-learned) about myself this year:

I don’t know enough about the customer. Yes, I know the job of people I’m helping. Yes, I know the subject area I’m trying to teach. But while I have general and in-depth knowledge, that ain’t cutting it. I need to climb inside my customer’s head, spend time with them doing their work from their point of view, not mine. I keep wanting to jump ahead to the business-building part of things, instead of product-market fit. If I get the right product-market fit, the market will “pull” the rest of the business from me. People will be clamoring at the door demanding that I take their money. If I don’t? Then I’m building something and trying to beat people over the head with it. Don’t do that, Daniel. It hurts.

Hey, I actually know this stuff. Since I’m trying to help people create, maintain, and optimize their to-do lists (backlogs), it would help if I knew what I was talking about. The more I interacted with fellow startup guys, the more I realized that I actually could help these folks too. I’m actually on to something very useful across a broad market. In fact, because book 3 is going to be on backlogs and startups, I can start “eating my own dogfood” and use the things I know in the work I’m doing. It sounds like I am an idiot for not knowing/doing this already, I know. But a lot of times there’s really obvious stuff right in front of you that you miss because you’re in the weeds. Jesse Mecham gave a great talk yesterday about this. He was running a profitable startup for years and still afraid to quit his day job because of his desire for security. It took going to a minister for somebody to convince him he had the answer all along.

Teach more to make more friends. This is one of those things that I understand intellectually, but I don’t think I really get it. Plus, as somebody who took years to learn a few things, I’m seeing these guys become experts in stuff that took a month or two, and then overgeneralizing what they know and saying things that aren’t necessarily true. This makes it tougher on the next guy who comes along who tries to take the reader to the next level. But I understand that taking somebody on a journey of learning with you is good for both of you. I need to keep banging this against my head until it sinks in.

My three action items:

Start talking to people on the phone. Once they sign up for the email course, ask them if they have a few minutes to chat. Get to know why people are interested in learning more about helping Agile teams. There’s no agenda: I’m not trying to sell or push products. Simply trying to gain as much loosely-formatted and unstructured data as possible that I can then go back and find themes and put some structure around.

Use my backlog principles to organize my own startup work. Hey, if the goal here is to complete a series of books that wraps up value creation from customer development to 20,000-person enterprises, time to put the stuff in book 3 to the test personally, instead of just reading about it and watching it. The startup part is probably part of the series I’m weakest on, so nothing like the present to fix that. Not only does this make sense, it’s only an hour or two of work, maybe every month or two. No biggie. (Which is one of the beauties of getting this out to folks, it should take a tough thing and make it much easier)

Revamp my email strategy. Right now I have an Agile Tune-Up email series that runs every week over a period of a year. The purpose is to get to know folks, start a conversation, let them know who I am and that I know some stuff that could help them. That’s about it. I really should 1) offer a free multi-part course or something else of value in return for folks joining the list, 2) re-vamp my weekly series so it’s more useful to the readers, 3) manually test out the emails that go out before I automate the process so that the formatting, message, and timing is exactly what I want, and 4) add in a bit of data around segmentation when people join the list so I can help them better. Are they coaches? Guys working at a big corporation? A startup? I need to create a trial scoring system and hook that into the people I interact with. Powerful idea from Brennan Dunn yesterday. Looking forward to implementing it.

As soon I finish creating the 17 clones I will need to do all of this.

But wait! There’s more! I have a couple of strategic bonus problems that need to be solved right away.

Videos or books? I’m halfway through Backlogs 2, and I only have an early version of the first of eight videos completed for the Backlogs 1 series. Looks like completing either of those two projects should take a solid 3 months. I can’t do both — or rather if it’s possible I don’t understand how. So what to do? Finish the video series so that folks can have the hard-nosed, detailed, deep-dive down on personal and team backlogs after they finish the first book? Or go ahead and continue the story with Smith and the plant, giving people more entertainment — perhaps even finishing the final book — then swinging back for the videos? Maybe neither? Maybe I just stop now and go start marketing the hell out of the book I’ve already completed. Beats me, and I’m the one supposed to be figuring this stuff out.

All-in-all, it was a great conference. Lots to think about, great folks, and some terrific tactical advice that I didn’t even get into here. Good stuff.

Several people have asked online lately, “Can you really write production code in F# and mono?”, “How are you doing architecture now that you’ve switched completely to F#”, and “Aren’t you doing some kind of weird thing now that doesn’t work with interactive websites?”

I usually don’t do coding blogs, because, frankly, after a while it all gets pretty boring. Same old bytes, just new bells and whistles on top. But my approach to architecture has changed in a major way over the last five years. Might as well document it.

Description of Problem Domain. For our sample project, let’s take Newspaper23. It’s a production system with a small number of users that’s been running for a few years and has evolved through my change in philosophy.

Newspaper 23 exists for one reason: I have a very short attention span. Websites are always trying to get me to hang around, click on related stories, sign-up for things, and so on. I find this very distracting. It’s very easy for me to spend a lot of time online that’s unproductive. So a few years ago I had an idea: since the news I consume is all from a dozen or two sites, why don’t I just harvest the titles and links from those sites and put them in a big list? Then I’m choosing what content to consume based on the site brand and the title of the article, not social cues or any other nonsense.

So I got out my credit card, went over to Amazon AWS, and fired up the site.

The Old, Old Way of doing things. The first time through, I did this in a “classic” manner. I sketched out a domain model, I elaborated a bit on a database model, I built a database model in MySQL, and I created a new C# app in Visual Studio and wrote a program to do everything that was needed — get some links, save the data about them, create a page with the links, do some “other stuff”.

The problem was, I was never really clear what the program should do. Should I collect the votes the article had from reddit? Don’t know, so I’ll throw that in there and save it. Might need it later. Should I provide the ability to comment, or vote? Don’t know. I’ll add space — after all, space is cheap — and if I need it, it’s there. Should there be some kind of authentication? Beats me. (Decided to skip that one)

Don’t get me wrong; none of this was a heavyweight or Big Design Up Front process. I’m only talking about an hour or two of doodling around with things. And the beauty of modern tools is that it really doesn’t matter whether your table has 5 fields or 10. Unless you’re writing the code to do something with the data, it’s just “out there”.

It did make the “link sucking” process a little more complicated, because with different sites I had to get different pieces of data. Fair enough.

What happened. After a couple of weeks of off-and-on coding, I got the program up and running, along with a unique system for identifying data on a web page to harvest. (It was a recursive RegEx language. Please, no “but you can’t use Regex on html!” It looked good.) The code base consisted of a data tier, some controller logic on top, and it seemed to work fine.

For a while.

But over a period of just a few days or weeks, it seemed like there were always these weird edge cases, both in the sites, and more disturbingly, in the tool version, setup, and configuration. Site X would change their format, and I’d be in the code changing the way I received links. Yes, I had hard-coded in the parameters for each site, but how difficult was it to change some hard-coded values once or twice a year? Except the more source code there was, and the more I touched the code, the more errors I got. And it seemed like for one reason or the other with this way of programming there was a boatload of code to wade through and I was always horsing around with it. [insert discussion here about TDD]

By using a MVC paradigm, I had written one piece of software that I could easily lay a bunch of different controllers down on. In fact, as we all know, I could develop a “controller library” over time that would take the data and do all sorts of wondrous and amazing things. That’s the beauty of a good model — it facilitates many different solutions.

I still think this might be the way to go, assuming 1) you use good testing practices, and 2) you’re going to constantly be under the hood in the code for a long time. But it introduced all kinds of problems related to having everything pass through one executable and code base. Did I really want to horse around with the code that grabbed links, for instance, simply because I was fixing the code that wrote the web page? Why was I putting all of my eggs in one basket? OOP is a great way of doing things, but it introduces a freaking huge amount of hidden interdependencies. Some times I wonder if 90% of the problems development teams have with programming is based on the fact that in the developer’s mind there is separation of model and controller, but that in the wiring itself, things can be all hooked together in completely insane ways, usually for what seemed like good reasons at the time. The black box of OOP has a lot of very sharp edges.

The Old Way of doing things. The first thing I did was ditch C#. It’s a great language, a better Java than Java, but it was the wrong paradigm for me. The second thing I did was start to build the app from the ground-up. Programming functionally required me to write functions that did things. This sounds blazingly obvious, but it’s actually a much different way of looking at things from creating a model and adding controllers on top. I still used the MySQL datastore — had to put stuff somewhere — but my code was a LOT smaller.

It also needed much less tweaking. Remember: this is personal programming. Instead of sitting on top of a model that could do many things and having to choose what to implement, which might cover calls all over the place, my code did a few things, and there was a clear path to the data for the things it did.

What happened. There was still that problem with everything running through one spot. If I wanted to tweak anything, anywhere, I’m opening up the entire codebase to make a change.

Interestingly, since this was revision 2, the code ran for much longer without needing tweaking. This meant that when I did get around to tweaking it, I had forgotten the build setup! So I’d have a 2-line change that would require screwing around with re-remembering the build-deploy environment. That might take several hours. Plus the sites were still hard-coded. The output was hard-coded. As much as I loved data-driven programming, why was everything so coupled?

Along this time I built HN Books, a social book site for hackers. It was all static. I found I could do a lot with static web pages and JSON files. Much more than most web programmers would believe.

Hmm. Fuctional programming. Static pages. Hmmm.

The way I do things now.

So now I have a new philosophy:

Third time through, I decided I write small programs. As small as I can. Instead of one big honking system, my system has four programs: GetLinks, UpdateArticleLibrary, CreateSelectedArticleList, and FormatSelectedArticleList. There’s a bonus utility, CheckXPath, which lets me check out XPath strings against websites to make sure they work (in case the site layout changes). I use as few external libraries, tools, or frameworks as possible. Over the years, I’ve found that every tool I picked up had 40 tons of features, of which I only needed 3. That meant that whenever I needed to do something simple that I had never done before, I had to wade through all sorts of forums, hearing about all sorts of arcane switches and other stuff. Screw that. I do not need to buy and operate a nuclear attack sub to go fishing from the pier. So — no Visual Studio, no MySQL, no complex dependencies at all. (Having said that, I actually dumped my custom link-sucking system and went with HtmlAgilityPack. I mean heck, once it’s installed, it’s just XPath. Time spent learning XPath is not as completely sunk as time spent learning WhizBang 7). Simplify.

My files read and write data on the local storage. I’m not doing transactions, therefore I don’t need a transactional datastore. It’s functional coding. Things come in, things go out. The O/S already has a wonderful tool for storing things. It’s called a file system, and unless I’m going to be moving around tens of thousands of these things, it’ll work just fine, thank you. Simplify.

I use the O/S to schedule when the programs run and to move things around. Instead of having one program that runs once a day, split the work up into a “pipeline” and have the O/S manage the pipeline. It’s already good at managing processes — it has all kinds of tools for it. Use them. Simplify.

I don’t “interact”. I process things at certain times. In the old days, I’d have an instance of my program spun up in Apache, waiting around in fastCGI for a client to come and connect. Then the program would do all sorts of things depending on which client it was, what the request was, and so on.

After many, many years of coding like this, I had to ask myself: why? Why the hell am I doing it this way? 9 times out of ten I’m delivering the same content. 9 times out of ten the client is just reading stuff. 9 times out of 10 I’m creating database connections, cursors, and all sorts of other cruft — just to send the same stuff back down the wire as I sent 2 seconds ago. For the huge majority of the use cases I can imagine, even for interactive sites, there’s no difference at all to the user between a program that sits waiting for connections and a series of programs that updates data every few seconds. This is stupid. Don’t do this any more. Decouple. Simplify.

I write simple queries to monitor how things are moving through the pipeline. Quite frankly, the system is running so well I don’t need to monitor it, but if I did, I’d simply monitor how things flow through the pipeline. I could even make a nice html page with graphs. Don’t need to, but it’s an easy option. In fact, I’d argue that the only things interesting to me as a owner/maintainer would be things visible at a system level, not a programming level. Huge win here: no programming skill required to look at CLR innards, just O/S skills. Simplify.

Nothing exists as hard-coded data. It’s all either config files or command-line parameters. Right now as I bring up the page, I can see that Hacker News isn’t returning any data. If I wanted, I could probably figure out what the problem was and fix it (assuming it was fixable on my end) in about 10 minutes. No programming required. All I need is a shell. I AM re-coding the system, kinda. I decided that since I have 10-15 functions that are the same across each executable, it would make sense to create one Visual Studio solution and share a couple of library source files. Welcome back to the days of shared C header files! I’m also making the logging more robust. Right now I log everything. (I don’t read the logs, but they are there.) This is using up too much disc space. Every few months I have to clean it out. So a more fine-tuned logging system would be nice. Maybe. Maybe not. Simplify.

TDD? TDD? We don’t need no stinking TDD. With purely functional programming, there are only 3 mistakes I can make: Failure to break down transforms into understandable atomic units, failure to describe the transform correctly, and failure to validate the data. If I do all three of those correctly? There’s nothing to test. It’s like trying to test an SQL select statement. The idea doesn’t make sense. Simplify.

It’s very nice. Here’s the main function for the first program in the chain, GetLinks, that gets links from sites (duh):

else"Number of links, titles, and other Data DOES NOT MATCH.\r\n" + max.Key + " has " + maxCount.ToString() + "entries while " + min.Key + " has " + minCount.ToString() + ".\r\nCheck your configuration file or run this program with the verbose option set /V"

else

"\r\n\r\nTHERE WERE NO SECTIONS OUTPUT.\r\nPlease check your configuration file.\r\nTry running this program with the /V option"

System.Console.WriteLine(numberOfDataPointsMatchMessage)

0 // return an integer exit code

with

| UserNeedsHelp as hex ->

System.Console.WriteLine("You'd like help")

System.Console.WriteLine("Good luck with that")

System.Console.WriteLine("/V for verbose output")

System.Console.WriteLine("/S:<url> to set the target site url")

System.Console.WriteLine("/C:<filename> to set the config file used. Config file is expected to have Windows line returns \r\n")

System.Console.WriteLine("/O:<filename> to set the output file created/overwritten")

System.Console.WriteLine("/N:<integer> number of links desired")

System.Console.WriteLine("")

0

This is 50 lines of code. But it’s really only 4, lines 204-207. It gets the configs passed in. It creates a new container to hold the output data, it processes the link, and then it saves the data (which happens in the ripLink function. Should be down in this one. Ugh.) The rest of it is logging and help system stuff.

I could show you ripLinks, but it’s the same deal: 50 lines of code which are really about 10. It gets the links from the page using the “ripLinksOnAPage” function (clever naming, eh?), then it processes the links according to the config file.

Let’s look at how it breaks up.

As you can see, there are really only 5 functions, ripLinks, http, loadHtml, recBuildLinksUp, writeLinksOut, and ripLinksOnAPage. The rest is either library calls or a few small helper functions. Take out the logging and some of the other broilerplate, and there’s maybe 50 lines of “real code” here.

We’ve done something simple, easily describable, and concrete. We took stuff from the file system, read a webpage, wrote stuff to the file system. Logged as we went along. That’s it. No need to solve world hunger. It has value.

This code has been running for more than a year with no modifications. I expect it to continue running forever. I’m done. Isn’t that nice?

Common objections.

It’s re-inventing the wheel. In most cases, I don’t need a wheel. I need a lug nut. Yes, I know what the wheel looks like, but the cognitive load of buying a big stack of wheels and carrying them around when I just need a couple of lug nuts? Tell you what. If I start thinking about round things to fit on cars, I’ll get a wheel. Otherwise I’m fine.

It’ll never scale. The funny part about this objection is the opposite is true: the more complex your stack, the more difficult and nuanced it is to scale. I can take this app and scale it out as far as I like. Heck, copy it to a CDN. We’re done. And for those of you thinking interactivity, think long and hard whether you need immediate feedback for the user or just something that changes every few seconds or once a minute. You could be spending a huge number of processor cycles worrying about concurrent clients when you really don’t need it.

It’s poorly-thought-out. My datastore? Line-delimited text files. Code complexity? Nothing more than a few hundred lines of code. Is there anything it can’t do? Not really. But (and this would be the objection I would have made several years ago) what if you want it to do something that required the data to change? Wouldn’t you have to re-jigger every piece of code in the pipeline?

Not really. First, because I’m using name-value pairs, I can always add data and not use it. So really, what would happen if I wanted some cool new feature would be 1) I’d change the data representation where it was created, 2) I’d add the code to store it, and 3) I’d add the code to use it. If you’ve worked in a functional environment, this is the way you make changes anyway. Nothing new here.

It’s not as cool as X. Probably not.

The deployment process is brittle. Actually, although I’m not continuously deploying what I write — there’s no need to automate it when it’s just me — I’m integrating DevOps directly into the solution. There’s no one part of this solution that’s “programming” and another part that’s “deployment”. It’s all integrated together. Instead of good coupling and cohesion at the function level, I have good coupling and cohesion at both the function and the executable level. Very cool stuff.

Continuing to add features. The very next thing I’m going to do is add/adjust the logging. Having to clean out the logs once every 3 or 4 months is a chore. Next up in my quest to eliminate distractions, I’ll probably go to the target site and rip the plain text and store it here. I’ve thought about adding voting and commenting, but it’s a personal site.

None of this will require a major change or a re-think of how the architecture works. Mostly the system just works and I don’t have to mess with it. O/S updates handle updating security and scaling problems. I worked a bit to make the solution, and now the solution works for me. And isn’t that the entire idea?

When I was a kid, going to secondary school and college, smart people consumed the news.

If you didn’t have much time, you “snacked” — consumed the few news segments on TV. If you considered yourself a serious person in society, wanting to be an informed voter and current on events and discoveries of the day, you got a morning newspaper and read it. Gave you wonderful material for conversation the rest of the day. Finally, if you really wanted to understand how the world around you worked, you subscribed to magazines, where long format and deep-dive articles took their time explaining to you why things were the way they were.

No more.

Nowadays most people don’t trust the news, but they still consume it. I find that odd.

The usual culprit is that the press has some sort of leaning to it — it’s either conservative or liberal. But press has always been biased, so I don’t buy that. People will tell you that it was the fault of cable TV and the 24-hour news cycle, but while that played a contributing factor, I don’t think it’s the entire story.

Nope, the reason consuming the news today sucks is that we live in a world of constant outrage.

At some point, news publishers realized that emotional engagement, not facts or solid background material, drove readers to consume and share. So all of our media channels are full of people who are either outraged about something or are using thinly-veiled logic to get us upset about something.

And so we have a treasure trove of material designed to drive “engagement”, which just means it’s stuff guaranteed to provoke an argument. Any news event can be spun half a dozen ways to try to generate anger — and it will be. Then, whichever angle works out the best will be mined for eyeballs until the next story comes along.

This consumption of material engineered to constantly outrage does not make for a healthy mind. Part of the reason is the constant emotional roller coaster it puts the consumer in, but part of the reason is that the media outlets are constantly trying to cover up and deny that this is why they’re running the stories to begin with. So most outlets well-known for “just the facts” reporting are anymore just presenting a light sheen on top of articles designed to enforce pre-existing attitudes.

Put another way, the reader is constantly being manipulated. The only question is the degree of manipulation and the honesty involved.

That’s why I’ve converted to reading tweets and opinion columns. Tweets are almost entirely too shallow to waste much of my time, and they’re wildly inaccurate, but they keep me apprised of the general gist of day-to-day conversation. Opinion columns are there to make a point regarding some pre-existing opinion. I find that to be perfectly fine. If you’re going to spin and slant the news to make your point, at least be a man about it and tell it to my face. Don’t hide behind “analysis” and pro and con segments.

With these two forms of news consumption, as long as I read opinion columns from all over the spectrum, I get a fairly good balanced diet of what’s going on. I don’t find all the drama in the news that my fellow consumers feel.

Consuming the news has changed. Smart people don’t do it like they used to.

I wish I could say long format pieces have survived this shift. They have not. More and more, I’m seeing long format articles that amount to nothing much more than extended arguments put forward by one special interest or another, many times with an interview of a token person holding an opposing position as some sort of fig leaf to “fairness”. What is needed here, as in tech and science news, is reporters that actually know their area and can write stories at length about important events happening there. Instead what we’re finding is reporters who are getting socially involved in issues, then try to pry meaningful news from their social network. You end up with four-thousand-word cocktail party chat. Not always, but more and more.

It’s sad that news is dead. As a former freelance writer who has written for both weekly, daily, and magazine outlets, I liked them. The TV guys were never hitting on much, but they had a fun, egocentric job to do as well. These guys as purveyors of what’s important to know are long gone. Their job positions and media outlets will go on for many decades longer, sadly. And dumb people will keep consuming them, keep getting upset every day, and keep wondering why the world is such a bad place to live in.

Working on wrapping up my third e-book. It’s a book on how to make and manage effective backlogs, or the lists of things both teams and developers do. The first two were kind of a lark — although I was trying as hard as I could, I realized that this was something that was going to take a while to learn.

This time around my e-book is much larger, and I’m hoping to hit the mainstream, both in terms of content and quality. Most all organizations have the job of coordinating their work both globally and locally, and I’ve made the book as a lesson wrapped in a story. I figure either you’ll like the lesson or the story, maybe both.

After spending several days fighting tools, I thought it’d be good to capture what I’ve learned. If you’re a tech person who wants to write an e-book, this’ll help you get from words to e-book format.