Here’s a subject of which I’m all too familiar and is in need of commentary. Since my profession is technical in nature, I’ve definitely run into various issues regarding software engineering, systems architecture and operations. Let’s Explore.

Software Engineering as a Profession

One thing that software engineers like is to be able to develop their code on their local laptops and computers. That’s great for rapid development, but it causes many problems later, particularly when it comes to security, deployment, systems architecture and operations.

For a systems engineer / devops engineer, the problem arises when that code needs to be productionalized. This is fundamentally a problem with pretty much any newly designed software system.

Having come from from a background of systems administration, systems engineering and devops, there are lots to be considered when wanting to deploy freshly designed code.

Designing in a Bubble

I’ve worked in many companies where development occurs offline on a notebook or desktop computer. The software engineer has built out a workable environment on their local system. The problem is, this local eneironment doesn’t take into account certain constraints which may be in place in a production environment such as internal firewalls, ACLs, web caching systems, software version differences, lack of compilers and other such security or software constraints.

What this means is that far too many times, deploying the code for the first time is fraught with problems. Specifically, problems that were not encountered on the engineer’s notebook… and problems that sometimes fail extremely bad. In fact, many of these failures are sometimes silent (the worst kind), where everything looks like it’s functioning normally, but the code is sending its data into a black hole and nothing is actually working.

This is the fundamental problem with designing in a bubble without any constraints.

I understand that building something new is fun and challenging, but not taking into account the constraints the software will be under when finally deployed is naive at best and reckless at the very worse. It also makes life as a systems engineer / devops engineer a living hell for several months until all of these little failures are sewn shut.

It’s like receiving a garment that looks complete, but on inspection, you find a bunch of holes all over that all need to be fixed before it can be worn.

Engineering as a Team

To me, this is situation means that software engineer is not a team player. They might be playing on the engineering team, but they’re not playing on the company team. Part of software design is designing for the full use case of the software, including not only code authoring, but systems deployment.

If systems deployment isn’t your specialty as a software engineer, then bring in a systems engineer and/or devops engineer to help guide your code during the development phase. Designing without taking the full scope of that software release into consideration means you didn’t earn your salary and you’re not a very good software engineer.

Yet, Silicon Valley is willing to pay “Principal Engineers” top dollar for these folks failing to do their jobs.

Building and Rebuilding

It’s an entirely a waste of time to get to the end of a software development cycle and claim “code complete” when that code is nowhere near complete. I’ve had so many situations where software engineers toss their code to us as complete and expect the systems engineer to magically make it all work.

It doesn’t work that way. Code works when it’s written in combination with understanding of the architecture where it will be deployed. Only then can the code be 100% complete because only then will it deploy and function without problems. Until that point is reached, it cannot be considered “code complete”.

Docker and Containers

More and more, systems engineers want to get out of the long drawn out business of integrating square code into a round production hole, eventually after much time has passed, molding the code into that round hole is possible. This usually takes months. Months that could have been avoided if the software engineer had designed the code in an environment where the production constraints exist.

That’s part of the reason for containers like Docker. When a container like Docker is used, the whole container can then be deployed without thought to square pegs in round holes. Instead, whatever flaws are in the Docker container are there for all to see because the developer put it there.

In other words, the middle folks who take code from engineering and mold it onto production gear don’t relish the thought of ironing out hundreds of glitchy problems until it seamlessly all works. Sure, it’s a job, but at some level it’s also a bit janitorial, wasteful and a unnecessary.

Planning

Part of the reason for these problems is the delineation between the engineering teams and the production operations teams. Because many organizations separate these two functional teams, it forces the above problem. Instead, these two teams should be merged into one and work together from project and code inception.

When a new project needs code to be built that will eventually be deployed, the production team should be there to move the software architecture onto the right path and be able to choose the correct path for that code all throughout its design and building phases. In fact, every company should mandate that its software engineers be a client of operations team. Meaning, they’re writing code for operations, not the customer (even though the features eventually benefit the customer).

The point here is that the code’s functionality is designed for the customer, but the deploying and running that code is entirely for the operations team. Yet, so many software engineers don’t even give a single thought to how much the operations team will be required support that code going forward.

Operational Support

For every component needed to support a specific piece of software, there needs to be a likewise knowledgeable person on the operations team to support that component. Not only do they need to understand that it exists in the environment, the need to understand its failure states, its recovery strategies, its backup strategies, its monitoring strategies and everything else in between.

This is also yet another problem that software engineers typically fail to address in their code design. Ultimately, your code isn’t just to run on your notebook for you. It must run on a set of equipment and systems that will serve perhaps millions of users. It must be written in ways that are fail safe, recoverable, redundant, scalable, monitorable, deployable and stable. These are the things that the operations team folks are concerned with and that’s what they are paid to do.

For each new code deployment, that makes the environment just that much more complex.

The Stacked Approach

This is an issue that happens over time. No software engineer wants to work on someone else’s code. Instead, it’s much easier to write something new and from scratch. It’s easy for software engineer, but it’s difficult for the operations team. As these new pieces of code get written and deployed, it drastically increases the technical debt and burden on the operations staff. Meaning, it pushes the problems off onto the operations team to continue supporting more and more and more components if none ever get rewritten or retired.

In one organization where I worked, we had such an approach to new code deployment. It made for a spider’s web mess of an environment. We had so many environments and so few operations staff to support it, the on-call staff were overwhelmed with the amount of incessant pages from so many of these components.

That’s partly because the environment was unstable, but that’s partly because it was a house of cards. You shift one card and the whole thing tumbles.

Software stacking might seem like a good strategy from an engineering perspective, but then the software engineers don’t have to first line support it. Sometimes they don’t have to support it at all. Yes, stacking makes code writing and deployment much simpler.

How many times can engineering team do this before the house of cards tumbles? Software stacking is not an ideal any software engineering team should endorse. In fact, it’s simply comes down to laziness. You’re a software engineer because writing code is hard, not because it is easy. You should always do the right thing even if it takes more time.

Burden Shifting

While this is related to software stacking, it is separate and must be discussed separately. We called this problem, “Throwing shit over the fence”. It happens a whole lot more often that one might like to realize. When designing in a bubble, it’s really easy to call “code complete” and “throw it all over the fence” as someone else’s problem.

While I understand this behavior, it has no place in any professionally run organization. Yet, I’ve seen so many engineering team managers endorse this practice. They simply want their team off of that project because “their job is done”, so they can move them onto the next project.

You can’t just throw shit over the fence and expect it all to just magically work on the production side. Worse, I’ve had software engineers actually ask my input into the use of specific software components in their software design. Then, when their project failed because that component didn’t work properly, they threw me under the bus for that choice. Nope, that not my issue. If your code doesn’t work, that’s a coding and architecture problem, not a component problem. If that open source component didn’t work in real life for other organizations, it wouldn’t be distributed around the world. If a software engineer can’t make that component work properly, that’s a coding and software design problem, not an integration or operational problem. Choosing software components should be the software engineer’s choice to use whatever is necessary to make their software system work correctly.

Operations Team

The operations team is the lifeblood of any organization. If the operations team isn’t given the tools to get their job done properly, that’s a problem with the organization as a whole. The operations team is the third hand recipient of someone else’s work. We step in and fix problems many times without any knowledge of the component or the software. We do this sometimes by deductive logic, trial and error, sometimes by documentation (if it exists) and sometimes with the help of a software engineer on the phone.

We use all available avenues at our disposal to get that software functioning. In the middle of the night the flow of information can be limited. This means longer troubleshooting times, depending on the skill level of the person triaging the situation.

Many organizations treat its operations team as a bane, as a burden, as something that shouldn’t exist, but does out of necessity. Instead of treating the operations team as second class citizens, treat this team with all of the importance that it deserves. This degrading view typically comes top down from the management team. The operations team is not a burden nor is it simply there out of necessity. It exists to keep your organization operational and functioning. It keeps customer data accessible, reliable, redundant and available. It is responsible for long term backups, storage and retrieval. It’s responsible for the security of that data and making sure spying eyes can’t get to it. It is ultimately responsible to make sure the customer experience remains at a high excellence standard.

If you recognize this problem in your organization, it’s on you to try and make change here. Operations exists because the company needs that job role. Computers don’t run themselves. They run because of dedicated personnel who make it their job and passion to make sure those computers stay online, accessible and remain 100% available.

Your company’s uptime metrics are directly impacted by the quality of your operations team staff members. These are the folks using the digital equivalent of chewing gum and shoelaces to keep the system operating. They spend many a sleepless night keeping these systems online. And, they do so without much, if any thanks. It’s all simply part of the job.

Software Engineer and Care

It’s on each and every software engineer to care about their fellow co-workers. Tossing code over the fence assuming there’s someone on the other side to catch it is insane. It’s an insanity that has run for far too long in many organizations. It’s an insanity that needs to be stopped and the trend needs to reverse.

In fact, by merging the software engineering and operations teams into one, it will stop. It will stop by merit of having the same bosses operating both teams. I’m not talking about at a VP level only. I’m talking about software engineering managers need to take on the operational burden of the components they design and build. They need to understand and handle day-to-day operations of these components. They need to wear pagers and understand just how much operational work their component is.

Only then can engineering organizations change for the positive.

As always, if you can identify with what you’ve read, I encourage you to like and leave a comment below. Please share with your friends as well.

I know I’ve been on a tirade with the number of rants recently, but here we are. I rant when there’s something to rant about. This time it’s about sharing Apple Music playlists on Twitter… and just how badly this feature is broken. Worse, just how Apple itself is broken. Let’s explore.

Twitter Cards

Twitter has a feature they call Twitter cards. It’s well documented and requires a number of meta tags to be present in an HTML page. When the page is shared via Twitter, Twitter goes looking at the HTML for its respective Twitter meta tags to generate a Twitter card.

A Twitter card comes in two sizes and looks something like this:

Small Twitter Card

Large Twitter Card

What determines the size of the Twitter card seems to be the size and ratio of the image. If the image is square in size (144×144 or larger), Twitter creates a small card as shown at the top. If the image ratio is not square and larger than 144×144, Twitter produces a large Twitter card. The difference between the cards is obvious:

Small card has an image to the left and text to the right

Large card has image above and text below

It’s up to the person sharing on Twitter to decide which size is most appropriate. Personally, I prefer the larger size because it allows for a much larger image.

What’s wrong with this Twitter card? If you guessed the image is way too tiny, you’d win. Apple doesn’tunderstand the concept of producing a 144×144 image properly. Here’s the fundamental problem. In iTunes, my playlist image is uploaded with a 1200×1200 size image. This image is well large enough for any use on the net. Here’s how it looks in iTunes, albeit scaled somewhat small:

Note, iTunes retains the full image size, but scales the image as needed. If you look at the playlist on the web, it looks like this with a much larger scaled image:

As you can see, the image scales properly and still looks good even larger. Yes, even large enough to produce a 144×144 image on a Twitter card.

You’ll notice that the text in red above is the piece that is relevant. Let’s look at that image now…

Scaled. Click to see 600×600 image

You’ll notice that the playlist image content is centered at 213×213 pixels in size centered in a light grey box that’s 600×600. Yes, that thick light grey border is part of the image. This is actually how the image is being produced by Apple on their servers. That would be okay if the image were scaled to the full 600×600 pixels. Unfortunately, it isn’t. Twitter will scale any image to its preferred size of 144×144 pixels for small Twitter cards. Here’s what a 144×144 image looks like when scaled by WordPress:

I have no idea what Twitter is using to scale its images, but it looks like absolute trash. The bigger problem isn’t that Twitter has scaled this image down, it’s that Apple has provided Twitter with such an already small and crap looking playlist image. Why have a 144×144 image if you’re only going to use 1/9th of the entire space? Apple, why wouldn’t you not want to use the entire 144×144 image space to make the image look like this:

That sized image would make the Twitter card look like this…

… instead of this absolute shit looking card…

How the Mighty Have Fallen

Apple used to be a well respected company who always prided itself on doing things correctly and producing high quality products. Today, they’re a shadow of their former selves. Producing products as crap as this only serves as a detriment to all of the other products they now offer. It’s clear, Apple Music is an afterthought and Apple seems to have only one engineer assigned to this software product… maybe none.

It’s also clear, Apple doesn’t respect the standards of anyone, not even themselves. I consider this absolute crapattention to detail. Seriously, who wants their images to be scaled to the point of unreadability? No one!

Yet, when I called Apple Support to report this issue, I was told, “This is expected behavior”. Expected by whom? Who would ever expect an image to be scaled the point of nonrecognition? No one. If this is the level of software development effort we’re now seeing from Apple, then I don’t even want to think what corners are being cut on their hardware products.

What’s next? Apple watches catching on fire and exploding on people’s wrists? Phones taking out people’s ears? If I can no longer trust Apple to uphold the standards of high quality, then the mighty have truly fallen. There is no hope for Apple no matter how much crap they try to peddle.

Apple, Hear Me!

If you are serious about your business, then you need to be serious about all aspects including offering high quality products, services and features. This goes all the way to playlist sharing on Twitter. My experience with dealing with Apple in this matter was so amateur including the way Apple Music itself is being handled, why should I continue to use this product? Give me a reason to pay you $99 for such shit! Seriously, in addition to the above, I’m also finding what appear to be bootlegged music products on Apple Music and yet you’re pawning it off as official releases?

And as suggested by your representative, why should I contact Twitter for this issue? Twitter’s features work properly when provided with the correct information. As has been stated for years in software engineering, “Garbage In, Garbage Out”. It is you, Apple, who are providing Twitter with garbage information. It’s not a Twitter problem, it’s an Apple problem. Also, because this is an Apple engineering problem to solve, why should I contact Twitter on Apple’s behalf? I don’t work for you. You need to have YOUR engineering team contact Twitter and have them explain to you the errors of your ways.

This is just the tip of the iceberg here. There’s so much wrong at Apple, if you continue to entrust your family’s safety into Apple’s products, you may find one of your family members injured or dead. Apple, wake up and learn to take quality seriously.

Then next time you are shopping for a computer or a watch devices, you need to ask yourself, “Do I really trust Apple to provide safe choices?”