STATUS: I built a Twitter search engine for businesses. The shit you’d do to get your startup off the ground.

MOOD: AWESOME

—————

1. I am SUPER happy that marriage doesn’t just mean man + woman anymore. We’ve just witnessed our own Brown vs Board of Education, and no blood was shed! What a time to be alive.

2. I made this post on reddit, thought it’d make for an interesting blog post:

Okay, so before I write my post, I’ll be upfront and say that I’ve had a few beers while trying to battle Twitter’s OAuth implementation. I won. Anyone interested in a business to Twitter username search engine?

If you’re a non-technical person, learn how to code. Seriously, just learn. It’s a time investment and will take a while for you to get right…but so is this startup building shit. I can’t imagine ever wanting to put the core technology of my business into someone else’s hands, and I can’t imagine hiring someone to build something that I can’t understand.

Programming languages are WAY easier to learn than they used to be. Python is an incredibly accessible language that is EXTREMELY powerful. Entire businesses (i.e. Google) were built on it. Entire businesses are devoted to teaching it (i.e. CodeAcademy). Again, it’s a time commitment and your code will BLOW CHUNKS at first, but it’ll get better.

Honestly, I wouldn’t even let this discourage you. You WON’T FUCKING BELIEVE just how much EXTREMELY powerful software used by household businesses is using 100% n00b code. The code isn’t important; delivering the product is

(more disclosure: I’m using PowerShell for my MVP though I’m planning on porting it to C# when it’s live after a few weeks of fund-raising)

EDIT: EVEN MORE DISCLOUSURE I “learned” Assembly, C and C++ in colleges. Those programming languages are hard to learn. If you’re reason for not learning how to program is “I took a class on C and it sucked ass,” things are better now, I promise. (Yes, I learned some very valuable programming idioms that would be difficult to learn in higher-level languages. Things like pointer arithmetic and manual memory allocation. Things that 9 times out of 10 WON’T prevent you from starting up your startup.)

STATUS: I played the role of Thinglistr and posted events on Twitter using my early thinglistr tech. This is when I go on Facebook, Yelp and Twitter for local businesses and find out if they’re doing anything, then post it on Twitter if they are. This process normally takes me two hours for 20 businesses.

Today, it took 15 minutes. Computers are great.

MOOD:

——————-

The subject of what I do for work often comes up when I go out. Despite having written tens of thousands of lines of code to do all sorts of business magic and doing software engineer-y things to prevent that code from circling out of control, I always felt that calling myself a software engineer would be lying. Those thoughts would look like these:

“I’m not good enough.”

“They do way more complicated things than I do.”

“I’d fail the interview in two seconds if I tried getting a real programming job.”

In other words, I felt much like this:

so I spent years telling people that I’m a systems administrator, and a god awfully many number of minutes attempting (and mostly failing) to explain what the hell that is.

The truth is that I spent a lot of time lying to myself. There are seemingly infintely many articles on the Internet that attempt to educate people on what a “good developer” looks like.

Want to know what I think? Good software developers are the ones that writes easily-manageable code for stuff that people want in reasonable time.

The number of programming languages (or the languages that you know, for that matter) doesn’t matter, though it helps.

The number of data structures and algorithms you know don’t matter, though it helps.

The number of git incantations you know or Linux kernel hacks you spout don’t matter, though it helps.

None of that matters.

Are you making stuff that people want? Are you make them on time? Are you listening to them and refining where needed? Can others on your team edit your code if/when needed?

If so, then congratulations: you are a great developer!

This seems tangential to my startup journey, but I wrote about it because the thoughts I mentioned above were the same ones that prevented me from traveling down this road. I thought I wasn’t smart enough to do something like this, but here I am.

There are plenty of people that can probably develop what I’m making in a weekend on nothing more than a 12-pack and some Ramen. There are also plenty of people that can roll their own Linux kernel in their sleep. I’m not one of them, and that’s okay. I know what I want to see, and I know that I can make it happen. That’s good enough for me.

STATUS: Welp, I wasted an entire day trying to write multi-threaded PowerShell code. In other words, I moved two steps forward and two steps back.MOOD: I actually feel surprisingly alright with this.

——————

Let’s get technical for a second.

I am using PowerShell to create the MVP for thinglistr’s backend because it’s the language that I know most extensively right now. I’m also a C# and F# developer (I miss writing in F# A LOT), but I needed to get this thing in the air as quickly as possible, and PowerShell is much easier to write and test in a SSH session than F# (despite it having a console, `fsi`) or C# are, so I chose that. (Before you say “Y U NO PYTHON,” my Python knowledge pales hard in comparison to my PowerShell knowledge and I wanted to spend more time building than learning the semantics of a lanuage. I’m also pretty set in using C# or F# for production (F# is great for transactions and multi-threaded code; immutability by default helps a lot).

This setup has worked pretty well so far. Working with RESTful queries and caching things (primitively) is pretty straightforward, and it does the backend stuff well enough to satisfy my needs right now. That didn’t stop me from wondering whether I could make things a little faster, though. It would be much more efficient for the business polling and review parsing code to be multi-threaded, so I spent some time in attempting to make the business polling code multi-threaded.

Big mistake. Sort of.

Unlike Python, PowerShell doesn’t have any native support for .NET threading. I’m guessing that this is so because much of the underlying language primitives and/or the runspace within which PowerShell session state is saved is not thread-safe nor were they designed to be consumed as raw .NET threads to begin with. In its place, PowerShell uses “Jobs,” which are essentially scriptblocks or scripts executed within forked PowerShell sessions whose results are streamed back to their parent PowerShell session. While they’re better than having nothing at all, they are quite limiting in a few ways:

– PowerShell sessions usually consume about 20MB of RAM (or more depending on your $PROFILE and Modules directory), whereas .NET threads take up about 1MB.
– Every PowerShell session loads your $PROFILE, which can be highly, highly wasteful depending on its size and can delay initialization.
– Scripblocks sent to every child session runs within isolated runspaces, so sharing objects or state from the parent session is impossile without some modification.

(I know that PowerShell 4.0+ supports workflows, but they are also based on the Jobs engine and have the same set of problems and a different set of quirks, and I know how to work with Jobs better.

Knowing these limitations, I decided on spending some time creating a custom job queue implementation for Powershell that:

– Creates the concept of job pools by creating jobs that share common identifiers,
– Throttles the addition of jobs into job pools automatically when they go above a certain threshold,
– Reloads dependencies in the runspaces for every job,
– Serializes function calls and their arguments so that the consumer of the pool doesn’t have to, and
– Outputs errors and results from jobs within the pool as a pscustomobject for easy manipulation.

It took >10 infuriating hours for me to get this working right, but I eventually got it…and found it to be WORSE than what I had before for a few reasons:

– My AWS instance is very memory-constrained (only has 1GB of RAM), so when taking OS overhead into account, I only needed a few concurrent PowerShell sessions to send my machine into swap city (10 powershell sessions = ~200 threads = 200 MB).
– PowerShell jobs don’t have access to the parent host console because of the whole isolated runspace thing, so there’s no way for me to easily debug issues within threads when they come up until AFTER they’re done executing. This could be problematic if issues arise while retrieving business data from Google sincetheir API request quota is really low for server-side requests (2000 requests/day).
– God it was so fucking slow.

What was funny was that I wasn’t that upset about this. I was much more upset whenever I ran into a bug and thought that I’d have to spend my entire weekend on this. Now that I know that this approach sucks (for now; I think it’ll be much more useful when I move this onto a real programming language), because I still have the original code that does this serially, I can spend today focusing on the real Goliath: event discovery within Twitter.