tag:blogger.com,1999:blog-49001866016612154662018-02-20T14:22:08.422-08:00Erik McClureThoughts from a code-wielding applied mathematicianErik McClurenoreply@blogger.comBlogger176125tag:blogger.com,1999:blog-4900186601661215466.post-14692326462406586912018-02-06T09:22:00.000-08:002018-02-06T09:22:36.505-08:00Migrating To A Static BlogI've finished constructing a new personal website for myself using hugo, and I'm moving my blog over there so I have more control over what gets loaded, and more importantly, so the page doesn't attempt to load Blogger's 5 MB worth of bloated javascript nonsense just to read some text. It also fixes math and code highlighting while reading on mobile. If you reached this post using Blogger, you'll be redirected or will soon be redirected to the corresponding post on my new website. All comments have been preserved from the original posts, but making new comments is currently disabled - I haven't decided if I want to use Disqus or attempt something else. An RSS feed is available on the bottom of the page for tracking new posts that should mimic the Blogger RSS feed, if you were using that. If something doesn't work, poke me on twitter and I'll try to fix it. I implemented share buttons with simple links, without embedding any crazy javascript bullshit. In fact, the only external resource loaded is a Google tracking ID for pageviews. Cloudflare is used to enforce an HTTPS connection over the custom domain even though the website is hosted on Github Pages. Hopefully, the new font and layout is easier to read than Blogger's tiny text and bullshit theme nonsense.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-91202136290828808862018-01-01T11:37:00.003-08:002018-01-01T11:49:02.956-08:00How To Avoid Memorizing Times TablesI was recently told that my niece was trying to memorize her times tables. As an applied mathematician whose coding involves plenty of multiplication, I was not happy to hear this. Nobody who does math actually memorizes times tables, and furthermore, forcing a child to memorize <i>anything</i> is probably the worst possible thing you can do in modern society. No one should <i>memorize</i> their times tables, they should learn how to <i>calculate them</i>. Forcing children to memorize useless equations for no reason is a great way to either ensure they hate math, teach them they should blindly memorize and believe anything adults tell them, or both. So for any parents who wish to teach their children how to be critical thinkers and give them an advantage on their next math test, I am going to describe how to derive the entire times tables with only 12 rules. <ol><li>Anything multiplied by 1 is itself. Note that I said <i>anything</i>, that includes fractions, pies, cars, the moon, or anything else you can think of. Multiplying it by 1 just gives you back the same result.</li><li>Any number multiplied by 10 has a zero added on the end. 1 becomes 10, 2 becomes 20, 72 becomes 720, 9999 becomes 99990, etc.</li><li>Any single digit multiplied by 11 simply adds itself on the end instead of 0. 1 becomes 11, 2 becomes 22, 5 becomes 55, etc. This is because you never need to multiply something by eleven. Instead, multiply it by 10 (add a zero to it) then add itself. \[11*11 = 11*(10 + 1) = 11*10 + 11 = 110 + 11 = 121\\ 12*11 = 12*(10 + 1) = 12*10 + 12 = 120 + 12 = 132\]</li><li>You can always reverse the numbers being multiplied and the same result comes out. $$12*2 = 2*12$$, $$8*7 = 7*8$$, etc. This is a simple rule, but it's very easy to forget, so keep it in mind.</li><li>Anything multiplied by 2 is doubled, or added to itself, but you only need to do this up to 9. For example, $$4*2 = 4 + 4 = 8$$. Alternatively, you can count up by 2 that many times: \[4*2 = 2 + 2 + 2 + 2 = 4 + 2 + 2 = 6 + 2 = 8\] To multiply any large number by two, double each individual digit and <i>carry the result</i>. Because you multiply each digit by 2 separately, the highest result you can get from this is 18, so you will only ever carry a 1, just like in addition. \[\begin{matrix} 3 & 6\\ & 2\\ \hline & \\ & \\ \hline & \end{matrix}\quad \begin{matrix} 3 & 6\\ & 2\\ \hline 1 & 2\\ & \\ \hline & \end{matrix}\quad \begin{matrix} 3 & 6\\ & 2\\ \hline 1 & 2\\ 6 & \\ \hline & \end{matrix}\quad \begin{matrix} 3 & 6\\ & 2\\ \hline 1 & 2\\ 6 & \\ \hline 7 & 2 \end{matrix}\] This method is why multiplying anything by 2 is one of the easiest operations in math, and as a result the rest of our times table rules are going to rely heavily on it. Don't worry about memorizing these results - you'll memorize them whether you want to or not simply because of how often you use them.</li><li>Any number multiplied by 3 is multiplied by 2 and then added to itself. For example: \[6*3 = 6*(2 + 1) = 6*2 + 6 = 12 + 6 = 18\]Alternatively, you can add the number to itself 3 times: $$3*3 = 3 + 3 + 3 = 6 + 3 = 9$$</li><li>Any number multiplied by 4 is simply multiplied by 2 twice. For example: $$7*4 = 7*2*2 = 14*2 = 28$$</li><li>Any number multiplied by 5 is the same number multiplied by 4 and then added to itself. \[6*5 = 6*(4 + 1) = 6*4 + 6 = 6*2*2 + 6 = 12*2 + 6 = 24 + 6 = 30\] Note that I used our rule for 4 here to break it up and calculate it using only 2. Once kids learn division, they will notice that it is often easier to calculate 5 by multiplying by 10 and halving the result, but we assume no knowledge of division.</li><li>Any number multiplied by 8 is multiplied by 4 and then by 2, which means it's actually just multiplied by 2 three times. For example: $$7*8 = 7*4*2 = 7*2*2*2 = 14*2*2 = 28*2 = 56$$ <li>Never multiply anything by 12. Instead, multiply it by 10, then add itself multiplied by 2. For example: $$12*12 = 12*(10 + 2) = 12*10 + 12*2 = 120 + 24 = 144$$</li><li>Multiplying any single digit number by 9 results in a number whose digits always add up to nine, and whose digits decrease in the right column while increasing in the left column. \[9 * 1 = 09\\ 9 * 2 = 18\\ 9 * 3 = 27\\ 9 * 4 = 36\\ 9 * 5 = 45\\ 9 * 6 = 54\\ 9 * 7 = 63\\ 9 * 8 = 72\\ 9 * 9 = 81\]10, 11, and 12 can be calculated using rules for those numbers.</li><li>For both 6 and 7, we already have rules for all the other numbers, so you just need to memorize 3 results: \[6*6 = 36\\ 6*7 = 42\\ 7*7 = 49\]Note that $$7*6 = 6*7 = 42$$. This is where people often forget about being able to reverse the numbers. Every single other multiplication involving 7 or 6 can be calculated using a rule for another number.</li></ol>And there you have it. Instead of trying to memorize a bunch of numbers, kids can learn <i>rules</i> that build on top of each other, each taking advantage of the rules established before it. It's much more engaging then trying to memorize a giant table of meaningless numbers, a task that's so mind-numbingly boring I can't imagine forcing an adult to do it, let alone a small child. More importantly, this task teaches you what math is <i>really</i> about. It's not about numbers, or adding things together, or memorizing a bunch of formulas. It's establishing simple rules, and then combining those rules together into more complex rules you can use to solve more complex problems. <br/><br/>This also establishes a fundamental connection to computer science that is often glossed over. Both math and programming are <b>repeated abstraction and generalization</b>. It's about combining simple rules into a more generalized rule, which can then be abstracted into a simpler form and combined to create even more complex rules. Programs start with machine instructions, while math starts with propositions. Programs have functions, and math has theorems. Both build on top of previous results to create more powerful and expressive tools. Both require a spark of creativity to recognize similarities between seemingly unrelated concepts and unite them in a more generalized framework. <br/><br/>We can demonstrate all of this simply by refusing to memorize our times tables.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com1tag:blogger.com,1999:blog-4900186601661215466.post-44422266676360927372017-11-01T15:26:00.000-07:002017-11-01T15:33:38.107-07:00Ignoring Outliers Creates Racist AlgorithmsHave you built an algorithm that <i>mostly</i> works? Does it account for <i>almost</i> everyone's needs, save for a few weird outliers that you ignore because they make up 0.0001% of the population? Congratulations, your algorithm is racist! To illustrate how this happens, let's take a recent example from Facebook. My friend's message was removed for "violating community standards". Now, my friend has had all sorts of ridiculous problems with Facebook, so to test my theory, I posted the exact same message on my page, and then had him report it.<br /><br /><img src="https://blackhole12.github.io/storage/fb1.png" alt="" width="637" /><br /><img src="https://blackhole12.github.io/storage/fb2.png" alt="" /><br /><img src="https://blackhole12.github.io/storage/fb3.png" alt="" /><br /><br />Golly gee, look at that, Facebook confirmed the message I sent does <b>not</b> violate community guidelines, but he's still banned for 12 hours for posting <i>the exact same thing</i>. What I suspect happened is this: Facebook has gotten mad at my friend for having a weird name multiple times, but he can't prove what his name is because he doesn't have access to his birth certificate because of family problems, <i>and</i> he thinks someone's been falsely reporting a bunch of his messages. The algorithm for determining whether or not something is "bad" probably took these misleading inputs, combined it with a short list of so-called "dangerous" topics like "terrorism", and then decided that if anyone reported one of his messages, it was <i>probably</i> bad. On the other hand, I have a very western name and nobody reports anything I post, so either the report actually made it to a human being, or the algorithm simply decided it was probably fine.<br /><br />Of course, the algorithm was wrong about my friend's message. But Facebook doesn't care. I'm sure a bunch of self-important programmers are just itching to tell me we can't deal with all the edge-cases in a commercial algorithm because it's infeasible to account for all of them. What I want to know is, have any of these engineers ever thought about <b>who the edge-cases are?</b> Have they ever thought about the kind of people who can't produce birth certificates, or don't have a driver's license, or have strange names <a href="https://modelviewculture.com/pieces/i-can-text-you-a-pile-of-poo-but-i-cant-write-my-name">that don't map to unicode properly</a> because they aren't western enough?<br /><br />Poor people. Minorities. Immigrants. Disabled people. All these people they claim to care about, all this talk of diversity and equal opportunity and inclusive policies, and they're building algorithms that <i>by their very nature</i> will exclude those less fortunate than them. Facebook's algorithm probably doesn't even know that my friend is asian, yet it's still discriminating against him. Do you know who can follow all those rules and assumptions they make about normal people? Rich people. White people. Privileged people. These algorithms benefit those who don't need help, and disproportionately punish those who don't need any more problems.<br /><br />What's truly terrifying is that <a href="https://www.nytimes.com/2017/10/18/upshot/taxibots-sensors-and-self-driving-shuttles-a-glimpse-at-an-internet-city-in-toronto.html">Silicon Valley wants to run the world</a>, and it wants to automate everything using a bunch of inherently flawed algorithms. Algorithms that might be <i>impossible</i> to perfect, given the almost unlimited number of edge-cases that reality can come up with. In fact, as I am writing this article, Chrome doesn't recognize "outlier" as a word, even though <a href="https://www.google.com/search?q=outlier">Google itself does</a>.<br /><br />Of course, despite this, Facebook already built an algorithm that tries to detect "toxicity" and silences "unacceptable" opinions. Even if they could build a perfect algorithm for detecting "bad speech", do these companies really think forcibly restricting free speech will accomplish anything other than improving their own self-image? A deeply cynical part of me thinks the only thing these companies actually care about is looking good. A slightly more optimistic part of me thinks a bunch of well-meaning engineers are simply being stupid.<br /><br />You can't change someone's mind by punching them in the face. Punching people in the face may shut them up, but it does not change their opinion. It doesn't <i>fix</i> anything. <i>Talking</i> to them does. I'm tired of this industry hiding problems behind shiny exteriors instead of fixing them. That's what used car salesmen do, not engineers. Programming has devolved into an art of deceit, where coders hide behind pretty animations and huge frameworks that sweep all their problems under the rug, while simultaneously screwing over the people who were supposed to benefit from an "egalitarian" industry that seems less and less egalitarian by the day.<br /><br />Either silicon valley needs to start dealing with people that don't fit in neat little boxes, or it will no longer be able to push humanity forward. If we're going to move forward as a species, we have to do it <b>together</b>. Launching a bunch of rich people into space doesn't accomplish anything. Curing cancer for rich people doesn't accomplish anything. Inventing immortality for rich people doesn't accomplish anything. If we're going to push humanity forward, we have to push <i>everyone</i> forward, and that means dealing with all 7 billion outliers.<br /><br />I hope silicon valley doesn't drag us back to the feudal age, but I'm beginning to think <a href="https://medium.com/@ebonstorm/feudalism-and-the-algorithmic-economy-62d6c5d90646">it already has</a>.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-53660975071399115152017-10-10T02:24:00.000-07:002017-10-10T02:24:11.101-07:00My Little Pony And The Downfall Of Western CivilizationOur current political climate is best described as a bunch of chimpanzees angrily throwing feces at each other. Many people recognize that there are many problems with modern politics, but few agree on what those problems actually are. Liberals, republicans, Trump, millenials, capitalism, communism, too many regulations, too few regulations, gerrymandering, illegal immigrants, rich people, poor people, schools, governments, religion, secularism, the list goes on and on and on. These, however, are not <i>problems</i>, they are <i>excuses</i>. They are scapegoats we have conjured up from our mental list of things we don't like so we can ignore the ugly truth.<br /><br />At some point, Americans are going to have to admit that the entire problem with this country has been summed up in a TV show for six-year-old girls designed to sell toys. My Little Pony was released seven years ago, and as the series has progressed, it has focused more and more on reforming villains instead of defeating them. Time and time again, ponies refuse to give up on those who seem lost to darkness and try to figure out what is causing them to lash out. A central theme of the show is that almost no one is truly a "villain", only misguided, misunderstood, or pushed towards lashing out after a traumatic experience. While a select few villains have been show to be truly irredeemable, this is rare, which is in line with reality. Only a very small percentage of the human population is deliberately evil as opposed to simply lashing out, being stupid, or being tragically misinformed.<br /><br />It is no longer possible to argue with anyone anymore. This is not because everyone suddenly became incapable of critical thought, but because <b>no one can agree on any facts</b>. When we built the internet, we thought having access to the sum of all human knowledge would bring infinite prosperity, but instead it brought us infinite <i>misinformation</i>. Russians have been making this worse, feeding false narratives to <i>both</i> sides to make us hate each other. <b>It worked</b>. When scientists themselves have been manipulated by corporations without anyone bothering to attempt to reproduce experiments, there simply isn't any good way to verify the truth of anything. The entire point of the scientific method is to make sure something is reproducible, but <a href="http://rogerpielkejr.blogspot.com/2011/08/surging-retractions-in-scientific.html">a staggering number of retractions in recent years</a> has demonstrated that barely anyone actually checks anyone else's work. This has fostered a general distrust in science, even for results that <i>have</i> been reproduced thousands of times, like the <i>thoroughly</i> debunked "Vaccines cause autism!" claim.<br /><br />This kind of political polarization is untenable. We no longer live in a world where we can get into a fight, stab someone else with a sword and call it good. If our political polarization goes unchecked, it will result in nothing less than the <b>total collapse of western civilization</b>. Humanity will have proven itself too dumb and tribalist to wield a tool as powerful as the internet. Whatever civilization comes after us will struggle to reclaim the technological progress we enjoy, now that we've stripped the planet of resources. If we fail now, humanity will never again be capable of reaching for the stars. We will have sentenced ourselves to live on this small rock until the sun boils the oceans away, doomed by our own stupidity.<br /><br />There was an episode of My Little Pony that explained the origin of their country, Equestria. The three races, Pegasi, Earth ponies, and Unicorns, <i>hated</i> each other. Evil forces fed on their hatred, smothering the land in snow and threatening to freeze them all to death. So, each race set out to find a new land to colonize, only to suddenly realize that each of them had wound up on the exact same new continent. The leaders of each race immediately started yelling at each other, and the blizzard returned, until each of them was encased in ice. Only when the assistants of each leader realized they didn't actually hate each other were they able to dispel the evil forces by starving them of hatred and talking sense into their leaders. <b>The moral of this story is very clear:</b> either we figure out how to get along, or <b>we're all going to die</b>. I really, honestly don't know how to explain this better than a saturday morning cartoon show about magical ponies.<br /><br />The problem is that I don't know if humanity is capable of moving past this. I've seen one of my friends rapidly devolve into insane conspiracy theory nonsense, and I simply don't have the mental willpower to engage with them. I eventually had to block them, and accomplished two things at once: reinforcing their echo chamber and reinforcing <i>my</i> echo chamber. I tried to hold on to them as a window into conservative nonsense so my twitter wasn't a complete echo chamber, but when the other side is saying truly horrible things about you, this becomes more and more difficult. It also make it more likely for me to say truly horrible things about the other side, in a vicious, endless cycle, and I simply have too many other things to worry about to be capable of dealing with that level of toxicity. At this stage, I'm not sure humanity has the strength to actually reconcile with itself. I think there is a real possibility that our worries about AI were misplaced - the technology that might ultimately destroy us could be the internet, simply because our tribalist brains are too desperate to find someone to hate. We may be fundamentally incapable of sustaining a global, interconnected society with instantaneous communication.<br /><br />At least then we'll have a very definitive answer to the Fermi Paradox.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-71099496752885799592017-09-05T01:04:00.001-07:002017-09-06T12:07:38.842-07:00I Used To Want To Work For GoogleA long time ago I thought Google was this magical company that truly cared about engineering and solving problems instead of maximizing shareholder value. Then Larry Page became CEO and <a href="https://blackhole12.blogspot.com/2013/11/googles-decline-really-bugs-me.html">I realized they were not a magical unicorn</a> and lamented the fact that they had been transformed into "just another large company". Several important things happened between that post and now: Microsoft got a new CEO, so I decided to give them a shot and got hired there. I quit right before Windows 10 came out because I knew it was going to be a disaster. More recently, it's become apparent that Google had gone far past simply being a behemoth unconcerned with the cries of the helpless and transformed into something outright malevolent. It's <a href="http://gizmodo.com/yes-google-uses-its-power-to-quash-ideas-it-doesn-t-li-1798646437">silenced multiple reporters</a>, <a href="https://thenextweb.com/insider/2013/01/05/calling-shenanigans-on-googles-windows-phone-8-maps-narrative/">blocked windows phone from accessing youtube out of spite</a>, and <a href="https://www.nytimes.com/2017/08/30/us/politics/eric-schmidt-google-new-america.html?_r=0">successfully gotten an entire group of researchers fired by threatening to pull funding</a> (<a href="https://citizensagainstmonopoly.org/">but that didn't stop them</a>).<br /><br />This is <b>evil</b>. This is <i>horrifying</i>. This is the kind of stuff Microsoft did in the 90s that made everyone hate it so much they still have to fight against the repercussions of decisions made two decades ago because of the sheer amount of damage they did and lives they ruined. I'm at the point where I'd rather go back to Microsoft, whose primary sin at this point is mostly just being incompetent instead of outright evil, rather than Google, who is actually doing things that are fundamentally morally wrong. These are the kinds of decisions that are bald-faced abuses of power, without any possible "good intention" driving them. It's <b>vile</b>. There is no excuse.<br /><br />As an ex-Microsoft employee, I can assure you that at no point did I think Microsoft was doing something <i>evil</i> while I was there. I haven't seen Microsoft do anything outright <i>evil</i> since I left, either. The few times they came close they backed off and apologized later. Microsoft didn't piss people off by being evil, it pissed people off by being <i>dumb</i>. I was approached by a Google recruiter shortly after I left and I briefly considered going to Google because I considered them vastly more competent, and I still do. However, no amount of engineering competency can make me want to work for a company that actively does things I consider morally reprehensible. This is the same reason I will never work for Facebook. I've drawn a line in the sand, and I find myself in the surprising situation of being on the opposite side of Google, and discovering that <i>Microsoft</i>, of all companies, isn't with them.<br /><br />I always thought I'd be able to mostly disregard the questionable things that Google and Microsoft were doing and compare them purely on the competency of their engineers. However, it seems that Google has every intention of driving me away by doing things so utterly disgusting I could <i>never</i> work there and still be able to sleep at night. This worries me deeply, because as these companies get larger and larger, they eat up all the other sources of employment. Working at a startup that isn't one of the big 5 won't help if it gets bought out next month. One friend of mine with whom I shared many horror stories with worked at LinkedIn. He was not happy when he woke up one day to discover he now worked for the very company he had heard me complaining about. Even now, he's thinking of quitting, and not because Microsoft is evil - they're just so goddamn <i>dumb</i>.<br /><br />The problem is that there aren't many other options, short of starting your own company. Google is evil, Facebook is evil, Apple is evil if you care about open hardware, Microsoft is too stupid to be evil but might at some point become evil again, and Amazon is probably evil and may or may not treat it's employees like shit depending on who you ask. Even if you don't work directly for them, you're probably using their products or services. At some point, you have to put food on the table. This is why I generally refuse to blame someone for working for an evil company because the economy punishes you for trying to stand up for your morals. It's not the workers fault, here, it's Wall Street incentivizing rotten behavior by rewarding short-term profits instead of long-term growth. A free market optimizes to a monopoly. Monopolies are bad. I don't know what people don't get about this. We're fighting over stupid shit like transgender troops or gay rights instead of just treating other human beings with decency, all the while letting rich people rob us blind as they decimate the economy. This is stupid. I would daresay it's almost more stupid than the guy at Microsoft who decided to fire all the testers.<br /><br />But I guess I'll take unrelenting stupidity over <b>retaliating against researchers for criticizing you</b>. At least until Microsoft remembers how to be evil. Then I don't know what I'll do.<br /><br />I don't know what anyone will do.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com3tag:blogger.com,1999:blog-4900186601661215466.post-18829547831154398992017-08-06T12:20:00.001-07:002017-09-05T18:08:49.895-07:00Sexist Programmers Are Awful EngineersMen and women <a href="https://stanmed.stanford.edu/2017spring/how-mens-and-womens-brains-are-different.html">are fundamentally different</a>. So are white people and black people and autistic people and gay people and transgender people and conservatives and liberals and every other human being along every imaginable axis of discrimination. Some of these differences are cultural. Others are genetic. Others depend on environmental factors. These differences mean that some of us are inherently better at certain tasks than others. On average, men are better at spatial temporal reasoning, women are better at reading comprehension and writing ability, and <a href="https://www.forbes.com/sites/jeffbercovici/2011/06/14/why-some-psychopaths-make-great-ceos/#749d149f261a">psychopaths can sometimes be excellent CEOs</a>.<br /><br />Whenever I meet a programmer who insists on doing everything a certain way, the chances I'll hire them drop off a cliff. Just as object-oriented programming didn't fix everything, neither will functional programming, or data-oriented programming or array-based programming or any other language. They are different tools that allow you to attack a problem from different directions, much like we have different classes of algorithms to attack certain classes of problems. Greedy algorithms, lazy evaluation, dynamic programming, recursive-descent, maximum flow, all of these are different ways to approach a problem. They represent looking at a problem from <i>different perspectives</i>. A problem that is difficult from one angle might be trivial when examined from a different angle.<br /><br />When I stumbled upon this <a href="http://gizmodo.com/exclusive-heres-the-full-10-page-anti-diversity-screed-1797564320">anti-diversity memo written by a Google employee</a>, I wonder just how dysfunctional of an engineer that person is. Problems are never solved by being closed-minded. They are solved by opening ourselves to new possibilities and exploring the problem space as an infinitely-dimensional fabric of possible configurations. You do not find possible edge-cases by being closed-minded. You find them by probing the outer edges of your solution, trying to find singularities and inflection points that hint at unusual behavior.<br /><br />You cannot build a great company by hiring people who are good at the same things you are. Attempting to maximize diversity only comes at a perceived cost of aptitude if you are measuring the wrong things. If your concept of what makes a "good programmer" is an extremely narrow set of skills, then you will inevitably select towards a specific ethnicity, culture, or sex, because the tiny statistical differences will be grossly magnified by the extremely narrow job requirements. Demand that all your programmers invert a binary tree on a whiteboard and you'll filter out <a href="https://twitter.com/mxcl/status/608682016205344768?lang=en">the guy who wrote the software 90% of your company uses</a>.<br /><br />If you think the field of computer science is really this narrow, you're a terrible programmer. Turing completeness is a fundamental property of the universe, and we are only just beginning to explore the full implications of information theory, the foundations of type theory, NP-completeness, and the nature of computation itself. Disregarding other people because they can't do something without ever considering what they can do will only hurt your team, and your company. Diversity inclusion programs shouldn't try to hire more women and ethnic groups because they're the same, they should be trying to hire them <i>because they are different</i>.<br /><br />When hiring someone to complete a job, you should hire whoever is the best fit for the job. In a vacuum where there is a single task that needs to be completed, gender and ethnicity should be ignored in favor of a purely meritocratic assessment. However, if you have a company that must respond to a changing world, diversity can reveal solutions you never even knew existed. An established company like Google must actively seek to increase diversity so that it can explore new perspectives that may give it an edge over its rivals. They cannot select on a purely meritocratic basis, because all measures of merit would be based on what the company is <i>already good at</i>, not what it could be good at. You cannot explore new opportunities by hiring the same people.<br /><br />Intelligent people value feedback from people who think differently than them. This is why many executives will deliberately hire people they disagree with so they can have someone challenge their views. This helps avoid creating an echo-chamber, which is the ultimate irony of a memo that's called "Google’s Ideological Echo Chamber", because scrapping the diversity inclusion programs as the memo suggests would itself create a new echo-chamber. You can't remove an echo-chamber by removing diversity - the author's premise is self-defeating. If they had stuck with only claiming that conservative ideologies should not be discriminated against, they would have been correct. Unfortunately, telling everyone they shouldn't discriminate against your perspective, which itself involves discriminating against other perspectives, is by definition a contradiction.<br /><br />We aren't going to write better programs by doing the same thing we've been doing for the past 10 years. To improve is to change, and those who seek to become better software engineers must themselves embrace change, or they will be left behind to rot in the sewers of forgotten programs, maintaining rancid enterprise code for the rest of their lives. If we are unwilling to change who is writing the programs, we'll be stuck making the same thing over and over again. A business that thinks nothing needs to change is one ripe for disruption. If you really think only hiring white males who correctly answer all your questions about graph theory and B-trees will help your business in the long-term, you're an idiot.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com1tag:blogger.com,1999:blog-4900186601661215466.post-1553825329274700892017-07-30T16:57:00.002-07:002017-07-30T17:04:49.990-07:00Why I Never Built My SoundCloud KillerWhile the news of SoundCloud imploding are fairly recent, musicians and producers have had a beef with the music uploading site's direction for years. I still remember when SoundCloud only gave you a paltry 2 hours worth of free upload time and managed to convert my high quality lossless WAV files to the shittiest 128 kbps MP3 I've ever heard in my life. What really pissed me off was that they demanded a ridiculous $7 a month just to <i>double</i> your upload time. This is in contrast to <a href="http://www.newgrounds.com/audio/">Newgrounds</a>, a tiny website run by a dozen people with an audio portal built almost as an afterthought that still manages to be superior to <i>every single other offering</i>. It gives you unlimited space, for free, and lets you upload <i>your own MP3</i>, which allows me to upload properly encoded joint-stereo 128 kbps MP3 files, or much higher quality MP3s for songs I'm giving out for free.<br /><br />Obviously, Newgrounds is only able to offer unlimited free uploads because the audio portal just piggybacks on the rest of the site. However, I was so pissed off at SoundCloud's disgusting subscription offering that I actually ran the numbers in terms of what it would cost to store lossless FLAC encodings of songs using Amazon S3. These calculations are now out of date, so I've redone them for the purposes of this blog.<br /><br />The average size of an FLAC encoded song is around 60 MB, but we'll assume it's 80 MB as an upper-bound, and to include the cost of storing the joint-stereo 128 kbps streaming MP3, which is usually less than 10% the size (using OPUS would reduce this even more, but it is not supported on all browsers yet). Amazon offers the first 50 TB of storage at $0.023 per gigabyte, per month. This comes down to about $0.00184 per month, per song, in order to store the full uncompressed version. Now, obviously, we must also stream the song, but we're only streaming the low-quality version, which is 10% the size, which is about 7 MB in our example (7 MB + 70 MB is about 80 MB for storage). The vast majority of music producers on the website have almost no following, and most will be lucky to get a single viral hit. As an example, after being on SoundCloud for over 7 years, I have managed to amass a mere 100000 views <i>total</i>. If I somehow got 20000 views of my songs <i>every single month</i>, the total cost of streaming 140 GB from Amazon S3 at $0.05 per GB would be $7 per month. That's how much SoundCloud is charging just to double my storage space!<br /><br />This makes even less sense when you calculate that 6 hours of FLAC would be 4.7 GB, or about 5 GB including the 128 kbps streaming MP3s. 5 GB of <i>storage space</i> costs a pathetic <b>$0.12 cents a month</b> to store on Amazon S3! All of the costs come down to <i>bandwidth</i>, which is relatively fixed by how many people are listening to songs, not how many songs there are. This means, if I'm paying any music service for the right to store music on their servers, I should get <i>near unlimited storage space</i> (maybe put in a sanity check of 10 GB max per month to limit abuse). I will point out that <a href="https://clyp.it/premium-pricing">Clyp.it actually does this properly</a>, giving you 6 hours of storage space for free and unlimited storage space if you pay them $6 a month.<br /><br />Unfortunately, Clyp.it does not try to be SoundCloud as it has no comments, no reshares, and well, not much of anything, really. It's like a giant pastebin for sounds where you can follow people or favorite things. <b>It's also probably screwed.</b><br /><br />Even though I had a name and a website design, I never launched it because even if I could find a way to identify a copyrighted song via some sort of ContentID system, I couldn't make it work without the record industry's cooperation. The problem is that the system has to know what songs are illegal in order to block them in the first place. Otherwise, people could upload Justin Bieber songs with impunity and I'd still get sued out of existence. The hard part about making a site like SoundCloud isn't actually making the website, it's dealing with the insane, litigation-happy oligarchs that own the entire music industry.<br /><br />SoundCloud's ordeal is <a href="https://www.buzzfeed.com/ryanmac/inside-the-storm-at-soundcloud">mentioned in this article</a>. Surprisingly, it took until 2012 for them to realize they had to start making deals with the major music labels. It took until 2014 for many of those deals to actually happen, and they were not in SoundCloud's favor. A deal with Warner Music Group, closed in 2014, gave Warner a <b>3-5% stake in the company</b> and an undisclosed cut of ad-revenue, just so SoundCloud could have the privilege of <i>not being sued out of existence</i>. This wasn't even an investment round, it was just so SoundCloud could have Warner Music Group's catalog on the site and not get sued!<br /><br />At this point, you have to be either very naive or very rich to go up against an industry that can and will send an army of lawyers after you. The legal system is not in your favor. It will be used to crush you like a bug and there is nothing you can do about it, because of one fundamental problem: <b>You can't detect copyright infringement without access to the original copy</b>.<br /><br />Because of this, the music industry holds the entire world hostage with a kind of Catch-22: They demand you take down all copyright infringing material, but in order to figure out if something is copyright infringement, you need access to <i>their</i> songs, which they only give out on <i>their</i> terms, which are never in your favor.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com3tag:blogger.com,1999:blog-4900186601661215466.post-91683676688305060802017-07-24T23:15:00.000-07:002017-08-24T01:07:03.201-07:00Integrating LuaJIT and Autogenerating C Bindings In Visual Studio<a href="https://www.lua.org/">Lua</a> is a popular scripting language due to its tight integration with C. <a href="http://luajit.org/">LuaJIT</a> is an extremely fast JIT compiler for Lua that can be integrated into your game, which also provides an <a href="http://luajit.org/ext_ffi.html">FFI Library</a> that directly interfaces with C functions, eliminating most overhead. However, the FFI library only accepts a subset of the C standard. Specifically, <b>"C declarations are not passed through a C pre-processor, yet. No pre-processor tokens are allowed, except for #pragma pack."</b> The website suggests running the header file through a preprocesser stage, but I have yet to find a LuaJIT tutorial that actually explains how to do this. Instead, all the examples simply copy+paste the function prototype into the Lua file itself. Doing this with makefiles and GCC is trivial, because you just have to add a compile step using <a href="https://gcc.gnu.org/onlinedocs/gcc-4.9.1/gcc/Preprocessor-Options.html">the <code>-E</code> option</a>, but integrating this with Visual Studio is more difficult. In addition, I'll show you how to properly load scripts and modify the PATH lookup variable so your game can have a proper <code>scripts</code> folder instead of dumping everything in <code>bin</code>.<br /><br /><h2>Compilation</h2>To begin, we need to <a href="http://luajit.org/download.html">download LuaJIT</a> and get it to actually compile. Doing this manually isn't too difficult, simply open an x64 Native Tools Command Prompt (or x86 Native Tools if you want 32-bit), navigate to <code>src/msvcbuild.bat</code> and run <code>msvcbuild.bat</code>. The default options will build an x64 or x86 dll with dynamic linking to the CRT. If you want a static lib file, you need to run it with the <code>static</code> option. If you want static linking to the CRT so you don't have to deal with that annoying Visual Studio Runtime Library crap, you'll have to modify the .bat file directly. Specifically, you need to find <code>%LJCOMPILE% /MD</code> and change it to <code>%LJCOMPILE% /MT</code>. This will then compile the static lib or dll with static CRT linking to match your other projects.<br /><br />This is a bit of a pain, and recently I've been trying to automate my build process and dependencies using <a href="">vcpkg</a> to act as a C++ package manager. A port of LuaJIT is included in the latest update of vcpkg, but if you want one that always statically links to the CRT, you can <a href="https://github.com/Black-Sphere-Studios/vcpkg/tree/master/ports/luajit">get it here</a>.<br /><br />An important note: the build instructions for LuaJIT state that you should copy the lua scripts contained in <code>src/jit</code> to your application folder. What it doesn't mention is that this is <i>optional</i> - those scripts contain debugging instructions for the JIT engine, which you probably don't need. It will work just fine without them.<br /><br />Once you have LuaJIT built, you should add it's library file to your project. This library file is called <b>lua51.lib</b> (and the dll is <b>lua51.dll</b>), because LuaJIT is designed as a drop-in replacement for the default Lua runtime. Now we need to actually load Lua in our program and integrate it with our code. To do this, use <code>lua_open()</code>, which returns a <code>lua_State*</code> pointer. You will need that <code>lua_State*</code> pointer for everything else you do, so store it somewhere easy to get to. If you are building a game using an Entity Component System, it makes sense to build a <code>LuaSystem</code> that stores your <code>lua_State*</code> pointer.<br /><br /><h2>Initialization</h2>The next step is to load in all the standard Lua libraries using <code>luaL_openlibs(L)</code>. Normally, you shouldn't do this if you need script sandboxing for player-created scripts. However, <b>LuaJIT's FFI library is inherently unsafe</b>. Any script with access to the FFI library can call any kernel API it wants, so you should be extremely careful about using LuaJIT if this is a use-case for your game. We can also register any C functions we want to the old-fashioned way via <code>lua_register</code>, but this is only useful for functions that don't have C analogues (due to having multiple return values, etc).<br /><br />There is one function in particular that you probably want to overload, and that is the <code>print()</code> function. By default, Lua will simply print to standard out, but if you aren't redirecting standard out to your in-game console, you probably have your own <code>std::ostream</code> (or even a custom stream class) that is sent all log messages. By overloading <code>print()</code>, we can have our Lua scripts automatically write to both our log file and our in-game console, which is extremely useful. Here is a complete re-implementation of <code>print</code> that outputs to an arbitrary <code>std::ostream&</code> object:<br /><pre><code class="language-cpp">int PrintOut(lua_State *L, std::ostream& out)<br />{<br /> int n = lua_gettop(L); /* number of arguments */<br /> if(!n)<br /> return 0;<br /> int i;<br /> lua_getglobal(L, "tostring");<br /> for(i = 1; i &lt;= n; i++)<br /> {<br /> const char *s;<br /> lua_pushvalue(L, -1); /* function to be called */<br /> lua_pushvalue(L, i); /* value to print */<br /> lua_call(L, 1, 1);<br /> s = lua_tostring(L, -1); /* get result */<br /> if(s == NULL)<br /> return luaL_error(L, LUA_QL("tostring") " must return a string to "<br /> LUA_QL("print"));<br /> if(i &gt; 1) out &lt;&lt; "\t";<br /> out &lt;&lt; s;<br /> lua_pop(L, 1); /* pop result */<br /> }<br /> out &lt;&lt; std::endl;<br /> return 0;<br />}<br /></code></pre>To overwrite the existing <code>print</code> function, we need to first define a Lua compatible shim function. In this example, I pass <code>std::cout</code> as the target stream:<br /><br /><pre><code class="language-cpp">int lua_Print(lua_State *L)<br />{<br /> return PrintOut(L, std::cout);<br />}<br /></code></pre>Now we simply register our <code>lua_Print</code> function using <code>lua_register(L, "print", &lua_Print)</code>. If we were doing this in a <code>LuaSystem</code> object, our constructor would look like this:<br /><br /><pre><code class="language-cpp">LuaSystem::LuaSystem()<br />{<br /> L = lua_open();<br /> luaL_openlibs(L);<br /> lua_register(L, "print", &lua_Print);<br />}<br /></code></pre>To clean up our Lua instance, we need to both trigger a final GC iteration to clean up any dangling memory, and then we call <code>lua_close(L)</code>, so our destructor would look like this:<br /><pre><code class="language-cpp">LuaSystem::~LuaSystem()<br />{<br /> lua_gc(L, LUA_GCCOLLECT, 0);<br /> lua_close(L);<br /> L = 0;<br />}<br /></code></pre><br /><h2>Loadings Scripts via Require</h2>At this point most tutorials skip to the part where you load a Lua script and write "Hello World", but we aren't done yet. Integrating Lua into your game means loading scripts and/or arbitrary strings as Lua code while properly resolving dependencies. If you don't do this, any one of your scripts that relies on another script will have to do <code>require("full/path/to/script.lua")</code>. We also face another problem - if we want to have a <code>scripts</code> folder where we simply automatically load every single script into our workspace, simply loading them all can cause duplicated code, because <code>luaL_loadfile</code> does <i>not</i> have any knowledge of <code>require</code>. You can solve this by simply loading a single <code>bootstrap.lua</code> script which then loads all your game's scripts via <code>require</code>, but we're going to build a much more robust solution.<br /><br />First, we need to modify Lua's <code>PATH</code> variable, or the variable that controls where it looks up scripts relative to our current directory. This function will append a path (which should be of the form <code>"path/to/scripts/?.lua"</code>) to the beginning of the <code>PATH</code> variable, giving it highest priority, which you can then use to add as many <code>script</code> directories as you want in your game, and any lua script from any of those folders will then be able to <code>require()</code> a script from any other folder in <code>PATH</code> without a problem. Obviously, you should probably only add one or two folders, because you don't want to deal with potential name conflicts in your script files.<br /><pre><code class="language-cpp">int AppendPath(lua_State *L, const char* path)<br />{<br /> lua_getglobal(L, "package");<br /> lua_getfield(L, -1, "path"); // get field "path" from table at top of stack (-1)<br /> std::string npath = path;<br /> npath.append(";");<br /> npath.append(lua_tostring(L, -1)); // grab path string from top of stack<br /> lua_pop(L, 1);<br /> lua_pushstring(L, npath.c_str());<br /> lua_setfield(L, -2, "path"); // set the field "path" in table at -2 with value at top of stack<br /> lua_pop(L, 1); // get rid of package table from top of stack<br /> return 0;<br />}<br /></code></pre>Next, we need a way to load all of our scripts using <code>require()</code> so that Lua properly resolves the dependencies. To do this, we create a function in C that literally calls the <code>require()</code> function for us:<br /><pre><code class="language-cpp">int Require(lua_State *L,const char *name)<br />{<br /> lua_getglobal(L, "require");<br /> lua_pushstring(L, name);<br /> int r = lua_pcall(L, 1, 1, 0);<br /> if(!r)<br /> lua_pop(L, 1);<br /> WriteError(L, r, std::cout);<br /> return r;<br />}<br /></code></pre>By using this to load all our scripts, we don't have to worry about loading them in any particular order - <code>require</code> will ensure everything gets loaded correctly. An important note here is <code>WriteError()</code>, which is a generic error handling function that processes Lua errors and writes them to a log. All errors in lua will return a nonzero error code, and will <i>usually</i> push a string containing the error message to the stack, which must then be popped off, or it'll mess things up later.<br /><pre><code class="language-cpp">void WriteError(lua_State *L, int r, std::ostream& out)<br />{<br /> if(!r)<br /> return;<br /> if(!lua_isnil(L, -1)) // Check if a string was pushed<br /> {<br /> const char* m = lua_tostring(L, -1);<br /> out &lt;&lt; "Error " &lt;&lt; r &lt;&lt; ": " &lt;&lt; m &lt;&lt; std::endl;<br /> lua_pop(L, 1);<br /> }<br /> else<br /> out &lt;&lt; "Error " &lt;&lt; r &lt;&lt; std::endl;<br />}<br /></code></pre><br /><h2>Automatic C Binding Generation</h2>Fantastic, now we're all set to load up our scripts, but we still need to somehow define a header file and also load that header file into LuaJIT's FFI library so our scripts have direct access to our program's exposed C functions. One way to do this is to just copy+paste your C function definitions into a Lua file in your scripts folder that is then automatically loaded. This, however, is a pain in the butt and is error-prone. We want to have a <i>single source of truth</i> for our function definitions, which means defining our entire LuaJIT C API in a single header file, which is then loaded directly into LuaJIT. Predictably, we will accomplish this by abusing the C preprocessor:<br /><br /><pre><code class="language-cpp">#ifndef __LUA_API_H__<br />#define __LUA_API_H__<br /><br />#ifndef LUA_EXPORTS<br />#define LUAFUNC(ret, name, ...) ffi.cdef[[ ret lua_##name(__VA_ARGS__); ]]; name = ffi.C.lua_##name<br />local ffi = require("ffi")<br />ffi.cdef[[ // Initial struct definitions<br />#else<br />#define LUAFUNC(ret, name, ...) ret __declspec(dllexport) lua_##name(__VA_ARGS__)<br />extern "C" { // Ensure C linkage is being used<br />#endif<br /><br />struct GameInfo<br />{<br /> uint64_t DashTail;<br /> uint64_t MaxDash;<br />};<br /><br />typedef const char* CSTRING; // VC++ complains about having const char* in macros, so we typedef it here<br /><br />#ifndef LUA_EXPORTS<br />]] // End struct definitions<br />#endif<br /><br /> LUAFUNC(CSTRING, GetGameName);<br /> LUAFUNC(CSTRING, IntToString, int);<br /> LUAFUNC(void, setdeadzone, float);<br /><br />#ifdef Everglade_EXPORTS<br />}<br />#endif<br /><br />#endif<br /></code></pre>The key idea here is to use macros such that, when we pass this through the preprocessor <i>without</i> any predefined constants, it will magically turn into a valid Lua script. However, when we compile it in our C++ project, our project defines <code>LUA_EXPORTS</code>, and the result is a valid C header. Our C <code>LUAFUNC</code> is set up so that we're using C linkage for our structs and functions, and that we're <i>exporting the function</i> via <code>__declspec(dllexport)</code>. This obviously only works for Visual Studio so you'll want to set up a macro for the GCC version, but I will warn you that VC++ got really cranky when i tried to use a macro for that in my code, so you may end up having to redefine the entire <code>LUAFUNC</code> macro for each compiler.<br /><br />At this point, we have a bit of a choice to make. It's more convenient to have the C functions available in the global namespace, which is what this script does, because this simplifies calling them from an interactive console. However, <b>using <code>ffi.C.FunctionName</code> is significantly faster</b>. Technically the fastest way is declaring <code>local C = ffi.C</code> at the top of a file and then calling the functions via <code>C.FunctionName</code>. Luckily, importing the functions into the global namespace does not preclude us from using the "fast" way of calling them, so our script here imports them into the global namespace for ease of use, but in our scripts we can use the <code>C.FunctionName</code> method instead. Thus, when outputting our Lua script, our <code>LUAFUNC</code> macro wraps our function definition in a LuaJIT <code>ffi.cdef</code> block, and then runs a <i>second</i> Lua statement that brings the function into the global namespace. This is why we have an initial <code>ffi.cdef</code> code block for the structs up top, so we can include that second lua statement after each function definition.<br /><br />Now we need to set up our compilation so that Visual Studio generates this file without any predefined constants and outputs the resulting lua script to our <code>scripts</code> folder, where our other in-game scripts can automatically load it from. We can accomplish this using a Post-Build Event (under Configuration Properties -> Build Events -> Post-Build Event), which then runs the following code:<br /><pre>CL LuaAPI.h /P /EP /u<br />COPY "LuaAPI.i" "../bin/your/script/folder/LuaAPI.lua" /Y<br /></pre>Visual Studio can sometimes be finicky about that newline, but if you put in two statements on two separate lines, it should run both commands sequentially. You may have to edit the project file directly to convince it to actually do this. The key line here is <code>CL LuaAPI.h /P /EP /u</code>, which tells the compiler to preprocess the file and output it to a <code>*.i</code> file. There is no option to configure the output file, it will always be the exact same file but with a <code>.i</code> extension, so we have to copy and rename it ourselves to our scripts folder using the <code>COPY</code> command.<br /><br /><h2>Loading and Calling Lua Code</h2>We are now set to load all our lua scripts in our script folder via <code>Require</code>, but what if we want an interactive Lua console? There are lua functions that read strings, but to make this simpler, I will provide a function that loads a lua script from an arbitrary <code>std::istream</code> and outputs to an arbitrary <code>std::ostream</code>:<br /><pre><code class="language-cpp">const char* _luaStreamReader(lua_State *L, void *data, size_t *size)<br />{<br /> static char buf[CHUNKSIZE];<br /> reinterpret_cast<std::istream*>(data)->read(buf, CHUNKSIZE);<br /> *size = reinterpret_cast<std::istream*>(data)->gcount();<br /> return buf;<br />}<br /><br />int Load(lua_State *L, std::istream& s, std::ostream& out)<br />{<br /> int r = lua_load(L, &_luaStreamReader, &s, 0);<br /><br /> if(!r)<br /> {<br /> r = lua_pcall(L, 0, LUA_MULTRET, 0);<br /> if(!r)<br /> PrintOut(L, out);<br /> }<br /><br /> WriteError(L, r, out);<br /> return r;<br />}<br /></code></pre><br />Of course, the other question is how to call Lua functions from our C++ code directly. There are many, <i>many</i> different implementations of this available, of varying amounts of safety and completeness, but to get you started, here is a very simple implementation in C++ using templates. Note that this does not handle errors - you can change it to use <code>lua_pcall</code> and check the return code, but <a href="http://www.knightsgame.org.uk/blog/2012/09/03/notes-on-luac-error-handling/">handling arbitrary Lua errors is nontrivial</a>.<br /><pre><code class="language-cpp">template&lt;class T, int N&gt;<br />struct LuaStack;<br /><br />template&lt;class T&gt; // Integers<br />struct LuaStack&lt;T, 1&gt;<br />{<br /> static inline void Push(lua_State *L, T i) { lua_pushinteger(L, static_cast&lt;lua_Integer&gt;(i)); }<br /> static inline T Pop(lua_State *L) { T r = (T)lua_tointeger(L, -1); lua_pop(L, 1); return r; }<br />};<br />template&lt;class T&gt; // Pointers<br />struct LuaStack&lt;T, 2&gt;<br />{<br /> static inline void Push(lua_State *L, T p) { lua_pushlightuserdata(L, (void*)p); }<br /> static inline T Pop(lua_State *L) { T r = (T)lua_touserdata(L, -1); lua_pop(L, 1); return r; }<br />};<br />template&lt;class T&gt; // Floats<br />struct LuaStack&lt;T, 3&gt;<br />{<br /> static inline void Push(lua_State *L, T n) { lua_pushnumber(L, static_cast&lt;lua_Number&gt;(n)); }<br /> static inline T Pop(lua_State *L) { T r = static_cast&lt;T&gt;(lua_touserdata(L, -1)); lua_pop(L, 1); return r; }<br />};<br />template&lt;&gt; // Strings<br />struct LuaStack&lt;std::string, 0&gt;<br />{<br /> static inline void Push(lua_State *L, std::string s) { lua_pushlstring(L, s.c_str(), s.size()); }<br /> static inline std::string Pop(lua_State *L) { size_t sz; const char* s = lua_tolstring(L, -1, &sz); std::string r(s, sz); lua_pop(L, 1); return r; }<br />};<br />template&lt;&gt; // Boolean<br />struct LuaStack&lt;bool, 1&gt;<br />{<br /> static inline void Push(lua_State *L, bool b) { lua_pushboolean(L, b); }<br /> static inline bool Pop(lua_State *L) { bool r = lua_toboolean(L, -1); lua_pop(L, 1); return r; }<br />};<br />template&lt;&gt; // Void return type<br />struct LuaStack&lt;void, 0&gt; { static inline void Pop(lua_State *L) { } };<br /><br />template&lt;typename T&gt;<br />struct LS : std::integral_constant&lt;int, <br /> std::is_integral&lt;T&gt;::value + <br /> (std::is_pointer&lt;T&gt;::value * 2) + <br /> (std::is_floating_point&lt;T&gt;::value * 3)&gt;<br />{};<br /><br />template&lt;typename R, int N, typename Arg, typename... Args&gt;<br />inline R _callLua(const char* function, Arg arg, Args... args)<br />{<br /> LuaStack&lt;Arg, LS&lt;Arg&gt;::value&gt;::Push(_l, arg);<br /> return _callLua&lt;R, N, Args...&gt;(function, args...);<br />}<br />template&lt;typename R, int N&gt;<br />inline R _callLua(const char* function)<br />{<br /> lua_call(_l, N, std::is_void&lt;R&gt;::value ? 0 : 1);<br /> return LuaStack&lt;R, LS&lt;R&gt;::value&gt;::Pop(_l);<br />}<br /><br />template&lt;typename R, typename... Args&gt;<br />inline R CallLua(lua_State *L, const char* function, Args... args)<br />{<br /> lua_getglobal(L, function);<br /> return _callLua&lt;R, sizeof...(Args), Args...&gt;(L, function, args...);<br />}<br /></code></pre>Now you have everything you need for an extensible Lua scripting implementation for your game engine, and even an interactive Lua console, all using LuaJIT. Good Luck!Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-69059431815842950332017-06-24T08:40:00.000-07:002017-06-25T12:26:28.962-07:00Discord: Rise Of The Bot Wars<!--<img src="https://blackhole12.github.io/storage/Discord-Logo-burning-web.png" alt="" style="float:right" />-->The most surreal experience I ever had on discord was when someone PMed me to complain that my anti-spam bot wasn't working against a 200+ bot raid. I pointed out that it was never designed for large-scale attacks, and that discord's own rate-limiting would likely make it useless. He revealed he was selling spambot accounts at a rate of about $1 for 100 unique accounts and that he was being attacked by a rival spammer. My anti-spam bot had been dragged into a turf war between two spambot networks. We discussed possible mitigation strategies for worst-case scenarios, but agreed that most of them would involve false-positives and that discord showed no interest in fixing how exploitable their API was. I hoped that I would never have to implement such extreme measures into my bot.<br /><br />Yesterday, our server was attacked by over 40 spambots, and after discord's astonishingly useless "customer service" response, I was forced to do exactly that.<br /><br /><h2>A Brief History of Discord Bots</h2>Discord is built on a REST API, which was reverse engineered by late 2015 and used to make unofficial bots. To test out their bots, they would hunt for servers to "raid", invite their bots to the server, then spam so many messages it would softlock the client, because discord still didn't have any rate limiting. Naturally, as the designated punching bags of the internet, furries/bronies/Twilight fans/slash fiction writers/etc. were among the first targets. The attack on our server was so severe it took us almost 5 minutes of wrestling with an unresponsive client to ban them. Ironically, a few of the more popular bots today, such as "BooBot", are banned as a result of that attack, because the first thing the bot creator did was use it to raid our server.<br /><br />I immediately went to work building an anti-spam bot that muted anyone sending more than 4 messages per second. Building a program in a hostile environment like this is much different from writing a desktop app or a game, because the bot had to be bulletproof - it had to rate-limit itself and could not be allowed to crash, <i>ever</i>. Any bug that allowed a user to crash the bot was treated as P0, because it could be used by an attacker to cripple the server. Despite using a very simplistic spam detection algorithm, this turned out to be highly effective. Of course, back then, discord didn't have rate limiting, or verification, or role hierarchies, or searching chat logs, or even a way to look up where your last ping was, so most spammers were probably not accustomed to having to deal with <i>any</i> kind of anti-spam system.<br /><br />I added raid detection, autosilence, an isolation channel, and join alerts, but eventually we were targeted by a group from 4chan's /pol/ board. Because this was a sustained attack, they began crafting spam attacks timed just below the anti-spam threshold. This forced me to implement a much more sophisticated anti-spam system, using a heat algorithm with a linear decay rate, which is still in use today. This improved anti-spam system eventually made the /pol/ group give up entirely. I'm honestly amazed the simplistic "X messages in Y seconds" approach worked as long as it did.<br /><br />Of course, none of this can defend against a large scale attack. As I learned by my chance encounter with an actual spammer, it was getting easier and easier to amass an army of spambots to assault a channel instead of just using one or two.<br /><br /><h2>Anatomy Of A Modern Spambot Attack</h2>At peak times (usually during summer break), our server gets raided 1-2 times <i>per day</i>. These minor raids are often just 2-3 tweens who either attempt to troll the chat, or use a basic user script to spam an offensive message. Roughly 60-70% of these raids are either painfully obvious or immediately trigger the anti-spam bot. About 20% of the raids involve slightly intelligent attempts to troll the chat by being annoying without breaking the rules, which usually take about 5-10 minutes to be "exposed". About 5-10% of the raids are large, involving 8 or more people, but they are also very obvious and can be easily confined to an isolation channel. Problems arise, however, with large <i>spambot</i> raids. Below is a timeline of the recent spambot attack on our server:<br /><br /><div style="width:100%;height:150px;background:#dfdfdf;border-radius:3px;"><div style="padding:10px;height:125px;position:relative;"><div style="transform: rotate(90deg);position:absolute;Left:-1em;Top:65px">messages</div><div class="datachart" style="display:flex;width:320px;height:100%;align-items: flex-end;"><div class="ychart" style="flex-shrink:0;font-size:0.8em;width:3em;height:100%;padding-right:0.2em;border-right:solid 1px #999;display:flex;justify-content:space-between;flex-direction:column;align-items: flex-end;"></div></div><div style="width:100%;"><div style="border-top:solid 1px #999;font-size:0.8em;position:relative;margin-left:3.2em;"><div style="position:absolute;left:0;">19:41:25</div><div style="position:absolute;left:80px;">19:41:45</div><div style="position:absolute;left:160px;">19:42:05</div><div style="position:absolute;left:240px;">19:42:25</div><div style="position:absolute;left:320px;">19:42:45</div></div></div></div></div><br />This was a botched raid, but the bots that actually worked started spamming within 5 seconds of joining, giving the moderators a very narrow window to respond. The real problem, however, is that so many of them joined, the bot's API calls to add a role to silence them were rate-limited. They also sent messages once every 0.9 seconds, which is designed to get around Discord's rate limiting. This amounted to 33 messages sent every second, but it was difficult for the anti-spam to detect. Had the spambots reduced their spam cadence to 3 seconds or more, this attack could have bypassed the anti-spam detection <i>entirely</i>. My bot now instigates a lockdown by raising the verification level when a raid is detected, but it simply can't silence users fast enough to deal with hundreds of spambots, so at some point the moderators must use a mass ban function. Of course, banning is restricted by the global rate limit, because <b><a href="https://discordapp.com/developers/docs/resources/guild#create-guild-ban">Discord has no mass ban API endpoint</a></b>, but luckily the global rate limit is something like 50 requests per second, so if you're <i>only</i> banning people, you're probably okay.<br /><br />However, a hostile attacker could sneak bots in one-by-one every 10 minutes or so, avoiding setting off the raid alarm, and then activate them all at once. 500 bots sending randomized messages chosen from an English dictionary once every 5 seconds after sneaking them in over a 48 hour period is the ultimate attack, and one that is almost impossible to defend against, because this also bypasses the 10-minute verification level. As a weapon of last resort, I added a command that immediately bans all users that sent their first message within the past two minutes, but, again, banning is subject to the <a href="https://discordapp.com/developers/docs/topics/rate-limits">global rate limit!</a> In fact, the rate limits can change at any time, and while message deletion has a higher rate limit for bots, bans don't. <br /><br />The only other option is to disable the @everyone role from being able to speak on any channel, but you have to do this on a <i>per channel basis</i>, because Discord ignores you if you attempt to globally disable sending message permissions for @everyone. Even then, creating an "approved" role doesn't work because any automated assignment could be defeated by adding bots one by one. The only defense a small Discord server has is to require moderator approval for every single new user, which isn't a solution - you've just given up having a public Discord server. It's only a matter of time until any angry 13-year-old can buy a sophisticated attack with a week's allowance. What will happen to public Discord servers then? Do we simply throw up our hands and admit that humanity is so awful we can't even have public communities anymore?<br /><br /><h2>The Discord API Hates You</h2>The rate-limits imposed on Discord API endpoints are exacerbated by temporary failures, and that's excluding network issues. Thus, if I attempt to set a silence role on a spammer that just joined, the API will repeatedly claim they do not exist. In fact, 3 separate API endpoints consistently fail to operate properly during a raid: A "member joined" event won't show up for several seconds, but if I fall back to calling <code>GetMember()</code>, this <i>also</i> claims the member doesn't exist, which means adding the role <i>also</i> fails! So I have to attempt to silence the user with every message they send until Discord actually adds the role, even though the API failures are also counted against the rate limit! This gets completely absurd once someone assaults your server with 1000 spambots, because this triggers all sorts of bottlenecks that normally aren't a problem. The alert telling you a user has joined? Rate limited. It'll take your bot 5-10 minutes to get through just <i>telling</i> you such a gigantic spambot army joined, unless you include code specifically designed to detect these situations and reduce the number of alerts. Because of this, a single user can trigger something like 5-6 API requests, all of which are counted against your global rate limit and can severely cripple a bot.<br /><br />The general advice that is usually given here is "just ban them", which is terrible advice because Discord's own awful message handling makes it incredibly easy to trigger a false positive. If a message fails to send, the client simply sends a completely new message, with it's own ID, and will continue re-sending the message until an Ack is received, at which point the user has probably send 3 or 4 copies of the same message, each of which have the same content, but completely unique IDs and timestamps, which looks <i>completely identical</i> to a spam attack.<br /><br />Technically speaking, this is done because Discord assigns snowflake IDs server-side, so each message attempt sent by the client must have a unique snowflake assigned after it is sent. However, it can also be trivially fixed by adding an optional "client ID" field to the message, with a client-generated ID that stays the same if the message is resent due to a network failure. That way, the server (or the other clients) can simply drop any duplicate messages with identical client IDs while still ensuring all messages have unique IDs across their distributed cluster. This would single-handedly fix all duplicate messages across the entire platform, and eliminate almost every single false-positive I've seen in my anti-spam bot.<br /><br /><h2>Discord Doesn't Care</h2>Sadly, Discord doesn't seem to care. The general advice in response to "how do I defend against a large scale spam attack" is "just report them to us", so we did exactly that, and then got what has to be one of the dumbest customer service e-mails I've ever seen in my life:<br /><br /><img src="https://blackhole12.github.io/storage/res4.PNG" alt="Discord Being Stupid" /><br /><br />Excuse me, <i><b>WHAT?!</b></i> Sorry about somebody spamming your service with horrifying gore images, but please don't delete them! What happens if the spammers just delete the messages themselves? What happens if they send <i>child porn?</i> "Sorry guys, please ignore the images that are literally illegal to even look at, but we can't delete them because Discord is fucking stupid." Does Discord understand the concept of <i>marking messages for deletion</i> so they are viewable for a short time as evidence for law enforcement?! My anti-spam bot's database currently has more information than <i>Discord's own servers!</i> If this had involved child porn, the FBI would have had to ask <b>me</b> for <b>my records</b> because Discord would have deleted them all!<br /><br />Obviously, we're not going to leave 500+ gore messages sitting in the chatroom while Discord's ass-backwards abuse team analyzes them. I just have to hope my own nuclear option can ban them quickly enough, or simply give up the entire concept of having a public Discord server.<br /><br />The problem is that the armies of spambots that were once reserved for the big servers are now so easy and so trivial to make that they're beginning to target smaller servers, servers that don't have the resources or the means to deal with that kind of large scale DDoS attack. So instead, I have to fight the growing swarm alone, armed with only a crippled, rate-limited bot of my own, and hope the dragons flying overhead don't notice.<br /><br />What the <i>fuck</i>, Discord.<br /><br /><script src="https://d3js.org/d3.v4.min.js"></script><br /><script>var data = [1,0,0,0,2,3,0,0,0,0,0,5,0,0,0,1,0,2,3,3,0,2,0,9,3,0,0,21,34,34,23,7,4,2,1,8,2,23,34,19,14,11,5,3,3,2,2,5,5,0,0,0,2,4,2,0,0,0,1,0,0,0,0,0,5,2,0,0,0,0,2,2,0,1,0,5,1,2,0,0,0,0,2,2,0,0,0,0,1,1]; var axis = [35, 28, 21, 14, 7, 0]; d3.select(".datachart") .selectAll("div") .data(data) .enter().append("div") .style("width", "2px") .style("margin-left", "2px") .style("flex-shrink", "0") .style("background", "#7289DA") .style("height", function(d) { return (d / axis[0])*100.0 + "%"; }); d3.select(".ychart") .selectAll("div") .data(axis) .enter().append("div") .style("flex-shrink", "0") .text(function(d) { return d; }); </script>Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com8tag:blogger.com,1999:blog-4900186601661215466.post-68554883483170524072017-06-01T15:48:00.001-07:002017-06-01T15:48:19.677-07:00Programmers Should Take LinguisticsThe older I get, the more I realize that 90% of all disagreements or social drama results from a miscommunication of some kind. Every time I wind up having to resolve a dispute, I'll try to get both sides of the story, only to realize that they're the same story, and both parties were actually either in agreement or fighting over a perceived insult that never actually existed. Unsurprisingly, a disproportionate amount of this miscommunication often involves programmers being far too strict with their interpretations of what certain words mean.<br /><br />A linguistics class teaches you about what a language actually is - a bunch of sounds that we mutually agree mean certain things. Language is, intrinsically, a social construct. The correct definition of a word is whatever the majority of your primary social circle thinks it is. However, this also means that if you interact with a secondary social circle, and they all think the word means something else, then whenever you interact with them, it really <i>does</i> mean something else. Language is inherently contextual, and the possible meanings of a word can change based on who is saying it and to whom they're saying it to. If everyone else on Earth has decided that 'literally' can also mean 'figuratively', then it does, even if the dictionary says otherwise. It also means most people don't actually care if you say <i>jif</i> or <i>gif</i>, they'll just say whatever pronunciation gets you to shut up about it.<br /><br />It's important to realize that a word's meaning is not defined by a dictionary, but rather by how people use it. The dictionary is simply a reflection of it's usage, and is generally a few years out of date. Just as the pronunciation of a word can vary by dialect, so can the potential meanings of a word. Meanings can be invented by regional subdialects and spread outward from there, which is the origin of many slang terms. Sometimes we invent entirely new words, like "dubstep", but young words may have fuzzy definitions. In some dialects of electronic music listeners, "dubstep" is not actually a a specific genre, but instead refers to all electronic music. Using dubstep to refer to any electronic song is <i>currently</i> incorrect if used in general parlance, because most people think it is referring to a very specific kind of music. However, if this usage of the word continues to be popularized, eventually the meaning of the word will change into a synonym for electronica, and the dictionaries will be updated to reflect this.<br /><br />The fluid nature of language is why prescriptive grammar is almost always unnecessary, unless you are deliberately conforming to a grammar standard for a specific medium, such as writing a story. In almost any other context, so long as everyone in your social group understands your 'dialect' of English, then it is valid grammar. However, if you attempt to use this dialect outside of your social circle with people who are not familiar with it, you will once again be in the wrong, as they will have no idea what you're talking about. This, however, does not mean there are no mandatory grammar rules, it's just that most of the rules that are actually necessary to speak the language properly are usually so ingrained that you don't even think about them.<br /><br />A fantastic example of this is a little known rule in English where all adjectives <i>must</i> come in a very specific order: <code>opinion-size-age-shape-color-origin-material-purpose Noun</code>. So you can have a lovely little old rectangular green French silver whittling knife, but if you switch the order of any of those adjectives you'll sound like a maniac. Conversely, you can never have a green great dragon. Despite the fact that this grammar rule is basically never talked about in any prescriptive grammar book, it is <i>mandatory</i> because if you don't follow it you won't be speaking proper English and people will have difficulty understanding you. True grammar rules are ones that, if not followed, result in nonsensical sentences that are difficult or impossible to parse correctly.<br /><br />However, this does not mean all sentences that are difficult to understand have incorrect grammar. In fact, even some words are completely ambiguous <i>by default</i>. If I say I'm "dusting an object", the meaning of the phrase is completely dependent on what the object is. If it's a cake, I'm probably dusting it with something. If it's a shelf, I'm probably dusting it to get rid of the dust.<br /><br />Programmers tend to be very literal minded people, and often like to think that language is a set of strict rules defined by their English class. In reality, language is a fluid, dynamic, ambiguous, constantly changing enigma that exists entirely because we all agree on what a bunch of sounds mean. We need to recognize this, and when we communicate to other people, we need to be on the lookout for potential misinterpretations of what we say, so we can provide clarifications when possible. If someone says something that seems ridiculous, ask them to clarify. I'm tired of resolving disagreements that exist only because nobody stopped to ask the other side to clarify what they meant.<br /><br />Stop demanding that everyone explain things in a way you'll understand. That's impossible, because everyone understands language slightly differently. Instead, ask for clarification if someone seems to be saying something unusual or before you debate a point they made. Maybe then we can keep the debates to <i>actual</i> disagreements, instead of arguing over communication failures.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-50408499471949354862017-05-04T14:06:00.000-07:002017-05-04T14:19:46.789-07:00Why Bother Making An App?I have an idea for an app. According to startup literature, I'm supposed to get initial fundraising from small-time investors, or "angel" investors, possibly with help from an incubator. Then, after using this money to build an MVP and push the product on the marketplace, I do a Series A round with actual venture capitalists. Now, the venture capitalists probably won't give me any money unless I can give them a proper financial outlook, user growth metrics, and a solid plan for expansion, along with a market cap estimation. Alternatively, I can just use enough meaningless buzzwords and complete bullshit to convince them to give me <a href="http://www.npr.org/sections/thetwo-way/2017/04/21/525055713/juicero-ceo-says-luxury-juicer-is-much-more-than-juice-internet-is-unimpressed">$120 million for a worthless piece of junk</a>.<br /><br />Either way, venture capitalists usually want a sizable 10-30% stake in your company (depending on if it's Series A, Series B, or Series C), given how much money they're pouring into a company that might fail. That's okay though, because my app does reasonably well and sells lots of copies on the app store and journalists write about it. Unfortunately, soon sales start tapering off, and ad revenue declines because customers either purchase the pro version or block the ads entirely. While the company is financially stable and making a modest profit, this isn't enough for the investors. They want <i>growth</i>, they need <i>user engagement</i>, they need <i>ever increasing profits</i>. Simply building a stable company isn't enough for them.<br /><br />So the investors start pushing for you to be bought out. You get lucky, and your app would make a great accessory to Google Assistant, or Cortana, and you get huge buyout offers from Microsoft, Google, and Amazon, because they have more money than most small countries. Investors immediately push for you to take the most lucrative offer from whoever is willing to give you the most cash for fucking over all of your customers. You can push back, but your power is limited, because those investors hold a significant chunk of your company. At best, you can pick the offer that is least likely to completely destroy your product.<br /><br />If you get lucky, your cross-platform app that worked on everything gets discontinued and re-integrated into one device that people have to buy due to vendor lock-in. If you aren't lucky, your app gets discontinued and completely forgotten about, until someone else comes up with the same idea and the process repeats. Maybe this time they'll get bought out and actually integrated into <i>something</i>.<br /><br />Either way, your customers lose. Every time. They are punished for believing that a new app, by some new company, could actually survive long enough to be useful to them without being consumed by the corporate monstrosities that run the world. If the company founders are nice, maybe some of the employees walk away rich, but most of them will probably just end up trapped inside a corporate behemoth until they can't take it anymore and finally quit. In your efforts to make the world a better place, you've managed to screw over your company, your customers, and even your employees, because investors don't care about your product, they care about milking you for all you're worth.<br /><br />But hey, at least you're rich, right?Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com3tag:blogger.com,1999:blog-4900186601661215466.post-70922945001146134982017-03-23T00:37:00.000-07:002017-03-23T00:37:10.427-07:00Companies Can't Be ApoliticalOne of the most common things I hear from people is that companies should be "apolitical". The most formal way this concept is expressed is that a company should make decisions based on what maximizes profits and not political opinions. Unfortunately, the statement "companies should only care about maximizing profits" is, itself, a political statement (and one I happen to disagree with). Thus, it is fundamentally impossible for a company to be <i>truly</i> apolitical, for the very act of attempting to be apolitical is a political statement.<br /><br />How much a company can avoid politics generally depends on both the type and size of the company. Once your company becomes large enough, it will influence politics simply by virtue of its enormous size, and eventually becomes an integral part of political debates whether or wants to or not. Large corporations must take into account the political climate when making business decisions, because simply attempting to blindly maximize profit may turn the public against them and destroy their revenue sources&mdash;thus, politics themselves become part of the profit equation, and cannot be ignored. Certain types of businesses embody political statements simply by <i>existing</i>. Grindr, for example, is a dating app for gay men. It's entire business model is dependent on enabling an activity that certain fundamentalists consider inherently immoral.<br /><br />You could, theoretically, try to solve part of this quandary by saying that companies should also be <i>amoral</i>, insofar that the free market should decide moral values. The fundamentalists would then protest the companies existence by not using it (but then, they never would have used it in the first place). However, the problem is that, once again, this very statement is itself political in nature. Thus, by either trying to be amoral or moral, a company is making a political statement.<br /><br />The issue at play here is that <i>literally everything is political</i>. When most everyone agrees on basic moral principles, it's easier to pretend that politics is really just about economic policy and lawyers, but our current political divisions have demonstrated that this is a fantasy. Politics <i>are</i> the fundamental morals that society has decided on. It's just a lot easier to argue about minor differences in economic policy instead of fundamental differences in basic morality.<br /><br />Of course, <i>how</i> companies participate in politics is also important to consider. Right now, a lot of companies participate in politics by spending exorbitant amounts of money on lobbyists. This is a symptom of money <i>in general</i>, and should be solved not by removing <i>corporate</i> money from politics, but removing <i>all</i> money, because treating spending money as a form of speech gives more speech to the rich, which inherently discriminates against the poor and violates the constitutional assertion that all men are created equal (but no one really seems to be paying attention to that line anyway).<br /><br />Instead of using money, corporations should <i>do</i> things that uphold whatever political values they believe in. As the saying goes, actions speak louder than words (or money, in this case). You could support civil rights activism by being more inclusive with your hiring and promoting a diverse work environment. Or, if you live in the Philippines, you could create an app that helps death squads hunt down drug users so they can be brutally executed. What's interesting is that most people consider the latter to be a moral issue as opposed to a political one, which seems to derive from the fact that once you agree on most fundamental morals, we humans simply make up a bunch of pointless rules to satisfy our insatiable desire to tell other humans they're wrong.<br /><br />We've lived in a civilized world for so long, we've forgotten the true roots of politics: a clash between our fundamental moral beliefs, not about how much parking fines should be. Your company will make a political statement whether you like it or not, so you'd better make sure it's the one you want.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-45680151651690341862017-03-07T15:06:00.001-08:002017-03-12T19:03:32.103-07:00I Can't Hear Anything Below 80 Hz*<blockquote><i>* at a comfortable listening volume.</i></blockquote><b>EDIT: I have confirmed all the results presented here by taking the low frequency test with someone standing physically next to me. They heard a tone beginning at 30 Hz, and by the time I could hear a very faint tone around 70 Hz, they described the tone as "conversation volume level", which is about 60 dB. I did not reach this perceived volume level until about 120 Hz, which strongly correlates with the experiment. More specific results would require a professional hearing test.</b><br /><br />For almost 10 years, I've suspected that something was wrong with my ability to hear bass tones. Unfortunately, while everyone is used to people having difficulty hearing high tones, nobody takes you seriously if you tell them you have difficulty hearing <i>low</i> tones, because most audio equipment has shitty bass response, and human hearing isn't very precise at those frequencies in the first place. People generally say "oh you're just supposed to <i>feel</i> the bass, don't worry about it." This was extremely frustrating, because one of my hobbies is writing music, and I have struggled for years and years to do proper bass mixing, which is basically the only activity on the entire planet that actually requires hearing subtle changes in bass frequencies. This is aggravated by the fact that most hearing tests are designed to detect issues with high frequencies, not low frequencies, so all the basic hearing tests I took at school gave test results back that said "perfectly normal". Since I now have professional studio monitor speakers, I'm going to use science to prove that I have an abnormal frequency sensitivity curve that severely hampers my ability to differentiate bass tones. Unfortunately, at the moment I live alone and nowhere near anyone else, so I will have to prove that my equipment is not malfunctioning without being able to actually hear it.<br /><br />Before performing the experiment, I did <a href="http://www.audiocheck.net/audiotests_frequencychecklow.php">this simple test</a> as a sanity check. At a normal volume level, I start to hear a very faint tone in that example at about 70 Hz. When I sent it to several other people, they all reported hearing a tone around 20-40 Hz, even when using consumer-grade hardware. This is clear evidence that something is very, very wrong, but I have to prove that my hardware is not malfunctioning before I can definitively state that I have a problem with my hearing.<br /><br />For this experiment, I will be using two JBL Professional LSR305 studio monitors plugged into a Focusrite Scarlett 2i2. Since these are studio monitors, they should have a roughly linear response all the way down to 20 Hz. I'm going to use a free sound pressure app on my android phone to prove that they have a relatively linear response time. The app isn't suitable for measuring very quiet or very loud sounds, but we won't be measuring anything past 75 dB in this experiment because I don't want to piss off my neighbors.<br /><br /><img src="https://blackhole12.github.io/storage/chart1.png" alt="Speaker Frequency Response Graph" width="700"/><br /><br />The studio monitor manages to put out relatively stable noise levels until it appears to fall off at 50 Hz. However, when I played a tone of 30 Hz at a volume loud enough for me to <i>feel</i>, the sound monitor still reported no pressure, which means the <i>microphone</i> can't detect anything lower than 50 Hz (I was later able to prove that the studio monitor is working properly when someone came to visit). Of course, I can't hear anything below 50 Hz anyway, no matter how loud it is, so this won't be a problem for our tests. To compensate for the variance in the frequency response volume, I use the sound pressure app to measure the <i>actual sound intensity</i> being emitted by the speakers.<br /><br />The first part of the experiment will detect the softest volume at which I can detect a tone at any frequency, starting from D3 (293 Hz) and working down note by note. The loudness of the tone is measured using the sound pressure app. For frequencies above 200 Hz, I can detect tones at volumes only slightly above the background noise in my apartment (15 dB). By the time we reach 50 Hz I was unwilling to go any louder (and the microphone would have stopped working anyway), but this is already enough for us to establish 50 Hz as the absolute limit of my hearing ability under normal circumstances.<br /><br /><img src="https://blackhole12.github.io/storage/chart2.png" alt="Threshold of Hearing Graph" width="700"/><br /><br />To get a better idea of my frequency response at more reasonable volumes, I began with a D4 (293 Hz) tone playing at a volume that corresponded to 43 dB SPL on my app, and then recorded the sound pressure level of each note once it's volume seemed to match with the other notes. This gives me a rough approximation of the 40 phon <a href="https://en.wikipedia.org/wiki/Equal-loudness_contour">equal loudness curve</a>, and allows me to overlay that curve on to the ISO 226:2003 standard:<br /><br /><img src="https://blackhole12.github.io/storage/chart3.png" alt="Equal Loudness Contour" width="700"/><br /><br />These curves make it painfully obvious that my hearing is severely compromised below 120 Hz, and becomes nonexistent past 50 Hz. Because I can still technically <i>hear</i> bass at extremely loud volumes, I can pass a hearing test trying to determine if I <i>can</i> hear low tones, but the instant the tones are not presented in isolation, they are drowned out by higher frequencies due to my impaired sensitivity. Because all instruments that aren't pure sine waves produce harmonics above the fundamental frequency, this means the only thing I'm hearing when a sub-bass is playing <i>are the high frequency harmonics</i>. Even then, I can still <i>feel</i> bass if it's loud enough, so the bass experience isn't completely ruined for me, but it makes mixing almost impossible because of how bass frequencies interact with the waveform. Bass frequencies take up lots of headroom, which is why in a trance track, you can tell where the kicks are just by looking at the waveform itself:<br /><br /><img src="https://blackhole12.github.io/storage/bass_example.png" alt="Bass Example" width="700"/><br /><br />When mixing, you must carefully balance the bass with the rest of the track. If you have too much bass, it will overwhelm the rest of the frequencies. Because of this, when I send my tracks to friends to get help on mixing, I can tell that the track sounds better, but I can't tell why. The reason is because they are adjusting bass frequencies <i>I literally cannot hear</i>. All I can hear is the end result, which has less frequency crowding, which makes the higher frequencies sound better, even though I can't hear any other difference in the track, so it seemes like black magic.<br /><br />It's even worse because I am almost completely incapable of differentiating tones below 120 Hz. You can play any note below B2 and I either won't be able to hear it or it'll sound the same as all the other notes. I can only consistently differentiate semitones above 400 Hz. Between 120-400 Hz, I can sometimes tell them apart, but only when playing them in total isolation. When they're embedded in a song, it's hopeless. This is why, in AP Music Theory, I was able to perfectly transcribe all the notes in the 4-part writing, <i>except the bass</i>, yet no other students seemed to have this problem. My impaired sensitivity to low frequencies mean they get drowned out by higher frequencies, making it more and more difficult to differentiate bass notes. In fact, in most rock songs, I can't hear the bass guitar <i>at all</i>. The only way for me to hear the bass guitar is for it to be played by itself.<br /><br />Incidentally, this is probably why I hate dubstep.<br /><br />For testing purposes, I've used the results of my sensitivity testing to create an EQ filter that mimics my hearing problems as best I can. I can't tell if the filter is on or off. For those of you that use FL Studio, the preset can be <a href="https://drive.google.com/file/d/0B_2aDNVL_NGmZEFkcFdlNmVxN1U/view?usp=sharing">downloaded here</a>.<br /><br /><img src="https://blackhole12.github.io/storage/EQ_demo.png" alt="EQ Curve"/><br /><br />Below is a song I wrote some time ago that was mastered by a friend who can actually hear bass, so hopefully the bass frequencies in this are relatively normal. I actually have a bass synth in this song I can only <i>barely</i> hear, and had to rely almost entirely on the sequencer to know which notes were which.<br /><br /><iframe width="100%" height="150" scrolling="no" frameborder="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/213925837&amp;auto_play=false&amp;hide_related=false&amp;show_comments=true&amp;show_user=true&amp;show_reposts=false&amp;visual=true"></iframe><br /><br />This is the same song with the filter applied:<br /><br /><iframe width="100%" height="160" src="https://clyp.it/4fkctaiv/widget" frameborder="0"></iframe><br /><br />By inverting this filter, I can attempt to "correct" for my bass hearing, although this is only effective down to about 70 Hz, which unfortunately means the entire sub-bass spectrum is simply inaudible to me. To accomplish this, I combine the inverted filter with a mastering plugin that completely removes all frequencies below 60 Hz (because I can't hear them) and then lowers the volume by about 8 dB so the amplified bass doesn't blow the waveform up. This doesn't seem to produce any audible effect on songs without significant bass, but when I tried it on a professionally mastered trance song, I was able to hear a small difference in the bass kick. I also tried it on <a href="https://www.youtube.com/watch?v=xllG3fSUAOw">Brothers In Arms</a> and, for the first time, noticed a very faint bass cello going on that I had never heard before. If you are interested, the FL studio mixer state track that applies the corrective filter is <a href="https://drive.google.com/file/d/0B_2aDNVL_NGmSWJEbE9YVGdrdnM/view?usp=sharing">available here</a>, but for normal human beings the resulting bass is probably offensively loud. For that same reason, it is unfortunately impractical for me to use, because listening to bass frequencies at near 70 dB levels is bad for your hearing, and for that matter it doesn't fix my impaired fidelity anyway, but at least I now know why bass mixing has been so difficult for me over the years.<br /><br />I guess if I'm going to continue trying to write music, I need to team up with one of my friends that can actually hear bass.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com4tag:blogger.com,1999:blog-4900186601661215466.post-30949059886888278982017-02-13T21:33:00.000-08:002017-02-14T03:31:45.709-08:00Windows Won't Let My Program CrashIt's been known for a while that windows has a bad habit of <a href="http://blog.paulbetts.org/index.php/2010/07/20/the-case-of-the-disappearing-onload-exception-user-mode-callback-exceptions-in-x64/">eating your exceptions</a> if you're inside a WinProc callback function. This behavior can cause all sorts of mayhem, like your program just vanishing into thin air without any error messages due to a stack overflow that terminated the program without actually throwing an exception. What I didn't realize is that it also eats <code>assert()</code>, which makes debugging hell, because the assertion would throw, the entire user callback would immediately terminate without any stack unwinding, and then windows would just... keep going, even though the program is now in a laughably corrupt state, because only half the function executed.<br /><br />While trying to find a way to fix this, I discovered that there are no less than 4 different ways windows can choose to eat exceptions from your program. I had already told the kernel to stop eating my exceptions using the following code:<br /><pre>HMODULE kernel32 = LoadLibraryA("kernel32.dll");<br /> assert(kernel32 != 0);<br /> tGetPolicy pGetPolicy = (tGetPolicy)GetProcAddress(kernel32, "GetProcessUserModeExceptionPolicy");<br /> tSetPolicy pSetPolicy = (tSetPolicy)GetProcAddress(kernel32, "SetProcessUserModeExceptionPolicy");<br /> if(pGetPolicy && pSetPolicy && pGetPolicy(&dwFlags))<br /> pSetPolicy(dwFlags & ~EXCEPTION_SWALLOWING); // Turn off the filter <br /></pre>However, despite this, COM itself was wrapping an entire <code>try {} catch {}</code> statement around my program, so I had to figure out how to turn that off, too. Apparently <a href="https://blogs.msdn.microsoft.com/oldnewthing/20110120-00/?p=11713/">some genius at Microsoft</a> decided the default behavior should be to just swallow exceptions whenever they were making COM, and now they can't change this default behavior because it'd break all the applications that now depend on COM eating their exceptions to run properly! So, I turned that off with this code:<br /><pre>CoInitialize(NULL); // do this first<br />if(SUCCEEDED(CoInitializeSecurity(NULL, -1, NULL, NULL, RPC_C_AUTHN_LEVEL_PKT_PRIVACY,<br /> RPC_C_IMP_LEVEL_IMPERSONATE, NULL, EOAC_DYNAMIC_CLOAKING, NULL)))<br />{<br /> IGlobalOptions *pGlobalOptions;<br /> hr = CoCreateInstance(CLSID_GlobalOptions, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pGlobalOptions));<br /> if(SUCCEEDED(hr))<br /> {<br /> hr = pGlobalOptions->Set(COMGLB_EXCEPTION_HANDLING, COMGLB_EXCEPTION_DONOT_HANDLE);<br /> pGlobalOptions->Release();<br /> }<br />}<br /></pre>There are two additional functions that could be swallowing exceptions in your program: <code>_CrtSetReportHook2</code> and <code>SetUnhandledExceptionFilter</code>, but both of these are for SEH or C++ exceptions, and I was throwing an assertion, not an exception. I was actually able to verify, by replacing the assertion <code>#define</code> with my own version, that throwing an actual C++ exception <i>did</i> crash the program... but an assertion didn't. Specifically, an assertion calls <code>abort()</code>, which raises <code>SIGABRT</code>, which crashes any normal program. However, it turns out that Windows was eating the abort signal, along with every other signal I attempted to raise, which is a problem, because half the library is written in C, and C obviously can't raise C++ exceptions. The assertion failure even showed up in the output... but didn't crash the program!<br /><pre>Assertion failed!<br /><br />Program: ...udio 2015\Projects\feathergui\bin\fgDirect2D_d.dll<br />File: fgEffectBase.cpp<br />Line: 20<br /><br />Expression: sizeof(_constants) == sizeof(float)*(4*4 + 2)<br /></pre>No matter what I do, Windows refuses to let the assertion failure crash the program, or even trigger a breakpoint in the debugger. In fact, calling the <code>__debugbreak()</code> intrinsic, which outputs an <code>int 3</code> CPU instruction, was <i>completely ignored</i>, as if it simply didn't exist. The only reliable way to actually crash the program without using C++ exceptions was to do something like divide by 0, or attempt to write to a null pointer, which triggers a segfault.<br /><br />Any good developer should be using assertions to verify their assumptions, so having assertions silently fail and then <i>corrupt the program</i> is even worse than ignoring they exist! Now you could have an assertion in your code that's firing, terminating that callback, leaving your program in a broken state, and then the next message that's processed blows up for strange and bizarre reasons that make no sense because they're impossible.<br /><br />I have a hard enough time getting my programs to <i>work</i>, I didn't think it'd be this hard to make them <i>crash</i>.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-64109488164008030732017-02-12T19:19:00.000-08:002017-02-12T19:19:18.366-08:00Owlboy And The Tragedy of Human Nature<a href="http://www.owlboygame.com/">Owlboy</a> is a game developed by <a href="http://www.dpadstudio.com/">D-Pad studios</a> over a protracted 9 year development cycle. Every aspect of the game is a work of art, meticulously pieced together with delicate care. While it has mostly gotten well-deserved positive reviews from critics, some people have voiced disappointment at the story arc and how the game was eventually resolved. Note: I'm about to talk about how the game ends, so an obligatory warning that there are <b>massive spoilers ahead</b>. If you haven't already played it, <a href="http://store.steampowered.com/app/115800/">go buy it, right now</a>.<br /><br /><hr><br />On the pirate mothership, it is revealed that the titular "owlboy" is <i>not</i> referring to Otus, but is actually referring to the mysterious cloaked figure, who is actually Solus. Otus merely serves as a distraction so that Solus could steal the relics from the pirates. Once the heroes follow Solus up to Mesos and beat him into submission, it is revealed that Solus masterminded the events of the entire game, promising the pirates power in exchange for retrieving the relics, then using Otus as a distraction to steal the relics and use them to power the Anti-Hex to save the world. <br /><br />Unfortunately, as the heroes immediately point out, this resulted in the destruction of Advent and the deaths of countless innocent people. Had Solus just <i>asked for help</i>, all of this could have been avoided, and the world could have been saved without incident. Solus admits that he felt he had no choice, as he simply had nobody he could trust. Our heroes offer to help finish the ritual, at which point Molstrom shows up and ruins everyone's day. Solus' methods have finally backfired on him, and he is now too injured to complete the ritual&mdash;but Otus isn't. In an act of desperation, he imbues Otus with the power of the artifacts, and Otus is able to complete the anti-hex in his stead, obliterating Molstrom in the process and saving the world.<br /><br />A lot of people take issue with this ending for two reasons: One, it means almost everything you fought for in the game technically meant nothing, because you are actually working against Solus the entire time. Two, the entire thing could have been avoided if Solus had just trusted someone, <i>anyone</i>, instead of engineering a ridiculously convoluted plot to get what he needed through deceit and betrayal. Otus and Solus here represent a hero and an anti-hero. They are both fundamentally good people with flawed goals for opposite reasons. Solus is <b>doing the right thing for the wrong reasons</b>, whereas Otus is <b>doing the wrong thing for the right reasons</b>. Solus is doing the right thing, which is saving the world, but he achieves it by sacrificing thousands of innocent lives and betraying everyone. Otus is unknowingly dooming the world to destruction, but only because he and everyone else is operating on faulty information. He always chooses to do the right thing, and to trust others.<br /><br />This is important, because the way the game ends is crucial to the narrative of the story and the underlying moral. Solus may have <i>nearly</i> saved the world, despite his questionable methods, but <i>he would have failed at the end</i>. Molstrom would have found him and easily defeated him, taking all the relics and then destroying whatever was left of the world as it rose into space. Only after Solus explained everything to Otus was Otus able to finally <b>do the right thing for the right reasons</b>, thanks to the friends he made on his journey holding Molstrom back. By doing this, Solus allows Otus to atone for failing to secure any of the relics, and Otus absolves Solus of the evil he had committed in the name of saving the world by annihilating Molstrom with the anti-hex.<br /><br />The whole point of this story is that it doesn't matter how many times you fail, so long as you eventually succeed. Otus may have failed to save the world over and over and over, but at the very end, as the world is coming apart at the seams, he is finally able to succeed, and that's what matters. This moral hit me particularly hard because I instantly recognized what it represented - failing to ship a game over and over and over. Owlboy's story is an allegory for <i>it's own development</i>, and on a broader scale, any large creative project that has missed deadlines and is falling behind. It doesn't matter how many times you've failed, because as long as you keep trying, eventually you'll succeed, and that's what really matters. The game acknowledges that humans are deeply flawed creatures, but if we work together, we can counteract each other's flaws and build something greater than ourselves.<br /><br />This is why I find it depressing that so many people object to Solus' behavior. Surely, no one would actually do that? However, at the beginning of the game, Solus' defining character moment is being bullied and abused by the other owls. His response to this abuse is to become withdrawn, trusting no one, determined to do whatever is necessary without relying on anyone else. <b>This is exactly what happens to people who have been abused or betrayed and develop trust issues</b>. These people will refuse help that they are offered, pushing other people away, often forcing an intervention to try and break through the emotional wall they have built to try and keep themselves from being hurt again. That's why you have to fight Solus first, because he's spent his whole life building a mental block to keep other people out, and Otus has to break through it to force him to finally accept help.<br /><br />Far from being unbelievable, Owlboy's plot is entirely too real, and like any good work of art, it forces us to confront truths that make us uncomfortable. Owlboy forces us to confront the tragedy of human nature, and deal with the consequences. At the same time, it shows us that, if we persevere, we can overcome our flaws, and build a better world, together.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-24467965999857712152017-02-01T19:35:00.001-08:002017-04-09T16:44:26.650-07:00DirectX Is TerrifyingAbout two months ago, I got a new laptop and proceeded to load all my projects on it. Despite compiling everything fine, my graphics engine that used DirectX mysteriously crashed upon running. I immediately suspected either a configuration issue or a driver issue, but this seemed weird because my laptop had a <i>newer</i> graphics card than my desktop. Why was it crashing on <i>newer</i> hardware? Things got even more bizarre once I narrowed down the issue - it was in my shader assignment code, which hadn't been touched in almost 2 years. While I initially suspected a shader compilation issue, there was no such error in the logs. All the shaders compiled fine, and then... didn't work.<br /><br />Now, if this error had also been happening on my desktop, I would have immediately started digging through my constant assignments, followed by the vertex buffers assigned to the shader, but again, all of this ran perfectly fine on my desktop. I was completely baffled as to why things weren't working properly. I had eliminated all possible errors I could think of that would have resulted from moving the project from my desktop to my laptop: none of the media files were missing, all the shaders compiled, all the relative paths were correct, I was using the exact same compiler as before with all the appropriate updates. I even updated drivers on both computers, but it stubbornly refused to work on the laptop while running fine on the desktop.<br /><br />Then I found something that nearly made me shit my pants.<br /><pre><code class="lang-cpp">if(profile &lt;= VERTEX_SHADER_5_0 && _lastVS != shader) {<br /> //...<br />} else if(profile &lt;= PIXEL_SHADER_5_0 && _lastPS != shader) { <br /> //...<br />} else if(profile &lt;= GEOMETRY_SHADER_5_0 && _lastGS != shader) {<br /> //...<br />}</code></pre>Like any sane graphics engine, I do some very simple caching by keeping track of the last shader I assigned and only setting the shader if it had actually changed. These if statements, however, have a very stupid but subtle bug that took me quite a while to catch. They're a standard range exclusion chain that figures out what type of shader a given shader version is. If it's less than say, 5, it's a vertex shader. Otherwise, if it's less than 10, that this means it's in the range 5-10 and is a pixel shader. Otherwise, if it's less than 15, it must be in the range 10-15, ad infinitum. The idea is that you don't need to check if the value is greater than 5 because the failure of the previous statement already implies that. However, adding that cache check on the end breaks all of this, because now you could be in the range 0-5, but the cache check could fail, throwing you down to the next statement checking to see if you're below 10. Because you're in the range 0-5, you're of course below 10, and the cache check will ALWAYS succeed, because no vertex shader would ever be in the pixel shader cache! <b>All my vertex shaders were being sent in to directX as pixel shaders after their initial assignment!</b><br /><br />For almost 2 years, I had been feeding DirectX <i>total bullshit</i>, and had even tested it on multiple other computers, and it had never given me a single warning, error, crash, or any indication whatsoever that my code was <b>completely fucking broken</b>, in either debug mode or release mode. Somehow, deep in the depths of nVidia's insane DirectX driver, it had managed to detect that I had just tried to assign a vertex shader to a pixel shader, and either ignored it completely, or silently fixed my catastrophic fuckup. However, my laptop had the <i>mobile</i> drivers, which for some reason did not include this failsafe, and so it actually crashed like it was supposed to.<br /><br />While this was an <i>incredibly</i> stupid bug that I must have written while sleep deprived or drunk (which is impressive, because I don't actually drink), it was simply impossible for me to catch because it produced <i>zero errors or warnings</i>. As a result, this bug has the honor of being both the dumbest and the longest-living bug of mine, <i>ever</i>. I've checked every location I know of for any indication that anything was wrong, including hooking into the debug log of directX and dumping all it's messages. Nothing. Nada. Zilch. Zero.<br /><br />I've heard stories about the insane bullshit nVidia's drivers do, but this is <b>fucking terrifying</b>.<br /><br />Alas, there is more. I had been experimenting with direct2D as an alternative because, well, it's a 2D engine, right? After getting text rendering working, a change in string caching suddenly broke the entire program. It broke in a particularly bizarre way, because it seemed to just stop rendering halfway through the scene. It took almost an hour of debugging for me to finally confirm that the moment I was rendering a particular text string, the direct2D driver just <i>stopped</i>. No errors were thrown. No warnings could be found. Direct2D's failure state was apparently to simply make every single function call silently fail with no indication that it was failing in the first place. It didn't even tell me that the device was missing or that I needed to recreate it. The text render call was made and then every single subsequent call was ignored and the backbuffer was forever frozen to that half-drawn frame.<br /><br />The error itself didn't seem to make any more sense, either. I was passing a perfectly valid string to Direct2D, but because that string originated in a <i>different DLL</i>, it apparently made Direct2D completely shit itself. Copying the string onto the stack, however, worked (which itself could only work if the original string was valid).<br /><br />The cherry on top of all this is when I discovered that Direct2D's matrix rotation constructor takes <b>degrees</b>, not radians, like <i>every single other mathematical function in the standard library.</i> <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/bb205362(v=vs.85).aspx">Even DirectX takes radians!</a><br /><br /><i><b>WHO WRITES THESE APIs?!</b></i>Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-8722887060551518792017-01-17T14:35:00.000-08:002017-02-01T17:07:54.228-08:00Our Software Is a Beacon of HopeAs I draw closer to releasing the first early alpha of a library I've been working on for several years, I've noticed that it's release time coincides with some rather depressing politics that I wish I could ignore. Unfortunately, it's difficult to ignore politicians who seek to destroy the lives of your closest friends. A common theme I hear from my friends is a feeling of helplessness, as though we all know terrible things are happening, but no one really knows what to do about it.<br /><br />But we are not helpless. It is so easy to lose ourselves in pessimism and cynicism, to think that everything will continue to be shit simply because it's been shit for so many years. A common refrain I heard while still working at a large corporation was, <i>we know our code is shit, but it's okay, because everyone else's code is shit</i>. This kind of mentality really bothers me, not because it isn't true, but because it seems to reflect a resigned, cynical view of the world, and never strives to do better. Yes, everything might end up being shit in the end, but if we don't even attempt to build something better, we never will. If we hadn't been trying to write better code we never would have invented structured programming, or object oriented programming, or functional programming. Each innovation builds on the last, and each innovation has it's faults, but each brings us one step closer to not being shit.<br /><br />What disturbs me is that the software industry's inability to strive for better code now mirrors a feeling of political resignation, a malaise that seems to be settling on our polarized political discourse. If democracy isn't working, what else can we do? If the entire system is corrupt, what hope do we have of fixing it? As wealth inequality rises to <a href="http://www.cbsnews.com/news/worlds-8-richest-men-are-worth-as-much-as-3-6-billion-people-davos-world-economic-forum/">ever more absurd levels</a>, our society is tumbling towards a breaking point. Just like a badly maintained software stack, our society is threatening to topple over under economic stresses it was never built to handle. How much longer can we keep it running with duct-tape and terrible hacks? It is our duty to push for real change, to build a better foundation for our future using whatever skills or resources we have.<br /><br />As I prepare my library for its initial release, I'm not just zipping up a bunch of files to upload to GitHub. I'm lighting a fire, a beacon of hope that I will cast out into the endless night, praying that it might lead us to a brighter future. I am joined by millions of other flickering flames, many of which will fade with time. A few, however, will grow brighter and brighter, until they bring forth a new dawn and shepherd us into a bright new future, even if only for a moment, before sinking below the horizon. Our entire society is built on these brief moments of clarity. Some are longer than others, but all are important. Even when the night seems to stretch on without end, the rays of hope from a thousand stars can still point us in the right direction.<br /><br />We all wish for a brighter future, a future free from disease and poverty and war and hunger. We seek the future that we envision for ourselves in our stories and movies. Sometimes, it seems like society is going backwards, when everything is going wrong and nothing seems to go right for anyone. But it is in these moments that we must not forget the most important thing: we cannot wait for the future to come to us, we must build it ourselves. 2016 was not a very good year for many people. 2017 shows signs of being even worse, but I don't care.<br /><br />I'm going to <i>make</i> 2017 be a better year. I'm going to <i>make</i> the future I want to live in. I'm going to <i>build</i> the tools I wish I had. We are not pawns in some cosmic game, we are human beings. We are free agents capable of making our own decisions, and I'm making a decision to make the future happen, whether it wants to or not.<br /><br />I'm not letting this year take me down without a fight.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-51921010201018264442016-12-20T15:26:00.000-08:002016-12-22T07:03:44.408-08:00Everyone Does sRGB Wrong Because Everyone Else Does sRGB WrongDid you know that CSS3 does all its linear gradients and color interpolation completely wrong? All color values in CSS3 are in the sRGB color space, because that's the color space that gets displayed on our monitor. However, the problem is that the sRGB color space looks like this:<br /><br /><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/e/ef/SRGB_gamma.svg/250px-SRGB_gamma.svg.png" alt="sRGB gamma curve" /><br /><br />Trying to do a linear interpolation along a nonlinear curve doesn't work very well. Instead, you're supposed to linearize your color values, transforming the sRGB curve to the linear RGB curve before doing your operation, and then transforming it back into the sRGB curve. This is gamma correction. Here are comparisons between gradients, transitions, alpha blending, and image resizing done directly in sRGB space (assuming your browser complies with the W3C spec) versus in linear RGB:<br /><br /><table><tr><th>sRGB</th><th>Linear</th></tr><tr><td><div style="background:linear-gradient( 0deg, #FF0000, #00FF00);width:100px;height:100px;"></div></td><td style="vertical-align:center"><img src="https://dl.dropboxusercontent.com/u/755994/srgb_gradient.png" style="padding:0;border:0;"/></td></tr><tr><td><div id="srgbani" style="background:#FF0000;width:100px;height:100px;"></div></td><td><div id="linearani" style="background:#FF0000;width:100px;height:100px;"></div></td></tr><tr><td><img src="https://dl.dropboxusercontent.com/u/755994/srgb_radial.png" style="background:#FF0000;width:100px;height:100px;margin:0;padding:0;border:0;" /></td><td><img src="https://dl.dropboxusercontent.com/u/755994/srgb_alpha.png" style="padding:0;border:0;" /></td></tr><tr><td><img src="https://dl.dropboxusercontent.com/u/755994/srgb_resize.png" style="padding:0;border:0;" height="100" width="100" /></td><td><img src="https://dl.dropboxusercontent.com/u/755994/srgb_resize_correct.png" style="padding:0; height="100" width="100" /></td></tr></table><br />At this point you've probably seen a bazillion posts about how you're doing color interpolation wrong, or gradients wrong, or alpha blending wrong, and the reason is because... you're doing it wrong. But one can hardly blame you, because <b>everyone</b> is doing it wrong. CSS does it wrong because SVG does it wrong because Photoshop does it wrong because everyone else does it wrong. It's all wrong.<br /><br />The amazing thing here is that the W3C is <a href="https://lists.w3.org/Archives/Public/www-style/2012Jan/0635.html">entirely aware of how wrong CSS3 linear gradients are</a>, but did it anyway to be consistent with everything else that does them wrong. It's interesting that while SVG is wrong by default, it <i>does</i> provide a way to fix this, via <code>color-interpolation</code>. Of course, CSS doesn't have this property yet, so literally all gradients and transitions on the web are wrong because there is no other choice. Even if CSS provided a way to fix this, it would still have to default to being wrong.<br /><br />It seems we have reached a point where, after years of doing sRGB interpolation incorrectly, we continue to do it wrong not because we don't know better, but because everyone else is doing it wrong. So everyone's doing it wrong because everyone else is doing it wrong. A single bad choice done long ago has locked us into <b>compatibility hell</b>. We got it wrong the first time so now we have to keep getting it wrong because everyone expects the wrong result.<br /><br />It doesn't help that we don't always necessarily <i>want</i> the correct interpolation. The reason direct interpolation in sRGB is wrong is because it changes the perceived luminosity. Notice that when going from red to green, the sRGB gradient gets darker in the middle of the transition, while the gamma-corrected one has constant perceived luminosity. However, an artist may prefer the sRGB curve to the linearized one because it puts more emphasis on red and green. This problem gets worse when artists use toolchains inside sRGB and unknowingly compensate for any gamma errors such that the result is roughly what one would expect. This is a particular issue in games, because games really do need gamma-correct lighting pipelines, but the GUIs were often built using incorrect sRGB interpolation, so doing gamma-correct alpha blending gives you the wrong result because the alpha values were already chosen to compensate for incorrect blending.<br /><br />In short, this is quite a mess we've gotten ourselves into, but I think the most important thing we can do is give people the <i>option</i> of having a gamma correct pipeline. It is not difficult to selectively blend things with proper gamma correction. We need to have things like SVG's <code>color-interpolation</code> property in CSS, and other software needs to provide optional gamma correct pipelines (I'm looking at you, photoshop).<br /><br />Maybe, eventually, we can dig ourselves out of this sRGB hell we've gotten ourselves into.<br /><br /><script>function decimalToHex(d) { var hex = Math.floor(d).toString(16); hex = "000000".substr(0, 6 - hex.length) + hex; return hex; } var start = null; function fromlinear(x) { return (x <= 0.0031308) ? (x * 12.92) : (1.055*Math.pow(x, 1/2.4)) - 0.055; } function transitionlinear(x) { if(!start) start = x; var delta = x - start; var v = document.getElementById("srgbani"); var v2 = document.getElementById("linearani"); var p = (delta*0.0005)%1.0; v.style.backgroundColor = "#" + decimalToHex(Math.floor(p*256)*256 + Math.floor((1-p)*256)*256*256); v2.style.backgroundColor = "#" + decimalToHex(Math.floor(fromlinear(p)*256)*256 + Math.floor(fromlinear(1-p)*256)*256*256); window.requestAnimationFrame(transitionlinear); } window.requestAnimationFrame(transitionlinear); </script>Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-63227290540440712542016-11-14T21:57:00.001-08:002016-11-16T04:39:25.454-08:00The Answer Is Not More Censorship, It's More InformationAfter being accused of censoring conservative news, Facebook <a href="http://arstechnica.com/business/2016/08/facebook-fires-human-editors-algorithm-immediately-posts-fake-news/">fired all it's human editors</a>, which was shortly followed by the algorithm being inundated with fake news. It now appears to be <a href="http://www.nytimes.com/2016/11/14/technology/facebook-is-said-to-question-its-influence-in-election.html">regretting that choice</a> as it came under fire for potentially contributing to the rise of Trump.<br /><br />This is consistent with a disturbing trend in liberal media, which is to outright censor any speech that is considered even remotely hurtful. Predictably, the definition of "hurtful" speech has gotten wider and wider to the <a href="http://www.washingtontimes.com/news/2016/aug/1/ben-shapiro-banned-from-depaul-university-over-sec/">point of absurdity</a>, because <i>that's what happens when you censor things!</i> What kind perversion of progressivism is this? We're supposed to stand for freedom of information, intellectual free thought, and above all, <i>free speech</i>. There is a <a href="http://blackhole12.blogspot.com/2016/03/the-right-to-ignore-difference-between.html">difference between free speech and harassment</a>. You can ask people to be respectful when debating you, but you <i>cannot ask them to censor their opinions</i>, even if you find those opinions morally reprehensible. If someone wants to respectfully debate the morality of eating babies, you can't censor them for harassment because their only crime is holding an opinion that you disapprove of.<br /><br />Fake news is yet another problem that is not going to be solved by more censorship. If a platform censors information that people want, no matter how wrong it is, they will simply <i>find another platform</i> that lets them share it. This is why Fox News exists. Instead of outright censoring fake stories, websites need to <i>bring attention to falsehoods</i>. A handy warning message prominently displayed on the article linking to an appropriate Snopes page would be a good step. This way, people are still free to share their fake news stories, but because they're doing it on your platform, you can use this opportunity to give anyone who sees it additional information that allows them to recognize the story as false. If you just censor the story, people will <i>use another platform</i> that you have no control over.<br /><br /><a href="http://www.snopes.com/media/notnews/tsarnaev.asp"><img src="https://i.imgur.com/3q7FsJV.png" alt="Example of flagged news story"/></a><br /><br />The answer to filter bubbles is not to simply crush the filter bubble you don't agree with. The fact that this notion is even being entertained is frightening. Culture wars are not won by suffocation, they are won by <i>changing minds</i>, and the only way to do that is to respect free speech and be willing to debate people who have views that are wildly and totally incompatible with yours, so long as they are respectful. Otherwise, all we will do is scream at each other, and President Trump may only be the beginning of a catastrophic chasm in America.<br /><br />You can't change minds by throwing stones. This applies to both sides.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com1tag:blogger.com,1999:blog-4900186601661215466.post-35791160584298778442016-08-18T17:55:00.000-07:002016-08-18T22:14:34.902-07:00Blaming Inslee for Washington's Early Prisoner Release Is UnconscionableFor a guy who keeps trying to convince me he's better than Jay Inslee, Bill Bryant seems determined to ensure I never vote for him in <i>any</i> election, <i>ever.</i><br /><br />As <a href="http://www.seattletimes.com/seattle-news/politics/we-fact-check-the-first-inslee-bryant-debate"/>The Seattle Times</a> so eloquently explains, the software bug leading to the early release of prisoners dates back to 2002, and wasn't discovered by the DOC until 2012, a year <b>before</b> Inslee took office! Even then, Inslee didn't learn of the problem until December 16th, after which he promptly held a news conference less than a week later. He fired multiple state workers and immediately instituted a <a href="http://www.seattletimes.com/seattle-news/crime/state-moves-to-review-and-confirm-prison-sentences/">rigorous double-checking procedure at the DOC</a> to ensure prisoners were being released on time. Given Jay Inslee's position, I simply cannot imagine a better way to handle this, short of Jay Inslee miraculously developing mind reading powers that told him to audit the DOC for no apparent reason.<br /><br />The fact that Bryant is <i>blaming</i> Inslee simply because Inslee was the one who <i>discovered</i> the problem is <b>absolutely disgusting</b>. I hear people learning about officials covering up mistakes all the time, and people <i>always</i> ask why they didn't just come forward about the problem and try to fix it. Well, that's <i>exactly what Inslee did</i>, and now Bryant is dragging him through the mud for it. I guess Bill Bryant is determined to make sure no good deed goes unpunished.<br /><br />What kind of depraved human being would crucify Inslee for something that isn't his fault? An even better question is why they think I would vote for them. There are plenty of other <i>real</i> issues facing Jay Inslee that would be entirely valid criticisms of his administration, but I honestly don't care about them anymore. The only thing I know is that I don't want Bill Bryant elected to any position in any public office, ever, until he publicly apologizes for his morally reprehensible behavior towards Jay Inslee. Washington state does not need a leader that attacks people for doing the right thing just so they can get a few votes. No state in the entire country deserves a leader like that.<br /><br />It's clear to me that Bill Bryant, and anyone who supports his shameless mudslinging, is the reason we can't have nice things.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-67974143859162687982016-07-30T22:18:00.002-07:002016-07-31T03:20:35.924-07:00Mathematical Notation Is AwfulToday, a friend asked me for help figuring out how to calculate the standard deviation over a discrete probability distribution. I pulled up <a href="http://blackhole12.blogspot.com/2013/05/course-notes.html">my notes from college</a> and was able to correctly calculate the standard deviation they had been unable to derive after hours upon hours of searching the internet and trying to piece together poor explanations from questionable sources. The crux of the problem was, as I had suspected, the astonishingly bad notation involved with this particular calculation. You see, the expected value of a given distribution $$X$$ is expressed as $$E[X]$$, which is calculated using the following formula:<br />\[ E[X] = \sum_{i=1}^{\infty} x_i p(x_i) \]<br />The standard deviation is the square root of the variance, and the variance is given in terms of the expected value.<br />\[ Var(X) = E[X^2] - (E[X])^2 \]<br />Except that $$E[X^2]$$ is of course completely different from $$(E[X])^2$$, but it gets worse, because $$E[X^2]$$ makes <i>no notational sense whatsoever</i>. In any other function, in math, doing $$f(x^2)$$ means going through and substitution $$x$$ with $$x^2$$. In this case, however, $$E[X]$$ actually doesn't have anything to do with the resulting equation, because $$X \neq x_i$$, and as a result, the equation for $$E[X^2]$$ is this:<br />\[ E[X^2] = \sum_i x_i^2 p(x_i) \]<br />Only the first $$x_i$$ is squared. $$p(x_i)$$ isn't, because it doesn't make any sense in the first place. It should really be just $$P_{Xi}$$ or something, because it's a <i>discrete value</i>, not a function! It would also explain why the $$x_i$$ inside $$p()$$ isn't squared - because it <i>doesn't even exist</i>, it's just a gross abuse of notation. This situation is so bloody confusing I even explicitely laid out the equation for $$E[X^2]$$ in my own notes, presumably to prevent me from trying to figure out what the hell was going on in the middle of my final.<br /><br />That, however, was only the beginning. Another question required them to find the covariance between two seperate discrete distributions, $$X$$ and $$Y$$. I have never actually done covariance, so my notes were of no help here, and I was forced to <a href="https://en.wikipedia.org/wiki/Covariance">return to wikipedia</a>, which gives this helpful equation.<br />\[ cov(X,Y) = E[XY] - E[X]E[Y] \]<br />Oh shit. I've already established that $$E[X^2]$$ is impossible to determine because the notation doesn't rely on any obvious rules, which means that $$E[XY]$$ could evaluate to <i>god knows what</i>. Luckily, wikipedia has an alternative calculation method:<br />\[ cov(X,Y) = \frac{1}{n}\sum_{i=1}^{n} (x_i - E(X))(y_i - E(Y)) \]<br />This almost works, except for two problems. One, $$\frac{1}{n}$$ doesn't actually work because we have a nonuniform discrete probability distribution, so we have to substitute multiplying in the probability mass function $$p(x_i,y_i)$$ instead. Two, wikipedia refers to $$E(X)$$ and $$E(Y)$$ as the <b>means</b>, not the expected value. This gets even more confusing because, at the beginning of the Wikipedia article, it used brackets ($$E[X]$$), and now it's using parenthesis ($$E(X)$$). Is that the same value? Is it something completely different? Calling it the mean would be confusing because the average of a given data set isn't necessarily the same as finding what the average <i>expected value</i> of a probability distribution is, which is why we call it the <i>expected value</i>. But naturally, I quickly discovered that <b>yes</b>, the mean and the average and the expected value are all <b>exactly the same thing!</b> Also, I still don't know why Wikipedia suddenly switched to $$E(X)$$ instead of $$E[X]$$ because it stills means <i>the exact same goddamn thing</i>.<br /><br />We're up to, what, five different ways of saying <i>the same thing?</i> At least, I'm assuming it's the same thing, but there could be some incredibly subtle distinction between the two that nobody ever explains anywhere except in some academic paper locked up behind a paywall that was published 30 years ago, because apparently mathematicians are okay with this.<br /><br />Even then, this is just <i>one instance</i> where the ambiguity and redundancy in our mathematical notation has caused enormous confusion. I find it particularly telling that the most difficult part about figuring out any mathematical equation for me is usually to simply figure out what all the goddamn notation even <i>means</i>, because usually most of it isn't explained at all. Do you know how many ways we have of taking the derivative of something?<br /><br />$$f'(x)$$ is the same as $$\frac{dy}{dx}$$ or $$\frac{df}{dx}$$ even $$\frac{d}{dx}f(x)$$ which is the same as $$\dot x$$ which is the same as $$Df$$ which is technically the same as $$D_xf(x)$$ and also $$D_xy$$ which is also the same as $$f_x(x)$$ provided x is the only variable, because taking the partial derivative of a function with only one variable is the exact same as taking the derivative in the first place, and I've actually seen math papers abuse this fact instead of use some other sane notation for the derivative. And that's just for the derivative!<br /><br />Don't even get me started on multiplication, where we use $$2 \times 2$$ in elementary school, $$*$$ on computers, but use $$\cdot$$ or simply stick two things next to each other in traditional mathematics. Not only is using $$\times$$ confusing as a multiplicative operator when you have $$x$$ floating around, but it's a <i>real operator!</i> It means <a href="https://en.wikipedia.org/wiki/Cross_product">cross product</a> in vector analysis. Of course, the $$\cdot$$ also doubles as meaning the Dot Product, which is at least nominally acceptable since a dot product does reduce to a simple multiplication of scalar values. The Outer Product is generally given as $$\otimes$$, unless you're in Geometric Algebra, in which case it's given by $$\wedge$$, which of course means <b>AND</b> in <a href="https://en.wikipedia.org/wiki/Logical_conjunction">binary logic</a>. Geometric Algebra then re-uses the cross product symbol $$\times$$ to instead mean <i>commutator product</i>, and also defines the <i>regressive product</i> as the dual of the outer product, which uses $$\nabla$$. This conflicts with the <a href="https://en.wikipedia.org/wiki/Gradient">gradient operator</a> in multivariable calculus, which uses the exact same symbol in a totally different context, and just for fun it also defined $$*$$ as the "scalar" product, just to make sure every possible operator has been violently hijacked to mean something completely unexpected.<br /><br />This is just <i>one area</i> of mathematics - it is common for many different subfields of math to redefine operators into their own meaning and god forbid any of these fields actually come into contact with each other because then no one knows what the hell is going on. Math is a language that is about as consistent as English, and that's on a good day.<br /><br />I am sick and tired of people complaining that nobody likes math when they refuse to admit that mathematical notation sucks, and is a major roadblock for many students. It is useful only for advanced mathematics that take place in university graduate programs and research laboratories. It's hard enough to teach people calculus, let alone expose them to something useful like statistical analysis or matrix algebra that is relevant in our modern world when the notation looks like Greek and makes about as much sense as the English pronunciation rules. We simply cannot introduce people to advanced math by writing a bunch of incoherent equations on a whiteboard. We need to find a way to separate the underlying mathematical <i>concepts</i> from the arcane scribbles we force students to deal with.<br /><br />Personally, I understand most of higher math by reformulating it in terms of lambda calculus and type theory, because they map to real world programs I can write and investigate and <i>explore</i>. Interpreting mathematical concepts in terms of computer programs is just one way to make math more tangible. There must be other ways we can explain math without having to explain the extraordinarily dense, outdated notation that we use.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com11tag:blogger.com,1999:blog-4900186601661215466.post-25003069453740868852016-04-28T14:23:00.002-07:002016-04-28T14:23:41.319-07:00The GPL Is Usually OverkillSomething that really bothers me about the GPL and free software crusaders in general is that they don't seem to understand the nuances behind the problem they are attempting to solve. I'm not entirely sure they are even trying to solve the right problem in the first place.<br /><br />The core issue at play here is <i>control</i>. In a world run by software, we need control over what code is being executed on our hardware. This issue is of paramount importance as we move into an age of autonomous vehicles and wearable devices. Cory Doctorow's brilliant essay, <a href="http://boingboing.net/2012/01/10/lockdown.html">Lockdown: The coming war on general-purpose computing</a>, gives a detailed explanation of precisely <i>why</i> it is of such crucial importance that <i>you</i> have control over what software gets executed on your machine, and not some corporation. Are you really going to buy a car and then get into it when you have no way of controlling what it does? This isn't some desktop computer anymore, it's a machine that can decide whether you live or die.<br /><br />It is completely true that, if everything was open source, we would then have total control over everything that is being run on our machines. However, proponents of open source software don't seem to realize that this is the <i>nuclear option</i>. Yes, it solves your problem, but it does so via massive overkill. There are much less extreme ways to achieve exactly what you want that don't involve requiring <i>literally everything</i> to be open source.<br /><br />Our goal is to ensure that only trusted code is run on our machine. In order to achieve this goal, the firmware and operating system <b>must be open source</b>. This is because both the operating system and the firmware act as gatekeepers - if we have something like Intel's Trusted Execution Technology, we absolutely must have access to the firmware's source code, because it is the first link in our <i>trust chain</i>. If we can make the TXT engine work for <i>us</i>, we can use it to ensure that only operating systems we approve of can be run on our hardware.<br /><br />We now reach the second stage of our trust chain. By using a boot-time validation of a cryptographic signature, we can verify that we are running an operating system of our choice. Because the operating system is what implements the necessary program isolation and protection mechanisms, it too is a required part of our trust chain and therefore <i>must be open source</i>. We also want the operating system to implement some form of permission restriction - ideally it would be like Android, except that <i>anything</i> running on the OS must explicitly tell you what resources it needs to access, not just apps you download from the store.<br /><br />And... <i>that's it</i>. Most free software zealots that are already familiar with this chain of events would go on to say that you should only install open source software and whatever, but in reality this is unnecessary. You certainly want open source drivers, but once you have control over the firmware and the operating system, no program can be run without your permission. Any closed-source application can only do what I have given it permission to do. It could phone home without my knowledge or abuse the permissions I have granted it in other ways, but the hardware is still under my control. I can simply uninstall the application if I decide I don't like it. Applications cannot control my hardware unless I give them explicit permission to do so.<br /><br />Is this enough? The FSF would argue that, no, it's not enough until your entire computer is running only open source code that can be verified and compiled by yourself, but I disagree. At some point, you have to trust <i>someone</i>, somewhere, as demonstrated by the <a href="http://c2.com/cgi/wiki?TheKenThompsonHack">Ken Thompson Hack</a>. I'm fine with trusting a corporation, but I need have total control over who I am trusting. If a corporation controls my firmware or my operating system, then <b>they control who I trust</b>. I can ask them politely to please trust or not trust some program, but I can't actually be sure of anything, because I do not control the root of the trust chain. <b>Open source software is important for firmware and operating systems, because it's the only way to establish a chain of trust that you control</b>. However, once I have an existing chain of trust, I don't need open-source programs anymore. <i>I now control what runs on my computer</i>. I can tell my computer to never run anything written by Oracle, and be assured that it will actually never trust anything written by Oracle.<br /><br />If I sell you a piece of software, you have a right to decompile it, modify it, hack it, or uninstall it. This does not, however, require me to explain how it works to you, nor how to assemble it. If you trust me, you can run this piece of software on your hardware, but if you don't trust me, then a verifiable chain of trust will ensure that my software <i>never</i> runs on your machine. <i>This</i> is what matters. This is what having control means. It means being able to say what does or doesn't run on your computer. It does not require that everything is open source. It only requires an open source, verifiable chain of trust, and faith in whatever company made the software you've chosen to run. If a company's software is proprietary and they do something you don't like, <i>use something else</i>.<br /><br />Open source code is important for <i>firmware and operating systems</i>, not <i>everything</i>.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0tag:blogger.com,1999:blog-4900186601661215466.post-63921895843644221702016-03-13T21:27:00.001-07:002016-03-13T21:27:47.065-07:00The Right To Ignore: The Difference Between Free Speech And HarassmentOn one hand, America was built on the values of <b>free speech</b>, which are obviously important. We cannot control what people are saying or our democracy falls apart. On the other hand, allowing <b>harassment</b> has a stifling effect on free speech, because it allows people to be bullied into silence. Before the internet, this distinction was fairly simple: Someone with a megaphone screaming hate speech in a park is exercising their right to free speech. Someone with a megaphone following a guy down the street is harassing them.<br /><br />The arrival of the internet has made this line much more vague. However, the line between free speech and harassment is not nearly as blurry as some people might think. Our concept of <i>reasonable</i> freedom of speech is guided primarily by our <i>ability to ignore it</i>. The idea is, someone can go to a public park and say whatever they want, because other people can simply go somewhere else. As long as people have the ability to <i>ignore</i> whatever you're saying, you can say pretty much whatever you want. We have some additional controls on this for safety reasons, so you can't go to a park and talk about how you're going to kill all the gay people, tomorrow, with a shotgun, because that is a death threat.<br /><br />Unfortunately, in the midst of defending free speech, a disturbing number of people have gotten it into their heads that other people <i>aren't allowed to ignore them</i>. <b>This is harassment</b>. The moment you take away someone's right to ignore what you are saying, you have crossed the line from free speech into harassment. Freedom of speech means that you have the right to say whatever you want, and <b>everyone else has the right to ignore you</b>. I'm not entirely sure <i>why</i> people think that free speech somehow lets them bypass blocking on a social network. Blocking people on the internet is what gives us our ability to ignore them. Blocking someone on the internet is the equivalent of walking away from some guy in the park who was yelling obscenities at you. If you bypass the blocking mechanism, this is basically <i>chasing them down as they run to their car, screaming your opinions at them with a megaphone</i>. Most societies think this is completely unacceptable behavior in real life, and it should be just as unacceptable on the internet.<br /><br />On the other hand, <b>enforcing political correctness is censorship</b>. Political correctness is obviously something that should be <i>encouraged</i>, but enforcing it when someone is saying things you don't like in a public place is a <b>clear violation of free speech</b>. This is not a blurry line. This is not vague. If a Nazi supporter is saying how much they hate Jews, and they are not targeting this message at any one individual, <i>this is clearly protected free speech</i>. Now, if the Nazi is actually saying we should <i>murder</i> all of the Jews, this turns into hate speech because it is inciting violence against a group of people, and is restricted for the same safety reasons that prevent you from shouting "FIRE!" in a crowded movie theater.<br /><br />Now, what if the Nazi supporter tells a <i>specific person</i> they are a dirty Jew? This gets into fuzzy territory. On one hand, we can say it isn't harassment so long as the targeted person can block the Nazi and the Nazi never attempts to circumvent this, or we can start classifying certain speech as being harassment if it is <i>ever</i> targeted at a specific individual. When we are debating what counts as harassment, <i>this</i> is the kind of stuff we should be debating. <i>This</i> is what society needs to figure out. Arguing that <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&cad=rja&uact=8&ved=0ahUKEwj0hOTPob_LAhVM5GMKHeYmC4AQFggtMAM&url=http%3A%2F%2Ffortune.com%2F2015%2F06%2F12%2Ftwitter-free-speech%2F&usg=AFQjCNEMd-RJPwTUw4eB3Pl_uj-ZgY3SqQ&sig2=Wi4Jh0D0eOVwn51TC5RGqQ">mass-block lists on twitter are hurting free speech</a> is absurd. If someone wants to disregard everything you're saying because you happen to be walking past a group of Nazi supporters, <i>they have a right to do that</i>. If someone wants to ignore <i>everyone else on the planet</i>, they have a right to do that.<br /><br />This kind of stuff is a problem when the vast majority of humanity uses a single <b>private</b> platform to espouse their opinions. Namely, Facebook. This is because Facebook has recently been <a href="http://www.vice.com/read/facebook-wont-let-faggots-say-faggot">banning people for saying faggot</a>, and because it's a private company, <i>this is completely legal</i>. This is also harmful to free speech. What appears to be happening is that websites, in an attempt to crack down on <i>harassment</i>, have instead accidentally started cracking down on <i>free speech</i> by outright banning people who say hateful things, instead of focusing on people who say hateful things <b>to specific individuals</b>.<br /><br />Both sides of the debate are to blame for this. Extreme harassment on the internet has caused a backlash, resulting in a politically correct movement that calls for tempering all speech that might be even slightly offensive to anyone. In response, free speech advocates overreact and start attacking them by bypassing blocking mechanisms and <b>harassing their members</b>. This causes SJWs to overcompensate and start clamping down on <b>all</b> hateful speech, even hateful speech that is <b>clearly protected free speech</b> and has nothing to do with harassment. This just pisses off the free speech movement even more, causing them to oppose <i>any</i> restrictions on free speech, even reasonable ones. This just encourages SJWs to censor even more stuff in an endless spiral of aggression and escalation, until on one side, everyone is screaming obscenities and racial slurs in an incoherent cacophony, and on the other side, no one is saying anything at all.<br /><br />If society is going to make any progress on this at all, we need to hammer out precisely what constitutes harassment and what is protected free speech. We need to make it absolutely clear that you can only harass an <i>individual</i>, and that <b>everyone</b> has the right to a block button. If a Nazi supporter starts screaming at you on twitter, you have the right to block him, and he has the right to block you. <b>We cannot solve harassment with censorship</b>. Instead of banning people we disagree with, we need to focus on building tools that let us ignore people we don't want to listen to. If you object to everyone blocking you because you use insulting racial epithets all the time, <i>maybe you should stop doing that</i>, and perhaps some of them might actually listen to you.<br /><br />Of course, if you disagree with me, you are welcome to exercise your right to tell me exactly how much you hate my guts and want me to die in the comments section below, and I will exercise my right to completely ignore everything you say.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com4tag:blogger.com,1999:blog-4900186601661215466.post-32600812708713166672016-02-08T03:41:00.001-08:002016-02-08T16:57:18.482-08:00Standard Intermediate Assembly For CPUsOver in GPU land, a magical thing is happening. All the graphics card vendors and big companies got together and came up with SPIR-V as the technological underpinning of <a href="https://www.khronos.org/vulkan/">Vulkan</a>, the as-of-yet unreleased new graphics API. SPIR-V is a cross-platform binary format for compiled shaders, which allows developers to use any language that can compile to SPIR-V to write shaders, and to run those compiled shaders on any architecture that supports SPIR-V. This is big news, and if it works as well as everyone's hoping it does, it will set the stage for a major change in how shaders are compiled in graphics engine toolchains.<br /><br />How is this possible? The <a href="https://www.khronos.org/registry/spir-v/specs/1.0/SPIRV.pdf">SPIR-V specification</a> tells us that it is essentially a <i>cross-platform intermediate assembly language</i>. It's higher level than conventional assembly, but lower than an actual language. The specifics of the language are fine-tuned towards modern graphics hardware, so that the instructions can encode sufficient metadata about what they're doing to enable hardware optimizations, while still allowing those instructions to be efficiently decoded by the hardware and implemented by the chip's microcode. <br /><br />While SPIR-V is specifically designed for GPUs, the specification bears some resemblance to another intermediate assembly language - <a href="http://llvm.org/docs/LangRef.html">LLVM IR</a>. This move by GPU vendors towards an intermediate assembly representation mirrors how modern language design is moving towards a standardized intermediate representation that many different languages compile to, which can then itself be compiled to any CPU architecture required. The LLVM IR is used for C, C++, Haskell, Rust, and <a href="http://llvm.org/ProjectsWithLLVM/">many others</a>. This intermediate representation decouples the underlying hardware from the high level languages, allowing <i>any</i> language that compiles down to LLVM IR to compile to any of it's supported CPU architectures - <a href="https://github.com/kripken/emscripten">even asm.js</a>.<br /><br />However, we have a serious problem looming over our heads in CPU land. Did you know that the x86 mov instruction <a href="https://www.cl.cam.ac.uk/~sd601/papers/mov.pdf">is turing complete</a>? In fact, even <a href="https://github.com/jbangert/trapcc">the page fault handler is turing complete</a>, so you can run programs on x86 <i>without actually executing any instructions!</i> The x86 architecture is so convoluted and bloated that it no longer has any predictable running time for any given set of instructions. <a href="http://research.google.com/pubs/pub37077.html">Inserting random useless mov instructions</a> can increase speed by 5% or more, <a href="http://stackoverflow.com/questions/25078285/replacing-a-32-bit-loop-count-variable-with-64-bit-introduces-crazy-performance">false dependencies</a> destroy performance, and Intel keeps introducing more and more complex instruction sets that <a href="http://techreport.com/news/26911/errata-prompts-intel-to-disable-tsx-in-haswell-early-broadwell-cpus">don't even work properly</a>. As a result, it's extremely difficult for any compiler to produce properly optimized assembly code, <i>even when it's targeting a specific architecture</i>.<br /><br />One way to attack this problem is to advocate for RISC - Reduced Instruction Set Computer. The argument is that fewer instructions will be easier to implement, reducing the chance of errors and making it easier for compilers to actually optimize the code in a meaningful way. Unfortunately, RISC has a serious problem: the laws of physics. A modern CPU is so fast that it can process an instruction faster than the electrical signal can get to the other side of the chip. Consequently, it spends the majority of it's time just <a href="http://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory">waiting for memory</a>. Both pipelining and branch prediction were created to deal with the memory latency problem, and it turns out that having complex instructions gives you a distinct advantage. The more complex your instruction is, the more the CPU has to do before it needs to fetch things from memory. This was the core observation of the Itanium instruction set, which relies on the compiler to determine which instructions can be executed in parallel in an attempt to remove the need for pipelining. Unfortunately, it turns out that removing dependency calculations is not enough - this is why many of Intel's new instructions are about <a href="https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions">encapsulating complex behaviors into single instructions</a> instead of simply adding more parallel operators.<br /><br />Of course, creating hardware that supports increasingly complex operations is very unsustainable, which is why modern CPUs don't execute assembly instructions directly. Instead, they use <a href="https://en.wikipedia.org/wiki/Microcode">Microcode</a>, which is the raw machine code that actually implements the "low-level" x86 assembly. Of course, at this point, x86 is so far removed from the underlying hardware it might as well be a (very crude) high level language all by itself. For example, the mov instruction usually doesn't actually move anything, it just <a href="https://en.wikipedia.org/wiki/Register_renaming">renames the internal register</a> being used. Because of this, the modern language stack looks something like this:<br /><br /><center><img src="https://googledrive.com/host/0B_2aDNVL_NGmcHlBbmV6T3lGdTg/" alt="Modern Language Stack" /></center><br />Even thought we're talking about four CPU architectures, what we really have is four competing intermediate layers. x86, x86-64, ARM and Itanium are all just crude abstractions above the CPU itself, which has it's own architecture dependent microcode that actually figures out how to run things. Since our CPUs will inevitably have complex microcode no matter what we do, why not implement something else with it? <b>What if the CPUs just executed LLVM IR directly?</b> Then we would have this:<br /><br /><center><img src="https://googledrive.com/host/0B_2aDNVL_NGmZVlyZVRoUlVtLVE/" alt="LLVM IR Microcode Stack" /></center><br />Instead of implementing x86-64 with microcode, implement the LLVM intermediate assembly code with microcode. This would make writing platform-independent code trivial, and would allow for way more flexibility for hardware designers to experiment with their CPU architecture. The high-level nature of the instructions would allow the CPU to load large chunks of data into registers for complex operations and perform more efficient optimizations with the additional contextual information.<br /><br /><b>Realistically, this will probably never happen</b>. For one, directly executing LLVM IR is probably a bad idea, because it was never developed with this in mind. Instead, Intel, AMD and ARM would have to cooperate to create something like SPIR-V that could be efficiently decoded and implemented by the hardware. Getting these competitors to actually cooperate with each other is the biggest obstacle to implementing something like this, and I don't see it happening anytime soon. Even then, a new standard architecture wouldn't replace LLVM IR, so you'd still have to compile to it.<br /><br />In addition, an entire new CPU architecture is extraordinarily unlikely to be widely adopted. One of the primary reasons x86-64 won out over Itanium was because it was capable of running x86 code at native speed, and Itanium's x86 emulation was <a href="https://en.wikipedia.org/wiki/Itanium#High-end_server_market">notoriously bad</a>. Even if we somehow moved to an industry-wide standard assembly language, the vast majority of the world's programs are still built for x86, so an efficient translation between x86 and our new intermediate representation would be paramount. That's without even considering that you'd have to recompile your OS to take advantage of the new assembly language, and modern OSes still have some platform-specific hand-written assembly in them.<br /><br />Sadly, as much as I like this concept, it will probably remain nothing more than a thought experiment. Perhaps as we move past the age of silicon and <a href="http://arstechnica.com/gadgets/2015/02/intel-forges-ahead-to-10nm-will-move-away-from-silicon-at-7nm/">look towards new materials</a>, we might get some new CPU architectures out of it. Maybe if we keep things like this in mind, next time we can do a better job than x86.Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com1tag:blogger.com,1999:blog-4900186601661215466.post-51460325123338890572015-09-28T03:07:00.000-07:002015-09-28T05:13:08.686-07:00There Will Never Be One True Programming LanguageA disturbing number of otherwise intelligent programmers seem to believe that, one day in the distant future, everyone will use the same magical programming language that solves everyone's problems at the same time. Often, this involves garbage collection, with lots of hand-waving about computers built with <a href="http://tvtropes.org/pmwiki/pmwiki.php/Main/AppliedPhlebotinum">Applied Phlebotinum</a>.<br /><br />For the sake of argument, let's assume it is the year 30XX, and we are a budding software developer on Mars. We have quantum computers that run on magic pixie dust and let us calculate almost anything we want as fast as we want so long as we don't break any laws of physics. Does everyone use the same programming language?<br /><br /><i><b>No.</b></i> Obviously, even if we built quantum computers capable of simulating a classical computer a thousand times faster than normal for some arbitrary reason<sup><a href="#foot1">[1]</a></sup>, we would still want to write software in a way that was easy to comprehend, and qubits are <a href="https://en.wikipedia.org/wiki/Qubit">anything but</a>. So, the standard programming language our Mars programmer would be using would not interact with the quantum computer at all. Perhaps it would be some form of functional language, which the cool kids seem to think will save the entire software industry and make everything faster, better, more reliable and probably cure cancer in the process<sup><a href="#foot2">[2]</a></sup>.<br /><br />But now, this same programming language that allows you to ignore the quantum computer must also be used to write it's own compiler that runs on... a quantum computer. So it needs to simultaneously be an elegant language for writing quantum programs. It also needs to be the perfect language for writing games and business applications and hypernet sites and machinery and brain implants and rocket trajectories and robot AI and planetary weather simulators and life support systems and hologram emittors and&mdash;<br /><br />... Wouldn't it be a lot easier to just have domain-specific languages?<br /><br />If you want to get technical, you can do all that with Lisp. It's just lambda calculus. It's also a <i>giant pain in the ass</i> when you use it for things that it was never meant for, like low level drivers or game development. You could try extending Lisp to better handle those edge-cases, and that's exactly what many lisp-derivatives do. The problem is that <b>we will always be able to find new edge-cases</b>. So, you would have to continue bolting things on to your magical language of ultimate power, playing a game of context-free whack-a-grammar for the rest of eternity.<br /><br />This is ultimately doomed to fail, because it ignores the underlying purpose of programming languages: A programming language is a <b>method of communication</b>. Given two grammars, one that can do everything versus one that can do the one thing you're trying to do <i>really well</i>, which one are you going to pick for that particular project? Trying to get everyone to use the same language while they are trying to solve radically different problems is simply <i>bad engineering</i>. Do you <i>really</i> want to write your SQL queries using Lisp? You can only stuff so many tools into a swiss army knife before it becomes <a href="http://www.wengerna.com/stuff/contentmgr/files/0/a45137daa224e9531cb3050458faee64/image/wenger_giant_knife.png">completely impractical</a>. <br /><br />Of course, a language that could do <i>everything</i> would also allow you to define any arbitrary new language within it, but this is equivalent to building an entire new language from scratch, because now everyone who wants to contribute to your project has to learn your unique grammar. However, having a common ground for different languages is a powerful tool, and we are headed in that direction with LLVM. You wouldn't want to write a program in LLVM, but it sure makes writing a compiler easier. <br /><br />Instead of a future with one true programming language, perhaps we should be focused on a future with <b>standard low-level assembly</b>. A future with many different languages that can all talk to each other. A future where we don't <i>need</i> to argue about which programming language is the best, because every language could be used for whatever task it is best suited for, all in the same project. Different compilers could interpret the standard low-level assembly in their own ways, optimizing for different use-cases.<br /><br />A universal standard for low-level assembly would <b>solve everyth&mdash;</b><br /><br /><center><img src="https://imgs.xkcd.com/comics/standards.png" alt="mandatory xkcd" title="Mars insisted on making semicolons optional, which rendered their code completely incompatible with Earth's code. The resulting Great Semicolon War left the entire northern hemisphere of Earth uninhabitable." /></center><br />... Actually, nevermind<sup><a href="#foot3">[3]</sup>.<br /><br /><hr><br /><span style="font-size:80%"><sup id="foot1">1</sup></a> As of right now, this is not actually true for the vast majority of computations we know how to do with qubits. We know that <i>certain classes of problems</i> are exponentially faster with quantum computers, but many other functions would get a mere linear increase in speed, <i>at most</i>. Whether or not this will change in the future is an active area of research.<br /><sup id="foot2">2</sup> They <i>totally</i> didn't say the same thing about object-oriented programming 30 years ago.<br /><sup id="foot3">3</sup> <i>&lt;/sarcasm&gt;</i><br /></span>Erik McClurehttps://plus.google.com/104896885003230920472noreply@blogger.com0