Alice GoldfussJekyll2018-12-27T21:11:51-05:00https://blog.alicegoldfuss.com/Alice Goldfusshttps://blog.alicegoldfuss.com/you@email.comhttps://blog.alicegoldfuss.com/2018-year-in-review2018-12-27T00:00:00-00:002018-12-27T00:00:00-05:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>2018, the year that felt like ten. Time hopped and jogged and stood still, twisted me all around, chewed me up and spat me out with every horrible update of the news cycle.</p>
<!-- more -->
<p>Still, this year wasn’t all bad! I gave <a href="https://www.youtube.com/watch?v=sJx_emIiABk" target="_blank">my first keynote</a>, saw some of <a href="https://githubengineering.com/glb-director-open-source-load-balancer/" target="_blank">my code open-sourced</a>, and attended a Frida Kahlo exhibit at the V&amp;A. I started my year with istio and kernel crashes, progressed to large Kubernetes cluster operations, and finished by starting a new perf project. I traveled, gardened, donated, baked, wrote, danced, built a CPU out of logic gates for funzies, and sipped tea. I ended up on <a href="https://www.huffingtonpost.com/entry/50-funniest-tweets-from-women-2018_us_5c195b7be4b08db990588964" target="_blank">this weird HuffPost list</a>. And, oh yeah, I went through a <a href="https://news.microsoft.com/2018/06/04/microsoft-to-acquire-github-for-7-5-billion/" target="_blank">teensy acquisition</a>.</p>
<p><img src="/images/nyc_garden.JPG" alt="Photo of small water lily pads in foreground, large lily pads in mid-ground, and a large glasshouse in the background." /></p>
<figcaption>The New York Botanical Garden</figcaption>
<p>In my <a href="http://blog.alicegoldfuss.com/2017-year-in-review/" target="_blank">2017 review post</a>, I said I wanted to do more lower level systems work, technical talks, and leisure reading. I’ve done that. For 2019, I’d like to find a balance between sharing my excitement over learning new things and broadcasting my already-considerable technical knowledge. I’m very bad at self-advertisement.</p>
<p>And now, some lists.</p>
<h2 id="talks-given">Talks given</h2>
<ol>
<li>The Lead Dev London (keynote)</li>
<li>Nike HQ</li>
<li>Portland DevOpsDays (emcee)</li>
<li>Velocity NY</li>
</ol>
<h2 id="concerts-attended">Concerts attended</h2>
<ol>
<li>Lorde</li>
<li>Janelle Monae</li>
<li>Lizzo</li>
</ol>
<p><img src="/images/Lorde.JPG" alt="Stage photo of Lorde being lifted and carried by a small group of dancers underneath a large projected display of five lighted panels fanned out. The panels are displaying a broken up image of a ship at sea." /></p>
<figcaption>Lorde</figcaption>
<h2 id="magazine-articles-written">Magazine articles written</h2>
<ol>
<li><a href="https://increment.com/security/they-know-where-you-live/" target="_blank">They Know Where You Live</a></li>
</ol>
<h2 id="books-enjoyed">Books enjoyed</h2>
<ol>
<li><a href="https://www.goodreads.com/book/show/8474434-linux-kernel-development" target="_blank">Linux Kernel Development</a> - great tour (albeit slightly outdated) of the kernel and operating systems in general</li>
<li><a href="https://www.goodreads.com/book/show/38496769-the-broken-earth-trilogy" target="_blank">The Broken Earth Trilogy</a> - expertly-crafted scifi about an enslaved people and a broken world</li>
<li><a href="https://www.goodreads.com/book/show/29751398-the-power" target="_blank">The Power</a> - <em>Inglourious Basterds</em> for women, best applied after remembering Brett Kavanaugh exists</li>
<li><a href="https://www.goodreads.com/book/show/35099718-so-you-want-to-talk-about-race" target="_blank">So You Want to Talk About Race</a> - you might think you don’t need to read this, but you’re wrong</li>
<li><a href="https://www.goodreads.com/book/show/12232938-the-lovely-bones" target="_blank">The Lovely Bones</a> - do you like to cry?</li>
</ol>
<p><img src="/images/ALU.JPG" alt="Whiteboard with an ALU drawn on it in green marker. The ALU is made up of different logic gate symbols." /></p>
<figcaption>My ALU</figcaption>
<h2 id="youtube-series-consumed">YouTube series consumed</h2>
<ol>
<li><a href="https://www.youtube.com/playlist?list=PLKtIunYVkv_S7LqWqRuGw1oz-1zG3dIL4" target="_blank">It’s Alive!</a></li>
<li><a href="https://www.youtube.com/watch?v=urk_kp76RoI&amp;list=PLsMtUWKCmBPTDOgDWhM-AWMhPS-XuLqNs" target="_blank">Anime Crimes Division</a></li>
<li><a href="https://www.youtube.com/user/bgfilms" target="_blank">Binging with Babish</a></li>
</ol>
<h2 id="best-tweet-according-to-twitter-analytics">Best tweet, according to Twitter Analytics</h2>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I&#39;m into<br /><br />⚪️ Men<br />⚪️ Women<br />🔘 Researching the menu ahead of time so I know exactly what to order, including side dishes and dessert</p>&mdash; ms claws (@alicegoldfuss) <a href="https://twitter.com/alicegoldfuss/status/1024405975363510272?ref_src=twsrc%5Etfw">July 31, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="best-tweet-according-to-me">Best tweet, according to me</h2>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Petition to treat genders like operating systems<br /><br />&quot;Are you a man or a woman?&quot;<br />&quot;Oh, I dual boot.&quot;</p>&mdash; ms claws (@alicegoldfuss) <a href="https://twitter.com/alicegoldfuss/status/1024331334661992448?ref_src=twsrc%5Etfw">July 31, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="some-numbers">Some numbers</h2>
<ul>
<li><strong>1</strong> - tarantulas held</li>
<li><strong>5</strong> - carnivorous plants tended</li>
<li><strong>7</strong> - jars of homemade pickles currently in my fridge</li>
<li><strong>9</strong> - scones eaten (estimated)</li>
<li><strong>23</strong> - gifts donated by the PDX DevOps community to the <a href="https://nayapdx.org/" target="_blank">NAYA</a> holiday drive</li>
</ul>
<p><a href="https://blog.alicegoldfuss.com/2018-year-in-review/">2018 Year in Review</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on December 27, 2018.</p>
https://blog.alicegoldfuss.com/enabling-perf-in-kubernetes2018-12-21T00:00:00-00:002018-12-21T00:00:00-05:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>Have you been trying to profile your Kubernetes applications with <code class="highlighter-rouge">perf</code>? Maybe you want to see what <a href="http://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java/" target="_blank">all the FlameGraphs fuss is about</a>? If your version of Docker was upgraded within the last year, you’ll likely run into issues.</p>
<!-- more -->
<p>Starting in v17.06 of Docker, <code class="highlighter-rouge">perf_event_open</code> is <a href="https://docs.docker.com/engine/security/seccomp/#significant-syscalls-blocked-by-the-default-profile" target="_blank">blocked by the default seccomp profile</a>. Which means running <code class="highlighter-rouge">perf</code> inside your container will get you this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>perf_event_open(..., PERF_FLAG_FD_CLOEXEC) failed with unexpected error 1 (Operation not permitted)
perf_event_open(..., 0) failed unexpectedly with error 1 (Operation not permitted)
You may not have permission to collect stats.
Consider tweaking /proc/sys/kernel/perf_event_paranoid:
-1 - Not paranoid at all
0 - Disallow raw tracepoint access for unpriv
1 - Disallow cpu events for unpriv
2 - Disallow kernel profiling for unpriv
</code></pre></div></div>
<p>Trying to alter the suggested <code class="highlighter-rouge">/proc/sys/kernel/perf_event_paranoid</code> from within the container gets you the expected:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash: /proc/sys/kernel/perf_event_paranoid: Read-only file system
</code></pre></div></div>
<p>What to do? You’ll need to enable <code class="highlighter-rouge">CAP_SYS_ADMIN</code>. This flag is one of many <a href="http://man7.org/linux/man-pages/man7/capabilities.7.html" target="_blank">Linux capabilities</a>, so named for the extra capabilities they grant. These flags grant scoped permission escalations for threads to perform specific tasks, from changing file attributes to altering the system clock. <code class="highlighter-rouge">CAP_SYS_ADMIN</code> is a particularly overloaded one, a kitchen sink of permissions escalations mostly geared toward profiling work.</p>
<p>If you’re only working with Docker, you can add <code class="highlighter-rouge">--cap-add SYS_ADMIN</code> to your <code class="highlighter-rouge">docker run</code> command, <a href="https://docs.docker.com/engine/reference/run/#additional-groups" target="_blank">as explored here</a>.</p>
<p>However, if you’re living that Kubernetes life, you’ll need to enable it using a <code class="highlighter-rouge">securityContext</code>. In the container spec of your deployment file, add:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>securityContext:
capabilities:
add: ["SYS_ADMIN"]
</code></pre></div></div>
<p>And you’ll be good to go!</p>
<p>Note that you need to strip the <code class="highlighter-rouge">CAP</code> prefix when adding capabilities in Kubernetes. You can read more about <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-capabilities-for-a-container" target="_blank">container privileges in Kubernetes here</a>.</p>
<p><strong>Remember to remove this setting when you’re done using it!</strong> <code class="highlighter-rouge">perf_event_open</code> is blocked by default because it grants user processes privileged access to the system. Branch deploy your change, use it, then rollback.</p>
<p><a href="https://blog.alicegoldfuss.com/enabling-perf-in-kubernetes/">Enabling `perf` in Kubernetes with Docker’s default seccomp profile</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on December 21, 2018.</p>
https://blog.alicegoldfuss.com/foot-candles2018-05-09T00:00:00-00:002018-05-09T00:00:00-04:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>Despite working in tech with an engineering title, my only degree is a BFA in Film and Animation. While some schools emphasize critique in their film programs, mine was more technical and hands-on. That meant I was shooting 16mm film by my second week, organizing set crews by my second quarter, and did everything from directing commercials to writing my own imaging codecs.</p>
<p>It’s also how I learned about foot-candles.</p>
<!-- more -->
<h2 id="set-the-scene">Set the scene</h2>
<p>A <strong>foot-candle</strong> is a unit of measurement describing the <strong>amount of light given off by one candle from one foot away</strong>.</p>
<p>You can measure the amount of foot-candles present in a space with a device called a <strong>light meter</strong> and then do math to convert them into lumens or watts, depending on what you need. In film and photography, you use foot-candles to calculate what f-stop to shoot at aka how wide your aperture should be.</p>
<p>I learned about foot-candles at the beginning of my freshman year. I used my manual light meter all the time, doing manual foot-candle conversions. Eventually we switched to using digital cameras with automatic exposure settings and the need for light meters, let alone foot-candle math, decreased. My light meter was always part of my kit, but the only people actually using theirs on set were the cinematographer and AC (assistant camera). I was an assistant director and didn’t need that knowledge anymore.</p>
<p><img src="/images/lightmeter.JPG" alt="" /></p>
<figcaption>I still have it</figcaption>
<h2 id="roll-action">Roll action</h2>
<p>Let’s say you want to get into photography and you ask for my help. <em>Fantastic!</em> I’m excited to teach you.</p>
<p>I get you started with a camera you already own: your cell phone. I teach you how to take better selfies and how to pay attention to basics like lighting, exposure, and composition. You apply these teachings to taking photos of other people, objects, and landscapes. You’re doing great! You even start an Instagram account because people love your photos so much.</p>
<p>Now you want to go deeper. You buy a digital camera with an automatic zoom lens. I teach you about long exposures, setting up your own lights, gels, and more advanced composition. Your craft is improving by leaps and bounds. You want to turn this into a career.</p>
<p>You’re deep into photography now and want to go whole hog: <strong>film</strong>. You buy an SLR and lenses. We cover film speeds, depth of field, paper types, unusual formats. You start going to workshops. You buy a textbook that covers shooting film, including how to calculate f-stops.</p>
<p>At one point, you look up from your book and say, “I just learned about foot-candles. How cool!”</p>
<p>I reply: “Oh, yeah! <strong>I learned about those in school</strong>.”</p>
<p>Record scratch. End scene.</p>
<h2 id="check-the-gate">Check the gate</h2>
<p>My off-hand comment started a chain reaction in your head. It goes like this:</p>
<ul>
<li>I’ve been doing photography for years and just learned about foot-candles, but Alice already knew about them.</li>
<li>Alice must have done all the things I’ve done as well, years ahead of me.</li>
<li>Alice is even more advanced than I am.</li>
<li>Alice will always be more advanced than I am.</li>
<li>I’m an <em>imposter</em>.</li>
</ul>
<p>In reality, however:</p>
<ul>
<li>I remembered the term but nothing else (I had to look up foot-candles to write this post).</li>
<li>I only learned about foot-candles because my film program was pedantic.</li>
<li>Half my classmates had digital light meters that did the math for them.</li>
<li>I haven’t used foot-candles since graduating.</li>
</ul>
<p>While knowing about foot-candles added some foundational understanding to my film degree, it wasn’t required to make films and the lack of it didn’t prevent you from becoming a photographer in your own right.</p>
<p>Foot-candles are specialized knowledge used by cinematographers and lighting experts. If you want to go into those or similar fields, awesome, you should know about foot-candles. But you didn’t need to know them to get as far as you did.</p>
<h2 id="rinse-your-negatives">Rinse your negatives</h2>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">True story bro...<a href="https://twitter.com/hashtag/programming?src=hash&amp;ref_src=twsrc%5Etfw">#programming</a> <a href="https://t.co/oGMQeF7hgr">pic.twitter.com/oGMQeF7hgr</a></p>&mdash; Paweł Zajączkowski (@gvaireth) <a href="https://twitter.com/gvaireth/status/962245805016010752?ref_src=twsrc%5Etfw">February 10, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Being a self-guided programmer means I go through the same mental cycle again and again. I’m always comparing myself to traditional CS grads. I’m always worried I’m missing something. I’m always worried I’m not enough.</p>
<p>This isn’t helped by some (not all) CS grads holding holier-than-thou attitudes regarding programmers from non-traditional backgrounds. Nothing like having your abilities questioned both internally and all across the entire span of your industry.</p>
<p>However, talking to CS grads over the years, I’ve learned the following:</p>
<ul>
<li>Not all CS degrees cover the same material.</li>
<li>Students can pick and choose their classes to an extent, and therefore graduate with different expertise.</li>
<li>Many CS grads don’t touch certain topics after graduating.</li>
<li>A lot of “real” computer science topics, such as compilers, are good for laying a foundation, but aren’t actually required to be a programmer.</li>
</ul>
<p>Hmm, how similar to my <strong>film degree!</strong></p>
<p>Therefore, it stands to reason there are some CS topics which would qualify as <strong>lighting and composition</strong> and some topics that qualify as <strong>foot-candles</strong>. Such as:</p>
<p><strong>Lighting &amp; Composition</strong></p>
<ul>
<li>Basic data structures (arrays, maps, tuples)</li>
<li>Iteration</li>
<li>OO vs procedural</li>
<li>Writing readable code</li>
<li>Writing tests</li>
</ul>
<p><strong>Foot-candles</strong></p>
<ul>
<li>Assembly</li>
<li>Memory management</li>
<li>Logic gates</li>
<li>Moore’s law</li>
<li>Kernel crashes</li>
</ul>
<p>Some of the foot-candles listed aren’t even covered in many CS programs, and the engineers who know them learned on their own. And you’d be hard-pressed to find a CS grad who still writes Assembly. That’s knowledge that helps with educational foundations but is only used in the professional world by specialists, such as malware researchers. Most traditional software engineers learn it in school and never touch it again.</p>
<h2 id="thats-a-wrap">That’s a wrap</h2>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">&quot;What about their four years of experience in college?&quot;<br /><br />Using Stanford as an example, an undergraduate degree is 180 units. At least 43 of those units must be in CS. So a BS spends about 25% of their 4 years actually doing CS classes. 🤔</p>&mdash; amy nguyen (@amyngyn) <a href="https://twitter.com/amyngyn/status/993702018496610306?ref_src=twsrc%5Etfw">May 8, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>The deeper you dive into programming, the more you will run into topics covered by CS degrees. This may make you feel extremely behind and out of your depth. When this happens, <strong>keep the following in mind:</strong></p>
<ul>
<li>Your lack of knowledge in these topics doesn’t negate the work you’ve already done.</li>
<li>You know things CS grads don’t.</li>
<li>It’s likely your understanding of the topic is fresher and more complete than a CS grad who hasn’t touched it in years.</li>
<li>Everyone learns things in different orders and at different times, including CS grads.</li>
<li>Some things you will never need to know.</li>
</ul>
<p>Also, use your CS grad friends as a resource! One of the biggest hurdles in being a self-guided programmer is not knowing what you don’t know. CS grads can point you to syllabi, books, articles, concepts, and terms that you may never have found on your own. And most of them are happy to teach.</p>
<p>And when you reach a new topic, something new and exciting to dig into, remember that you’ve come this far without it and it just may be a foot-candle.</p>
<p><em>Thanks to Marc Hedlund, Brad Greenlee, Julia Evans, B Byrne, Marco Rogers, Amy Nguyen, and Keeley Hammond for their feedback on this post and Camille Fournier for inspiring it.</em></p>
<p><a href="https://blog.alicegoldfuss.com/foot-candles/">Foot-candles: the different paths to tech</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on May 09, 2018.</p>
https://blog.alicegoldfuss.com/2017-year-in-review2017-02-20T00:00:00-00:002017-12-22T00:00:00-05:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>I have a tradition of writing down my accomplishments from the past year and sticking them on the wall in my favorite depression zone-out corner to comfort me in the winter.</p>
<p>This year, I thought I would share the results. It’s a nice way for people to catch up with my recent shenanigans, as well as a chance for me to categorize and reflect on the parts I liked the most and would like to continue in the new year.</p>
<p>Perhaps adopting this tradition will help you, too.</p>
<!-- more -->
<p><strong>Contents</strong><br />
<a href="#professional">Professional</a><br />
<a href="#community">Community</a><br />
<a href="#personal">Personal</a><br />
<a href="#things">Things I Like</a></p>
<h2 id="professional"><a name="professional"></a>Professional</h2>
<p><img src="/images/LISA-flamegraphs.png" alt="" /></p>
<figcaption>Brendan Gregg discussing my work at LISA17.</figcaption>
<ul>
<li><strong>I got a new job! I left New Relic and joined GitHub as an SRE.</strong>
<ul>
<li>I’d had my eye on GitHub for a while, specifically the team that owns the custom load balancer (written in C). I was excited to start on this very team.</li>
<li>My job hunt included interviewing at Google, one of those career goals most tech people at least think about. I really enjoyed the troubleshooting and architecture interviews, but could do without the literally-no-sleep-the-night-before-constantly-feel-like-vomiting anxiety.</li>
</ul>
</li>
<li><strong>I shipped my first user-facing code!</strong>
<ul>
<li>Being an SRE means my work always has user-impact, but usually in background ways.</li>
<li>At GitHub, I shipped <a href="http://github-debug.com" target="_blank">github-debug.com</a>, a small tool to help users diagnose networking issues.</li>
</ul>
</li>
<li><strong>I wrote my first professional C code!</strong>
<ul>
<li>As part of my increased love for lower-level systems engineering, I started learning C at the end of 2016. Almost a year to that day, I had a PR merged into GitHub’s custom load balancer. Paid to write C!</li>
</ul>
</li>
<li><strong>I took a Data Structures class</strong>
<ul>
<li>I don’t have a formal CS education, and that’s never actually been a problem for me professionally. However, I wanted to take an actual DS course to help my systems engineering work.</li>
<li>I took <a href="https://www.coursera.org/learn/data-structures" target="_blank">this one through Coursera</a> (and passed). The material wasn’t bad, but I found a distinct lack of support in the forums. Also paying $50 to submit homework assignments is ridiculous.</li>
</ul>
</li>
<li><strong>Oh yeah, I kinda innovated some kernel performance work.</strong>
<ul>
<li>My last team at New Relic ran Dockerized Cassandra clusters, and I wanted to dive into performance work on them using FlameGraphs. However, getting FlameGraphs out of containerized processes wasn’t a known path at the time.</li>
<li>So, I figured it out. And wrote up a <a href="http://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java/" target="_blank">blog post</a> detailing how I did it.</li>
<li>I found out at LISA17 that Netflix adopted my work and runs it in production(!!). I’m honestly still shocked.</li>
</ul>
</li>
</ul>
<h2 id="community"><a name="community"></a>Community</h2>
<p><img src="/images/increment.png" alt="" /></p>
<figcaption>My first professional technical publication.</figcaption>
<p>This section was a bit gnarly, so I broke it down even further.</p>
<h3 id="talks">Talks</h3>
<ul>
<li><strong>I only gave two conference talks this year:</strong>
<ul>
<li><a href="https://vimeo.com/221050366" target="_blank">Martyrs on Film: learning to hate the #oncallselfie</a> at Monitorama</li>
<li><a href="https://dockercon.docker.com/watch/ams9ztREHRBU8bHpvZ6fr6" target="_blank">Rock stars, Builders, and Janitors: you’re doing it wrong</a> at Dockercon EU</li>
</ul>
</li>
<li><strong>I had a great time as the lunch and learn guest of a local women tech leaders group</strong></li>
<li><strong>I was a guest on the <a href="https://www.arresteddevops.com/alice-fireside-chat/" target="_blank">Arrested DevOps podcast!</a></strong></li>
</ul>
<h3 id="organizing">Organizing</h3>
<ul>
<li><strong>I once again organized <a href="https://www.devopsdays.org/events/2017-portland/welcome/" target="_blank">DevOpsDays Portland</a>.</strong>
<ul>
<li>It took place during a record-breaking heat wave. Do not recommend,</li>
<li>Last year’s success meant we had a lot more funds this year to funnel back into the conference. It really showed!</li>
</ul>
</li>
<li><strong>I was a talks co-chair for <a href="https://www.usenix.org/conference/lisa17/" target="_blank">LISA17</a>.</strong>
<ul>
<li>I was really honored to be part of such a large and long-running event.</li>
<li>I was also very happy to flex my network and bring in a group of speakers not usually seen at this event. We had some amazing talks.</li>
</ul>
</li>
<li><strong>I continued to organize <a href="https://twitter.com/pdxdevops" target="_blank">PDX DevOps</a>.</strong>
<ul>
<li>I’ve been organizing this group for years, and I think we’ve finally hit our stride.</li>
<li>We have a new meeting space, new meeting night, an official sponsor, and average 50-80 attendees now. I love it!</li>
</ul>
</li>
</ul>
<h3 id="charity">Charity</h3>
<ul>
<li><strong>I found my charity shirts a new home on <a href="https://interruptdesigns.threadless.com/" target="_blank">Threadless</a>, which has generously taken over the actual donations part.</strong></li>
<li><strong>I launched a <a href="https://interruptdesigns.threadless.com/designs/manic-pxe-dream-girl" target="_blank">new shirt</a> this year, with proceeds going to <a href="https://www.freegeek.org/" target="_blank">Free Geek</a>.</strong></li>
<li><strong>I donated $1500 to <a href="http://www.hackthehood.org/" target="_blank">Hack the Hood</a> and organized a Twitter donations challenge that resulted in over $18k donated to the org overall.</strong></li>
</ul>
<h3 id="publications">Publications</h3>
<ul>
<li><strong>I wrote an article for <a href="https://increment.com/development/center-stage-best-practices-for-staging-environments/" target="_blank">Increment Magazine</a> about best practices for staging environments!</strong>
<ul>
<li>I was so touched to be asked for technical content. I’m not used to it and I had a blast writing that article and would love to do it again.</li>
<li>Susan Fowler runs a good operation, just saying.</li>
</ul>
</li>
<li><strong>I’m now cited in the <em>The Practice of Network and System Administration</em>!</strong>
<ul>
<li>It’s just one line about documentation practices, but still.</li>
</ul>
</li>
</ul>
<h3 id="social-media">Social Media</h3>
<ul>
<li><strong>I started the hashtag <a href="https://twitter.com/hashtag/witbragday" target="_blank"><em>#WITBragDay</em></a> in the wake of the so-called Google Manifesto.</strong>
<ul>
<li>This hashtag encouraged women in tech to share their technical accomplishments, shouting back in the face of those who would push us out of the industry.</li>
<li>This event was <a href="https://particle.scitech.org.au/tech/women-in-technology-are-truly-awesome-particle/" target="_blank">featured</a> in <a href="http://www.thehindu.com/sci-tech/technology/witbragday-the-gender-bug-in-the-software-industry/article19485972.ece" target="_blank">multiple</a> <a href="https://www.smartcompany.com.au/startupsmart/news-analysis/ten-of-the-best-tweets-from-witbragday-about-women-in-tech/" target="_blank">publications</a>.</li>
<li>I was so happy to bring women together after a horrid week of once again being told we don’t belong.</li>
</ul>
</li>
<li><strong>I also started the <a href="https://twitter.com/hashtag/unqualifiedfortech" target="_blank"><em>#unqualifiedfortech</em></a> hashtag</strong>
<ul>
<li>This was in response to people critiquing the former CSO of Equifax having a music degree.</li>
<li>This hashtag was also <a href="https://mic.com/articles/184537/unqualifiedfortech-this-viral-hashtag-is-exposing-how-the-tech-industry-rewards-white-privilege#.0Hjr7eJYB" target="_blank">covered</a> by <a href="https://www.edsurge.com/news/2017-09-20-tech-employees-question-credentials-prerequisites-and-privilege-with-unqualifiedfortech" target="_blank">multiple</a> <a href="https://officechai.com/stories/unqualified-for-tech/#sthash.PeADcsJw.dpbs" target="_blank">publications</a>.</li>
</ul>
</li>
</ul>
<h2 id="personal"><a name="personal"></a>Personal</h2>
<p><img src="/images/hair.png" alt="" /></p>
<figcaption>From anime boy to anime boy.</figcaption>
<p>You don’t actually have to read this section.</p>
<ul>
<li><strong>I cut my hair!</strong>
<ul>
<li>I’ve been wanting to do this for a while and took the plunge during a summer break from Twitter. Not being easily spotted in a crowd has definitely improved my public experience.</li>
<li>Highly recommended for decrease in maintenance and stalkers.</li>
</ul>
</li>
<li><strong>I went to Europe!</strong>
<ul>
<li>Thanks in large part to DockerCon EU, I visited mainland Europe for the first time.</li>
<li>Ate many pastries, drank lots of tea, rode many trains, saw the bust of Nefertiti, and visited the Jewish History Museum in Berlin.</li>
</ul>
</li>
<li><strong>I did more baking.</strong>
<ul>
<li>Baking is a soothing activity for when I’m sick of thinking.</li>
<li>This year I baked a cake, new cookies, and successfully made English scones for the first time.</li>
</ul>
</li>
</ul>
<h2 id="things-i-like"><a name="things"></a>Things I Like</h2>
<p><img src="/images/chairemanime.png" alt="" /></p>
<figcaption>ProZD is a delight.</figcaption>
<p>When I do this exercise for myself, I always end with a list of activities that cheer me up (since this whole thing is a way to beat the winter blues). So, here’s a list of things you can try yourself:</p>
<ul>
<li><strong>Tea</strong>
<ul>
<li>I’ve been loving <a href="https://www.amazon.com/Bodum-Assam-17-Ounce-Stainless-Steel-Filter/dp/B000JWFH5M/" target="_blank">this teapot</a> (pours nicely and you can easily reuse the leaves).</li>
<li>I like many teas, but a solid favorite is the <a href="http://www.thejasminepearl.com/English_Breakfast_Black_Tea_p/1013.htm" target="_blank">English Breakfast</a> blend from Jasmine Pearl Tea Co.</li>
</ul>
</li>
<li><strong>Lorde: Melodrama</strong>
<ul>
<li>I’m not usually an artist-focused fan, but I’ve really enjoyed Lorde’s work.</li>
<li>Her new album has a really good sound and lots of replay value.</li>
</ul>
</li>
<li><strong>Brooklyn Nine-Nine</strong>
<ul>
<li>This is my new <em>Parks and Rec</em>. Much like P&amp;R, it takes a while to find its groove.</li>
<li><em>Features:</em> black men in positions of power, a gay black man, multiple latina characters, a not-terrible white male protag, considerate humor.</li>
</ul>
</li>
<li><strong><em>Ancillary Justice</em> by Ann Leckie</strong>
<ul>
<li>You know those books you wish you could climb inside and live in forever? Yeah. That.</li>
<li><em>Features:</em> genderblind society, endless social strategic warfare, mysteries, lots of tea drinking.</li>
</ul>
</li>
<li><strong><em>Dawn</em> by Octavia Butler</strong>
<ul>
<li>Not actually a happy book. Not especially comforting. However, stays with you forever.</li>
<li><em>Features:</em> black woman protagonist, extremely unique aliens, haunting decisions regarding sex and the human race.</li>
</ul>
</li>
<li><strong><a href="https://www.youtube.com/user/lisaeldridgedotcom" target="_blank">Lisa Eldridge</a></strong>
<ul>
<li>Famous makeup artist who makes really nice tutorials and offers insight into the method behind the application.</li>
<li>Very soothing voice. I’ll put her videos on repeat just for her voice.</li>
</ul>
</li>
<li><strong><a href="https://www.youtube.com/user/ProZD" target="_blank">ProZD</a></strong>
<ul>
<li>Voice actor whose work ranges from hilarious short videos to snack reviews.</li>
<li>Another source of soothing videos, especially his food review ones.</li>
</ul>
</li>
</ul>
<h2 id="in-conclusion">In Conclusion</h2>
<p>The professional work I am most proud of this year revolved around deeper systems thinking, whether writing C or tinkering with FlameGraphs. I am also very proud of my <em>Increment</em> article, because it allowed me to showcase my technical thoughts. I would like to do more work like this in 2018 and perhaps only give technical talks at conferences.</p>
<p>I’m happy with my charity work but always want to do more. I’m hoping the more Twitter followers I get, the more money I can funnel into excellent causes.</p>
<p>I’d like to read more books like <em>Ancillary Justice</em> and <em>Dawn</em> and continue to expand my interests outside of tech. And I really need to make a dent in my tea stash.</p>
<p>Happy 2017! May it never rise from the grave.</p>
<p><a href="https://blog.alicegoldfuss.com/2017-year-in-review/">2017 Year in Review</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on December 22, 2017.</p>
https://blog.alicegoldfuss.com/tea-and-tech-culture2017-02-20T00:00:00-00:002017-02-20T00:00:00-05:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>For the majority of my life, I was not a social drinker. Not only did I abstain from alcohol, but also any sort of caffeine not found in a good hot chocolate. I made it through high school, college, and the first years of my professional life without a beer, latte, or late-night crunch energy drink.</p>
<p>Some of it was personal morals, but most of it was personal taste. I would sip friends’ drinks, make a face, and return to my water and cake. I was fine with this, because I was staying true to myself. And the people worth keeping in my life didn’t care. But I knew I was missing out.</p>
<!-- more -->
<h2 id="first-sip">First sip</h2>
<p>In early 2016, I had the fantastic opportunity to speak in London. This was my first time traveling abroad, and my small-town USA heart was overjoyed.</p>
<p>I had traveled between time zones before. I had worked nighttime on-call pages before. Combined, those experiences almost prepared me for West Coast to UK jet lag.</p>
<p>The first morning felt like dying. I couldn’t walk straight, couldn’t think straight, and almost vomited into the spacious marble sink. But I knew I needed breakfast. I knew this was required to adjust. So I dragged myself to the restaurant of London’s Mayfair Hotel wearing jeans, a hoodie, and bright orange slippers.</p>
<p><img src="/images/slippers.JPG" alt="" /></p>
<p>Looking back on that meal, I’m surprised I wasn’t escorted out by posh security guards in tuxedo jackets. I walked in without a reservation (didn’t realize I needed one), past the hostess (didn’t realize there was one), sat at a table of my own choosing (probably meant for someone else), and stared into space. Compared to the elegant travelers around me I looked deranged, but I was trying not to throw up while also figuring out how everyone else got food (a buffet twenty feet behind me). Each second was a decade in my own personal hell, and if I didn’t eat soon I was going to pass out.</p>
<p>At some point one of the waiters must have taken pity on me, because they brought me a pot of tea.</p>
<p>I had tried tea in the past multiple times. Peppermint tea had always smelled so good yet tasted so disappointing. Chamomile helped me sleep on anxious nights, but was a gross brew. A friend had even made me a cup of chai with extra sugar, but even that was too bitter for me. It seemed tea would never be my thing.</p>
<p>But on that morning, I needed sustenance. I still hadn’t figured out where the luscious dishes at neighboring tables were coming from, but I was conscious enough to know that stealing them with my grubby American hands was a no-no. And the pot of tea had come with milk, sugar, and honey. Which was food.</p>
<p>So, I poured a cup of tea, added some milk and honey, and took a sip.</p>
<p>That tea was my <em>salvation</em>.</p>
<p>It was one of the best things I had ever tasted. It was food, it was freedom, it was sanity. I no longer felt like throwing up. I poured another cup, burned my tongue, kept drinking anyway. Another waiter brought me a menu (I must have looked so pathetic) and I ordered Eggs Benedict. I then learned the joy of cutting a hearty egg dish with a good cup of tea.</p>
<p>I lived off tea the rest of the week. I drank several cups with breakfast, again at lunch, and with an afternoon break when possible. Around Day 3 I remembered that tea has caffeine in it and maybe that’s why I still couldn’t sleep at night. I backed off to give my poor virgin system a break.</p>
<p><img src="/images/sohotea.JPG" alt="" /></p>
<p>I brought my obsession home with me. I bought tea, cups, a cart. I had tea with friends and family. It tasted the same as before my London trip, but my newfound Stockholm Syndrome now associated it with comfort, excitement, and not dying in a ritzy overseas restaurant. I was hooked.</p>
<h2 id="gather-round-kids">Gather round, kids</h2>
<p>So, why did I write a blog post about tea? First of all, this is my blog and I’ll do what I want. And don’t even think I got all my tea thoughts out. Not even close.</p>
<p>I wrote about tea because tea doesn’t make me better at my job. At all. I don’t usually drink tea at work, preferring to save it for the weekends when I can savor it. I definitely don’t down a cup every morning to help me computer with confidence.</p>
<p>Yet, drinking tea has made my job much easier. People give it to me, offer to drink it with me, use it as an icebreaker. It’s both a gateway to interesting conversations and a reward for doing favors. I now participate in a social experience that is easy to quantify, repeat, and leverage.</p>
<p><img src="/images/teacart.JPG" alt="" /></p>
<p>Think of how many times social drinking has been a part of your career. Getting coffee before scheduling a formal interview. Grabbing a beer after work. Receiving a nice bottle of Scotch to celebrate meeting a difficult deadline. Social drinking allows us to easily categorize and connect with people through experiences and gifts. And for most of my life I operated without it.</p>
<p>This made everything, from networking to connecting with my coworkers, difficult and awkward. I know I missed out on opportunities because my water and brownie were out of the norm. I was even told at one point that I couldn’t be trusted until I drank alcohol. I didn’t fit in, in the simplest of ways, so I was left out.</p>
<p>You know what other social shortcuts I don’t have? Building a computer with my dad. Learning to solder in Boy Scouts. Traveling abroad as a kid. All common experiences among men in tech. All used to connect at networking events and in interviews.</p>
<p><strong>Here are some experiences that I have, but other tech workers don’t:</strong></p>
<ul>
<li>Watching <em>Star Trek</em>, <em>Star Wars</em>, or <em>Lord of the Rings</em></li>
<li>Reading reddit and Hacker News</li>
<li>Playing video and board games</li>
</ul>
<p>None of these things make someone better at working in tech (let’s face it, reddit and HN make you worse). But they are often used as a litmus test to judge competency. I’ve even heard “ask them if they like &lt;geek hobby&gt;” as an interview tip. If someone has a background similar to yours, you can relate to them easier. Surely this person is smart and nice, because they like the same movies you do, and you are smart and nice. This is <strong>culture fit</strong>.</p>
<p>A lot of people missed out on my skills because I didn’t fit their culture. The same is true for many underrepresented people in tech. Asking those in power to not only look past a difference in appearance but also a difference in experience can be too much to handle. This is why “diversity in tech” so often looks like a white woman who speaks like the typical male nerd.</p>
<p><img src="/images/bletchleytea.JPG" alt="" /></p>
<p>Many tech companies are getting into the groove of doing cultural interviews alongside technical ones. But many, usually startups, interpret “cultural” the wrong way. Instead of evaluating interpersonal communication, they look for people who also enjoy a good game of beer pong. They use social shortcuts, assuming it will get them the best people (aka people like them). And they miss out on awesome talent.</p>
<p>How can you avoid this? In addition to technical skills, interview people based on communication, teamwork, problem solving, ability to ask good questions, and how fast they pick up new concepts. Don’t use hobbies as a litmus test.</p>
<p>Think of it this way: if someone was a smart, kind, and considerate coworker, who enjoyed mentoring and gave great code reviews, would it really matter if they’ve never seen <em>Star Wars</em>? If you can’t imagine something causing interpersonal issues on your team, why interview for it?</p>
<p>Consider how your team interacts together at and around work. Are you reinforcing an environment where only one or two types of people would thrive? Every time your team celebrates at the bar, you’re not only excluding people like me, but former alcoholics and members of some religious faiths. At the same time, don’t make it awkward when someone doesn’t get a pop culture reference or doesn’t want coffee with the morning bagel order. This is your opportunity to make them feel welcome.</p>
<p>Don’t think of it as culture fit. Think of it as <strong>culture add</strong>. We learn when things are new, and things are new when they’re different. Think of how much you can learn from someone of a different ethnicity or socioeconomic class. There are swaths of foods, music, philosophies, traditions, and fashions that you don’t know. Why limit your life to the several shades of your norm?</p>
<p>I mean, unless you want to miss out on someone like me.</p>
<p>*sips tea*</p>
<p><em>For more insight on alcohol as tech currency, please read this <a href="https://modelviewculture.com/pieces/alcohol-and-inclusivity-planning-tech-events-with-non-alcoholic-options" target="_blank">excellent article</a> by the fantastic <a href="https://twitter.com/feynudibranch" target="_blank">Kara Sowles</a>.</em></p>
<p><a href="https://blog.alicegoldfuss.com/tea-and-tech-culture/">Tea and Tech Culture</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on February 20, 2017.</p>
https://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java2017-01-29T00:00:00-00:002017-01-29T00:00:00-05:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>About a month ago, I had the pleasure of taking a tutorial led by the fantastic Brendan Gregg on creating <a href="https://github.com/brendangregg/FlameGraph" target="_blank">FlameGraphs</a> using the Linux <code class="highlighter-rouge">perf</code> toolset. I recommend reading his many <a href="http://www.brendangregg.com/flamegraphs.html" target="_blank">blog posts</a> on the subject, but in short: while <code class="highlighter-rouge">perf</code> is an excellent resource for debugging kernel and user space processes, FlameGraphs make the data even easier to consume.</p>
<p>Now, if the process you’re trying to profile is Java, there are some extra hoops to jump through, which Brendan has <a href="http://techblog.netflix.com/2015/07/java-in-flames.html" target="_blank">also detailed online</a>.</p>
<p>But if the Java process is in a container, it’s even more annoying. That’s where this post comes in.</p>
<!-- more -->
<h2 id="some-context">Some context</h2>
<p>As explained in Brendan’s blog post <a href="http://techblog.netflix.com/2015/07/java-in-flames.html" target="_blank">here</a>, <code class="highlighter-rouge">perf</code> doesn’t work out of the box on Java, because Java doesn’t automatically expose stacks and method names. Running <code class="highlighter-rouge">perf</code> without these gives you something like this:</p>
<p><img src="/images/flamegraph1.png" alt="" /></p>
<p>Notice the nondescript frame dedicated to “java”? Not very helpful.</p>
<p>Running Java with the option <code class="highlighter-rouge">-XX:+PreserveFramePointer</code> (starting in JDK8u60) will expose the stacks. However, without the method name symbols, you get this:</p>
<p><img src="/images/flamegraph2.png" alt="" /></p>
<p>You need to also collect and dump the symbols of the running Java process, so <code class="highlighter-rouge">perf</code> can apply them to the correct stacks. This is made easier by Johannes Rudolph’s <a href="https://github.com/jrudolph/perf-map-agent" target="_blank">perf-map-agent</a> repo. It has some scripts that will dump the Java process symbols and even integrate with the <a href="https://github.com/brendangregg/FlameGraph" target="_blank">FlameGraph</a> repo to make the graphs for you with one command. It’s pretty slick.</p>
<p>Enter containers.</p>
<h2 id="containers">Containers</h2>
<p>Containers, for all their hype and mystery, are still processes on a host. Run a <code class="highlighter-rouge">ps</code> and you can see all container processes running the same as noncontainerized ones.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>ps <span class="nt">-ef</span> | <span class="nb">grep </span>java
103 88834 88800 33 Jan27 ? 10:05:13 /usr/java/default/bin/java</code></pre></figure>
<p>That Java process is running inside a Docker container, and from the point of view of the host, it has PID <code class="highlighter-rouge">88834</code> and UID <code class="highlighter-rouge">103</code>.</p>
<p>Inside the container, that Java process has PID <code class="highlighter-rouge">27</code> and is owned by the <code class="highlighter-rouge">cassandra</code> user.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>ps <span class="nt">-ef</span> | <span class="nb">grep </span>java
cassand+ 27 1 33 Jan27 ? 10:05:20 /usr/java/default/bin/java</code></pre></figure>
<p>Herein lies the issue. Due to a bug in Java, you must dump the process symbols while operating as the owner of the Java process. The <code class="highlighter-rouge">perf-map-agent</code> scripts require it. But the process owner (<code class="highlighter-rouge">cassandra</code>) only exists within the container. Meanwhile, the <code class="highlighter-rouge">perf</code> toolkit must be run as root, and it’s common practice not to allow root within running containers.</p>
<p>So, how can you dump the symbols?</p>
<h2 id="the-hack">The hack</h2>
<p>The hack (“workaround” is too elegant a word) is to run <code class="highlighter-rouge">perf</code> outside on the host, dump the symbols inside the container, and marry the two resulting files in the same space to make a FlameGraph.</p>
<p>More specifically:</p>
<ol>
<li>Setup the FlameGraph repo on your host and the <code class="highlighter-rouge">perf-map-agent</code> repo inside the container where the Java process owner can access it. I also had to alter <code class="highlighter-rouge">/etc/passwd</code> inside the container to give my <code class="highlighter-rouge">cassandra</code> user a shell (use <code class="highlighter-rouge">vipw</code> for safety).</li>
<li>Capture a system profile on the host with something like
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sudo perf record -F 99 -a -g -- sleep 30
</code></pre></div> </div>
</li>
<li>From inside the container (easier to have this running already in another shell) dump the symbols for the Java process with
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> java -cp attach-main.jar:$JAVA_HOME/lib/tools.jar \
</code></pre></div> </div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> net.virtualvoid.perf.AttachOnce PID
</code></pre></div> </div>
</li>
<li>You will now have a <code class="highlighter-rouge">perf-PID.map</code> file inside <code class="highlighter-rouge">/tmp</code> of the container. Move this file to the host (I used a mounted volume).</li>
<li>Now on the host, rename the <code class="highlighter-rouge">perf-PID.map</code> file to match the PID of the Java process <em>as seen by the host</em>. For example, my file was named <code class="highlighter-rouge">perf-27.map</code> but the host has that PID as <code class="highlighter-rouge">88834</code>, so I renamed it to <code class="highlighter-rouge">perf-88834.map</code></li>
<li>Move the re-named <code class="highlighter-rouge">perf-PID.map</code> file to your host’s <code class="highlighter-rouge">/tmp</code> directory and <code class="highlighter-rouge">chown</code> it to <code class="highlighter-rouge">root</code></li>
<li>You can now proceed with the directions as though containers are not involved. So, create a FlameGraph with
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> sudo perf script | stackcollapse-perf.pl | flamegraph.pl \
</code></pre></div> </div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> --color=java --hash &gt; flamegraph.svg
</code></pre></div> </div>
</li>
</ol>
<p>You will need to alter this command depending on where your <code class="highlighter-rouge">perf.data</code> file resides in relation to the FlameGraph repo.</p>
<p><em>Voila!</em> A containerized Java FlameGraph.</p>
<p><img src="/images/flamegraph3.png" alt="" /></p>
<p><strong>Tips:</strong></p>
<ul>
<li>Let Java warm up before profiling it to ensure less churn in symbol creation. I let mine run for 15 minutes.</li>
<li>Run the <code class="highlighter-rouge">perf</code> profile before dumping the symbols. Switching the order might result in empty stacks, because the symbols were created in the JVM after the <code class="highlighter-rouge">perf-PID.map</code> file.</li>
</ul>
<h2 id="why">Why?</h2>
<p>Why is this hack needed? Why can’t we dump the symbols outside the container?</p>
<p>At first glance, it seems easy enough to just create a <code class="highlighter-rouge">cassandra</code> user on the host with UID 103. But trying to dump the Java symbols gives us an error:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="o">[</span>cassandra@hostname]<span class="nv">$ </span>java <span class="nt">-cp</span> attach-main.jar:<span class="nv">$JAVA_HOME</span>/lib/tools.jar net.virtualvoid.perf.AttachOnce 88834
Exception <span class="k">in </span>thread <span class="s2">"main"</span> com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
at sun.tools.attach.LinuxVirtualMachine.&lt;init&gt;<span class="o">(</span>LinuxVirtualMachine.java:106<span class="o">)</span>
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine<span class="o">(</span>LinuxAttachProvider.java:63<span class="o">)</span>
at com.sun.tools.attach.VirtualMachine.attach<span class="o">(</span>VirtualMachine.java:208<span class="o">)</span>
at net.virtualvoid.perf.AttachOnce.loadAgent<span class="o">(</span>AttachOnce.java:37<span class="o">)</span>
at net.virtualvoid.perf.AttachOnce.main<span class="o">(</span>AttachOnce.java:33<span class="o">)</span></code></pre></figure>
<p>This is the same behavior you get if you try to dump the symbols as a user who doesn’t own the Java process. So, the host’s <code class="highlighter-rouge">cassandra</code> user can’t attach to a socket. What kind of socket? JMX or UNIX? Not sure. <a href="https://docs.oracle.com/javase/6/docs/jdk/api/attach/spec/com/sun/tools/attach/VirtualMachine.html#attach(java.lang.String)" target="_blank">The documentation isn’t super clear.</a></p>
<p>Even <code class="highlighter-rouge">nsenter</code> fails here:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="o">[</span>root@hostname]<span class="nv">$ </span>nsenter <span class="nt">-t</span> 88834 <span class="nt">-n</span> java <span class="nt">-cp</span> attach-main.jar:<span class="nv">$JAVA_HOME</span>/lib/tools.jar net.virtualvoid.perf.AttachOnce 88834
Exception <span class="k">in </span>thread <span class="s2">"main"</span> com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
at sun.tools.attach.LinuxVirtualMachine.&lt;init&gt;<span class="o">(</span>LinuxVirtualMachine.java:106<span class="o">)</span>
at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine<span class="o">(</span>LinuxAttachProvider.java:63<span class="o">)</span>
at com.sun.tools.attach.VirtualMachine.attach<span class="o">(</span>VirtualMachine.java:208<span class="o">)</span>
at net.virtualvoid.perf.AttachOnce.loadAgent<span class="o">(</span>AttachOnce.java:37<span class="o">)</span>
at net.virtualvoid.perf.AttachOnce.main<span class="o">(</span>AttachOnce.java:33<span class="o">)</span></code></pre></figure>
<p>Walking the same network namespace as process <code class="highlighter-rouge">88834</code> still doesn’t access the socket.</p>
<p>I talked to several people about this and each conversation ended in puzzlement. Usually I would only post once I had all the answers, but I think it’s good to illustrate that everyone gets stuck sometimes. And it’s better to get the hack out there as a stopgap in the meantime, clunky though it might be. I look forward to a more elegant solution.</p>
<h2 id="special-thanks">Special thanks</h2>
<p>I want to thank Brendan Gregg, Johannes Rudolph, and Nitsan Wakart for creating and maintaining the FlameGraph and perf-map-agent repos, as well as helping me initially troubleshoot. Thank you to Jérôme Petazzoni for his unique container systems knowledge and my colleague Mike Hix for poking at namespaces. I am proud to work with all of you and delighted to occasionally stump you.</p>
<p><a href="https://blog.alicegoldfuss.com/making-flamegraphs-with-containerized-java/">Making FlameGraphs with Containerized Java</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on January 29, 2017.</p>
https://blog.alicegoldfuss.com/ladies-is-gender-neutral2016-09-15T00:00:00-00:002016-09-15T00:00:00-04:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>On Monday evening I launched a <a href="https://teespring.com/ladies-is-gender-neutral" target="_blank">Teespring campaign</a> to raise money for <a href="https://www.gnome.org/outreachy/" target="_blank">Outreachy</a>, a fantastic nonprofit that helps underrepresented people in tech find internships. The shirt in question reads “‘LADIES’ IS GENDER NEUTRAL” and is only offered in women’s sizes. However, I presented the women’s sizes as unisex and replied to requests for men’s sizing with phrases women have heard many times.</p>
<p><img src="/images/ladies_neutral_1.png" alt="" /></p>
<p>This rustled some jimmies.</p>
<p>Regardless, the campaign took off and, as of writing this post, we have raised <em>over $2k</em> for Outreachy. Many people jumped onboard, excited to turn the tables on the many uses of “guys” as a gender-neutral term of address.</p>
<p>However, “ladies” is <strong>not</strong> gender-neutral. Neither is “guys.”</p>
<p><strong>That’s the point.</strong></p>
<!-- more -->
<p>Neither term is correct. Both erase some section of the population, and choosing one over the other isn’t the correct action. Even if you think using “ladies” is transgressive, you’re still erasing some part of the gender spectrum.</p>
<p>When addressing a group of people, no matter how small, you should use a true gender-neutral form of address. For example:</p>
<ul>
<li>Folks</li>
<li>Team</li>
<li>People</li>
<li>Y’all</li>
<li>Friends</li>
<li>Everyone</li>
</ul>
<p>These are all great ways to include people in both personal and professional settings.</p>
<p>So, why didn’t I make a shirt that says “JUST USE ‘FOLKS’” and offer it in every cut? Because, sometimes, the best way to expose privilege is to <em>take it away</em>. Many men expected me to include men’s sizing by request. By telling them no, I gave them a choice: don’t participate in something you enjoy <em>or</em> adapt to the only option given.</p>
<p>This is a choice marginalized people face every day.</p>
<p><img src="/images/ladies_neutral_2.png" alt="" /></p>
<p>Usually I am all for inclusion, but I used this campaign as a way to make a point. And, let’s be clear, all I did was exclude men’s sizing for a shirt. I didn’t deny anyone jobs or access to healthcare. That would be cruel.</p>
<p>I hope this has opened some people’s eyes to what it feels like to be excluded, and how something so simple as a shirt that fits can make an impact. I hope people have a better understanding of how easy it is to be erased from public dialogue. I hope this influences actions going forward.</p>
<p><img src="/images/ladies_neutral_3.png" alt="" /></p>
<p>Something this campaign also helped expose was society’s very limited view on what it means to be a woman. Society expects women to be short and slight, and any deviation from those rules is not supported. Despite offering women’s shirts up to 4XL in size, some women still couldn’t buy them due to women’s sizes being smaller and shorter than men’s. Usually these women have to buy men’s shirts, because they have no other options.</p>
<p><img src="/images/ladies_neutral_4.png" alt="" /></p>
<p>To those women (and nonbinary individuals, and people with gender dysphoria) I accidentally excluded with this campaign, <strong>I am truly sorry</strong>. You have my permission to take the design <a href="/images/ladies_neutral_landscape.png">(found here)</a> and make a shirt for yourself that fits.</p>
<p>My next shirt campaign will have both women’s and men’s sizes, but I want to emphasize that this is <em>bullshit</em>. Labeling clothing this way forces our bodies into a binary that doesn’t exist. I know men who wear women’s cuts and women who prefer men’s and there are many, many nonbinary people who feel comfortable in both or neither. In a perfect world, everything would be individually tailored to everyone’s bodies; however, the next best thing would be to use “fitted” vs “straight” labeling. Unfortunately, this is not yet an option in most shirt resellers.</p>
<p>In closing, to those of you who bought a shirt: <strong>thank you!</strong> You are donating money to an excellent cause. However, please do not use this shirt to end discussions. Let it be the beginning of a new one.</p>
<p><a href="https://blog.alicegoldfuss.com/ladies-is-gender-neutral/">'Ladies' Is Gender Neutral</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on September 15, 2016.</p>
https://blog.alicegoldfuss.com/hacking-pagerduty2016-09-08T00:00:00-00:002016-09-08T00:00:00-04:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>Back in January I decided to toy around with some internal tooling, so our support staff could directly page specific engineering teams. To do this, I dug into PagerDuty’s REST API with the aim of making a simple UI in front of it.</p>
<p>That’s when I noticed something strange.</p>
<p>After a few minutes of staring at the screen, cogs slowly turning, I had my weekend planned.</p>
<!-- more -->
<h2 id="the-setup">The Setup</h2>
<p>Here is an example call from the <a href="https://v2.developer.pagerduty.com/page/events-api-reference#!/Event/post_create_event_json" target="_blank">PagerDuty API docs</a>:</p>
<figure class="highlight"><pre><code class="language-curl" data-lang="curl">curl -X POST --header 'Content-Type: application/json'
--header 'Accept: application/json' -d
'{ "service_key": "e93facc04764012d7bfb002500d5d1a6",
"incident_key": "srv01/HTTP",
"event_type": "trigger",
"description": "FAILURE for production/HTTP on machine srv01.acme.com",
"client": "Sample Monitoring Service",
"client_url": "https://monitoring.service.com",
"details":
{ "ping time": "1500ms",
"load avg": 0.75 },
"contexts": [ ] }'
'https://events.pagerduty.com/generic/2010-04-15/create_event.json'</code></pre></figure>
<p>See that field <code class="highlighter-rouge">service_key</code>? That takes a 32 character hex value, and it’s all you need to trigger an alert for a service. Not just one of your services. Any service.</p>
<p>No account ID, no user API key, no form of 2FA. Just a 32 character string.</p>
<p>Which means, if you can guess the string, you can trigger an alert for any PagerDuty customer.</p>
<p>Very, very unethical.</p>
<p>Also, very fun.</p>
<p>Now I just needed proof.</p>
<h2 id="the-execution">The Execution</h2>
<p>At first I tried to figure out the hashing algorithm. How did PagerDuty derive this 32 character string? Was it a mashup of the account and service ID? Did it have to do with the customer registration date?</p>
<p>I messed around with MD5 and SHA1, combining different seed values, with no such luck. These are known weak hash functions, and I didn’t expect to hit gold. Most likely whatever algorithm PagerDuty used included entropy I had no way of recreating. But it’s always good to try the doorknob before breaking off the hinges.</p>
<p>Now, onto my battering ram.</p>
<p>If I couldn’t reverse-engineer a key, I would need to generate all possibilities. Then, hypothetically, I could send triggers to all of them, and strike true for some percentage of attempts. This is called a <strong>brute force attack</strong>.</p>
<p>I wrote a Python script to iterate through all possibilities in the [a-f][0-9] range and pipe them to a text file. I knew there would be a lot of entries and I would get impatient waiting for the script to complete. However, I figured I would get far enough to get some reasonable matches. I wasn’t going to actually spam PagerDuty’s API, just prove that I could get a positive hit on one of the service keys in my account.</p>
<p>I let the script run for 80 minutes before killing it. The resulting text file was 43G in size.</p>
<p>The last generated entry? <strong>aaaaaaaaaaaaaaaaaaaaaaaafc74a7b1</strong></p>
<h2 id="the-reality">The Reality</h2>
<p>So, I finally decided to do the math. Just how many results was I looking at here?</p>
<p>The 32 character string has values in range [a-f] and [0-9]. That’s 16 possible values for 32 spots.</p>
<p>16<sup>32</sup> = <strong>3.4028237e+38</strong></p>
<p>A big number, sure, but not very meaningful. Let’s look at it this way:</p>
<p>If I generated 100 keys per second for 100,000 years, I would generate only <strong>1.0790283e+24</strong> results.</p>
<p>“But Alice,” you may say, “no key will have twenty a’s in it. You’re wasting a lot of compute time on unnecessary repeats.”</p>
<p>Very astute assessment! If I generated all possible service keys while only allowing each value to repeat three times, I would be generating about <strong>883,413,014,065,530,374,354,632,704,000,000 entries</strong>.</p>
<p>At a rate of 100 per second, that’s <strong>280,128,429,117,684,669,696,421 years</strong>.</p>
<p>Much more reasonable.</p>
<h2 id="the-possibility">The Possibility</h2>
<p>So, can it be done? Obviously my weekend experiment was a failure. But could someone else, someone with more resources, hack PagerDuty?</p>
<p>The <a href="http://calc.opensecurityresearch.com/" target="_blank">Brute Force Calculator</a> is a fun tool that can tell us just that. You give it your password length, keys per second, and charset, and it will tell you how long it takes to brute force the entire keyspace.</p>
<p>I fed it a length of 32 and my custom charset (abcdef1234567890) and chose the fastest calculation rate listed (929803 keys/sec).</p>
<p>The calculator spat out <strong>12 septillion years</strong>.</p>
<p>But, let’s stretch a bit further. In 2012 a <a href="http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/" target="_blank">25-GPU cracking cluster</a> was able to compute every possible 8 character password combination in 5.5 hours. Its 350 billion guesses-per-second speed, applied to our entire keyspace, would take about 30,829,380,202,302,897,683 years. Limiting each character to three repeats per key would take about <strong>80,036,694,033,624 years</strong>.</p>
<p>I’ll wait.</p>
<p>Even if someone did generate all possible service keys, the amount of time and effort needed to send triggers using all of them is practically not worth it. After all, PagerDuty has around 7,000 customers, a microscopic drop in the bucket of possible keys, even if every customer has 50,000 services in their account.</p>
<p>The answer is right in front of you, only it’s the size of a grain of sand, and you’re standing on the beach. Good luck.</p>
<h2 id="the-conclusion">The Conclusion</h2>
<p>So, can you hack PagerDuty? The best way would be to reverse-engineer the keys, if you can. Otherwise, you’re better off randomly unplugging rack cables and paging people that way.</p>
<p>For what it’s worth, I did reach out to PagerDuty in January and inform them of my weekend shenanigans. A security engineer told me that, while they know the 32 character string is secure, they would consider increasing the key length to prevent possible attacks.</p>
<p>Guess my next long weekend is planned.</p>
<p><a href="https://blog.alicegoldfuss.com/hacking-pagerduty/">Hacking PagerDuty</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on September 08, 2016.</p>
https://blog.alicegoldfuss.com/debugging-hardware-panic2016-08-04T00:00:00-00:002016-08-04T00:00:00-04:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>So, a server rebooted. Perhaps I’ve been lucky in the developers I work with, but whenever a host reboots, I always think hardware failure first. I treated this case no differently and jumped onto the host to run some quick diagnostics.</p>
<p>Since I deal in Dell equipment, my first stop is always <code class="highlighter-rouge">check_openmanage</code>. It gives a quick report on CPUs, DIMMs, chassis, power supplies, etc from the command line and takes about 30 seconds to run. It rarely fails me.</p>
<p>However, this time it did. Everything returned <code class="highlighter-rouge">[OK]</code>. No hardware failures.</p>
<p>I double-checked this claim on the iDRAC, just in case this was the one time in history a GUI was more correct than the modules running directly on the box. Nope, everything was green.</p>
<p>So I loaded up the kernel dump.</p>
<!-- more -->
<h2 id="fatal-hardware-error">Fatal Hardware Error!</h2>
<p>There are plenty of guides out there on how to setup and run <code class="highlighter-rouge">crash</code>. If you’re reading this blog post, I assume you already got that far. If not, <a href="http://blog.zedroot.org/linux-kernel-debuging-using-kdump-and-crash/" target="_blank">this guide is pretty good</a>. Of course, if you haven’t configured your system to record dumps in the first place, take this as a lesson learned and do that ASAP.</p>
<p>I ran <code class="highlighter-rouge">crash</code> on the dump and it spat out this:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">KERNEL: /usr/lib/debug/lib/modules/2.6.32-504.16.2.el6.x86_64/vmlinux
<span class="nb">.</span>
<span class="nb">.</span>
<span class="nb">.</span>
RELEASE: 2.6.32-504.16.2.el6.x86_64
VERSION: <span class="c">#1 SMP Wed Apr 22 06:48:29 UTC 2015</span>
MACHINE: x86_64 <span class="o">(</span>2500 Mhz<span class="o">)</span>
MEMORY: 96 GB
PANIC: <span class="s2">"[139839.578542] Kernel panic - not syncing: Fatal hardware error!"</span></code></pre></figure>
<p><code class="highlighter-rouge">“Fatal hardware error!"</code> so something had failed. But my diagnostics didn’t agree.</p>
<p>I spent the next several hours down in the weeds.</p>
<p>Running <code class="highlighter-rouge">bt</code> gives you the stack trace of the active tasks at the time of the panic:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">crash&gt; bt
PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: <span class="s2">"swapper"</span>
<span class="c">#0 [ffff880053a06cc0] machine_kexec at ffffffff8103b5bb</span>
<span class="c">#1 [ffff880053a06d20] crash_kexec at ffffffff810c9942</span>
<span class="c">#2 [ffff880053a06df0] panic at ffffffff81529723</span>
<span class="c">#3 [ffff880053a06e70] ghes_notify_nmi at ffffffff8131c691</span>
<span class="c">#4 [ffff880053a06ea0] notifier_call_chain at ffffffff815304d5</span>
<span class="c">#5 [ffff880053a06ee0] atomic_notifier_call_chain at ffffffff8153053a</span>
<span class="c">#6 [ffff880053a06ef0] notify_die at ffffffff810a4f5e</span>
<span class="c">#7 [ffff880053a06f20] do_nmi at ffffffff8152e1c9</span>
<span class="c">#8 [ffff880053a06f50] nmi at ffffffff8152da60</span>
<span class="o">[</span>exception RIP: intel_idle+177]
RIP: ffffffff812eaae1 RSP: ffffffff81a01e38 RFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffffffff81a01fd8 RDI: ffffffff81a90500
RBP: ffffffff81a01ea8 R8: 0000000000000002 R9: 000000000000009c
R10: 00007f0efb0ee285 R11: 0000000000000000 R12: 0000000000000000
R13: 145631f3e34bac61 R14: 0000000000000001 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
<span class="nt">---</span> &lt;NMI exception stack&gt; <span class="nt">---</span>
<span class="c">#9 [ffffffff81a01e38] intel_idle at ffffffff812eaae1</span>
<span class="c">#10 [ffffffff81a01eb0] cpuidle_idle_call at ffffffff81426117</span>
<span class="c">#11 [ffffffff81a01ed0] cpu_idle at ffffffff81009fc6</span></code></pre></figure>
<p>In this case, the failure happened in driver method <code class="highlighter-rouge">intel_idle</code> at offset 177. This is usually where you decompile the kernel code (or throw the server away and go back to sleep) but I found the code online instead. The <code class="highlighter-rouge">intel_idle</code> driver forces CPUs into the lowest activity level when not in use instead of keeping the core warm. It’s meant to boost energy efficiency on your hardware, but I don’t care for the performance hit. In this case it looked like the method had failed to store CPU state for some reason, causing the panic. However, the server was running now and this seemed to be a one-off machine goblin occurrence.</p>
<p>Except a few hours later, when it crashed again.</p>
<p>And again.</p>
<p>And again.</p>
<p><code class="highlighter-rouge">“Fatal hardware error!"</code> each time. Yet everything was coming up <code class="highlighter-rouge">[OK]</code>. What was failing? Was it the <code class="highlighter-rouge">intel_idle</code> driver? It ran on other hosts without issue. Why was it breaking here?</p>
<p><strong>Protip:</strong> trust the kernel.</p>
<h2 id="the-esm-log">The ESM Log</h2>
<p>So, let’s cut to the chase. What was the issue and how did I find it?</p>
<p>The secret here is the ESM log. ESM stands for Embedded System Management and is sometimes known as the System Event Log. It logs every hardware-level event, even harmless ones, and it showed a critical failure on this host.</p>
<p>How did I find this? I didn’t. It turns out, an engineer on a completely different team had seen <code class="highlighter-rouge">esmlog</code> show up in <code class="highlighter-rouge">dmesg</code> one day and didn’t know what it was. And as he was learning about it, he just so happened to pick my troubled host to investigate.</p>
<p>Curiosity is awesome.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>omreport system esmlog
Severity : Critical
Date and Time : <span class="nt">--</span> <span class="nt">--</span> <span class="nt">--</span> <span class="nt">--</span>:--:-- <span class="nt">----</span>
Description : A bus fatal error was detected on a component at bus 0 device 3 <span class="k">function </span>0.</code></pre></figure>
<p><code class="highlighter-rouge">bus 0 device 3 function 0</code> refers to a location on the PCI. You can dig into this further with <code class="highlighter-rouge">lspci</code>.</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>lspci <span class="nt">-s</span> 00:03.0
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a <span class="o">(</span>rev 04<span class="o">)</span></code></pre></figure>
<p>So the failure was on the PCI itself. The <code class="highlighter-rouge">intel_idle</code> failure made more sense now: the processes were unable to share state across the bus.</p>
<p>You can also run a super detailed command on this device location:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nv">$ </span>lspci <span class="nt">-vvxs</span> 00:03:0</code></pre></figure>
<p>Which gave me, among many other things, this:</p>
<figure class="highlight"><pre><code class="language-shell" data-lang="shell">03:00.0 PCI bridge: Renesas Technology Corp. SH7757 PCIe Switch <span class="o">[</span>PS] <span class="o">(</span>prog-if 00 <span class="o">[</span>Normal decode]<span class="o">)</span></code></pre></figure>
<p>I put in a request for a new SH7757 PCIe Switch and that was that.</p>
<h2 id="futureproofing">Futureproofing</h2>
<p>So, I debugged the kernel panic. But, there were gaps in my diagnostics that had me looking in the wrong places for hours. How do I prevent that from happening again?</p>
<p>You can get ESM log data from the <code class="highlighter-rouge">check_openmanage</code> command, but it isn’t included by default because the log itself is so noisy. However, I wanted visibility into the log, so I wrote a custom <code class="highlighter-rouge">nrpe</code> check to poll the ESM log and port it into Nagios. It doesn’t give details to the failures themselves, just if there are <code class="highlighter-rouge">CRITICAL</code> events on a host. It also doesn’t page people; if the failure is bad enough that the host reboots, we’ll know anyway.</p>
<p>So, why have the check if you’re not going to page on it? The developers I work with don’t have root access or deep systems knowledge, but they do have Nagios logins. And I would much prefer getting a page that says, “A server rebooted and I see a <code class="highlighter-rouge">CRITICAL</code> ESM log message in Nagios” vs “A server rebooted and lol idk.” Proper alerting is an art.</p>
<p>We haven’t needed the ESM log monitoring yet (fingers crossed!) but I’m glad the developers have access to it anyway. My future wishlist includes writing automation to clear the ESM log after a period of time, so only fresh events appear. Otherwise you need to manually clear the log after an incident.</p>
<p>I hope this post has been helpful. Now go back to bed.</p>
<p><a href="https://blog.alicegoldfuss.com/debugging-hardware-panic/">Debugging a Hardware Panic</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on August 04, 2016.</p>
https://blog.alicegoldfuss.com/slapping-back-imposter-syndrome2016-06-14T00:00:00-00:002016-06-14T00:00:00-04:00Alice Goldfusshttps://blog.alicegoldfuss.comyou@email.com
<p>Imposter Syndrome used to rule my life. After all, I was an engineer (hah!) with a film degree, and everyone I worked with was much more intelligent and experienced than I’d ever be. I would never get to their level, everything I said was stupid, and if I ever received a promotion it was due to diversity quotas or a one-time fluke.</p>
<p>My fixation was so bad that it was brought up in my yearly performance review. I was told by my manager to get it under control, or she would have to write it down in my file.</p>
<p>So I worked on it.</p>
<p>And I got <em>better</em>.</p>
<p>This isn’t a magical antidote that will make you good as new. My Imposter Syndrome is like Lyme disease; it never fully goes away. But I have gone from bathroom panic attacks and Sunday night tears to a more balanced view of myself and others. I still have bad days, sure, but the depths are not as deep.</p>
<p>I hope it helps you as well.</p>
<!-- more -->
<h2 id="filing-down-the-pedestals">Filing down the pedestals</h2>
<p>Imposter Syndrome distorts how you see yourself as compared to others. Everyone else is on a pedestal marked Knows Everything and you’re in a dark pit that doesn’t even deserve a sign.</p>
<p>So, the first step is take those people off their pedestals.</p>
<p><strong>Please note:</strong> this advice sounds mean because you are focusing on less-than-flattering things in others. However, we are not trying to put others down. We are simply trying to see them as human.</p>
<p>Whenever you’re around people you admire, <strong>pay attention</strong> to when they say the following phrases:</p>
<ol>
<li>“I don’t know.”</li>
<li>“Let me check the docs.”</li>
<li>“Let me look that up.”</li>
<li>“Oops.”</li>
<li>“I asked so-and-so and they think ___”</li>
<li>“I tried this but it didn’t work.”</li>
<li>“Can you take a look?”</li>
<li>“I need to rollback.”</li>
<li>“I’m sorry.”</li>
<li>“I broke prod.”</li>
</ol>
<p>They will say them. They say them right now, but you don’t notice. They say them in standups, in meetings, in post-mortems. They say them to coworkers and managers. They say them to you. And each of these phrases is them admitting they aren’t perfect.</p>
<p>I had to train myself to listen for these phrases. I found that the engineers I respected the most say them all the time. If you’re an experienced engineer, you’re probably working on complex systems or trying something new. That means you’re more likely to make mistakes. Good engineers admit mistakes and move on.</p>
<p>Noting when my heroes admitted fault didn’t make me respect them any less; it just made it easier to talk to them. It was much easier to learn from someone when I wasn’t terrified they would see me for the fake that I am.</p>
<p>So, keep track of your coworkers’ mistakes. That time they broke the build, or didn’t test thoroughly enough, or had to make major revisions to a PR. They might know more than you in some areas, but they don’t know everything.</p>
<h2 id="climbing-out-of-the-pit">Climbing out of the pit</h2>
<p>This part makes the assumption that you are a being impacted by a linear expression of time. That is to say, you move forward and change.</p>
<p>Climbing out of the pit involves recognizing progress within yourself.</p>
<p>Pay attention to when you do the following:</p>
<ol>
<li><strong>Skip a section in the docs</strong> or an online forum answer. You don’t need to read how to install a gem, you know that already. Wait, did you always know how to install a gem? No, you learned that.</li>
<li><strong>Pass on a meeting or meetup</strong> because you already know the material. You know how to setup your Python environment. You know how to use Docker. Even if it’s an internal project brief, if you already know it, then you are ahead of the people RSVPing to that meeting.</li>
<li><strong>Teach someone something.</strong> This doesn’t have to be a big formal talk or mentor relationship. Teaching the person next to you how to use the VPN, where to find that block of code, or even how to create a new color scheme for their IDE all count. You are imparting knowledge that this person does not have. You are the expert in this exchange.</li>
</ol>
<p>In addition to the above passive observations, here are some more active things to do:</p>
<ol>
<li><strong>Keep a logbook</strong> of your projects and accomplishments. When you’re working on a project, record your contributions in this logbook with a date. Include things like technical books you’ve read, workshops you’ve attended, talks you’ve seen. They all indicate growth.</li>
<li><strong>Go look at job openings</strong> at other companies and note how your skillset compares. You might not have qualified a year ago, but you are probably closer to qualifying now. Understand where your value lies in the industry.</li>
<li><strong>Look at the resume</strong> you used to get your current job and update it to reflect what you know now.</li>
</ol>
<p>Basically, compare yourself today to yourself 3 years ago. Is Past You intimidated by Present You? Good. You’re doing it right.</p>
<p>Recognizing you’ve grown is beneficial in many ways. It helps you understand the gap between yourself and those you admire, because they most likely had a head start. It helps you realize opportunities where you can help people who don’t know as much as you do. And it helps you prepare for performance reviews and promotion cycles (which you deserve and should pursue).</p>
<p>You know more today than you did yesterday and a ton more than you did a year ago. It takes a while to notice you’ve improved, but I promise you have.</p>
<h2 id="finally">Finally</h2>
<p>I’ve been using these techniques for over a year. They haven’t made me cocky or eliminated all doubt. But I do feel like more of an equal to my coworkers, and I spend less time burning the midnight oil trying to catch up on the sum of all technical knowledge before people catch on.</p>
<p>So, give them a try. Adjust the imbalance between your perception and reality.</p>
<p>You’re right where you need to be.</p>
<p><a href="https://blog.alicegoldfuss.com/slapping-back-imposter-syndrome/">Slapping Back Imposter Syndrome</a> was originally published by Alice Goldfuss at <a href="https://blog.alicegoldfuss.com">Alice Goldfuss</a> on June 14, 2016.</p>