Category Archives: Red Hat

I get quite a bit of email notification from Pagure, the free (as in freedom!), Python-based git forge. I have and participate in several repos there. Pagure sends me email when there’s a new or updated issue, pull request, or commit to a repo I monitor.

Previously, all that email for me has gone into a “Pagure” folder. But Pagure offers a special header, X-pagure-project, which indicates the specific project that triggered the email. Unfortunately, Gmail’s filters notoriously lack the ability to parse arbitrary headers. This is actually a limitation of the Gmail API. Why there’s no getHeaders() call, I have no idea — but maybe it has something to do with worries about arbitrary content there.

Now, I could probably set up filters to look for mail From: pagure@pagure.io and then look in the subject line for [Project-name]. But this means I have to make a new filter for every project manually. YUCK!

Enter Google Apps Script

So I wrote my very first Google Apps Script (this script is MIT licensed).

Along with several people on the Fedora Engineering team, I recently attended the DevConf.cz 2017 event. The conference has grown into an amazingly successful gathering of open source developers. Most attendees live in Europe but there were some from every continent. The coverage spanned all the big open source buzz-generating technologies. Session topics included containers, PaaS, orchestration and automation, and DevOps.

One of the most frequent topics I heard at and around the conference was CI — continuous integration. The idea of CI isn’t new in and of itself. Many open source projects put it to use already. If you do any work on Github, you’ve already seen it at work. (Also notably, OpenStack features one of the biggest and arguably most successful at-scale efforts.) However, it hasn’t taken strong root in Fedora, yet.

Based on the discussions I had and heard at DevConf.cz, that’s likely to change soon. Why? Well, for one thing, Fedora still breaks too often. This is especially (but not exclusively) true about Rawhide. But in many cases, we can easily understand these breakages. That means we can detect and prevent them before they occur. While update testing in Bodhi helps in the case of stable releases, that’s not a cure-all. That process still depends heavily on manual efforts that aren’t guaranteed. Moreover, those efforts form a gate of several days at a minimum, between building, karma, and tagging. This process no longer scales with new, fast-moving deliverables like Fedora’s edition of the Atomic Host. One size no longer fits all. We need a more flexible, automated approach.

What does that mean? For Fedora, the CI concept means gating on automated tests that guarantee some level of validity to a change. These tests are run prior to introducing changes into a tree, ostree, module, container, etc. That way, we don’t pass breakage on to the consumer. But we can message that breakage back to the maintainer directly. The maintainer then quickly acts to fix the issues. And they see more immediate results from their fix.

We must do some work in the Fedora infrastructure to make this process work for Fedora contributors (maintainers especially). Some app work is also required to support our packaging and deliverables. We must make sure it’s easy for people to contribute to tests. Maintainers need to easily understand the status of their build requests. And of course the community must be able to continue contributing to higher levels of testing that make for a better Fedora. So CI isn’t the end of the story. It’s just another strong link in a chain of improvements.

Fortunately, we’re well situated to do great work here. For instance, the Fedora Infrastructure team already plans to front our dist-git package control with Pagure. (You probably know Pagure is a full-featured collaborative git forge.) This ensures the Fedora dist-git acts as a source for CI, tested with each new proposed change. But we believe this is unlikely to cause more complexity. Packagers can expect to use tools they already know. Furthermore, they could use the popular, effective pull request model to maintain packages and develop testing. This lowers the bar to contribution while maintaining high quality.

Another point in our favor is the entire Fedora application infrastructure has been running on the fedmsg messaging bus for years. We already can orchestrate and coordinate a huge number of automated activities around events involving our apps. Therefore a higher level of continuous integration and testing is well within our grasp.

Of course, this blog post is light on details. (What else would you expect from a manager?) 😛 Those details are obviously important. As the least technical person in the Fedora Engineering team, though, I’m the last person you’d want to describe them in detail. But the folks in the Infrastructure community team will be launching a series of discussions on the mailing list to explore those details.

We’re all passionate about making Fedora better. So hearing feedback from our contributors and colleagues at DevConf.cz has us very excited and enthusiastic about this next level. I encourage you to get involved and contribute constructively to the discussions!

It’s sad I don’t get more time to post here these days. Being a manager is a pretty busy job, although I have no complaints! It’s enjoyable, and fortunately I have one of the best teams imaginable to work with, the Fedora Engineering team.

Since we’re coming to the close of another calendar year, I wanted to take a moment to remind people about what the holidays mean to availability. I’m going to crib from an earlier post of mine:

My good friend John Poelstra is fond of saying, “It’s OK to disappoint people, but it’s not OK to surprise them.” That wisdom is a big reason why I like to set expectations over the holidays.

Working at Red Hat is a fast paced and demanding job. Working full time in Fedora is itself demanding on top of that. These demands can make downtime at the holiday important for our team. At Red Hat, there’s a general company shutdown between Christmas and the New Year. This lets the whole organization relax and step away from the keyboard without guilt or fear.

Of course, vital functions are always staffed. Red Hat’s customers will always find us there to support them. Similarly, our Fedora infrastructure team will monitor over the holidays to ensure our services are working nominally, and jump in to fix them if not.

Some people like to spend time over the holidays hacking on projects. Others prefer to spend the time with family and friends. I’ve encouraged our team to use the Fedora vacation calendar to mark their expected “down time.” I encourage other community members to use the calendar, too, especially if they carry some expectations or regular responsibilities around the project.

So all this to say, don’t be surprised if it’s harder to reach some people over the holidays. I’m personally taking several weeks around this holiday shutdown as time off, to relax with my family and recharge for what’s sure to be another busy year. Whatever your plans, I hope the holiday season is full of joy and peace for you and yours.

I’m thrilled to announce that Jeremy Cline has joined the Fedora Engineering team, effective today. Like our other recent immigrant, Randy Barlow, Jeremy was previously a member of Red Hat’s Pulp team. (This is mostly coincidental — the Pulp team’s a great place to work, and people there don’t just move to Fedora automatically.) Jeremy is passionate about and has a long history of open source contribution. We had many excellent applicants for our job opening, and weren’t even able to interview every qualified candidate before we had to make a decision. I’m very pleased with the choice, and I hope the Fedora community joins me in welcoming Jeremy!

I’m extremely happy to announce that Randy Barlow has joined the Fedora Engineering team, effective today. Randy was previously a member of Red Hat’s team working on technologies like Pulp. You can find his fingerprints in many upstream repositories as a frequent contributor. This is fortunate for our team, since Randy will be contributing both to our applications infrastructure and to our release team. His experience with Pulp may come in handy too, since it may play a part in making next-generation Fedora content available to the public. I hope the Fedora community will join me in welcoming Randy!

My day job, as you probably know, is at Red Hat, where I manage the Fedora Engineering team. Our team provides engineering and design services for the Fedora Project, a collaborative community project which is the upstream source for a number of influential products. Not the least of these is Red Hat Enterprise Linux, of course.

We now have a job opening for a senior engineer in the USA to be part of our team.

About the job

Collaborating with the rest of the team, the community at large, and your colleagues in Red Hat, you’ll:

Build tools and web services that will in turn build a more modular next-generation Fedora

Do full-stack development, with a hand in everything from working with designers, to architecting and writing code, to deploying in test/stage/production, to maintenance and enhancement

Automate, automate, automate all the things

Co-create and develop our next community engagement system, Fedora Hubs

Stay tuned into exciting work going on throughout Red Hat related to platform and other technologies

As with all Fedora Engineering jobs, communicate openly and continually with the whole community, and build community around everything you do using open source best practices

We do that work openly: collaboration via git repositories, rapid and constant communication via IRC, frequent discussion through our mailing lists, and gathering and building community around our work. Simply put, we love open.

Who we’re looking for

The right candidate is a team player, positive and constructive, fully engaged and passionately committed to delivering results with their colleagues, wherever they might be. This is a senior position, so we’re looking for someone who’s a proactive, detail-oriented self-starter, and has the ability to lead from the side on a few projects. We want you to be really good at Python and being Pythonic, and hopefully have some similar high-level language experience like Ruby or Java where you can help us grow, too.

Not interested in working in a Red Hat office? No sweat. We’re a distributed team, and this job is perfect for an experienced remote candidate. Remote doesn’t mean aloof, though. You need to be somewhat of a people person to crush this job, because good open source means collaboration. We meet up at least once a year as a team for Flock, but we work together constantly online.

I would be remiss in not pointing out this position back-fills some mighty big shoes. That’s why I’ve outlined some ideas for who we’re looking for in this post. That being said, you’ll also be working for a manager who’s a super guy. Just check the sidebar of this blog for proof. In all seriousness though, we all care about our teammates’ success, so you’re never on your own when you need a hand.

And don’t forget, Red Hat is consistently rated one of the best and most innovative places to work.

Enough already, where do I apply?

I realized that some folks around the Fedora community may wonder why they don’t see me around as often this week and next week. I’m still alive and well, but I’m traveling in the Czech Republic. I’m currently in Brno for some Red Hat departmental meetings. Following that, I’ll be attending the Devconf.cz event. Then I’ll be back in the Brno office for a few days of other work and team meetings. I’ll be back in my home office on Monday, February 15th and around as usual at that point.

There are three USA internship positions open at Red Hat on the Fedora Engineering team. These internships are all available this coming summer (2016). Read below for some more details.

I feel like my recent blog posts have all been about openings. But it’s nice to be able to offer them. Each internship on our job site is linked in the bullets below, so you can apply for any for which you feel qualified.

Software engineering internship – This internship focuses primarily on Python software development. You’ll take designs and build them into widgets for our Fedora Hubs project. You’ll collaborate with our applications team on back-end technology for the community. You’ll also work with our infrastructure team on production deployment. This internship also includes other related projects, not limited to just Hubs. The intern is preferably in Westford, MA for the summer. But that’s not strictly required for a well-qualified candidate with experience working in a global open source project. (In other words, if you can succeed at keeping touch daily with the team as a remotee, don’t let the location throw you.)

User experience and design internship – This intern will work directly with our designers Máirín Duffy and Ryan Lerch on research and design for Fedora projects such as Hubs. You’ll do user research and work on improving interaction in our web apps and other associated projects. You’ll collaborate with the other interns, our whole development team, and the Fedora community at large. This job is in Westford, MA.

Web development internship – This intern will work primarily (but not exclusively) on web apps at the front end. You’ll turn design into reality, and streamline existing applications to designer specs to build a more unified project. As with our other internships, you’ll be able to explore and work on other projects too. You’ll work with our design team, the community Websites team and our application developers to create and improve web tools our contributors will use for years. This job is preferably in Westford, MA for the summer. But we’ll consider remote for an exceptionally qualified candidate.

These jobs should end up being quite competitive. So I encourage you to get your application in as soon as you can!

Last year, we were fortunate to have superb interns who produced fantastic results. Meghan Richardson worked on many of the initial designs for Hubs in its early stages, interviewing potential users and then creating the workflows and feel for this new tool. Nate Yazdani worked on statscache, which compiles data from our messaging bus to make it available quickly to other apps, without constantly crushing our data store.

Red Hat is a fantastic company at which to intern, and we pride ourselves on an open and inclusive culture. These are not “fetch coffee” jobs — you’ll work on real problems alongside real experts.

Are you familiar with Outreachy? It’s a program designed to boost participation in open source by underrepresented groups. My employer, Red Hat, is sponsoring this internship. The Fedora Project, which my team at Red Hat supports, is proud to partner in offering it. This Fedora internship is an opportunity to work directly with our senior software engineers on a major project called Fedora Hubs.

Hubs is an initiative to bring better and more timely information to Fedora’s large community of participants and contributors. Several of our engineers will collaborate with you throughout the internship period on this project. We’ll use Hubs as a foundation for regular, daily mentoring of our Outreachy intern. You’ll use best of breed technologies like Python, Bootstrap, jinja templates, and more. Best of all, you’ll work alongside the team in real time.

You can read more about the internship and qualifications here on our wiki. The page also has general information about Fedora, such as our mission and how we work together.

But this internship isn’t just extending your skills, or working with awesome people. (Even though that’s a great incentive!) It’s also about bringing the open source way to thousands of people. This project lets contributors build sub-communities, or hubs, around their interests. “Standing on the shoulders of giants” is one of the greatest benefits of open source. The same goes for using open source to build communities. It’s a natural fit and a wonderful outcome for a successful internship.

Are you interested in diving deeper into Hubs, to see what this internship is about? Check out this recent blog post from our own Máirín Duffy on the purpose, design, and mockups of the Hubs project.

Flock had a team of organizers within OSAS (and Josh also assisted throughout). As a former FUDCon organizer, though, I know the value of extra hands showing up to do work. Since old habits die hard, I showed up expecting to help out behind the scenes. That means I didn’t get to see a huge amount of content I was personally interested in. But in return hopefully everyone had a smoother Flock experience, especially speakers.

When I arrived, I reported to Tom Callaway, Ruth Suehle, and Josh. They got the conference rooms opened, and I helped set up the speaker workstations. We worked pretty late, well after midnight. Things were looking a little bleak at that point, with execrable network bandwidth, no projectors, no screens, and no audio for the ballroom.

Fears and worries abate

Nevertheless, the next morning Josh and I got up early and grabbed coffee at nearby Tedward’s. This place was a godsend, although their 7:00am opening time forced us to walk around a bit until we could get in.

We went down to do some additional setup. The organizers had worked with Remy DeCausemaker to get a bunch of loaner projectors from RIT so we’d be ready for the first sessions at 10:00am. (EDIT: According to Remy, Tim Duffy and Dan Schneiderman are the heroes of this particular day; see comments below.) So at least our speakers would be in OK shape. I helped Josh and Tom get everything ready in those rooms, while Ruth made sure registration and other logistics were under control. I missed Matthew Miller’s keynote, but I’d seen at least some of the material previously.

After lunchtime, things continued to drastically improve. The rental projectors showed up, along with small screens for each room and big speakers for the ballroom. The wireless internet improved quite a bit when a switch flip occurred due to our conference starting up. (It was dismal Tuesday night!) We had all the speakers trained on how to record their talks locally, to get around the constrained network bandwidth.

Suddenly things were looking up! Not surprisingly, the Fedora Engineering team dinner that night at The Old Toad was much more enjoyable. Since I wasn’t overly worried about the conference experience for the speakers and attendees any longer, it was easier to relax and enjoy the company of the team. I was so happy that we were able to get together in one place, since we really only get to do that once a year. (Incidentally, our friend Stephen Smoogen was absent from Flock due to family commitments — we missed you, Smooge!)

Fedora contributors at Flock gather at Victoire for dinner

I continued to monitor speaker rooms most of Wednesday and Thursday. I managed to make it to a couple sessions where I wasn’t sure there would be any senior Fedora leadership around. For example, I attended the Fedora Magazine session by Chris Roberts as well as most of the Fedora Hubs session by Máirín Duffy and Meghan Richardson.

I attended and loved Major Hayden‘s (of Rackspace fame) Thursday keynote on fighting impostor syndrome. It was one of the most practical that I’ve seen on this topic. I feel impostor syndrome is just a fancy way to refer to insecurity, a common trait for conscientious people. But that doesn’t make the strategies Major outlined any less useful or thoughtful. He gave a great talk — engaging and humorous without diluting the material. If you have a chance to invite him to a conference to speak, definitely do so!

I gave my own talk on Remote Ninjutsu on Thursday afternoon. The slides for the talk are here, although the video will be more useful for context. All the Flock 2015 videos are supposed to be available at some point in the next couple of weeks. Stay tuned for announcements about them.

The Thursday night social event at the Strong National Museum of Play was fantastic. It was a great way to blow off steam and enjoy the company of fellow Fedorans. I’m not sure how the organizers managed to find such a perfect venue!

Workshops and Flock wrap-up

On Friday I enjoyed the keynote by Jon Schull of eNable, the community that is flipping the script on prosthetics provision through 3D printing. It was a very moving look at how people are applying open source to make the world better for people in need.

Then the workshops beckoned. Now that I’d finished my Flock duties helping speakers and attendees, I was able to attend several sessions that were relevant to me personally, including the Fedora/CentOS rel-eng joint session, and my own on revamping the Flock software stack.

Once again, the Friday night social event at the George Eastman House was marvelous. It was a beautiful, grand mansion and the tour was quite interesting. I’d love to go back there sometime to see the exhibits I missed!

The music parlor in the George Eastman House

Flock conferences are always especially great for their hallway track. So many discussions can be had or progressed that way with high bandwidth. The challenge is always to move that discussion to a transparent context if it involves people not present, though. I’ve been seeing many trip reports from people’s blogs about Flock, and resulting list discussions, so I think that process is well underway.

Of course, that means Flock is a very engaging event. It takes a lot of attention and brainpower to shift focus for all those conversations! As a result, by Saturday afternoon I know I was fairly exhausted — in a good way, though. Several other people I know felt likewise, and commented on how well the conference had gone. In fact, I heard a number of comments that this was the best Flock, and even Fedora premier event, yet. The OSAS folks deserve special recognition for pulling off a fantastic conference.

Sunday started with a couple meetings, including with Matthew Miller and Jan Ku?ík, our new Fedora program manager. Then, after seeing a few other friends and colleagues off, I got to the airport. I relaxed in a lounge over beers with Kevin Fenzi, Jan Zeleny, and Stephen Tweedie, before we went to our respective flights. Then after a quick flight home, it was the usual “fun” making my way down I-95 from the airport to home. Monday morning was right around the corner…

Here’s to another great Flock, and to doing it again next year!

Flock attendees wind down after the conference ends… with more hacking!