August 29, 2016

Kevin Fenzi : two vpns and NetworkManager? no longer a problem…

NetworkManager has pretty handy vpn handling for laptops. You can setup all different kinds, it can prompt you for passphrases, it can set up specific vpns on specific networks, etc.

However, if you had more than 1 vpn you wanted to run at a time you had to pick one for NetworkManager to handle and do the other(s) outside NetworkManager. This is happily no longer the case (at least with NetworkManager 1.4.0): It can bring up as many vpns as you like and manage them all.

This weekend I took my home openvpn connection (which I had setup as a system wide openvpn service) and moved it into a NetworkManager vpn profile. It works just fine with my work vpn and now NetworkManager can bring it up automatically after it connects to the network.

So, if you need two (or more) vpns on your laptop, you can now happily move them all into NetworkManager. Many thanks to the hard working NetworkManager folks. ūüôā

July 25, 2016

Kevin Fenzi : Looking forward to flock 2016

Just over one week until flock ( ), Fedora’s main yearly conference. This time it’s in Krak√≥w, Poland. This of course means a long time traveling for myself and other North American Fedorans, but it’s always well worth it.

In addition to seeing old and new friends, I’m looking forward to quite a lot of the talks (see for the full schedule) along with workshops and hallway discussions.

I’m going to be giving a talk on using Fedora Rawhide as your daily driver OS. Hopefully lots of tips and workflows that people will find useful. Also, I am co-presenting with Pierre-Yves Chibon (pingou) on the current state of Fedora Infrastructure along with lots of upcoming plans.

If you aren’t able to make it to Krak√≥w in person, do follow along on #fedora-flock ( on freenode ) and in blogs and other project communications.

June 27, 2016

Kevin Fenzi : Zodbot… upgraded

Kneel before the new Zodbot!

We have upgraded our beloved evil super villain IRC bot on freenode from an old version of supybot-gribble to a new shiny version of limnoria (¬† ).¬† This doesn’t change much in the interface, but it does mean we are using something that is maintained and gets updates and is a good deal more secure. If you notice problems please do let us know with a Fedora Infrastructure ticket.

Also, as one of the maintainers of supybot-gribble in Fedora and EPEL, I will be retiring that in favor of the limnoria package very soon. It should be now available in updates repos for your upgrading pleasure.

Ursa: You are master of all you survey.

General Zod: [bored] So I was yesterday. And the day before.

June 01, 2016

Kevin Fenzi : What didn’t happen today?

Today, astute observers may have noticed that and didn’t participate in the long running, annoying, insecure and exasperating tradition called “mailman day”. This is where mailman spams you with a reminder for every list you are on with your plaintext “password” in clear email.

Since we have now fully switched over to mailman 3, you can relax and rest easy that we DID NOT spam you with your mailman password today, nor will we from now on. ūüėČ

Our mailman3 instances authenticate users via Fedora Account System OpenID, persona or yahoo, no local throwaway passwords to look for or remember.

May 11, 2016

Kevin Fenzi : Mailman3: no topics support (with alternatives!)

In mailman2 there is a feature called ‘topics’. What this allowed you to do was create some filters on the list and when those filters matched mailman would add a header telling the user that this post was part of a ‘topic’ or not. You can then use the mailman interface to tell it you only wanted posts on a specific list of topics and no others. As far as I know there aren’t too many lists that used them, but there are a few we run in the Fedora Project that do: The prime example being the ‘package-announce’ list.

The package-announce list gets emails from bodhi (the Fedora updates system) about each update when it’s pushed into the stable updates repo. As you can imagine with a fast moving distro like Fedora, this list gets a LOT of posts. This list has topics for each Fedora release (22, 23, 24, etc) and also “Security” and “New package” updates. Using topics users could get a subset of these emails that they wanted.

Sadly, topic support isn’t yet available in mailman3, and it’s unclear right now when it might be implemented. Not wanting to hold up our migration to mailman3 forever, we want to go ahead and move lists like package-announce over as soon as we can. Of course this means those folks using topics are going to be in a bind. However, there are some alternatives:

  • You can switch to using a RSS feed directly from bodhi itself. For example: will give you all stable Fedora 23 updates. will give you all newpackage type updates, etc. Pretty much any search or page with a rss symbol on it on bodhi is an RSS feed.
  • You can set up notifications in the Fedora notifications service: to either email or IRC. For packages you are interested in, all updates, security or newpackage updates or any combo.
  • You can of course just stay on the package-announce list and filter things locally. You will be getting emails about some things you don’t care about, but emails are pretty small in this day of streaming video. ūüôā

Hopefully those interested can switch over to one of the alternative methods above until topic support once again lands in mailman.

May 04, 2016

Kevin Fenzi : Fedora account system and FreeIPA

Over the years, a number of times, people have asked us about migrating from our own custom Fedora Account System (FAS) to FreeIPA.

We would love to, but right now as things stand it’s just not a great fit for us. In the past we have had trouble articulating what the issues are, but now Patrick has written up a wiki page with all the items we know of:

Hopefully we can keep open a dialog with FreeIPA folks and keep the above wiki page synced up so perhaps someday we can migrate when blockers are no longer in our way. ūüôā

April 16, 2016

Kevin Fenzi : ansible 2.0 going stable in f23/f22/epel7/epel6

Just a quick heads up: ansible- is heading out to the stable updates repos for Fedora 23, Fedora 22, EPEL7 and EPEL6.

There’s a few outstanding issues folks might hit, but we figured it was finally time to push to stable. If you use includes for handlers you might hit ( ). This one will be fixed in 2.1 by allowing you to add a config option to set includes as static or dynamic. Also, you might see some intermittent issues with some hosts saying “unable to resolve remote temp directory” ( ). Hopefully this will be fixed in, and there are some workarounds in the issue.

If you run into another issue, please do check if it’s reported upstream and if not report it.

We have also pushed out a ansible1.9 compat package for those folks that need to stick with 1.9 for now for whatever reasons. You can simply remove the ansible package and install ansible1.9 and you can use that version. It’s currently at 1.9.4, but there’s a 1.9.6 update pending updates-testing now also.

Happy ansiblizing!

April 03, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-04-03 edition

Well, it’s been a while again, so I thought I would update everyone on recent goings on in Rawhide, Fedora’s rolling development release.

Since my last post we finally moved over to having Rawhide and Branched (what will be Fedora24) done by the new pungi4. There are sadly lots of little issues that have been cropping up and getting fixed by Release Engineering, but it’s been a bit of a bumpy ride. As of this post, the last Rawhide compose that finished was the 2016-03-31 one, all the ones since then have failed. Hopefully this will get sorted out tomorrow. The current cause of problems is adding atomic/ostree to the compose (which we very much need/want), it’s just been more difficult that we first hoped. Once things settle down we will really be in a better place than before, with full composes (tested by openqa!) every night.

There was a bit of delay since the Branching event on signing, but last week we got back on track and (almost) everything in rawhide should be signed by the Fedora-25 key now.

With compose issues and general breakage in the install path it’s been difficult to install rawhide recently. Again, once we get the normal composes back on track this should clear up. As usual, once you have a rawhide install there hasn’t been really much breakage of the day to day use path.

Astute observers would have noticed that the rawhide-source mirrormanager repo wasn’t working after the switch to pungi4. I’ve fixed that up the other day and rawhide-source should be working normally again.

Kevin Fenzi : How long does it take for that Fedora ssh key to sync?

There’s been some confusion about how long it takes from changing your ssh key in the Fedora Account System (FAS) and being able to use it to access things like or or The method has also changed a while back so I thought I would write up this short post on it.

TLDR version: for ssh access to fedorapeople, pkgs or fedorahosted, it should take about 2-17minutes. (usually 2). For access to other machines (if you are in a sysadmin group), it should take 15 to 30min (usually 15). So, you would be safe saying “within 30minutes”.

In the past, we simply had a cron job on every machine that would run every 30min or so and sync everything. This was a massive waste of resources. We had to size our fas servers to handle the load of every machine +/- a few minutes hitting them and for the most part, absolutely nothing changed. Then, along came fedmsg and we now had easily available data to know when something has changed and what it might affect. The new process is run from a fedmsg listener: It waits for ssh keys changes to be mentioned on the fedmsg bus, then it looks to see if they are from someone in a sysadmin group or not. It will then wait for about 30 seconds (if there’s more changes coming in, might as well do them at once). If not, run a ansible playbook that updates just those machines that they might have access to. If they are, use a different playbook that updates all the machines in turn.

The non sysadmin playbook takes about a minute to run. So, the vast majority of these changes will appear in about 2 minutes.

The sysadmin playbook takes much longer (There’s a LOT more machines in this playbook): 15minutes or so. The vast majority of changes here will appear in 15minutes (but it depends on which machine you are waiting for access on of course).

As a side note this applies to ssh keys and mail aliases, but not koji certificates. Those are active immediately as soon as they are issued (and your old certs are revoked).

Since the script will only run one playbook run at a time, the worst case is your changes just miss a sysadmin playbook run and you are yourself a sysadmin. Then you may have to wait 30 minutes. For the case most people will hit, it should just be a minute or two.

March 30, 2016

Kevin Fenzi : taskwarrior tips and notes

A year or so ago some co-workers pointed me at taskwarrior ( ) as a neat way to do todos and task tracking. At the time I tried to dive into it and it just didn’t seem to work out for various reasons ( I didn’t give myself free time to really dig into it, tried to make it do time tracking as well, etc). Last week and over this last long weekend I gave it another go and I think it may stick this time. I thought this might be a good time to share the tips and workflows I came up with in case anyone out there wants to give taskwarrior a try.

First a bit of background: I’m a sysadmin by trade, so my todos and tasks are often tons of short tasks and reminders along with meetings and longer term planning/thinking items. For many years my workflow has been to use my mail client inbox along with a text file for things. If it was in my inbox it means I need to take some action or reply to something. If it’s in the text file it’s something I have already done (to keep a record) or some short term todos of things that don’t have email associated. This setup works pretty well, but it’s bad at followups (where I replied/did something and the other party didn’t), anoying for repeat items (have to use a template for the text file or copy and paste), and bad at searching for particular things.

Enter taskwarrior: The new workflow I’ve settled on pretty much replaces my text file with taskwarrior, so I still have mail inbox, but taskwarrior helping it out. There’s extensive taskwarrior docs on the web and the man page is pretty nice too. It’s very nice that taskwarrior is flexable, it can be used a bunch of different ways with different workflows and methods. You can set things up to use it the way you want to. Above and beyond those things:

  • Setting up meetings confused me, because I don’t want to see them until near the meeting time/day, until I realized I can use wait to make them wait until I want to see them. For example, here’s the EPEL meeting thats every wed at 18UTC: ‘task add ‘EPEL meeting’ project:work recur:weekly due:2016-03-30-12:00 wait:soww’ (soww is start of work week). So, I won’t see this on my task next list until monday when I am looking at my upcoming week.
  • tags are very handy. For example I have been adding +nagios to any task where I am looking at or acting on an alert. That way I can go back after a while and look and see if there’s any trends or things we really want to try and fix because they re-occur. ‘task completed +nagios’ will show all these completed nagios tagged tasks. (likewise, ansible commits, bugzilla, infratrac and internal only tickets)
  • ‘task edit <id|uid>’ is pretty handy to see what task has on a task or tweak it if you added it wrong.
  • If you really do want to wait forever for something, you can use ‘wait:someday’ or ‘wait:later’ and it will set it to 2038. ūüôā
  • You can also add ‘until’ on meetings for example for just after they would end or the like, and task will delete that task if it reaches the until time. This is handy for example if you miss a meeting, you didn’t do it, it should just be removed, but you don’t even have to think about it if you add it with an until.
  • Another cool one from Paul Frields: You can use ‘task log …’ to create a task thats already marked done. This is handy if you already did something and didn’t have a chance to make a task before hand. Saves you adding a new task only to mark it done.
  • Out of the box, task doesn’t keep track of how much time you spent on any given task. You can use some hooks to record this, but it doesn’t do it by default. I think the last time I tried it out this might have confused me, but this time thinking about it, I am not sure I care exactly how many minutes I spend on each task.
  • I’ve found it handy to add text as much as I can. When adding the task ‘task add …’ (describe it as simply and completely as you can), when starting the task ‘task N start …’ (usually here a link if it’s to a ticket or bugzilla thing where I know the url when starting)’, and when completing the task ‘task N done …’ (here a commit url or the like with the work done). Or ‘task N annotate …’ anytime you made some progress and it’s an interesting or noteworthy thing to add.

Taskwarrior does have a server you can sync to (taskd). It’s pretty easy to setup and uses TLS certs to authenticate clients. There’s also an android app, which is kind of painful to configure, but seems to work otherwise. I’ve not done all that much with it yet, but it might be handy if you are at a event or the like and want to record a task but don’t have your laptop out. There’s also bugwarrior that lets you pull from trac/bugzilla/pagure/etc into task. I did that the last time I tried things out, but I think it’s not a good workflow for me. When I do work on a ticket or bug I can add that task manually, and leave most/many of them that are not waiting on me to sit in their space.

Reports are quite handy for seeing how much you have done and what you have done in a time period, and pretty easy to setup. Just add them to your .taskrc with what to filter on and what cols to show. I setup one for what I have completed for the current work week and what I completed for last work week (useful in weekly reports to management).

Still getting the hang of things, but I am hopeful that I’ll stick with taskwarrior at least for now.

March 09, 2016

Kevin Fenzi : encrypt all the things: blogs

So, with ssl certificates pretty easily available these days from it’s more and more worth looking at making sure you are using https instead of http for everything you can.

There are however some corner cases that are… difficult. One of those is blog aggregators. A while back we moved our to This was to get it out of the domain because we send HSTS headers for telling browsers they should always contact that domain with https. Blog aggregators are in a tough spot, because they simply pull content from a bunch of different sites and put it in one place and link to it. Unless 100% of the blogs that the site is aggregating are also https, if the site itself uses https most browsers will show you a nasty warning about mixed encrypted/non encrypted content. So, for now, since most of the blogs on are http, we are leaving itself as http.

However, we would love to get to the point where all the blogs we aggregate are https. I don’t think it’s an impossible journey. Here’s what you can do if your blog is listed in

  • Check to see if your blog site already supports https and has a valid cert. If so, you simply need to login to and edit your .planet file to use the https link instead of http. Done.
  • If your blog site doesn’t support https (yet), ask your blog provider about adding it. They should be able to add a cert pretty easily.
  • If your blog site doesn’t support https(yet) and you run your own blog, why not add https support?

If we can get a critical mass of blogs using https, we can look at switching the site over too. Help us out. ūüôā

February 24, 2016

Kevin Fenzi : A Fedora Distribution download primer

With the fresh news of a compromise in the Linux Mint distribution images, I thought I would take a few minutes to explain how Fedora handles image downloads and what you can do as an end user to make sure you have the correct and official Fedora images.

First, lets take a look at what happens in each step if you open your browser to (our install images download site):

  • You type ‘’ in your browser.
  • First, your operating system asks your dns servers for the IP address of If your OS is using dnssec, then it will get a cryptographically signed answer. If not, it will get whatever answer your dns servers give it.
  • Next your browser may try and connect to via http. We have setup to redirect all http requests to https, so this would get you a redirect.
  • On the first https connection to, we send a HSTS header. This tells your browser (if supported by it) that it should ALWAYS use https to talk to this site. Even if you enter, it would just correct that and connect on https.
  • Once connected you can download distro images by clicking on the download link for the image you like. Once you click on a download (unless you have completely disabled javascript), there’s a screen describing how to verify your download:
  • Once you have downloaded your image, you need to do two things to make sure it’s the valid and official image: First, check the gpg signature of the checksum file. Official checksum files in Fedora are always signed. You can get the gpg key for that Fedora release from, most any keyserver, or from the fedora-repos package if you already have a Fedora install. Additionally, if you import this key and then refresh (gpg2 –refresh-keys) you can see the signatories of that key and decide based on all that if you trust it. If thats correct, then you can use sha256sum to check the checksum of the image. YOU SHOULD ALWAYS DO THESE CHECKS. ūüôā

So, we have dnssec, hsts and signed checksum files. Would that have helped us any if we suffered a attack similar to the one the Linux Mint folks suffered? In that attack, their download machines were compromised and intruders replaced checksum and download links to their own version. If that happened to Fedora, the only step above that would protect people would have been the gpg signature check (which sadly, many folks never do, and for good reason, it’s hard and anoying and manual).

In Fedora 24 the workstation and server editions will be moving to preferring the usb media creator application instead of preferring direct downloads of images. We will need to make sure it’s as secure as we can make it, but there may well be a manual step of checking the application after you download it. (Unless you already have Fedora and install it as a normal package). In it’s current form it already downloads checksum files from the Fedora master mirror via https and uses that to check downloads, but more can be done.

February 21, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-02-21 edition

Once again I have been busy and haven’t gotten a notes from the trail out recently, time to correct that!

Just a bit over two weeks ago we had a mass rebuild event in rawhide. This one was for gcc6 which just landed in rawhide. The actual rebuilding took (much like the last one we did) about 36 hours, but thats just the tip of the iceberg. From the devel-announce post about the mass rebuild completing: “16129 builds have been tagged into f24, there s currently 1155 failed builds that need to be addressed by the package maintainers.” Dealing with those failed builds is always the intensive part of any mass rebuild cycle, since it takes maintainer time to sort out whats going on (and often upstream communication) to fix.

In just a few days we will be branching off F24 from rawhide. Do make sure when this happens on tuesday that you follow the direction you want: Either keep on rawhide or follow Fedora 24 until it’s release in June.

With the mass rebuild and a bunch of other changes people are landing before the branch event there’s been some issues with the rawhide install path again. If you need to install rawhide, do check for the last image that passed install tests and save yourself time and energy before downloading.

Finally we are close to landing the changes in the compose process around pungi4. This will make every day’s rawhide just like a release compose. It will have all the trees, images, checksums, metadata and such every day. This will also be used for the Fedora 24 cycle composes, so everything should be a good deal more consistent.

January 24, 2016

Kevin Fenzi : A tale of that most hated hardware: printers

Ask any sysadmin what hardware they have to deal with is the worst and I think universally they will all tell you it’s printers. There’s a lot of reasons for this, but I think the biggest ones are:

  • They use / misuse consumable items like ink, paper, etc.
  • They have a ton of moving parts to jam or break or get blocked.
  • They tend to be in places where end users are asked to service them and often put in incorrect paper, jam things or misrefill ink, etc.
  • Recent trends have made printers lowest margin items. If your printer doesn’t work, throw it away and get a new one, it’s cheaper than fixing or refilling your existing one.

A few years ago I got a HP multifunction printer (when it’s predecessor broke and it turned out to be cheaper to just buy a new one). The scanner part of it works great and always has. The printer part on the other hand has been nothing but horror. I don’t tend to print much at all, but a few times a year for some reason or another I need to print a document or directions to something or whatever and every single time this printer has failed me on the first print attemps. Reasons for it’s breakage have included:

  • It’s been so long since I printed the ink “dried up”. Once I was able to get it working by cleaning it with a q-tip, once by making the printer run it’s clean print heads cycle about 20 times, once by buying a new cartridge (almost as much as a new printer).
  • Once it decided to just not turn on at all. Unplugging it completely and waiting 5minutes and replugging it “fixed” it.
  • Once the color ink cartridge was low so it wouldn’t print anything (even just using the black cart that was full!) which required a new color cartridge.

Over the holidays I cleaned out a lot of my computer room, so I could see all the printers I still had that I hadn’t managed to throw away. 2 old HP inkjets, an old epson, and… what is that at the back of the room? why it’s my old HP laserjet 4P. I used that printer extensively during college and after and finally retired it when the tonor ran out and I wanted to move to a new shiny color inkjet printer.

So that got me thinking. The laserjet 4P only has a parallel port, but a quick check found a parallel to usb cable for $8. At the time I retired it, a tonor cartridge for the 4P cost around $100, but looking now, I found one for $12. A quick amazon order and a few days the items arrived. I hooked up the printer to my main server and put in the new tonor cart, added it to cups and printed a test page. A lovely 600dpi test page.

So, hopefully this laserjet 4P will now print reliably for me for the rest of my life and I can avoid the inkjet “buy a new one” everytime I need to print something.

January 18, 2016

Jeffrey Haemer : Another Sale!

The last one was "Best Sellers." Now the video is also "Best Reviewed." Nice! - Best of 2015 Sale

Kevin Fenzi : A vist to a Fedora datacenter

I just got back from a visit to the largest of the datacenters used by Fedora Infrastructure, and I thought I would share a bit about why we do such visits what what we do on them.

We have a number of sites where Fedora Infrastructure has at least one machine, but we have one main datacenter (sponsored by our main sponsor, Red Hat) where all our build system machines are, our fedorainfracloud openstack cloud, and most of our main servers are. This site does have some on site folks who can handle most of our day to day hands on needs, but sometimes we have things that need a lot of coordination, or extra time to complete, so we try and ‘save up’ those things for one of our on-site visits.

(Side note: If you are interested in donating machines/colocation space for the Fedora Project, please do mail We would love to hear from you!)

On this particular visit last week we got quite a few things done:

  • Package maintainers will be happy to hear we racked a new HP Moonshot chassis. This will hopefully allow us soon to run armv7 builder vm’s on aarch64 virthosts which will vastly decrease build times.
  • We moved 5 machines to a new rack to be used for testing storage solutions. We want to see if we can move to more gluster or open source storage solutions. These machines will be used to test out those solutions as they mature.
  • We added memory to a number of cloud compute nodes, allowing us to run more and larger cloud instances (and have more copr builders).
  • We added a new equallogic storage device to our cloud network. This is a smaller one than we have now, but it will allow us to test things and also give us some more cloud storage room.
  • We fixed some firmware on some power supplies (yes, power supplies have firmware too!). This involved an elaborate dance of moving power supplies around in various machines to get them all upgraded (which would have been very tedious remotely).
  • Moved an old remote kvm and added a new kvm. We don’t tend to use these all that much, but they are very handy for machines that don’t have proper out of band management.
  • Pulled tons and tons of unused cables out of all our racks. On site folks don’t tend to want to remove cables, only add new cables for new devices. There was a bunch of old cables that were only connected to a switch or serial port or power device and were no longer needed.
  • Setup a power7 machine (many thanks to IBM) in our cloud network. As soon as we have it installed and setup we should be able to have ppc64 and ppc64le cloud instances and also do copr builds of those arches locally instead of against a remote server.
  • Updated our records of which servers were on which serial, power and switch ports. This information is very important when making remote requests or power cycling servers so you don’t accidentally get the wrong one.
  • Pulled out a few older servers we were no longer using.
  • Racked a few new servers we will be using for releng/buildsystem stuff.
  • Checked all the servers for any error lights or alarms. Cleared a few we had already fixed, and fixed a few we didn’t know about yet.

All in all a good weeks work that should set us up well for the next few months. I would like to thank Steven Smoogen (for being on-site and getting all the above done with me) and Patrick Uiterwijk (who held down the Fedora Infrastructure fort while we were off at the datacenter and helped get all the above done from the remote side) as well as Matthew Galgoci (Red Hat networking guru extrodinare) and our On-site folks, Jesse Iashie and Pedro Munoz. It’s a pleasure to work with you all.

January 02, 2016

Kevin Fenzi : Rawhide year in review

Looking back over 2015, I see I only managed 7 rawhide posts. Will see if I can do better for 2016. In any case, looking back at these posts and emails I have sent to mailing lists with rawhide topics I thought I would do a bit of a year end roundup.

Overall we have made some great improvements:

  • We now have openqa running on rawhide images every day to let us know when the install path breaks so we can fix it and so people know when doing a rawhide install will work for them.
  • Rawhide is “mostly” signed now. When a large set of rpms lands right before the compose the autosigner sometimes doesn’t sign them all before the compose, but for the most part all the rpms are signed. (For example, all rawhide composes since 2015-12-29 are fully signed, that one had a qt5 build that landed right before compose).
  • Slowly it seems like more people are using rawhide day to day and helping report issues or problems. Rawhide is not for everyone, but it’s good to have a pool of folks using things and testing and reporting bugs so they can get fixed.

Coming up in 2016:

  • releng hopes to soon change the rawhide compose process to use pungi4. This would be the same process then used for real Alpha/Beta/RC/Final composes. It would mean we would have a full compose of all things every night we could test and use.
  • We are working through bugs with sigul batch signing mode (we have already made some great progress fixing several issues), and once those are all solved the autosigner should be able to handle things much quicker.
  • Hopefully we can start doing some more automated testing on composes as well.

Looking forward to another fun year running rawhide and making it better for everyone!

December 23, 2015

Kevin Fenzi : Rawhide: Notes from the trail (2015-12-23)

A short note from the trail for those folks riding along with rawhide this winter:

Openqa (¬† is running on daily rawhide composes, letting everyone know when the install path is not working as expected. This has already resulted in several quick fixes and bringing things back on track in the last month. It’s really nice to get the install path to rawhide to be as stable as the day to day trail ride.

There is currently an issue with gtk3 and gtkmm30 using applications crashing on start. The gtk3 folks are working on a fix. If you need something that uses gtk3/gtkmm30 right now, downgrade to gtk3-3.19.4-1.fc24.

Happy Holidays and Merry Christmas from the trail!

December 22, 2015

Kevin Fenzi : On giving

Toward the end of the year, many folks thoughts turn to giving. Either giving a monetary donation, or time and energy. I think there’s a number of reasons for that: In the US at least tax deductible donations are something people want to get in before the end of the year, people know more about how much they might have to donate, as well as having more time away from the pressures of work to think about groups that might be worthy to donate to. Whatever the case, I thought I would share a few thoughts around such donations.

First it’s good to ponder on what kind of gift you might be able to provide. Sure, if you have some money you can allocate thats great, but there’s lots of other ways to give back. Try asking what you can do to help and most any non profit group will have a long list of helpful things to choose from.

Next take a look at what groups you want to support. This of course is highly personal, but do take a look at information about a group before donating to them. For example, many of us in the US will see people outside shops and stores ringing a bell and asking for donations to the Salvation Army. At first glance you might think this a worthy group helping the less fortunate, but looking for further information will show you that the group is highly religious ( for example requiring people they help to listen to them proselytize their beliefs ) and discriminatory to LGBT folks. ¬†Additionally, look at how much of your donation might actually get to the people the group purports to help and not get used for “overhead”. Legit non profit groups are required to publish their finances.

Once you have decided to give to some groups, look and see if there’s any matching fundraisers going on or planned. This is where existing members will match new member donations over a time period to try and help the group and add more members. Your donation can go further to help the group if you wait and donate at such a time. Also, check with your employer, many companies have matching programs where they will match your donation to non profit groups. This is another great way to get more money to the non profit you are wanting to help.

I’ve decided to donate some money to three groups this year:

  • Electronic Frontier Foundation – I was able to donate a few weeks ago when they were doing a matching fund drive, and also my wonderful employer (Red Hat) offers matching funds, so I was able to get them 3x what I donated. If you are at all active in on line rights and freedoms you know who the eff is. These folks do great work and everyone should support them.
  • The Software Freedom Conservancy-¬† They are also currently holding a matching funds fundraiser and my wonderful employer (Red Hat) offers matching funds, so again I was able to get them 3X the funds of just my donation. These folks are doing great work handling legal and paperwork for open source groups as well as being the only ones willing/able to fight for enforcement of the GPL. They lost a number of spineless corporate backers this year with their GPL enforcement battles and its very important that we help them to keep going.
  • Colorado Greyhound Adoption – This is the greyhound group that I have worked with in the past and got my wonderful pups from. They do great work, but just one dog with high medical bills can wipe out their entire medical budget. Sadly, I wasn’t paying attention and there was a matching day on the 8th that I missed, but at least my employer (Red Hat) will match my donation to them, so they get 2X as much as my donation.

I hope you all will think about giving back to the groups that help advance causes you believe in.

November 16, 2015

Kevin Fenzi : Rawhide: notes from the trail (2015-11-16)

There has been a lot of things going on in rawhide of late, so past due for another note from the trail. :)

The python 3.5 rebuild has landed. The vast majority of it was done in a side tag by Peter Robinson, Kalev Lember and Robert Kuska (and others!), then merged back into rawhide on friday (the 13th). There are still a number of packages that need fixes to build against python 3.5, expect most of them to get fixed up this week. If you have some of those installed, dnf may well hold back python3 and all the newly rebuilt packages until the ones you have installed are all fixed or you remove them.

There was a dracut bug over the weekend (landed friday and fixed this morning) that might result in newly updated kernels not having an initramfs. Make sure you get the version from this morning before installing/upgrading kernels. (That was: bug 1281917)

On the kernel front, the 4.4.0 merge cycle has been running the last few weeks, resulting in tons of changes and some fun bugs. If your tun/tap devices don’t work, that is¬†bug 1281674¬†which already has a fix, hopefully upstreamed soon. I’ve also seen plymouth unable to prompt for decryption on boot (not yet filed), and there was breakage in video for vm’s (already fixed upstream hopefully). Likely there will be a 4.4.0 rc1 kernel for rawhide today with many of those fixes.

There were no rawhide livecd’s or arm appliance images for a few days late last week. This was due to the builders all being moved to Fedora 23 and missing a few packages they needed for koji to be able to create those images. That should be all solved now, however some images still may not compose due to lingering python3 deps.

On the graphical front, Gnome logins switched from being Xorg by default and having a special “Gnome wayland” session for wayland to the reverse. Wayland is now the default with the normal “Gnome” session. There is a “Gnome Xorg” session now if you need to fall back to Xorg for any reason. How well this ends up working out will decide if Fedora 24 Workstation will default to wayland or not. In my testing, Gnome/Wayland is prefectly usable day to day now, many of the issues in the past have been fixed up. There’s still a lot of polish type things to get right I think, for example I use sloppy focus and it sometimes doesn’t act right with wayland.

Whew. As usual lots going on in rawhide and always a fun bit of trail riding. Until next time, keep them doggies movin!

November 06, 2015

Kevin Fenzi : Lenovo Yoga 900 and Fedora Review

Just about 2 years ago now I picked up a Lenovo Yoga 2 Pro, which I have been using happily since then. My full review was:¬†Lenovo yoga 2 pro and Fedora review¬†and then a followup a year later at¬†yoga 2 pro: a year later. It’s been a great little laptop, but I decided it might be time to replace it soon. Mostly because the warentee on the yoga 2 pro was done, the cpu was showing a bit of age and I was hoping there would be some advancements in laptops over the last 2 years.

A few weeks ago, Lenovo came out with the Yoga 900, which was the successor to last years Yoga 3 pro and it in turn my Yoga 2 pro. The stats and early reviews looked pretty nice, so I ordered one.

I was hoping for a smooth Fedora experience, but sadly I ran into two issues right away after booting from a Fedora Live USB:

  • The wireless didn’t work. It didn’t even see the interface. Of course I ran into this same problem 2 years ago with the yoga 2 pro: it was the ideapad_notebook module not knowing about this laptop and thinking it had hardware rfkill set on all the wireless modules. A quick bit of hacking on that module and it no longer showed the interfaces as hardware blocked. Unfortunately, it still didn’t work. I looked then to confirm that the intel 8260 wireless was supported by Linux, and indeed it is. Looking closer, I saw that the pci id of the card I had was not in fact listed in the driver. They had “0x24F4, 0x1130” and mine was “0x24F3, 0x1130”. Fixing that quickly in the module got the iwlwifi module working with the interface and all was well. This bug tracks these issues (both of which should be headed upstream):¬†bug 1275490
  • The touchpad and touchscreen didn’t work. This seemed to be a probing issue with the i2c-designware platform. A few posts to the linux i2c-devel list and some exchanges with an intel engineer and I had a one line patch that gets it working. Hopefully that will be upstreamable soon.

With those issues out of the way, I went ahead and did a rawhide install on it, copied my /home over to it and then updated my ansible playbooks that setup all the things I want on my laptop. It was a pretty painless process overall and then I was up and running on the new machine. :)

Good things:

  • All the nice things about the yoga 2 pro came over: It’s actually a bit smaller and lighter, the screen resolution is still 3200×1800, it fits in the laptop sleve I got for the yoga 2 pro.
  • It’s really fast. The latest gen i7-6500U’s are really zippy.
  • I’ve not traveled too much with it yet, but in the short times I have the battery life has been great. Estimating around 12 hours or so, almost 2x the battery life on the yoga 2 pro.
  • The hinge is a new “watchband” style. I never had any issues with the yoga 2 pro’s hinge, but this one looks a good deal sturdier.
  • 16GB of ram (over the yoga 2 pro’s 8GB) is nice. Might run some vm’s now. :)
  • The laptop backlight now has 3 levels: bright, low, off. I could see using the low setting in some cases.
  • It has a USB-C port instead of a mini-hdmi on the yoga 2 pro. I have no USB-C cables, but this seems like it will be a win allowing me to connect (with the right cables) to a DP, hdml, usb, whatever via the USB-C port. I am not clear on if I could charge from this port as well, but I’ll see when/if I get a cable. :)
  • The power port is also a USB 2 port. Could be handy in some times when you don’t have power and need to attach another usb device.
  • The firmware actually has a way to enter secure boot setup mode and add your own keys. The yoga 2 pro didn’t have that.

Bad things:

  • Just like the yoga 2 pro, firmware¬†updates seem to require windows. ;( Lenovo provides handy iso updaters for many laptops, but not this one. Likely because they outsourced the firmware to some 3rd party. ;(

Just different:

  • The keyboard feel is different. I am not sure if I like it better, don’t like it, or just need to get more used to it.
  • 512GB ssd instead of a 128GB in the yoga 2 pro. I wasn’t even using the 128, so it doesn’t really seem that much different. :)

Overall I think the upgrade is well worth it, but it’s all mostly incremental improvements, nothing jaw droppingly awesome. The active warentee, cpu speed, memory and battery life all make it worthwhile IMHO.

October 12, 2015

Kevin Fenzi : The 5 states of the modern sysadmin

I think there’s (at least) 5 states you might find yourself in as a sysadmin in these days:

  1. Day to day things that aren’t (yet) automated.
  2. Automating and designing for the future.
  3. Fires and outages
  4. Interruptions
  5. Time to dream

In general you¬†will¬†want to move things from the first bucket to the second as much as you¬†can. Automate all the things that happen often, or (re)design things so you no longer have to do such things day to day. So if you find yourself restarting a webserver every ¬†day, figure out why it needs that and fix that. Or if you have to spend lots of time processing new lists or resetting passwords, make those so they are self service. This might also include reading emails and lists and feeds, if you spend a lot of time here that you don’t get any benefit from, perhaps it’s time to drop some of those?

Of course outages and critical problems are another chunk of time. When things stop working at 2pm or 2am, you¬†start working on bringing things back to normal. Bucket two helps here as well, if you find out things are often causing outages or problems, redesigning them or fixing the underlying problem saves stressfull outage time. It’s important after outages and critical problems to try and have some time from bucket 5 as well. Often just thinking about how the problem happened and spending some time looking at it can give you a idea for a design to use in bucket 2 to fix it once and for all.

Interruptions are another chunk of time. While you might think it “only takes a minute” to ping your sysadmin about something, it ¬†takes a while to get back to the second and final buckets above where (hopefully) you are¬†spending your¬†time. This is why filing tickets or using some other tracker for your non critical things helps a great deal. Then, they can be processed in the first bucket when you have time to do that and also have a nice record of what you need to work on in the second one.

Finally another place that is great for you¬†to have time is a category I call “time to dream”. This is usually unstructured time where you as a sysadmin can ponder on how things are setup, look at logs or machines you don’t usually deal with, read about tools that you might be able to use to make things better, or just a totally new workflow for something.

If you find yourself in a sysadmin job where you spend all or most of your time in buckets 1, 3 and 4 you have a definite problem.

So, all you sysadmins out there: are there other states you find yourself in often that I haven’t mentioned yet?

October 03, 2015

Kevin Fenzi : Fun with wifi

The last few weeks I ran into some really nasty problems with wifi on my laptop: It started cutting out and dropping me offline for several minutes at a time and was slow and pretty unusable.

Of course the first thing I thought of was a bug in the rawhide Linux kernels (I run rawhide full time on my primary laptop), but going back to previous kernels or even Fedora 23 or 22 stable kernels showed the same behavior. Looking at the disconnects and re-associations I was pointed to it possibly being just very congested. So, I moved the channel my AP was on (man are there a lot of access points in this area now) and played around with the settings (My access point is a WNDR3700v2 running OpenWrt). That seemed ¬†to help a bit, but not all that much. I also noticed that all the other devices in the house seemed fine (sadly, most of them don’t have a good way to show signal strength or status, but they didn’t seem to be reconnecting).

Along this journey I got to play with the ‘iw’ and ‘nmcli’ commands a lot. I would recommend you all take a look at iw, it can sure tell you a ton of information about your wireless connection and how it’s working or not working. nmcli has also come a long way. ‘nmcli d wifi’ is something I am now using all the time rather than looking a the pull down menu of networks.

Finally, I swapped out the wireless card in my laptop. I had gotten a replacement one years ago when I got my laptop when I read reports of poor wireless performance, but never swapped it in when the existing card seemed to work. The new card worked TONS better. I can only conclude that the old one was going bad in some way. :(

Of course being a computer geek I took this as a great time to look into setting up WDS (Wireless Distribution System, basically a way for one AP to become a client of another and bridge all the connections to/from it, extending your range) in my house. My main access point is upstairs on one end of the house, and I am downstairs a fair bit of the time. I picked up some little NEXX WT3020’s for travel (also running OpenWRT), so it seemed it would be easy to setup one of them downstairs as a WDS client connected to the main AP. OpenWRT makes it pretty easy to set this up these days, but there were a few gotchas I ran into: Make sure you set the frequency width to 40 on both the main and client WDS client, and make sure you set the static ip for the client ap right (or else you won’t be able to easily get to it to configure it). Otherwise the instructions at¬† worked fine. I’m not sure how much better it really makes things, but it’s kind of nice to have if I ever need it.

September 25, 2015

Kevin Fenzi : A fully ansible powered Fedora Infrastructure

Just a week or so shy of 3 years ago, Fedora Infrastructure embarked on a journey to migrate to using ansible to manage all our hosts. It’s been a long road and one we could have done faster, but we wanted to do things in a transparent and measured way, not rushing and making quick decisions and changes that we would have to clean up later. Today that journey is completed.

So why ansible? What made us start this journey? We had a CM setup and it was working, but after weighing the advantages and disadvantages we definitely felt moving to ansible would be worth it. Just a few of the reasons:

  • We are very much a python shop, ruby syntax isn’t hard, but it’s just another thing to remember when making changes.
  • Not having to have agents on all our nodes is a big win memory and resource wise.
  • Not having to have yet another ssl cert setup for our CM is a great win.
  • We could drop func (that we were using for some ad-hoc things in favor of just using ansible for those things)
  • Not having to worry about version skew between our management host and clients was a big win.
  • Easier to explain and show how things work to new folks was a big win.
  • Not having to worry about scaling central host to number of managed nodes was a big win.
  • … and I’m sure more things I am forgetting now…

Of course just because we have finished this migration doesn’t mean we don’t have lots and lots of other things ahead of us. In 3 years ansible has changed some and we need to go back and clean up some of those things that currently work, but aren’t ideal anymore. Ansible 2.0 should be out before too long and we need to look at moving to that and helping out with it’s testing whereever we can.

I’d like to thank the Fedora Infrastructure team and all those who contributed to our migration efforts as well as the excellent ansible developers and community. We could not have done it without any of you!

September 23, 2015

Kevin Fenzi : Rawhide: Notes from the trail: dnf-1.1.2-2.fc24 will plum rustle your cattle

Just a heads up for you intrepid rawhide users:

The latest (as of this writing) dnf update, version 1.1.2-2.fc24 seems to break doing a lot of things you might want dnf to do (like update packages or list them or anything).

Bug is open to track the issue.

In the mean time you may want to manually download 1.1.1-2.fc24 from and downgrade and add a ‘exclude=python3-dnf-1.1.2-2.fc24’ to your /etc/dnf/dnf.conf

Keep em movin

September 20, 2015

Kevin Fenzi : Bodhi2 and it’s much faster updates flow

Now that we have had bodhi2 in production use for a while, I thought I would talk some about it’s backend and how it manages updates pushes.

First, for a bit of background, lets mention how bodhi1 did things:

  • release engineers would gather a list of pending updates and look over them.
  • release engineers would sign those updates
  • release engineers would tell bodhi1 to push them. Typically in freezes (alpha, beta, final) the branched updates-testing ones were pushed, then all the stable releases (both testing and stable). Outside of freezes, all branches were pushed at the same time (both testing and stable).
  • bodhi1 would start going through the list one by one and mashing the packages into repos.
  • At the end when all branches were done it would go through them one by one and gather updateinfo and other data it needed to inject.
  • It would then create rpm-ostree/atomic updates for each branch that has one one by one.
  • Then it would wait for all of them to be synced to the master mirrors.
  • Then it would go through them one by one and update bugs and note things in the bodhi updates.

As you might imagine this could take a very long time. It wasn’t unusual to see this be 12-18 hours or more for the full set.

Bodhi2 instead of being single threaded is very much mult-threaded:

  • release engineers gather a list of pending updates and look them over.
  • release engineers sign updates
  • release engineers tell bodhi2 to push them.
  • bodhi2 looks at all the pending updates: Are any of them security? It will do all those branches that have security updates first.
  • Threads are fired off for each branch that is to be done (first all security holding ones, then all the rest).
  • In each of those threads, subthreads are fired off to: mash updates (using createrepo_c instead of createrepo that bodhi1 used), gather updateinfo, create digests, ie, anything that can be done and wait to be collected together at the end is done.
  • When one branch is done, atomic trees are updated, it’s synced out and all work is completed on it (updating bugs, etc).

So, where bodhi1 took 12-18 (or sometimes more!) hours to push all Fedora branches, bodhi2 is taking around¬†6 hours to push them all. The slowest of the branches is of course fedora-21 stable updates (It takes about 4.5 hours to complete), which makes sense as it’s been out for a very long time now and has a very large pile of updates in it.

Since things are multi threaded in bodhi2, failures and issues are better as well. If something broke somewhere in bodhi1 pushes, everything ground to a halt until it was fixed and resumed. In bodhi2 if there’s an issue, only that particular branch will fail and can be resumed anytime. Everything else completes.

Moving forward there’s even more improvements we hope to roll out: Right now bodhi handles the mashing of updates on it’s own instance, and can be somewhat limited by it’s I/O bandwith, so we would like to move that out into koji and have koji do the mashes and create the updates trees. That way it can be farmed out to a bunch of seperate builders. There’s some createrepo_c changes hopefully we can get working to add multithreading to delta rpm creation, which may speed things up too. Finally we may tweak the threading to do all the security and non security branches at once (memory and I/O permitting).

September 18, 2015

Kevin Fenzi : rawhide: notes from the trail (2015-09-18)

Long overdue again for another “This week in rawhide”, I realized that I haven’t been able to do them weekly for quite some time, so I’m changing the name to ‘notes from the trail’ to play on the rawhide/cattle drive metaphor.

There was a good bit of rawhide discussion at flock this year and some of the things discussed there have already come to pass: Namely, Adam Williamson has setup openQA to run tests on branched (what will be Fedora 23) and rawhide. So, we now have some better idea when things are not working or didn’t generate images to test, etc. It’s sending emails to both fedora devel list and fedora test list.

I posted about the flock discussions in but sadly it devolved into a discussion if we should rename rawhide or not and if so, to what. We still have not yet landed pungi4 for doing the rawhide composes, but hopefully that will happen before too long, and I am not sure the progress of adding taskotron checks to rawhide builds is.

Current rawhide should be back to ‘mostly signed’ packages. I have been signing things as time permits.

Finally, I would like to touch on something that has been mentioned in the various renaming rawhide to something else threads: perception. I cannot count the number of times I have seen someone appear in a end user support channel (#fedora, fedora-forum, users mailing list, etc) and say “Hi, I want to run rawhide, what do you think?” to which the answer is variants of “Oh no, rawhide breaks all the time” “It will blow up on you” or the like. On one side I can understand support folks saying that. They don’t want someone hitting bugs and getting upset and they don’t want someone who isn’t experienced in troubleshooting to get in over their head. On the other hand, its just not true anymore. I cannot recall the last serious breakage I ran into. The last bug I filed was about selinux preventing my root /var/spool/cron/root job from running (which was easy to work around and notice). When I last redid the rawhide wiki page I made sure to add a “Audience” to it:¬† So, next time someone asks you if they should run rawhide, point them there and ask them if that describes them.

September 03, 2015

Jeffrey Haemer : Sale on My Git Under the Hood LiveLessons!

Well, I haven't blogged much lately, have I? :-)

Heather Fox, from Pearson, tells me that my Git Under the Hood, LiveLessons, is a best-seller, and ison sale this weekend.

InformIT Labor Day Sale - Save up to 70%

August 16, 2015

Kevin Fenzi : Flock 2015 day 4 ‚Äď saturday

Saturday came a bit less early than the previous days as we had no keynote, just workshops, so everyone seemed a bit more refreshed.

Again infrastructure folks met up and worked on various projects. Dennis got the builders all upgraded so we could test some dnf/mock patches, but that caused things to break in another area. Happily Ralph was able to iterate on a patch and we applied it as a hotfix to get all the builders up and building again.

There was some interesting discussion about using PCP (performance co-pilot). Very interested to see what kind of information about our infrastructure this could bring.

John containerized out pastebin service as a proof of concept. I am looking forward to looking it all over and seeing how it’s done and how we can better use containers and such for our applications.

After workshops everyone split out to various places for dinner. I went with a group to the Genosee brew house. Very nice view and great beer!

Kevin Fenzi : Flock 2015 day 3 ‚Äď friday

Friday started with a keynote by John Schull at eNable. They basically are a community around designing, building and distributing upper limb prognostics. With 3d printers getting so common and good, it’s enabled people to design and print their own. There’s tons of places people could contribute to the project, from designing things, printing them and shipping them, to helping gather statistics about the community and help them move on to other exciting areas. Really a very cool project that I had never heard of before. Kudos to the flock organizers for bringing John in.

After that we moved on to workshops (both today and tomorrow). The infrastructure team gathered up and started using a gobby document to do a sort of bar camp thing so we could discuss the things we wanted. Then we went through them. We didn’t mange to get them all, but we got a lot of them. Look for mailman3, bodhi2 and other exciting things very very soon.

After a lunch break we came and everyone started hacking on various things. I wrote down many of the things I discussed, but will need to gather them and go through them to make them coherent. :)

The friday evening event was at the Eastman house. Very cool place, lots of great pictures and old cameras, as well as a chocolate fountain. :)

August 14, 2015

Kevin Fenzi : Flock 2015 day 2 ‚Äď thursday

Thursday came all too early. After a quick breakfast, it was time for the keynote:

Major Hayden talked about impostor syndrome and related difficulties (like the Dunning-Kruger effect). He suggested a method used by airplane pilots might help some people to move forward when they are affected by these sorts of things: The OODA loop: observe, orient, decide, and act. Interesting topic and many people seemed to know it in the room. 

Next up was my talk on open source etiquette. It wasn’t completely horrible, but while giving it I saw a bunch of places I should change the pacing and what I wanted to mention and what pictures I might use. There wasn’t a big crowd, but there were at least a few people and one person did thank me after the talk so it can’t be completely bad. ūüėČ

I went back to the hallway track until lunch (another nice one at the hotel).

Then on to Paul Frields talk about remote work. I’ve worked remotely for the last 15 years, so I think I do ok, but there were some nice ideas Paul had about how to prioritize things and simplify workflow. I may well try a few of the things he mentioned in the coming weeks.

Next was a talk by Mike Mcgrath on Atomic and Openshift and docker and how all the things fit together. It was pretty high level, but it did clarify for me some things about what things were used in others and how they all fit.

Then off to a talk from Ralph Bean about technical debt and microservices. Some good background for other people, and some ideas we should definitely try out in coming months. In particular I liked the idea of doing a week where we just try and do cleanup and fighting down tech debt every few months.

The last session of the day was the “meet your fesco” session. Sadly, we didn’t get all FESCo there, but we did have some good discussions and questions. There was talk about how to better handle secondary arches, how to better deal with security updates, how we should handle the old “Incomplete” changes that never got actually submitted anywhere, and what the role of FESCo vs the council was. Great stuff.

The evening event was then at the Strong museum of fun. It was awesome. There were a bunch of old video and arcade games. I had fun playing centipede and air hocky and hercules (The worlds largest pinball). The winner tho was a game called “Whirly Bird” which was this mechanical helicoper game from 1969, it was quite cool. The food, drink and company was great as well.

Tomorrow will start out at 9am with another keynote and then we will go into workshops. Looking forward to actually getting things done tomorrow instead of just talking about them. :)

Kevin Fenzi : Flock 2015 day 1 ‚Äď wed

Flock is finally here! :)

The conference is in the hotel, so there was pretty much no travel time or chance to get lost. Registration was quick and easy and well run.

The keynote was from our Fedora Project Leader, Matt Miller. He had a number of great slides of statistics about how Fedora is doing in various ways, showing things are growing and looking pretty good.

I caught a few talks, and they were all pretty good:

10 ideas that are killing open source had some great discussion and a nice way of looking at the various reasons people don’t open source their code, and how to convince them to change their minds.

After that I went to the “hallway track” and talked with a bunch of folks about a bunch of things. It’s great talking to people in real life in high bandwith to try and explain or learn about something. It’s also of course great to talk to old and new friends.

Lunch was pretty nice there at the hotel as well, then back to catch part of Patrick’s talk about the state of Fedora Infrastructure. There were a number of questions which Patrick handled just fine. It was great to see that he knows the infrastructure setup so well and can field questions nicely.

The final talk of the day I went to was “What does Red Hat want?” Which was a great talk by¬†Denise Dumas. She went over how Red Hat culture is setup and noted that there’s even a clause for employees that if they participate in an open source project, they can put that projects needs over the company if they need to. There were a lot of questions on how to better communication from Red Hat to Fedora and vice versa. It would be great to hear more clearly from Red Hat what they really do want so the Fedora community could decide what and how to help them.

A bunch of us then went off for a team dinner at the Old Toad. It’s a nice british pub type place. The beer, food and company was all excellent.

Back at the hotel there was a games night. I didn’t manage to play any games, but I did talk to a number of folks to round out the evening.


August 13, 2015

Kevin Fenzi : Fedora 23 Alpha release day ‚Äď tuesday ‚Äď 2015-08-11

Morning came far too soon, but on the plus side there was a nice blanket of fog over the city.

The Fedora 23 Alpha release went out nice and smoothly. I merged websites branch into master, fixed a prerelease redirect we had in place and acked the announcement that was written by marketing and FPL and sent by our Release Engineering leader. There were a few minor link issues, but quickly fixed. BW climbed over the day, it seemed a pretty popular release.

We went over to java’s, a local coffee joint. I had the Aztek coffee. It was awesome! Then we wandered over to the Old Toad for some lunch. Also very nice. After that we decided to go back to java’s and hack the afternoon away on various projects.

Dinner was at Dinosaur BBQ just down the road. It was pretty excellent as well.


August 05, 2015

Kevin Fenzi : Flock 2015 Rochester coming up next week!

The summer is just flying by and next week is the 2015, North America version of Flock, this time in Rochester, NY.

Flock is the annual Fedora conference, which alternates between North American and Europe each year. It’s a great time to listen in talks what other Fedora folks are working on and have less format hallway talks with people you work with all the time but don’t live near.

Some information for this years flock:

Will be Aug 12th to the 15th in Rochester, NY.

If you can’t make it in person, there will be likely live streaming and transcripting going on.

Please Join #fedora-flock on for general conversation with other flock goers and as a place to find more information for remote attendees.

Registration and conference rate hotel room are long closed, but if you still find yourself able to make it to there in person you are welcome to attend. (You may not be able to get a t-shirt or lunches or evening events, but the conference is open to everyone!)

Evening events have been announced:

The schedule of talks:

Please thank our wonderful sponsors:

I’m looking forward to it. :)

August 04, 2015

Kevin Fenzi : This week in rawhide, the early August edition

It’s been a little while since my last rawhide post. Things have been busy.

Fedora 23 has branched off rawhide heading for it’s Alpha release hopefully next week.

A number of people have hit a common issue with rawhide recently and I thought I would talk some about that. Namely, it’s getting a rawhide release installed, ie, the install path. Once you have an install, things are usually not that bad. DNF hides broken deps from you, you get the updates you can apply and things roll right along. However, the upgrade path is a really hard one which we should fix.

First of all, in order to produce a rawhide compose at all, the following have to work:

  1. The rawhide comps xml file from has to be valid and usable. If this breaks, the compose fails and nothing is produced.
  2. The chroot that rawhide compose runs it needs to have no broken deps and is installable. This is what you get when you do a mock –init
  3. The chroot has to be able to install: “koji yum createrepo make intltool findutils mash yum-utils rsync repoview hardlink”
  4. mash (in the chroot) has to run and complete ok.

Once that happens the compose will finish (even if parts of it are broken after that).

In order to produce a boot.iso for the day:

  1. mock has to be able to –init a chroot
  2. All of: “pungi nfs-utils setarch” have to be installable and have no broken deps.
  3. lorax has to be able to run and produce images (This requires no broken deps from anything in the set of things on the boot.iso).

So, as you can tell things can (and do) break down in various places. Recently there was a rpm build that changed soname. This meant that a few of the tools in step 2 of the base case above had broken deps, so result was no rawhide compose at all for those days. Then there was a broken dep in grub2, which meant the boot.iso couldn’t be created. Then, finally today there was a broken dep in tigervnc which meant no boot.iso.

So, there’s two obvious approaches to try and make all these parts better, we could take one or both of them. :)

We could gate builds entering rawhide and run some kind of dep check on them. I have a plan for how we could do the gating part pretty simply today, but dependency checking is pretty hard. It might be we could leverage taskotron for this, but not sure.

And/or, we could just try and fail faster/quicker. Identify the packages in all the above sets we need to be working and anytime any of them change we try and build a new iso or test compose. If it fails we ring loud alarm bells. We could then even untag the offending build if it cannot be fixed quickly. Right now failures are not very visible, and they really should be so that everyone in the community can work to make sure they get fixed.

I’m running a workshop at flock next week on friday afternoon (yeah, I know, but still). I do hope interested parties will be there and we can try and make rawhide installable for everyone all the time. :)¬†

July 31, 2015

Kevin Fenzi : We moved the planet!

The Fedora planet that is. :)

The old site: will continue to work and redirect to the new site: for quite some time to come, but you should go update your feed links now while you are seeing this. :)

Why did we do this change? It’s part of us trying to make sure everything in * is using https. This allows us to set a HSTS header to get all browsers to use https with sites and also get that preseeded in some browsers. shows blog posts of the entire Fedora community and those posts may well have http links in them, or https links to self signed certs or the like, so there isn’t any way to make fully valid https, so we moved it to it’s own domain where it can happily use http until browsers no longer accept it.

This brings us one step closer to a https future. :)

July 09, 2015

Kevin Fenzi : new

We just switched the server over from a old rhel6, puppet managed instance to a new shiny rhel7 ansible managed one. :)

Hopefully everything is humming along fine, but if you run into problems with something that used to work but no longer does, please do file a ticket in the fedora infrastructure trac and we will try and fix it up.

This is also a great time to go and look over what files you have there and delete old/unneeded items. Personally, I had some fedora 12 and 13 repos that didn’t need to be around anymore. If everyone could clean up files they no longer need it would save us time and backup space.

Thanks and enjoy the new

July 06, 2015

Kevin Fenzi : gnome on wayland testing

With the recent release of gnome 3.17.3 I decided to take a look at running gnome under wayland again.

I run rawhide on my laptop, so it was pretty easy to just select gnome on wayland from gdm and give it a go.

The good news is that progress is being made from the last time I tried. For pluses:

  • No crashing or hanging. Last I tried my session hung after a short time.
  • All the extensions I use worked fine.
  • Copy/paste worked fine between apps

However, there’s still things to fix up:

  • gnome-terminal is basically unusable due to resizing itself all the time on focus in or out events. The window gets smaller and smaller and jumps back to normal, then super small again, etc. I had to run xfce4-terminal to have any kind of sanity.
  • There’s some weird spacing issues. I usually run with 2 terminals, each taking 1/2 of the screen. If I do this under gnome/X11 they fill up the entire screen, in gnome/wayland there is about a 1/2 in space around each terminal when I side tile them.
  • There is some serious focus lag. I switch applications/windows using alt-tab, and there’s sometimes a >5 second delay after I have switched before the new app has focus. This makes me send things to the old application I didn’t mean to. It may be related to me having sloppy focus enabled.
  • No ssh-agent, so you have to start one or type in your ssh passphrase a billion times.
  • Some things seemed oddly sized, like the display brightness widget. (Ie, the OSD when you hit up or down brightness). Much larger in gnome/wayland than gnome/X11

Overall things are moving forward to a bright wayland future.

June 20, 2015

Kevin Fenzi : mirrorlist/metalink woes now fixed

As many of you may have noticed, we have had some issues the last few days in Fedora Infrastructure, in particular with metalinks (the files used by dnf/yum to pull down repository data).

First, I’d like to let you know that the mirrorlists/metalink issues should all be fixed now, so please do ‘dnf clean all’ or ‘yum clean all’ and retry your operations and let us know (via a Fedora Infrastructure ticket or #fedora-admin on irc) if you still see any problems. If you changed your repo files while this issue was happening, I would strongly urge you to change it back to the metalink. It has a number of great advantages over a specific mirror or mirrorlist.

We don’t yet have a detailed root cause write up, but should have something next week. The problems seem to have been a number of issues, starting with a network problem that seems to have caused some issue with a virtualhost server that had our main vpn hub and postgresql server on it, then on to a possible mirrormanager bug where it removed the wrong alternative repomd data, breaking metalinks. Most of the metalinks were fixed last night, but we wanted to make sure and have everything back to normal before returning to normal status.

We did create some valuable tools out of this unfortunate outage, including scripting to update specific repomd.xml’s and not wait for an entire crawl of the tree (which can take many many hours), scripting to check all metalinks against the master mirror and yell when they are not completely in sync, and more.

I’d like to thank everyone for being patient while we work on this issue and especially thank all the hard working folks on the Fedora Engineering team that worked on this around the clock! I’m proud to work with such great folks.

June 07, 2015

Kevin Fenzi : dnf (or yum) and metalinks

Like yum before it, dnf also uses (by default) metalinks served by Fedora Mirrormanager mirrorlist servers. Metalinks are a great thing, but it seems like many people don’t understand them or realize what great benefits they provide, so I thought I would do this post to help.

A metalink is a xml document that includes checksums and lists of mirrors that have repodata. In Fedora you can see it in the repos provided by the fedora-repos rpm under /etc/yum.repos.d/ like fedora.repo for example:


Note that the metalink is fetched over https, meaning if you trust the ssl cert CA setup, you will know that it comes from the Fedora mirrorlist servers and hasn’t been intercepted or tampered with. You will also note that it’s passing some data to the mirrorlist server. What repo you want, what your arch is, and also your IP address.

Once yum or dnf pulls this url, it goes to mirrorlist servers. They take the request and build a metalink xml for that request based on a cache of mirrormanager data that they have.

At the top of the metalink is included the current repomd.xml file’s timestamp and checksums. Below this there may be serveral “alternate” checksums for repomd.xml from previous repodata. Any of these are considered valid. This is to allow for syncing time for mirrors. If an updates push has just finished, it means only the master mirror is likely using the very newest repomd.xml, allowing the previous one means people will still be able to use mirrors until they sync up to the new one. After some time, the older alternative repomd.xml is dropped, leaving the current one.

Next a list of mirrors is generated: Is there some mirror that has marked itself as always preferred for your ip block? Is there a mirror that has marked itself as preferred for your ASN? Is there a mirror geographically near you? What mirrors are currently marked “up to date” (the mirrorlists have a cache, but it’s refreshed every hour and the crawler marks mirrors up to date or not all the time)? It weights things by the amount of bandwith mirrors have indicated they are able to handle (so a fast mirror with small BW doesn’t get overrun by requests). It then takes the output of this query and makes a list of mirrors in a preferred order (most preferred at the top, then down). Finally the master mirrors are added at the lowest preference¬†to the list, so things should fall through to them if no other mirrors work.

Once you have the metalink, yum or dnf then tries the mirrors in preference for the repomd.xml. If it finds one with a matching checksum, it uses that mirror. In the repomd.xml is checksums for all the rest of the repodata, and thus all the packages in the repo. So, if you have the correct/valid metalink, you have the correct/valid repomd.xml, which means even if you download over ftp or rsync or http the packages you get MUST match the valid checksums. Of course additionally after you download and before you update or install the gpg keys are checked on the signed packages, but if you use the metalink you will never need to worry about a corrupt/invalid/unsigned package in the first place.

I’ve seen a number of people lately suggesting you should just replace the metalink in your repos with a direct link to a local mirror you like. This is really a bad idea except as a very short term workaround. If you do that:

  • If the mirror you are pointing to is out of date or needs to have some kind of maint on it, mirror admins can remove it or mark it not up to date and all the metalink using folks will no longer hit it, but you will.
  • You no longer have a way to tell if the metadata on that mirror has been corrupted/tampered with. True, the packages are still signed, but a evil mirror might give you known vulnerable (but signed in the past) packages. You have no way of telling. Or withholding packages with fixes from you.
  • If the mirror you are pointing to is http or ftp someone could man in the middle attack it and tamper with what you get.

Finally, now that we have mirrormanager 2 deployed we are able to start looking at some enhancements for it. Some ideas I have had:

  • A way to tell the mirrorlists you only want https (or http/https/rsync ok, but no ftp), etc. This should be pretty easy to do.
  • Consider just dropping ftp. It’s a horrible protocol, perhaps we don’t need to keep it alive anymore.
  • More comments in the metalink users get that will help us debug metalink problems. ie, how it decided what was included, etc.
  • We have already been working on lots of great improvements on the crawler to do quicker crawls and detect out of date mirrors more easily, including a canary mode (check just repomd.xml), multithreading rsync checks and more.
  • I’d like a way to tell what mirrors have iso/qcow2/raw images. We point people to mirrors to download those, but sometimes they get a otherwise updated mirror that doesn’t carry images and get a 404. If we kept track of which exact mirrors had that exact image we could direct people more accurately.
  • <your ideas here>: please file ideas at¬†

I’d like to thank¬†Pierre-YvesChibon and¬†Adrian Reber for all there recent work writing, rolling out and debugging mirrormanager2. Thanks!

May 24, 2015

Kevin Fenzi : Attention Fedora 22 prerelease users

Just a note for everyone/anyone who installed Fedora 22 from anything before the Release Candidate (RC) composes: (If you install from the final release on tuesday you are not affected of course):

Before this point the updates-testing repository was enabled and you very likely installed some things from it if you did any installs or updates after you installed. A fedora-release update came along and disabled this repo now, so you have packages from it installed, but that repo is no longer enabled. This can show up as weird issues around mismatched devel packages or other strange looking dependency issues.

Please do one of the following if you see any such issues:

1. You can re-enable updates-testing and help us test updates. See:


2. You can run a ‘dnf distro-sync’ and downgrade your packages to all the correct versions available in updates and the base repo.

April 19, 2015

Kevin Fenzi : Colorhug ALS

I ordered a colorhug ALS right after they were announced. What is that you might ask? Well, it’s a USB Automatic Light Sensor, useful for laptops that do not have a built in one. It allows software to see just how bright your environment is and adjust your laptop screen (saving lots of battery life).

The device came pretty quickly (considering it was shipped from the UK), and had a nice little instruction sheet, 1GB usb stick with docs and info and the device itself. It sticks out of the USB port a bit more than I would like, but it’s reasonably unobtrusive once it’s plugged in. Fedora already includes the colorhug-backlight package, so that was simple to install and run. Right now it’s tied to gnome, but I suspect it might not be hard to interface into other desktops.

After playing a bit with the default level, things seemed to work quite nicely. I did have a few times when I had the laptop on my lap and the sensor got obscured or put in shadow and adjusted the screen when perhaps it shouldn’t have, but it was easy to see what was happening and that didn’t occur in most normal use.

It would be cool if something like this could be added to yubikey nanos or the like (since I already have a port with one of those if it had a sensor it would be great) but not sure how practical that would be.

If your laptop doesn’t have a light sensor and battery life is important to you, I would definitely recommend picking one of these up.

See: for more info and ordering instructuions.

February 16, 2015

Kevin Fenzi : Rawhide: Beloved and vital member of the Fedora family

I thought I would post somewhat of a rebuttal to with some of my thoughts around Fedora rawhide and try and clear up some misconceptions.

There are indeed people using Rawhide day to day. I myself have for the last few years, and I know there are a number of others (based on IRC conversations and posts to the test list). Regarding the KF5 issues, this is a somewhat unstable time for KF5, as they are just now landing things and integrating them and also gcc just updated to 5.0, causing them some issues. Perhaps some of this work could have been done in a copr or the like, but sometimes it’s really hard to anticipate what will happen when you finally build in the official Fedora buildsystem. I don’t think the common answer here should be “you should expect that in rawhide”, but instead “You should understand that at times various parts of rawhide may be under more work and help them work around those issues”. I’ve definitely run into situations in the last few years where something was broken and I couldn’t use it, but I reported bugs on them and people fixed them up. In the mean time it’s always good to have alternatives.

As for IRC, rawhide issues are 100% on-topic for both the #fedora-devel and #fedora-qa channels. It’s very true that people in #fedora will tell you you are asking in the wrong place (they only support stable releases there), but if anyone in #fedora-devel or #fedora-qa tells you that, they are wrong. Also, realize that IRC is a async media. You may well have better luck filing a bug or posting to a mailing list depending on the time and nature of your question.

I tried to make sure and capture the audience for rawhide on the Fedora rawhide wiki page:

Finally, in coming months, some improvements/plans for rawhide:

  • We hope to finish our automated signing for rawhide. Once this is in place it would allow us to know that everything is signed in rawhide, and as a bonus would give us a gating point via koji tags (ie, right now builds just go to f22 tag, but after we land this things will go to a f22-pending tag or the like, then get signed or have other things run on them).
  • Once we have that in place we have a gating point and we can hook automated qa type things into it. Basic problems could be detected and the build never pushed out.
  • I’d personally like to revist the idea of shipping previous versions in rawhide (ie, foo-1.0-1.fc23 and foo-1.0-2.fc23) to allow for easier downgrades. Will need to try and convince the rest of releng to do that however, and it would make things larger.
  • There’s likely to be a mass rebuild in rawhide in the next few weeks. Both for the gcc5 change and also for hardened_build by default.

If anyone has questions about rawhide, feel free to ask me in #fedora-devel on irc, the devel list or via email, happy to help out.

January 25, 2015

Kevin Fenzi : ansible idempotence tips

So, what better to do on a lazy sunday than clean up some of the Fedora Infrastructure ansible playbooks to be idempotent. I ran into several common cases in our playbooks for something to not be idempotent, so I thought I would share them in case others run into the same issues.

First, for those of you wondering what ‘idempotence’ is, wikipedia describes it thusly: “the property of certain operations in mathematics and¬†computer science, that can be applied multiple times without changing the result beyond the initial application”. More practically in the ansible world it means after 1 run of a playbook to set things to the desired state, further runs of the same playbook should result in 0 changes.

In Fedora Infrastructure we run a ansible-playbook run of each of our host or group playbooks with ‘–check’ and ‘–diff’ every night from cron. Then, a handy report is mailed to our sysadmin-logs members showing all the playbooks that showed something would have changed. In the ideal state this report would be empty, but there’s various reasons things show up in there.

Of course the most common reason a playbook would show up in our check/diff report is that something actually changed and the playbook has simply not yet been run on the host(s). This could result from someone checking in a change to git and not running the playbook, some variable or global change was checked in and the person doing it didn’t realize they needed to run this particular playbook, someone made a manual change on the host (BAD sysadmin!), or perhaps the host was not reachable or up when the playbook was last run. This is the easiest case to fix up: just run the playbook(s) and everything gets back to the steady state.

Another common case I see is when a file is changed multiple times in the same playbook run. This can result from a simple copy paste error, or a conditional that isn’t correct (a ‘when:’ in ansible terms), or having the same template or file in multiple roles or directories and not realizing that you are setting it in two places. This can be fixed by only changing the file once, either by collapsing the task into one place, or adding conditionals so it only applies once on each host. Sometimes in rare cases you DO need to change a file multiple times (for example, put a admin config in place, run some admin commands, put the normal user config in place). In these cases you can use ansibles “when_changed” to just tell it this isn’t really a change and don’t bother marking itself as well. Do be careful using this.

Another case I see a good deal is when you run a command or shell (which is fine, –check simply ignores them because they always change) but register a variable thats used later in the playbook. In this case there’s a problem when running with –check because the command/shell is never run, the variable isn’t set and when you try and use it the playbook fails. In these cases you can use: “always_run: yes” and the command/shell will run. Of course you may also want to add a “when_changed:” so it doesn’t always show as changed.

I reduced our non idempotent playbooks today a good deal, but there’s still more work to be done. I do look forward to the day the report is empty, and I don’t think it will be all that far off. ūüėČ

January 19, 2015

Kevin Fenzi : Fedora Infrastructure DB dumps

In Fedora Infrastructure, all our applications are Free software. It’s one of our base requirements, allowing anyone out there to examine source, improve or modify things. Sometimes, just having the source of an application isn’t enough, you need the raw data to figure out some issue or generate some metric or support a theory.

Happily in some cases, our applications databases don’t contain any sensitive data. In those cases we provide dumps of these databases for any interested parties.

Currently available:

datanommer – This is the application that consumes all messages on our fedmsg bus. If there were messages in the past, it would have recorded them. You want this db if you want to look at fedmsg metrics or information.

fedoratagger – This application allows users to tag and rate packages in Fedora. This db is usefull if you want to look at application ratings or tags.

pkgdb2 – This is the application that controls access to Fedora packages for maintainers. You want this db if you want to look at stats about packages or packages related to acls.

And finally, new today: koji. This is the db that is used by It’s got information about all official builds back to the dark ages. You want this if you want to look at history about builds or buildroots or old fedora release versions.

All of these are available at:

If you find these useful or see other applications we run where there’s no sensitive data stored in the db, please let us know!

January 12, 2015

Kevin Fenzi : Fedora Xfce: Versions and other questions answered

In the last few months I’ve seen several questions around Xfce in Fedora, so I thought I would share some thoughts around versions and other questions related to Fedora’s Xfce efforts. Note that I am only one of several maintainers of Fedora Xfce packages, so these thoughts are my own (Although I think my co-maintainers share them too).

The current stable released version of Xfce is 4.10, which was released in April of 2012. A rough plan was made for a 4.12 release, but Xfce is an all volunteer project (as far as I know, no one is paid to work on it full time), so timelines have slipped. There’s a number of components with 4.11 development releases out, but currently no hard plan for a 4.12 release. There are also from time to time 4.10.x bugfix releases for various components and plugins.

Several people have asked why we don’t land the Xfce 4.11 builds in rawhide. The problem is that since there’s no timeline for 4.12 to be released, we could easily end up shipping those in the next stable Fedora release (Fedora releases branch off rawhide). While in general these builds work fine, they still aren’t good to ship out in a stable release. Since these are unstable development releases, Xfce developers could change the way the interface works, drop or add new functionality or otherwise make end user visible changes. While these are fine between Fedora releases (people set aside time and realize that they are going to have to learn the new setup), they would not be acceptable to change in a existing stable release. This sort of thing is an excellent use for the Fedora copr system, and indeed we have a 4.11 copr ready and waiting for those willing to use it.

I’ve also seen from time to time lately people saying that “Xfce is dead”. This is completely untrue. There’s still a fair bit of development going on. There’s bugfixes, new releases and lists activity. True there’s not a 4.12 release in sight yet, but I’m not sure thats really such a bad thing.

The Fedora Xfce spin is of course alive and well. We did have to finally give up on limiting to 700MB size with Fedora 21. It was just too difficult to trim things and have a good experience. We re-targeted for 1GB usb, which hopefully most everyone can find these days. :)

Xfce in EPEL continues to be around for EPEL users: 4.6 in EPEL5, 4.8 in EPEL6 and 4.10 in EPEL7.

There have also been some questions around Xfce and HIDPI displays. The support isn’t ideal at all, but Xfce an work fine with some tweaking on a HIDPI display. Here (on a 3200×1800 display) I adjust the dpi in Settings -> appearance -> fonts. This doesn’t get set completely right on login (sometimes some autostarted apps don’t see the change and need to be restarted). Also, you get very small window decorations (hard to resize windows). Otherwise it looks fine and works well. Ideally once Xfce goes to gtk3 they can take advantage of the HIDPI support there.

December 10, 2014

Kevin Fenzi : Fedora Infrastructure release day retrospective

Fedora 21 was released yesterday. (If you haven’t already, go get it: )

This release was not as smooth for infrastructure as previous releases have been, for which I apologize.

Here’s what happened: For the last few weeks we had been seeing sporadic slowdowns in the bodhi application, but had been unable to isolate what was causing them. This last week was the Fedora Infrastructure Mirrormanager 2 / Ansible FAD, and there we added some more debugging in, but still couldn’t see where the problem was. It wasn’t in bodhi itself, but somewhere in it’s integration with the authentication system and getting to that via proxy01 (our main datacenter proxy). Proxy01 seemed busier than usual, but it gets a lot of traffic anyhow. We bumped memory up on it to make sure it could better cope with release day.

Then, release day: proxy02 (a server in england) started being unable to cope with load and we removed it from DNS. Then, proxy01 started having problems. Since most services were slow in any case, we updated our status page that it was release day and to expect slowdowns. Most services (aside bodhi) were actually up and fine, just slower than normal. Some folks took this to mean we were completely down, but this was not the case. Next release we probibly will make a special banner telling people it’s release day and to expect things to be slow, but up and all working.

Finally this morning Patrick discovered a problem in our DNS setup. It had been there all along, but the amount of traffic we had been seeing in the last few weeks and especially on release day made it much worse: There were only proxy02 and proxy01 available for EU dns. This means that EU folks would always get those 2 proxies, and with one out, always get that single one. There were 2 other proxies that should have been in DNS for EU, but were not. We quickly added them, added proxy02 back in and things have been very quiet since then. With proxy01 not having to handle all of the EU traffic, bodhi was happy again and with 2 more proxies closer to EU, EU users should be happy again. Many thanks to Patrick for tracking this down finally.

Sorry for the slowdowns and issues on release day. Everything should be back to normal now and we should not have this problem on the next release.

In the last week, our master mirrors have pushed out around 50TB of data. Not bad.

December 09, 2014

Kevin Fenzi : Mirrormanager and ansible FAD (day 5)

Last day to get things done. Tomorrow folks head home (on release day morning even).

I cleaned up the postgres server ansible role we had to handle things more nicely. Ralph kept slugging away at the proxy playbooks. Patrick worked on single sign out, then openid for bodhi. Pingou got mirrormanager2 rpms built and playbooks setup to install them in staging. I looked over keepalived, and it should work for our limited needs.

So, moving forward, we need to:

  • Finish getting proxy setup in staging and test it a bunch before rolling out to production.
  • Finish mirrormanager2 deployment in stg and test a bunch and roll to production.
  • Next week likely we will schedule some outages and migrate some machines that need downtime for that… fas servers, db servers, virthosts, etc.
  • mailman3 and bodhi2 need to land somewhat soonish before f22 alpha so we can not have to port the old bodhi1/mailman2 servers to ansible.
  • We need to finish koji in staging and finish keepalived.
  • Some more misc playbooks need doing.

We then got mirrormanager2 frontend, backend and crawler working in stg. Still need mirrorlists and a lot more testing. Then we helped out with some release setup for redirects and such for tomorrow.

Tomorrow we all fly back home. It’s been a great FAD, we got a ton of things done… I’m looking forward to the next one. :)

December 08, 2014

Kevin Fenzi : Mirrormanager and ansible FAD (day 4)

Two more days of FAD to go. Today and tomorrow and then folks head home (on Fedora 21 release day).

Today we really worked on figuring out our proxy setup in ansible. We divided up some things and got to work. I made a proxy02.stg instance that we could configure with ansible and then compare against the proxy01.stg made by puppet. Ralph worked on the templates for the websites and Pingou worked on varnish and haproxy ansible roles. Patrick worked on migrating all our nameservers from puppet to ansible. Luke worked on test coverage for mailman 2 and got it really going on the mirrorlists setup. Smooge worked on people and planet ansible migrating. David finished getting fedimg all working in stg.

Later in the afternoon we all took a break to go up to Duram and visit Seth Vidals ghost bike. It was a somber occasion. I still very much miss him. ;(

After that we met up for dinner with Tom Callaway and Ruth, then everyone headed to sleep.

Last day tomorrow. Hope we can get the proxy setup mostly done. :)


December 07, 2014

Kevin Fenzi : Mirrormanager and ansible FAD (day 3)

Today we started in looking at the big picture where we are in our migration of ansible and what things we need to do to complete it.

There’s 67 hosts still in puppet right now. 27 are all ready to move, we just need to schedule downtime for them and do them. 27 more just need some playbooks written then migrated. The rest are special in one way or another. lockbox01 needs to be the last one we do as it’s the puppet master. The proxy hosts are tracky as they have a LOT of apache config on them. We discussed various ways of moving forward on it.

I started working on landing the playbooks for fas (fedora account system) which turned out to be a bit tricky, but I got our staging instance going. We can do the production ones as soon as we schedule a downtime. Pingou made several playbooks for hosts and got them ready to go. Patrick got the nameservers sorted out and ready to move.

Smooge got all our non logging hosts logging! This was great… of course as reward we get some error messages from a host that he then had to go and fix up, but it’s great to have them all logging away correctly again.

There was some work by Ralph and David to get fedimg all humming along in ansible in staging.

Luke got most of the work to get atomic composes going with bodhi updates pushes setup. Hopefully we can land that next week after release.

We talked about our existing proxy setup (httpd, varnish, haproxy, applications, memcached) and determined that we should change how we are doing memcached. Instead of some shared instances, just give applications a local one that they only talk to. This eliminates the external ones as a point of failure, and should help all the apps.

Tomorrow more ansible. Hopefully we can at least prototype the proxy setup.

December 06, 2014

Kevin Fenzi : Mirrormanager and ansible FAD (day 2)

The second day of the FAD was also devoted to Mirrormanager work (as we only have Matt Domsch around today).

I created 4 new staging instances for us: a mirrorlist server, a frontend (to run the web frontend), a backend ( to see new content, make mirrorlists, etc) and a crawler (to crawl mirrors and find out of date ones). Took a bit of work to get the production nfs mount setup (read only of course) on the new backend.

Next was fun with databases for me, I copied the production mirrormanager 1 db over to staging and got staging mirrormanager 1 working. (It had a config issue). Then, I made a mirrormanager 2 db from the mirrormanager 1 dump and pingou took that and migrated it to the mm2 schema.

Smooge looked over our stats and found some strange issues, for which we added a bunch of logging to try and isolate what was going on and what people were really getting.

There was a lot of discussion around atomic trees and how to add them properly to mirrormanager.

Then, off to a quick run by the Red Hat holiday party that happened to be going on this evening and off to dinner with Greg from ansible. It was awesome that the ansible folks took us out for dinner, they are great and you should all be using ansible. ;)

Tomorrow on to the ansible part of our hackfest.



December 05, 2014

Kevin Fenzi : Mirrormanager and ansible FAD (days 0 and 1)

On Wed afternoon I flew out to Raleigh for our Mirrormanager2 and Ansible FAD. Luckily this time I got a direct flight (from denver) so no connections to miss. My flight ended up being about 40min late in the end (possibly due to fog, they never said) and I got into the hotel after midnight and crashed.

A bit of background on this FAD. We had planned it many months ago hoping it would be after the Fedora 21 release (alas, due to slips this is not the case). Also, we had hoped to work on FAS3 instead of ansible, but we couldn’t pull all the folks who work on FAS3 in, so we decided to use the second half of the FAD for ansible work that we would really like to get done sooner rather than later. Mirrormanager2 is a re-write of our aging mirrormanager1 setup and we wanted a chance to work with the author of MirrorManager1 (Matt Domsch) to get historical reasoning for choices and where we can redo design.

Thursday was the first day of the FAD (which is hosted over at the Red Hat Tower in downtown Raleigh). We all gathered and did a bit of overview on what was implememented so far on mirrormanager2 (basically the user interface, db and backend) and what we still needed to work on (all the scripts, the crawler, the mirrorlists server, etc).

Then, us sysadmin types stepped away for a few hours for some meetings with folks we work with in Red Hat IT. Some great discussion about plans for datacenters and making things better for our Fedora users via Red Hat infrastructure we can leverage.

Finally we got back to the FAD and went over more information from the past and code started to flow.

We ended the day by going out to a irish pub for dinner, then back to the hotel for some much needed sleep.

November 30, 2014

Kevin Fenzi : 30 days! whew!

Well, looks like I managed to do it: a blog post every day for 30days (all of november). :)

I did have a few times, especially toward the end where I was at a loss for topics. I didn’t want to repeat myself or go over the same things, but yet wanted them to be interesting.

I hope folks found them interesting, thought provoking, or at least amusing.

I think coming out of this I will probibly try and blog more, but will cut back from daily for a while at least.

November 29, 2014

Kevin Fenzi : Some handy systemd/journal tips

I thought I would share some nifty little systemd/journal commands I have run into in the past year that are handy.

  • easy way to make sure something restarts on fail: mkdir /etc/systemd/systemd/nameofthing.conf.d then make a nameofthing-override.conf in that dir with:¬†[Service]
  • systemd-delta – Gives you the list of things that have changed from the ‘default’ units. Ones overridden, diff of any that were edited, etc.
  • journalctl –list-boots – Gives you a list of boots (with also the number you can use with the next command) and times/dates.
  • journalctl -b N – where N is the boot you want information from. I use this all the time with -1 or -2 to see how things changed in the previous 2 boots.
  • systemctl status PID – you can pass this any PID and it will give you the status of the unit that started it/controls it. Very handy to see what some random process is a part of.
  • systemctl suspend -i – Ignore any inhibitors and just suspend. This is like a –force, so be careful if you are in the middle of something like a package update.
  • systemd-inhibit –list – shows you all suspend inhibitions and who asked for them.
  • journalctl –disk-usage – show usage of journal logs on disk. You can tune them then in /etc/systemd/journald.conf

Hope you all find some of these as handy as I did. ;)

November 28, 2014

Kevin Fenzi : Fedora 21 RC1

For once we managed to make an RC before the last minute for a release. ;)

So, please if you have a few minutes this weekend and want to help us out, go and test the Fedora 21 Release Candidate 1.

  • See the test announce list post:¬† for lots of links and more information.
  • See #fedora-qa on to discuss findings with other QA folks.
  • See the blockerbugs application to see where things stand:¬†

Fedora 21 looks like it’s going to be a nice stable release. ;)

November 27, 2014

Kevin Fenzi : Thanks

I’ll take today’s blog post as a chance to co-opt a silly holiday and turn it into a chance to be thankful to some folks:

A big thanks to my co-workers: It’s awesome working with all of you to make Fedora and the world a better place.

A big thanks to the Fedora Community: I’ve met many wonderful¬†people via Fedora and it’s great anytime I interact with them. The Fedora community is like a large extended family. (including crazy uncles, great cooks, world travelers and all in between).

A thanks to my company: Paying me to work on something I love working on and having a open source culture that makes me welcome.

And finally of course a thank you to my family: My Girl and my dogs.

Hope everyone out there has a chance to think and be thankfull for the people they have in their life.

November 26, 2014

Kevin Fenzi : How packages are added to EPEL-7

I’ve seen a number of people ask things like: “Foo is in EPEL-6, why isn’t it in EPEL-7?” so I thought I would share a detailed answer:

In Fedora, all new releases are branched off rawhide, the ever moving forward release. This means every Fedora release starts out with a version of every package that was in rawhide at the time. This works great for Fedora, but EPEL is just a collection of add on packages, not a full distribution. New EPEL branches/releases appear only after the new RHEL version beta appears. Between RHEL6 and RHEL7 this was something like 3 years. A LOT can happen in 3 years in the open source world. Packages that were in EPEL-6, may make no sense in EPEL-7 (the base things they depend on may no longer exist in the base os, the maintainer may not want to commit to maintaining them there, they may have been renamed, etc). For these reasons, packages are added to EPEL-7 only on actual request of a maintainer. There was no mass branching of all packages in EPEL-6. They are different things and have different package needs.

Some examples: EPEL-6 has Xfce 4.8 in it. By the time EPEL-7 was created, Xfce 4.10 was out. Packages like ‘Terminal’ were re-named xfce4-terminal, new base library packages were added, etc. If we mass branched from EPEL-6 there would have been a bunch of packages we would immediately have to just remove because they could never build or exist.

If you want to get some package in EPEL-7 thats not currently there, file a bug on the package and ask that it be added to EPEL-7. If the maintainer doesn’t respond in a week and you are yourself a package maintainer, you can request the branch. If you aren’t you can ask on the epel-devel list for anyone interested in maintaining it for you.

Hope that clarifies things somewhat…

November 24, 2014

Kevin Fenzi : Fedora 21 weekend upgrading

With the release of a new Fedora getting close (Fedora 21 is currently scheduled for release December 9th), I figured this last weekend was a great time to upgrade the ‘production’ side of my home machines to Fedora 21.

I have 2 pretty important machines. One is my production virthost. It also acts as my firewall and router for all my networks (external, wireless, the test network to the test virthost, the internal network, etc). The second is a vm running on that virthost that is my mail hub, vpn end point and other important functions.

Of course the first thing to do before any major upgrade is to check backups. I run nightly backups with rdiff-backup to my main virthost, but also I have a encrypted external drive I sync those updates to from time to time, and off site where I can upload them too. I made sure everything was synced up and looked good.

The official way to upgrade between Fedora releases is ‘fedup’. However, my main virthost is a bit of a weird setup with raid and encryption and such, so I usually just do a yum update on it, and since I am doing that on the virthost, I usually just do the same on the vm as well. This time (like the last few upgrades) everything went very smoothly. No conflicts or issues with the upgrade transactions on either host, took a bit of time to do them, but finished with no issues.

Then, a quick reboot and I was up in Fedora 21. Well, mostly. I ran into a issue with my ‘internal’ bridge. I had not listed the HWADDR for the network interface that should be on the internal bridge, and Fedora 20’s NetworkManager was fine with figuring it out, but for some reason Fedora 21’s didn’t get it added right. I had to set the HWADDR and restart a number of things. I thne ran into a minor issue with my ipv6 tunnel, but got it sorted in short order.

So, at least for me, Fedora 21 upgrades were as easy as they have always been.


November 23, 2014

Kevin Fenzi : Some simple ansible tricks

ansible is very handy for one-off tasks like gathering quick info or running commands over a group, so I thought I would share some of those today:

Filtering on facts:

  • ansible -m setup -a ‘filter=ansible_distribution_version’ -o all (This runs the setup module on all hosts and filters on the ansible_distribution_version, so ’20’ for Fedora 20, etc. Note also that we are passing -o which means to put all the information on one line, this is very handy for grepping output).
  • ansible -m setup -a ‘filter=ansible_selinux*’ all (Runs the setup module and filters out the selinux info. Note that this is a array with more info, like config and runtime, etc)

Looking for specific information:

  • ansible -a ‘ps aux | grep puppet’ -m shell proxies (This runs the command on all hosts in the proxies group and shows output. Note that here we are using the ‘shell’ module instead of the default command module. This is because command doesn’t understand pipes and such, so you need a full shell module.)
  • ansible -m virt -a ‘command=freemem’ virthost (This uses the virt module and shows free memory on the virthost)

Docs on ansible modules:

  • Of course there’s google and duckduckgo and the ansible project web pages, but if you want to look up something quickly about an ansible module, use ‘ansible-doc’ command line. Just ‘ansible-doc shell’ for example to get all the info about the shell module, it’s arguments, etc.

It’s super easy to get started using ansible this way. All you have to do is install it on your control host and have ssh connectivity to the hosts you want to run things on.

November 22, 2014

Kevin Fenzi : Some Fedora Infrastructure stats ‚Äď 2014-11-22

Some possibly interesting stats for folks about Fedora’s Infrastructure (collected this morning by ansible, look for a post about that soon)

  • Amusingly, we currently have exactly 400 “instances”. Thats hardware servers, vm’s, cloud instances, etc. There’s more cloud instances that are transitory that aren’t covered in this as well.
  • 2 instances are down (1’s a arm SOC with a bad drive, 1 is a persistent cloud instance we haven’t spun up yet).
  • OS breakdown: 154 are Fedora 20, 135 are RHEL 6, 105 are RHEL 7, 3 are Fedora 19, 3 are Fedora 21 (Thats right, we have more Fedora instances than RHEL anymore)
  • Selinux breakdown: 229 are enforcing, 166 permissive, and 3 are disabled (one of those needs to be, the other 2 should get fixed)
  • Hosts still in our old puppet setup: 67 (expect this to drop after our FAD in early december)

Expect RHEL6 numbers to go down to 0 over the next few months. We are already almost at 50% RHEL7, and so far it’s been great to work with.

Hopefully early next year we can finally retire puppet as well.

UPDATE: I meant to say we had more Fedora Instances than any other single OS. (Seeing RHEL6 and RHEL7 as different). Of course total RHEL is still higher than Fedora instances.