October 03, 2017

Kevin Fenzi : ansible retired from epel7


Just a note for anyone looking for ansible in epel7.

It’s been retired there because with the release of RHEL 7.4 it’s now
int the rhel-extras channel.


Accordingly, you can get ansible now from rhel extras channel, or CentOS
extras repo.

You can also get ansible rpms now from

Note that ansible continues to be available from epel6.

September 03, 2017

Kevin Fenzi : Flock2017: day 4 and home

The last day of Flock started even earlier with a team breakfast. The Fedora Engineering team (which I am on) really only gets together once a year at flock, so we gathered up for a team breakfast to catch up on things and see each other in person. Nice breakfast at the Egg and I.

Then back to the conference where folks were showing off what they did or telling people how their sessions went. I think a number of folks had to depart before this, so I am not sure how well it worked, but it was nice to hear how sessions I was unable to attend went.

Next up was the weekly FESCo meeting. Since most of us were at flock, we gathered in the hotel lobby with laptops and had a IRC meeting while all together. The meeting went pretty quickly and then it was time to catch a ride to the airport. The ride was smooth and trouble free until we got close to the airport. Then an accident made the last few miles take an hour or so. I had time to catch a sandwitch and a beer before my flight (did you know Sam Adams makes a nitro vanilla porter?). Flight was long, but uneventfull and then just the hour or so drive home. Finally got home around midnight local time.

Some final thoughts:

  • I’m not sure the recap works on the last day. Might be better the end of the previous day so everyone is still around?
  • I’m not sure I feel like I got much “do”in done. I definitely got a lot of nice discussions and learned some very nice things, but not sure how much I really did while there. Perhaps it would be good if we want to do that focus again to have workshops/sessions decide up front what they want to do and then report on if they did it or more than that or less than that. Have a concrete goal to work on. It’s so easy to get distracted.
  • I met a number of new folks or people I only knew via IRC, but perhaps we could do something to mix things up more and get people to meet up with new people instead of talking only to old friends. Perhaps some game/badge you can only get by scanning people’s badges or a event where random teams are made for something? Or assign everyone a conference buddy that is the oldest fas account holder with the newest and so on.

Anyhow, I had a great time as always. Many kudos to the flock team and all the attendees! +1 would flock again. 😉

Kevin Fenzi : Flock2017: day 3

Day 3 started again at 8:30, so nice and early.

First up was Mike McGrath giving the “What does Red Hat want?” talk. He had a very interesting (and likely right on the money) concept: Fedora lives in the space right after the very early adopters of something and before the bulk of people start adopting something. In other words, if we aren’t adopting new things, stablizing them a bit and then moving on, we are not in the right place. We don’t want something too soon, it has to be starting to gain adoption, and we also don’t want something thats too stable and going on to the masses because it’s boring. Excellent food for thought.

Next was my workshop (luckily nearby Mikes talk). The room was pretty packed, which was great. I first went over plans for the rest of this year, next year and beyond as I see them. I’d like to see us get more apps in openshift. A lot of the simple ones shouldn’t be hard at all. Then, I’d like to look at leveraging some cloud providers some more. Since we have openshift and apps in it we could look at migrating some of them to cloud providers and reduce our dependency on our hardware in our main datacenter. This would also allow us to spread out more and handle more traffic and load more easily. I’d like to also work more with other open source community infrastructures: in particular CentOS folks, who we already talked to about running our jenkins instance and setting up better integration between pagure and centosci. We share a lot of needs and there’s no reason we can’t also share resources to meet those needs. I’d also like to get more metrics than we have these days (about bwandwith used, cpu consumed, etc… which moving to a cloud provider will give us). There will of course be lots of things where that doesn’t make sense (like the buildsystem/releng machines), but it should make us a lot more flexable.

After going over that I opened things up to questions and ideas. I was hoping for some more crazy off the wall ones, but we did get a number of good ones. We talked jenkins and CI, we talked about metrics and things that do that, monitoring (if we wanted to move away from our current setup), how we might best make use of modules (perhaps have infra specific ones we use for our apps, which makes us not be tied to a specific Fedora release), making docs or other artifacts via openshift, or even reworking how packages are built to use it. Then we broke up to hack on various things and let other folks have some time back if there were things they needed to do.

Then after a bit of hallway/fixing compose issues time, it was lunch again.

After lunch was a general talk on modularity. I think I had a handle on things before this, but it was good to see the current state and there were some great questions from the audience.

After that it was back to the hallway to discuss things with folks and work on composes and other fires.

There was no planned dinner this evening, so went out and had a relaxing dinner with some folks at a local pasta place.

Kevin Fenzi : Flock2017: day 2

Day two of flock started too early for non morning people (8:30am), so I was up around 7am to shower, get some breakfast (at the hotel resturant, it was ok) and get over to the conference rooms.

There were a ton of interesting talks in the morning, but we were also having some rawhide and branched compose issues, so I opted to go to the hallway track and work on that and discuss all kinds of things with all kinds of people. It was great to be able to have full bandwith discussions with someone about something.

Had lunch outside on the deck. It was very nice there, but a bit hot and bright for a bunch of nerds. Lunch discussion was all over the place, including new arm laptop (pinebook) and Fedora builders and how various things were made to laptops and support from various vendors.

Then it was time to rush to the EPEL for the future talk from Smooge. The room was standing room only, so perhaps people are realizing how popular and important EPEL is. Smooge presented some proposals for dealing with various problems in EPEL, but I had to go and get a server that crashed back up and running, so I missed the discussion part of this. Hopefully we will see more discussion in EPEL meetings and on the epel-devel list.

Next up was Ricky’s talk about our recent openshift deployment. He went over how we are doing things and the various caveats and issues around any new tech. The slot was only 30min, but he covered things very well.

Next (in the same room) was Jeremy’s talk about the future of fedmsg. I was a bit confused here I admit (sorry Jeremy!) about the scope and nature of changes. Really it was a bunch of things we could (or decide not to) change. I somehow thought of it as all tied together. I am sure this will also have a bunch more discussion soon and we will see what we all want to do here.

I then went back to the hallway to work on compose issues and some urgent tickets.

Then it was time for the evening event: Wackenhammers arcade. It was a pretty amusing old time arcade with beer and food from a food truck. Got some nice visiting in here with the Fedora kernel team and tons of other folks.

August 30, 2017

Kevin Fenzi : Flock2017: day 1

Day 1 started out way too early. 7am (which feels like 4am for me!).

After a quick breakfast downstairs, and run by registration I made it to the ballroom just in time for opening announcements then our Glorious Fedora Project Leaders “State of the Fedora” talk. The room was pretty packed actually, and as allways Matt had some pretty charts and graphs.

I found the premise interesting: That Fedora lives on the innovation curve at the very front left. A small bit behind the very bleeding edge, and back to where it starts getting boring. If thats true (and it seems likely), then we always need to be pulling in the very new stuff from one side and integrating it. So the state of things now is a bit on fire, but in a good way.

After that was the group picture (interested to see how it turned out this time) and then everyone running a session got a minute (or less) to pitch their session and try and get folks interested in going to them.

Then, on to sessions:

  • Lunch: Not bad. Sandwitches on the deck looking over the resorts gold course.
  • Pagure hackfest: There were not many of us, but we made some progress. Pingou and I fixed the incoming email processing on pagure.io so emails can comment.
  • State of the Fedora Server: A quick overview of server and what is going on. Amazingly we ran out of time. If folks still have more questions, please ask on the server list.
  • Fedora Legal: this is why I drink: a great session as always from Spot.
  • Lots of “hallway” discussion and talk on all kinds of topics.

Then time for games night + dinner (pizza) and beer. The evening ended with us hanging out in the hotel bar and chatting on all sort of topics.


Kevin Fenzi : Flock2017: day 0 (and -1)

It’s almost time for flock 2017. This year it’s in Cape Cod. Also this year I am heading there not from Denver, but from Salem, OR.

My best option for flights this year was a red-eye from Portland to Boston, then the bus out to Hyannis. It was a long, but uneventfull journey: 10:45pm out of Portland, 7am arrival at Boston, and 10am arrival at the Hyannis bus station.

I then started the walk (about a mile) to the hotel, but turns out I ran into some Fedorans I know, and they had a car, so I had a nice ride to the hotel.

The hotel is pretty large, but it’s pretty weird to get to/from our rooms as you have to go up and down stairs and around. It’s definitely past it’s glory days, but still seems reasonable to me.

After making the hotel, some hacking in the lobby and visiting and then lunch and dinner at some local main street places (burger and beer and then italian for dinner). After the long day, I was asleep before 10pm. 🙂

Tomorrow: Day 1!

August 26, 2017

Kevin Fenzi : Public Service Announcement: Fedora 26 and delta rpms

Just a quick public service announcement:

Delta rpms (The much smaller files that contain just the changes from one package version to another) are currently not working for Fedora 26 (all other releases should be working fine).

They actually were not fully working before, but the problem wasn’t detected fully until a few weeks after the release. The issue is that we now push out a bunch of alternative arch builds in the same updates as before and we place those in another location on the master mirrors. This means that mash and bodhi (The things that make the updates repos/updates) need to know where to look for the older package rpms in order to make delta rpms against the new ones, but currently we just pass it one location for everything.

We hope to fix this soon, but if you don’t see delta rpms in F26 (yet) this is why.

EDITED: As of 2016-09-01 we have F26 delta rpms fixed. Everything should be back to normal. Thanks for your patience.

August 25, 2017

Kevin Fenzi : Rawhide notes from the trail, the long road edition (2017-08-24)

Hey look, it’s been exactly 30 days since my last rawhide post. What a ride it’s been this last month:

The mass rebuild was delayed a bit due to tooling issues, then releng availablity, but finally happened around the beginning of the month. Then, sadly, another one was called for due to a binutils issue on ppc64. Luckily this was just all the archfull packages (the noarch ones were unaffected and could be reused from the first rebuild).

There have been a bunch of issues plaguing composes, including, but not limited to:

  • rdma-core replacing a number of other packages and needing lorax to adjust to it pulling perl into the base images.
  • rdma-core not building for armv7 and lorax needing to not try and install it on that platform.
  • Storage backend issues. We are now thinking it’s some interaction between Linux kernel and NFSv4.1. But the slowness has made composes and new build repos very slow sometimes.
  • New rpm that switched to sha256 headers breaking signing because old rpm didn’t correectly read the new headers (in an update this has been fixed now).
  • New rpm needing a bunch of things that linked to it rebuilt.
  • After the mass rebuild, all f28 needed signing with the new f28 key. (took a few days, but done).

Due to all that we have had many fewer actual finished composes than we normally had in the past (now that pungi will fail a compose if any required part fails). I’m not sure if thats making things better or worse, but hopefully they will get better. It does mean when we do get a compose there’s a better chance of it being usable, but also when we don’t theres no new repos for people to use day to day.

And tons of new things landing:

  • Bunch of changes to debuginfo. Now there are split debuginfo per subpackage and sources are all in a debugsource package.
  • Kerberos KCM is setup by default.
  • Tons of new versions of things: rpm, glibc, gcc, kernel, etc.

F27 branched off rawhide to go it’s way to release, and f24 went end of life (We thank it for it’s long and valient service).

Next week is flock (Fedora’s big yearly conference). I look forward to seeing my Fedora family and working on tons and tons of things. (This years theme is more ‘do’ than ‘talk’). Hope to see many of you there!

July 24, 2017

Kevin Fenzi : rawhide notes from the trail, the 2017-07-24 edition

Greetings! Once again it’s been a long spell since one of these posts, so lets jump right in and rope that calf.

Rawhide has been marching along to the next branching point, when Fedora 27 will branch off for it’s release. There’s a mass rebuild that should be happening very very soon now, there was a bit of delay while tools were all sorted out. Look for a very very big rawhide update soon once that mass rebuild finishes.

All the alternative arches are now in the same koji. We finally got s390x added in and going. Initally it was only 5 builders with 4GB ram, so things got stopped up from time to time, as all it would take was 5 long builds to show up and everything else waited for them. A few weeks ago we increased things to 15 builders and 8GB of ram each so hopefully that will keep up with builds as we go. We will see what the mass rebuild looks like with all the alternative arches included this time. I suspect it’s going to take longer than before, but not sure how much longer.

Thanks to Patrick you may also notice that uploading sources to our lookaside cache is no longer uploading them twice. Instead it’s using the kerberos cache to authenticate on the first go so it just needs to upload once.

We have missed some composes in the last few months (once for quite a while), and this is due to the compose process now failing the compose if all the release blocking deliverables are not there. On one hand thats nice because it means we always have those if the compose finishes, on the other it’s not so nice because if it fails we need to go fix whatever breakage is there.

Look for a new note soon, and ride safe out there!

May 15, 2017

Kevin Fenzi : CI and infrastructure hackfest 2017

I was hoping to write a bit of a summary for each day of the CI and Infrastructure hackfest last week, but there just wasn’t enough time to sit down and write up blog posts. Each day we started bright and early gathering in the hotel lobby at 8am, got to the Red Hat tower, got some breakfast and got to work until 5 or 6 and then we went and got dinner and back to the hotel to do it all over the next day. We definitely had some excellent discussions and got a lot of work done.

First a few thank yous: Red Hat provided us the use of the fittingly named “Fedora” room on the 9th floor to work in. It was a perfect size for our group. I think once we had to scare up an extra chair, but usually we fit just fine. OSAS (Red Hats Open Source And Standards group) funded travel and lodging for folks. Paul Frields (stickster) handled logistics and tried to keep us all scheduled, on track and in general cat herded in the right direction.

We had 3 BIG goals for this hackfest and a bunch of little ones. I think we did great on all counts. First on larger goals:

  • Monday we got a detailed dump of information about our authentication systems, past, present and future. Both from a perspective of sysadmins wanting to manage and fix things and application developers wanting their app to authenticate right. The high level overview is that we have FAS2 currently as the source of all our authentication knowledge. It syncs (one way) to freeipa (a pair of replicated masters). This sync can sometimes fail, so we now know more about what to do in those cases from the sysadmin side. Then we have ipsilon that manages openid, openid-connect and (used to) handle persona. We got some detailed workflows for each of these cases. Moving forward we want to get apps using OpenID-Connect. Down the road we talked about replacing fas with a very thin layer community access api app. Not sure thats going to happen, but might be an interesting way to go.
  • We wanted to harness the CentOS CI pipeline for testing a subset of optin packages from Fedora. I wasn’t directly working in this area, but I know by the end of the week we had CentOS-CI sending fedmsgs in our staging env and had setup a ci instance near it with resultsdb and taskotron to manage tests. I think there’s some more hooking up of things to go, but overall it should be pretty cool.
  • Finally we wanted to look into and setup our own OpenShift instance. We had some very nice discussions with Red Hat OpenShift folks who manage and deploy things bigger than we can imagine. They gave us some very helpful information. Then we talked out initial policy questions and so forth. By friday we had a OpenShift cluster up and running in our staging env. We still need to get some official certs, sort out the ansible managing applications setup end, and figure out what we want to do for persistent storage, but otherwise we made a vast amount of progress. You can find out questions and answers at https://fedoraproject.org/wiki/Infrastructure/OpenShift

On smaller items:

  • We met with the Red Hat Storage folks who manage our backend storage. There’s some nice improvements coming along later this year if we can adjust things to take advantage of it. Mostly that means splitting up our gigantic koji volume into a ‘archive’ and ‘active’ volumes and splitting our ftp volume into a ‘active’ and ‘archive’ volumes. Then we can hopefully move the active volumes over to flash storage, which should be a great deal faster. All still up in the air, but promising.
  • We met with the folks who manage our main datacenter, and it’s looking possible that we will be moving all our stuff to a new area a short distance away. If this comes to pass, it would be later in the summer and there will likely be some downtime. On the plus side we would get a chance to organize machines better, get new racks and better power and all around be in a better place moving forward.
  • We had a nice discussion around making bodhi faster. I’ve been over that many a time, but we did come up with a few new ideas, like generating drpms at update submission time instead of at mashing time (but that would need createrepo_c changes). Or perhaps triggering pushes only on critpath or security updates and other leaf node packages would just wait.
  • There were a number of discussions around factory 2.0, modularity, branching, koji namespaces, etc.
  • We had several discussions on postgres BDR (Bi directional replication). I’ve moved a number of our apps to it in staging and hoped to roll it out in production, but there were some concerns. In the end we decided to look at deploying it and then deciding on a app by app basis which ones were ready to move over. In the end we hope to have everything using it, but some apps need time to make sure they follow all of BDR’s rules and do the right thing. Additionally some apps may want to use BDR, but in sync mode to avoid possible bad data on a node crash. koji upstream needs to support things before we can move koji, but some of our smaller apps may be able to move soon. We also decided we wanted a script that could detect problems by looking at an applications schema. Pingou already went and had this done by the end of the week.
  • Tim Flink (Fedora QA) and I had a nice discussion about scaling the existing QA setup. We agreed that it might make good sense to see if we could migrate this into our cloud, provided we can get it up on a modern version we could actually support. That is likely to happen in the next few months as we have some new hardware for it and are retiring some of the old hardware, so it’s a good time to force an new setup.
  • We went to the ansible offices and had beer and pizza and watched a baseball game. 🙂

All in all a super productive week. Look for lots of the above to come up in meetings, tickets, mailing lists and so forth as we share plans we made, and make sure everyone is on board with them or if we need to adjust anything.

May 08, 2017

Kevin Fenzi : CI and infrastructure hackfest day -1

Ah travel, such fun, especially these days.

Happily things went pretty nicely… security was not a hassle, the flight was a bit early and went well and no problems meeting up with folks once I landed. Then off to a lovely dinner and sleep in time for the hacking to begin tomorrow.

May 06, 2017

Kevin Fenzi : CI and Infrastructure hackfest 2017 next week

Tomorrow I’m traveling out to Raleigh, NC for a gathering to work on CI and Infrastructure for Fedora and will be out there all next week. We will of course be around on IRC and hope to pull in remote folks that are interested in participating, but if you need us for something and can’t find anyone, please file a ticket and we will get back to you as soon as we can.

https://fedoraproject.org/wiki/CI_and_Infrastructure_Hackathon_2017 has a list of the things we hope to work on, but a short summary:

  • Get a bunch of information from Patrick on OIDC (Open ID Connect, basically the current spec for OpenID), both for application developers who might need to interact with it and sysadmins who need to manage it.
  • Work on a bunch of items related to Continuous Integration in Fedora, koji and bodhi integration and being able to use the CentOS CI hardware to run more tests.
  • Figure out a plan to setup a OpenShift instance in Fedora Infrastructure. This will help us deploy some new apps that expect that sort of setup as well as look at moving some of our apps to this new and exciting workflow. We have a big list of stuff to figure out at: https://fedoraproject.org/wiki/Infrastructure/OpenShift
  • Some misc other discussions about database replications and other things that high bandwith talks would help.

It should be a fun and busy week… I’m going to try and summarize each days discussions here, either daily or at the end of the week.

April 01, 2017

Kevin Fenzi : Rawhide notes from the trail, 2017-04-01 (no foolin!)

Just a few quick notes for those folks riding along the rawhide trail:

There’s some pretty bad brokenness in the polkit package that landed in rawhide on thursday. Your best bet is to downgrade to polkit-0.113-7.fc26 until it gets sorted out. Symptoms include not being able to login with gdm, lots of things being broken because they cannot start, etc. This is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1438086

FESCo approved the Fedora 27 schedule not log ago: https://fedoraproject.org/wiki/Releases/27/Schedule and so the next mass rebuild will happen 2017-07-12. Mark your calendars.

As many of you may know, I switch off between gdm/Gnome and lightdm/Xfce pretty often. I finally tracked down one annoyance that has been hitting me for a while: On login to Gnome I was getting 2 polkit dialogs every time: asking for perms to handle rfkill and networking. Turns out this was due to blueman autostarting in Gnome and not having those permissions. There was no reason for it to need to start there since Gnome has it’s own built in bluetooth, so I filed: https://bugzilla.redhat.com/show_bug.cgi?id=1432555 on that. Workaround is to add a NotShowIN Gnome for it’s autostart file.

Until next time, ride safe!

March 25, 2017

Kevin Fenzi : MUA++ (or on to thunderbird)

So, a few weeks ago I moved my MUA (Mail User Agent) from claws-mail over to evolution. Last week I decided to move on and give thunderbird a try.

Mostly evolution worked, but it had some bugs I hit that were quite annoying: From time to time it would duplicate emails on my work imap server. Suddenly I would get 50 copies of some list post. I could of course select them and ‘delete duplicates’ but this was pretty anoying. I tried all sorts of tuning to get it to not do that, but nothing seemed to work. Additionally I found the keyboard shortcuts difficult to get used.

So, thunderbird. For this I needed to make a change I had been meaning to for a long time, but kept never getting around to: I needed to switch to using my home server’s imap server instead of delivering email to my laptop (thunderbird only does imap for incoming emails). Fortunately, it was easy to just change my .procmailrc to deliver to the server and serve it via imap. However then I ran into some real confusion: I had setup my server (dovecot) a number of years back to provide 2 ‘namespaces’ in imap: The first would be a Mailbox that it would deliver email to, and the second would be a Maildir that it used for folders (this was due to having some friends using my mail server who insisted on using mail clients via shell that didn’t understand Maildir). I had to do a bit of tweaking to get it working for me, but not breaking it for others. The mbox namespace also meant there was a mail directory with mailbox folders and if you were not careful how you set things up you would get those (mailbox is a good deal less permofrmant than Maildir, so I wanted to avoid them). Finally, I got it all working there.

So, after now using thunderbird for a week or so, the good:

  • thunderbird has no problems talking to various imap servers. No duplicate emails, no errors, everything works nicely and pretty quickly.
  • lightning plugin is now built in/included in thunderbird and it has had no problems talking to all my vaious calendars.
  • enigmail seems to do a fine job with encrypted emails and signing my outgoing emails.
  • The keyboard commands seem a lot easier to get used to, and with the Nostalgy extension it’s pretty easy to file emails and go to places.
  • The search features seem very fast and work well. I ‘star’ mails I want to deal with later, and have a search folder that shows all starred emails. I can from there easily open a tab with the entire conversation if I want to read the thread the email was in again.
  • There’s a handy sort by ‘Grouped’ thats nice for some things. It will show you for example todays emails and let you expand previous days if you like.

The bad:

  • I cannot quite seem to get the message view to look the way I want. It seems to change what fonts it uses sometimes based on “I am not sure what”. Possibly if the email is html only? Will keep looking into it.
  • I had to enter all my stupid filtering rules _again_. I just redid them for evolution, but now again for thunderbird. I really need to look into sieve and just doing it on the server. There outta be a standard!

and the things that are just related, but not directly thunderbird:

  • My mail is now in imap on my main server and I can read it via thunderbird, or roundcubemail.
  • I’ve unsubscribed or otherwise removed myself from a bunch of lists or things that were sending me email that I never read anymore or cared about. There’s still some more of these to go, but its good every once in a while to drop all your filters and rebuild them to see what should just never come in at all.

Will I stick with thunderbird? Time will tell. So far this week indications are good, but we will see.

March 13, 2017

Kevin Fenzi : Email and RSS and their readers and writers

I’ve been using claws-mail to both read/compose emails and to read all the vast pile of RSS feeds I try and keep up with for quite a while now and it’s for the most part been fine. I never liked claws-mail use of mh folders instead of something more standard like Maildir, and it’s calendar integration is… not there, but it did a pretty good job.

Unfortunately, claws-mail uses a plugin called ‘fancy’ to render html (which sees heavy use when loading rss feeds). This plugin uses the old old webkit1 (webkitgtk package in Fedora). This package hasn’t been maintained upstream in quite a long while and the number of vulnerabilities in it has just grown and grown. Other distros have dropped it entirely, and Fedora is finally following suit very soon. This has caused the Fedora claws-mail maintainers to drop the fancy plugin to prevent dropping claws-mail entirely. However, without that plugin, RSS reading is… not very pretty.

So, after seeing Jiri Eischmann’s recent post ( https://eischmann.wordpress.com/2017/03/10/nextcloud-linux-desktop/ ) about using the nextcloud News app, I decided to give that a try. Installing was pretty easy, but note that it’s not in stable apps yet. Importing my feeds from claws-mail’s RSSyl was pretty easy, but the only want to put feeds in categories is drag and drop. Drag and drop when you have many many feeds is a real pain, I really wished for a ‘move to folder’ option. The Fedora “feedreader” app looks pretty nice and after a bit of tweaking handled things pretty well. The android OwnNews app also seems pretty nice and functional. I did notice some times when it seemed all three readers were not 100% in sync as to what was read or not, but things stayed pretty close. Being able to use my phone to browse feeds will be handy. I also took this time to cull a bunch of feeds that didn’t have any posts in the last few years or I didnt care about anymore or the like. Overall I think this will be a pretty reasonable replacement.

Next up was mail. I could have just stuck with claws-mail for just mail, but without the fancy plugin any html emails would not render really at all. So, I figured I would bite the bullet and look at evolution again. I switched over my work and gmail accounts first as they use imap and state is (mostly) remote anyhow. They setup pretty easily. Next I added a account that was all my old claws-mail folders in mh format. In the past when I tried this evolution crashed, but this time it seemed to handle it ok. I later disabled this account for now, as it caused fetching new emails to take a while and I only need it if I need to find an old email and can enable it then. Finally I switched over my main mail stream. Unfortunately, I couldn’t find a way to import my old claws-mail filters, but this did give me a chance to cull emails that I get. As things came in saturday and sunday I added filter rules, unsubscribed, or otherwise made a note to stop whatever it was sending me email. There’s definitely some oddities I will have to get used to with evolution, but overall I think I can make it work. The biggest frustration is that the shortcut keys to do things don’t seem to work for me. (ie, . and , are supposed to go to the next or previous unread message, but they do not here. They don’t do anything). There doesn’t seem to be a way to quickly go to a particular folder. After a bit of use, I figured out the problem: The shortcuts are focus specific, and I don’t really see any indicator which pane I have focused. ;(

Right now I think the RSS change is likely to stick, not entirely sure on evolution, but if it doesn’t there’s a number of other clients to try out (thunderbird, mutt, or back to emacs 🙂

March 05, 2017

Kevin Fenzi : Rawhide notes from the trail, the 2017-03-04 edition

Well, branching of Fedora 26 off rawhide has come and gone, and it was a bit of a rocky ride this time sadly.

The branched composes were failing at first, then working, but not actually syncing to the master mirrors in order for mirrormanager to notice it and people to you know, actually use it.

This time we tried to make sure both branched f26 and rawhide were signed the entire time by the right keys. They currently are, but sadly, there’s a old/obsolete/bad/nolonger used/no good/varmit of a Fedora 26 key thats in the fedora-release-25 package. So if you are trying to upgrade you may need to find/get the right f26 key before doing so. Also, the resigning meant that mirrors had to pull basically every package again, so it’s going to be a bit before everyone is synced up on rawhide and f26 content.

If you want to keep following Fedora 26 to it’s release, you need to run a ‘dnf repo-sync –releasever=26’ to make sure you have the right releases and repos. If you want to stay riding on the rawhide trail, you can just dnf update and should get the fedora-27 release and repos.

Until next time, ride safe.

February 23, 2017

Kevin Fenzi : Rawhide notes from the trail, 2017-02-22

Some recent gotchas in rawhide:

  • firefox doesn’t seem to compile correctly with gcc7 (and the optimization levels Fedora uses by default). The current rawhide version will install and run fine, but looks horrible. As a work around you can install a flatpak or a binary version from upstream or downgrade to firefox-51.0.1-2.fc26 from koji. This is tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1422532
  • A few weeks ago, python2-jinja2 was updated to 1.9.4, then 1.9.5. Unfortunately 1.9.4 broke ansible templating, and all the problems were not fixed by 1.9.5. As a consequence, ansible hasn’t been runable the last few weeks. Until today: I pushed ansible- into rawhide. This isn’t the final, but it hopefully fixes the jinja compatibility issues and gets rawhide ansible users back on track.
  • After my last post about composes failing due to an unsigned package, we ran into 3 other issues we had to fix to actually get a compose working: First, an nss update caused lorax to break. Turns out it was due to the mock chroot used not having /dev/urandom and nss noq requires that to be there. Next was the rdma-core package breaking things. It was untagged a while back to fix, but then the mass rebuild rebuilt it again and it got pushed out again. Finally policycoreutils-python started pulling a ton of more deps which broke the minimal installer chroot.
  • Even after we now have composes, livecd’s are not working due to an anaconda bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425827 Hopefully we will have a fix for that soon and livemedia will be back.
  • There was a pungi bug preventing the ostree composes working, there’s a proposed fix for that one.
  • i386 images and media are all failing due to a weird dep solving issue where it says “will not install src.rpm”. https://bugzilla.redhat.com/show_bug.cgi?id=1416699

Lots of bugs, but we are moving forward on quashing them now. I am hoping we have some good composes later this week and then we can try and keep them that way.

February 20, 2017

Kevin Fenzi : Fedora macbook pro testers++

In the final run-up to the Fedora 25 release, we slipped a week because there was a bug in installs on apple osx (now macos again) hardware. This was (and is) a use case the Workstation working group cares about, as they would love for folks with apple hardware to install Fedora and use it on that hardware. Sadly, we don’t have too many testers with this hardware to help our testing cycles, and many community members with this hardware also are using it day to day and cannot afford to reinstall and test at the drop of a hat.

I use my personal yoga 900 for my main machine, so my work laptop has always been for me a test machine or a backup in case of failure. There are a number of reasons for this: I prefer to pick and use laptops that aren’t standard corporate offerings, I like to know that I can do anything I choose with the laptop, and if (god forbid) I moved to a new job I wouldn’t have to give my primary laptop back.

So, when I was up for laptop refresh and I saw that a macbook pro 13″ model (12,1 apparently) was available, I decided to choose that and help out with Fedora testing efforts on this hardware. I do feel a bit bad about this as I am not a big fan of Apple and this does mean giving them money, but on the other hand, hopefully I can test Fedora and help avoid slips due to lack of hardware. Also, I can run rawhide on it and see if I can get everything working fully.

The macbook arrived the other day and yesterday I unpacked it and did some initial testing:

First, the hardware: Seems nice enough, but I am not sure why anyone would get one of these over a Dell XPS 13 or a yoga 900/910. It’s got a lower resolution screen than those, 8gb memory instead of 16, and only an i5 cpu instead of an i7. The feel is very solid, but it makes it just makes it seem too heavy to me. Otherwise the screen is bright and nice, the keyboard backlight is nice (The yoga 900 only has off, dim, bright for the keyboard backlight, but this macbook you can set it to whatever you like), the power connector is neat in that it has a light on it telling you if it’s charging (orange/amber) or fully charged (green). The keyboard and trackpad seem fine.

I decided I would go through the normal macos setup first and then try and setup Fedora to dual boot, as I imagine that would be a common setup. Part of the setup instructions that came from my corporate overlords was mention of enabling File Vault full disk encryption. So, I did that and got everything installed and seemed to be working fine (at least as far as I could tell, not being a macos user normally).

Still following what I would think would be the more traveled path, I went to https://getfedora.org and downloaded the Fedora Media Writer. Download went fine and it was no trouble to run it, but it did give me a warning that this was something I had “DOWNLOADED” from the internet. I don’t guess we have much way around that warning as we are signing the binary fine and it’s not saying it’s unsigned, just that it was downloaded and are you sure you want to run it. Download and burn to usb went just fine, no problems at all with FMW. It might be nice if there is an option to download Rawhide images if you really wanted them.

Hold down the option key and power on and you get the boot selector thing. Choose FedoraMedia and there’s the Fedora Live USB. Everything booted up nicely and I poked around to see what was working or not working. Turns out so far the only thing that doesn’t appear to be working out of the box is the webcam. It seems to be a broadcom model that some folks are working on reverse engineering, but haven’t gotten that done enough to merge into the mainline kernel ( https://github.com/patjak/bcwc_pcie ). I might try that out sometime down the road. The power connector activity seems a bit odd as well: when you plug or unplug the power it seems to take a minute or two before it notices and starts updating.

Next I pulled up the Anaconda installer and looked to install Fedora along side the existing OS. I needed some space to install in, so I selected the largest partition which I knew was the main macos volume (but oddly was showing up as Unknown), and told anaconda to shrink it. That seemed to work fine, and I installed Fedora in the newly freed space. The install finished fine and I rebooted.

On reboot, I now got grub and Fedora booted and worked fine. The macos entries however, errored and didn’t work at all. Turns out this is a long standing, known bug: https://bugzilla.redhat.com/show_bug.cgi?id=893179 I tried various things suggested in the bug to chainload, but with no luck. Luckily you can still hold down option and get the native bootloader.

So, I did that and tried to boot the macos install, but it would think for a while and then error with a picture that was a circle and crossbar and fail to boot. After poking around it seems I now hit: https://bugzilla.redhat.com/show_bug.cgi?id=1033778 ( https://fedoraproject.org/wiki/Common_F24_bugs#apple-core-storage-wipe ). So, my macos partition was unknown to anaconda, so it let me shrink it, but it messed it up completely and nuked all data that was on it. Had I not encrypted things it would have worked, but since I did it was gone.

Luckily apple has the advantage of controlling the hardware platform in this case, so I just needed to boot with option command r to get a network rescue. This let me wipe the drive, repartition it, reinstall macos, then boot the Fedora USB again and install Fedora in the space I left for it. So, it wasn’t the end of the world, but it would sure have been anoying if I had data there.

After all that installing, I moved the Fedora install from F25 to rawhide. No issues there, everything seems to work and be fine after that.

So, if we could somehow fix the two bugs I ran into for f26 I think it would help macbook folks a fair bit. If anyone needs me to install or test anything on this platform, just let me know. I plan to keep the macbook ready to (re)install most anytime and hopefully can provide more test coverage for upcoming cycles.

February 19, 2017

Kevin Fenzi : Rawhide notes from the trail: 2017-02-18 edition

Greetings everyone, lets dive right into the latest changes in the rawhide world:

  • The Fedora 26 mass rebuild ran and finished last weekend. 16,352 successfull builds happened, along with around 1,000 that failed to build. Now we have a few weeks until we branch f26 off to fix things up.
  • The mass rebuild did disrupt signing of normal updates. Perhaps next mass rebuild we should look at standing up another set of signing servers to just sign the mass rebuild.
  • Composes for the last few days have failed. Turns out its due to an unsigned package. But how could that happen? We passed all the builds through the regular signing process. Turns out when builds were tagged in, there were a few builds that overrode newer versions already in rawhide, so releng ran a custom script to retag newer builds back in. However, there was a package where the maintainer built a new version, decided for some reason it was unusable and untagged it. Thats fine, but the custom script mistakenly tagged this “newer” build in and it was long enough ago that it’s signature was removed. Just a short note here about “newer”: koji has no concept of package versions. To koji if you ask for all the ‘newest’ builds in a tag, it will give you the most recently tagged ones. This importantly has nothing at all to do with the package-epoch-version-release, thats just not on a level koji knows or cares about.

February 13, 2017

Kevin Fenzi : A tale of a bluetooth headset bug…

I find tracking down obscure bugs interesting, so I thought I would share another tale of one I just tracked down.

Ever since coming back from the holiday break this year, I found my bluetooth headsets had stopped working. (I have 2 very different models: a plantronics M50 that I have had for a long time, and a Bose QuietComfort 35 (yes, the headphones have a headset profile!)).

At first when I noticed the problem I tried the obvious things:

  • Downgraded pulseaudio
  • Downgraded bluez
  • Downgraded and tried various kernels.
  • Looked though tons of upstream pulseaudio and bluez bugs.

No change at all. I added myself to a existing bluez bug and provided the bluez maintainer a bunch of debugging output, but it all looked pretty normal, there was just no sound. So I gave up and moved on to more urgent things.

Fast forward to today: what better way to spend a sunday afternoon that more debugging. I repeated my former tests and had the same result. Then I decided to try some LiveUSB’s to isolate it. I downloaded and booted a F25 updated workstation iso and it still had the issue. So, I thought, Ah ha it has to be the updated pulseaudio since that landed in f25 updates. But to be sure, I downloaded and booted the stock f25 release iso. Still had the issue. Boggling. Then I went looking around in the kernel bugzilla and found a report that talked about issues with bluetooth and firmware. That was just the hint I was looking for. So, I downgraded the linux-firmware package and tried things and… it still didn’t work! There was one more trick: The firmware only reloads on cold boot. If it’s already loaded (or misloaded) it doesn’t even try again. So, cold boot and then everthing started working.

The key kernel boot message on the failing firmware: Bluetooth: hci0: Failed to send Intel_Write_DDC (-22) and the working firmware: Bluetooth: hci0: Applying Intel DDC parameters completed

For bonus points of course the failing firmware file is: ibt-11-5.ddc and the working one is: ibt-11-5.ddc (yes, thats right, they are changing the file around without changing the version any).

Next of course is to figure out where to report this to get it fixed, but some poking around and testing found that http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/commit/?id=581f24500138f5e410d51ab63b205be9a52f4c77 already had a fix. I just need to wait for a update to the linux-firmware in Fedora and everything should be back to normal.

So, troubleshooting lessons for today: Sometimes cold boot matters. When you think you have downgraded everything that could cause a problem, remember firmware. Reading around on bug reports will sometimes give you ideas on how to solve yours.

February 07, 2017

Kevin Fenzi : Rawhide notes from the trail, the 2017-02-06 edition

Some more notes from the rawhide trail…

Astute readers may have noticed that there was a Mass Rebuild scheduled for last week ( Feb 1st) and it’s not yet happened. There were a few reasons for the delay: The gcc folks wanted a few more days to quash some more bugs in gcc 7.0 and additional time was allocated to a change in how debuginfo packages would be generated. Hopefully this rebuild will fire off tomorrow (the 7th) and result in a 1 week push in the Fedora 26 schedule. Just a reminder about Mass rebuilds: These are done in a side tag by an automated release engineering script. It simply uses the rpmdev-bumpspec script to increase the release tag by 1 and add a changelog entry about the mass rebuild and then initiate a build. Packages that fail to build will have FTBFS (Fails to build from source) bugs filed on them after the mass rebuild is over. There’s two weeks alloted to cleaning up and fixing these FTBFS bugs and any other fallout from the Mass Rebuild. If you are using rawhide, note that there will be a day after the Mass rebuild where most all of your packages will be updated. Allot extra time to download and upgrade on that day. The last Mass rebuild took around 1.5 days to complete. This one may be slower due to all the additional alternative arches we now have.

Another thing that needed to land before the Mass Rebuild was the targeted boost rebuild. This has completed and the side tag merged back into rawhide. Tomorrow’s rawhide should have a bunch of packages that use boost updated. There will also be a few that still need fixing for the new boost version by their maintainers.

After the mass rebuild is over and cleaned up after, we have the branching off of Fedora 26 from rawhide on the 28th, and then rawhide will march on to Fedora 27.

January 21, 2017

Kevin Fenzi : kojipkgs: what it is and how it works

The last week or so I have spent a ton of time on kojipkgs (sorry again for any build failures this may have caused), so I thought it would be good to outline what it is and how it’s used and finally the working setup we have now.

kojipkgs is a concept that koji has of a host/url to go to to download packages. On small koji installs this host/url can be simply the koji hub host. Or it can be a seperate host or hosts, as long as it has access to all the packages koji wants. So in practice this means it has to share a NFS mount or other shared storage with the koji hub.

For many years Fedora infrastructure had a kojipackages instance that NFS mounted the koji storage, ran a apache web server locally on port 8080 and then had a squid instance in front of that, listening on port 80 and 443 and talking to apache for any items that were not in it’s cache. This worked out well because squid could cache commonly used objects and save access to the NFS server as well as return more quickly. However, it also has drawbacks: It’s a single point of failure and it’s exposed to the outside world and squid’s SSL code has not gotten anywhere near the same scrutiny that apache’s has.

A bit of a side note, but related, lets talk about how koji does builds for a minute. If you tell koji to do an official build of some package, first it makes a mock chroot and downloads and installs the packages in the srpm-build koji group (you can see these and what packages are in them with a ‘koji list-groups’ command). Note that this install is done with the HOST dnf into the chroot. The src.rpm is then built from pkgs git and lookaside cache and then koji starts builds for any needed arches. These builds first download the src.rpm via a koji urllib2 call and the make a mock chroot and download and install all the packages in the ‘build’ group. Then rpmbuild is called, etc.

So, week before last I updated builders (as I often do) and also upgraded the kojipkgs01 squid server (running rhel7) to the latest versions. However, we started seeing downloading problems at build time. Sometimes (but rarely) dnf would just fail to download a package it needed and fail the build. I figured this would be a great time to try and remove the single point of failure of that one server.

I spun up a second kojipkgs instance (kojipkgs02) setup just like the first one (rhel7, squid, apache, NFS). Then I setup our two proxies in that main datacenter to handle kojipkgs requests and use haproxy to send them in to kojipkgs01/02. Then, a switch of dns and it was switched over. Now, however we hit a new problem. Sometimes the src.rpm download via koji was not working. It was silently downloading only part of the src.rpm and then failing the build. Additionally, the build download issues were still happening. I tried isolating to just one proxy. Just one backend. Various haproxy balancer setups so that requests that went to one backend kept on that backend. I replaced both kojipkgs instances with Fedora 25 (much newer squid version). Koji has the ability to pass several urls to the mock setup it uses for builds, so we passed it multiple urls to kojipkgs there and that seemed to get rid of that problem, but the src.rpm download issue was still there.

Finally, we tried disabling squid’s “smp mode”. Suddenly everything started working nicely.

So in the end of this saga we have:

  • Nice Highy Available kojipkgs. (2 proxies using 2 backends, so we can update, restart, reboot a proxy/backend anytime we like without causing problems).
  • A bug report on librepo to retry downloads the number of times it claims to in dnf.conf default even if there’s only one baseurl and it was slow/timed out: https://bugzilla.redhat.com/show_bug.cgi?id=1414116
  • A koji ticket to make the src.rpm more robust: https://pagure.io/koji/issue/290 and some PR’s to change it use requests and validate the downloaded rpm already.
  • I still need to file an upstream bug on squid about smp mode, but I don’t have too much info to give them (there were no errors anywhere, just some small number of connections would hang/not finish).

January 05, 2017

Kevin Fenzi : Tools I use daily

I recently saw a post from my co-worker Kushal Das: tools-i-use-daily and I thought it would be fun to share the same information:

Operating System

I use Fedora rawhide on my laptop, which is my primary client machine. I don’t have a desk and desktop and monitors, just the laptop and a comfy chair. At home I have two dell CS24 servers (one is a test instance and has all my test instances for Fedora maintainers on it, the other is my ‘production’ server and is the firewall and hosts the primary server vm that handles all my email and other services for scrye.com). I have various small arm devices, some of which are on and some of which are off at any given time.


Since I run rawhide and update and reboot often (usually 1x a day during the week), I usually switch off between Xfce and Gnome / Fedora Workstation. I do admit Gnome has been growing on me over time and I might give it the edge these days, but Xfce is like an old friend that I can easily also do all my work in. I did play around some with sway (wayland based i3 clone) over the break, but it seemed like it would be too much work to build up a config that would be fully usable.


I still use a pretty crazy mail setup, but it keeps working so I don’t replace it. Mail comes into my main server (using postfix/sqlgrey/spamassassin, etc) then goes to UUCP over vpn (openvpn) tcp to my laptop. Then, on the laptop I use claws-mail to read the email which is locally delivered by postfix. You may think I am completely crazy to use UUCP, but it really fits this use case nicely: I can compose and send emails on my laptop even if I don’t have any internet, they will simply queue and send the next time I connect. Emails wait for my laptop/vpn and then deliver. I can set larger emails to smaller priority so I get the smaller ones sooner. It resumes just fine. It handles slow links just fine. I keep meaning to just switch to imap, but it ends up not really being enough win for me to care. 🙂


vim all the way. I used to be a xemacs/emacs user, but I haven’t touched either in years. vim is just so nice and quick and works everywhere there’s no contest.


firefox now. I was using midori full time, but upstream has really not had much time of late and things have gotten pretty dire. I did move the Fedora packages to webkit2 (via an unreleased upstream branch), but many things still don’t work right there. I would love to go back, but it would need some real releases and people working on it.


hexchat. Anyone out there still using xchat really should switch. I have a znc server on my main server at home I connect to.


wordpress. It’s just easy and works. Sure it has security issues from time to time, but as long as you keep updated you are usually in pretty good shape. I use the Fedora packages and they tend to do well at keeping up to date.


Just a few misc items: I use pass all the time (password-safe), it has most of my passwords in it. ssh, tmux, task (taskwarriror) and ansible are also things I use all the time.

December 28, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-28 edition

Just a few rawhide notes to end out the year:

  • Python 3.6 rebuild completed (mostly) and was merged back into rawhide. There were a number of things that needed cleanup, but we seem to have gotten them all handled in a few days. Next time we should really allow a day or two before we merge the side tag in and fix up things that are important for composes before we merge into rawhide.
  • Astute folks may have noticed: Builds in koji now will complete on all arches, even if one fails. In the past if one arch failed, the rest were canceled and the build came back as failed right away. Now all the arches will complete before the failure comes back. This allows you to see not only the problem that failed the build, but any other problems it would have hit on other arches as well. This should be of help especially to larger builds.
  • There is still a bit of fallout from the rawhide always signed changes we need to fix up in the new year: chain-builds no longer work, the rawhide fedora-release needs to be modified to always expect repos are signed, and we need to discuss what sort of testing we want to add in now that we have a handy place for adding it.

Have a smooth ride the rest of the year, no matter where your trail takes you.

Kevin Fenzi : 2016 Holiday hacking (so far)

I hope everyone is having a lovely holiday season and doing the things they find fun or recharging.

I’ve finally gotten to a few hacking projects that I thought might be interesting to share with others:

  • I finally cleaned up my local ansible repo some and set it up as a pagure project: ( https://pagure.io/scrye-ansible/ ). It still needs a bunch more cleanup (these playbooks were some of the first I ever wrote, so many years ago), but they should help anyone who wants to setup test machines like I have here or provide some more ansible examples.
  • For 4+ years I have had a CentOS6 digitial ocean droplet I have been using as a secondary nameserver for my domains. It actually predates IPV6 support at digitial ocean and had a manually setup he.net ipv6 tunnel on it. I finally looked at the ansible digitial_ocean module and setup a new shiny Fedora 25 droplet, created entirely via ansible as well as putting my secondary nsd configuration in it. This is all now in the above scrye-ansible repo. I may well be using more digital ocean droplets now that I see how easy it is via ansible.
  • I looked around for a new rom for my phone (nexus 6) and didn’t find much that filled me with joy. So, for now I just moved to the latest nightly CM version and will see if this new fork ( http://lineageos.org/ ) pans out. The other roms out there are really unprofessional looking: (homepage is a xda forum post, binaries have no checksums and download from bigfiles.com or the like, last commits to source repos was years ago, etc).
  • One big issue I had had with my phone of late is it quitting when it has 30-40% battery left and suddenly saying it has 0%. I found that you can reset the battery calibration (boot with power, down volume, select ‘bootloader logs’, press power for 10+ seconds until the phone reboots). That seems to have gotten things a good deal happier now, but time will tell. Unfortunately, the battery on the nexus 6 is not very easily replaceable.
  • I got all my home machines rebooted into the latest update kernel. I sometimes delay on my firewall/gateway machine because it means a short outage, but everything is updated now.
  • Pulled all the old backups off my phone and freed up a fair pile of space. There’s no reason to store them there, I can copy them back if I need to restore something from them (which I never really do after the initial restores).
  • Setup usbguard on my home machines. The chances of someone plugging in a evil usb device to them is small, but no harm in setting it up and denying all the non known to me devices. I’ve been using it on my laptop for a long while now.
  • Updated my drop-keys (gnome) and xscreensaver-keys (Xfce) scripts and pushed them to the pagure repo: https://pagure.io/scrye-ansible/c/37610c85918125d92bedff6673b58310605e90dd?branch=master basically these scripts can watch for the screen to lock and drop all your unlocked ssh keys. For gnome there’s an upstream bug that might make this land in upstream gnome-keyring: https://bugzilla.gnome.org/show_bug.cgi?id=735149

In the next few days I hope to catch up on some packaging backlog and perhaps get some armv7 installs done.

I hope everyone is having an enjoyable, relaxing holiday!

December 18, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-18 edition

Hello from the Rawhide trail.

With the recent Flag day (on Dec 12th), we switched all rawhide builds to allow us to sign (and hopefully eventually test) all packages. Here’s how it works:

  • Your rawhide build used to just be tagged into the f26 (currently rawhide) tag. Now, it tags into the f26-pending tag instead.
  • The autosigner sees the build, signs it and moves it to the f26 tag for the next rawhide compose.

Unfortunately, there is a backlog of packages we needed to sign, many from the ppc{le} bringup, so thats why there hasn’t been any rawhide composes the last few days. The one today should be out later today however, and we should be back on track from there.

So currently we are just signing things at the $release-pending tag, but we would like to try and start doing some automated QA there at some point. Nothing that will hold up builds for too long, but something that will catch obviously broken builds from landing. Now that we have everything otherwise in place we can start figuring out what we want to run there.

Also, coming soon to rawide will be the first rebuild of python packages for the upcoming python 3.6. Hopefully that will all be smooth sailing, but a number of package updates will land for that soon.

December 03, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-03 edition

Well, it’s been a while but I thought I would share news from rawhide both from the previous month or so and upcoming.

With the release of Fedora 25 a number of developers and testers are switching back to rawhide in the run up to the Fedora 26 branching event. Welcome back to the trail!

In the last month there’s been various small rawhide issues:

  • openssl 1.1.0c came out and had a number of issues. In particular it broke authentication to koji, so any package maintainers had to downgrade back. It also broke a bunch of tests in python, which has delayed the landing of python 3.6 in rawhide. There’s a patched version of 1.1.0c that landed last week, so everyone should be ok to update openssl again.
  • The latest systemd version landed, but then had to be untagged again because it broke a number of parts of compose. Hopefully those will get worked through and fixed.
  • More recently (The last week) a kf5/qt5 update is in progress and isn’t fully finished. Rawhide users will notice a number of packages dnf says it’s skipping due to broken dependencies. This should get fixed up soon, so just keep skipping those items.
  • Finally, the last few days we have run into an odd problem that looks like it might be related to RHEL7.3 squid (which is whats on kojipkgs.fedoraproject.org and used for package downloads for builds). Sometimes packages just don’t seem to download and it breaks the compose, but testing later they work fine. We downgraded squid back to the 7.2 version for now and composes seem to be working again.

Finally in exciting upcoming news: There’s a “flag day” on december 12th (2016-12-12) for a bunch of changes that release engineering has been wanting to make for a long time. One of these changes will finally allow us to do the last thing we need to do to get a 100% signed and gated rawhide. There will be a change in fedpkg (as well as some koji server changes) that will make rawhide builds land in a ‘f26-pending’ tag instead of just going directly to ‘f26’. Then, our autosign process will sign them and retag them to f26. We also can add in automated QA here to decide if a build is ready to be tagged in or not. I hope next year that we can add a bunch of tests here and make rawhide more and more stable.

October 24, 2016

Kevin Fenzi : Security Score Card For Fedora Infrastructure

Josh is asking folks to send him their security score card via twitter. Since I’ve been trying to blog more and like pontificating, I thought I would respond here in a blog post. 😉

There’s 4 parts to the scorecard:

  1. Number of staff
  2. Number of “systems”
  3. Lines of code
  4. Number of security people

For Fedora Infrastructure, some of these are pretty hard to answer, but here’s some attempts:

  1. Fedora Infrastructure is a Open organization. People who show up and start doing things are granted more and more permissions based on their merit. Sometimes people drift away to other things, sometimes new people show up. There’s some people employed by Fedora’s primary sponsor Red Hat, specifically to work on Fedora. Those account for 3.5 sysadmins, 5 applications developers, 2 release engineers, and 2 design folks. Specific areas will have potentially lots more community folks working on them. So, answer: 13-130?
  2. This one is easier to quantify. We have (almost) everything in ansible, so right now our ansible inventory + some misc non ansible hosts is around 616 hosts.
  3. This is another one thats difficult. We have a lot of applications (see https://apps.fedoraproject.org/) Some of them are just upstream projects we have instances of (mediawiki, askbot, etc). Others are things where we are primary developers on (fedocal, pagure, etc). It would be a fun project to look at all these and count up lines of code. Answer: dunno. ;(
  4. If this is full time security people working only on security issues, then 0. We do have a excellent security office in Patrick who is super smart and good at auditing and looking for issues before they bite us, but he’s not doing that full time. Others of the sysadmin teams do security updates and monitoring lists/errata and watching logs for out of the ordinary behavior, but thats also not full time. So, answer: 0 or 1 0r 3?

So, from this I think it would be nice to have a better idea of our applications (not lines of code), but just where to keep track of things better and who knows that application. It would be awesome to get some full time security folks, but I am not sure that will be in the cards.

I’d like to thank Josh for bringing up the discussion… it’s an interesting one for sure.

October 23, 2016

Kevin Fenzi : Another Fedora cycle, another painless Fedora upgrade

As we near the release of Fedora 25, as always I upgrade my main servers to the new release before things come out so I can share any problems or issues I hit and possibly get them fixed before the official release.

As with the last few cycles, there was almost no problems. The one annoying item I hit was a configuration change in squid. In the past squid allowed you to have a ‘http_port NNNN intercept’ and no regular http_port defined, however the Fedora 25 version ( squid-4.0.11-1.fc25 ) fails to start with a cryptic: “mimeLoadIcon: cannot parse internal URL: http://myhost:0/
squid-internal-static/icons/…” error. It took me a while to find out that I needed to add a ‘http_port NNNN’ also now.

Otherwise everything went just fine. Many thanks to our excellent QA and upgrade developers.

October 22, 2016

Kevin Fenzi : some IOT ideas

With the massive denial of service against a large DNS provider (in turn causing a lot of other things to have outages) on friday using a network of insecure IOT (Internet of things) devices (mostly cameras), a lot of folks are thinking about how to address IOT problems. There are a lot of problems and no easy answers, but I thought I would throw out a few ideas here and see if any resonate with others.

First, the problems: IOT devices are insecure and easily taken over for denial of service or other attacks, there is little to no economic incentive to make them more secure, consumers largely don’t care as long as they are still doing whatever their main function is, devices seldom get updates and when they do only for a short time before the company that made them moves on to something else.

Some ideas:

  • Make things vastly more cheap. Yeah, thats right, more cheap. To the point where you can dispose of or toss in recycling the device that just went out of support or was found to be used in some botnet. Or 3d print a new device. Of course this is not going to happen for a long while.
  • Make updates very reliable. I think we can do this with something like ostree/atomic + some wrapper hardware. Upgrade, reboot, if things don’t come back in X minutes, reboot back to the last working version and send for help.
  • Get IOT devices to use mainstream open Linux distros. These would provide source, upgrades, and device makers wouldn’t have to care about not supporting things. This would probibly take legislation and a big push for Linux distros to cater to IOT devices, and there would likely still be some hardware specific code (but it could be open and maintained in distros).
  • Require Internet insurance for users of ISPs. ISPs that police botnet and other harmful devices would pay lower raates than ones that didn’t care, money raised could be used to shut down harmful devices.

It’s a pretty difficult problem sadly, and there’s no good answer, but we are going to have to start doing something or start getting used to large DDOSes taking out a bunch of things we use often.

Kevin Fenzi : xfce4-terminal: lots of progress recently

Most Linux desktop’s have their own terminal programs, and Xfce is no exception. There’s been a number of releases of xfce4-terminal recently, so I thought I would share some of the changes with people who perhaps haven’t tried it recently.

The biggest change is that it’s been ported to gtk3 and vte291. This is great news as it means it’s no longer using the no longer supported at all ancient version of vte. Unlimited scrollback has been added, wayland support, tons of bug fixes, translation updates, and memory leaks quashed.

More specifics in the NEWS file: https://git.xfce.org/apps/xfce4-terminal/tree/NEWS

Give it a try today and see if it handles all your terminal needs.

October 20, 2016

Kevin Fenzi : keepalived: Simple HA

We have been using keepalived in Fedora Infrastructure for a while now. It’s a pretty easy to use and simple way to do some basic HA. Keepalived can keep track of which machine is “master” for a IP address and quickly fail over and back when moving that IP address around. You can also run scripts on state change. Keepalived uses VRRP and handles updating arp tables when IP addresses move around. It also supports weighting so you can prefer one or another server to “normally” have the master IP/scripts.

Right now we are keepalived on our main koji server pair. We have a koji01 and koji02. Normally 01 is primary/master and has the application IP address on it so all traffic goes to it. If for some reason it was turned off, the keepalived on 02 would see that and take the IP address and run a script to become master. If 01 came back up, 02 would see that and transfer back to it. Right now we have the scripts setting up on the secondary server a bunch of cron jobs (garbage collection) and kojira (the process that regenerates build roots).

We are also using keepalived on some new paired postgresql instances we are working on. More on that in a later blog post. If you need simple HA with a IP address and script(s), keepalived does a banner job.

October 19, 2016

Kevin Fenzi : pass – The standard unix password manager

In my line of work (sysadmin), I have to deal with a LOT of passwords. For a number of years I was a fan of keepassx, but the upgrade to version 2.0 there didn’t thrill me. (There were some features I liked that it dropped and in general it seemed to be less nice), so I decided to look around and was pointed to pass.

Pass (or password-store as it’s sometimes called) can be found at https://www.passwordstore.org/ and in most major Linux distributions package set. (It’s called just pass in fedora). It’s a simple command line tool and uses gnupg and git (if you like). Each of your passwords / sites is a gnupg encrypted file, setup in a tree under ~/.password-store. You can tell pass also to use git here so every change you make it a git commit of encrypted files. So, you should be able to find any old passwords you changed in git history if needed.

Setup is pretty simple:

  • pass init yourgpgid@example.com – this sets up the base directory for pass. Note that you can actually add additional gpg keyids to encrypt files to later if you have a team or the like.
  • pass git init – this sets up the git repo

After setup you can use ‘pass insert path/to/name’ to add your own password, ‘pass generate /path/to/name’ to have pass generate a random password for you (using pwgen), ‘pass ls’ to list the tree of sites, ‘pass -c /path/to/name’ to copy the password to your clipboard for easy inserting into another application or website. (by default it will stay in the clipboard for 45seconds or 1 paste and then vanish). Note that you never even need to know the password here, you just get it from pass and paste it in and are done. If for some reason you do need to see it (a broken app that won’t let you paste for example), you can just do ‘pass /path/to/name’ and it will output the password. You can do ‘pass edit /path/to/name’ to edit a password, and note that you can add whatever you like to the pass file. pass -c will use the first line as the password, but you can add more lines with security questions and random answers, usernames or any other notes you like about the site. ‘pass git ‘ will let you run any arbitrary git command on your pass git repo, if you wanted to look at history or go back to some previous commit.

There is a android app that is supposed to work with pass files, but I have not tried it. It requires you to copy your gnupg private key to your phone, which is not something I am really wanting to do. Since I have my laptop with me most all the time anyhow, it’s not a big deal.

October 18, 2016

Kevin Fenzi : debugging koji build failures

From time to time builds fail in koji (The Fedora build system), and it’s good to know how to figure out where to look for the reason. Koji has a central hub that manages jobs and a bunch of builders that actually do the builds. When someone initiates a build you are talking to the hub and either uploading a src.rpm (for a scratch build) or telling it to use a particular git hash/repo for an official Fedora build. For official builds, koji will first generate a job to build the src.rpm from git and the packages lookaside cache with the source. This job will run on some builder thats ready and has capacity for it. Once the src.rpm is generated (or if you are providing it for a scratch build), the hub will generate build tasks for all the arches that are set in the target tag you are building to. Each one of those will go out to a builder of the right arch type that is enabled and has capacity, etc. If any of these fail, the entire build fails.

Once you have gotten notice of a failure, the first thing to do is go to the link for the build in the koji web interface and then click on the task link to show the build tasks. There you will see one (or more) links in red that failed. Don’t click on any of them yet though, instead click on the “show result” link. That will tell you exactly which task failure failed the build. When one failure happens koji cancels the other builds, but sometimes they fail at nearly the same time and you are looking at the wrong place for the real failure. Sometimes all the links are green and the reason for the failure is only visible in the “show result” area (for example, if you have a archfull package with noarch subpackages and they are different on different arches, thats a failure, or if you don’t have permissions to build the thing you are building, thats a tagging failure).

Next click on that task that you saw in the “show result” link (if you haven’t already seen the error as above), and again click on the “show result” link. You should see something like:

"BuildError: error building package (arch i686), mock exited with status 1; 
see build.log for more information"

Or it could refer you to the root.log. Go to that log and look toward the end for the error. If the error is in the root.log it means there was a problem setting up the mock chroot. That could be that there is a package you BuildRequire that in not installable, or some package in the build root is completely broken.

If the issue is in the build.log its the actual output from the rpmbuild, so this would be where to look at your general compiler issues, etc.

If you ask someone to help you out and look at the error(s) you are getting, it’s important to give them the top level link to the task instead of just a link to the root.log or build.log. If you only have those links it’s impossible to back up and look at the higher levels, but if you have the top level task you can look at the entire thing.

Aside from normal rpm package builds, koji also builds a bunch of other things like livemedia and other images. Finding issues there is very similar. Always check the “show results” area for which logs to look at for the actual failure.

October 17, 2016

Kevin Fenzi : gnome extensions

In general when using Gnome I try and avoid extensions. Most of the ‘default’ setup is fine or I was able to get used to it and it works well enough. There are a few extensions I do use however for various reasons. All of these work with Wayland.

  • AlternateTab – I use alt-tab a LOT. I just found myself unable to get used to the default behavior of showing applications and windows under those, when I use alt-tab I want to switch to a specific window quickly, not have to switch to a application and arrow key around to find the one I want.
  • Openweather – This is just fluff, but nice to have. Gives you a nice little weather report in the top bar. You do have to pick a provider and get an API key, but it’s pretty painless. You can also setup multiple locations.
  • PanelOSD – This one I use to move notifications over to the top right side. This is where I am used to them being from Xfce and that way they are not in my way.
  • TopIcons – This allows you to see systray icons on the top bar. There’s still a few applications I use from time to time that have a systray and it’s handy to see them if they do.

I’d probibly include the redshift extension here, but it doesn’t work with Wayland. 🙁

October 16, 2016

Kevin Fenzi : Some interesting gnome 3.22.x gsettings

As I have mentioned in the past, I switch off between Xfce and Gnome desktops here. While in Gnome, I have a number of settings I have picked up in various places (bugs, talking to maintainers, various docs and blogs) and I thought I would share a few in the hopes that they would be of interest to others. You can use the ‘gsettings’ tool or the ‘dconf-editor’ GUI to set or check the current values of these keys. Of course some of them can also be set from the normal settings guis.

  • org.gnome.shell.overrides edge-tiling true – This lets you drag windows to the right or left edge and have them tile to 1/2 of the desktop. This goes along with dragging to the top of the screen to maximize an application.
  • org.gnome.software download-updates false – Disable gnome-softwares downloading of updates. Since I use dnf this saves me a bit of BW and also messages about needing to apply updates I already have applied.
  • org.gnome.nm-applet suppress-wireless-networks-available true – This disables the ‘There are open wireless networks” message. I only want to join specific wireless networks if I need something.
  • org.gnome.settings-daemon.plugins.xsettings hinting ‘slight’ – slight hinting works best with the fonts I like. I think this may now be the default.
  • org.gnome.desktop.wm.preferences focus-mode ‘sloppy’ – I want to change focus when I move the mouse. I am pretty used to this from years and years now.
  • org.gnome.desktop.wm.preferences auto-raise-delay 125 – and autoraise on focus change.
  • org.gnome.shell.window-switcher app-icon-mode ‘both’ – I want both the icon and the name in the alt-tab switching thing.
  • org.gnome.desktop.interface clock-format ’24h’ – 24 hour clock is what I am used to. 🙂
  • org.gnome.desktop.interface show-battery-percentage true – shows the battery % left in the menu bar. Handy when you are wanting to see how long you have left and you don’t want to pull down the menu.
  • org.gnome.desktop.interface cursor-blink false – blinking cursors are evil. They suck power are distracting and should be disabled. (I could swear that I saw a site a while back that showed how to disable blinking cursors in any application or system you could name, but I can’t seem to find it now)
  • org.gnome.desktop.interface clock-show-date true – I like the date in the menu bar.
  • org.gnome.shell disable-extension-version-validation true – This disables the version check for extensions. Sometimes it helps, but usually if an extension doesn’t work for me setting this doesn’t help much.
  • org.gnome.desktop.peripherals.touchpad two-finger-scrolling-enabled true – I used to always use Edge scrolling, but it went away for a while and I retrained myself on 2 finger scrolling and it’s fine now.
  • org.gnome.desktop.peripherals.touchpad tap-to-click true – must have tap to click. Saves so much wear and tear on my hands and the touchpad. Just a gentile tap is all thats needed.

Do you have some favorite ones?

October 15, 2016

Kevin Fenzi : Running ansible from a git checkout

The Fedora and EPEL ansible packages are as up to date as possible, but sometimes you want to run a newer, not yet released ansible version for any of a number of reasons:

  • To confirm a bug is fixed.
  • To test an RC for the next release or look for regressions on the devel branch.
  • To see if a change caused any performance improvements or regressions.

Happily this is very easy to do with ansible:

  1. clone the ansible repo: ‘git clone https://github.com/ansible/ansible.git’
  2. checkout the branch or thing you want to test. For example now stable-2.1 branch has the latest 2.1.x rc, and stable-2.2 has the code for the latest 2.2 rc.
  3. ‘source ansible/hacking/env-setup’ This will set your ansible/ansible-playbook commands to use your checkout instead of any locally installed version.

I use this setup to test new versions/RC’s over the Fedora Infrastructure playbooks. It’s very handy to catch last minute bugs in RC’s and/or to clean up depreciations so when the next version is actually released you are already ready for it.

Happy ansible hacking.

Kevin Fenzi : The greatest game in history…

I am of course talking about nethack.

If you haven’t played in a little while (and you HAVE played nethack right?), it’s as fun as you remember it. Running around the dungeon fighting monsters and picking up magical devices. It’s pretty hard to believe that next year will be the 30th anniversary of nethack.

It’s friday, you deserve some nethack today.

October 14, 2016

Kevin Fenzi : ss: the netstat replacement

Many of the old network tools in the net-tools package have been deprecated for a while now. The replacement for the traditional netstat command is ‘ss’.

It of course has a ton of options (like netstat). I particularly like ‘ss -e’ (extended) and ‘ss -E’ (show connections as they appear).

Do take a look today and see if you can finally replace your netstat usage today.

October 13, 2016

Kevin Fenzi : The Fedora infinote server

Many of you may not know that Fedora Infrastructure runs a infinote server instance at infinote.fedoraproject.org.

Infinote is a collaborative text server. You can connect to it with the ‘gobby-0.5’ client located in the gobby05 package in Fedora. Once connected you can create documents and multiple people can work on them at the same time. The server takes a git snapshot of all documents every few minutes so you can see history. There’s even a cgit instance at https://infinote.fedoraproject.org/cgit/

Of course these days there’s etherpad and the like that have more features, but from time to time there’s a good Fedora use for a server we run. Fedora Infrastructure uses it to coordinate their weekly meetings and update/reboot planned outages. Everyone is welcome to use it for their Fedora related collaborative text editing. Or you can of course install the libinfinity package and run your own server.

October 11, 2016

Kevin Fenzi : Fedora 25 Beta is here

This morning we released Fedora 25 Beta: https://fedoraproject.org/wiki/F25_Beta_release_announcement

Release mornings are always a bit stressful for us in Fedora Infrastructure, even though we have gotten pretty good at doing them. Lots of things need to be lined up and ready to go when the announcement goes out: torrents, websites, announcement, etc. Also any networking or hardware issues are particularly unwelcome on these days.

Today everything has gone quite smoothly so far, and I hope everyone enjoys Fedora 25 Beta.

October 10, 2016

Kevin Fenzi : Fedora go/no-go meetings

Before each Fedora milestone release (Alpha, Beta, Final) there is at least one Go/No-Go meeting. As the name suggests this is the meeting where folks from QA, FESCo, Release engineering, Program Management and others gather to see if the release is ready to go out the following week. The meetings are held on IRC and open to everyone (Although go/no-go votes may only be taken from representatives of specific groups). You can see the logs of the last one (last week to decide if Fedora 25 Beta was ready to release this week) this meetbot.fedoraproject.org log link

In the event the decision is “No-go” then the Milestone release slips a week and a new Go/No-Go meeting is scheduled a week out. This allows whatever issues were blocking the release to (hopefully) be solved.

If the decision is “Go”, then the release happens the following week. Why so long between the decision to release and the actual release? Well, there are a number of things that need to happen to get things ready for release, including, but not limited to:

  • The release has to be put in the correct place on master mirrors (moved from the alt/stage directory to releases/test or releases) and time is needed for all our mirrors to pick up this content and sync it.
  • Common bugs need to be identified and written up: https://fedoraproject.org/wiki/Common_F25_bugs
  • The websites team needs to land any changes they need for the release and get them ready to go live on release morning.
  • Torrents need to be synced and setup, ready to live for release morning.
  • In the buildsystem, the release needs to be tagged so we know exactly what packages were in that release and so alternative arches can build the exact same set for their releases.
  • Checksum files and so forth need to be verified and signed by release engineering.

I hope everyone is looking forward to the Fedora 25 Beta release tomorrow!

October 09, 2016

Kevin Fenzi : squeezing out a bit more battery life under Fedora

In my experience, battery life on laptops has been getting better all the time with Linux/Fedora. However, there’s a few things you can do if you need just a bit more battery life than you are getting with default settings.

First let me say that in the ideal world you wouldn’t have to do any of this. Things should just work and give you the best battery savings they can. Sadly, thats not fully the case (yet), so sometimes you have to take matters into your own hands.

First, there’s some packages that try and set things for you: tuned and tlp (and possibly others, but those are the two I know of). If you are using tuned, you will want to run a ‘sudo tuned-adm profile powersave’ to switch to the powersave profile. With tlp it should switch to this automatically when it sees you are running on battery.

powertop has a bunch of recommended settings for power saving which you can automatically apply with ‘sudo powertop –auto-tune’ or by enabling the powertop service. You do need to be careful here however as some of these settings are not set that way by default because they cause lockups or other bad behavior. Do test this on your particular hardware before you trust it.

Finally there’s a bunch of common sense items: lower the screen brightness as much as you can (the screen is by far the largest power user on a laptop), put your bluetooth/wireless in airplane mode so they don’t consume power (if you don’t need them), remove usb devices (some of them suck up power even while not doing much), turn off keyboard backlight and such.

October 08, 2016

Kevin Fenzi : Fedora Infrastructure Outages (and the one on 2016-10-08)

Astute folks may have noticed that we had a pretty severe outage this morning that lasted just about an hour. This seems like a good time to talk about outages in general and this one in specific.

In general these days our outages fall into roughly 3 ‘buckets’:

  1. Planned outages. We do these every few months to reboot servers and apply updates, or when we have to migrate something that will take some downtime. This is by far the largest bucket.
  2. Unplanned outages, but that are small in scope. Typically this might be one of our many datacenters having network problems or a single service having some issue. These typically aren’t noticed by very many people (in most cases only the infrastructure noc folks who disable a proxy or restart a service or whatever is needed).
  3. Unplanned outages that are large in scope. These are thankfully rare. These are were we loose connectivity to our main datacenter (that has the majority of our hardware in it), or severe hardware or software failure (storage servers not working, database needs reloading, etc). Luckily these are pretty rare.

So what happens in an outage? Normally nagios lets us know about it pretty quickly and someone starts taking a look. Usually until we have some idea, we treat things as being in bucket 2 if they aren’t planned. That is, we assume it’s a minor issue until we investigate and see it’s larger. Once the scope of the issue is determined, we update https://status.fedoraproject.org/ This is out status indicator. If you are wondering if there is a known outage, you can look at this. Note that this is MANUALLY updated after initial innvestigation, so do give a few minutes after seeing something before expecting status to be updated. Next, folks work on the outage, this would be anyone in the sysadmin-noc group and the primary infrastructure admins. Status is then updated when the outage is over.

Today (2016-10-08) at approximately 16:15UTC our primary datacenter (PHX2) became unreachable. This turned out to be due to a firewall upgrade that did not go as planned. Service was fully restored at approximately 17:15UTC. Right now when our primary datacenter isn’t reachable most things are also not working as we have our primary vpn hub located there, and thats what our various proxies use to talk to applications. Unfortunately this includes our most active service: mirrorlists. We have planned to move that service to a container and run it at each datacenter, but just have not gotten to deploying that yet. Once we do, at least mirrorlists will stay up if the primary datacenter is down.

Additionally, we have plans to eliminate the first type of outage or reduce it greatly. For the most part our applications are deployed with pairs (or more) of instances so no service needs to go down, however the sticking point is rebooting database servers. I’ve been working on setting up some replication so hopefully we don’t need to schedule any outages for that either.

October 07, 2016

Kevin Fenzi : Rawhide kernels back on track now

Just another quick note that rawhide kernels are back to booting and working normally here at least. With 4.9.0-0.rc0.git3.2.fc26.x86_64 at least. That should be in today’s rawhide.

October 06, 2016

Kevin Fenzi : rawhide 4.9.0 pre rc1 kernels

Just a quick note that with the upstream kernel folks opening up the merge window for the 4.9 linux kernel, rawhide also has been getting these merge kernels. Of course there’s a ton of churn when the kernel merge window is open, vast amounts of things are merged, so it’s a wonder they usually work as well as they do.

In this case there seems to be some pretty severe issues with at least kernel-4.9.0-0.rc0.git1.1.fc26 and kernel-4.9.0-0.rc0.git2.1.fc26 at least. In vm’s they seem to boot and then start oopsing. On my laptop it does boot, but then waits about 2 minutes at waiting for usb devices, boots the rest of the way up and then has no wireless or usb devices.

https://bugzilla.redhat.com/1382134 is tracking at least part of this issue and the kernel maintainers are digging into it. Of course while they are, more things are being merged (possibly even fixes to these issues), so hopefully it will all shake out in the next few days.

In the mean time, rawhide users are advised to just switch back to the 4.8.0 final kernel for now.

October 05, 2016

Kevin Fenzi : 10 basic linux security measures everyone should be doing

Akin to locking your doors and closing your windows there’s some really basic things everyone should be doing with their Linux installs (This is of course written from a Fedora viewpoint, but I think this pretty much applies to all computer OSes).

  1. Choose nice long passphrases you can remember. Most any modern system will have a pretty long limit on passphrases, so pick something nice and long that you can remember. Don’t think of them as passwords, they are phrases with many words.
  2. When installing, encrypt your drive(s). The performance hit is not noticeable and if you ever throw away a broken drive or someone steals your computer they won’t have your data.
  3. Apply updates regularly. If you aren’t someone who remembers to do so, setup something like dnf-automatic to just apply them every day for you in the middle of the night. Otherwise try and get into the habit of letting gnome-software do offline updates at some regular time.
  4. Along with (3), reboot when needed for new kernels or glibc or other things you use. Get used to rebooting on a regular schedule. Don’t be afraid of rebooting, get used to doing it.
  5. If you are in a place with untrusted people roaming around, do setup a screen locker and lock your computer when you are away from it.
  6. Make (and sometimes test) regular backups. You may not think of backups as a security measure, but they sure are. Think of the new fad of ‘ransomware’ where someone encrypts your data and sells you the key. If you have good backups you can just wipe that all out and restore from those. They are handy for lots of other reasons too.
  7. Don’t open weird attachments or links sent to you in email. If you didn’t ask for it, delete it.
  8. Don’t plugin weird devices you run across to your machine. (USB or otherwise). You can use a neat package called ‘usbguard’ to make sure no one else does while you are not around too.
  9. Use a passphrase manager or have some system to allow you to have long, not easily guessed passphrases at all the various applications you login to. There’s tons of these out there: Password managers: pass, keepassx, gpg encrypted file, etc. Schemes: Diceware, etc. Pick one that works for you.
  10. If you use a laptop/travel a lot, consider using a VPN for all your network needs. As long as you have an endpoint to connect to (your home server, your work, a vpn provider) you can send (almost) all your traffic over the vpn and thus avoid problems with people sniffing local traffic.

Some of these things require an initial investment of time (backups, vpn, passphrase manager, screen lock) and some require just making something a habit (long passphrases, apply updates regularly, reboot regularly, don’t open weird things in email or the physical world), but they are all worth it.

Will this make your computer “secure”? No. There’s no such thing. “secure” is not a binary state, it’s a process of assessing threats and deciding what you can or want to do about them. Doing the above things will protect you from some threats nicely (guessable passwords, untrusted people tampering with your computer, sniffing traffic, vulnerabilities that have already been fixed in software you use, etc), but will basically do nothing against others ( someone installing a keylogging device and recording everything you type, someone threatening you with harm to tell them some information, someone installing a spy cam and recording everything on your computer screen, someone using a non public vulnerability in software you use, someone social engineering access to your computer, etc).

October 04, 2016

Kevin Fenzi : Short note: fedoraproject.org smtp sessions now using TLS

Just a quick note for everyone who gets emails from fedoraproject.org or it’s various other domains. Just before we entered freeze for Fedora 25 Beta we landed changes to make our smtp servers use TLS where possible, so emails to servers that support it should now be fully encrypted.

Please let us know if you see any issues related to this change.

October 03, 2016

Kevin Fenzi : How to debug Fedora rawhide compose problems

From time to time rawhide composes fail and are not announced or synced to mirrors.

In the past this would happen only if the very basic setup (a mock chroot with the ‘buildsys-build’ group installed in it) broke. Additionally, in the past rawhide composes where many deliverables failed to compose were still synced out and announced, leading to days when no images were available until the issue was fixed.

Now, with the latest version of pungi (The tool that composes Fedora releases, including rawhide), composes can fail if some deliverables (Those marked in the configuration as not failable) didn’t complete. So, while rawhide can fail more easily, it also means it’s much easier to revert some change that broke images and get that fixed before it lands, and images should be always available.

So, how can you tell if a rawhide compose (or some part of it) failed and why? All the pungi logs are avilable and since all the builds take place in koji, anyone can look at them as well. Rawhide composes are of course fedmsg enabled, so you want to look for the https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.pungi.compose.status.change topic. Composes can finish with 3 states:

  1. FINISHED – This means the compose finished and everything in it completed successfully. I am not sure we have yet seen this status in real life. 😉
  2. FINISHED_INCOMPLETE – This means the compose finished and only failable things failed. This is the “normal” status we see day to day.
  3. DOOMED – This means the compose failed it’s initial very basic setup and/or some deliverable marked as not failable failed. When this happens, it means the compose isn’t synced out or advertised. This is the status where we need to find out what caused the problem and fix it and either restart the compose or wait for the next days compose. In IRC or on fedmsg you may see this status as “failed in a horrible fire” as thats what our fedmsg translates it to.

So if you have a compose and you want to see why some part of it failed, you can look at the fedmsg and it provides a location url, but they are always the same format. Lets look at an example: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20161002.n.0/ This means it’s a rawhide compose (there’s a Branched directory for the Branched Fedora 25 right now), then the particular compose was Fedora-Rawhide-YYYYMMDD.n.0. Yeah, month and day, then a ‘n’ to mean ‘nightly’ and a ‘0’ to mean it was the first compose of the day. If there’s more composes that same day, they are in n.1, n.2, etc. For Alpha/Beta/Final releases there’s no ‘n’ there, as they are not nightly.

The first place I go to look for issues is the global log file. This would be in logs/global/pungi.global.log and you will want to look for the image or deliverable you are interested in or search generally for tracebacks. Usually this will note a koji task id or build which you can then look up on koji. Since this is already getting long, I’ll post about tracking down koji build problems another day.

October 02, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-10-02 edition

It’s once again been a while since one of these posts, but again I’m going to try and do them more often again.

This last week had a few big changes in rawhide:

  • With the Fedora 26 change approval last week ( https://fedoraproject.org/wiki/Changes/DNF-2.0 ) DNF-2.0 has landed in rawhide. Things were a bit shaky at first as the compose tools were not also updated, but anaconda folks were ready with a patch and lorax got a patch from the DNF maintainers and everything landed. There’s still one big issue outstanding: dnf 2.0 sets strict group installs back to true, which causes all the live media to fail to compose. Some of our comps groups contain packages that only exist in some arches, not all of them, and when dnf does strict group installs it errors when it cannot find all the packages in the group. I’ve filed a bug for figuring this out: https://bugzilla.redhat.com/show_bug.cgi?id=1380945
  • Last year we setup a quick and dirty autosigning setup for rawhide, which was just a script in a loop run by releng folks when they remembered to do so. This meant it wasn’t all that reliable. Now, thanks to Patrick Uiterwijk, we have a real autosigning setup implemented and are very very close to an always signed rawhide. The final bit of gating needs a update to fedpkg to go out first, but as of last week, most everything should be signed with only the rare issue with a build that lands right before the compose starts.
  • There has been some work the last few weeks to produce a rpm ostree for rawhide that is updated every few minutes. Once this is available it will be a great help to testing and running rawhide.
  • Xorg server 1.19 almost landed, but we need to fix tigervnc before it can (at least without breaking all the install paths for rawhide). That is being fixed in a side tag and should land as soon as tigervnc is fixed. https://bugzilla.redhat.com/show_bug.cgi?id=1380879 is tracking that work
  • A while back changes were made to cause composes that fail when some deliverables don’t compose. This has been very helpful in making sure changes that land don’t break things. New package builds that cause the entire compose to fail can be untagged until the breakage is fixed.

That’s it for now from the rawhide trail…

Kevin Fenzi : How to reignite a flamewar in one tweet (and I still don’t get it)

disSome of you may have seen this post last week:


I saw it I think shortly after it was posted to hacker news (aside: I read hacker news via it’s rss feed that has just the raw links and no comments or ratings, much nicer IMHO).

My first thought on reading it was “Did they report this upstream?” and upon some digging, yes they did ( https://github.com/systemd/systemd/issues/4234 ). It was filed the same time they posted their blog post/tweet (no responsible disclosure here). My next thought was to see if it worked, so I tried it on various of my test machines and it did not. Turns out you have to run it in a loop (as noted now in the blog and seen in the systemd issue). At that point I saw that there was a Fedora tracking bug, someone requested a CVE and there were patches proposed on the systemd issue (which has since been closed/fix merged), so I moved on with life.

Next someone posted the blog link to the fedora users list along with a rebuttal from a systemd maintainer: https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post, and the discussion went downhill quickly from there. Finally, I saw a slashdot post today with the calm, journalistic title of Multiple Linux Distributions Affected By Crippling Bug In Systemd (which points to the blog post above that started all this).

So, everyone is flaming everyone else again about systemd, and I just don’t get it. My personal relationship with systemd is well within the range of all the other Open Source projects I use every day. Mostly I am happy with it, sometimes it has bugs and I report them, sometimes the project makes decisions I don’t like or agree with, sometimes the project doesn’t communicate as well with downstream as I would like (see fedora devel list post about systemd and TasksMax ), sometimes I have to learn (again) how to do what I want to do, sometimes it’s slow when I want it to be fast and so on. Would a local denial of service attack in another project have (re)opened such a bunch of flames? There was in fact just the day before the post a remote denial of service in bind (the most popular dns server out there). Did you even hear about it?

I just don’t get the passionate hatred systemd has. I guess perhaps I never will, and I guess thats ok. I would kindly ask those who do passionately dislike systemd to just move on to somewhere without it and leave the rest of us alone, but I don’t expect thats likely to happen. In the mean time if you have some technical problem with systemd, I will be happy to help you isolate it and file a bug or teach you how to do whatever you are trying to do with it, but I’m going to try and stay out of your flamewars.

September 25, 2016

Kevin Fenzi : 6 months a task warrior

A while back I added a task to my taskwarrior database: evaluate how things were going at 6 months of use. Today is that day. 🙂

A quick recap: about 6 months ago I switched from my ad-hoc list in a vim session and emails in a mailbox to using task warrior for tracking my various tasks ( http://taskwarrior.org/ )

Some stats:

Category Data
Pending 17
Waiting 20
Recurring 12
Completed 1094
Deleted 18
Total 1161
Annotations 1059
Unique tags 45
Projects 10
Blocked tasks 0
Blocking tasks 0
Data size 2.0 MiB
Undo transactions 3713
Sync backlog transactions 0
Tasks tagged 39.8%
Oldest task 2016-03-24-13:45
Newest task 2016-09-25-10:03
Task used for 6mo
Task added every 3h
Task completed every 4h
Task deleted every 10d
Average time pending 4d
Average desc length 32 characters

Overall I have gotten a pretty good amount of use from task. I do find it a bit sad that I have only been completing a task every 4 hours and adding one every 3 hours. At that rate things aren’t going to be great after a while. I have been using annotations a lot, which I think is a good thing. Looking back at old tasks I can get a better idea of what I did to solve a task or more context around it (I always try and add links to bugzilla or trac or pagure if there’s a ticket or bug involved).

I’d say I am happier for using task and will continue using it. It’s very nice to be able to see what all is pending and easily add things when people ask you for things and you are otherwise busy. I’d recommend it to anyone looking for a nice way to track tasks.

August 29, 2016

Kevin Fenzi : two vpns and NetworkManager? no longer a problem…

NetworkManager has pretty handy vpn handling for laptops. You can setup all different kinds, it can prompt you for passphrases, it can set up specific vpns on specific networks, etc.

However, if you had more than 1 vpn you wanted to run at a time you had to pick one for NetworkManager to handle and do the other(s) outside NetworkManager. This is happily no longer the case (at least with NetworkManager 1.4.0): It can bring up as many vpns as you like and manage them all.

This weekend I took my home openvpn connection (which I had setup as a system wide openvpn service) and moved it into a NetworkManager vpn profile. It works just fine with my work vpn and now NetworkManager can bring it up automatically after it connects to the network.

So, if you need two (or more) vpns on your laptop, you can now happily move them all into NetworkManager. Many thanks to the hard working NetworkManager folks. 🙂

July 25, 2016

Kevin Fenzi : Looking forward to flock 2016

Just over one week until flock ( https://flocktofedora.org ), Fedora’s main yearly conference. This time it’s in Kraków, Poland. This of course means a long time traveling for myself and other North American Fedorans, but it’s always well worth it.

In addition to seeing old and new friends, I’m looking forward to quite a lot of the talks (see https://flock2016.sched.org/ for the full schedule) along with workshops and hallway discussions.

I’m going to be giving a talk on using Fedora Rawhide as your daily driver OS. Hopefully lots of tips and workflows that people will find useful. Also, I am co-presenting with Pierre-Yves Chibon (pingou) on the current state of Fedora Infrastructure along with lots of upcoming plans.

If you aren’t able to make it to Kraków in person, do follow along on #fedora-flock ( on freenode ) and in blogs and other project communications.

June 27, 2016

Kevin Fenzi : Zodbot… upgraded

Kneel before the new Zodbot!

We have upgraded our beloved evil super villain IRC bot on freenode from an old version of supybot-gribble to a new shiny version of limnoria ( https://github.com/ProgVal/Limnoria ).  This doesn’t change much in the interface, but it does mean we are using something that is maintained and gets updates and is a good deal more secure. If you notice problems please do let us know with a Fedora Infrastructure ticket.

Also, as one of the maintainers of supybot-gribble in Fedora and EPEL, I will be retiring that in favor of the limnoria package very soon. It should be now available in updates repos for your upgrading pleasure.

Ursa: You are master of all you survey.

General Zod: [bored] So I was yesterday. And the day before.

June 01, 2016

Kevin Fenzi : What didn’t happen today?

Today, astute observers may have noticed that lists.fedoraproject.org and lists.fedorahosted.org didn’t participate in the long running, annoying, insecure and exasperating tradition called “mailman day”. This is where mailman spams you with a reminder for every list you are on with your plaintext “password” in clear email.

Since we have now fully switched over to mailman 3, you can relax and rest easy that we DID NOT spam you with your mailman password today, nor will we from now on. 😉

Our mailman3 instances authenticate users via Fedora Account System OpenID, persona or yahoo, no local throwaway passwords to look for or remember.

May 11, 2016

Kevin Fenzi : Mailman3: no topics support (with alternatives!)

In mailman2 there is a feature called ‘topics’. What this allowed you to do was create some filters on the list and when those filters matched mailman would add a header telling the user that this post was part of a ‘topic’ or not. You can then use the mailman interface to tell it you only wanted posts on a specific list of topics and no others. As far as I know there aren’t too many lists that used them, but there are a few we run in the Fedora Project that do: The prime example being the ‘package-announce’ list.

The package-announce list gets emails from bodhi (the Fedora updates system) about each update when it’s pushed into the stable updates repo. As you can imagine with a fast moving distro like Fedora, this list gets a LOT of posts. This list has topics for each Fedora release (22, 23, 24, etc) and also “Security” and “New package” updates. Using topics users could get a subset of these emails that they wanted.

Sadly, topic support isn’t yet available in mailman3, and it’s unclear right now when it might be implemented. Not wanting to hold up our migration to mailman3 forever, we want to go ahead and move lists like package-announce over as soon as we can. Of course this means those folks using topics are going to be in a bind. However, there are some alternatives:

  • You can switch to using a RSS feed directly from bodhi itself. For example: https://bodhi.fedoraproject.org/rss/updates/?releases=F23 will give you all stable Fedora 23 updates. https://bodhi.fedoraproject.org/rss/updates/?releases=F23&type=newpackage will give you all newpackage type updates, etc. Pretty much any search or page with a rss symbol on it on bodhi is an RSS feed.
  • You can set up notifications in the Fedora notifications service: https://apps.fedoraproject.org/notifications/ to either email or IRC. For packages you are interested in, all updates, security or newpackage updates or any combo.
  • You can of course just stay on the package-announce list and filter things locally. You will be getting emails about some things you don’t care about, but emails are pretty small in this day of streaming video. 🙂

Hopefully those interested can switch over to one of the alternative methods above until topic support once again lands in mailman.

May 04, 2016

Kevin Fenzi : Fedora account system and FreeIPA

Over the years, a number of times, people have asked us about migrating from our own custom Fedora Account System (FAS) to FreeIPA.

We would love to, but right now as things stand it’s just not a great fit for us. In the past we have had trouble articulating what the issues are, but now Patrick has written up a wiki page with all the items we know of:


Hopefully we can keep open a dialog with FreeIPA folks and keep the above wiki page synced up so perhaps someday we can migrate when blockers are no longer in our way. 🙂

April 16, 2016

Kevin Fenzi : ansible 2.0 going stable in f23/f22/epel7/epel6

Just a quick heads up: ansible- is heading out to the stable updates repos for Fedora 23, Fedora 22, EPEL7 and EPEL6.

There’s a few outstanding issues folks might hit, but we figured it was finally time to push to stable. If you use includes for handlers you might hit ( https://github.com/ansible/ansible/issues/13485 ). This one will be fixed in 2.1 by allowing you to add a config option to set includes as static or dynamic. Also, you might see some intermittent issues with some hosts saying “unable to resolve remote temp directory” ( https://github.com/ansible/ansible/issues/13876 ). Hopefully this will be fixed in, and there are some workarounds in the issue.

If you run into another issue, please do check if it’s reported upstream and if not report it.

We have also pushed out a ansible1.9 compat package for those folks that need to stick with 1.9 for now for whatever reasons. You can simply remove the ansible package and install ansible1.9 and you can use that version. It’s currently at 1.9.4, but there’s a 1.9.6 update pending updates-testing now also.

Happy ansiblizing!

April 03, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-04-03 edition

Well, it’s been a while again, so I thought I would update everyone on recent goings on in Rawhide, Fedora’s rolling development release.

Since my last post we finally moved over to having Rawhide and Branched (what will be Fedora24) done by the new pungi4. There are sadly lots of little issues that have been cropping up and getting fixed by Release Engineering, but it’s been a bit of a bumpy ride. As of this post, the last Rawhide compose that finished was the 2016-03-31 one, all the ones since then have failed. Hopefully this will get sorted out tomorrow. The current cause of problems is adding atomic/ostree to the compose (which we very much need/want), it’s just been more difficult that we first hoped. Once things settle down we will really be in a better place than before, with full composes (tested by openqa!) every night.

There was a bit of delay since the Branching event on signing, but last week we got back on track and (almost) everything in rawhide should be signed by the Fedora-25 key now.

With compose issues and general breakage in the install path it’s been difficult to install rawhide recently. Again, once we get the normal composes back on track this should clear up. As usual, once you have a rawhide install there hasn’t been really much breakage of the day to day use path.

Astute observers would have noticed that the rawhide-source mirrormanager repo wasn’t working after the switch to pungi4. I’ve fixed that up the other day and rawhide-source should be working normally again.