July 24, 2017

Kevin Fenzi : rawhide notes from the trail, the 2017-07-24 edition

Greetings! Once again it’s been a long spell since one of these posts, so lets jump right in and rope that calf.

Rawhide has been marching along to the next branching point, when Fedora 27 will branch off for it’s release. There’s a mass rebuild that should be happening very very soon now, there was a bit of delay while tools were all sorted out. Look for a very very big rawhide update soon once that mass rebuild finishes.

All the alternative arches are now in the same koji. We finally got s390x added in and going. Initally it was only 5 builders with 4GB ram, so things got stopped up from time to time, as all it would take was 5 long builds to show up and everything else waited for them. A few weeks ago we increased things to 15 builders and 8GB of ram each so hopefully that will keep up with builds as we go. We will see what the mass rebuild looks like with all the alternative arches included this time. I suspect it’s going to take longer than before, but not sure how much longer.

Thanks to Patrick you may also notice that uploading sources to our lookaside cache is no longer uploading them twice. Instead it’s using the kerberos cache to authenticate on the first go so it just needs to upload once.

We have missed some composes in the last few months (once for quite a while), and this is due to the compose process now failing the compose if all the release blocking deliverables are not there. On one hand thats nice because it means we always have those if the compose finishes, on the other it’s not so nice because if it fails we need to go fix whatever breakage is there.

Look for a new note soon, and ride safe out there!

May 15, 2017

Kevin Fenzi : CI and infrastructure hackfest 2017

I was hoping to write a bit of a summary for each day of the CI and Infrastructure hackfest last week, but there just wasn’t enough time to sit down and write up blog posts. Each day we started bright and early gathering in the hotel lobby at 8am, got to the Red Hat tower, got some breakfast and got to work until 5 or 6 and then we went and got dinner and back to the hotel to do it all over the next day. We definitely had some excellent discussions and got a lot of work done.

First a few thank yous: Red Hat provided us the use of the fittingly named “Fedora” room on the 9th floor to work in. It was a perfect size for our group. I think once we had to scare up an extra chair, but usually we fit just fine. OSAS (Red Hats Open Source And Standards group) funded travel and lodging for folks. Paul Frields (stickster) handled logistics and tried to keep us all scheduled, on track and in general cat herded in the right direction.

We had 3 BIG goals for this hackfest and a bunch of little ones. I think we did great on all counts. First on larger goals:

  • Monday we got a detailed dump of information about our authentication systems, past, present and future. Both from a perspective of sysadmins wanting to manage and fix things and application developers wanting their app to authenticate right. The high level overview is that we have FAS2 currently as the source of all our authentication knowledge. It syncs (one way) to freeipa (a pair of replicated masters). This sync can sometimes fail, so we now know more about what to do in those cases from the sysadmin side. Then we have ipsilon that manages openid, openid-connect and (used to) handle persona. We got some detailed workflows for each of these cases. Moving forward we want to get apps using OpenID-Connect. Down the road we talked about replacing fas with a very thin layer community access api app. Not sure thats going to happen, but might be an interesting way to go.
  • We wanted to harness the CentOS CI pipeline for testing a subset of optin packages from Fedora. I wasn’t directly working in this area, but I know by the end of the week we had CentOS-CI sending fedmsgs in our staging env and had setup a ci instance near it with resultsdb and taskotron to manage tests. I think there’s some more hooking up of things to go, but overall it should be pretty cool.
  • Finally we wanted to look into and setup our own OpenShift instance. We had some very nice discussions with Red Hat OpenShift folks who manage and deploy things bigger than we can imagine. They gave us some very helpful information. Then we talked out initial policy questions and so forth. By friday we had a OpenShift cluster up and running in our staging env. We still need to get some official certs, sort out the ansible managing applications setup end, and figure out what we want to do for persistent storage, but otherwise we made a vast amount of progress. You can find out questions and answers at https://fedoraproject.org/wiki/Infrastructure/OpenShift

On smaller items:

  • We met with the Red Hat Storage folks who manage our backend storage. There’s some nice improvements coming along later this year if we can adjust things to take advantage of it. Mostly that means splitting up our gigantic koji volume into a ‘archive’ and ‘active’ volumes and splitting our ftp volume into a ‘active’ and ‘archive’ volumes. Then we can hopefully move the active volumes over to flash storage, which should be a great deal faster. All still up in the air, but promising.
  • We met with the folks who manage our main datacenter, and it’s looking possible that we will be moving all our stuff to a new area a short distance away. If this comes to pass, it would be later in the summer and there will likely be some downtime. On the plus side we would get a chance to organize machines better, get new racks and better power and all around be in a better place moving forward.
  • We had a nice discussion around making bodhi faster. I’ve been over that many a time, but we did come up with a few new ideas, like generating drpms at update submission time instead of at mashing time (but that would need createrepo_c changes). Or perhaps triggering pushes only on critpath or security updates and other leaf node packages would just wait.
  • There were a number of discussions around factory 2.0, modularity, branching, koji namespaces, etc.
  • We had several discussions on postgres BDR (Bi directional replication). I’ve moved a number of our apps to it in staging and hoped to roll it out in production, but there were some concerns. In the end we decided to look at deploying it and then deciding on a app by app basis which ones were ready to move over. In the end we hope to have everything using it, but some apps need time to make sure they follow all of BDR’s rules and do the right thing. Additionally some apps may want to use BDR, but in sync mode to avoid possible bad data on a node crash. koji upstream needs to support things before we can move koji, but some of our smaller apps may be able to move soon. We also decided we wanted a script that could detect problems by looking at an applications schema. Pingou already went and had this done by the end of the week.
  • Tim Flink (Fedora QA) and I had a nice discussion about scaling the existing QA setup. We agreed that it might make good sense to see if we could migrate this into our cloud, provided we can get it up on a modern version we could actually support. That is likely to happen in the next few months as we have some new hardware for it and are retiring some of the old hardware, so it’s a good time to force an new setup.
  • We went to the ansible offices and had beer and pizza and watched a baseball game. 🙂

All in all a super productive week. Look for lots of the above to come up in meetings, tickets, mailing lists and so forth as we share plans we made, and make sure everyone is on board with them or if we need to adjust anything.

May 08, 2017

Kevin Fenzi : CI and infrastructure hackfest day -1

Ah travel, such fun, especially these days.

Happily things went pretty nicely… security was not a hassle, the flight was a bit early and went well and no problems meeting up with folks once I landed. Then off to a lovely dinner and sleep in time for the hacking to begin tomorrow.

May 06, 2017

Kevin Fenzi : CI and Infrastructure hackfest 2017 next week

Tomorrow I’m traveling out to Raleigh, NC for a gathering to work on CI and Infrastructure for Fedora and will be out there all next week. We will of course be around on IRC and hope to pull in remote folks that are interested in participating, but if you need us for something and can’t find anyone, please file a ticket and we will get back to you as soon as we can.

https://fedoraproject.org/wiki/CI_and_Infrastructure_Hackathon_2017 has a list of the things we hope to work on, but a short summary:

  • Get a bunch of information from Patrick on OIDC (Open ID Connect, basically the current spec for OpenID), both for application developers who might need to interact with it and sysadmins who need to manage it.
  • Work on a bunch of items related to Continuous Integration in Fedora, koji and bodhi integration and being able to use the CentOS CI hardware to run more tests.
  • Figure out a plan to setup a OpenShift instance in Fedora Infrastructure. This will help us deploy some new apps that expect that sort of setup as well as look at moving some of our apps to this new and exciting workflow. We have a big list of stuff to figure out at: https://fedoraproject.org/wiki/Infrastructure/OpenShift
  • Some misc other discussions about database replications and other things that high bandwith talks would help.

It should be a fun and busy week… I’m going to try and summarize each days discussions here, either daily or at the end of the week.

April 01, 2017

Kevin Fenzi : Rawhide notes from the trail, 2017-04-01 (no foolin!)

Just a few quick notes for those folks riding along the rawhide trail:

There’s some pretty bad brokenness in the polkit package that landed in rawhide on thursday. Your best bet is to downgrade to polkit-0.113-7.fc26 until it gets sorted out. Symptoms include not being able to login with gdm, lots of things being broken because they cannot start, etc. This is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1438086

FESCo approved the Fedora 27 schedule not log ago: https://fedoraproject.org/wiki/Releases/27/Schedule and so the next mass rebuild will happen 2017-07-12. Mark your calendars.

As many of you may know, I switch off between gdm/Gnome and lightdm/Xfce pretty often. I finally tracked down one annoyance that has been hitting me for a while: On login to Gnome I was getting 2 polkit dialogs every time: asking for perms to handle rfkill and networking. Turns out this was due to blueman autostarting in Gnome and not having those permissions. There was no reason for it to need to start there since Gnome has it’s own built in bluetooth, so I filed: https://bugzilla.redhat.com/show_bug.cgi?id=1432555 on that. Workaround is to add a NotShowIN Gnome for it’s autostart file.

Until next time, ride safe!

March 25, 2017

Kevin Fenzi : MUA++ (or on to thunderbird)

So, a few weeks ago I moved my MUA (Mail User Agent) from claws-mail over to evolution. Last week I decided to move on and give thunderbird a try.

Mostly evolution worked, but it had some bugs I hit that were quite annoying: From time to time it would duplicate emails on my work imap server. Suddenly I would get 50 copies of some list post. I could of course select them and ‘delete duplicates’ but this was pretty anoying. I tried all sorts of tuning to get it to not do that, but nothing seemed to work. Additionally I found the keyboard shortcuts difficult to get used.

So, thunderbird. For this I needed to make a change I had been meaning to for a long time, but kept never getting around to: I needed to switch to using my home server’s imap server instead of delivering email to my laptop (thunderbird only does imap for incoming emails). Fortunately, it was easy to just change my .procmailrc to deliver to the server and serve it via imap. However then I ran into some real confusion: I had setup my server (dovecot) a number of years back to provide 2 ‘namespaces’ in imap: The first would be a Mailbox that it would deliver email to, and the second would be a Maildir that it used for folders (this was due to having some friends using my mail server who insisted on using mail clients via shell that didn’t understand Maildir). I had to do a bit of tweaking to get it working for me, but not breaking it for others. The mbox namespace also meant there was a mail directory with mailbox folders and if you were not careful how you set things up you would get those (mailbox is a good deal less permofrmant than Maildir, so I wanted to avoid them). Finally, I got it all working there.

So, after now using thunderbird for a week or so, the good:

  • thunderbird has no problems talking to various imap servers. No duplicate emails, no errors, everything works nicely and pretty quickly.
  • lightning plugin is now built in/included in thunderbird and it has had no problems talking to all my vaious calendars.
  • enigmail seems to do a fine job with encrypted emails and signing my outgoing emails.
  • The keyboard commands seem a lot easier to get used to, and with the Nostalgy extension it’s pretty easy to file emails and go to places.
  • The search features seem very fast and work well. I ‘star’ mails I want to deal with later, and have a search folder that shows all starred emails. I can from there easily open a tab with the entire conversation if I want to read the thread the email was in again.
  • There’s a handy sort by ‘Grouped’ thats nice for some things. It will show you for example todays emails and let you expand previous days if you like.

The bad:

  • I cannot quite seem to get the message view to look the way I want. It seems to change what fonts it uses sometimes based on “I am not sure what”. Possibly if the email is html only? Will keep looking into it.
  • I had to enter all my stupid filtering rules _again_. I just redid them for evolution, but now again for thunderbird. I really need to look into sieve and just doing it on the server. There outta be a standard!

and the things that are just related, but not directly thunderbird:

  • My mail is now in imap on my main server and I can read it via thunderbird, or roundcubemail.
  • I’ve unsubscribed or otherwise removed myself from a bunch of lists or things that were sending me email that I never read anymore or cared about. There’s still some more of these to go, but its good every once in a while to drop all your filters and rebuild them to see what should just never come in at all.

Will I stick with thunderbird? Time will tell. So far this week indications are good, but we will see.

March 13, 2017

Kevin Fenzi : Email and RSS and their readers and writers

I’ve been using claws-mail to both read/compose emails and to read all the vast pile of RSS feeds I try and keep up with for quite a while now and it’s for the most part been fine. I never liked claws-mail use of mh folders instead of something more standard like Maildir, and it’s calendar integration is… not there, but it did a pretty good job.

Unfortunately, claws-mail uses a plugin called ‘fancy’ to render html (which sees heavy use when loading rss feeds). This plugin uses the old old webkit1 (webkitgtk package in Fedora). This package hasn’t been maintained upstream in quite a long while and the number of vulnerabilities in it has just grown and grown. Other distros have dropped it entirely, and Fedora is finally following suit very soon. This has caused the Fedora claws-mail maintainers to drop the fancy plugin to prevent dropping claws-mail entirely. However, without that plugin, RSS reading is… not very pretty.

So, after seeing Jiri Eischmann’s recent post ( https://eischmann.wordpress.com/2017/03/10/nextcloud-linux-desktop/ ) about using the nextcloud News app, I decided to give that a try. Installing was pretty easy, but note that it’s not in stable apps yet. Importing my feeds from claws-mail’s RSSyl was pretty easy, but the only want to put feeds in categories is drag and drop. Drag and drop when you have many many feeds is a real pain, I really wished for a ‘move to folder’ option. The Fedora “feedreader” app looks pretty nice and after a bit of tweaking handled things pretty well. The android OwnNews app also seems pretty nice and functional. I did notice some times when it seemed all three readers were not 100% in sync as to what was read or not, but things stayed pretty close. Being able to use my phone to browse feeds will be handy. I also took this time to cull a bunch of feeds that didn’t have any posts in the last few years or I didnt care about anymore or the like. Overall I think this will be a pretty reasonable replacement.

Next up was mail. I could have just stuck with claws-mail for just mail, but without the fancy plugin any html emails would not render really at all. So, I figured I would bite the bullet and look at evolution again. I switched over my work and gmail accounts first as they use imap and state is (mostly) remote anyhow. They setup pretty easily. Next I added a account that was all my old claws-mail folders in mh format. In the past when I tried this evolution crashed, but this time it seemed to handle it ok. I later disabled this account for now, as it caused fetching new emails to take a while and I only need it if I need to find an old email and can enable it then. Finally I switched over my main mail stream. Unfortunately, I couldn’t find a way to import my old claws-mail filters, but this did give me a chance to cull emails that I get. As things came in saturday and sunday I added filter rules, unsubscribed, or otherwise made a note to stop whatever it was sending me email. There’s definitely some oddities I will have to get used to with evolution, but overall I think I can make it work. The biggest frustration is that the shortcut keys to do things don’t seem to work for me. (ie, . and , are supposed to go to the next or previous unread message, but they do not here. They don’t do anything). There doesn’t seem to be a way to quickly go to a particular folder. After a bit of use, I figured out the problem: The shortcuts are focus specific, and I don’t really see any indicator which pane I have focused. ;(

Right now I think the RSS change is likely to stick, not entirely sure on evolution, but if it doesn’t there’s a number of other clients to try out (thunderbird, mutt, or back to emacs 🙂

March 05, 2017

Kevin Fenzi : Rawhide notes from the trail, the 2017-03-04 edition

Well, branching of Fedora 26 off rawhide has come and gone, and it was a bit of a rocky ride this time sadly.

The branched composes were failing at first, then working, but not actually syncing to the master mirrors in order for mirrormanager to notice it and people to you know, actually use it.

This time we tried to make sure both branched f26 and rawhide were signed the entire time by the right keys. They currently are, but sadly, there’s a old/obsolete/bad/nolonger used/no good/varmit of a Fedora 26 key thats in the fedora-release-25 package. So if you are trying to upgrade you may need to find/get the right f26 key before doing so. Also, the resigning meant that mirrors had to pull basically every package again, so it’s going to be a bit before everyone is synced up on rawhide and f26 content.

If you want to keep following Fedora 26 to it’s release, you need to run a ‘dnf repo-sync –releasever=26’ to make sure you have the right releases and repos. If you want to stay riding on the rawhide trail, you can just dnf update and should get the fedora-27 release and repos.

Until next time, ride safe.

February 23, 2017

Kevin Fenzi : Rawhide notes from the trail, 2017-02-22

Some recent gotchas in rawhide:

  • firefox doesn’t seem to compile correctly with gcc7 (and the optimization levels Fedora uses by default). The current rawhide version will install and run fine, but looks horrible. As a work around you can install a flatpak or a binary version from upstream or downgrade to firefox-51.0.1-2.fc26 from koji. This is tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1422532
  • A few weeks ago, python2-jinja2 was updated to 1.9.4, then 1.9.5. Unfortunately 1.9.4 broke ansible templating, and all the problems were not fixed by 1.9.5. As a consequence, ansible hasn’t been runable the last few weeks. Until today: I pushed ansible- into rawhide. This isn’t the final, but it hopefully fixes the jinja compatibility issues and gets rawhide ansible users back on track.
  • After my last post about composes failing due to an unsigned package, we ran into 3 other issues we had to fix to actually get a compose working: First, an nss update caused lorax to break. Turns out it was due to the mock chroot used not having /dev/urandom and nss noq requires that to be there. Next was the rdma-core package breaking things. It was untagged a while back to fix, but then the mass rebuild rebuilt it again and it got pushed out again. Finally policycoreutils-python started pulling a ton of more deps which broke the minimal installer chroot.
  • Even after we now have composes, livecd’s are not working due to an anaconda bug: https://bugzilla.redhat.com/show_bug.cgi?id=1425827 Hopefully we will have a fix for that soon and livemedia will be back.
  • There was a pungi bug preventing the ostree composes working, there’s a proposed fix for that one.
  • i386 images and media are all failing due to a weird dep solving issue where it says “will not install src.rpm”. https://bugzilla.redhat.com/show_bug.cgi?id=1416699

Lots of bugs, but we are moving forward on quashing them now. I am hoping we have some good composes later this week and then we can try and keep them that way.

February 20, 2017

Kevin Fenzi : Fedora macbook pro testers++

In the final run-up to the Fedora 25 release, we slipped a week because there was a bug in installs on apple osx (now macos again) hardware. This was (and is) a use case the Workstation working group cares about, as they would love for folks with apple hardware to install Fedora and use it on that hardware. Sadly, we don’t have too many testers with this hardware to help our testing cycles, and many community members with this hardware also are using it day to day and cannot afford to reinstall and test at the drop of a hat.

I use my personal yoga 900 for my main machine, so my work laptop has always been for me a test machine or a backup in case of failure. There are a number of reasons for this: I prefer to pick and use laptops that aren’t standard corporate offerings, I like to know that I can do anything I choose with the laptop, and if (god forbid) I moved to a new job I wouldn’t have to give my primary laptop back.

So, when I was up for laptop refresh and I saw that a macbook pro 13″ model (12,1 apparently) was available, I decided to choose that and help out with Fedora testing efforts on this hardware. I do feel a bit bad about this as I am not a big fan of Apple and this does mean giving them money, but on the other hand, hopefully I can test Fedora and help avoid slips due to lack of hardware. Also, I can run rawhide on it and see if I can get everything working fully.

The macbook arrived the other day and yesterday I unpacked it and did some initial testing:

First, the hardware: Seems nice enough, but I am not sure why anyone would get one of these over a Dell XPS 13 or a yoga 900/910. It’s got a lower resolution screen than those, 8gb memory instead of 16, and only an i5 cpu instead of an i7. The feel is very solid, but it makes it just makes it seem too heavy to me. Otherwise the screen is bright and nice, the keyboard backlight is nice (The yoga 900 only has off, dim, bright for the keyboard backlight, but this macbook you can set it to whatever you like), the power connector is neat in that it has a light on it telling you if it’s charging (orange/amber) or fully charged (green). The keyboard and trackpad seem fine.

I decided I would go through the normal macos setup first and then try and setup Fedora to dual boot, as I imagine that would be a common setup. Part of the setup instructions that came from my corporate overlords was mention of enabling File Vault full disk encryption. So, I did that and got everything installed and seemed to be working fine (at least as far as I could tell, not being a macos user normally).

Still following what I would think would be the more traveled path, I went to https://getfedora.org and downloaded the Fedora Media Writer. Download went fine and it was no trouble to run it, but it did give me a warning that this was something I had “DOWNLOADED” from the internet. I don’t guess we have much way around that warning as we are signing the binary fine and it’s not saying it’s unsigned, just that it was downloaded and are you sure you want to run it. Download and burn to usb went just fine, no problems at all with FMW. It might be nice if there is an option to download Rawhide images if you really wanted them.

Hold down the option key and power on and you get the boot selector thing. Choose FedoraMedia and there’s the Fedora Live USB. Everything booted up nicely and I poked around to see what was working or not working. Turns out so far the only thing that doesn’t appear to be working out of the box is the webcam. It seems to be a broadcom model that some folks are working on reverse engineering, but haven’t gotten that done enough to merge into the mainline kernel ( https://github.com/patjak/bcwc_pcie ). I might try that out sometime down the road. The power connector activity seems a bit odd as well: when you plug or unplug the power it seems to take a minute or two before it notices and starts updating.

Next I pulled up the Anaconda installer and looked to install Fedora along side the existing OS. I needed some space to install in, so I selected the largest partition which I knew was the main macos volume (but oddly was showing up as Unknown), and told anaconda to shrink it. That seemed to work fine, and I installed Fedora in the newly freed space. The install finished fine and I rebooted.

On reboot, I now got grub and Fedora booted and worked fine. The macos entries however, errored and didn’t work at all. Turns out this is a long standing, known bug: https://bugzilla.redhat.com/show_bug.cgi?id=893179 I tried various things suggested in the bug to chainload, but with no luck. Luckily you can still hold down option and get the native bootloader.

So, I did that and tried to boot the macos install, but it would think for a while and then error with a picture that was a circle and crossbar and fail to boot. After poking around it seems I now hit: https://bugzilla.redhat.com/show_bug.cgi?id=1033778 ( https://fedoraproject.org/wiki/Common_F24_bugs#apple-core-storage-wipe ). So, my macos partition was unknown to anaconda, so it let me shrink it, but it messed it up completely and nuked all data that was on it. Had I not encrypted things it would have worked, but since I did it was gone.

Luckily apple has the advantage of controlling the hardware platform in this case, so I just needed to boot with option command r to get a network rescue. This let me wipe the drive, repartition it, reinstall macos, then boot the Fedora USB again and install Fedora in the space I left for it. So, it wasn’t the end of the world, but it would sure have been anoying if I had data there.

After all that installing, I moved the Fedora install from F25 to rawhide. No issues there, everything seems to work and be fine after that.

So, if we could somehow fix the two bugs I ran into for f26 I think it would help macbook folks a fair bit. If anyone needs me to install or test anything on this platform, just let me know. I plan to keep the macbook ready to (re)install most anytime and hopefully can provide more test coverage for upcoming cycles.

February 19, 2017

Kevin Fenzi : Rawhide notes from the trail: 2017-02-18 edition

Greetings everyone, lets dive right into the latest changes in the rawhide world:

  • The Fedora 26 mass rebuild ran and finished last weekend. 16,352 successfull builds happened, along with around 1,000 that failed to build. Now we have a few weeks until we branch f26 off to fix things up.
  • The mass rebuild did disrupt signing of normal updates. Perhaps next mass rebuild we should look at standing up another set of signing servers to just sign the mass rebuild.
  • Composes for the last few days have failed. Turns out its due to an unsigned package. But how could that happen? We passed all the builds through the regular signing process. Turns out when builds were tagged in, there were a few builds that overrode newer versions already in rawhide, so releng ran a custom script to retag newer builds back in. However, there was a package where the maintainer built a new version, decided for some reason it was unusable and untagged it. Thats fine, but the custom script mistakenly tagged this “newer” build in and it was long enough ago that it’s signature was removed. Just a short note here about “newer”: koji has no concept of package versions. To koji if you ask for all the ‘newest’ builds in a tag, it will give you the most recently tagged ones. This importantly has nothing at all to do with the package-epoch-version-release, thats just not on a level koji knows or cares about.

February 13, 2017

Kevin Fenzi : A tale of a bluetooth headset bug…

I find tracking down obscure bugs interesting, so I thought I would share another tale of one I just tracked down.

Ever since coming back from the holiday break this year, I found my bluetooth headsets had stopped working. (I have 2 very different models: a plantronics M50 that I have had for a long time, and a Bose QuietComfort 35 (yes, the headphones have a headset profile!)).

At first when I noticed the problem I tried the obvious things:

  • Downgraded pulseaudio
  • Downgraded bluez
  • Downgraded and tried various kernels.
  • Looked though tons of upstream pulseaudio and bluez bugs.

No change at all. I added myself to a existing bluez bug and provided the bluez maintainer a bunch of debugging output, but it all looked pretty normal, there was just no sound. So I gave up and moved on to more urgent things.

Fast forward to today: what better way to spend a sunday afternoon that more debugging. I repeated my former tests and had the same result. Then I decided to try some LiveUSB’s to isolate it. I downloaded and booted a F25 updated workstation iso and it still had the issue. So, I thought, Ah ha it has to be the updated pulseaudio since that landed in f25 updates. But to be sure, I downloaded and booted the stock f25 release iso. Still had the issue. Boggling. Then I went looking around in the kernel bugzilla and found a report that talked about issues with bluetooth and firmware. That was just the hint I was looking for. So, I downgraded the linux-firmware package and tried things and… it still didn’t work! There was one more trick: The firmware only reloads on cold boot. If it’s already loaded (or misloaded) it doesn’t even try again. So, cold boot and then everthing started working.

The key kernel boot message on the failing firmware: Bluetooth: hci0: Failed to send Intel_Write_DDC (-22) and the working firmware: Bluetooth: hci0: Applying Intel DDC parameters completed

For bonus points of course the failing firmware file is: ibt-11-5.ddc and the working one is: ibt-11-5.ddc (yes, thats right, they are changing the file around without changing the version any).

Next of course is to figure out where to report this to get it fixed, but some poking around and testing found that http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/commit/?id=581f24500138f5e410d51ab63b205be9a52f4c77 already had a fix. I just need to wait for a update to the linux-firmware in Fedora and everything should be back to normal.

So, troubleshooting lessons for today: Sometimes cold boot matters. When you think you have downgraded everything that could cause a problem, remember firmware. Reading around on bug reports will sometimes give you ideas on how to solve yours.

February 07, 2017

Kevin Fenzi : Rawhide notes from the trail, the 2017-02-06 edition

Some more notes from the rawhide trail…

Astute readers may have noticed that there was a Mass Rebuild scheduled for last week ( Feb 1st) and it’s not yet happened. There were a few reasons for the delay: The gcc folks wanted a few more days to quash some more bugs in gcc 7.0 and additional time was allocated to a change in how debuginfo packages would be generated. Hopefully this rebuild will fire off tomorrow (the 7th) and result in a 1 week push in the Fedora 26 schedule. Just a reminder about Mass rebuilds: These are done in a side tag by an automated release engineering script. It simply uses the rpmdev-bumpspec script to increase the release tag by 1 and add a changelog entry about the mass rebuild and then initiate a build. Packages that fail to build will have FTBFS (Fails to build from source) bugs filed on them after the mass rebuild is over. There’s two weeks alloted to cleaning up and fixing these FTBFS bugs and any other fallout from the Mass Rebuild. If you are using rawhide, note that there will be a day after the Mass rebuild where most all of your packages will be updated. Allot extra time to download and upgrade on that day. The last Mass rebuild took around 1.5 days to complete. This one may be slower due to all the additional alternative arches we now have.

Another thing that needed to land before the Mass Rebuild was the targeted boost rebuild. This has completed and the side tag merged back into rawhide. Tomorrow’s rawhide should have a bunch of packages that use boost updated. There will also be a few that still need fixing for the new boost version by their maintainers.

After the mass rebuild is over and cleaned up after, we have the branching off of Fedora 26 from rawhide on the 28th, and then rawhide will march on to Fedora 27.

January 21, 2017

Kevin Fenzi : kojipkgs: what it is and how it works

The last week or so I have spent a ton of time on kojipkgs (sorry again for any build failures this may have caused), so I thought it would be good to outline what it is and how it’s used and finally the working setup we have now.

kojipkgs is a concept that koji has of a host/url to go to to download packages. On small koji installs this host/url can be simply the koji hub host. Or it can be a seperate host or hosts, as long as it has access to all the packages koji wants. So in practice this means it has to share a NFS mount or other shared storage with the koji hub.

For many years Fedora infrastructure had a kojipackages instance that NFS mounted the koji storage, ran a apache web server locally on port 8080 and then had a squid instance in front of that, listening on port 80 and 443 and talking to apache for any items that were not in it’s cache. This worked out well because squid could cache commonly used objects and save access to the NFS server as well as return more quickly. However, it also has drawbacks: It’s a single point of failure and it’s exposed to the outside world and squid’s SSL code has not gotten anywhere near the same scrutiny that apache’s has.

A bit of a side note, but related, lets talk about how koji does builds for a minute. If you tell koji to do an official build of some package, first it makes a mock chroot and downloads and installs the packages in the srpm-build koji group (you can see these and what packages are in them with a ‘koji list-groups’ command). Note that this install is done with the HOST dnf into the chroot. The src.rpm is then built from pkgs git and lookaside cache and then koji starts builds for any needed arches. These builds first download the src.rpm via a koji urllib2 call and the make a mock chroot and download and install all the packages in the ‘build’ group. Then rpmbuild is called, etc.

So, week before last I updated builders (as I often do) and also upgraded the kojipkgs01 squid server (running rhel7) to the latest versions. However, we started seeing downloading problems at build time. Sometimes (but rarely) dnf would just fail to download a package it needed and fail the build. I figured this would be a great time to try and remove the single point of failure of that one server.

I spun up a second kojipkgs instance (kojipkgs02) setup just like the first one (rhel7, squid, apache, NFS). Then I setup our two proxies in that main datacenter to handle kojipkgs requests and use haproxy to send them in to kojipkgs01/02. Then, a switch of dns and it was switched over. Now, however we hit a new problem. Sometimes the src.rpm download via koji was not working. It was silently downloading only part of the src.rpm and then failing the build. Additionally, the build download issues were still happening. I tried isolating to just one proxy. Just one backend. Various haproxy balancer setups so that requests that went to one backend kept on that backend. I replaced both kojipkgs instances with Fedora 25 (much newer squid version). Koji has the ability to pass several urls to the mock setup it uses for builds, so we passed it multiple urls to kojipkgs there and that seemed to get rid of that problem, but the src.rpm download issue was still there.

Finally, we tried disabling squid’s “smp mode”. Suddenly everything started working nicely.

So in the end of this saga we have:

  • Nice Highy Available kojipkgs. (2 proxies using 2 backends, so we can update, restart, reboot a proxy/backend anytime we like without causing problems).
  • A bug report on librepo to retry downloads the number of times it claims to in dnf.conf default even if there’s only one baseurl and it was slow/timed out: https://bugzilla.redhat.com/show_bug.cgi?id=1414116
  • A koji ticket to make the src.rpm more robust: https://pagure.io/koji/issue/290 and some PR’s to change it use requests and validate the downloaded rpm already.
  • I still need to file an upstream bug on squid about smp mode, but I don’t have too much info to give them (there were no errors anywhere, just some small number of connections would hang/not finish).

January 05, 2017

Kevin Fenzi : Tools I use daily

I recently saw a post from my co-worker Kushal Das: tools-i-use-daily and I thought it would be fun to share the same information:

Operating System

I use Fedora rawhide on my laptop, which is my primary client machine. I don’t have a desk and desktop and monitors, just the laptop and a comfy chair. At home I have two dell CS24 servers (one is a test instance and has all my test instances for Fedora maintainers on it, the other is my ‘production’ server and is the firewall and hosts the primary server vm that handles all my email and other services for scrye.com). I have various small arm devices, some of which are on and some of which are off at any given time.


Since I run rawhide and update and reboot often (usually 1x a day during the week), I usually switch off between Xfce and Gnome / Fedora Workstation. I do admit Gnome has been growing on me over time and I might give it the edge these days, but Xfce is like an old friend that I can easily also do all my work in. I did play around some with sway (wayland based i3 clone) over the break, but it seemed like it would be too much work to build up a config that would be fully usable.


I still use a pretty crazy mail setup, but it keeps working so I don’t replace it. Mail comes into my main server (using postfix/sqlgrey/spamassassin, etc) then goes to UUCP over vpn (openvpn) tcp to my laptop. Then, on the laptop I use claws-mail to read the email which is locally delivered by postfix. You may think I am completely crazy to use UUCP, but it really fits this use case nicely: I can compose and send emails on my laptop even if I don’t have any internet, they will simply queue and send the next time I connect. Emails wait for my laptop/vpn and then deliver. I can set larger emails to smaller priority so I get the smaller ones sooner. It resumes just fine. It handles slow links just fine. I keep meaning to just switch to imap, but it ends up not really being enough win for me to care. 🙂


vim all the way. I used to be a xemacs/emacs user, but I haven’t touched either in years. vim is just so nice and quick and works everywhere there’s no contest.


firefox now. I was using midori full time, but upstream has really not had much time of late and things have gotten pretty dire. I did move the Fedora packages to webkit2 (via an unreleased upstream branch), but many things still don’t work right there. I would love to go back, but it would need some real releases and people working on it.


hexchat. Anyone out there still using xchat really should switch. I have a znc server on my main server at home I connect to.


wordpress. It’s just easy and works. Sure it has security issues from time to time, but as long as you keep updated you are usually in pretty good shape. I use the Fedora packages and they tend to do well at keeping up to date.


Just a few misc items: I use pass all the time (password-safe), it has most of my passwords in it. ssh, tmux, task (taskwarriror) and ansible are also things I use all the time.

December 28, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-28 edition

Just a few rawhide notes to end out the year:

  • Python 3.6 rebuild completed (mostly) and was merged back into rawhide. There were a number of things that needed cleanup, but we seem to have gotten them all handled in a few days. Next time we should really allow a day or two before we merge the side tag in and fix up things that are important for composes before we merge into rawhide.
  • Astute folks may have noticed: Builds in koji now will complete on all arches, even if one fails. In the past if one arch failed, the rest were canceled and the build came back as failed right away. Now all the arches will complete before the failure comes back. This allows you to see not only the problem that failed the build, but any other problems it would have hit on other arches as well. This should be of help especially to larger builds.
  • There is still a bit of fallout from the rawhide always signed changes we need to fix up in the new year: chain-builds no longer work, the rawhide fedora-release needs to be modified to always expect repos are signed, and we need to discuss what sort of testing we want to add in now that we have a handy place for adding it.

Have a smooth ride the rest of the year, no matter where your trail takes you.

Kevin Fenzi : 2016 Holiday hacking (so far)

I hope everyone is having a lovely holiday season and doing the things they find fun or recharging.

I’ve finally gotten to a few hacking projects that I thought might be interesting to share with others:

  • I finally cleaned up my local ansible repo some and set it up as a pagure project: ( https://pagure.io/scrye-ansible/ ). It still needs a bunch more cleanup (these playbooks were some of the first I ever wrote, so many years ago), but they should help anyone who wants to setup test machines like I have here or provide some more ansible examples.
  • For 4+ years I have had a CentOS6 digitial ocean droplet I have been using as a secondary nameserver for my domains. It actually predates IPV6 support at digitial ocean and had a manually setup he.net ipv6 tunnel on it. I finally looked at the ansible digitial_ocean module and setup a new shiny Fedora 25 droplet, created entirely via ansible as well as putting my secondary nsd configuration in it. This is all now in the above scrye-ansible repo. I may well be using more digital ocean droplets now that I see how easy it is via ansible.
  • I looked around for a new rom for my phone (nexus 6) and didn’t find much that filled me with joy. So, for now I just moved to the latest nightly CM version and will see if this new fork ( http://lineageos.org/ ) pans out. The other roms out there are really unprofessional looking: (homepage is a xda forum post, binaries have no checksums and download from bigfiles.com or the like, last commits to source repos was years ago, etc).
  • One big issue I had had with my phone of late is it quitting when it has 30-40% battery left and suddenly saying it has 0%. I found that you can reset the battery calibration (boot with power, down volume, select ‘bootloader logs’, press power for 10+ seconds until the phone reboots). That seems to have gotten things a good deal happier now, but time will tell. Unfortunately, the battery on the nexus 6 is not very easily replaceable.
  • I got all my home machines rebooted into the latest update kernel. I sometimes delay on my firewall/gateway machine because it means a short outage, but everything is updated now.
  • Pulled all the old backups off my phone and freed up a fair pile of space. There’s no reason to store them there, I can copy them back if I need to restore something from them (which I never really do after the initial restores).
  • Setup usbguard on my home machines. The chances of someone plugging in a evil usb device to them is small, but no harm in setting it up and denying all the non known to me devices. I’ve been using it on my laptop for a long while now.
  • Updated my drop-keys (gnome) and xscreensaver-keys (Xfce) scripts and pushed them to the pagure repo: https://pagure.io/scrye-ansible/c/37610c85918125d92bedff6673b58310605e90dd?branch=master basically these scripts can watch for the screen to lock and drop all your unlocked ssh keys. For gnome there’s an upstream bug that might make this land in upstream gnome-keyring: https://bugzilla.gnome.org/show_bug.cgi?id=735149

In the next few days I hope to catch up on some packaging backlog and perhaps get some armv7 installs done.

I hope everyone is having an enjoyable, relaxing holiday!

December 18, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-18 edition

Hello from the Rawhide trail.

With the recent Flag day (on Dec 12th), we switched all rawhide builds to allow us to sign (and hopefully eventually test) all packages. Here’s how it works:

  • Your rawhide build used to just be tagged into the f26 (currently rawhide) tag. Now, it tags into the f26-pending tag instead.
  • The autosigner sees the build, signs it and moves it to the f26 tag for the next rawhide compose.

Unfortunately, there is a backlog of packages we needed to sign, many from the ppc{le} bringup, so thats why there hasn’t been any rawhide composes the last few days. The one today should be out later today however, and we should be back on track from there.

So currently we are just signing things at the $release-pending tag, but we would like to try and start doing some automated QA there at some point. Nothing that will hold up builds for too long, but something that will catch obviously broken builds from landing. Now that we have everything otherwise in place we can start figuring out what we want to run there.

Also, coming soon to rawide will be the first rebuild of python packages for the upcoming python 3.6. Hopefully that will all be smooth sailing, but a number of package updates will land for that soon.

December 03, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-03 edition

Well, it’s been a while but I thought I would share news from rawhide both from the previous month or so and upcoming.

With the release of Fedora 25 a number of developers and testers are switching back to rawhide in the run up to the Fedora 26 branching event. Welcome back to the trail!

In the last month there’s been various small rawhide issues:

  • openssl 1.1.0c came out and had a number of issues. In particular it broke authentication to koji, so any package maintainers had to downgrade back. It also broke a bunch of tests in python, which has delayed the landing of python 3.6 in rawhide. There’s a patched version of 1.1.0c that landed last week, so everyone should be ok to update openssl again.
  • The latest systemd version landed, but then had to be untagged again because it broke a number of parts of compose. Hopefully those will get worked through and fixed.
  • More recently (The last week) a kf5/qt5 update is in progress and isn’t fully finished. Rawhide users will notice a number of packages dnf says it’s skipping due to broken dependencies. This should get fixed up soon, so just keep skipping those items.
  • Finally, the last few days we have run into an odd problem that looks like it might be related to RHEL7.3 squid (which is whats on kojipkgs.fedoraproject.org and used for package downloads for builds). Sometimes packages just don’t seem to download and it breaks the compose, but testing later they work fine. We downgraded squid back to the 7.2 version for now and composes seem to be working again.

Finally in exciting upcoming news: There’s a “flag day” on december 12th (2016-12-12) for a bunch of changes that release engineering has been wanting to make for a long time. One of these changes will finally allow us to do the last thing we need to do to get a 100% signed and gated rawhide. There will be a change in fedpkg (as well as some koji server changes) that will make rawhide builds land in a ‘f26-pending’ tag instead of just going directly to ‘f26’. Then, our autosign process will sign them and retag them to f26. We also can add in automated QA here to decide if a build is ready to be tagged in or not. I hope next year that we can add a bunch of tests here and make rawhide more and more stable.

October 24, 2016

Kevin Fenzi : Security Score Card For Fedora Infrastructure

Josh is asking folks to send him their security score card via twitter. Since I’ve been trying to blog more and like pontificating, I thought I would respond here in a blog post. 😉

There’s 4 parts to the scorecard:

  1. Number of staff
  2. Number of “systems”
  3. Lines of code
  4. Number of security people

For Fedora Infrastructure, some of these are pretty hard to answer, but here’s some attempts:

  1. Fedora Infrastructure is a Open organization. People who show up and start doing things are granted more and more permissions based on their merit. Sometimes people drift away to other things, sometimes new people show up. There’s some people employed by Fedora’s primary sponsor Red Hat, specifically to work on Fedora. Those account for 3.5 sysadmins, 5 applications developers, 2 release engineers, and 2 design folks. Specific areas will have potentially lots more community folks working on them. So, answer: 13-130?
  2. This one is easier to quantify. We have (almost) everything in ansible, so right now our ansible inventory + some misc non ansible hosts is around 616 hosts.
  3. This is another one thats difficult. We have a lot of applications (see https://apps.fedoraproject.org/) Some of them are just upstream projects we have instances of (mediawiki, askbot, etc). Others are things where we are primary developers on (fedocal, pagure, etc). It would be a fun project to look at all these and count up lines of code. Answer: dunno. ;(
  4. If this is full time security people working only on security issues, then 0. We do have a excellent security office in Patrick who is super smart and good at auditing and looking for issues before they bite us, but he’s not doing that full time. Others of the sysadmin teams do security updates and monitoring lists/errata and watching logs for out of the ordinary behavior, but thats also not full time. So, answer: 0 or 1 0r 3?

So, from this I think it would be nice to have a better idea of our applications (not lines of code), but just where to keep track of things better and who knows that application. It would be awesome to get some full time security folks, but I am not sure that will be in the cards.

I’d like to thank Josh for bringing up the discussion… it’s an interesting one for sure.

October 23, 2016

Kevin Fenzi : Another Fedora cycle, another painless Fedora upgrade

As we near the release of Fedora 25, as always I upgrade my main servers to the new release before things come out so I can share any problems or issues I hit and possibly get them fixed before the official release.

As with the last few cycles, there was almost no problems. The one annoying item I hit was a configuration change in squid. In the past squid allowed you to have a ‘http_port NNNN intercept’ and no regular http_port defined, however the Fedora 25 version ( squid-4.0.11-1.fc25 ) fails to start with a cryptic: “mimeLoadIcon: cannot parse internal URL: http://myhost:0/
squid-internal-static/icons/…” error. It took me a while to find out that I needed to add a ‘http_port NNNN’ also now.

Otherwise everything went just fine. Many thanks to our excellent QA and upgrade developers.

October 22, 2016

Kevin Fenzi : some IOT ideas

With the massive denial of service against a large DNS provider (in turn causing a lot of other things to have outages) on friday using a network of insecure IOT (Internet of things) devices (mostly cameras), a lot of folks are thinking about how to address IOT problems. There are a lot of problems and no easy answers, but I thought I would throw out a few ideas here and see if any resonate with others.

First, the problems: IOT devices are insecure and easily taken over for denial of service or other attacks, there is little to no economic incentive to make them more secure, consumers largely don’t care as long as they are still doing whatever their main function is, devices seldom get updates and when they do only for a short time before the company that made them moves on to something else.

Some ideas:

  • Make things vastly more cheap. Yeah, thats right, more cheap. To the point where you can dispose of or toss in recycling the device that just went out of support or was found to be used in some botnet. Or 3d print a new device. Of course this is not going to happen for a long while.
  • Make updates very reliable. I think we can do this with something like ostree/atomic + some wrapper hardware. Upgrade, reboot, if things don’t come back in X minutes, reboot back to the last working version and send for help.
  • Get IOT devices to use mainstream open Linux distros. These would provide source, upgrades, and device makers wouldn’t have to care about not supporting things. This would probibly take legislation and a big push for Linux distros to cater to IOT devices, and there would likely still be some hardware specific code (but it could be open and maintained in distros).
  • Require Internet insurance for users of ISPs. ISPs that police botnet and other harmful devices would pay lower raates than ones that didn’t care, money raised could be used to shut down harmful devices.

It’s a pretty difficult problem sadly, and there’s no good answer, but we are going to have to start doing something or start getting used to large DDOSes taking out a bunch of things we use often.

Kevin Fenzi : xfce4-terminal: lots of progress recently

Most Linux desktop’s have their own terminal programs, and Xfce is no exception. There’s been a number of releases of xfce4-terminal recently, so I thought I would share some of the changes with people who perhaps haven’t tried it recently.

The biggest change is that it’s been ported to gtk3 and vte291. This is great news as it means it’s no longer using the no longer supported at all ancient version of vte. Unlimited scrollback has been added, wayland support, tons of bug fixes, translation updates, and memory leaks quashed.

More specifics in the NEWS file: https://git.xfce.org/apps/xfce4-terminal/tree/NEWS

Give it a try today and see if it handles all your terminal needs.

October 20, 2016

Kevin Fenzi : keepalived: Simple HA

We have been using keepalived in Fedora Infrastructure for a while now. It’s a pretty easy to use and simple way to do some basic HA. Keepalived can keep track of which machine is “master” for a IP address and quickly fail over and back when moving that IP address around. You can also run scripts on state change. Keepalived uses VRRP and handles updating arp tables when IP addresses move around. It also supports weighting so you can prefer one or another server to “normally” have the master IP/scripts.

Right now we are keepalived on our main koji server pair. We have a koji01 and koji02. Normally 01 is primary/master and has the application IP address on it so all traffic goes to it. If for some reason it was turned off, the keepalived on 02 would see that and take the IP address and run a script to become master. If 01 came back up, 02 would see that and transfer back to it. Right now we have the scripts setting up on the secondary server a bunch of cron jobs (garbage collection) and kojira (the process that regenerates build roots).

We are also using keepalived on some new paired postgresql instances we are working on. More on that in a later blog post. If you need simple HA with a IP address and script(s), keepalived does a banner job.

October 19, 2016

Kevin Fenzi : pass – The standard unix password manager

In my line of work (sysadmin), I have to deal with a LOT of passwords. For a number of years I was a fan of keepassx, but the upgrade to version 2.0 there didn’t thrill me. (There were some features I liked that it dropped and in general it seemed to be less nice), so I decided to look around and was pointed to pass.

Pass (or password-store as it’s sometimes called) can be found at https://www.passwordstore.org/ and in most major Linux distributions package set. (It’s called just pass in fedora). It’s a simple command line tool and uses gnupg and git (if you like). Each of your passwords / sites is a gnupg encrypted file, setup in a tree under ~/.password-store. You can tell pass also to use git here so every change you make it a git commit of encrypted files. So, you should be able to find any old passwords you changed in git history if needed.

Setup is pretty simple:

  • pass init yourgpgid@example.com – this sets up the base directory for pass. Note that you can actually add additional gpg keyids to encrypt files to later if you have a team or the like.
  • pass git init – this sets up the git repo

After setup you can use ‘pass insert path/to/name’ to add your own password, ‘pass generate /path/to/name’ to have pass generate a random password for you (using pwgen), ‘pass ls’ to list the tree of sites, ‘pass -c /path/to/name’ to copy the password to your clipboard for easy inserting into another application or website. (by default it will stay in the clipboard for 45seconds or 1 paste and then vanish). Note that you never even need to know the password here, you just get it from pass and paste it in and are done. If for some reason you do need to see it (a broken app that won’t let you paste for example), you can just do ‘pass /path/to/name’ and it will output the password. You can do ‘pass edit /path/to/name’ to edit a password, and note that you can add whatever you like to the pass file. pass -c will use the first line as the password, but you can add more lines with security questions and random answers, usernames or any other notes you like about the site. ‘pass git ‘ will let you run any arbitrary git command on your pass git repo, if you wanted to look at history or go back to some previous commit.

There is a android app that is supposed to work with pass files, but I have not tried it. It requires you to copy your gnupg private key to your phone, which is not something I am really wanting to do. Since I have my laptop with me most all the time anyhow, it’s not a big deal.

October 18, 2016

Kevin Fenzi : debugging koji build failures

From time to time builds fail in koji (The Fedora build system), and it’s good to know how to figure out where to look for the reason. Koji has a central hub that manages jobs and a bunch of builders that actually do the builds. When someone initiates a build you are talking to the hub and either uploading a src.rpm (for a scratch build) or telling it to use a particular git hash/repo for an official Fedora build. For official builds, koji will first generate a job to build the src.rpm from git and the packages lookaside cache with the source. This job will run on some builder thats ready and has capacity for it. Once the src.rpm is generated (or if you are providing it for a scratch build), the hub will generate build tasks for all the arches that are set in the target tag you are building to. Each one of those will go out to a builder of the right arch type that is enabled and has capacity, etc. If any of these fail, the entire build fails.

Once you have gotten notice of a failure, the first thing to do is go to the link for the build in the koji web interface and then click on the task link to show the build tasks. There you will see one (or more) links in red that failed. Don’t click on any of them yet though, instead click on the “show result” link. That will tell you exactly which task failure failed the build. When one failure happens koji cancels the other builds, but sometimes they fail at nearly the same time and you are looking at the wrong place for the real failure. Sometimes all the links are green and the reason for the failure is only visible in the “show result” area (for example, if you have a archfull package with noarch subpackages and they are different on different arches, thats a failure, or if you don’t have permissions to build the thing you are building, thats a tagging failure).

Next click on that task that you saw in the “show result” link (if you haven’t already seen the error as above), and again click on the “show result” link. You should see something like:

"BuildError: error building package (arch i686), mock exited with status 1; 
see build.log for more information"

Or it could refer you to the root.log. Go to that log and look toward the end for the error. If the error is in the root.log it means there was a problem setting up the mock chroot. That could be that there is a package you BuildRequire that in not installable, or some package in the build root is completely broken.

If the issue is in the build.log its the actual output from the rpmbuild, so this would be where to look at your general compiler issues, etc.

If you ask someone to help you out and look at the error(s) you are getting, it’s important to give them the top level link to the task instead of just a link to the root.log or build.log. If you only have those links it’s impossible to back up and look at the higher levels, but if you have the top level task you can look at the entire thing.

Aside from normal rpm package builds, koji also builds a bunch of other things like livemedia and other images. Finding issues there is very similar. Always check the “show results” area for which logs to look at for the actual failure.

October 17, 2016

Kevin Fenzi : gnome extensions

In general when using Gnome I try and avoid extensions. Most of the ‘default’ setup is fine or I was able to get used to it and it works well enough. There are a few extensions I do use however for various reasons. All of these work with Wayland.

  • AlternateTab – I use alt-tab a LOT. I just found myself unable to get used to the default behavior of showing applications and windows under those, when I use alt-tab I want to switch to a specific window quickly, not have to switch to a application and arrow key around to find the one I want.
  • Openweather – This is just fluff, but nice to have. Gives you a nice little weather report in the top bar. You do have to pick a provider and get an API key, but it’s pretty painless. You can also setup multiple locations.
  • PanelOSD – This one I use to move notifications over to the top right side. This is where I am used to them being from Xfce and that way they are not in my way.
  • TopIcons – This allows you to see systray icons on the top bar. There’s still a few applications I use from time to time that have a systray and it’s handy to see them if they do.

I’d probibly include the redshift extension here, but it doesn’t work with Wayland. 🙁

October 16, 2016

Kevin Fenzi : Some interesting gnome 3.22.x gsettings

As I have mentioned in the past, I switch off between Xfce and Gnome desktops here. While in Gnome, I have a number of settings I have picked up in various places (bugs, talking to maintainers, various docs and blogs) and I thought I would share a few in the hopes that they would be of interest to others. You can use the ‘gsettings’ tool or the ‘dconf-editor’ GUI to set or check the current values of these keys. Of course some of them can also be set from the normal settings guis.

  • org.gnome.shell.overrides edge-tiling true – This lets you drag windows to the right or left edge and have them tile to 1/2 of the desktop. This goes along with dragging to the top of the screen to maximize an application.
  • org.gnome.software download-updates false – Disable gnome-softwares downloading of updates. Since I use dnf this saves me a bit of BW and also messages about needing to apply updates I already have applied.
  • org.gnome.nm-applet suppress-wireless-networks-available true – This disables the ‘There are open wireless networks” message. I only want to join specific wireless networks if I need something.
  • org.gnome.settings-daemon.plugins.xsettings hinting ‘slight’ – slight hinting works best with the fonts I like. I think this may now be the default.
  • org.gnome.desktop.wm.preferences focus-mode ‘sloppy’ – I want to change focus when I move the mouse. I am pretty used to this from years and years now.
  • org.gnome.desktop.wm.preferences auto-raise-delay 125 – and autoraise on focus change.
  • org.gnome.shell.window-switcher app-icon-mode ‘both’ – I want both the icon and the name in the alt-tab switching thing.
  • org.gnome.desktop.interface clock-format ’24h’ – 24 hour clock is what I am used to. 🙂
  • org.gnome.desktop.interface show-battery-percentage true – shows the battery % left in the menu bar. Handy when you are wanting to see how long you have left and you don’t want to pull down the menu.
  • org.gnome.desktop.interface cursor-blink false – blinking cursors are evil. They suck power are distracting and should be disabled. (I could swear that I saw a site a while back that showed how to disable blinking cursors in any application or system you could name, but I can’t seem to find it now)
  • org.gnome.desktop.interface clock-show-date true – I like the date in the menu bar.
  • org.gnome.shell disable-extension-version-validation true – This disables the version check for extensions. Sometimes it helps, but usually if an extension doesn’t work for me setting this doesn’t help much.
  • org.gnome.desktop.peripherals.touchpad two-finger-scrolling-enabled true – I used to always use Edge scrolling, but it went away for a while and I retrained myself on 2 finger scrolling and it’s fine now.
  • org.gnome.desktop.peripherals.touchpad tap-to-click true – must have tap to click. Saves so much wear and tear on my hands and the touchpad. Just a gentile tap is all thats needed.

Do you have some favorite ones?

October 15, 2016

Kevin Fenzi : Running ansible from a git checkout

The Fedora and EPEL ansible packages are as up to date as possible, but sometimes you want to run a newer, not yet released ansible version for any of a number of reasons:

  • To confirm a bug is fixed.
  • To test an RC for the next release or look for regressions on the devel branch.
  • To see if a change caused any performance improvements or regressions.

Happily this is very easy to do with ansible:

  1. clone the ansible repo: ‘git clone https://github.com/ansible/ansible.git’
  2. checkout the branch or thing you want to test. For example now stable-2.1 branch has the latest 2.1.x rc, and stable-2.2 has the code for the latest 2.2 rc.
  3. ‘source ansible/hacking/env-setup’ This will set your ansible/ansible-playbook commands to use your checkout instead of any locally installed version.

I use this setup to test new versions/RC’s over the Fedora Infrastructure playbooks. It’s very handy to catch last minute bugs in RC’s and/or to clean up depreciations so when the next version is actually released you are already ready for it.

Happy ansible hacking.

Kevin Fenzi : The greatest game in history…

I am of course talking about nethack.

If you haven’t played in a little while (and you HAVE played nethack right?), it’s as fun as you remember it. Running around the dungeon fighting monsters and picking up magical devices. It’s pretty hard to believe that next year will be the 30th anniversary of nethack.

It’s friday, you deserve some nethack today.

October 14, 2016

Kevin Fenzi : ss: the netstat replacement

Many of the old network tools in the net-tools package have been deprecated for a while now. The replacement for the traditional netstat command is ‘ss’.

It of course has a ton of options (like netstat). I particularly like ‘ss -e’ (extended) and ‘ss -E’ (show connections as they appear).

Do take a look today and see if you can finally replace your netstat usage today.

October 13, 2016

Kevin Fenzi : The Fedora infinote server

Many of you may not know that Fedora Infrastructure runs a infinote server instance at infinote.fedoraproject.org.

Infinote is a collaborative text server. You can connect to it with the ‘gobby-0.5’ client located in the gobby05 package in Fedora. Once connected you can create documents and multiple people can work on them at the same time. The server takes a git snapshot of all documents every few minutes so you can see history. There’s even a cgit instance at https://infinote.fedoraproject.org/cgit/

Of course these days there’s etherpad and the like that have more features, but from time to time there’s a good Fedora use for a server we run. Fedora Infrastructure uses it to coordinate their weekly meetings and update/reboot planned outages. Everyone is welcome to use it for their Fedora related collaborative text editing. Or you can of course install the libinfinity package and run your own server.

October 11, 2016

Kevin Fenzi : Fedora 25 Beta is here

This morning we released Fedora 25 Beta: https://fedoraproject.org/wiki/F25_Beta_release_announcement

Release mornings are always a bit stressful for us in Fedora Infrastructure, even though we have gotten pretty good at doing them. Lots of things need to be lined up and ready to go when the announcement goes out: torrents, websites, announcement, etc. Also any networking or hardware issues are particularly unwelcome on these days.

Today everything has gone quite smoothly so far, and I hope everyone enjoys Fedora 25 Beta.

October 10, 2016

Kevin Fenzi : Fedora go/no-go meetings

Before each Fedora milestone release (Alpha, Beta, Final) there is at least one Go/No-Go meeting. As the name suggests this is the meeting where folks from QA, FESCo, Release engineering, Program Management and others gather to see if the release is ready to go out the following week. The meetings are held on IRC and open to everyone (Although go/no-go votes may only be taken from representatives of specific groups). You can see the logs of the last one (last week to decide if Fedora 25 Beta was ready to release this week) this meetbot.fedoraproject.org log link

In the event the decision is “No-go” then the Milestone release slips a week and a new Go/No-Go meeting is scheduled a week out. This allows whatever issues were blocking the release to (hopefully) be solved.

If the decision is “Go”, then the release happens the following week. Why so long between the decision to release and the actual release? Well, there are a number of things that need to happen to get things ready for release, including, but not limited to:

  • The release has to be put in the correct place on master mirrors (moved from the alt/stage directory to releases/test or releases) and time is needed for all our mirrors to pick up this content and sync it.
  • Common bugs need to be identified and written up: https://fedoraproject.org/wiki/Common_F25_bugs
  • The websites team needs to land any changes they need for the release and get them ready to go live on release morning.
  • Torrents need to be synced and setup, ready to live for release morning.
  • In the buildsystem, the release needs to be tagged so we know exactly what packages were in that release and so alternative arches can build the exact same set for their releases.
  • Checksum files and so forth need to be verified and signed by release engineering.

I hope everyone is looking forward to the Fedora 25 Beta release tomorrow!

October 09, 2016

Kevin Fenzi : squeezing out a bit more battery life under Fedora

In my experience, battery life on laptops has been getting better all the time with Linux/Fedora. However, there’s a few things you can do if you need just a bit more battery life than you are getting with default settings.

First let me say that in the ideal world you wouldn’t have to do any of this. Things should just work and give you the best battery savings they can. Sadly, thats not fully the case (yet), so sometimes you have to take matters into your own hands.

First, there’s some packages that try and set things for you: tuned and tlp (and possibly others, but those are the two I know of). If you are using tuned, you will want to run a ‘sudo tuned-adm profile powersave’ to switch to the powersave profile. With tlp it should switch to this automatically when it sees you are running on battery.

powertop has a bunch of recommended settings for power saving which you can automatically apply with ‘sudo powertop –auto-tune’ or by enabling the powertop service. You do need to be careful here however as some of these settings are not set that way by default because they cause lockups or other bad behavior. Do test this on your particular hardware before you trust it.

Finally there’s a bunch of common sense items: lower the screen brightness as much as you can (the screen is by far the largest power user on a laptop), put your bluetooth/wireless in airplane mode so they don’t consume power (if you don’t need them), remove usb devices (some of them suck up power even while not doing much), turn off keyboard backlight and such.

October 08, 2016

Kevin Fenzi : Fedora Infrastructure Outages (and the one on 2016-10-08)

Astute folks may have noticed that we had a pretty severe outage this morning that lasted just about an hour. This seems like a good time to talk about outages in general and this one in specific.

In general these days our outages fall into roughly 3 ‘buckets’:

  1. Planned outages. We do these every few months to reboot servers and apply updates, or when we have to migrate something that will take some downtime. This is by far the largest bucket.
  2. Unplanned outages, but that are small in scope. Typically this might be one of our many datacenters having network problems or a single service having some issue. These typically aren’t noticed by very many people (in most cases only the infrastructure noc folks who disable a proxy or restart a service or whatever is needed).
  3. Unplanned outages that are large in scope. These are thankfully rare. These are were we loose connectivity to our main datacenter (that has the majority of our hardware in it), or severe hardware or software failure (storage servers not working, database needs reloading, etc). Luckily these are pretty rare.

So what happens in an outage? Normally nagios lets us know about it pretty quickly and someone starts taking a look. Usually until we have some idea, we treat things as being in bucket 2 if they aren’t planned. That is, we assume it’s a minor issue until we investigate and see it’s larger. Once the scope of the issue is determined, we update https://status.fedoraproject.org/ This is out status indicator. If you are wondering if there is a known outage, you can look at this. Note that this is MANUALLY updated after initial innvestigation, so do give a few minutes after seeing something before expecting status to be updated. Next, folks work on the outage, this would be anyone in the sysadmin-noc group and the primary infrastructure admins. Status is then updated when the outage is over.

Today (2016-10-08) at approximately 16:15UTC our primary datacenter (PHX2) became unreachable. This turned out to be due to a firewall upgrade that did not go as planned. Service was fully restored at approximately 17:15UTC. Right now when our primary datacenter isn’t reachable most things are also not working as we have our primary vpn hub located there, and thats what our various proxies use to talk to applications. Unfortunately this includes our most active service: mirrorlists. We have planned to move that service to a container and run it at each datacenter, but just have not gotten to deploying that yet. Once we do, at least mirrorlists will stay up if the primary datacenter is down.

Additionally, we have plans to eliminate the first type of outage or reduce it greatly. For the most part our applications are deployed with pairs (or more) of instances so no service needs to go down, however the sticking point is rebooting database servers. I’ve been working on setting up some replication so hopefully we don’t need to schedule any outages for that either.

October 07, 2016

Kevin Fenzi : Rawhide kernels back on track now

Just another quick note that rawhide kernels are back to booting and working normally here at least. With 4.9.0-0.rc0.git3.2.fc26.x86_64 at least. That should be in today’s rawhide.

October 06, 2016

Kevin Fenzi : rawhide 4.9.0 pre rc1 kernels

Just a quick note that with the upstream kernel folks opening up the merge window for the 4.9 linux kernel, rawhide also has been getting these merge kernels. Of course there’s a ton of churn when the kernel merge window is open, vast amounts of things are merged, so it’s a wonder they usually work as well as they do.

In this case there seems to be some pretty severe issues with at least kernel-4.9.0-0.rc0.git1.1.fc26 and kernel-4.9.0-0.rc0.git2.1.fc26 at least. In vm’s they seem to boot and then start oopsing. On my laptop it does boot, but then waits about 2 minutes at waiting for usb devices, boots the rest of the way up and then has no wireless or usb devices.

https://bugzilla.redhat.com/1382134 is tracking at least part of this issue and the kernel maintainers are digging into it. Of course while they are, more things are being merged (possibly even fixes to these issues), so hopefully it will all shake out in the next few days.

In the mean time, rawhide users are advised to just switch back to the 4.8.0 final kernel for now.

October 05, 2016

Kevin Fenzi : 10 basic linux security measures everyone should be doing

Akin to locking your doors and closing your windows there’s some really basic things everyone should be doing with their Linux installs (This is of course written from a Fedora viewpoint, but I think this pretty much applies to all computer OSes).

  1. Choose nice long passphrases you can remember. Most any modern system will have a pretty long limit on passphrases, so pick something nice and long that you can remember. Don’t think of them as passwords, they are phrases with many words.
  2. When installing, encrypt your drive(s). The performance hit is not noticeable and if you ever throw away a broken drive or someone steals your computer they won’t have your data.
  3. Apply updates regularly. If you aren’t someone who remembers to do so, setup something like dnf-automatic to just apply them every day for you in the middle of the night. Otherwise try and get into the habit of letting gnome-software do offline updates at some regular time.
  4. Along with (3), reboot when needed for new kernels or glibc or other things you use. Get used to rebooting on a regular schedule. Don’t be afraid of rebooting, get used to doing it.
  5. If you are in a place with untrusted people roaming around, do setup a screen locker and lock your computer when you are away from it.
  6. Make (and sometimes test) regular backups. You may not think of backups as a security measure, but they sure are. Think of the new fad of ‘ransomware’ where someone encrypts your data and sells you the key. If you have good backups you can just wipe that all out and restore from those. They are handy for lots of other reasons too.
  7. Don’t open weird attachments or links sent to you in email. If you didn’t ask for it, delete it.
  8. Don’t plugin weird devices you run across to your machine. (USB or otherwise). You can use a neat package called ‘usbguard’ to make sure no one else does while you are not around too.
  9. Use a passphrase manager or have some system to allow you to have long, not easily guessed passphrases at all the various applications you login to. There’s tons of these out there: Password managers: pass, keepassx, gpg encrypted file, etc. Schemes: Diceware, etc. Pick one that works for you.
  10. If you use a laptop/travel a lot, consider using a VPN for all your network needs. As long as you have an endpoint to connect to (your home server, your work, a vpn provider) you can send (almost) all your traffic over the vpn and thus avoid problems with people sniffing local traffic.

Some of these things require an initial investment of time (backups, vpn, passphrase manager, screen lock) and some require just making something a habit (long passphrases, apply updates regularly, reboot regularly, don’t open weird things in email or the physical world), but they are all worth it.

Will this make your computer “secure”? No. There’s no such thing. “secure” is not a binary state, it’s a process of assessing threats and deciding what you can or want to do about them. Doing the above things will protect you from some threats nicely (guessable passwords, untrusted people tampering with your computer, sniffing traffic, vulnerabilities that have already been fixed in software you use, etc), but will basically do nothing against others ( someone installing a keylogging device and recording everything you type, someone threatening you with harm to tell them some information, someone installing a spy cam and recording everything on your computer screen, someone using a non public vulnerability in software you use, someone social engineering access to your computer, etc).

October 04, 2016

Kevin Fenzi : Short note: fedoraproject.org smtp sessions now using TLS

Just a quick note for everyone who gets emails from fedoraproject.org or it’s various other domains. Just before we entered freeze for Fedora 25 Beta we landed changes to make our smtp servers use TLS where possible, so emails to servers that support it should now be fully encrypted.

Please let us know if you see any issues related to this change.

October 03, 2016

Kevin Fenzi : How to debug Fedora rawhide compose problems

From time to time rawhide composes fail and are not announced or synced to mirrors.

In the past this would happen only if the very basic setup (a mock chroot with the ‘buildsys-build’ group installed in it) broke. Additionally, in the past rawhide composes where many deliverables failed to compose were still synced out and announced, leading to days when no images were available until the issue was fixed.

Now, with the latest version of pungi (The tool that composes Fedora releases, including rawhide), composes can fail if some deliverables (Those marked in the configuration as not failable) didn’t complete. So, while rawhide can fail more easily, it also means it’s much easier to revert some change that broke images and get that fixed before it lands, and images should be always available.

So, how can you tell if a rawhide compose (or some part of it) failed and why? All the pungi logs are avilable and since all the builds take place in koji, anyone can look at them as well. Rawhide composes are of course fedmsg enabled, so you want to look for the https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.pungi.compose.status.change topic. Composes can finish with 3 states:

  1. FINISHED – This means the compose finished and everything in it completed successfully. I am not sure we have yet seen this status in real life. 😉
  2. FINISHED_INCOMPLETE – This means the compose finished and only failable things failed. This is the “normal” status we see day to day.
  3. DOOMED – This means the compose failed it’s initial very basic setup and/or some deliverable marked as not failable failed. When this happens, it means the compose isn’t synced out or advertised. This is the status where we need to find out what caused the problem and fix it and either restart the compose or wait for the next days compose. In IRC or on fedmsg you may see this status as “failed in a horrible fire” as thats what our fedmsg translates it to.

So if you have a compose and you want to see why some part of it failed, you can look at the fedmsg and it provides a location url, but they are always the same format. Lets look at an example: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20161002.n.0/ This means it’s a rawhide compose (there’s a Branched directory for the Branched Fedora 25 right now), then the particular compose was Fedora-Rawhide-YYYYMMDD.n.0. Yeah, month and day, then a ‘n’ to mean ‘nightly’ and a ‘0’ to mean it was the first compose of the day. If there’s more composes that same day, they are in n.1, n.2, etc. For Alpha/Beta/Final releases there’s no ‘n’ there, as they are not nightly.

The first place I go to look for issues is the global log file. This would be in logs/global/pungi.global.log and you will want to look for the image or deliverable you are interested in or search generally for tracebacks. Usually this will note a koji task id or build which you can then look up on koji. Since this is already getting long, I’ll post about tracking down koji build problems another day.

October 02, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-10-02 edition

It’s once again been a while since one of these posts, but again I’m going to try and do them more often again.

This last week had a few big changes in rawhide:

  • With the Fedora 26 change approval last week ( https://fedoraproject.org/wiki/Changes/DNF-2.0 ) DNF-2.0 has landed in rawhide. Things were a bit shaky at first as the compose tools were not also updated, but anaconda folks were ready with a patch and lorax got a patch from the DNF maintainers and everything landed. There’s still one big issue outstanding: dnf 2.0 sets strict group installs back to true, which causes all the live media to fail to compose. Some of our comps groups contain packages that only exist in some arches, not all of them, and when dnf does strict group installs it errors when it cannot find all the packages in the group. I’ve filed a bug for figuring this out: https://bugzilla.redhat.com/show_bug.cgi?id=1380945
  • Last year we setup a quick and dirty autosigning setup for rawhide, which was just a script in a loop run by releng folks when they remembered to do so. This meant it wasn’t all that reliable. Now, thanks to Patrick Uiterwijk, we have a real autosigning setup implemented and are very very close to an always signed rawhide. The final bit of gating needs a update to fedpkg to go out first, but as of last week, most everything should be signed with only the rare issue with a build that lands right before the compose starts.
  • There has been some work the last few weeks to produce a rpm ostree for rawhide that is updated every few minutes. Once this is available it will be a great help to testing and running rawhide.
  • Xorg server 1.19 almost landed, but we need to fix tigervnc before it can (at least without breaking all the install paths for rawhide). That is being fixed in a side tag and should land as soon as tigervnc is fixed. https://bugzilla.redhat.com/show_bug.cgi?id=1380879 is tracking that work
  • A while back changes were made to cause composes that fail when some deliverables don’t compose. This has been very helpful in making sure changes that land don’t break things. New package builds that cause the entire compose to fail can be untagged until the breakage is fixed.

That’s it for now from the rawhide trail…

Kevin Fenzi : How to reignite a flamewar in one tweet (and I still don’t get it)

disSome of you may have seen this post last week:


I saw it I think shortly after it was posted to hacker news (aside: I read hacker news via it’s rss feed that has just the raw links and no comments or ratings, much nicer IMHO).

My first thought on reading it was “Did they report this upstream?” and upon some digging, yes they did ( https://github.com/systemd/systemd/issues/4234 ). It was filed the same time they posted their blog post/tweet (no responsible disclosure here). My next thought was to see if it worked, so I tried it on various of my test machines and it did not. Turns out you have to run it in a loop (as noted now in the blog and seen in the systemd issue). At that point I saw that there was a Fedora tracking bug, someone requested a CVE and there were patches proposed on the systemd issue (which has since been closed/fix merged), so I moved on with life.

Next someone posted the blog link to the fedora users list along with a rebuttal from a systemd maintainer: https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post, and the discussion went downhill quickly from there. Finally, I saw a slashdot post today with the calm, journalistic title of Multiple Linux Distributions Affected By Crippling Bug In Systemd (which points to the blog post above that started all this).

So, everyone is flaming everyone else again about systemd, and I just don’t get it. My personal relationship with systemd is well within the range of all the other Open Source projects I use every day. Mostly I am happy with it, sometimes it has bugs and I report them, sometimes the project makes decisions I don’t like or agree with, sometimes the project doesn’t communicate as well with downstream as I would like (see fedora devel list post about systemd and TasksMax ), sometimes I have to learn (again) how to do what I want to do, sometimes it’s slow when I want it to be fast and so on. Would a local denial of service attack in another project have (re)opened such a bunch of flames? There was in fact just the day before the post a remote denial of service in bind (the most popular dns server out there). Did you even hear about it?

I just don’t get the passionate hatred systemd has. I guess perhaps I never will, and I guess thats ok. I would kindly ask those who do passionately dislike systemd to just move on to somewhere without it and leave the rest of us alone, but I don’t expect thats likely to happen. In the mean time if you have some technical problem with systemd, I will be happy to help you isolate it and file a bug or teach you how to do whatever you are trying to do with it, but I’m going to try and stay out of your flamewars.

September 25, 2016

Kevin Fenzi : 6 months a task warrior

A while back I added a task to my taskwarrior database: evaluate how things were going at 6 months of use. Today is that day. 🙂

A quick recap: about 6 months ago I switched from my ad-hoc list in a vim session and emails in a mailbox to using task warrior for tracking my various tasks ( http://taskwarrior.org/ )

Some stats:

Category Data
Pending 17
Waiting 20
Recurring 12
Completed 1094
Deleted 18
Total 1161
Annotations 1059
Unique tags 45
Projects 10
Blocked tasks 0
Blocking tasks 0
Data size 2.0 MiB
Undo transactions 3713
Sync backlog transactions 0
Tasks tagged 39.8%
Oldest task 2016-03-24-13:45
Newest task 2016-09-25-10:03
Task used for 6mo
Task added every 3h
Task completed every 4h
Task deleted every 10d
Average time pending 4d
Average desc length 32 characters

Overall I have gotten a pretty good amount of use from task. I do find it a bit sad that I have only been completing a task every 4 hours and adding one every 3 hours. At that rate things aren’t going to be great after a while. I have been using annotations a lot, which I think is a good thing. Looking back at old tasks I can get a better idea of what I did to solve a task or more context around it (I always try and add links to bugzilla or trac or pagure if there’s a ticket or bug involved).

I’d say I am happier for using task and will continue using it. It’s very nice to be able to see what all is pending and easily add things when people ask you for things and you are otherwise busy. I’d recommend it to anyone looking for a nice way to track tasks.

August 29, 2016

Kevin Fenzi : two vpns and NetworkManager? no longer a problem…

NetworkManager has pretty handy vpn handling for laptops. You can setup all different kinds, it can prompt you for passphrases, it can set up specific vpns on specific networks, etc.

However, if you had more than 1 vpn you wanted to run at a time you had to pick one for NetworkManager to handle and do the other(s) outside NetworkManager. This is happily no longer the case (at least with NetworkManager 1.4.0): It can bring up as many vpns as you like and manage them all.

This weekend I took my home openvpn connection (which I had setup as a system wide openvpn service) and moved it into a NetworkManager vpn profile. It works just fine with my work vpn and now NetworkManager can bring it up automatically after it connects to the network.

So, if you need two (or more) vpns on your laptop, you can now happily move them all into NetworkManager. Many thanks to the hard working NetworkManager folks. 🙂

July 25, 2016

Kevin Fenzi : Looking forward to flock 2016

Just over one week until flock ( https://flocktofedora.org ), Fedora’s main yearly conference. This time it’s in Kraków, Poland. This of course means a long time traveling for myself and other North American Fedorans, but it’s always well worth it.

In addition to seeing old and new friends, I’m looking forward to quite a lot of the talks (see https://flock2016.sched.org/ for the full schedule) along with workshops and hallway discussions.

I’m going to be giving a talk on using Fedora Rawhide as your daily driver OS. Hopefully lots of tips and workflows that people will find useful. Also, I am co-presenting with Pierre-Yves Chibon (pingou) on the current state of Fedora Infrastructure along with lots of upcoming plans.

If you aren’t able to make it to Kraków in person, do follow along on #fedora-flock ( on freenode ) and in blogs and other project communications.

June 27, 2016

Kevin Fenzi : Zodbot… upgraded

Kneel before the new Zodbot!

We have upgraded our beloved evil super villain IRC bot on freenode from an old version of supybot-gribble to a new shiny version of limnoria ( https://github.com/ProgVal/Limnoria ).  This doesn’t change much in the interface, but it does mean we are using something that is maintained and gets updates and is a good deal more secure. If you notice problems please do let us know with a Fedora Infrastructure ticket.

Also, as one of the maintainers of supybot-gribble in Fedora and EPEL, I will be retiring that in favor of the limnoria package very soon. It should be now available in updates repos for your upgrading pleasure.

Ursa: You are master of all you survey.

General Zod: [bored] So I was yesterday. And the day before.

June 01, 2016

Kevin Fenzi : What didn’t happen today?

Today, astute observers may have noticed that lists.fedoraproject.org and lists.fedorahosted.org didn’t participate in the long running, annoying, insecure and exasperating tradition called “mailman day”. This is where mailman spams you with a reminder for every list you are on with your plaintext “password” in clear email.

Since we have now fully switched over to mailman 3, you can relax and rest easy that we DID NOT spam you with your mailman password today, nor will we from now on. 😉

Our mailman3 instances authenticate users via Fedora Account System OpenID, persona or yahoo, no local throwaway passwords to look for or remember.

May 11, 2016

Kevin Fenzi : Mailman3: no topics support (with alternatives!)

In mailman2 there is a feature called ‘topics’. What this allowed you to do was create some filters on the list and when those filters matched mailman would add a header telling the user that this post was part of a ‘topic’ or not. You can then use the mailman interface to tell it you only wanted posts on a specific list of topics and no others. As far as I know there aren’t too many lists that used them, but there are a few we run in the Fedora Project that do: The prime example being the ‘package-announce’ list.

The package-announce list gets emails from bodhi (the Fedora updates system) about each update when it’s pushed into the stable updates repo. As you can imagine with a fast moving distro like Fedora, this list gets a LOT of posts. This list has topics for each Fedora release (22, 23, 24, etc) and also “Security” and “New package” updates. Using topics users could get a subset of these emails that they wanted.

Sadly, topic support isn’t yet available in mailman3, and it’s unclear right now when it might be implemented. Not wanting to hold up our migration to mailman3 forever, we want to go ahead and move lists like package-announce over as soon as we can. Of course this means those folks using topics are going to be in a bind. However, there are some alternatives:

  • You can switch to using a RSS feed directly from bodhi itself. For example: https://bodhi.fedoraproject.org/rss/updates/?releases=F23 will give you all stable Fedora 23 updates. https://bodhi.fedoraproject.org/rss/updates/?releases=F23&type=newpackage will give you all newpackage type updates, etc. Pretty much any search or page with a rss symbol on it on bodhi is an RSS feed.
  • You can set up notifications in the Fedora notifications service: https://apps.fedoraproject.org/notifications/ to either email or IRC. For packages you are interested in, all updates, security or newpackage updates or any combo.
  • You can of course just stay on the package-announce list and filter things locally. You will be getting emails about some things you don’t care about, but emails are pretty small in this day of streaming video. 🙂

Hopefully those interested can switch over to one of the alternative methods above until topic support once again lands in mailman.

May 04, 2016

Kevin Fenzi : Fedora account system and FreeIPA

Over the years, a number of times, people have asked us about migrating from our own custom Fedora Account System (FAS) to FreeIPA.

We would love to, but right now as things stand it’s just not a great fit for us. In the past we have had trouble articulating what the issues are, but now Patrick has written up a wiki page with all the items we know of:


Hopefully we can keep open a dialog with FreeIPA folks and keep the above wiki page synced up so perhaps someday we can migrate when blockers are no longer in our way. 🙂

April 16, 2016

Kevin Fenzi : ansible 2.0 going stable in f23/f22/epel7/epel6

Just a quick heads up: ansible- is heading out to the stable updates repos for Fedora 23, Fedora 22, EPEL7 and EPEL6.

There’s a few outstanding issues folks might hit, but we figured it was finally time to push to stable. If you use includes for handlers you might hit ( https://github.com/ansible/ansible/issues/13485 ). This one will be fixed in 2.1 by allowing you to add a config option to set includes as static or dynamic. Also, you might see some intermittent issues with some hosts saying “unable to resolve remote temp directory” ( https://github.com/ansible/ansible/issues/13876 ). Hopefully this will be fixed in, and there are some workarounds in the issue.

If you run into another issue, please do check if it’s reported upstream and if not report it.

We have also pushed out a ansible1.9 compat package for those folks that need to stick with 1.9 for now for whatever reasons. You can simply remove the ansible package and install ansible1.9 and you can use that version. It’s currently at 1.9.4, but there’s a 1.9.6 update pending updates-testing now also.

Happy ansiblizing!

April 03, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-04-03 edition

Well, it’s been a while again, so I thought I would update everyone on recent goings on in Rawhide, Fedora’s rolling development release.

Since my last post we finally moved over to having Rawhide and Branched (what will be Fedora24) done by the new pungi4. There are sadly lots of little issues that have been cropping up and getting fixed by Release Engineering, but it’s been a bit of a bumpy ride. As of this post, the last Rawhide compose that finished was the 2016-03-31 one, all the ones since then have failed. Hopefully this will get sorted out tomorrow. The current cause of problems is adding atomic/ostree to the compose (which we very much need/want), it’s just been more difficult that we first hoped. Once things settle down we will really be in a better place than before, with full composes (tested by openqa!) every night.

There was a bit of delay since the Branching event on signing, but last week we got back on track and (almost) everything in rawhide should be signed by the Fedora-25 key now.

With compose issues and general breakage in the install path it’s been difficult to install rawhide recently. Again, once we get the normal composes back on track this should clear up. As usual, once you have a rawhide install there hasn’t been really much breakage of the day to day use path.

Astute observers would have noticed that the rawhide-source mirrormanager repo wasn’t working after the switch to pungi4. I’ve fixed that up the other day and rawhide-source should be working normally again.

Kevin Fenzi : How long does it take for that Fedora ssh key to sync?

There’s been some confusion about how long it takes from changing your ssh key in the Fedora Account System (FAS) and being able to use it to access things like fedorapeople.org or pkgs.fedoraproject.org or fedorahosted.org. The method has also changed a while back so I thought I would write up this short post on it.

TLDR version: for ssh access to fedorapeople, pkgs or fedorahosted, it should take about 2-17minutes. (usually 2). For access to other machines (if you are in a sysadmin group), it should take 15 to 30min (usually 15). So, you would be safe saying “within 30minutes”.

In the past, we simply had a cron job on every machine that would run every 30min or so and sync everything. This was a massive waste of resources. We had to size our fas servers to handle the load of every machine +/- a few minutes hitting them and for the most part, absolutely nothing changed. Then, along came fedmsg and we now had easily available data to know when something has changed and what it might affect. The new process is run from a fedmsg listener: It waits for ssh keys changes to be mentioned on the fedmsg bus, then it looks to see if they are from someone in a sysadmin group or not. It will then wait for about 30 seconds (if there’s more changes coming in, might as well do them at once). If not, run a ansible playbook that updates just those machines that they might have access to. If they are, use a different playbook that updates all the machines in turn.

The non sysadmin playbook takes about a minute to run. So, the vast majority of these changes will appear in about 2 minutes.

The sysadmin playbook takes much longer (There’s a LOT more machines in this playbook): 15minutes or so. The vast majority of changes here will appear in 15minutes (but it depends on which machine you are waiting for access on of course).

As a side note this applies to ssh keys and mail aliases, but not koji certificates. Those are active immediately as soon as they are issued (and your old certs are revoked).

Since the script will only run one playbook run at a time, the worst case is your changes just miss a sysadmin playbook run and you are yourself a sysadmin. Then you may have to wait 30 minutes. For the case most people will hit, it should just be a minute or two.

March 30, 2016

Kevin Fenzi : taskwarrior tips and notes

A year or so ago some co-workers pointed me at taskwarrior ( https://taskwarrior.org/ ) as a neat way to do todos and task tracking. At the time I tried to dive into it and it just didn’t seem to work out for various reasons ( I didn’t give myself free time to really dig into it, tried to make it do time tracking as well, etc). Last week and over this last long weekend I gave it another go and I think it may stick this time. I thought this might be a good time to share the tips and workflows I came up with in case anyone out there wants to give taskwarrior a try.

First a bit of background: I’m a sysadmin by trade, so my todos and tasks are often tons of short tasks and reminders along with meetings and longer term planning/thinking items. For many years my workflow has been to use my mail client inbox along with a text file for things. If it was in my inbox it means I need to take some action or reply to something. If it’s in the text file it’s something I have already done (to keep a record) or some short term todos of things that don’t have email associated. This setup works pretty well, but it’s bad at followups (where I replied/did something and the other party didn’t), anoying for repeat items (have to use a template for the text file or copy and paste), and bad at searching for particular things.

Enter taskwarrior: The new workflow I’ve settled on pretty much replaces my text file with taskwarrior, so I still have mail inbox, but taskwarrior helping it out. There’s extensive taskwarrior docs on the web and the man page is pretty nice too. It’s very nice that taskwarrior is flexable, it can be used a bunch of different ways with different workflows and methods. You can set things up to use it the way you want to. Above and beyond those things:

  • Setting up meetings confused me, because I don’t want to see them until near the meeting time/day, until I realized I can use wait to make them wait until I want to see them. For example, here’s the EPEL meeting thats every wed at 18UTC: ‘task add ‘EPEL meeting’ project:work recur:weekly due:2016-03-30-12:00 wait:soww’ (soww is start of work week). So, I won’t see this on my task next list until monday when I am looking at my upcoming week.
  • tags are very handy. For example I have been adding +nagios to any task where I am looking at or acting on an alert. That way I can go back after a while and look and see if there’s any trends or things we really want to try and fix because they re-occur. ‘task completed +nagios’ will show all these completed nagios tagged tasks. (likewise, ansible commits, bugzilla, infratrac and internal only tickets)
  • ‘task edit <id|uid>’ is pretty handy to see what task has on a task or tweak it if you added it wrong.
  • If you really do want to wait forever for something, you can use ‘wait:someday’ or ‘wait:later’ and it will set it to 2038. 🙂
  • You can also add ‘until’ on meetings for example for just after they would end or the like, and task will delete that task if it reaches the until time. This is handy for example if you miss a meeting, you didn’t do it, it should just be removed, but you don’t even have to think about it if you add it with an until.
  • Another cool one from Paul Frields: You can use ‘task log …’ to create a task thats already marked done. This is handy if you already did something and didn’t have a chance to make a task before hand. Saves you adding a new task only to mark it done.
  • Out of the box, task doesn’t keep track of how much time you spent on any given task. You can use some hooks to record this, but it doesn’t do it by default. I think the last time I tried it out this might have confused me, but this time thinking about it, I am not sure I care exactly how many minutes I spend on each task.
  • I’ve found it handy to add text as much as I can. When adding the task ‘task add …’ (describe it as simply and completely as you can), when starting the task ‘task N start …’ (usually here a link if it’s to a ticket or bugzilla thing where I know the url when starting)’, and when completing the task ‘task N done …’ (here a commit url or the like with the work done). Or ‘task N annotate …’ anytime you made some progress and it’s an interesting or noteworthy thing to add.

Taskwarrior does have a server you can sync to (taskd). It’s pretty easy to setup and uses TLS certs to authenticate clients. There’s also an android app, which is kind of painful to configure, but seems to work otherwise. I’ve not done all that much with it yet, but it might be handy if you are at a event or the like and want to record a task but don’t have your laptop out. There’s also bugwarrior that lets you pull from trac/bugzilla/pagure/etc into task. I did that the last time I tried things out, but I think it’s not a good workflow for me. When I do work on a ticket or bug I can add that task manually, and leave most/many of them that are not waiting on me to sit in their space.

Reports are quite handy for seeing how much you have done and what you have done in a time period, and pretty easy to setup. Just add them to your .taskrc with what to filter on and what cols to show. I setup one for what I have completed for the current work week and what I completed for last work week (useful in weekly reports to management).

Still getting the hang of things, but I am hopeful that I’ll stick with taskwarrior at least for now.

March 09, 2016

Kevin Fenzi : encrypt all the things: blogs

So, with ssl certificates pretty easily available these days from letsencrypt.org it’s more and more worth looking at making sure you are using https instead of http for everything you can.

There are however some corner cases that are… difficult. One of those is blog aggregators. A while back we moved our planet.fedoraproject.org to fedoraplanet.org. This was to get it out of the fedoraproject.org domain because we send HSTS headers for fedoraproject.org telling browsers they should always contact that domain with https. Blog aggregators are in a tough spot, because they simply pull content from a bunch of different sites and put it in one place and link to it. Unless 100% of the blogs that the site is aggregating are also https, if the site itself uses https most browsers will show you a nasty warning about mixed encrypted/non encrypted content. So, for now, since most of the blogs on fedoraplanet.org are http, we are leaving fedoraplanet.org itself as http.

However, we would love to get to the point where all the blogs we aggregate are https. I don’t think it’s an impossible journey. Here’s what you can do if your blog is listed in fedoraplanet.org:

  • Check to see if your blog site already supports https and has a valid cert. If so, you simply need to login to fedorapeople.org and edit your .planet file to use the https link instead of http. Done.
  • If your blog site doesn’t support https (yet), ask your blog provider about adding it. They should be able to add a letsencrypt.org cert pretty easily.
  • If your blog site doesn’t support https(yet) and you run your own blog, why not add https support?

If we can get a critical mass of blogs using https, we can look at switching the site over too. Help us out. 🙂

February 24, 2016

Kevin Fenzi : A Fedora Distribution download primer

With the fresh news of a compromise in the Linux Mint distribution images, I thought I would take a few minutes to explain how Fedora handles image downloads and what you can do as an end user to make sure you have the correct and official Fedora images.

First, lets take a look at what happens in each step if you open your browser to getfedora.org (our install images download site):

  • You type ‘getfedora.org’ in your browser.
  • First, your operating system asks your dns servers for the IP address of getfedora.org. If your OS is using dnssec, then it will get a cryptographically signed answer. If not, it will get whatever answer your dns servers give it.
  • Next your browser may try and connect to getfedora.org via http. We have getfedora.org setup to redirect all http requests to https, so this would get you a redirect.
  • On the first https connection to getfedora.org, we send a HSTS header. This tells your browser (if supported by it) that it should ALWAYS use https to talk to this site. Even if you enter http://getfedora.org, it would just correct that and connect on https.
  • Once connected you can download distro images by clicking on the download link for the image you like. Once you click on a download (unless you have completely disabled javascript), there’s a screen describing how to verify your download: https://getfedora.org/verify
  • Once you have downloaded your image, you need to do two things to make sure it’s the valid and official image: First, check the gpg signature of the checksum file. Official checksum files in Fedora are always signed. You can get the gpg key for that Fedora release from getfedora.org, most any keyserver, or from the fedora-repos package if you already have a Fedora install. Additionally, if you import this key and then refresh (gpg2 –refresh-keys) you can see the signatories of that key and decide based on all that if you trust it. If thats correct, then you can use sha256sum to check the checksum of the image. YOU SHOULD ALWAYS DO THESE CHECKS. 🙂

So, we have dnssec, hsts and signed checksum files. Would that have helped us any if we suffered a attack similar to the one the Linux Mint folks suffered? In that attack, their download machines were compromised and intruders replaced checksum and download links to their own version. If that happened to Fedora, the only step above that would protect people would have been the gpg signature check (which sadly, many folks never do, and for good reason, it’s hard and anoying and manual).

In Fedora 24 the workstation and server editions will be moving to preferring the usb media creator application instead of preferring direct downloads of images. We will need to make sure it’s as secure as we can make it, but there may well be a manual step of checking the application after you download it. (Unless you already have Fedora and install it as a normal package). In it’s current form it already downloads checksum files from the Fedora master mirror via https and uses that to check downloads, but more can be done.

February 21, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-02-21 edition

Once again I have been busy and haven’t gotten a notes from the trail out recently, time to correct that!

Just a bit over two weeks ago we had a mass rebuild event in rawhide. This one was for gcc6 which just landed in rawhide. The actual rebuilding took (much like the last one we did) about 36 hours, but thats just the tip of the iceberg. From the devel-announce post about the mass rebuild completing: “16129 builds have been tagged into f24, there s currently 1155 failed builds that need to be addressed by the package maintainers.” Dealing with those failed builds is always the intensive part of any mass rebuild cycle, since it takes maintainer time to sort out whats going on (and often upstream communication) to fix.

In just a few days we will be branching off F24 from rawhide. Do make sure when this happens on tuesday that you follow the direction you want: Either keep on rawhide or follow Fedora 24 until it’s release in June.

With the mass rebuild and a bunch of other changes people are landing before the branch event there’s been some issues with the rawhide install path again. If you need to install rawhide, do check https://openqa.fedoraproject.org/ for the last image that passed install tests and save yourself time and energy before downloading.

Finally we are close to landing the changes in the compose process around pungi4. This will make every day’s rawhide just like a release compose. It will have all the trees, images, checksums, metadata and such every day. This will also be used for the Fedora 24 cycle composes, so everything should be a good deal more consistent.

January 24, 2016

Kevin Fenzi : A tale of that most hated hardware: printers

Ask any sysadmin what hardware they have to deal with is the worst and I think universally they will all tell you it’s printers. There’s a lot of reasons for this, but I think the biggest ones are:

  • They use / misuse consumable items like ink, paper, etc.
  • They have a ton of moving parts to jam or break or get blocked.
  • They tend to be in places where end users are asked to service them and often put in incorrect paper, jam things or misrefill ink, etc.
  • Recent trends have made printers lowest margin items. If your printer doesn’t work, throw it away and get a new one, it’s cheaper than fixing or refilling your existing one.

A few years ago I got a HP multifunction printer (when it’s predecessor broke and it turned out to be cheaper to just buy a new one). The scanner part of it works great and always has. The printer part on the other hand has been nothing but horror. I don’t tend to print much at all, but a few times a year for some reason or another I need to print a document or directions to something or whatever and every single time this printer has failed me on the first print attemps. Reasons for it’s breakage have included:

  • It’s been so long since I printed the ink “dried up”. Once I was able to get it working by cleaning it with a q-tip, once by making the printer run it’s clean print heads cycle about 20 times, once by buying a new cartridge (almost as much as a new printer).
  • Once it decided to just not turn on at all. Unplugging it completely and waiting 5minutes and replugging it “fixed” it.
  • Once the color ink cartridge was low so it wouldn’t print anything (even just using the black cart that was full!) which required a new color cartridge.

Over the holidays I cleaned out a lot of my computer room, so I could see all the printers I still had that I hadn’t managed to throw away. 2 old HP inkjets, an old epson, and… what is that at the back of the room? why it’s my old HP laserjet 4P. I used that printer extensively during college and after and finally retired it when the tonor ran out and I wanted to move to a new shiny color inkjet printer.

So that got me thinking. The laserjet 4P only has a parallel port, but a quick check found a parallel to usb cable for $8. At the time I retired it, a tonor cartridge for the 4P cost around $100, but looking now, I found one for $12. A quick amazon order and a few days the items arrived. I hooked up the printer to my main server and put in the new tonor cart, added it to cups and printed a test page. A lovely 600dpi test page.

So, hopefully this laserjet 4P will now print reliably for me for the rest of my life and I can avoid the inkjet “buy a new one” everytime I need to print something.

January 18, 2016

Jeffrey Haemer : Another Sale!

The last one was "Best Sellers." Now the video is also "Best Reviewed." Nice!

InformIT.com - Best of 2015 Sale

Kevin Fenzi : A vist to a Fedora datacenter

I just got back from a visit to the largest of the datacenters used by Fedora Infrastructure, and I thought I would share a bit about why we do such visits what what we do on them.

We have a number of sites where Fedora Infrastructure has at least one machine, but we have one main datacenter (sponsored by our main sponsor, Red Hat) where all our build system machines are, our fedorainfracloud openstack cloud, and most of our main servers are. This site does have some on site folks who can handle most of our day to day hands on needs, but sometimes we have things that need a lot of coordination, or extra time to complete, so we try and ‘save up’ those things for one of our on-site visits.

(Side note: If you are interested in donating machines/colocation space for the Fedora Project, please do mail admin@fedoraproject.org. We would love to hear from you!)

On this particular visit last week we got quite a few things done:

  • Package maintainers will be happy to hear we racked a new HP Moonshot chassis. This will hopefully allow us soon to run armv7 builder vm’s on aarch64 virthosts which will vastly decrease build times.
  • We moved 5 machines to a new rack to be used for testing storage solutions. We want to see if we can move to more gluster or open source storage solutions. These machines will be used to test out those solutions as they mature.
  • We added memory to a number of cloud compute nodes, allowing us to run more and larger cloud instances (and have more copr builders).
  • We added a new equallogic storage device to our cloud network. This is a smaller one than we have now, but it will allow us to test things and also give us some more cloud storage room.
  • We fixed some firmware on some power supplies (yes, power supplies have firmware too!). This involved an elaborate dance of moving power supplies around in various machines to get them all upgraded (which would have been very tedious remotely).
  • Moved an old remote kvm and added a new kvm. We don’t tend to use these all that much, but they are very handy for machines that don’t have proper out of band management.
  • Pulled tons and tons of unused cables out of all our racks. On site folks don’t tend to want to remove cables, only add new cables for new devices. There was a bunch of old cables that were only connected to a switch or serial port or power device and were no longer needed.
  • Setup a power7 machine (many thanks to IBM) in our cloud network. As soon as we have it installed and setup we should be able to have ppc64 and ppc64le cloud instances and also do copr builds of those arches locally instead of against a remote server.
  • Updated our records of which servers were on which serial, power and switch ports. This information is very important when making remote requests or power cycling servers so you don’t accidentally get the wrong one.
  • Pulled out a few older servers we were no longer using.
  • Racked a few new servers we will be using for releng/buildsystem stuff.
  • Checked all the servers for any error lights or alarms. Cleared a few we had already fixed, and fixed a few we didn’t know about yet.

All in all a good weeks work that should set us up well for the next few months. I would like to thank Steven Smoogen (for being on-site and getting all the above done with me) and Patrick Uiterwijk (who held down the Fedora Infrastructure fort while we were off at the datacenter and helped get all the above done from the remote side) as well as Matthew Galgoci (Red Hat networking guru extrodinare) and our On-site folks, Jesse Iashie and Pedro Munoz. It’s a pleasure to work with you all.