December 03, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-12-03 edition

Well, it’s been a while but I thought I would share news from rawhide both from the previous month or so and upcoming.

With the release of Fedora 25 a number of developers and testers are switching back to rawhide in the run up to the Fedora 26 branching event. Welcome back to the trail!

In the last month there’s been various small rawhide issues:

  • openssl 1.1.0c came out and had a number of issues. In particular it broke authentication to koji, so any package maintainers had to downgrade back. It also broke a bunch of tests in python, which has delayed the landing of python 3.6 in rawhide. There’s a patched version of 1.1.0c that landed last week, so everyone should be ok to update openssl again.
  • The latest systemd version landed, but then had to be untagged again because it broke a number of parts of compose. Hopefully those will get worked through and fixed.
  • More recently (The last week) a kf5/qt5 update is in progress and isn’t fully finished. Rawhide users will notice a number of packages dnf says it’s skipping due to broken dependencies. This should get fixed up soon, so just keep skipping those items.
  • Finally, the last few days we have run into an odd problem that looks like it might be related to RHEL7.3 squid (which is whats on kojipkgs.fedoraproject.org and used for package downloads for builds). Sometimes packages just don’t seem to download and it breaks the compose, but testing later they work fine. We downgraded squid back to the 7.2 version for now and composes seem to be working again.

Finally in exciting upcoming news: There’s a “flag day” on december 12th (2016-12-12) for a bunch of changes that release engineering has been wanting to make for a long time. One of these changes will finally allow us to do the last thing we need to do to get a 100% signed and gated rawhide. There will be a change in fedpkg (as well as some koji server changes) that will make rawhide builds land in a ‘f26-pending’ tag instead of just going directly to ‘f26’. Then, our autosign process will sign them and retag them to f26. We also can add in automated QA here to decide if a build is ready to be tagged in or not. I hope next year that we can add a bunch of tests here and make rawhide more and more stable.



October 24, 2016

Kevin Fenzi : Security Score Card For Fedora Infrastructure

Josh is asking folks to send him their security score card via twitter. Since I’ve been trying to blog more and like pontificating, I thought I would respond here in a blog post. 😉

There’s 4 parts to the scorecard:

  1. Number of staff
  2. Number of “systems”
  3. Lines of code
  4. Number of security people

For Fedora Infrastructure, some of these are pretty hard to answer, but here’s some attempts:

  1. Fedora Infrastructure is a Open organization. People who show up and start doing things are granted more and more permissions based on their merit. Sometimes people drift away to other things, sometimes new people show up. There’s some people employed by Fedora’s primary sponsor Red Hat, specifically to work on Fedora. Those account for 3.5 sysadmins, 5 applications developers, 2 release engineers, and 2 design folks. Specific areas will have potentially lots more community folks working on them. So, answer: 13-130?
  2. This one is easier to quantify. We have (almost) everything in ansible, so right now our ansible inventory + some misc non ansible hosts is around 616 hosts.
  3. This is another one thats difficult. We have a lot of applications (see https://apps.fedoraproject.org/) Some of them are just upstream projects we have instances of (mediawiki, askbot, etc). Others are things where we are primary developers on (fedocal, pagure, etc). It would be a fun project to look at all these and count up lines of code. Answer: dunno. ;(
  4. If this is full time security people working only on security issues, then 0. We do have a excellent security office in Patrick who is super smart and good at auditing and looking for issues before they bite us, but he’s not doing that full time. Others of the sysadmin teams do security updates and monitoring lists/errata and watching logs for out of the ordinary behavior, but thats also not full time. So, answer: 0 or 1 0r 3?

So, from this I think it would be nice to have a better idea of our applications (not lines of code), but just where to keep track of things better and who knows that application. It would be awesome to get some full time security folks, but I am not sure that will be in the cards.

I’d like to thank Josh for bringing up the discussion… it’s an interesting one for sure.



October 23, 2016

Kevin Fenzi : Another Fedora cycle, another painless Fedora upgrade

As we near the release of Fedora 25, as always I upgrade my main servers to the new release before things come out so I can share any problems or issues I hit and possibly get them fixed before the official release.

As with the last few cycles, there was almost no problems. The one annoying item I hit was a configuration change in squid. In the past squid allowed you to have a ‘http_port NNNN intercept’ and no regular http_port defined, however the Fedora 25 version ( squid-4.0.11-1.fc25 ) fails to start with a cryptic: “mimeLoadIcon: cannot parse internal URL: http://myhost:0/
squid-internal-static/icons/…” error. It took me a while to find out that I needed to add a ‘http_port NNNN’ also now.

Otherwise everything went just fine. Many thanks to our excellent QA and upgrade developers.



October 22, 2016

Kevin Fenzi : some IOT ideas

With the massive denial of service against a large DNS provider (in turn causing a lot of other things to have outages) on friday using a network of insecure IOT (Internet of things) devices (mostly cameras), a lot of folks are thinking about how to address IOT problems. There are a lot of problems and no easy answers, but I thought I would throw out a few ideas here and see if any resonate with others.

First, the problems: IOT devices are insecure and easily taken over for denial of service or other attacks, there is little to no economic incentive to make them more secure, consumers largely don’t care as long as they are still doing whatever their main function is, devices seldom get updates and when they do only for a short time before the company that made them moves on to something else.

Some ideas:

  • Make things vastly more cheap. Yeah, thats right, more cheap. To the point where you can dispose of or toss in recycling the device that just went out of support or was found to be used in some botnet. Or 3d print a new device. Of course this is not going to happen for a long while.
  • Make updates very reliable. I think we can do this with something like ostree/atomic + some wrapper hardware. Upgrade, reboot, if things don’t come back in X minutes, reboot back to the last working version and send for help.
  • Get IOT devices to use mainstream open Linux distros. These would provide source, upgrades, and device makers wouldn’t have to care about not supporting things. This would probibly take legislation and a big push for Linux distros to cater to IOT devices, and there would likely still be some hardware specific code (but it could be open and maintained in distros).
  • Require Internet insurance for users of ISPs. ISPs that police botnet and other harmful devices would pay lower raates than ones that didn’t care, money raised could be used to shut down harmful devices.

It’s a pretty difficult problem sadly, and there’s no good answer, but we are going to have to start doing something or start getting used to large DDOSes taking out a bunch of things we use often.



Kevin Fenzi : xfce4-terminal: lots of progress recently

Most Linux desktop’s have their own terminal programs, and Xfce is no exception. There’s been a number of releases of xfce4-terminal recently, so I thought I would share some of the changes with people who perhaps haven’t tried it recently.

The biggest change is that it’s been ported to gtk3 and vte291. This is great news as it means it’s no longer using the no longer supported at all ancient version of vte. Unlimited scrollback has been added, wayland support, tons of bug fixes, translation updates, and memory leaks quashed.

More specifics in the NEWS file: https://git.xfce.org/apps/xfce4-terminal/tree/NEWS

Give it a try today and see if it handles all your terminal needs.



October 20, 2016

Kevin Fenzi : keepalived: Simple HA

We have been using keepalived in Fedora Infrastructure for a while now. It’s a pretty easy to use and simple way to do some basic HA. Keepalived can keep track of which machine is “master” for a IP address and quickly fail over and back when moving that IP address around. You can also run scripts on state change. Keepalived uses VRRP and handles updating arp tables when IP addresses move around. It also supports weighting so you can prefer one or another server to “normally” have the master IP/scripts.

Right now we are keepalived on our main koji server pair. We have a koji01 and koji02. Normally 01 is primary/master and has the application IP address on it so all traffic goes to it. If for some reason it was turned off, the keepalived on 02 would see that and take the IP address and run a script to become master. If 01 came back up, 02 would see that and transfer back to it. Right now we have the scripts setting up on the secondary server a bunch of cron jobs (garbage collection) and kojira (the process that regenerates build roots).

We are also using keepalived on some new paired postgresql instances we are working on. More on that in a later blog post. If you need simple HA with a IP address and script(s), keepalived does a banner job.



October 19, 2016

Kevin Fenzi : pass – The standard unix password manager

In my line of work (sysadmin), I have to deal with a LOT of passwords. For a number of years I was a fan of keepassx, but the upgrade to version 2.0 there didn’t thrill me. (There were some features I liked that it dropped and in general it seemed to be less nice), so I decided to look around and was pointed to pass.

Pass (or password-store as it’s sometimes called) can be found at https://www.passwordstore.org/ and in most major Linux distributions package set. (It’s called just pass in fedora). It’s a simple command line tool and uses gnupg and git (if you like). Each of your passwords / sites is a gnupg encrypted file, setup in a tree under ~/.password-store. You can tell pass also to use git here so every change you make it a git commit of encrypted files. So, you should be able to find any old passwords you changed in git history if needed.

Setup is pretty simple:

  • pass init yourgpgid@example.com – this sets up the base directory for pass. Note that you can actually add additional gpg keyids to encrypt files to later if you have a team or the like.
  • pass git init – this sets up the git repo

After setup you can use ‘pass insert path/to/name’ to add your own password, ‘pass generate /path/to/name’ to have pass generate a random password for you (using pwgen), ‘pass ls’ to list the tree of sites, ‘pass -c /path/to/name’ to copy the password to your clipboard for easy inserting into another application or website. (by default it will stay in the clipboard for 45seconds or 1 paste and then vanish). Note that you never even need to know the password here, you just get it from pass and paste it in and are done. If for some reason you do need to see it (a broken app that won’t let you paste for example), you can just do ‘pass /path/to/name’ and it will output the password. You can do ‘pass edit /path/to/name’ to edit a password, and note that you can add whatever you like to the pass file. pass -c will use the first line as the password, but you can add more lines with security questions and random answers, usernames or any other notes you like about the site. ‘pass git ‘ will let you run any arbitrary git command on your pass git repo, if you wanted to look at history or go back to some previous commit.

There is a android app that is supposed to work with pass files, but I have not tried it. It requires you to copy your gnupg private key to your phone, which is not something I am really wanting to do. Since I have my laptop with me most all the time anyhow, it’s not a big deal.



October 18, 2016

Kevin Fenzi : debugging koji build failures

From time to time builds fail in koji (The Fedora build system), and it’s good to know how to figure out where to look for the reason. Koji has a central hub that manages jobs and a bunch of builders that actually do the builds. When someone initiates a build you are talking to the hub and either uploading a src.rpm (for a scratch build) or telling it to use a particular git hash/repo for an official Fedora build. For official builds, koji will first generate a job to build the src.rpm from git and the packages lookaside cache with the source. This job will run on some builder thats ready and has capacity for it. Once the src.rpm is generated (or if you are providing it for a scratch build), the hub will generate build tasks for all the arches that are set in the target tag you are building to. Each one of those will go out to a builder of the right arch type that is enabled and has capacity, etc. If any of these fail, the entire build fails.

Once you have gotten notice of a failure, the first thing to do is go to the link for the build in the koji web interface and then click on the task link to show the build tasks. There you will see one (or more) links in red that failed. Don’t click on any of them yet though, instead click on the “show result” link. That will tell you exactly which task failure failed the build. When one failure happens koji cancels the other builds, but sometimes they fail at nearly the same time and you are looking at the wrong place for the real failure. Sometimes all the links are green and the reason for the failure is only visible in the “show result” area (for example, if you have a archfull package with noarch subpackages and they are different on different arches, thats a failure, or if you don’t have permissions to build the thing you are building, thats a tagging failure).

Next click on that task that you saw in the “show result” link (if you haven’t already seen the error as above), and again click on the “show result” link. You should see something like:

"BuildError: error building package (arch i686), mock exited with status 1; 
see build.log for more information"

Or it could refer you to the root.log. Go to that log and look toward the end for the error. If the error is in the root.log it means there was a problem setting up the mock chroot. That could be that there is a package you BuildRequire that in not installable, or some package in the build root is completely broken.

If the issue is in the build.log its the actual output from the rpmbuild, so this would be where to look at your general compiler issues, etc.

If you ask someone to help you out and look at the error(s) you are getting, it’s important to give them the top level link to the task instead of just a link to the root.log or build.log. If you only have those links it’s impossible to back up and look at the higher levels, but if you have the top level task you can look at the entire thing.

Aside from normal rpm package builds, koji also builds a bunch of other things like livemedia and other images. Finding issues there is very similar. Always check the “show results” area for which logs to look at for the actual failure.



October 17, 2016

Kevin Fenzi : gnome extensions

In general when using Gnome I try and avoid extensions. Most of the ‘default’ setup is fine or I was able to get used to it and it works well enough. There are a few extensions I do use however for various reasons. All of these work with Wayland.

  • AlternateTab – I use alt-tab a LOT. I just found myself unable to get used to the default behavior of showing applications and windows under those, when I use alt-tab I want to switch to a specific window quickly, not have to switch to a application and arrow key around to find the one I want.
  • Openweather – This is just fluff, but nice to have. Gives you a nice little weather report in the top bar. You do have to pick a provider and get an API key, but it’s pretty painless. You can also setup multiple locations.
  • PanelOSD – This one I use to move notifications over to the top right side. This is where I am used to them being from Xfce and that way they are not in my way.
  • TopIcons – This allows you to see systray icons on the top bar. There’s still a few applications I use from time to time that have a systray and it’s handy to see them if they do.

I’d probibly include the redshift extension here, but it doesn’t work with Wayland. 🙁



October 16, 2016

Kevin Fenzi : Some interesting gnome 3.22.x gsettings

As I have mentioned in the past, I switch off between Xfce and Gnome desktops here. While in Gnome, I have a number of settings I have picked up in various places (bugs, talking to maintainers, various docs and blogs) and I thought I would share a few in the hopes that they would be of interest to others. You can use the ‘gsettings’ tool or the ‘dconf-editor’ GUI to set or check the current values of these keys. Of course some of them can also be set from the normal settings guis.

  • org.gnome.shell.overrides edge-tiling true – This lets you drag windows to the right or left edge and have them tile to 1/2 of the desktop. This goes along with dragging to the top of the screen to maximize an application.
  • org.gnome.software download-updates false – Disable gnome-softwares downloading of updates. Since I use dnf this saves me a bit of BW and also messages about needing to apply updates I already have applied.
  • org.gnome.nm-applet suppress-wireless-networks-available true – This disables the ‘There are open wireless networks” message. I only want to join specific wireless networks if I need something.
  • org.gnome.settings-daemon.plugins.xsettings hinting ‘slight’ – slight hinting works best with the fonts I like. I think this may now be the default.
  • org.gnome.desktop.wm.preferences focus-mode ‘sloppy’ – I want to change focus when I move the mouse. I am pretty used to this from years and years now.
  • org.gnome.desktop.wm.preferences auto-raise-delay 125 – and autoraise on focus change.
  • org.gnome.shell.window-switcher app-icon-mode ‘both’ – I want both the icon and the name in the alt-tab switching thing.
  • org.gnome.desktop.interface clock-format ’24h’ – 24 hour clock is what I am used to. 🙂
  • org.gnome.desktop.interface show-battery-percentage true – shows the battery % left in the menu bar. Handy when you are wanting to see how long you have left and you don’t want to pull down the menu.
  • org.gnome.desktop.interface cursor-blink false – blinking cursors are evil. They suck power are distracting and should be disabled. (I could swear that I saw a site a while back that showed how to disable blinking cursors in any application or system you could name, but I can’t seem to find it now)
  • org.gnome.desktop.interface clock-show-date true – I like the date in the menu bar.
  • org.gnome.shell disable-extension-version-validation true – This disables the version check for extensions. Sometimes it helps, but usually if an extension doesn’t work for me setting this doesn’t help much.
  • org.gnome.desktop.peripherals.touchpad two-finger-scrolling-enabled true – I used to always use Edge scrolling, but it went away for a while and I retrained myself on 2 finger scrolling and it’s fine now.
  • org.gnome.desktop.peripherals.touchpad tap-to-click true – must have tap to click. Saves so much wear and tear on my hands and the touchpad. Just a gentile tap is all thats needed.

Do you have some favorite ones?



October 15, 2016

Kevin Fenzi : Running ansible from a git checkout

The Fedora and EPEL ansible packages are as up to date as possible, but sometimes you want to run a newer, not yet released ansible version for any of a number of reasons:

  • To confirm a bug is fixed.
  • To test an RC for the next release or look for regressions on the devel branch.
  • To see if a change caused any performance improvements or regressions.

Happily this is very easy to do with ansible:

  1. clone the ansible repo: ‘git clone https://github.com/ansible/ansible.git’
  2. checkout the branch or thing you want to test. For example now stable-2.1 branch has the latest 2.1.x rc, and stable-2.2 has the code for the latest 2.2 rc.
  3. ‘source ansible/hacking/env-setup’ This will set your ansible/ansible-playbook commands to use your checkout instead of any locally installed version.

I use this setup to test new versions/RC’s over the Fedora Infrastructure playbooks. It’s very handy to catch last minute bugs in RC’s and/or to clean up depreciations so when the next version is actually released you are already ready for it.

Happy ansible hacking.



Kevin Fenzi : The greatest game in history…

I am of course talking about nethack.

If you haven’t played in a little while (and you HAVE played nethack right?), it’s as fun as you remember it. Running around the dungeon fighting monsters and picking up magical devices. It’s pretty hard to believe that next year will be the 30th anniversary of nethack.

It’s friday, you deserve some nethack today.



October 14, 2016

Kevin Fenzi : ss: the netstat replacement

Many of the old network tools in the net-tools package have been deprecated for a while now. The replacement for the traditional netstat command is ‘ss’.

It of course has a ton of options (like netstat). I particularly like ‘ss -e’ (extended) and ‘ss -E’ (show connections as they appear).

Do take a look today and see if you can finally replace your netstat usage today.



October 13, 2016

Kevin Fenzi : The Fedora infinote server

Many of you may not know that Fedora Infrastructure runs a infinote server instance at infinote.fedoraproject.org.

Infinote is a collaborative text server. You can connect to it with the ‘gobby-0.5’ client located in the gobby05 package in Fedora. Once connected you can create documents and multiple people can work on them at the same time. The server takes a git snapshot of all documents every few minutes so you can see history. There’s even a cgit instance at https://infinote.fedoraproject.org/cgit/

Of course these days there’s etherpad and the like that have more features, but from time to time there’s a good Fedora use for a server we run. Fedora Infrastructure uses it to coordinate their weekly meetings and update/reboot planned outages. Everyone is welcome to use it for their Fedora related collaborative text editing. Or you can of course install the libinfinity package and run your own server.



October 11, 2016

Kevin Fenzi : Fedora 25 Beta is here

This morning we released Fedora 25 Beta: https://fedoraproject.org/wiki/F25_Beta_release_announcement

Release mornings are always a bit stressful for us in Fedora Infrastructure, even though we have gotten pretty good at doing them. Lots of things need to be lined up and ready to go when the announcement goes out: torrents, websites, announcement, etc. Also any networking or hardware issues are particularly unwelcome on these days.

Today everything has gone quite smoothly so far, and I hope everyone enjoys Fedora 25 Beta.



October 10, 2016

Kevin Fenzi : Fedora go/no-go meetings

Before each Fedora milestone release (Alpha, Beta, Final) there is at least one Go/No-Go meeting. As the name suggests this is the meeting where folks from QA, FESCo, Release engineering, Program Management and others gather to see if the release is ready to go out the following week. The meetings are held on IRC and open to everyone (Although go/no-go votes may only be taken from representatives of specific groups). You can see the logs of the last one (last week to decide if Fedora 25 Beta was ready to release this week) this meetbot.fedoraproject.org log link

In the event the decision is “No-go” then the Milestone release slips a week and a new Go/No-Go meeting is scheduled a week out. This allows whatever issues were blocking the release to (hopefully) be solved.

If the decision is “Go”, then the release happens the following week. Why so long between the decision to release and the actual release? Well, there are a number of things that need to happen to get things ready for release, including, but not limited to:

  • The release has to be put in the correct place on master mirrors (moved from the alt/stage directory to releases/test or releases) and time is needed for all our mirrors to pick up this content and sync it.
  • Common bugs need to be identified and written up: https://fedoraproject.org/wiki/Common_F25_bugs
  • The websites team needs to land any changes they need for the release and get them ready to go live on release morning.
  • Torrents need to be synced and setup, ready to live for release morning.
  • In the buildsystem, the release needs to be tagged so we know exactly what packages were in that release and so alternative arches can build the exact same set for their releases.
  • Checksum files and so forth need to be verified and signed by release engineering.

I hope everyone is looking forward to the Fedora 25 Beta release tomorrow!



October 09, 2016

Kevin Fenzi : squeezing out a bit more battery life under Fedora

In my experience, battery life on laptops has been getting better all the time with Linux/Fedora. However, there’s a few things you can do if you need just a bit more battery life than you are getting with default settings.

First let me say that in the ideal world you wouldn’t have to do any of this. Things should just work and give you the best battery savings they can. Sadly, thats not fully the case (yet), so sometimes you have to take matters into your own hands.

First, there’s some packages that try and set things for you: tuned and tlp (and possibly others, but those are the two I know of). If you are using tuned, you will want to run a ‘sudo tuned-adm profile powersave’ to switch to the powersave profile. With tlp it should switch to this automatically when it sees you are running on battery.

powertop has a bunch of recommended settings for power saving which you can automatically apply with ‘sudo powertop –auto-tune’ or by enabling the powertop service. You do need to be careful here however as some of these settings are not set that way by default because they cause lockups or other bad behavior. Do test this on your particular hardware before you trust it.

Finally there’s a bunch of common sense items: lower the screen brightness as much as you can (the screen is by far the largest power user on a laptop), put your bluetooth/wireless in airplane mode so they don’t consume power (if you don’t need them), remove usb devices (some of them suck up power even while not doing much), turn off keyboard backlight and such.



October 08, 2016

Kevin Fenzi : Fedora Infrastructure Outages (and the one on 2016-10-08)

Astute folks may have noticed that we had a pretty severe outage this morning that lasted just about an hour. This seems like a good time to talk about outages in general and this one in specific.

In general these days our outages fall into roughly 3 ‘buckets’:

  1. Planned outages. We do these every few months to reboot servers and apply updates, or when we have to migrate something that will take some downtime. This is by far the largest bucket.
  2. Unplanned outages, but that are small in scope. Typically this might be one of our many datacenters having network problems or a single service having some issue. These typically aren’t noticed by very many people (in most cases only the infrastructure noc folks who disable a proxy or restart a service or whatever is needed).
  3. Unplanned outages that are large in scope. These are thankfully rare. These are were we loose connectivity to our main datacenter (that has the majority of our hardware in it), or severe hardware or software failure (storage servers not working, database needs reloading, etc). Luckily these are pretty rare.

So what happens in an outage? Normally nagios lets us know about it pretty quickly and someone starts taking a look. Usually until we have some idea, we treat things as being in bucket 2 if they aren’t planned. That is, we assume it’s a minor issue until we investigate and see it’s larger. Once the scope of the issue is determined, we update https://status.fedoraproject.org/ This is out status indicator. If you are wondering if there is a known outage, you can look at this. Note that this is MANUALLY updated after initial innvestigation, so do give a few minutes after seeing something before expecting status to be updated. Next, folks work on the outage, this would be anyone in the sysadmin-noc group and the primary infrastructure admins. Status is then updated when the outage is over.

Today (2016-10-08) at approximately 16:15UTC our primary datacenter (PHX2) became unreachable. This turned out to be due to a firewall upgrade that did not go as planned. Service was fully restored at approximately 17:15UTC. Right now when our primary datacenter isn’t reachable most things are also not working as we have our primary vpn hub located there, and thats what our various proxies use to talk to applications. Unfortunately this includes our most active service: mirrorlists. We have planned to move that service to a container and run it at each datacenter, but just have not gotten to deploying that yet. Once we do, at least mirrorlists will stay up if the primary datacenter is down.

Additionally, we have plans to eliminate the first type of outage or reduce it greatly. For the most part our applications are deployed with pairs (or more) of instances so no service needs to go down, however the sticking point is rebooting database servers. I’ve been working on setting up some replication so hopefully we don’t need to schedule any outages for that either.



October 07, 2016

Kevin Fenzi : Rawhide kernels back on track now

Just another quick note that rawhide kernels are back to booting and working normally here at least. With 4.9.0-0.rc0.git3.2.fc26.x86_64 at least. That should be in today’s rawhide.



October 06, 2016

Kevin Fenzi : rawhide 4.9.0 pre rc1 kernels

Just a quick note that with the upstream kernel folks opening up the merge window for the 4.9 linux kernel, rawhide also has been getting these merge kernels. Of course there’s a ton of churn when the kernel merge window is open, vast amounts of things are merged, so it’s a wonder they usually work as well as they do.

In this case there seems to be some pretty severe issues with at least kernel-4.9.0-0.rc0.git1.1.fc26 and kernel-4.9.0-0.rc0.git2.1.fc26 at least. In vm’s they seem to boot and then start oopsing. On my laptop it does boot, but then waits about 2 minutes at waiting for usb devices, boots the rest of the way up and then has no wireless or usb devices.

https://bugzilla.redhat.com/1382134 is tracking at least part of this issue and the kernel maintainers are digging into it. Of course while they are, more things are being merged (possibly even fixes to these issues), so hopefully it will all shake out in the next few days.

In the mean time, rawhide users are advised to just switch back to the 4.8.0 final kernel for now.



October 05, 2016

Kevin Fenzi : 10 basic linux security measures everyone should be doing

Akin to locking your doors and closing your windows there’s some really basic things everyone should be doing with their Linux installs (This is of course written from a Fedora viewpoint, but I think this pretty much applies to all computer OSes).

  1. Choose nice long passphrases you can remember. Most any modern system will have a pretty long limit on passphrases, so pick something nice and long that you can remember. Don’t think of them as passwords, they are phrases with many words.
  2. When installing, encrypt your drive(s). The performance hit is not noticeable and if you ever throw away a broken drive or someone steals your computer they won’t have your data.
  3. Apply updates regularly. If you aren’t someone who remembers to do so, setup something like dnf-automatic to just apply them every day for you in the middle of the night. Otherwise try and get into the habit of letting gnome-software do offline updates at some regular time.
  4. Along with (3), reboot when needed for new kernels or glibc or other things you use. Get used to rebooting on a regular schedule. Don’t be afraid of rebooting, get used to doing it.
  5. If you are in a place with untrusted people roaming around, do setup a screen locker and lock your computer when you are away from it.
  6. Make (and sometimes test) regular backups. You may not think of backups as a security measure, but they sure are. Think of the new fad of ‘ransomware’ where someone encrypts your data and sells you the key. If you have good backups you can just wipe that all out and restore from those. They are handy for lots of other reasons too.
  7. Don’t open weird attachments or links sent to you in email. If you didn’t ask for it, delete it.
  8. Don’t plugin weird devices you run across to your machine. (USB or otherwise). You can use a neat package called ‘usbguard’ to make sure no one else does while you are not around too.
  9. Use a passphrase manager or have some system to allow you to have long, not easily guessed passphrases at all the various applications you login to. There’s tons of these out there: Password managers: pass, keepassx, gpg encrypted file, etc. Schemes: Diceware, etc. Pick one that works for you.
  10. If you use a laptop/travel a lot, consider using a VPN for all your network needs. As long as you have an endpoint to connect to (your home server, your work, a vpn provider) you can send (almost) all your traffic over the vpn and thus avoid problems with people sniffing local traffic.

Some of these things require an initial investment of time (backups, vpn, passphrase manager, screen lock) and some require just making something a habit (long passphrases, apply updates regularly, reboot regularly, don’t open weird things in email or the physical world), but they are all worth it.

Will this make your computer “secure”? No. There’s no such thing. “secure” is not a binary state, it’s a process of assessing threats and deciding what you can or want to do about them. Doing the above things will protect you from some threats nicely (guessable passwords, untrusted people tampering with your computer, sniffing traffic, vulnerabilities that have already been fixed in software you use, etc), but will basically do nothing against others ( someone installing a keylogging device and recording everything you type, someone threatening you with harm to tell them some information, someone installing a spy cam and recording everything on your computer screen, someone using a non public vulnerability in software you use, someone social engineering access to your computer, etc).



October 04, 2016

Kevin Fenzi : Short note: fedoraproject.org smtp sessions now using TLS

Just a quick note for everyone who gets emails from fedoraproject.org or it’s various other domains. Just before we entered freeze for Fedora 25 Beta we landed changes to make our smtp servers use TLS where possible, so emails to servers that support it should now be fully encrypted.

Please let us know if you see any issues related to this change.



October 03, 2016

Kevin Fenzi : How to debug Fedora rawhide compose problems

From time to time rawhide composes fail and are not announced or synced to mirrors.

In the past this would happen only if the very basic setup (a mock chroot with the ‘buildsys-build’ group installed in it) broke. Additionally, in the past rawhide composes where many deliverables failed to compose were still synced out and announced, leading to days when no images were available until the issue was fixed.

Now, with the latest version of pungi (The tool that composes Fedora releases, including rawhide), composes can fail if some deliverables (Those marked in the configuration as not failable) didn’t complete. So, while rawhide can fail more easily, it also means it’s much easier to revert some change that broke images and get that fixed before it lands, and images should be always available.

So, how can you tell if a rawhide compose (or some part of it) failed and why? All the pungi logs are avilable and since all the builds take place in koji, anyone can look at them as well. Rawhide composes are of course fedmsg enabled, so you want to look for the https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.pungi.compose.status.change topic. Composes can finish with 3 states:

  1. FINISHED – This means the compose finished and everything in it completed successfully. I am not sure we have yet seen this status in real life. 😉
  2. FINISHED_INCOMPLETE – This means the compose finished and only failable things failed. This is the “normal” status we see day to day.
  3. DOOMED – This means the compose failed it’s initial very basic setup and/or some deliverable marked as not failable failed. When this happens, it means the compose isn’t synced out or advertised. This is the status where we need to find out what caused the problem and fix it and either restart the compose or wait for the next days compose. In IRC or on fedmsg you may see this status as “failed in a horrible fire” as thats what our fedmsg translates it to.

So if you have a compose and you want to see why some part of it failed, you can look at the fedmsg and it provides a location url, but they are always the same format. Lets look at an example: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20161002.n.0/ This means it’s a rawhide compose (there’s a Branched directory for the Branched Fedora 25 right now), then the particular compose was Fedora-Rawhide-YYYYMMDD.n.0. Yeah, month and day, then a ‘n’ to mean ‘nightly’ and a ‘0’ to mean it was the first compose of the day. If there’s more composes that same day, they are in n.1, n.2, etc. For Alpha/Beta/Final releases there’s no ‘n’ there, as they are not nightly.

The first place I go to look for issues is the global log file. This would be in logs/global/pungi.global.log and you will want to look for the image or deliverable you are interested in or search generally for tracebacks. Usually this will note a koji task id or build which you can then look up on koji. Since this is already getting long, I’ll post about tracking down koji build problems another day.



October 02, 2016

Kevin Fenzi : Rawhide notes from the trail, the 2016-10-02 edition

It’s once again been a while since one of these posts, but again I’m going to try and do them more often again.

This last week had a few big changes in rawhide:

  • With the Fedora 26 change approval last week ( https://fedoraproject.org/wiki/Changes/DNF-2.0 ) DNF-2.0 has landed in rawhide. Things were a bit shaky at first as the compose tools were not also updated, but anaconda folks were ready with a patch and lorax got a patch from the DNF maintainers and everything landed. There’s still one big issue outstanding: dnf 2.0 sets strict group installs back to true, which causes all the live media to fail to compose. Some of our comps groups contain packages that only exist in some arches, not all of them, and when dnf does strict group installs it errors when it cannot find all the packages in the group. I’ve filed a bug for figuring this out: https://bugzilla.redhat.com/show_bug.cgi?id=1380945
  • Last year we setup a quick and dirty autosigning setup for rawhide, which was just a script in a loop run by releng folks when they remembered to do so. This meant it wasn’t all that reliable. Now, thanks to Patrick Uiterwijk, we have a real autosigning setup implemented and are very very close to an always signed rawhide. The final bit of gating needs a update to fedpkg to go out first, but as of last week, most everything should be signed with only the rare issue with a build that lands right before the compose starts.
  • There has been some work the last few weeks to produce a rpm ostree for rawhide that is updated every few minutes. Once this is available it will be a great help to testing and running rawhide.
  • Xorg server 1.19 almost landed, but we need to fix tigervnc before it can (at least without breaking all the install paths for rawhide). That is being fixed in a side tag and should land as soon as tigervnc is fixed. https://bugzilla.redhat.com/show_bug.cgi?id=1380879 is tracking that work
  • A while back changes were made to cause composes that fail when some deliverables don’t compose. This has been very helpful in making sure changes that land don’t break things. New package builds that cause the entire compose to fail can be untagged until the breakage is fixed.

That’s it for now from the rawhide trail…



Kevin Fenzi : How to reignite a flamewar in one tweet (and I still don’t get it)

disSome of you may have seen this post last week:

https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_tweet

I saw it I think shortly after it was posted to hacker news (aside: I read hacker news via it’s rss feed that has just the raw links and no comments or ratings, much nicer IMHO).

My first thought on reading it was “Did they report this upstream?” and upon some digging, yes they did ( https://github.com/systemd/systemd/issues/4234 ). It was filed the same time they posted their blog post/tweet (no responsible disclosure here). My next thought was to see if it worked, so I tried it on various of my test machines and it did not. Turns out you have to run it in a loop (as noted now in the blog and seen in the systemd issue). At that point I saw that there was a Fedora tracking bug, someone requested a CVE and there were patches proposed on the systemd issue (which has since been closed/fix merged), so I moved on with life.

Next someone posted the blog link to the fedora users list along with a rebuttal from a systemd maintainer: https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post, and the discussion went downhill quickly from there. Finally, I saw a slashdot post today with the calm, journalistic title of Multiple Linux Distributions Affected By Crippling Bug In Systemd (which points to the blog post above that started all this).

So, everyone is flaming everyone else again about systemd, and I just don’t get it. My personal relationship with systemd is well within the range of all the other Open Source projects I use every day. Mostly I am happy with it, sometimes it has bugs and I report them, sometimes the project makes decisions I don’t like or agree with, sometimes the project doesn’t communicate as well with downstream as I would like (see fedora devel list post about systemd and TasksMax ), sometimes I have to learn (again) how to do what I want to do, sometimes it’s slow when I want it to be fast and so on. Would a local denial of service attack in another project have (re)opened such a bunch of flames? There was in fact just the day before the post a remote denial of service in bind (the most popular dns server out there). Did you even hear about it?

I just don’t get the passionate hatred systemd has. I guess perhaps I never will, and I guess thats ok. I would kindly ask those who do passionately dislike systemd to just move on to somewhere without it and leave the rest of us alone, but I don’t expect thats likely to happen. In the mean time if you have some technical problem with systemd, I will be happy to help you isolate it and file a bug or teach you how to do whatever you are trying to do with it, but I’m going to try and stay out of your flamewars.



September 25, 2016

Kevin Fenzi : 6 months a task warrior

A while back I added a task to my taskwarrior database: evaluate how things were going at 6 months of use. Today is that day. 🙂

A quick recap: about 6 months ago I switched from my ad-hoc list in a vim session and emails in a mailbox to using task warrior for tracking my various tasks ( http://taskwarrior.org/ )

Some stats:

Category Data
Pending 17
Waiting 20
Recurring 12
Completed 1094
Deleted 18
Total 1161
Annotations 1059
Unique tags 45
Projects 10
Blocked tasks 0
Blocking tasks 0
Data size 2.0 MiB
Undo transactions 3713
Sync backlog transactions 0
Tasks tagged 39.8%
Oldest task 2016-03-24-13:45
Newest task 2016-09-25-10:03
Task used for 6mo
Task added every 3h
Task completed every 4h
Task deleted every 10d
Average time pending 4d
Average desc length 32 characters

Overall I have gotten a pretty good amount of use from task. I do find it a bit sad that I have only been completing a task every 4 hours and adding one every 3 hours. At that rate things aren’t going to be great after a while. I have been using annotations a lot, which I think is a good thing. Looking back at old tasks I can get a better idea of what I did to solve a task or more context around it (I always try and add links to bugzilla or trac or pagure if there’s a ticket or bug involved).

I’d say I am happier for using task and will continue using it. It’s very nice to be able to see what all is pending and easily add things when people ask you for things and you are otherwise busy. I’d recommend it to anyone looking for a nice way to track tasks.



August 29, 2016

Kevin Fenzi : two vpns and NetworkManager? no longer a problem…

NetworkManager has pretty handy vpn handling for laptops. You can setup all different kinds, it can prompt you for passphrases, it can set up specific vpns on specific networks, etc.

However, if you had more than 1 vpn you wanted to run at a time you had to pick one for NetworkManager to handle and do the other(s) outside NetworkManager. This is happily no longer the case (at least with NetworkManager 1.4.0): It can bring up as many vpns as you like and manage them all.

This weekend I took my home openvpn connection (which I had setup as a system wide openvpn service) and moved it into a NetworkManager vpn profile. It works just fine with my work vpn and now NetworkManager can bring it up automatically after it connects to the network.

So, if you need two (or more) vpns on your laptop, you can now happily move them all into NetworkManager. Many thanks to the hard working NetworkManager folks. 🙂



July 25, 2016

Kevin Fenzi : Looking forward to flock 2016

Just over one week until flock ( https://flocktofedora.org ), Fedora’s main yearly conference. This time it’s in Kraków, Poland. This of course means a long time traveling for myself and other North American Fedorans, but it’s always well worth it.

In addition to seeing old and new friends, I’m looking forward to quite a lot of the talks (see https://flock2016.sched.org/ for the full schedule) along with workshops and hallway discussions.

I’m going to be giving a talk on using Fedora Rawhide as your daily driver OS. Hopefully lots of tips and workflows that people will find useful. Also, I am co-presenting with Pierre-Yves Chibon (pingou) on the current state of Fedora Infrastructure along with lots of upcoming plans.

If you aren’t able to make it to Kraków in person, do follow along on #fedora-flock ( on freenode ) and in blogs and other project communications.



June 27, 2016

Kevin Fenzi : Zodbot… upgraded

Kneel before the new Zodbot!

We have upgraded our beloved evil super villain IRC bot on freenode from an old version of supybot-gribble to a new shiny version of limnoria ( https://github.com/ProgVal/Limnoria ).  This doesn’t change much in the interface, but it does mean we are using something that is maintained and gets updates and is a good deal more secure. If you notice problems please do let us know with a Fedora Infrastructure ticket.

Also, as one of the maintainers of supybot-gribble in Fedora and EPEL, I will be retiring that in favor of the limnoria package very soon. It should be now available in updates repos for your upgrading pleasure.

Ursa: You are master of all you survey.

General Zod: [bored] So I was yesterday. And the day before.



June 01, 2016

Kevin Fenzi : What didn’t happen today?

Today, astute observers may have noticed that lists.fedoraproject.org and lists.fedorahosted.org didn’t participate in the long running, annoying, insecure and exasperating tradition called “mailman day”. This is where mailman spams you with a reminder for every list you are on with your plaintext “password” in clear email.

Since we have now fully switched over to mailman 3, you can relax and rest easy that we DID NOT spam you with your mailman password today, nor will we from now on. 😉

Our mailman3 instances authenticate users via Fedora Account System OpenID, persona or yahoo, no local throwaway passwords to look for or remember.



May 11, 2016

Kevin Fenzi : Mailman3: no topics support (with alternatives!)

In mailman2 there is a feature called ‘topics’. What this allowed you to do was create some filters on the list and when those filters matched mailman would add a header telling the user that this post was part of a ‘topic’ or not. You can then use the mailman interface to tell it you only wanted posts on a specific list of topics and no others. As far as I know there aren’t too many lists that used them, but there are a few we run in the Fedora Project that do: The prime example being the ‘package-announce’ list.

The package-announce list gets emails from bodhi (the Fedora updates system) about each update when it’s pushed into the stable updates repo. As you can imagine with a fast moving distro like Fedora, this list gets a LOT of posts. This list has topics for each Fedora release (22, 23, 24, etc) and also “Security” and “New package” updates. Using topics users could get a subset of these emails that they wanted.

Sadly, topic support isn’t yet available in mailman3, and it’s unclear right now when it might be implemented. Not wanting to hold up our migration to mailman3 forever, we want to go ahead and move lists like package-announce over as soon as we can. Of course this means those folks using topics are going to be in a bind. However, there are some alternatives:

  • You can switch to using a RSS feed directly from bodhi itself. For example: https://bodhi.fedoraproject.org/rss/updates/?releases=F23 will give you all stable Fedora 23 updates. https://bodhi.fedoraproject.org/rss/updates/?releases=F23&type=newpackage will give you all newpackage type updates, etc. Pretty much any search or page with a rss symbol on it on bodhi is an RSS feed.
  • You can set up notifications in the Fedora notifications service: https://apps.fedoraproject.org/notifications/ to either email or IRC. For packages you are interested in, all updates, security or newpackage updates or any combo.
  • You can of course just stay on the package-announce list and filter things locally. You will be getting emails about some things you don’t care about, but emails are pretty small in this day of streaming video. 🙂

Hopefully those interested can switch over to one of the alternative methods above until topic support once again lands in mailman.



May 04, 2016

Kevin Fenzi : Fedora account system and FreeIPA

Over the years, a number of times, people have asked us about migrating from our own custom Fedora Account System (FAS) to FreeIPA.

We would love to, but right now as things stand it’s just not a great fit for us. In the past we have had trouble articulating what the issues are, but now Patrick has written up a wiki page with all the items we know of:

https://fedoraproject.org/wiki/Infrastructure/fas_freeipa

Hopefully we can keep open a dialog with FreeIPA folks and keep the above wiki page synced up so perhaps someday we can migrate when blockers are no longer in our way. 🙂



April 16, 2016

Kevin Fenzi : ansible 2.0 going stable in f23/f22/epel7/epel6

Just a quick heads up: ansible-2.0.1.0 is heading out to the stable updates repos for Fedora 23, Fedora 22, EPEL7 and EPEL6.

There’s a few outstanding issues folks might hit, but we figured it was finally time to push to stable. If you use includes for handlers you might hit ( https://github.com/ansible/ansible/issues/13485 ). This one will be fixed in 2.1 by allowing you to add a config option to set includes as static or dynamic. Also, you might see some intermittent issues with some hosts saying “unable to resolve remote temp directory” ( https://github.com/ansible/ansible/issues/13876 ). Hopefully this will be fixed in 2.0.2.0, and there are some workarounds in the issue.

If you run into another issue, please do check if it’s reported upstream and if not report it.

We have also pushed out a ansible1.9 compat package for those folks that need to stick with 1.9 for now for whatever reasons. You can simply remove the ansible package and install ansible1.9 and you can use that version. It’s currently at 1.9.4, but there’s a 1.9.6 update pending updates-testing now also.

Happy ansiblizing!



April 03, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-04-03 edition

Well, it’s been a while again, so I thought I would update everyone on recent goings on in Rawhide, Fedora’s rolling development release.

Since my last post we finally moved over to having Rawhide and Branched (what will be Fedora24) done by the new pungi4. There are sadly lots of little issues that have been cropping up and getting fixed by Release Engineering, but it’s been a bit of a bumpy ride. As of this post, the last Rawhide compose that finished was the 2016-03-31 one, all the ones since then have failed. Hopefully this will get sorted out tomorrow. The current cause of problems is adding atomic/ostree to the compose (which we very much need/want), it’s just been more difficult that we first hoped. Once things settle down we will really be in a better place than before, with full composes (tested by openqa!) every night.

There was a bit of delay since the Branching event on signing, but last week we got back on track and (almost) everything in rawhide should be signed by the Fedora-25 key now.

With compose issues and general breakage in the install path it’s been difficult to install rawhide recently. Again, once we get the normal composes back on track this should clear up. As usual, once you have a rawhide install there hasn’t been really much breakage of the day to day use path.

Astute observers would have noticed that the rawhide-source mirrormanager repo wasn’t working after the switch to pungi4. I’ve fixed that up the other day and rawhide-source should be working normally again.



Kevin Fenzi : How long does it take for that Fedora ssh key to sync?

There’s been some confusion about how long it takes from changing your ssh key in the Fedora Account System (FAS) and being able to use it to access things like fedorapeople.org or pkgs.fedoraproject.org or fedorahosted.org. The method has also changed a while back so I thought I would write up this short post on it.

TLDR version: for ssh access to fedorapeople, pkgs or fedorahosted, it should take about 2-17minutes. (usually 2). For access to other machines (if you are in a sysadmin group), it should take 15 to 30min (usually 15). So, you would be safe saying “within 30minutes”.

In the past, we simply had a cron job on every machine that would run every 30min or so and sync everything. This was a massive waste of resources. We had to size our fas servers to handle the load of every machine +/- a few minutes hitting them and for the most part, absolutely nothing changed. Then, along came fedmsg and we now had easily available data to know when something has changed and what it might affect. The new process is run from a fedmsg listener: It waits for ssh keys changes to be mentioned on the fedmsg bus, then it looks to see if they are from someone in a sysadmin group or not. It will then wait for about 30 seconds (if there’s more changes coming in, might as well do them at once). If not, run a ansible playbook that updates just those machines that they might have access to. If they are, use a different playbook that updates all the machines in turn.

The non sysadmin playbook takes about a minute to run. So, the vast majority of these changes will appear in about 2 minutes.

The sysadmin playbook takes much longer (There’s a LOT more machines in this playbook): 15minutes or so. The vast majority of changes here will appear in 15minutes (but it depends on which machine you are waiting for access on of course).

As a side note this applies to ssh keys and mail aliases, but not koji certificates. Those are active immediately as soon as they are issued (and your old certs are revoked).

Since the script will only run one playbook run at a time, the worst case is your changes just miss a sysadmin playbook run and you are yourself a sysadmin. Then you may have to wait 30 minutes. For the case most people will hit, it should just be a minute or two.



March 30, 2016

Kevin Fenzi : taskwarrior tips and notes

A year or so ago some co-workers pointed me at taskwarrior ( https://taskwarrior.org/ ) as a neat way to do todos and task tracking. At the time I tried to dive into it and it just didn’t seem to work out for various reasons ( I didn’t give myself free time to really dig into it, tried to make it do time tracking as well, etc). Last week and over this last long weekend I gave it another go and I think it may stick this time. I thought this might be a good time to share the tips and workflows I came up with in case anyone out there wants to give taskwarrior a try.

First a bit of background: I’m a sysadmin by trade, so my todos and tasks are often tons of short tasks and reminders along with meetings and longer term planning/thinking items. For many years my workflow has been to use my mail client inbox along with a text file for things. If it was in my inbox it means I need to take some action or reply to something. If it’s in the text file it’s something I have already done (to keep a record) or some short term todos of things that don’t have email associated. This setup works pretty well, but it’s bad at followups (where I replied/did something and the other party didn’t), anoying for repeat items (have to use a template for the text file or copy and paste), and bad at searching for particular things.

Enter taskwarrior: The new workflow I’ve settled on pretty much replaces my text file with taskwarrior, so I still have mail inbox, but taskwarrior helping it out. There’s extensive taskwarrior docs on the web and the man page is pretty nice too. It’s very nice that taskwarrior is flexable, it can be used a bunch of different ways with different workflows and methods. You can set things up to use it the way you want to. Above and beyond those things:

  • Setting up meetings confused me, because I don’t want to see them until near the meeting time/day, until I realized I can use wait to make them wait until I want to see them. For example, here’s the EPEL meeting thats every wed at 18UTC: ‘task add ‘EPEL meeting’ project:work recur:weekly due:2016-03-30-12:00 wait:soww’ (soww is start of work week). So, I won’t see this on my task next list until monday when I am looking at my upcoming week.
  • tags are very handy. For example I have been adding +nagios to any task where I am looking at or acting on an alert. That way I can go back after a while and look and see if there’s any trends or things we really want to try and fix because they re-occur. ‘task completed +nagios’ will show all these completed nagios tagged tasks. (likewise, ansible commits, bugzilla, infratrac and internal only tickets)
  • ‘task edit <id|uid>’ is pretty handy to see what task has on a task or tweak it if you added it wrong.
  • If you really do want to wait forever for something, you can use ‘wait:someday’ or ‘wait:later’ and it will set it to 2038. 🙂
  • You can also add ‘until’ on meetings for example for just after they would end or the like, and task will delete that task if it reaches the until time. This is handy for example if you miss a meeting, you didn’t do it, it should just be removed, but you don’t even have to think about it if you add it with an until.
  • Another cool one from Paul Frields: You can use ‘task log …’ to create a task thats already marked done. This is handy if you already did something and didn’t have a chance to make a task before hand. Saves you adding a new task only to mark it done.
  • Out of the box, task doesn’t keep track of how much time you spent on any given task. You can use some hooks to record this, but it doesn’t do it by default. I think the last time I tried it out this might have confused me, but this time thinking about it, I am not sure I care exactly how many minutes I spend on each task.
  • I’ve found it handy to add text as much as I can. When adding the task ‘task add …’ (describe it as simply and completely as you can), when starting the task ‘task N start …’ (usually here a link if it’s to a ticket or bugzilla thing where I know the url when starting)’, and when completing the task ‘task N done …’ (here a commit url or the like with the work done). Or ‘task N annotate …’ anytime you made some progress and it’s an interesting or noteworthy thing to add.

Taskwarrior does have a server you can sync to (taskd). It’s pretty easy to setup and uses TLS certs to authenticate clients. There’s also an android app, which is kind of painful to configure, but seems to work otherwise. I’ve not done all that much with it yet, but it might be handy if you are at a event or the like and want to record a task but don’t have your laptop out. There’s also bugwarrior that lets you pull from trac/bugzilla/pagure/etc into task. I did that the last time I tried things out, but I think it’s not a good workflow for me. When I do work on a ticket or bug I can add that task manually, and leave most/many of them that are not waiting on me to sit in their space.

Reports are quite handy for seeing how much you have done and what you have done in a time period, and pretty easy to setup. Just add them to your .taskrc with what to filter on and what cols to show. I setup one for what I have completed for the current work week and what I completed for last work week (useful in weekly reports to management).

Still getting the hang of things, but I am hopeful that I’ll stick with taskwarrior at least for now.



March 09, 2016

Kevin Fenzi : encrypt all the things: blogs

So, with ssl certificates pretty easily available these days from letsencrypt.org it’s more and more worth looking at making sure you are using https instead of http for everything you can.

There are however some corner cases that are… difficult. One of those is blog aggregators. A while back we moved our planet.fedoraproject.org to fedoraplanet.org. This was to get it out of the fedoraproject.org domain because we send HSTS headers for fedoraproject.org telling browsers they should always contact that domain with https. Blog aggregators are in a tough spot, because they simply pull content from a bunch of different sites and put it in one place and link to it. Unless 100% of the blogs that the site is aggregating are also https, if the site itself uses https most browsers will show you a nasty warning about mixed encrypted/non encrypted content. So, for now, since most of the blogs on fedoraplanet.org are http, we are leaving fedoraplanet.org itself as http.

However, we would love to get to the point where all the blogs we aggregate are https. I don’t think it’s an impossible journey. Here’s what you can do if your blog is listed in fedoraplanet.org:

  • Check to see if your blog site already supports https and has a valid cert. If so, you simply need to login to fedorapeople.org and edit your .planet file to use the https link instead of http. Done.
  • If your blog site doesn’t support https (yet), ask your blog provider about adding it. They should be able to add a letsencrypt.org cert pretty easily.
  • If your blog site doesn’t support https(yet) and you run your own blog, why not add https support?

If we can get a critical mass of blogs using https, we can look at switching the site over too. Help us out. 🙂



February 24, 2016

Kevin Fenzi : A Fedora Distribution download primer

With the fresh news of a compromise in the Linux Mint distribution images, I thought I would take a few minutes to explain how Fedora handles image downloads and what you can do as an end user to make sure you have the correct and official Fedora images.

First, lets take a look at what happens in each step if you open your browser to getfedora.org (our install images download site):

  • You type ‘getfedora.org’ in your browser.
  • First, your operating system asks your dns servers for the IP address of getfedora.org. If your OS is using dnssec, then it will get a cryptographically signed answer. If not, it will get whatever answer your dns servers give it.
  • Next your browser may try and connect to getfedora.org via http. We have getfedora.org setup to redirect all http requests to https, so this would get you a redirect.
  • On the first https connection to getfedora.org, we send a HSTS header. This tells your browser (if supported by it) that it should ALWAYS use https to talk to this site. Even if you enter http://getfedora.org, it would just correct that and connect on https.
  • Once connected you can download distro images by clicking on the download link for the image you like. Once you click on a download (unless you have completely disabled javascript), there’s a screen describing how to verify your download: https://getfedora.org/verify
  • Once you have downloaded your image, you need to do two things to make sure it’s the valid and official image: First, check the gpg signature of the checksum file. Official checksum files in Fedora are always signed. You can get the gpg key for that Fedora release from getfedora.org, most any keyserver, or from the fedora-repos package if you already have a Fedora install. Additionally, if you import this key and then refresh (gpg2 –refresh-keys) you can see the signatories of that key and decide based on all that if you trust it. If thats correct, then you can use sha256sum to check the checksum of the image. YOU SHOULD ALWAYS DO THESE CHECKS. 🙂

So, we have dnssec, hsts and signed checksum files. Would that have helped us any if we suffered a attack similar to the one the Linux Mint folks suffered? In that attack, their download machines were compromised and intruders replaced checksum and download links to their own version. If that happened to Fedora, the only step above that would protect people would have been the gpg signature check (which sadly, many folks never do, and for good reason, it’s hard and anoying and manual).

In Fedora 24 the workstation and server editions will be moving to preferring the usb media creator application instead of preferring direct downloads of images. We will need to make sure it’s as secure as we can make it, but there may well be a manual step of checking the application after you download it. (Unless you already have Fedora and install it as a normal package). In it’s current form it already downloads checksum files from the Fedora master mirror via https and uses that to check downloads, but more can be done.



February 21, 2016

Kevin Fenzi : Rawhide: notes from the trail, 2016-02-21 edition

Once again I have been busy and haven’t gotten a notes from the trail out recently, time to correct that!

Just a bit over two weeks ago we had a mass rebuild event in rawhide. This one was for gcc6 which just landed in rawhide. The actual rebuilding took (much like the last one we did) about 36 hours, but thats just the tip of the iceberg. From the devel-announce post about the mass rebuild completing: “16129 builds have been tagged into f24, there s currently 1155 failed builds that need to be addressed by the package maintainers.” Dealing with those failed builds is always the intensive part of any mass rebuild cycle, since it takes maintainer time to sort out whats going on (and often upstream communication) to fix.

In just a few days we will be branching off F24 from rawhide. Do make sure when this happens on tuesday that you follow the direction you want: Either keep on rawhide or follow Fedora 24 until it’s release in June.

With the mass rebuild and a bunch of other changes people are landing before the branch event there’s been some issues with the rawhide install path again. If you need to install rawhide, do check https://openqa.fedoraproject.org/ for the last image that passed install tests and save yourself time and energy before downloading.

Finally we are close to landing the changes in the compose process around pungi4. This will make every day’s rawhide just like a release compose. It will have all the trees, images, checksums, metadata and such every day. This will also be used for the Fedora 24 cycle composes, so everything should be a good deal more consistent.



January 24, 2016

Kevin Fenzi : A tale of that most hated hardware: printers

Ask any sysadmin what hardware they have to deal with is the worst and I think universally they will all tell you it’s printers. There’s a lot of reasons for this, but I think the biggest ones are:

  • They use / misuse consumable items like ink, paper, etc.
  • They have a ton of moving parts to jam or break or get blocked.
  • They tend to be in places where end users are asked to service them and often put in incorrect paper, jam things or misrefill ink, etc.
  • Recent trends have made printers lowest margin items. If your printer doesn’t work, throw it away and get a new one, it’s cheaper than fixing or refilling your existing one.

A few years ago I got a HP multifunction printer (when it’s predecessor broke and it turned out to be cheaper to just buy a new one). The scanner part of it works great and always has. The printer part on the other hand has been nothing but horror. I don’t tend to print much at all, but a few times a year for some reason or another I need to print a document or directions to something or whatever and every single time this printer has failed me on the first print attemps. Reasons for it’s breakage have included:

  • It’s been so long since I printed the ink “dried up”. Once I was able to get it working by cleaning it with a q-tip, once by making the printer run it’s clean print heads cycle about 20 times, once by buying a new cartridge (almost as much as a new printer).
  • Once it decided to just not turn on at all. Unplugging it completely and waiting 5minutes and replugging it “fixed” it.
  • Once the color ink cartridge was low so it wouldn’t print anything (even just using the black cart that was full!) which required a new color cartridge.

Over the holidays I cleaned out a lot of my computer room, so I could see all the printers I still had that I hadn’t managed to throw away. 2 old HP inkjets, an old epson, and… what is that at the back of the room? why it’s my old HP laserjet 4P. I used that printer extensively during college and after and finally retired it when the tonor ran out and I wanted to move to a new shiny color inkjet printer.

So that got me thinking. The laserjet 4P only has a parallel port, but a quick check found a parallel to usb cable for $8. At the time I retired it, a tonor cartridge for the 4P cost around $100, but looking now, I found one for $12. A quick amazon order and a few days the items arrived. I hooked up the printer to my main server and put in the new tonor cart, added it to cups and printed a test page. A lovely 600dpi test page.

So, hopefully this laserjet 4P will now print reliably for me for the rest of my life and I can avoid the inkjet “buy a new one” everytime I need to print something.



January 18, 2016

Jeffrey Haemer : Another Sale!

The last one was "Best Sellers." Now the video is also "Best Reviewed." Nice!

InformIT.com - Best of 2015 Sale


Kevin Fenzi : A vist to a Fedora datacenter

I just got back from a visit to the largest of the datacenters used by Fedora Infrastructure, and I thought I would share a bit about why we do such visits what what we do on them.

We have a number of sites where Fedora Infrastructure has at least one machine, but we have one main datacenter (sponsored by our main sponsor, Red Hat) where all our build system machines are, our fedorainfracloud openstack cloud, and most of our main servers are. This site does have some on site folks who can handle most of our day to day hands on needs, but sometimes we have things that need a lot of coordination, or extra time to complete, so we try and ‘save up’ those things for one of our on-site visits.

(Side note: If you are interested in donating machines/colocation space for the Fedora Project, please do mail admin@fedoraproject.org. We would love to hear from you!)

On this particular visit last week we got quite a few things done:

  • Package maintainers will be happy to hear we racked a new HP Moonshot chassis. This will hopefully allow us soon to run armv7 builder vm’s on aarch64 virthosts which will vastly decrease build times.
  • We moved 5 machines to a new rack to be used for testing storage solutions. We want to see if we can move to more gluster or open source storage solutions. These machines will be used to test out those solutions as they mature.
  • We added memory to a number of cloud compute nodes, allowing us to run more and larger cloud instances (and have more copr builders).
  • We added a new equallogic storage device to our cloud network. This is a smaller one than we have now, but it will allow us to test things and also give us some more cloud storage room.
  • We fixed some firmware on some power supplies (yes, power supplies have firmware too!). This involved an elaborate dance of moving power supplies around in various machines to get them all upgraded (which would have been very tedious remotely).
  • Moved an old remote kvm and added a new kvm. We don’t tend to use these all that much, but they are very handy for machines that don’t have proper out of band management.
  • Pulled tons and tons of unused cables out of all our racks. On site folks don’t tend to want to remove cables, only add new cables for new devices. There was a bunch of old cables that were only connected to a switch or serial port or power device and were no longer needed.
  • Setup a power7 machine (many thanks to IBM) in our cloud network. As soon as we have it installed and setup we should be able to have ppc64 and ppc64le cloud instances and also do copr builds of those arches locally instead of against a remote server.
  • Updated our records of which servers were on which serial, power and switch ports. This information is very important when making remote requests or power cycling servers so you don’t accidentally get the wrong one.
  • Pulled out a few older servers we were no longer using.
  • Racked a few new servers we will be using for releng/buildsystem stuff.
  • Checked all the servers for any error lights or alarms. Cleared a few we had already fixed, and fixed a few we didn’t know about yet.

All in all a good weeks work that should set us up well for the next few months. I would like to thank Steven Smoogen (for being on-site and getting all the above done with me) and Patrick Uiterwijk (who held down the Fedora Infrastructure fort while we were off at the datacenter and helped get all the above done from the remote side) as well as Matthew Galgoci (Red Hat networking guru extrodinare) and our On-site folks, Jesse Iashie and Pedro Munoz. It’s a pleasure to work with you all.



January 02, 2016

Kevin Fenzi : Rawhide year in review

Looking back over 2015, I see I only managed 7 rawhide posts. Will see if I can do better for 2016. In any case, looking back at these posts and emails I have sent to mailing lists with rawhide topics I thought I would do a bit of a year end roundup.

Overall we have made some great improvements:

  • We now have openqa running on rawhide images every day to let us know when the install path breaks so we can fix it and so people know when doing a rawhide install will work for them.
  • Rawhide is “mostly” signed now. When a large set of rpms lands right before the compose the autosigner sometimes doesn’t sign them all before the compose, but for the most part all the rpms are signed. (For example, all rawhide composes since 2015-12-29 are fully signed, that one had a qt5 build that landed right before compose).
  • Slowly it seems like more people are using rawhide day to day and helping report issues or problems. Rawhide is not for everyone, but it’s good to have a pool of folks using things and testing and reporting bugs so they can get fixed.

Coming up in 2016:

  • releng hopes to soon change the rawhide compose process to use pungi4. This would be the same process then used for real Alpha/Beta/RC/Final composes. It would mean we would have a full compose of all things every night we could test and use.
  • We are working through bugs with sigul batch signing mode (we have already made some great progress fixing several issues), and once those are all solved the autosigner should be able to handle things much quicker.
  • Hopefully we can start doing some more automated testing on composes as well.

Looking forward to another fun year running rawhide and making it better for everyone!



December 23, 2015

Kevin Fenzi : Rawhide: Notes from the trail (2015-12-23)

A short note from the trail for those folks riding along with rawhide this winter:

Openqa ( https://openqa.fedoraproject.org/ is running on daily rawhide composes, letting everyone know when the install path is not working as expected. This has already resulted in several quick fixes and bringing things back on track in the last month. It’s really nice to get the install path to rawhide to be as stable as the day to day trail ride.

There is currently an issue with gtk3 and gtkmm30 using applications crashing on start. The gtk3 folks are working on a fix. If you need something that uses gtk3/gtkmm30 right now, downgrade to gtk3-3.19.4-1.fc24.

Happy Holidays and Merry Christmas from the trail!



December 22, 2015

Kevin Fenzi : On giving

Toward the end of the year, many folks thoughts turn to giving. Either giving a monetary donation, or time and energy. I think there’s a number of reasons for that: In the US at least tax deductible donations are something people want to get in before the end of the year, people know more about how much they might have to donate, as well as having more time away from the pressures of work to think about groups that might be worthy to donate to. Whatever the case, I thought I would share a few thoughts around such donations.

First it’s good to ponder on what kind of gift you might be able to provide. Sure, if you have some money you can allocate thats great, but there’s lots of other ways to give back. Try asking what you can do to help and most any non profit group will have a long list of helpful things to choose from.

Next take a look at what groups you want to support. This of course is highly personal, but do take a look at information about a group before donating to them. For example, many of us in the US will see people outside shops and stores ringing a bell and asking for donations to the Salvation Army. At first glance you might think this a worthy group helping the less fortunate, but looking for further information will show you that the group is highly religious ( for example requiring people they help to listen to them proselytize their beliefs ) and discriminatory to LGBT folks.  Additionally, look at how much of your donation might actually get to the people the group purports to help and not get used for “overhead”. Legit non profit groups are required to publish their finances.

Once you have decided to give to some groups, look and see if there’s any matching fundraisers going on or planned. This is where existing members will match new member donations over a time period to try and help the group and add more members. Your donation can go further to help the group if you wait and donate at such a time. Also, check with your employer, many companies have matching programs where they will match your donation to non profit groups. This is another great way to get more money to the non profit you are wanting to help.

I’ve decided to donate some money to three groups this year:

  • Electronic Frontier Foundation – eff.org. I was able to donate a few weeks ago when they were doing a matching fund drive, and also my wonderful employer (Red Hat) offers matching funds, so I was able to get them 3x what I donated. If you are at all active in on line rights and freedoms you know who the eff is. These folks do great work and everyone should support them.
  • The Software Freedom Conservancy- https://sfconservancy.org They are also currently holding a matching funds fundraiser and my wonderful employer (Red Hat) offers matching funds, so again I was able to get them 3X the funds of just my donation. These folks are doing great work handling legal and paperwork for open source groups as well as being the only ones willing/able to fight for enforcement of the GPL. They lost a number of spineless corporate backers this year with their GPL enforcement battles and its very important that we help them to keep going.
  • Colorado Greyhound Adoption – https://coloradogreyhoundadoption.org. This is the greyhound group that I have worked with in the past and got my wonderful pups from. They do great work, but just one dog with high medical bills can wipe out their entire medical budget. Sadly, I wasn’t paying attention and there was a matching day on the 8th that I missed, but at least my employer (Red Hat) will match my donation to them, so they get 2X as much as my donation.

I hope you all will think about giving back to the groups that help advance causes you believe in.



November 16, 2015

Kevin Fenzi : Rawhide: notes from the trail (2015-11-16)

There has been a lot of things going on in rawhide of late, so past due for another note from the trail. :)

The python 3.5 rebuild has landed. The vast majority of it was done in a side tag by Peter Robinson, Kalev Lember and Robert Kuska (and others!), then merged back into rawhide on friday (the 13th). There are still a number of packages that need fixes to build against python 3.5, expect most of them to get fixed up this week. If you have some of those installed, dnf may well hold back python3 and all the newly rebuilt packages until the ones you have installed are all fixed or you remove them.

There was a dracut bug over the weekend (landed friday and fixed this morning) that might result in newly updated kernels not having an initramfs. Make sure you get the version from this morning before installing/upgrading kernels. (That was: bug 1281917)

On the kernel front, the 4.4.0 merge cycle has been running the last few weeks, resulting in tons of changes and some fun bugs. If your tun/tap devices don’t work, that is bug 1281674 which already has a fix, hopefully upstreamed soon. I’ve also seen plymouth unable to prompt for decryption on boot (not yet filed), and there was breakage in video for vm’s (already fixed upstream hopefully). Likely there will be a 4.4.0 rc1 kernel for rawhide today with many of those fixes.

There were no rawhide livecd’s or arm appliance images for a few days late last week. This was due to the builders all being moved to Fedora 23 and missing a few packages they needed for koji to be able to create those images. That should be all solved now, however some images still may not compose due to lingering python3 deps.

On the graphical front, Gnome logins switched from being Xorg by default and having a special “Gnome wayland” session for wayland to the reverse. Wayland is now the default with the normal “Gnome” session. There is a “Gnome Xorg” session now if you need to fall back to Xorg for any reason. How well this ends up working out will decide if Fedora 24 Workstation will default to wayland or not. In my testing, Gnome/Wayland is prefectly usable day to day now, many of the issues in the past have been fixed up. There’s still a lot of polish type things to get right I think, for example I use sloppy focus and it sometimes doesn’t act right with wayland.

Whew. As usual lots going on in rawhide and always a fun bit of trail riding. Until next time, keep them doggies movin!



November 06, 2015

Kevin Fenzi : Lenovo Yoga 900 and Fedora Review

Just about 2 years ago now I picked up a Lenovo Yoga 2 Pro, which I have been using happily since then. My full review was: Lenovo yoga 2 pro and Fedora review and then a followup a year later at yoga 2 pro: a year later. It’s been a great little laptop, but I decided it might be time to replace it soon. Mostly because the warentee on the yoga 2 pro was done, the cpu was showing a bit of age and I was hoping there would be some advancements in laptops over the last 2 years.

A few weeks ago, Lenovo came out with the Yoga 900, which was the successor to last years Yoga 3 pro and it in turn my Yoga 2 pro. The stats and early reviews looked pretty nice, so I ordered one.

I was hoping for a smooth Fedora experience, but sadly I ran into two issues right away after booting from a Fedora Live USB:

  • The wireless didn’t work. It didn’t even see the interface. Of course I ran into this same problem 2 years ago with the yoga 2 pro: it was the ideapad_notebook module not knowing about this laptop and thinking it had hardware rfkill set on all the wireless modules. A quick bit of hacking on that module and it no longer showed the interfaces as hardware blocked. Unfortunately, it still didn’t work. I looked then to confirm that the intel 8260 wireless was supported by Linux, and indeed it is. Looking closer, I saw that the pci id of the card I had was not in fact listed in the driver. They had “0x24F4, 0x1130” and mine was “0x24F3, 0x1130”. Fixing that quickly in the module got the iwlwifi module working with the interface and all was well. This bug tracks these issues (both of which should be headed upstream): bug 1275490
  • The touchpad and touchscreen didn’t work. This seemed to be a probing issue with the i2c-designware platform. A few posts to the linux i2c-devel list and some exchanges with an intel engineer and I had a one line patch that gets it working. Hopefully that will be upstreamable soon.

With those issues out of the way, I went ahead and did a rawhide install on it, copied my /home over to it and then updated my ansible playbooks that setup all the things I want on my laptop. It was a pretty painless process overall and then I was up and running on the new machine. :)

Good things:

  • All the nice things about the yoga 2 pro came over: It’s actually a bit smaller and lighter, the screen resolution is still 3200×1800, it fits in the laptop sleve I got for the yoga 2 pro.
  • It’s really fast. The latest gen i7-6500U’s are really zippy.
  • I’ve not traveled too much with it yet, but in the short times I have the battery life has been great. Estimating around 12 hours or so, almost 2x the battery life on the yoga 2 pro.
  • The hinge is a new “watchband” style. I never had any issues with the yoga 2 pro’s hinge, but this one looks a good deal sturdier.
  • 16GB of ram (over the yoga 2 pro’s 8GB) is nice. Might run some vm’s now. :)
  • The laptop backlight now has 3 levels: bright, low, off. I could see using the low setting in some cases.
  • It has a USB-C port instead of a mini-hdmi on the yoga 2 pro. I have no USB-C cables, but this seems like it will be a win allowing me to connect (with the right cables) to a DP, hdml, usb, whatever via the USB-C port. I am not clear on if I could charge from this port as well, but I’ll see when/if I get a cable. :)
  • The power port is also a USB 2 port. Could be handy in some times when you don’t have power and need to attach another usb device.
  • The firmware actually has a way to enter secure boot setup mode and add your own keys. The yoga 2 pro didn’t have that.

Bad things:

  • Just like the yoga 2 pro, firmware updates seem to require windows. ;( Lenovo provides handy iso updaters for many laptops, but not this one. Likely because they outsourced the firmware to some 3rd party. ;(

Just different:

  • The keyboard feel is different. I am not sure if I like it better, don’t like it, or just need to get more used to it.
  • 512GB ssd instead of a 128GB in the yoga 2 pro. I wasn’t even using the 128, so it doesn’t really seem that much different. :)

Overall I think the upgrade is well worth it, but it’s all mostly incremental improvements, nothing jaw droppingly awesome. The active warentee, cpu speed, memory and battery life all make it worthwhile IMHO.



October 12, 2015

Kevin Fenzi : The 5 states of the modern sysadmin

I think there’s (at least) 5 states you might find yourself in as a sysadmin in these days:

  1. Day to day things that aren’t (yet) automated.
  2. Automating and designing for the future.
  3. Fires and outages
  4. Interruptions
  5. Time to dream

In general you will want to move things from the first bucket to the second as much as you can. Automate all the things that happen often, or (re)design things so you no longer have to do such things day to day. So if you find yourself restarting a webserver every  day, figure out why it needs that and fix that. Or if you have to spend lots of time processing new lists or resetting passwords, make those so they are self service. This might also include reading emails and lists and feeds, if you spend a lot of time here that you don’t get any benefit from, perhaps it’s time to drop some of those?

Of course outages and critical problems are another chunk of time. When things stop working at 2pm or 2am, you start working on bringing things back to normal. Bucket two helps here as well, if you find out things are often causing outages or problems, redesigning them or fixing the underlying problem saves stressfull outage time. It’s important after outages and critical problems to try and have some time from bucket 5 as well. Often just thinking about how the problem happened and spending some time looking at it can give you a idea for a design to use in bucket 2 to fix it once and for all.

Interruptions are another chunk of time. While you might think it “only takes a minute” to ping your sysadmin about something, it  takes a while to get back to the second and final buckets above where (hopefully) you are spending your time. This is why filing tickets or using some other tracker for your non critical things helps a great deal. Then, they can be processed in the first bucket when you have time to do that and also have a nice record of what you need to work on in the second one.

Finally another place that is great for you to have time is a category I call “time to dream”. This is usually unstructured time where you as a sysadmin can ponder on how things are setup, look at logs or machines you don’t usually deal with, read about tools that you might be able to use to make things better, or just a totally new workflow for something.

If you find yourself in a sysadmin job where you spend all or most of your time in buckets 1, 3 and 4 you have a definite problem.

So, all you sysadmins out there: are there other states you find yourself in often that I haven’t mentioned yet?



October 03, 2015

Kevin Fenzi : Fun with wifi

The last few weeks I ran into some really nasty problems with wifi on my laptop: It started cutting out and dropping me offline for several minutes at a time and was slow and pretty unusable.

Of course the first thing I thought of was a bug in the rawhide Linux kernels (I run rawhide full time on my primary laptop), but going back to previous kernels or even Fedora 23 or 22 stable kernels showed the same behavior. Looking at the disconnects and re-associations I was pointed to it possibly being just very congested. So, I moved the channel my AP was on (man are there a lot of access points in this area now) and played around with the settings (My access point is a WNDR3700v2 running OpenWrt). That seemed  to help a bit, but not all that much. I also noticed that all the other devices in the house seemed fine (sadly, most of them don’t have a good way to show signal strength or status, but they didn’t seem to be reconnecting).

Along this journey I got to play with the ‘iw’ and ‘nmcli’ commands a lot. I would recommend you all take a look at iw, it can sure tell you a ton of information about your wireless connection and how it’s working or not working. nmcli has also come a long way. ‘nmcli d wifi’ is something I am now using all the time rather than looking a the pull down menu of networks.

Finally, I swapped out the wireless card in my laptop. I had gotten a replacement one years ago when I got my laptop when I read reports of poor wireless performance, but never swapped it in when the existing card seemed to work. The new card worked TONS better. I can only conclude that the old one was going bad in some way. :(

Of course being a computer geek I took this as a great time to look into setting up WDS (Wireless Distribution System, basically a way for one AP to become a client of another and bridge all the connections to/from it, extending your range) in my house. My main access point is upstairs on one end of the house, and I am downstairs a fair bit of the time. I picked up some little NEXX WT3020’s for travel (also running OpenWRT), so it seemed it would be easy to setup one of them downstairs as a WDS client connected to the main AP. OpenWRT makes it pretty easy to set this up these days, but there were a few gotchas I ran into: Make sure you set the frequency width to 40 on both the main and client WDS client, and make sure you set the static ip for the client ap right (or else you won’t be able to easily get to it to configure it). Otherwise the instructions at http://wiki.openwrt.org/doc/howto/clientmode worked fine. I’m not sure how much better it really makes things, but it’s kind of nice to have if I ever need it.



September 25, 2015

Kevin Fenzi : A fully ansible powered Fedora Infrastructure

Just a week or so shy of 3 years ago, Fedora Infrastructure embarked on a journey to migrate to using ansible to manage all our hosts. It’s been a long road and one we could have done faster, but we wanted to do things in a transparent and measured way, not rushing and making quick decisions and changes that we would have to clean up later. Today that journey is completed.

So why ansible? What made us start this journey? We had a CM setup and it was working, but after weighing the advantages and disadvantages we definitely felt moving to ansible would be worth it. Just a few of the reasons:

  • We are very much a python shop, ruby syntax isn’t hard, but it’s just another thing to remember when making changes.
  • Not having to have agents on all our nodes is a big win memory and resource wise.
  • Not having to have yet another ssl cert setup for our CM is a great win.
  • We could drop func (that we were using for some ad-hoc things in favor of just using ansible for those things)
  • Not having to worry about version skew between our management host and clients was a big win.
  • Easier to explain and show how things work to new folks was a big win.
  • Not having to worry about scaling central host to number of managed nodes was a big win.
  • … and I’m sure more things I am forgetting now…

Of course just because we have finished this migration doesn’t mean we don’t have lots and lots of other things ahead of us. In 3 years ansible has changed some and we need to go back and clean up some of those things that currently work, but aren’t ideal anymore. Ansible 2.0 should be out before too long and we need to look at moving to that and helping out with it’s testing whereever we can.

I’d like to thank the Fedora Infrastructure team and all those who contributed to our migration efforts as well as the excellent ansible developers and community. We could not have done it without any of you!



September 23, 2015

Kevin Fenzi : Rawhide: Notes from the trail: dnf-1.1.2-2.fc24 will plum rustle your cattle

Just a heads up for you intrepid rawhide users:

The latest (as of this writing) dnf update, version 1.1.2-2.fc24 seems to break doing a lot of things you might want dnf to do (like update packages or list them or anything).

Bug https://bugzilla.redhat.com/show_bug.cgi?id=1265336 is open to track the issue.

In the mean time you may want to manually download 1.1.1-2.fc24 from koji.fedoraproject.org and downgrade and add a ‘exclude=python3-dnf-1.1.2-2.fc24’ to your /etc/dnf/dnf.conf

Keep em movin



September 20, 2015

Kevin Fenzi : Bodhi2 and it’s much faster updates flow

Now that we have had bodhi2 in production use for a while, I thought I would talk some about it’s backend and how it manages updates pushes.

First, for a bit of background, lets mention how bodhi1 did things:

  • release engineers would gather a list of pending updates and look over them.
  • release engineers would sign those updates
  • release engineers would tell bodhi1 to push them. Typically in freezes (alpha, beta, final) the branched updates-testing ones were pushed, then all the stable releases (both testing and stable). Outside of freezes, all branches were pushed at the same time (both testing and stable).
  • bodhi1 would start going through the list one by one and mashing the packages into repos.
  • At the end when all branches were done it would go through them one by one and gather updateinfo and other data it needed to inject.
  • It would then create rpm-ostree/atomic updates for each branch that has one one by one.
  • Then it would wait for all of them to be synced to the master mirrors.
  • Then it would go through them one by one and update bugs and note things in the bodhi updates.

As you might imagine this could take a very long time. It wasn’t unusual to see this be 12-18 hours or more for the full set.

Bodhi2 instead of being single threaded is very much mult-threaded:

  • release engineers gather a list of pending updates and look them over.
  • release engineers sign updates
  • release engineers tell bodhi2 to push them.
  • bodhi2 looks at all the pending updates: Are any of them security? It will do all those branches that have security updates first.
  • Threads are fired off for each branch that is to be done (first all security holding ones, then all the rest).
  • In each of those threads, subthreads are fired off to: mash updates (using createrepo_c instead of createrepo that bodhi1 used), gather updateinfo, create digests, ie, anything that can be done and wait to be collected together at the end is done.
  • When one branch is done, atomic trees are updated, it’s synced out and all work is completed on it (updating bugs, etc).

So, where bodhi1 took 12-18 (or sometimes more!) hours to push all Fedora branches, bodhi2 is taking around 6 hours to push them all. The slowest of the branches is of course fedora-21 stable updates (It takes about 4.5 hours to complete), which makes sense as it’s been out for a very long time now and has a very large pile of updates in it.

Since things are multi threaded in bodhi2, failures and issues are better as well. If something broke somewhere in bodhi1 pushes, everything ground to a halt until it was fixed and resumed. In bodhi2 if there’s an issue, only that particular branch will fail and can be resumed anytime. Everything else completes.

Moving forward there’s even more improvements we hope to roll out: Right now bodhi handles the mashing of updates on it’s own instance, and can be somewhat limited by it’s I/O bandwith, so we would like to move that out into koji and have koji do the mashes and create the updates trees. That way it can be farmed out to a bunch of seperate builders. There’s some createrepo_c changes hopefully we can get working to add multithreading to delta rpm creation, which may speed things up too. Finally we may tweak the threading to do all the security and non security branches at once (memory and I/O permitting).



September 18, 2015

Kevin Fenzi : rawhide: notes from the trail (2015-09-18)

Long overdue again for another “This week in rawhide”, I realized that I haven’t been able to do them weekly for quite some time, so I’m changing the name to ‘notes from the trail’ to play on the rawhide/cattle drive metaphor.

There was a good bit of rawhide discussion at flock this year and some of the things discussed there have already come to pass: Namely, Adam Williamson has setup openQA to run tests on branched (what will be Fedora 23) and rawhide. So, we now have some better idea when things are not working or didn’t generate images to test, etc. It’s sending emails to both fedora devel list and fedora test list.

I posted about the flock discussions in https://lists.fedoraproject.org/pipermail/devel/2015-August/213527.html but sadly it devolved into a discussion if we should rename rawhide or not and if so, to what. We still have not yet landed pungi4 for doing the rawhide composes, but hopefully that will happen before too long, and I am not sure the progress of adding taskotron checks to rawhide builds is.

Current rawhide should be back to ‘mostly signed’ packages. I have been signing things as time permits.

Finally, I would like to touch on something that has been mentioned in the various renaming rawhide to something else threads: perception. I cannot count the number of times I have seen someone appear in a end user support channel (#fedora, fedora-forum, users mailing list, etc) and say “Hi, I want to run rawhide, what do you think?” to which the answer is variants of “Oh no, rawhide breaks all the time” “It will blow up on you” or the like. On one side I can understand support folks saying that. They don’t want someone hitting bugs and getting upset and they don’t want someone who isn’t experienced in troubleshooting to get in over their head. On the other hand, its just not true anymore. I cannot recall the last serious breakage I ran into. The last bug I filed was about selinux preventing my root /var/spool/cron/root job from running (which was easy to work around and notice). When I last redid the rawhide wiki page I made sure to add a “Audience” to it: https://fedoraproject.org/wiki/Releases/Rawhide#Audience So, next time someone asks you if they should run rawhide, point them there and ask them if that describes them.



September 03, 2015

Jeffrey Haemer : Sale on My Git Under the Hood LiveLessons!

Well, I haven't blogged much lately, have I? :-)

Heather Fox, from Pearson, tells me that my Git Under the Hood, LiveLessons, is a best-seller, and ison sale this weekend.

InformIT Labor Day Sale - Save up to 70%


August 16, 2015

Kevin Fenzi : Flock 2015 day 4 – saturday

Saturday came a bit less early than the previous days as we had no keynote, just workshops, so everyone seemed a bit more refreshed.

Again infrastructure folks met up and worked on various projects. Dennis got the builders all upgraded so we could test some dnf/mock patches, but that caused things to break in another area. Happily Ralph was able to iterate on a patch and we applied it as a hotfix to get all the builders up and building again.

There was some interesting discussion about using PCP (performance co-pilot). Very interested to see what kind of information about our infrastructure this could bring.

John containerized out pastebin service as a proof of concept. I am looking forward to looking it all over and seeing how it’s done and how we can better use containers and such for our applications.

After workshops everyone split out to various places for dinner. I went with a group to the Genosee brew house. Very nice view and great beer!



Kevin Fenzi : Flock 2015 day 3 – friday

Friday started with a keynote by John Schull at eNable. They basically are a community around designing, building and distributing upper limb prognostics. With 3d printers getting so common and good, it’s enabled people to design and print their own. There’s tons of places people could contribute to the project, from designing things, printing them and shipping them, to helping gather statistics about the community and help them move on to other exciting areas. Really a very cool project that I had never heard of before. Kudos to the flock organizers for bringing John in.

After that we moved on to workshops (both today and tomorrow). The infrastructure team gathered up and started using a gobby document to do a sort of bar camp thing so we could discuss the things we wanted. Then we went through them. We didn’t mange to get them all, but we got a lot of them. Look for mailman3, bodhi2 and other exciting things very very soon.

After a lunch break we came and everyone started hacking on various things. I wrote down many of the things I discussed, but will need to gather them and go through them to make them coherent. :)

The friday evening event was at the Eastman house. Very cool place, lots of great pictures and old cameras, as well as a chocolate fountain. :)



August 14, 2015

Kevin Fenzi : Flock 2015 day 2 – thursday

Thursday came all too early. After a quick breakfast, it was time for the keynote:

Major Hayden talked about impostor syndrome and related difficulties (like the Dunning-Kruger effect). He suggested a method used by airplane pilots might help some people to move forward when they are affected by these sorts of things: The OODA loop: observe, orient, decide, and act. Interesting topic and many people seemed to know it in the room. 

Next up was my talk on open source etiquette. It wasn’t completely horrible, but while giving it I saw a bunch of places I should change the pacing and what I wanted to mention and what pictures I might use. There wasn’t a big crowd, but there were at least a few people and one person did thank me after the talk so it can’t be completely bad. 😉

I went back to the hallway track until lunch (another nice one at the hotel).

Then on to Paul Frields talk about remote work. I’ve worked remotely for the last 15 years, so I think I do ok, but there were some nice ideas Paul had about how to prioritize things and simplify workflow. I may well try a few of the things he mentioned in the coming weeks.

Next was a talk by Mike Mcgrath on Atomic and Openshift and docker and how all the things fit together. It was pretty high level, but it did clarify for me some things about what things were used in others and how they all fit.

Then off to a talk from Ralph Bean about technical debt and microservices. Some good background for other people, and some ideas we should definitely try out in coming months. In particular I liked the idea of doing a week where we just try and do cleanup and fighting down tech debt every few months.

The last session of the day was the “meet your fesco” session. Sadly, we didn’t get all FESCo there, but we did have some good discussions and questions. There was talk about how to better handle secondary arches, how to better deal with security updates, how we should handle the old “Incomplete” changes that never got actually submitted anywhere, and what the role of FESCo vs the council was. Great stuff.

The evening event was then at the Strong museum of fun. It was awesome. There were a bunch of old video and arcade games. I had fun playing centipede and air hocky and hercules (The worlds largest pinball). The winner tho was a game called “Whirly Bird” which was this mechanical helicoper game from 1969, it was quite cool. The food, drink and company was great as well.

Tomorrow will start out at 9am with another keynote and then we will go into workshops. Looking forward to actually getting things done tomorrow instead of just talking about them. :)



Kevin Fenzi : Flock 2015 day 1 – wed

Flock is finally here! :)

The conference is in the hotel, so there was pretty much no travel time or chance to get lost. Registration was quick and easy and well run.

The keynote was from our Fedora Project Leader, Matt Miller. He had a number of great slides of statistics about how Fedora is doing in various ways, showing things are growing and looking pretty good.

I caught a few talks, and they were all pretty good:

10 ideas that are killing open source had some great discussion and a nice way of looking at the various reasons people don’t open source their code, and how to convince them to change their minds.

After that I went to the “hallway track” and talked with a bunch of folks about a bunch of things. It’s great talking to people in real life in high bandwith to try and explain or learn about something. It’s also of course great to talk to old and new friends.

Lunch was pretty nice there at the hotel as well, then back to catch part of Patrick’s talk about the state of Fedora Infrastructure. There were a number of questions which Patrick handled just fine. It was great to see that he knows the infrastructure setup so well and can field questions nicely.

The final talk of the day I went to was “What does Red Hat want?” Which was a great talk by Denise Dumas. She went over how Red Hat culture is setup and noted that there’s even a clause for employees that if they participate in an open source project, they can put that projects needs over the company if they need to. There were a lot of questions on how to better communication from Red Hat to Fedora and vice versa. It would be great to hear more clearly from Red Hat what they really do want so the Fedora community could decide what and how to help them.

A bunch of us then went off for a team dinner at the Old Toad. It’s a nice british pub type place. The beer, food and company was all excellent.

Back at the hotel there was a games night. I didn’t manage to play any games, but I did talk to a number of folks to round out the evening.

 



August 13, 2015

Kevin Fenzi : Fedora 23 Alpha release day – tuesday – 2015-08-11

Morning came far too soon, but on the plus side there was a nice blanket of fog over the city.

The Fedora 23 Alpha release went out nice and smoothly. I merged websites branch into master, fixed a prerelease redirect we had in place and acked the announcement that was written by marketing and FPL and sent by our Release Engineering leader. There were a few minor link issues, but quickly fixed. BW climbed over the day, it seemed a pretty popular release.

We went over to java’s, a local coffee joint. I had the Aztek coffee. It was awesome! Then we wandered over to the Old Toad for some lunch. Also very nice. After that we decided to go back to java’s and hack the afternoon away on various projects.

Dinner was at Dinosaur BBQ just down the road. It was pretty excellent as well.

 



August 05, 2015

Kevin Fenzi : Flock 2015 Rochester coming up next week!

The summer is just flying by and next week is the 2015, North America version of Flock, this time in Rochester, NY.

Flock is the annual Fedora conference, which alternates between North American and Europe each year. It’s a great time to listen in talks what other Fedora folks are working on and have less format hallway talks with people you work with all the time but don’t live near.

Some information for this years flock:

Will be Aug 12th to the 15th in Rochester, NY.

If you can’t make it in person, there will be likely live streaming and transcripting going on.

Please Join #fedora-flock on irc.freenode.net for general conversation with other flock goers and as a place to find more information for remote attendees.

Registration and conference rate hotel room are long closed, but if you still find yourself able to make it to there in person you are welcome to attend. (You may not be able to get a t-shirt or lunches or evening events, but the conference is open to everyone!)

Evening events have been announced: http://tinyurl.com/flock2015evening

The schedule of talks: http://flock2015.sched.org/

Please thank our wonderful sponsors: http://www.flocktofedora.org/sponsors/

I’m looking forward to it. :)