GPG: Strong Keys, Rotation, and Keybase

2016-11-23 14:09 EST

I have sort of a love/hate relationship with strong crypto. I love it and hate that my sphere doesn't use it much. That lack of use sometimes leads me to complacency and my GPG key is no exception. I've used the same key since 2008 with no rotation. It's a 1024 DSA primary key with a single RSA subkey. The key's bounced around from machine to machine and who knows if it stayed secure the whole time. Also, DSA has fallen out of favor as has keylengths that short. Time to generate a new key.

Generating a new key is as straightforward as 'gpg --gen-key'. I've got a few additional goals, though. I also use which adds some complications since it's badly documented and uses gpg signatures for its magic.


  • New key with strong modern settings
  • Distinct subkeys for signing, encryption, and authentication
  • A sane transition

Why distinct subkeys?


First, it's important to remember that a GPG key isn't a single entity. It's more like a keyring. A subkey has its own identity (fingerprint) and function but it is associated with (and signed by) the primary key. A GPG key can have infinite subkeys, all tied together by the primary key.

A primary key can have four functions:

  • Certify - The ability to claim or trust other keys. This is how subkeys are tied together

  • Signing - Indicate that your key either created or approves a specific piece of content, like an email

  • Encryption - Transforming content into a secure form that can only decrypted by another specific key

  • Authentication - Allows the key to be used in place of a password in some cases. For instance, a GPG key can be used in place of an SSH key if the right magic is applied. (At some point, I'll write about this because it's hella difficult to get going on gpg versions below 2.1 (aka pretty much every OS except for FreeBSD.)

My 2008 key allowed the primary key to execute all four functions. That's a 'minimum viable' setup and is the easiest for first-timers. Unfortunately, this allows an attacker to gain full control of your GPG key from any machine that contains your keyring. In the setup I'm about to describe, the biggest risk is that an attacker could impersonate you.

Because Reasons

I decided to separate these functions into distinct subkeys. The primary key only has the ability to certify while subkeys contain the ability to sign, encrypt, and authenticate. This enables a few shiny tricks.

First, you can create what are sometimes called "laptop keys". Laptop keys contain only the subkeys for signing, encryption, and authentication. They cannot modify the main keyring in any way. This allows a secondary computer to perform the usual day-to-day actions without exposing the primary key. Laptop keys also can't modify their own expiration date. If a laptop is lost, the subkeys can be revoked or expired, telling the world not to trust or use those keys in the future.

Second, you can rotate subkeys as often as you want without needing to generate a whole new keyring. This is particularly handy for my use case at the moment where I'd prefer to avoid this whole-key rotation dance for as long as possible.

Third, you can move the primary key (the one with Certify ability) onto a USB stick or Yubico key and hide it away in a safe, keeping it hidden from an online attack.

Key Generation

Rather than copy and paste half of someone else's blog post, I recommend following the instructions in Mike English's article "Generating More Secure GPG Keys: A Step-by-Step Guide". Specifically, use the sections entitled "Generating The Primary Key" and "Adding Subkeys".

Since we are obsoleting an existing key, I strongly recommend adding a comment to the new primary key to provide a textual link. For instance, the comment on my new primary key is "(Obsoletes C134314B)". We'll deal with GPG trust in a moment but a comment allows a human to see the link between keys and know what's going on.

Web Of Trust

Since we've generated an entirely new key, we need to form a trust link between the old and new keys by cross-signing. (I'm assuming you are working on a host that has the secret parts of both your old and new keys.)

First, we sign the new key with the old key:

gpg --default-key ${old_key} --sign-key ${new_key}

Then we sign the old key with the new key:

gpg --default-key ${new_key} --sign-key ${old_key}


To upload our keys, we need a public key server. I use Most of the key servers synchronize so this is largely a matter of preference.

Upload the keys like so:

gpg --keyserver hkp:// --send-keys $oldkeyid $newkeyid

Expiring and/or Revoking The Old Key

We need to tell the world to not use the old key anymore. There are two ways to do this.

Expiration is sort of the soft way. It tells the world not to use the key anymore but that you still control the secret parts. Expiration can be changed at any time with the secret parts so you could theoretically bring the old key back to life later.

The hard line is revocation. This tells the world to not trust the key and to scream bloody murder if they see content encrypted or signed with this key. Most clients throw up huge warnings or simply refuse to acknowledge the existence of revoked keys.

The "right" way is to revoke the old key.

First, you need to generate the revocation certificate:

gpg -a --gen-revoke $oldkey > $oldkey.supersede

sec  1024D/C134314B 2008-10-01 Matt Cashner (sungo) <>

Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
  0 = No reason specified
  1 = Key has been compromised
  2 = Key is superseded
  3 = Key is no longer used
  Q = Cancel
(Probably you want to select 1 here)
Your decision? 2
Enter an optional description; end it with an empty line:
> Superseded by 0xAC70D9A5
Reason for revocation: Key is superseded
Superseded by 0xAC70D9A5
Is this okay? (y/N) y

Notice how I added a description that points to the new key id.

(You should also generate a revocation key using the reason "Key has been compromised". Store that in a safe offline location for use later if you ever lose control of the secret parts of your key.)

Now, import the revocation certificate and send it to the public key server:

gpg --import $oldkey.supersede
gpg --send-keys $oldkey

Laptop Keys

Once everything's squared away, we can create our "laptop keys". The laptop keyring contains the three subkeys we created via Mike's instructions but not the secret parts of the primary key. As mentioned previously, this allows the laptop to sign, encrypt, and authenticate but does not allow the laptop to certify or modify the basic properties of the keys.

First, we export the public side of the GPG keyring. Then we export the subkeys. Since the subkeys have their own identity, they have their own secrets.

gpg -a --export $mykeyid > ${mykeyid}.pub
gpg -a --export-secret-subkeys $mykeyid > ${mykeyid}_subkeys.gpg

On the laptop, first we import the secret side of the subkeys. Then we import the public key to fill in all the details.

gpg --import ${mykeyid}_subkeys.gpg
gpg --import ${mykeyid}.pub

Make sure to transport the 'mykeyid_subkeys.gpg' file securely since those subkeys can encrypt and decrypt content.

Keybase aims to make life with GPG easier. It does a really good job but it encourages what I consider bad behavior in key management. By default, it creates and stores a private key on their server. We can never truly verify the condition of their server so a Keybase-hosted private key should always be considered untrustworthy.

I keep my key locally and use the keybase CLI client. This leads us to some undocumented waters as the Keybase folks are focused on the usability of the web site. (I'm assuming from here on out that you've got the keybase CLI client installed and configured.)

If you already have a GPG key attached to your Keybase profile, it's possible to add additional keys. You can import your new key like so:

keybase pgp select --multi $newkeyid

If you go to your Keybase profile on the web, you can see two GPG keys listed.

Since we've revoked the old key, we need to remove it from our Keybase profile. On the website, hit "edit" next to the signature for the old key and select "Revoke this public key". A popup will appear, detailing the CLI commands to run to drop the old key. 'keybase pgp select' uses GPG key IDs but 'keybase pgp drop' uses Keybase internal IDs. The popup will tell you which ID to use.

keybase pgp drop "$keybase_id"

Adding and revoking keys in this fashion does not store your private GPG key on Keybase's servers. The inner workings of Keybase are complicated at best, involving GPG signatures, Merkle roots, and the bitcoin blockchain. The important fact here is that this process tells Keybase, and its users, that this GPG key is yours and can be used to sign and encrypt. A Keybase user can track/follow you and retrieve your GPG public key using the 'keybase pgp pull $userid' command.

Keybase added a UI that shows a bit about how everything ties together. Looking at my profile, it's possible to see how the signatures and devices all tie together via a directed graph or the signature chain.


Whew. That's a lot of words for a handful of CLI commands. I'm hoping, though, that I conveyed the "why" of my approach without getting too much into the theory. If this is still confusing as hell, hit the "Contact Info" link in the header and let me know how I can improve it. There's so much about GPG on the web and so little of it makes sense to people who don't care about the math or details of public key cryptography.

Credit Where Credit Is Due

I used a series of articles by Mike English to get through this process mostly unscathed.

Today's Maintenance

2016-11-09 19:16 EST

As mentioned on Twitter, had a big maintenance today. It was probably the most noisy maintenance we've had in a long time.

So, what did you get?

  • Two old crusty servers were retired and three brand spanking new ones were brought into service. That brings our publicly-accessible node count to 5.

  • New SSL cert. It's still self-signed but all nodes now have a non-expired cert for on port 7062.

  • IPv6. We have two nodes available via IPv6 via and We hope to add more in the near future.

  • Less sungo-SPOFs. The backend details are boring and tedious but I spent a bunch of time enabling more people to help build, deploy, and maintain our servers. The network is in much better shape now, should I disappear.


I want to take a moment to address network security. Right now, every node offers SSL on port 7062. That's fantastic and makes me happy. There is bad news, however, and I want to make sure I'm crystal clear on this.

Despite client-to-server encryption, the links between the servers are NOT encrypted. By default, do NOT treat this network as a secure safe place from network sniffing.

I was really hoping to be able to announce the opposite today but it didn't work out. There's a lot of work ahead of us to build a secure network for you but that work is underway now that this maintenance is behind us.

The End

With today's work, I'm happier with the state of our infrastructure than I have been in a while. The lack of server link encryption makes me sad. But we know now what needs to be done. That's progress.

As always, if you have questions or concerns or if you have an irc issues following today's work, please hit me up on irc or twitter.

Bad Ideas: Cron Replacement on OSX 10.11

2015-10-02 23:24 EDT

OSX 10.11, aka El Capitan, was released a few days ago and introduced System Integrity Protection (SIP). SIP is a security mechanism which, at its most basic, prevents the root user from performing all sorts of actions. It's conceptually very similar to SELinux but with none of the configuration ability. For the purposes of this post, the key "issue" is that SIP prevents root from writing to protected directories, namely /usr (amongst others).

In setting up a new machine tonight, I ran into an issue. I was unable to create a new crontab. After some digging, I realized that all of cron's magic is in /usr/lib on OSX. SIP is guarding /usr now so there's no way to get a new crontab going. That's … problematic.

Years ago, Apple transitioned all of its tooling away from cron into launchd. Cron-like functions are configured via plist files in ~/Library/LaunchAgents or /Library/LaunchAgents (for apps to be executed as root). The syntax is clunky as all hell but the system works. The "right way" would be to rewrite my desired crontab entries into individual launchd plists. That sounds really boring. Besides, why do things the "right" way when I could do something terrible instead?

I decided to build a replacement for cron via launchd and some shell. It ended up working like /etc/cron.hourly and /etc/cron.daily that anacron usually provides.

Here's the launchd file for the 5 minute execution. This lives in ~/Library/LaunchAgents/com.sungo.fivemincron.plist. Load it up by running launchctl load com.sungo.fivemincron.plist. If you are a tmux user, you must run this command outside of tmux or you'll get a permission denied error.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">







Hopefully it's obvious but that runs /Users/sungo/bin/five_minute_cron every 300 seconds. For a normal person, you can probably stop here. Put whatever code you want to run in-series in that script and you're good.

I'm not normal.

With cron, if I set several commands to run every five minutes, they run in parallel. Putting those commands into a script gets them to run in series which is not what I want.

gnu-parallel to the rescue.

Here's what five_minute_cron looks like with a dash of crazy.

#!/usr/local/bin/parallel --shebang --will-cite :::


-shebang tells parallel that the following lines make up its config file. -will-cite gets rid of the really fucking annoying plea for citations. (Yeah, I don't even, either.) The ::: tells parallel to use the remaining arguments as command input source rather than waiting for commands via stdin.

All in all, that script runs the commands listed in parallel. Launchd will run the script every 300 seconds.

So now I have a cron-ish replacement using launchd and gnu parallel.

Surviving A Botswarm - A User's Guide

2014-07-18 21:58 EDT

So yeah. has been under a sustained bot incursion for a few days now. The network opers are actively working the problem the best we can. At the end of the day, though, with the existence of cheap instant virtual machines and the wonderful world of Tor, this sort of incident is difficult to stop completely.

(For the record, I don't view this as an attack. Near as I can tell, this is one kid getting his jollies by occasionally spamming channels. This thing is mostly automated except when he gets bored and tries to taunt me with rainbow colors. Which, incidentally, is why hereafter this person will be known as The Rainbow Warrior.)

Anyway, I don't really want to talk about bored 14 yr olds.

Let's talk instead about what you can do to survive a botswarm.

First, set your client to ignore joins, parts, quits, and maybe modes. In irssi, that command is "/ignore * JOINS PARTS QUITS MODES". I have this set in my client in most channels, particularly the quiet ones. It cuts the noise significantly.

Some clients provide a function to ignore nick changes as well. In irssi, add NICKS to that command above. I don't recommend this in normal use but right now, with bots that change nicks constantly, it might be a good idea.

Second, figure out how to make your irc client strip color codes. If you're an adult hanging out with adults, chances are you forgot you could do color in irc. It's safe to strip out and unless you're making a foray to the wonderful land of Dalnet, you'll never miss it. (I'd give you the irssi command but I don't know it yet. If you know, clue me in and I'll update this.)

Third, ignore CTCP. (Add CTCPS to the ignore command listed above.) This is 'client-to-client protocol', again something you probably forgot about. It's mostly used to negotiate DCC sessions and annoy people via the 'notice' message type.

Side note: You really should ignore DCC as well, just from a security perspective. In this day and age, it is a ridiculously bad idea to allow a direct connection to your irc client from another client on the internet. Historically, DCC was used to transmit files. You'd join #warez on dalnet, see a bot's message (in color, no doubt) and use DCC to download files from it. I have no idea why we did this instead of usenet but whatevs. This was always a bad idea. Add "DCC" to that ignore command I keep referencing to ignore this blast from the past too.

As a user, that's really all you can do, sadly. Ignore the junk as best you can. If it drives you nuts, it's ok if you walk away for a while. We understand. I'm on the response team for this shit and I often slam the laptop lid closed and walk away for a while.

Let's talk about what channel operators can do to help protect their users.

First, go read this New IRC Channel Operator's Guide from Even if you're not a new operator, go read it anyway.

If you suddenly get hit by a flood of joins from people you don't know, just make the channel +im temporarily so they can't keep coming in and can't flood in the channel. Note they can still cause flooding such as by rapidly changing their nicknames. Now just kick them without bans since they cannot rejoin while you are +i, that gives you time to set proper bans after you've kicked them all out.

There are three basic defensive postures. (These can be aided by the use of control bots but do not require it.)

  • Set the channel +k. This "keys" the channel and requires the user to know a special string to join. Only give that key to people you like and if the bots get in, you know one of the people you like is a jackass.

  • Set the channel to +i. This sets the channel to "invite only". Users must be allowed in via the /invite command to access. On, users can use the /knock command to request an invite. If you do this, I suggest leaving the channel -s and put this info in the topic. That way, a user can do '/list #yourchannel' and see that they need to /knock first.

  • Set the channel to +m. This moderates the channel and only people with +o (ops), +h (halfops) or +v (voice) can speak. People can join but won't be able to speak until given +v. If someone, bot or not, gets out of control, you can just -v them. It's pretty much the same as "hellbanning", to use the modern parlance. Many IRC clients can be configured to automatically give +v to people when they join. (That is typically much easier and a better idea than dealing with channel control bots.)

  • Make heavy use of halfops. The +h (halfops) user mode is one of the most underutilized feature of, in my opinion. Halfops have the ability to do all the normal ops stuff. Kick people, ban people, voice people, etc. They cannot however perform those actions against anyone who is +o. It's a really nice way to create a class of users who can help police the channel but are visibly different from the channel leadership.

That's all I have for now. If you find any tips I should know about or blog about, grab me on First Community Opers

2014-05-31 15:14 EDT

It is my privilege to announce the first batch of community opers. They are tasked with drafting a charter and maintaining the peace on under the terms of the Standards Of Conduct. We'll have our first meeting at YAPC::NA 2014.

  • Coordinating Network Opers: sungo (me), mst (Matt Trout)
  • Supervising Network Oper: perigrin (Chris Prather)
  • Community Opers (in alpha order):
    • ether (Karen Etheridge)
    • garu (Breno G. de Oliveira)
    • genehack (John Anderson)

Sometimes, when governance is put in place and when the leaders are announced, the shady parts of the community come forward to harass them. Let me be really clear about this. Folks, you had plenty of time to discuss governance sensibly. If you have issues with these community opers, we can discuss it. But if you do more than discuss it, if you harass them in any way, I will take it as a threat to network safety and security and I will respond as strongly as I am allowed under the SoC and governance rules.

It's really unfortunate that I feel the need to document my stance but we've dealt with this before and I lost my sense of humor about it a long time ago.