Effect of Let's Encrypt root certificate authority changes on ODK ecosystem?

Support
1

1. What is the problem? Be very detailed.

Over the weekend, Let's Encrypt posted an article about how they are changing their root certificate authority in 2021 (https://letsencrypt.org/2020/11/06/own-two-feet.html). Let's Encrypt has been cross signing with a broadly accepted root authority IdenTrust, but that is about to change in 2021. They will no longer be cross signing but using only their own root certificate authority. Specifically, they say that Android versions prior to version 7.1.1 do not yet trust Let's Encrypt's root certificate authority. The article says that this affects software / operating systems not updated since 2016.

I'm wondering how much of an impact this will have on the ODK community, given that ODK server software use Let's Encrypt for TLS certificates. Worst case scenario is that these old phones would not be able to connect in order to submit data or get forms.

2. What app or server are you using and on what device and operating system? Include version numbers.

This may not affect our organization. We are still checking within our network to see what Android versions are in use. However, we have held onto old Android phones for a long time previously, so it would not surprise me if we have old, vulnerable Android phones being used for data collection.

3. What you have you tried to fix the problem?

It seems according to Odk Collect with Custom Certificate: Android 5 (works), Android 7,8 (fails), ODK uses the operating system's root certificate authorities.

4. What steps can we take to reproduce the problem?

This isn't a problem yet, but a potential problem.

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.

Nothing for now.

1 Like
2

Thanks for bringing this up, @jpringle. We are hoping that the disruption will be minimal. To summarize:

  • ISRG Root X1 is a Let's Encrypt's root and is recognized by Android 7.1.1+
  • DST Root X3 is a IdenTrust root and is recognized more broadly
  • Jan 2021: by default, certs issued by Let's Encrypt will be rooted by ISRG Root X1 but can still use DST Root X3 with a configuration change when requesting a certificate
  • Sept 2021: DST Root X3 will expire so all certs issued by Let's Encrypt will be rooted by ISRG Root X1

Our current plan is to explicitly specify trust for ISRG Root X1 in Collect starting with v1.29. This will allow devices running Android 6+ to connect to any server that uses any certificate issued by Let's Encrypt either before or after the Jan 2021 change. Devices will need to upgrade Collect to benefit from this change. We expect v1.29 to go out early to mid December.

This leaves a question about what to do for Android 5.0 and 5.1. These versions represent about 5% of Collect's active users. We have no way of knowing how many of those currently connect to a server that uses a certificate issued by Let's Encrypt but it's probably not all of them.

Here are some possible answers:

  • Do nothing. It's a relatively small slice of the user base. We can document how users can manually add ISRG Root X1 to Android and that will be recognized by Collect (on Android 5.0 and 5.1)
  • Provide more helpful documentation. Even if it's a small slice of the user base, it's still up to ~45k people. Detect the specific certificate failure and provide a link to explicit guidance for adding the certificate as described above.
  • Programmatically register support for ISRG Root X1 with our HTTP library (okhttp). I don't know exactly what this looks like but it should be possible.

If you use Android 5.0 or 5.1 devices and certificates issued by Let's Encrypt, it would be helpful to hear from you.

As always, any other ideas on how to address this are welcome.

3

I was inclined to do nothing for Android <= 5.1 but the @TAB discussed and felt supporting all Android versions was important. I'm close to a solution for that so there should be no disruption to users who update to Collect v1.29+.

3 Likes
4

We asked all of our various country teams what Android versions are in use, and it turns out that one country is using Android 6, and their data collection is starting now and scheduled to last until the end of January or possibly February 2021. So too late to update Collect or purchase new phones. But there is a solution!

We are modifying the behavior of certbot in order to keep cross signing with DST Root CA X3 until at least July 2021. The instructions are at https://community.letsencrypt.org/t/certbot-users-preparing-for-the-isrg-root-transition-january-11-2021/138059 (I'll refer to this as the "Let's Encrypt notice" later) and I will relay what I have learned here.

First: we are using Digital Ocean and ODK Aggregate (I promise we will upgrade to Central for the next wave of data collection!). So we followed https://docs.getodk.org/aggregate-digital-ocean/ exactly. Including the cloud config script. The certbot commands from the cloud config script from when we installed the server way back:

add-apt-repository -y ppa:certbot/certbot
apt-get -y update
apt-get -y install python-certbot-nginx
(crontab -l 2>/dev/null; echo "0 0 1 * * /usr/bin/certbot renew > /var/log/letsencrypt/letsencrypt.log") | crontab -

When I ssh'd into the machine and ran certbot --version I got in response: certbot 0.31.0 and according to the Let's Encrypt notice, I need v1.6.0 or newer.

I tried the default for upgrading installed software: apt-get update && apt-get upgrade but that did not upgrade certbot, so I followed the instructions (linked in the Let's Encrypt notice) at https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx (the Digital Ocean droplet is running Ubuntu 18.04 Bionic). This method installs certbot as a snap using the snap package repository. My Ubuntu came with snap already installed!

First, uninstall certbot:

apt-get remove certbot

That also removed python-certbot-nginx and python3-certbot-nginx, so I made a note to check this at the end of installation.

Second, install certbot with snap

snap install --classic certbot

Third, make a link so that we can run certbot from the old location

ln -s /snap/bin/certbot /usr/bin/certbot

Finally, check the version and do a dry run of a renewal:

certbot --version           # Result: certbot 1.9.0
certbot renew --dry-run

And from the output of the dry run:

...
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
...

it appears that this new certbot knows about the connection to nginx.

Now that certbot is updated, it is able to do the desired certificate continuing from January 2021. The last step is to add a line to /etc/letsencrypt/cli.ini so that we activate that functionality. So I added

preferred-chain = DST Root CA X3

to that file.

I hope this works for me! I'll report back if anything unexpected comes up.

1 Like
5

Thanks for sharing your experience and solution, @jpringle!

To summarize, if you use Android devices with versions < 7.1.1 AND they connect to a server that uses a certificate issued by Let's Encrypt :

  • If you can upgrade to Collect v1.29+ before June 2021 (tentative date from Let's Encrypt), do so, and no other changes are needed.
  • If you can't upgrade to Collect v1.29+ before June 2021 but you will be able to before January 2024 or won't be using older Android devices by then, you can continue to cross-sign your certs until you can upgrade Collect. @jpringle's instructions may be helpful if you're on Digital Ocean. Otherwise refer to the Let's Encrypt post
  • If you can't upgrade to Collect v1.29+ before January 2024, I believe you will need to switch to a certificate authority other than Let's Encrypt. That's far enough in the future that things may very well change a lot before then.
2 Likes
6

Let's Encrypt certs will now be cross-signed beyond the expiration date of the IdenTrust root certificate. I have updated the dates in my summary above to reflect this. The major change is that no one needs to make any changes before at least June 2021.