In my day job I spend a lot of time dealing with sketchy TLS setups. The error messages provided by OpenSSL tend to be quite opaque – fortunately there’s tools to help you.

In the concrete case I needed to send a client certificate and the connections failed with SSLV3_ALERT_CERTIFICATE_UNKNOWN. I have already disabled all client-side verification, but it kept happening, so I suspected it’s a server-side problem. A server that I don’t control.

To verify, I reached out to my friend and tireless ssl maintainer Christian Heimes and he taught me two new-ish debugging techniques for cases like this.

Wireshark

First, it’s surprisingly easy to debug SSL/TLS handshake errors in Wireshark. Capture the failing session and then filter out the packages that went between you and the TLS peer.

For instance if your peer’s IP is 1.2.3.4, you’d add filter like this:

ip.addr == 1.2.3.4 and tls

And this is how I verified my suspicion:

Wireshark showing me the culprit.
Wireshark showing me the culprit.

As suspected, the server was refusing my client certificate. But now I could send it to the other party to help them debug it on their end.

While my particular troubles ended here, let’s look further what else you can do! For instance, for troubleshooting handshakes this is all good, sometimes you need to look into the encrypted traffic and that’s where it gets interesting.

Bonus: Peeking Into Encrypted TLS Traffic

While encryption is there to exactly not do that, you actually can snoop into encrypted traffic.

This is well-described in Everything curl for curl, but it works with every Go program and Python script too:

Set the well-known environment variable SSLKEYLOGFILE to a file name and software that supports it will write the connection-specific secret keys into it.

Then you can use that file to decrypt captured TLS traffic by loading it in Wireshark: Preferences → Protocols → TLS → (Pre)-Master Secret log filename

SSL added and removed here! 😀

Python Secret: ssl.SSLContext has a Message Callback

In Python 3.8 and later, you can set a super secret callback attribute on ssl.SSLContext called _msg_callback. This is something the ssl maintainers added for themselves as a private debugging tool. Thus you can absolutely not rely on its future existence and behavior. However, when in a pickle, it can be very useful.

It gets called with 6 arguments on each TLS message:

  1. The connection object (either ssl.SSLObject or ssl.SSLSocket).
  2. The direction ("read" or "write").
  3. The protocol message version (ssl.TLSVersion). This is not the version of the current connection and varies depending on the message type.
  4. The content type which is a private enum called _TLSContentType.
  5. The message type that can be – depending on the type of the message – one of the three private enums:
  6. The decrypted payload as bytes.

You can use that to either log them out or to put a pdb.set_trace() into it. This kind of introspection is groundbreaking compared to what we used to have!