A little bit more information from wpasupplicant:
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: TLS: tls_verify_cb - preverify_ok=1 err=0 (ok) ca_cert_verify=1 depth=0 buf='/CN=subject'
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: EAP: Status notification: remote certificate verification (param=success)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: (where=0x1001 ret=0x1)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: SSL_connect:SSLv3/TLS read server certificate
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: RX ver=0x301 content_type=22 (handshake/server key exchange)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: Message - hexdump(len=331): [REMOVED]
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: TX ver=0x301 content_type=256 (TLS header info/)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: Message - hexdump(len=5): [REMOVED]
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: TX ver=0x301 content_type=21 (alert/)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: Message - hexdump(len=2): [REMOVED]
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: (where=0x4008 ret=0x250)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: SSL3 alert: write (local SSL3 detected an error):fatal:internal error
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: EAP: Status notification: local TLS alert (param=internal error)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: (where=0x1002 ret=0xffffffff)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: SSL_connect:error in error
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: OpenSSL: openssl_handshake - SSL_connect error:0A0C0103:SSL routines::internal error
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: 7 bytes pending from ssl_out
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: Using TLS version TLSv1
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: Failed - tls_out available to report error (len=7)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: SSL: 7 bytes left to be sent out (of total 7 bytes)
Mär 28 16:26:26 apollo13 wpa_supplicant[1855]: EAP-PEAP: TLS processing failed

It seems as if it successfully validated the cert?

This bug is also present in Ubuntu: jhttps://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/1104476. It seems that "system-ca-certs=true" is added to the configuration regardless of what is selected in the UI. A workaround is to edit this field after creating the configuration.

Not sure, the ubuntu bug is from 2013 and seems to be popping up again now. Fwiw I do not have system-ca-certs=true since Fedora seems to use the sysconfig/ifcfg backend. This is my current (broken) configuration:

ESSID=MY_NETWORK
MODE=Managed
KEY_MGMT=WPA-EAP
MAC_ADDRESS_RANDOMIZATION=default
TYPE=Wireless
IEEE_8021X_EAP_METHODS=PEAP
IEEE_8021X_IDENTITY=myuser
IEEE_8021X_INNER_AUTH_METHODS=MSCHAPV2
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=MY_NETWORK
UUID=de0d8ea2-0732-4d81-8ca5-c79b4b2e67a4
ONBOOT=yes
IEEE_8021X_CA_CERT=/home/florian/.pki/ROOT-CA.crt
IPV6_PRIVACY=rfc3041
MACADDR=permanent
DHCP_CLIENT_ID=mac

I tried https://bugzilla.redhat.com/show_bug.cgi?id=2071615 because I needed LEGACY in fedora 35 for it to work. Updating to the new openssl from testing did not help either. Any idea on how to verify "system-ca-certs=true" for the ifcfg plugin?

I have switched from ifcfg to keyfile for this connection and system-ca-certs is not to be found.

Got more debug information from wpa_supplicant:

Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: wlp0s20f3: CTRL-EVENT-EAP-PEER-CERT depth=0 subject='/CN=dc1.domain.com' hash=
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: wlp0s20f3: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:dc1.domain.com
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: wlp0s20f3: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:domain.com
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: wlp0s20f3: CTRL-EVENT-EAP-PEER-ALT depth=0 DNS:DOMAIN
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: TLS: tls_verify_cb - preverify_ok=1 err=0 (ok) ca_cert_verify=1 depth=0 buf='/CN=dc1.domain.com'
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: TLS: Match domain against suffix domain.com
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: TLS: Certificate dNSName - hexdump(len=18): XX XX XX XX XX
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: TLS: Suffix match in dNSName found
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAP: Status notification: remote certificate verification (param=success)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: (where=0x1001 ret=0x1)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: SSL_connect:SSLv3/TLS read server certificate
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: RX ver=0x301 content_type=22 (handshake/server key exchange)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: Message - hexdump(len=331): [REMOVED]
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: TX ver=0x301 content_type=256 (TLS header info/)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: Message - hexdump(len=5): [REMOVED]
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: TX ver=0x301 content_type=21 (alert/)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: Message - hexdump(len=2): [REMOVED]
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: (where=0x4008 ret=0x250)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: SSL3 alert: write (local SSL3 detected an error):fatal:internal error
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAP: Status notification: local TLS alert (param=internal error)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: (where=0x1002 ret=0xffffffff)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: SSL_connect:error in error
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: OpenSSL: openssl_handshake - SSL_connect error:0A0C0103:SSL routines::internal error
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: 7 bytes pending from ssl_out
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: Using TLS version TLSv1
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: Failed - tls_out available to report error (len=7)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: SSL: 7 bytes left to be sent out (of total 7 bytes)
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAP-PEAP: TLS processing failed
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAP: method process -> ignore=FALSE methodState=DONE decision=FAIL eapRespData=0x55b2b219ae10
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAP: EAP entering state SEND_RESPONSE
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAP: EAP entering state IDLE
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAPOL: SUPP_BE entering state RESPONSE
Apr 22 08:47:07 apollo13 wpa_supplicant[20904]: EAPOL: txSuppRsp

CCing some openssl people to see if they can help with debugging this. Reporter is happy to help debug, just needs some pointers.

https://bugs.launchpad.net/ubuntu/+source/wpa/+bug/1958267 seems related. Have you tried https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1963834/comments/7?

I have not, I will try that next week when I am in the office (sadly I only can try it there). That said as comment #23 indicates there (https://bugs.launchpad.net/ubuntu/+source/wpa/+bug/1958267/comments/23) the fix does not help and they get "internal error" like I do. Users where the suggested fix did help seemed to get a clear mention of unsafe renegotiation in the error.

I'd appreciate any other ideas you have, will collect them and try them next week.

You could try setting a breakpoint on ossl_statem_fatal() and see if the backtrace gives any indication on which specific invocation of SSLfatal with SSL_AD_INTERNAL_ERROR out of the roughly 600 in the OpenSSL source code caused this.

Perfect, will do. One question though, TLS is handled by radius here (I guess?). Could I try peap authentication like that manually against our Radius server without involving WIFI -- this way I could test this remotely (I guess).

I'd say yes, but I'm not an expert on wpa_supplicant and WiFi. Maybe the wpa_supplicant maintainers can help on this question?

Ok, I managed to get access to the radius server and simulate wpa_supplicant via eapol_test. It errors out here:

https://github.com/openssl/openssl/blob/1bf649b5998ac98511203f54ce954eccfaf75467/ssl/statem/statem_clnt.c#L2248

and inside tls1_set_peer_legacy_sigalg in: 

https://github.com/openssl/openssl/blob/1bf649b5998ac98511203f54ce954eccfaf75467/ssl/t1_lib.c#L1336-L1338

I'll be debugging further, but I am not 100% sure what to query exactly.

A few observations:

1. You're probably using TLS 1.1 or lower. In TLS 1.2 or above, the client should send the signature_algorithms TLS extension, SSL_USE_SIGALGS(s) would be true, and this code wouldn't run.
2. This code path is supposed to determine the signature algorithm used by the server to sign part of the communication with the server certificate. The (TLS 1.2) RFC says this:

   If the client does not send the signature_algorithms extension, the
   server MUST do the following:

   -  If the negotiated key exchange algorithm is one of (RSA, DHE_RSA,
      DH_RSA, RSA_PSK, ECDH_RSA, ECDHE_RSA), behave as if client had
      sent the value {sha1,rsa}.

   -  If the negotiated key exchange algorithm is one of (DHE_DSS,
      DH_DSS), behave as if the client had sent the value {sha1,dsa}.

   -  If the negotiated key exchange algorithm is one of (ECDH_ECDSA,
      ECDHE_ECDSA), behave as if the client had sent value {sha1,ecdsa}.

   Note: this is a change from TLS 1.1 where there are no explicit
   rules, but as a practical matter one can assume that the peer
   supports MD5 and SHA-1.
3. It depends on the server's certificate which algorithm will be chosen. I'm assuming your server is offering an RSA certificate, since that's most common, but you should check this.
4. If the server uses RSA and now SSL_USE_SIGALGS(s) is false, the default is rsa_pkcs1_md5_sha1.
5. Before the end of tls1_get_legacy_sigalg(), there's an invocation of tls12_sigalg_allowed() with the chosen signature algorithm. That eventually checks whether sigalg_security_bits() is high enough for your chosen SECLEVEL. As a special case due to bug 2071615, Fedora allows SHA1 in SECLEVEL 1 (i.e. crypto-policy LEGACY). For MD5-SHA1, the security bits value is 67, for MD5 it's 39. For security level 1, the minimum accepted value is 80. Try setting your security level to 0 and see if that helps.

I'd recommend asking your radius server admin to support TLS 1.2 or newer.

I'll start a discussion whether we also want to allow rsa_pkcs1_md5_sha1 in LEGACY.

I just wanted to post that SECLEVEL=0 fixes it, I also verified via GDB (took me a while since the value is optimized out) that the value is indeed 67. I will see about getting that updated.

Internal discussion resulted in agreement that we want to support rsa_pkcs1_md5_sha1 in LEGACY, so I'll follow up with a patch similar to bug 2071615, which also allows that.

Hi Clemens, many thanks for your support on this. Is there a chance that you can shed some light on the internal discussion? I am wondering why you are going to enable this -- do you happen to have data at hand that suggests that this is widely used by windows radius or so? As for me I will see about using that to push a TLS update, which would be more in line with out company policies anyways…

It's a simple discussion:
- In LEGACY crypto-policy, Fedora supports TLS down to TLS 1.0
- TLS 1.0 expects to use rsa_pkcs1_md5_sha1

This means our current configuration is inconsistent. Either it should require TLS >= 1.2 and not support rsa_pkcs1_md5_sha1, or it should support rsa_pkcs1_md5_sha1 if TLS 1.0 is allowed.

In LEGACY crypto policy for F36, we've made the decision to allow TLS 1.0, because there is probably a myriad of devices out there that doesn't speak newer TLS. We can revisit this decision for F37 or F38, but for now we'd probably annoy many users if LEGACY didn't even support those devices anymore.

If you get the update to TLS 1.2, you should be able to switch to the DEFAULT crypto policy, which requires TLS >= 1.2 and does not allow SHA1 or MD5-SHA1.

Thank you for the explanation. And I somewhat doubt that I can switch back to DEFAULT because as you say there is a myriad of other annoying things out there :/

Thanks for the prompt investigation on this, both of you!

I'm gonna propose this as an FE just in case we slip again, as this seems like something people might want to work on live and installer images. I don't think it's likely to be common enough to make it a blocker, especially as you already have to change the default crypto policy, so changing one more thing (the SECLEVEL) to make it work isn't too onerous of an extra step; I'll flag this as commonbugs so we can document the need to do that.

(In reply to Clemens Lang from comment #18)
> It's a simple discussion:
> - In LEGACY crypto-policy, Fedora supports TLS down to TLS 1.0
> - TLS 1.0 expects to use rsa_pkcs1_md5_sha1
> 
> This means our current configuration is inconsistent. Either it should
> require TLS >= 1.2 and not support rsa_pkcs1_md5_sha1, or it should support
> rsa_pkcs1_md5_sha1 if TLS 1.0 is allowed.

looking at the wpa_supplicant code, I see that it does not load the legacy provider before trying to compute MD5 (there is a commit that fixes DES and RC4, but MD5 looks not "patched" [1]).
If we add a proper call to  That should make the configuration "consistent" in case TLS-1.0 is allowed, looking at the wpa_supplicant code, I see that wpa_supplicant does not load the legacy provider before trying to compute MD5 (there is a commit that fixes DES and RC4, but MD5 looks not "patched" [1]).

Adding a proper call to openssl_load_legacy_provider() in md5_vector() should at least make configuration consistent, correct? So you just need to allow TLS-1.0 in the onfiguration, and don't care about manually enabling the legacy provider. Unfortunately I can't reproduce the faulty scenario easily, so any help in testing this patch would be appreciated.

@cllang  do you think it makes sense to try a small patch that fixes this?

thanks,
-- 
davide

[1] https://w1.fi/cgit/hostap/commit/?id=ff2eccbdf

(In reply to Davide Caratti from comment #21)
> (In reply to Clemens Lang from comment #18)

... wow. I managed to totally scramble words in comment #21. Let's retry with less words.
is it worth testing a patch like the one below? 



--- a/src/crypto/crypto_openssl.c
+++ b/src/crypto/crypto_openssl.c
@@ -391,6 +391,8 @@ out:
 #ifndef CONFIG_FIPS
 int md5_vector(size_t num_elem, const u8 *addr[], const size_t *len, u8 *mac)
 {
+       openssl_load_legacy_provider();
+
        return openssl_digest_vector(EVP_md5(), num_elem, addr, len, mac);
 }
 #endif /* CONFIG_FIPS */

Just to record some things I have noticed:

 * Changing the SECLEVEL to 0 in opensslcnf.config did *not* work.
 * Changing the SECLEVEL to 0 in openssl.config did work. (I have no idea what the difference between those two files is and when one or the other is used)
 * I tried to uncomment the following section in /etc/pki/tls/openssl.cnf

[provider_sect]
##default = default_sect
##legacy = legacy_sect
##
##[default_sect]
##activate = 1
##
##[legacy_sect]
##activate = 1

but that didn't help either. All in all I really appreciate the help from Clemens, without it I would have been lost. There seem to be quite a few knobs to tune openssl and it is hard for an enduser to figure out what tunes what.

(In reply to Davide Caratti from comment #21)
> looking at the wpa_supplicant code, I see that it does not load the legacy
> provider before trying to compute MD5

The MD5 digest is part of the OpenSSL default provider: https://github.com/openssl/openssl/blob/openssl-3.0/providers/defltprov.c#L146-L149

> (there is a commit that fixes DES and RC4, but MD5 looks not "patched" [1]).

That is necessary because RC4 is only available in the OpenSSL legacy provider: https://github.com/openssl/openssl/blob/openssl-3.0/providers/legacyprov.c#L121-L127

> Adding a proper call to openssl_load_legacy_provider() in md5_vector()
> should at least make configuration consistent, correct?

No, that call is unnecessary, as the default provider already supports MD5.


I believe you may be confusing two concepts that have a "legacy" keyword, but are otherwise unrelated:

1. OpenSSL 3 did split its algorithms into providers, specifically the default, legacy, and FIPS providers. Those contain implementations of cryptographic algorithms with different properties, some of them for everyday use (default), some of them that should no longer be used (legacy), and some of them validated (FIPS). While MD5 is a legacy algorithm (in that you should really not use it anymore), the OpenSSL developers have not decided to put it into the legacy provider, so there is no need to explicitly load that.
2. The crypto-policies package provides system-wide default settings for all cryptographic libraries. It supports three major configurations, LEGACY, DEFAULT, and FUTURE, as well as sub-configurations such as, for example, DEFAULT:SHA-1. The chosen crypto-policy changes what configuration settings apply in the default OpenSSL configuration file. Specifically, crypto-policies configures the following settings in OpenSSL:

 - Acceptable ciphers, and the OpenSSL SECLEVEL setting
 - Acceptable TLS1.3 cipher suites
 - The minimum and maximum TLS and DTLS versions
 - Acceptable TLS signature algorithms

Setting the crypto-policy to LEGACY sets the minimum TLS version to 1.0 and the SECLEVEL to 1. However, in OpenSSL 3, that does not enable SHA1-based signature algorithms, so TLS 1.0 does not actually end up working. We could either set SECLEVEL to 0, which would further reduce security, or patch SECLEVEL 1 to allow SHA-1 signature algorithms (when a fedora-specific setting rh-allow-sha1-signatures = yes is set, which is the default). This is what we're doing, but we missed also including MD5-SHA1 signature algorithms in this patch.

FEDORA-2022-2f54478579 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-2f54478579

FEDORA-2022-2f54478579 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Re-opening since this was filed against F36 and should be fixed there.

See https://bodhi.fedoraproject.org/updates/FEDORA-2022-bc51e23571. Florian, could you test to see whether this solves your issue with an unmodified OpenSSL config in LEGACY crypto-policy?

Will do, but can only do it on Monday -- sorry.

In today's Go/No-Go meeting, we agreed to grant a freeze exception. Including this fix on GA images is beneficial for users of affected WiFi networks
https://meetbot.fedoraproject.org/fedora-meeting/2022-04-28/f36-final-go_no_go-meeting.2022-04-28-17.01.log.html#l-644

FEDORA-2022-bc51e23571 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-bc51e23571

FEDORA-2022-bc51e23571 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Clemens: I have verified that the latest release fixes the issue on TLS 1.0.

Thanks for checking.