How to check OpenSSL: alert internal error, handshake failure for CA

Question

Context is our Mosquitto broker, running on a certain domain. User can connect via TLS only.

We are using a self-signed certificate for this purpose, because we want to sign client certificates by ourselves.

Command on my machine

mosquitto_pub -h mqtt.sustainable-data-platform.org -p 8883 --cafile ./ca.cert.pem -t test -m "fff aaa" -u <user> -P <pssw>

The CA cert

openssl x509 -noout -text -in ./ca.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=DE, ST=NRW, L=Koeln, O=sdp, CN=sdp Server/emailAddress=...
        Validity
            Not Before: Jun 18 09:48:29 2024 GMT
            Not After : Jun 13 09:48:29 2044 GMT
        Subject: C=DE, ST=NRW, L=Koeln, O=sdp, CN=sdp Server/emailAddress=...
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ...

SSL Error

OpenSSL Error[0]: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
06-20T12:46:09: OpenSSL Error[1]: error:140940E5:SSL routines:ssl3_read_bytes:ssl handshake failure

Guess

The domain does not match with CN of my CA certificate, do we need these to match? Any other way to fix the problem, because I am not familiar with using self-signed certificates for mosquitto?

Edit

openssl s_client

I have checked this with openssl s_client --CAfile ./ca.cert.pem -connect mqtt.sustainable-data-platform.org:8883 and it returns (snippets):

CONNECTED(00000003)
depth=1 C = DE, ST = NRW, L = Koeln, O = sdp, CN = sdp Server, emailAddress = <mail>
verify return:1
depth=0 C = DE, ST = NRW, O = sdp, OU = sdp Server, CN = sdp server interm
verify return:1
---
Certificate chain
 0 s:C = DE, ST = NRW, O = sdp, OU = sdp Server, CN = sdp server interm
   i:C = DE, ST = NRW, L = Koeln, O = sdp, CN = sdp Server, emailAddress = <mail>
 1 s:C = DE, ST = NRW, L = Koeln, O = sdp, CN = sdp Server, emailAddress = <mail>
   i:C = DE, ST = NRW, L = Koeln, O = sdp, CN = sdp Server, emailAddress = <mail>
---
Server certificate
-----BEGIN CERTIFICATE-----

[...]

---
SSL handshake has read 3163 bytes and written 452 bytes
Verification: OK
---

[...]

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:  
        [...]

[...]

    Start Time: 1718884057
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---