elukey@cumin1002:~$ sudo cookbook sre.network.tls lsw1-d1-codfw
Acquired lock for key /spicerack/locks/cookbooks/sre.network.tls: {'concurrency': 20, 'created': '2024-07-17 07:36:21.455628', 'owner': 'elukey@cumin1002 [964742]', 'ttl': 1800}
START - Cookbook sre.network.tls for network device lsw1-d1-codfw
lsw1-d1-codfw: ❌ Can't connect to device, assuming initial bootstrap.
lsw1-d1-codfw: 🔏 cfssl called with operation: gencert.
lsw1-d1-codfw: ⚙️ Deploy needed.
lsw1-d1-codfw: 👍 All done.
Released lock for key /spicerack/locks/cookbooks/sre.network.tls: {'concurrency': 20, 'created': '2024-07-17 07:36:21.455628', 'owner': 'elukey@cumin1002 [964742]', 'ttl': 1800}
END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d1-codfw

Wed, Jul 17, 7:39 AM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Change #1054618 merged by Elukey:

[operations/cookbooks@master] sre.network.tls: use a different client certificate to authenticate

https://gerrit.wikimedia.org/r/1054618

Wed, Jul 17, 7:33 AM · CFSSL-PKI, Infrastructure-Foundations

Tue, Jul 16

gerritbot added a project to T355750: CFSSL gencert "remote error: tls: certificate require": Patch-For-Review.

Tue, Jul 16, 4:52 PM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Change #1054618 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.network.tls: use a different client certificate to authenticate

https://gerrit.wikimedia.org/r/1054618

Tue, Jul 16, 4:52 PM · CFSSL-PKI, Infrastructure-Foundations

elukey added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

I managed to get the certificate via:

Tue, Jul 16, 4:47 PM · CFSSL-PKI, Infrastructure-Foundations

Maintenance_bot removed a project from T355750: CFSSL gencert "remote error: tls: certificate require": Patch-For-Review.

Tue, Jul 16, 1:31 PM · CFSSL-PKI, Infrastructure-Foundations

elukey added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

In T355750#9976497, @elukey wrote:

Completely different use case: traffic-cache-upload-bullseye.traffic.eqiad1.wikimedia.cloud

In there purged needs to get a cfssl discovery cert, but I see the same error reported in the task's description. I suspect this could be related to mutual-tls-cert and mutual-tls-key, since when I try to run the correspondent cfssl gencert command I get the same error with/without those options.

Tue, Jul 16, 1:07 PM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Change #1053937 merged by Elukey:

[operations/puppet@production] pki: add the Traffic's project Puppet CA to client_auth_CA.pem in cloud

https://gerrit.wikimedia.org/r/1053937

Tue, Jul 16, 12:36 PM · CFSSL-PKI, Infrastructure-Foundations

Mon, Jul 15

elukey added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

After some digging, it seems to me that the issue is httpd on pki1001: it rejects the client authentication from cumin1002. I added a bit more logging to the mod-ssl module, and this is what I see in the ssl error log:

Mon, Jul 15, 11:02 AM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Change #1054289 merged by Elukey:

[operations/puppet@production] profile::pki::multirootca: use info in the client auth vhost

https://gerrit.wikimedia.org/r/1054289

Mon, Jul 15, 10:05 AM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Change #1054289 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::pki::multirootca: use info in the client auth vhost

https://gerrit.wikimedia.org/r/1054289

Mon, Jul 15, 9:59 AM · CFSSL-PKI, Infrastructure-Foundations

Fri, Jul 12

elukey added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

I tried to validate client_auth_CA.pem on pkiXXXX and it looks good (allowing Puppet5/7/PKI client certs), so it must be something client-cert related but I am still missing what.

Fri, Jul 12, 3:36 PM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a project to T355750: CFSSL gencert "remote error: tls: certificate require": Patch-For-Review.

Fri, Jul 12, 1:59 PM · CFSSL-PKI, Infrastructure-Foundations

gerritbot added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Change #1053937 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] pki: add the Traffic's project Puppet CA to client_auth_CA.pem in cloud

https://gerrit.wikimedia.org/r/1053937

Fri, Jul 12, 1:59 PM · CFSSL-PKI, Infrastructure-Foundations

elukey added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Completely different use case: traffic-cache-upload-bullseye.traffic.eqiad1.wikimedia.cloud

Fri, Jul 12, 1:07 PM · CFSSL-PKI, Infrastructure-Foundations

May 31 2024

cmooney added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Actually something tangentially related, lsw1-a1-codfw had a cert generated previously. This switch has now been moved and re-purposed with a new name, lsw1-d1-codfw.

May 31 2024, 10:53 AM · CFSSL-PKI, Infrastructure-Foundations

cmooney added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

In T355750#9825850, @CDanis wrote:

Hi Arzhel, for when I do have time to look at this, do you have a recommended way of reproducing without breaking anything or potentially actually affecting a network device?

May 31 2024, 10:45 AM · CFSSL-PKI, Infrastructure-Foundations

May 27 2024

joanna_borun triaged T365362: Alert and automate the renewal of CFSSL intermediate CAs as Medium priority.

May 27 2024, 2:27 PM · CFSSL-PKI, Infrastructure-Foundations

joanna_borun triaged T365361: Establish a process to periodically upgrade the CFSSL infrastructure as Medium priority.

May 27 2024, 2:27 PM · CFSSL-PKI, Infrastructure-Foundations

May 23 2024

ayounsi added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

sudo cookbook sre.network.tls --system lsw1-f8-eqiad

May 23 2024, 2:24 PM · CFSSL-PKI, Infrastructure-Foundations

CDanis added a comment to T355750: CFSSL gencert "remote error: tls: certificate require".

Hi Arzhel, for when I do have time to look at this, do you have a recommended way of reproducing without breaking anything or potentially actually affecting a network device?

May 23 2024, 2:20 PM · CFSSL-PKI, Infrastructure-Foundations