Paste P63281

move-vlan dry-run error
ActivePublic

Authored by ayounsi on May 27 2024, 9:12 AM.
Tags
None
Referenced Files
F54438687: move-vlan dry-run error
May 27 2024, 9:12 AM
Subscribers
None
DRY-RUN: Acquiring lock for key spicerack.dhcp.DHCP:codfw: {'concurrency': 1, 'created': '2024-05-27 09:11:16.334191', 'owner': 'ayounsi@cumin1002 [3059137]', 'ttl': 60}
DRY-RUN: Reduce tries from 27 to 1 in DRY-RUN mode
DRY-RUN: Issuing read for key /spicerack/locks/modules/spicerack.dhcp.DHCP:codfw with args {'timeout': 60}
DRY-RUN: Skipping lock acquire/release in DRY-RUN mode
DRY-RUN: Acquired lock for key /spicerack/locks/modules/spicerack.dhcp.DHCP:codfw: {'concurrency': 1, 'created': '2024-05-27 09:11:16.348363', 'owner': 'ayounsi@cumin1002 [3059137]', 'ttl': 60}
DRY-RUN: Executing commands ['/bin/rm -v /etc/dhcp/automation/ttyS1-115200/wikikube-worker2001.conf'] on 1 hosts: install2004.wikimedia.org
DRY-RUN: Executing commands ['/usr/local/sbin/dhcpincludes -r commit'] on 1 hosts: install2004.wikimedia.org
DRY-RUN: Releasing lock for key spicerack.dhcp.DHCP:codfw with ID e6935a70-7784-4032-89b9-af5c065ce106
DRY-RUN: Issuing read for key /spicerack/locks/modules/spicerack.dhcp.DHCP:codfw with args {'timeout': 60}
DRY-RUN: Lock for key /spicerack/locks/modules/spicerack.dhcp.DHCP:codfw and ID e6935a70-7784-4032-89b9-af5c065ce106 not found. Unable to release it. Was expired?
DRY-RUN: Deleting local Puppet certificate on 1 hosts: wikikube-worker2001.codfw.wmnet
DRY-RUN: Executing commands ['rm -rfv /var/lib/puppet/ssl'] on 1 hosts: wikikube-worker2001.codfw.wmnet
DRY-RUN: Generating a new Puppet certificate on 1 hosts: wikikube-worker2001.codfw.wmnet
DRY-RUN: Executing commands [cumin.transports.Command('puppet agent --test --color=false', ok_codes=[])] on 1 hosts: wikikube-worker2001.codfw.wmnet
DRY-RUN: Exception raised while executing cookbook sre.hosts.reimage:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 250, in _run
raw_ret = runner.run()
File "/home/ayounsi/cookbooks_testing/cookbooks/cookbooks/sre/hosts/reimage.py", line 671, in run
fingerprint = self.puppet_installer.regenerate_certificate()[self.fqdn]
File "/usr/lib/python3/dist-packages/spicerack/puppet.py", line 294, in regenerate_certificate
raise PuppetHostsError(
spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are:
DRY-RUN: Executing commands ['/usr/bin/facter -p networking.mac'] on 1 hosts: wikikube-worker2001.codfw.wmnet
DRY-RUN: Executing commands ['clear dhcp relay binding 10.192.16.39 routing-instance PRODUCTION', 'clear ethernet-switching mac-ip-table d0:8e:79:f5:6f:94'] on 1 hosts: lsw1-b5-codfw.mgmt.codfw.wmnet
DRY-RUN: Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
DRY-RUN: **The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" wikikube-worker2001.codfw.wmnet to get a root shellbut depending on the failure this may not work.**
DRY-RUN: Reimage executed with errors:
- wikikube-worker2001 (**FAIL**)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
- **The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" wikikube-worker2001.codfw.wmnet to get a root shellbut depending on the failure this may not work.**
DRY-RUN: Releasing lock for key sre.hosts.reimage:wikikube-worker2001 with ID 9d705424-8ed2-4517-b2ed-c6505904aee0
DRY-RUN: Issuing read for key /spicerack/locks/cookbooks/sre.hosts.reimage:wikikube-worker2001 with args {'timeout': 60}
DRY-RUN: Lock for key /spicerack/locks/cookbooks/sre.hosts.reimage:wikikube-worker2001 and ID 9d705424-8ed2-4517-b2ed-c6505904aee0 not found. Unable to release it. Was expired?
DRY-RUN: __COOKBOOK_STATS__:name=sre.hosts.reimage,exit_code=99,duration=166.331
DRY-RUN: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2001.codfw.wmnet with OS bullseye