Ansible hangs when running playbook second time

Question

I'm new to ansible and am trying to automate some steps.

This task runs well the first time.

# Copy Root ssh keys within all devices
- hosts: etall
  tasks:
    - name: ssh keygen
      command: ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P ""

The second time it just hangs after gathering facts. Setting ANSIBLE_DEBUG=1 shows the step hanging at:

16584 1554867575.39541: _low_level_execute_command(): executing: /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1554867574.524454-229535370546451/AnsiballZ_command.py && sleep 0'

While tracing on the client machine:

/usr/bin/python -m trace --trace /root/.ansible/tmp/ansible-tmp-1554873279.0459487-106888464136363/AnsiballZ_command.py

<---output snippet---->
AnsiballZ_command.py(16):     import base64
 --- modulename: trace, funcname: _unsettrace
trace.py(80):         sys.settrace(None)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib64/python2.7/trace.py", line 819, in <module>
    main()
  File "/usr/lib64/python2.7/trace.py", line 807, in main
    t.runctx(code, globs, globs)
  File "/usr/lib64/python2.7/trace.py", line 513, in runctx
    exec cmd in globals, locals
  File "/root/.ansible/tmp/ansible-tmp-1554873279.0459487-106888464136363/AnsiballZ_command.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1554873279.0459487-106888464136363/AnsiballZ_command.py", line 16, in _ansiballz_main
    import base64
ImportError: No module named base64

I can import base64 after /usr/bin/python

This was not a problem during first run. Also works after reinstalling OS on client machine.

Version:

ansible 2.7.6
  config file = /user/user1/plays/ansible.cfg
  configured module search path = ['/user/user1/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /user/user1/p363/lib/python3.6/site-packages/ansible-2.7.6-py3.6.egg/ansible
  executable location = /user/user1/p363/bin/ansible
  python version = 3.6.3 (default, Jul  8 2018, 21:13:48) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Python version in client machin is 2.7.5

CONFIGURATION

ansible-config dump --only-changed
DEFAULT_FORKS(/user/user1/plays/ansible.cfg) = 1
DEFAULT_HOST_LIST(/user/user1/plays/ansible.cfg) = ['/user/user1/plays/hosts.ini']
DEFAULT_LOG_PATH(/user/user1/plays/ansible.cfg) = /user/user1/plays/ansible.log
DEFAULT_ROLES_PATH(/user/user1/plays/ansible.cfg) = ['/user/user1/plays/hosts']
RETRY_FILES_ENABLED(/user/user1/plays/ansible.cfg) = False

ENVIRONMENT: ansible master

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Workstation"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Workstation"
VARIANT_ID="workstation"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:workstation"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"

ansible client

# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

playbook:

# Copy Root ssh keys within all machines
- hosts: etall
  tasks:
    - name: ssh keygen
      command: ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P "" 

    - name: Fetch keyfile
      fetch:
        src: "~/.ssh/id_rsa.pub"
        dest: "buffer/{{ ansible_hostname }}-id_rsa.pub"
        flat: yes

    - name: Copy keyfile to destination
      authorized_key:
        user: root
        state: present
        key: "{{ lookup('file','buffer/{{ item }}-id_rsa.pub') }}"
      when: item != ansible_hostname
      with_items:
        - "{{ groups['etall'] }}"

    - name: ssh keyscan
      shell: ssh-keyscan {{ item }} >> /root/.ssh/known_hosts
      when: item != ansible_hostname
      with_items:
        - "{{ groups['etall'] }}"

Verbose output snippet right after gathering facts:

TASK [ssh keygen] *************************************************************************************************************************************************************************************************
task path: /user/user1/plays/auth_keys.yml:4
<linux-user5> ESTABLISH SSH CONNECTION FOR USER: root
<linux-user5> SSH: EXEC sshpass -d11 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/user/user1/.ansible/cp/0969431db6 linux-user5 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<linux-user5> (0, b'/root\n', b'')
<linux-user5> ESTABLISH SSH CONNECTION FOR USER: root
<linux-user5> SSH: EXEC sshpass -d11 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/user/user1/.ansible/cp/0969431db6 linux-user5 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890 `" && echo ansible-tmp-1554878730.6050982-46540706020890="` echo /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890 `" ) && sleep 0'"'"''
<linux-user5> (0, b'ansible-tmp-1554878730.6050982-46540706020890=/root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890\n', b'')
Using module file /user/user1/p363/lib/python3.6/site-packages/ansible-2.7.6-py3.6.egg/ansible/modules/commands/command.py
<linux-user5> PUT /user/user1/.ansible/tmp/ansible-local-312468ua784d6/tmp9l2vw34v TO /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890/AnsiballZ_command.py
<linux-user5> SSH: EXEC sshpass -d11 sftp -o BatchMode=no -b - -C -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/user/user1/.ansible/cp/0969431db6 '[linux-user5]'
<linux-user5> (0, b'sftp> put /user/user1/.ansible/tmp/ansible-local-312468ua784d6/tmp9l2vw34v /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890/AnsiballZ_command.py\n', b'')
<linux-user5> ESTABLISH SSH CONNECTION FOR USER: root
<linux-user5> SSH: EXEC sshpass -d11 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/user/user1/.ansible/cp/0969431db6 linux-user5 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890/ /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890/AnsiballZ_command.py && sleep 0'"'"''
<linux-user5> (0, b'', b'')
<linux-user5> ESTABLISH SSH CONNECTION FOR USER: root
<linux-user5> SSH: EXEC sshpass -d11 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o User=root -o ConnectTimeout=10 -o ControlPath=/user/user1/.ansible/cp/0969431db6 -tt linux-user5 '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1554878730.6050982-46540706020890/AnsiballZ_command.py && sleep 0'"'"''

Hangs after gathering facts.

This completed fine during first run. Also worked after reinstalling OS on client machine.

Answer 1

Have you tried running your ssh-keygen command twice manually ?

First run:

user@host:~$ ssh-keygen -t rsa -f /tmp/id_rsa -q -P ""
user@host:~$

Second run:

user@host:~$ ssh-keygen -t rsa -f /tmp/id_rsa -q -P ""
/tmp/id_rsa already exists.
Overwrite (y/n)? 

Your command is simply waiting forever an answer that will never come.

Solution 1: answer the question(s)

You could use the expect module to answer the interactive prompts if you know them in advance

- name: ssh keygen
  expect:
    command: ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P ""
    responses:
      Overwrite \(y/n\)\?: y

Although this will work on every run, it's a bad solution as it is not idempotent: there will be a change on your server every time you run your playbook although no changes have been made to your inventory or variables

Solution 2: generate only when needed.

A better approach would be to create the key only if it does not exists or if it needs to be refreshed. There are several ways to do this and it greatly depends on your needs, your wishes, the size of your server park...

With a relatively small park, you could generate the keys locally and store them in your inventory (using ansible-vault to obfuscate the private keys). You then simply have to copy the files to the servers. No changes will be made if the file already exist and has the same content.

If the park is bigger, the keys could be stored in Hashicorp Vault, Cyberark...

If you still want to create the keys on each server, here is my take to do it idem-potently:

- name: remove key files if refresh was asked
  file:
    path: "/root/.ssh/id_rsa{{ item }}"
    state: absent
  loop:
    - ""
    - ".pub"
  when: refresh_keys | default(false) | bool

- name: check if key exists
  stat:
    path: /root/.ssh/id_rsa
  register: ssh_key_file

- name: generate keys if needed
  command: ssh-keygen -t rsa -f /root/.ssh/id_rsa -q -P ""
  when: not (ssh_key_file.stat.exists)
  changed_when: true

Now if your run your playbook normally, the keys will be created only if they don't already exists. If you want to refresh the keys on servers already having one you can pass an extra var to the playbook

ansible-playbook /your/inventory.ini your_playbook.yml -e refresh_keys=true