So here is the answer, initially posted by me on r/vmware on reddit but figured it's better to post it here too:
TLDR;
Here are is a simple fix to the issue on Ubuntu 19 / 20 or any other system where attempts to access shared virtual machines is crashing (i.e. /var/lib/vmware/bin/hostd) is mysteriously crashing.
cp /etc/pam.d/vmware-authd ~/vmware-authd.backup
sed -e '/pam_cap/s/^.*$/# -- pam_cap does not work for multithreaded apps -- /' /etc/pam.d/common-auth | sudo tee /etc/pam.d/common-auth-mt
sudo -i -e 's/common-auth/common-auth-mt/' /etc/pam.d/vmware-authd
Details on why this works,
The culprit is then entry in /etc/pam.d/common-auth that includes pam_cap.so. It turns out /usr/lib/vmware/bin/hostd is a MULTITHREADED app and pam_cap.so does NOT work with them:
DESCRIPTION
The pam_cap PAM module sets the current process' inheritable capabilities.
Capabilities are read from the /etc/security/capability.conf config file, or alternate file specified with the config= option.
The module must not be called by a multithreaded application.
(source: pam_cap(8) man page)
and it turns out hostd is multithreaded :P
So we create a "multithreaded" version of all the "common-*" files (it turns out that it was only common-auth) and use that for our multithreaded apps.
Of course you can do what TLDR; above did by hand as under:
❯ /bin/cat /etc/pam.d/vmware-authd
#%PAM-1.0
auth include common-auth-mt
account include common-account
password include common-password
session include common-session
and
❯ cat common-auth-mt
#
# /etc/pam.d/common-auth - authentication settings common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of the authentication modules that define
# the central authentication scheme for use on the system
# (e.g., /etc/shadow, LDAP, Kerberos, etc.). The default is to use the
# traditional Unix authentication mechanisms.
#
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules. See
# pam-auth-update(8) for details.
# here are the per-package modules (the "Primary" block)
auth [success=1 default=ignore] pam_unix.so nullok_secure
# here's the fallback if no module succeeds
auth requisite pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
auth required pam_permit.so
# and here are more per-package modules (the "Additional" block)
# -- pam_cap does not work for multithreaded apps --
# end of pam-auth-update config
Hopefully this is clear enough.
More info
BTW this also works for mysql mariadb and other multithreaded app auth issues by checking if their pam modules somehow include pam_cap.
