Mysql error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)'

Question

I had a space issue in my server, so I added more space and now MySQL won't start.

trying to start mysql:

root@vps:~# /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.service
Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.

Checking the error with systemctl status mysql.service:

root@vps412690:~# systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: failed (Result: exit-code) since Thu 2018-11-08 09:33:42 CET; 11s ago
  Process: 21650 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Nov 08 09:33:42 vps mysql[21650]: .Warning: World-writable config file '/etc/mysql/my.cnf' is ignored
Nov 08 09:33:42 vps /etc/init.d/mysql[22203]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Nov 08 09:33:42 vps /etc/init.d/mysql[22203]: [61B blob data]
Nov 08 09:33:42 vps /etc/init.d/mysql[22203]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)'
Nov 08 09:33:42 vps /etc/init.d/mysql[22203]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Nov 08 09:33:42 vps /etc/init.d/mysql[22203]:
Nov 08 09:33:42 vps mysql[21650]: failed!
Nov 08 09:33:42 vps systemd[1]: mysql.service: control process exited, code=exited status=1
Nov 08 09:33:42 vps systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Nov 08 09:33:42 vps systemd[1]: Unit mysql.service entered failed state.

If I'm trying to apt-get (maybe i'm missing some file or whatever), it doesn't work:

Errors were encountered while processing:
 mysql-server-5.5
 mysql-server
 nginx-full
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

Found interesting log in my var/log/mysql/error.log file:

181108 10:05:31 InnoDB: The InnoDB memory heap is disabled
181108 10:05:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins
181108 10:05:31 InnoDB: Compressed tables use zlib 1.2.8
181108 10:05:31 InnoDB: Using Linux native AIO
181108 10:05:31 InnoDB: Initializing buffer pool, size = 128.0M
181108 10:05:31 InnoDB: Completed initialization of buffer pool
181108 10:05:31 InnoDB: highest supported file format is Barracuda.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is 5F884400
InnoDB: Cleaning up trx with id 5F87FA74
181108 10:05:31  InnoDB: Waiting for the background threads to start
181108 10:05:32 InnoDB: 5.5.62 started; log sequence number 3969391541705
181108 10:05:32 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
181108 10:05:32 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
181108 10:05:32 [Note] Server socket created on IP: '0.0.0.0'.
181108 10:05:32  InnoDB: Assertion failure in thread 140603552634624 in file trx0purge.c line 843
InnoDB: Failing assertion: purge_sys->purge_trx_no <= purge_sys->rseg->last_trx_no
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:05:32 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346701 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0x55b6743790f3]
/usr/sbin/mysqld(handle_fatal_signal+0x3e4)[0x55b674264b14]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7fe0e913c890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fe0e7b32067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe0e7b33448]
/usr/sbin/mysqld(+0x5944d5)[0x55b6744244d5]
/usr/sbin/mysqld(+0x59501e)[0x55b67442501e]
/usr/sbin/mysqld(+0x662d92)[0x55b6744f2d92]
/usr/sbin/mysqld(+0x658fb8)[0x55b6744e8fb8]
/usr/sbin/mysqld(+0x596ec5)[0x55b674426ec5]
/usr/sbin/mysqld(+0x587f2c)[0x55b674417f2c]
/usr/sbin/mysqld(+0x58c723)[0x55b67441c723]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7fe0e9135064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe0e7be562d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

df -h /var/lib/mysql

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G   19G   19G  51% /

Not sure where to start debugging this issue, any clue?

Answer 1

This answer expands on the discussion in the comments section:

To make sure your InnoDB files are healthy, you really need to recreate them - it could really bite you later, if you don't do this now. To do so:

• Make a complete DB level backup (including all users, stored procedures, ...)
• Since you now have enough space, use something along the lines of tar -cjf /var/libbackup-mysql.tar.bz2 /var/lib/mysql to create a file-level backup (just in case ...)
• Stop your database server
• Remove everything from /var/lib/mysql
• Start the database server again, this will take some time and recreate an empty InnoDB structure
• Reimport the DB level backup

If this doesn't give you back all your data, you can use the file level backup to start over.

Answer 2

This could have litterally hundred of explanations but the first line of systemctl status is very likely to be a useful hint:

Nov 08 09:33:42 vps mysql[21650]: .Warning: World-writable config file '/etc/mysql/my.cnf' is ignored

If your configuration file isn't read there will be weird issues indeed.

Also did you try connecting to the mysql forcefully on the IP address which seem to be binded and not the unix socket ?

Did you read the recommendation about recovery also:

InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.