Busybox execs says libc.so can't be found

Question

I'm making some Busybox/Linux operating system. It works with static busybox version, initramfs and sysroot have the same stuff except the init script. The one from initramfs sets PATH and mount /sys, /proc and /dev and new_root and then switches to it. It works but I wanted to make the main sysroot use shared version of busybox so external executables could use the same libc as busybox, so there is no wasted disk space. There are steps my Makefile does:

1. Fetch sources of linux, busybox, musl and binary i686-linux-musl toolchain.
2. Configure and compile linux, musl and busybox (install initramfs' static one to _initramfs_sysroot and the shared one to _sysroot) with the toolchain.
3. Copy data from busybox/_initramfs_sysroot to initramfs/ and make a working cpio.gz archive from it.
4. Copy data from busybox/_sysroot and copy that to sysroot/. Install musl to sysroot/usr and move sysroot/usr/lib to sysroot/usr/lib32 (because for some reason busybox wants libc to be in here).
5. Make the raw disk image with extlinux on it.
6. It doesn't work.

What I can see: Linux gives me a kernel panic when trying to run switch_root /new_root /etc/rc.d/rc from the initramfs saying it's not present but all I have changed was that busybox is shared now so I mount the filesystem on my host OS and execute ldd /mnt/etc/rc.d/rc and it says that there is no such file `/usr/lib32/libc.so and then I decided to move /usr/lib to /usr/lib32 as I said before. But I got the same error anyways, even if I set LD_LIBRARY_PATH to /usr/lib (libbusybox) and /usr/lib32 (libc) just like PATH - in the initramfs' init script, right before switching root to /new_root.

I discovered that the musl's libc.so is also an executable which can run programs that require it. For example, I succeeded to run /bin/sh (from busybox) as a shared executable through /usr/lib32/libc.so /bin/sh and it works but only for the sh program. I can't run any other program even through the shell, but I can use the libc.so's "Dynamic Program Loader" again (that is the name of the libc's embedded executable or anything, it says that as help instructions).

Answer 1

Understanding the Problem:

Kernel panic: Occurs when the kernel encounters a critical error it can't recover from. Root cause: Inability to locate libc.so during switch_root. Potential factors: Incorrect library paths or environment variables. Missing dependencies in the initramfs or sysroot. Permissions or ownership issues. Configuration errors in Busybox or the kernel. Troubleshooting Steps:

1. Verify Library Paths and Environment Variables: Double-check LD_LIBRARY_PATH and PATH in the initramfs init script. Ensure they point to the correct locations of libc.so and libbusybox.so. Print these variables before switch_root to confirm their values.
2. Inspect Initramfs Contents: Use lsinitramfs to list the files in the initramfs. Verify that libc.so and libbusybox.so (if needed) are present. Check for any missing dependencies using ldd on essential binaries.
3. Examine Shared Library Dependencies: Use ldd /mnt/etc/rc.d/rc on the mounted sysroot to list dependencies. Ensure all required libraries are present in the sysroot or initramfs.
4. Check File Permissions and Ownership: Ensure libc.so and other libraries have appropriate permissions (usually 755). Verify ownership, especially if files are copied from different sources.
5. Consider Configuration Issues: Review Busybox and kernel configuration options related to shared libraries. Ensure they are compatible with your setup.
6. Review Build and Installation Steps: Double-check the Makefile for errors or inconsistencies. Ensure correct installation of Busybox, Musl, and other components.