Verify there's enough memory and storage to compile a file?

Question

I'm testing a C++ library under Solaris 11 with Sun Studio 12.5. The library is trouble free under Sun Studio 12.2 through 12.4, but its having trouble under 12.5.

Under 12.5, the pain point appears when compiling an non-optimized and debug build of file bench.cpp. The tools complain they have run out of space. The output of the failed compile is below.

I've seen this symptom before, but its usually an ARM device with little memory and no swap file.

The machine has dual Xeon's, 8 GB of RAM and two 136 GB disk drives in a RAID 1 config. Used disk space appears to be around 20 GB. I fear I may have mis-configured when installing the system, but I can't spot what it is. The configuration information is below.

My questions are very basic: do I have enough memory and storage to compile a file that taxes the tools? Is something mis-configured?

---

$ prtconf | grep -i memory
Memory size: 8190 Megabytes

$ zpool list
NAME   SIZE  ALLOC  FREE  CAP  DEDUP  HEALTH  ALTROOT
rpool  136G  19.6G  116G  14%  1.00x  ONLINE  -

$ df -h
Filesystem             Size   Used  Available Capacity  Mounted on
rpool/ROOT/solaris     134G    11G       113G    10%    /
/devices                 0K     0K         0K     0%    /devices
/dev                     0K     0K         0K     0%    /dev
ctfs                     0K     0K         0K     0%    /system/contract
proc                     0K     0K         0K     0%    /proc
mnttab                   0K     0K         0K     0%    /etc/mnttab
swap                   6.7G   1.6M       6.7G     1%    /system/volatile
objfs                    0K     0K         0K     0%    /system/object
sharefs                  0K     0K         0K     0%    /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap1.so.1
                       124G    11G       113G    10%    /lib/libc.so.1
fd                       0K     0K         0K     0%    /dev/fd
rpool/ROOT/solaris/var
                       134G   346M       113G     1%    /var
swap                   6.8G   114M       6.7G     2%    /tmp
rpool/VARSHARE         134G   164K       113G     1%    /var/share
rpool/export           134G    32K       113G     1%    /export
rpool/export/home      134G    36K       113G     1%    /export/home
rpool/export/home/apolyakov
                       134G    38K       113G     1%    /export/home/apolyakov
rpool/export/home/jwalton
                       134G   2.6G       113G     3%    /export/home/jwalton
rpool/export/home/mcaswell
                       134G    38K       113G     1%    /export/home/mcaswell
rpool/export/home/pgutmann
                       134G    38K       113G     1%    /export/home/pgutmann
rpool                  134G   4.9M       113G     1%    /rpool
rpool/VARSHARE/zones   134G    31K       113G     1%    /system/zones
rpool/VARSHARE/pkg     134G    32K       113G     1%    /var/share/pkg
rpool/VARSHARE/pkg/repositories
                       134G    31K       113G     1%    /var/share/pkg/repositories

And:

# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <HP-LOGICAL VOLUME-5.26-136.70GB>
          /pci@0,0/pci8086,25e3@3/pci103c,3235@0/sd@0,0
Specify disk (enter its number): 0
selecting c1t0d0
[disk formatted]
/dev/dsk/c1t0d0s1 is part of active ZFS pool rpool. Please see zpool(1M).

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        ...

format> partition

PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        ...
partition> print
Current partition table (original):
Total disk sectors available: 286660669 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector         Size         Last Sector
  0  BIOS_boot    wm               256      256.00MB          524543    
  1        usr    wm            524544      136.44GB          286660702    
  2 unassigned    wm                 0           0               0    
  3 unassigned    wm                 0           0               0    
  4 unassigned    wm                 0           0               0    
  5 unassigned    wm                 0           0               0    
  6 unassigned    wm                 0           0               0    
  8   reserved    wm         286660703        8.00MB          286677086   

---

$ cat bench-compile.txt
...
/opt/developerstudio12.5/bin/CC -DDEBUG -g3 -xO0 -native -m64 -KPIC -template=no%extdef -c bench2.cpp

DBG_GEN FATAL ERROR: dbg_tables.c:171 - fwrite() failed to write required bytes [DBG_GEN 5.6.3]

CC: Fatal error in /opt/developerstudio12.5/lib/compilers/bin/previse
CC: Status 134
gmake: *** [test.o] Error 134
gmake: *** Waiting for unfinished jobs....
Assembler: bench2.cpp
    "<null>", line 821548 : Trouble writing; probably out of temporary file space
/opt/developerstudio12.5/lib/compilers/sys/amd64/libsunir.so'ir_proc_write+0x5d [0xffff80ffbb0eeffd]
/opt/developerstudio12.5/lib/compilers/sys/amd64/libsunir.so'ir_mod_write_and_close+0x2f [0xffff80ffbb10633f]
/opt/developerstudio12.5/lib/compilers/bin/iropt'0x3fee3f [0x7fee3f]
/opt/developerstudio12.5/lib/compilers/bin/iropt'main+0x567 [0x8058f7]
/opt/developerstudio12.5/lib/compilers/bin/iropt'0x12d324 [0x52d324]
compiler(iropt) error:  libsunir File IO (read / write IR) error using /tmp/iropt.1468062981.23519.05.ir (No space left on device).
CC: cannot copy temporaries: /tmp/previse.1468062981.23519.07.err
CC: Fatal error in /opt/developerstudio12.5/lib/compilers/bin/previse
CC: Status 134
gmake: *** [bench2.o] Error 134
ERROR: failed to make cryptest.exe

$ 

Answer 1

According to the error message, your system is lacking virtual memory. This can happen even while having enough disk space and enough RAM.

I would start by creating a second zvol, say a 8 GB one, and add it as a swap device. e.g.:

zfs create -V 8gb rpool/swap1
swap -a /dev/zvol/dsk/rpool/swap1
echo "/dev/zvol/dsk/rpool/swap1 - - swap - no -" >> /etc/vfstab

Monitoring the virtual memory usage can be done with the swap -s command.

Answer 2

There are two similar problems that can happen when building programs on Solaris with Studio compilers.

1) /tmp space runs out

The symptom above shows that this was happening for you. Since the /tmp directory is allocated out of virtual memory (memory-based file system) the answer is to add swap space, as shown above.

2) malloc returns a failure code

If the system is unable to find heap space (malloc) in a running process, one of the common error modes will be a message about "fork failed" which happens when the OS cannot allocate enough virtual memory to make a new process image. This message will come from the OS, not the compiler.

The Solaris kernel will sometimes return a malloc() failure to a user program even if there is plenty of swap space assigned to the system. This happens when the request for memory times out. If the kernel can't find available space (by swapping out memory to disk for example) within a reasonable time, it will return a failure.

This will be confusing because it looks like there is no more swap, but it's just a timing issue.

In summary:

Running out of VM via the /tmp directory has different symptoms and different behavior than running out of VM via the malloc() call. So keep this in mind.