Can someone help me to analyze these valgrind output?

Question

I wrote a C program on linux to process large amount of data, it run OK most of the time. But when processing a set of particular data it always told me the 'segmentation fault' error, I tried to use gdb without luck (see my another question How to deal segmentation fault with GDB under particular circumstance?), so I tried to use valgrind. It didn't tell me the 'segmentation fault' error when run with valgrind. So how to find the bug according to these valgrind output?

==2441== Invalid write of size 4
==2441==    at 0x404893: nodes_term32_flush (tyn_indexer.c:227)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid write of size 4
==2441==    at 0x4048D8: nodes_term32_flush (tyn_indexer.c:254)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x40450D: nodes_term32_flush (tyn_indexer.c:260)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x404878: nodes_term32_flush (tyn_indexer.c:224)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid write of size 4
==2441==    at 0x40487F: nodes_term32_flush (tyn_indexer.c:224)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x4048BD: nodes_term32_flush (tyn_indexer.c:251)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c094 is not stack'd, malloc'd or (recently) free'd
==2441==
==2441== Invalid write of size 4
==2441==    at 0x4048C4: nodes_term32_flush (tyn_indexer.c:251)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c094 is not stack'd, malloc'd or (recently) free'd
==2441==
==2441== Invalid read of size 4
==2441==    at 0x4093A0: tyn_p4d_encode32 (tyn_coder.c:645)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x409490: tyn_p4d_encode32 (tyn_coder.c:669)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x4094A7: tyn_p4d_encode32 (tyn_coder.c:667)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c088 is 8 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x409574: tyn_p4d_encode32 (tyn_coder.c:694)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x4095A6: tyn_p4d_encode32 (tyn_coder.c:708)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c088 is 8 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x409524: tyn_p4d_encode32 (tyn_coder.c:697)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c08c is 12 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x40953A: tyn_p4d_encode32 (tyn_coder.c:700)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1233c08c is 12 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x409552: tyn_p4d_encode32 (tyn_coder.c:702)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1243c2fc is 12 bytes after a block of size 16 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x40785B: tyn_exsorter_sort (tyn_exsorter.c:106)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x407CB7: tyn_iS16_encode32 (tyn_coder.c:96)
==2441==    by 0x409621: tyn_p4d_encode32 (tyn_coder.c:725)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1243d880 is 0 bytes after a block of size 16 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x40785B: tyn_exsorter_sort (tyn_exsorter.c:106)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x407D20: tyn_iS16_encode32 (tyn_coder.c:109)
==2441==    by 0x409621: tyn_p4d_encode32 (tyn_coder.c:725)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1243d880 is 0 bytes after a block of size 16 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x40785B: tyn_exsorter_sort (tyn_exsorter.c:106)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Invalid read of size 4
==2441==    at 0x407D37: tyn_iS16_encode32 (tyn_coder.c:108)
==2441==    by 0x409621: tyn_p4d_encode32 (tyn_coder.c:725)
==2441==    by 0x4045F0: nodes_term32_flush (tyn_indexer.c:132)
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==  Address 0x1243d8a8 is not stack'd, malloc'd or (recently) free'd
==2441==

==2441== Use of uninitialised value of size 8
==2441==    at 0x3AE9C4726B: _itoa_word (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C49852: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
==2441==    by 0x4071EF: tyn_build_index (tyn_indexer.c:888)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Conditional jump or move depends on uninitialised value(s)
==2441==    at 0x3AE9C47275: _itoa_word (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C49852: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
==2441==    by 0x4071EF: tyn_build_index (tyn_indexer.c:888)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
document id: 0
==2441== Conditional jump or move depends on uninitialised value(s)
==2441==    at 0x3AE9C4774E: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
==2441==    by 0x4073AD: tyn_build_index (tyn_indexer.c:900)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Use of uninitialised value of size 8
==2441==    at 0x3AE9C4726B: _itoa_word (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C49852: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
==2441==    by 0x4073AD: tyn_build_index (tyn_indexer.c:900)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
==2441== Conditional jump or move depends on uninitialised value(s)
==2441==    at 0x3AE9C47275: _itoa_word (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C49852: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
==2441==    by 0x4073AD: tyn_build_index (tyn_indexer.c:900)
==2441==    by 0x40384F: main (tyn_indexer.c:943)
==2441==
int32_t category: 0
bytes_collected: 972467429
196220 bytes/s
tyn_config->indexer->dl_directory: /dragon/tyan
tyn_config->indexer->memory_limit: 10240000
==2441==
==2441== HEAP SUMMARY:
==2441==     in use at exit: 214,695,668 bytes in 399,272 blocks
==2441==   total heap usage: 401,095 allocs, 1,823 frees, 219,225,806 bytes allocated
==2441==
==2441== LEAK SUMMARY:
==2441==    definitely lost: 9,442,580 bytes in 27 blocks
==2441==    indirectly lost: 34,682,771 bytes in 81 blocks
==2441==      possibly lost: 170,557,809 bytes in 399,140 blocks
==2441==    still reachable: 12,508 bytes in 24 blocks
==2441==         suppressed: 0 bytes in 0 blocks
==2441== Rerun with --leak-check=full to see details of leaked memory
==2441==
==2441== For counts of detected and suppressed errors, rerun with: -v
==2441== Use --track-origins=yes to see where uninitialised values come from
==2441== ERROR SUMMARY: 32680 errors from 23 contexts (suppressed: 2 from 2)

0xC0000022L · Accepted Answer · 2012-03-23 03:30:22Z

That's a lot of output you have here. You should have condensed it more to the areas you are interested. I'll try to give a few general pointers, though.

==2441== Invalid write of size 4

opens a "block" that tells you details about the allocated memory and the stack backtrace of its allocation and the error site. The ==2441== is the PID of the process, which is useful in cases where multiple processes run in parallel.

Error site (comments added with #). The stack backtrace is always in reverse order, meaning the crash (or what would have become a crash normally) site is at the top of the listing and the calls that led there are listed in reverse order below:

# This points to a function nodes_term32_flush() in file tyn_indexer.c on line 227
==2441==    at 0x404893: nodes_term32_flush (tyn_indexer.c:227)
# the format is the same, at this line tyn_exsorter_sort() calls nodes_term32_flush()
==2441==    by 0x407B77: tyn_exsorter_sort (tyn_exsorter.c:131)
# ... and so on
==2441==    by 0x406DDE: tyn_build_index (tyn_indexer.c:731)
# ... leading up to the process "entry point"
==2441==    by 0x40384F: main (tyn_indexer.c:943)

The memory block that was affected. The first line tells us that a block of exactly one MiB was allocated and that (together with the output above) you were reading the first 4 bytes (likely a 32bit value) after the last allocated byte of that block. The rest of the format follows what you know from the stack backtrace above.

==2441==  Address 0x1233c080 is 0 bytes after a block of size 1,048,576 alloc'd
==2441==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==2441==    by 0x406BEB: tyn_build_index (tyn_indexer.c:663)
==2441==    by 0x40384F: main (tyn_indexer.c:943)

All the repetitions afterwards are - from experience - most often a consequence of the first error. So always start fixing the first listed issue in a sequence of reported issues.

Now for the other error class that appears in your output:

==2441== Use of uninitialised value of size 8
# All library functions here ...
==2441==    at 0x3AE9C4726B: _itoa_word (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C49852: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
# ... but this one should be yours. Check out this file/line to see what can be the problem with the printf() call
==2441==    by 0x4071EF: tyn_build_index (tyn_indexer.c:888)
==2441==    by 0x40384F: main (tyn_indexer.c:943)

The following is likely again just a consequence of the output above:

==2441== Conditional jump or move depends on uninitialised value(s)
==2441==    at 0x3AE9C47275: _itoa_word (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C49852: vfprintf (in /lib64/libc-2.14.90.so)
==2441==    by 0x3AE9C51FE8: printf (in /lib64/libc-2.14.90.so)
==2441==    by 0x4071EF: tyn_build_index (tyn_indexer.c:888)
==2441==    by 0x40384F: main (tyn_indexer.c:943)

I strongly suggest two things: read the Valgrind manual (I know this sounds patronizing, but it's worth it) and make use of its many options. I created a few functions that I am using on my development machines:

# vim: set autoindent smartindent tabstop=2 shiftwidth=2 expandtab filetype=sh:

function vgrun
{
  local COMMAND="$1"
  local NAME="$2"
  [[ -n "$COMMAND" ]] || { echo "Syntax: vgrun <command> <name>"; return; }
  [[ -n "$NAME" ]] || { echo "Syntax vgrun <command> <name>"; return; }
  valgrind \
        --leak-check=full --error-limit=no --track-origins=yes \
        --undef-value-errors=yes --log-file=valgrind-${NAME}.log \
        --read-var-info=yes \
        $COMMAND | tee valgrind-${NAME}-output.log 2>&1
}

function vgtrace
{
  local COMMAND="$1"
  local NAME="$2"
  [[ -n "$COMMAND" ]] || { echo "Syntax: vgtrace <command> <name>"; return; }
  [[ -n "$NAME" ]] || { echo "Syntax vgtrace <command> <name>"; return; }
  valgrind \
        --leak-check=full --error-limit=no --track-origins=yes \
        --undef-value-errors=yes --log-file=valgrind-${NAME}.log \
        --read-var-info=yes --trace-children=yes \
        $COMMAND | tee valgrind-${NAME}-output.log 2>&1
}

function vgdbg
{
  [[ -n "$*" ]] || { echo "Syntax: vgrun <command>"; return; }
  valgrind \
        --leak-check=full --error-limit=no --track-origins=yes \
        --undef-value-errors=yes --read-var-info=yes --db-attach=yes \
        "$@"
}

vgrun simply runs a command with Valgrind. Note that the command needs to be in quotes with all its parameters for this one to work. vgtrace is merely a variation of vgrun, adding --trace-children=yes to the Valgrind command line. The most useful, by far, is vgdbg, which will ask you to attach GDB to the running program and thus lets you interactively debug the issue at hand, including properly inspecting stack frames, values etc - if you speak GDB, that is.

and a +1 back for taking the time to dig deeper into the data. I was really going for the general approach since there was a lot of info there and I haven't used valgrind in quite a long time. Just over time debugging segfaults I find uninitialized variables tend to lead in that direction, and of course overwriting a buffer can do all sorts of fun things. — pstrjds, Commented Mar 23, 2012 at 4:39

pstrjds · Accepted Answer · 2012-03-23 03:08:55Z

2

I would take a look at the errors talking about uninitialized values first. I would also look at the invalid writes. Segfaults can often be a result of using an uninitialized value. They can also be caused by invalid writes that cause heap corruption. The output there is giving you line numbers. Start looking in those areas. The output there is also showing you several memory leaks. Follow the output advice and run it again with --leak-check=full to get more detailed analysis of the memory leaks so you can track those down and take care of them.

answered Mar 23, 2012 at 3:08

pstrjds

17.2k6 gold badges52 silver badges61 bronze badges

Well, there need not be any such errors (uninitialized) in order to get the output of the OP. And those below the actual error often tend to be a consequence of the error in the first place. Though admittedly it also happens the other way around. Still +1 for the general direction.
– 0xC0000022L
Commented Mar 23, 2012 at 3:10
Thank you for your answer and I am gonna do a carefully check
– Mickey Shine
Commented Mar 23, 2012 at 3:14

Add a comment |

Collectives™ on Stack Overflow

Can someone help me to analyze these valgrind output?

2 Answers 2

Not the answer you're looking for? Browse other questions tagged
c
linux
debugging
valgrind
or ask your own question.

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Not the answer you're looking for? Browse other questions tagged clinuxdebuggingvalgrind or ask your own question.

Linked

Related

Not the answer you're looking for? Browse other questions tagged
c
linux
debugging
valgrind
or ask your own question.