How to Find and Fix Memory Leaks in Linux Applications

Detecting a memory leak is the easy part. Running one command tells you that bytes went missing, and then the real work starts: deciding which frame in the stack is the line you should actually change, whether the other records are separate bugs or fallout from the same one, and what to do when the numbers climb but every tool says the program is clean.

This guide walks that whole path on one small leaky C program. Each tool section shows the report, points at the frame that matters, and lists the options that make the report say more. The last sections cover the two hard cases: memory that no detector calls a leak because it is still reachable, and a long-lived service that is growing right now and cannot be restarted.

In containers or Kubernetes, rising cgroup usage is not always a heap leak. Page cache, tmpfs, and the limit itself can dominate before the heap looks wrong, so rule out container memory limits before you treat a graph spike as a bug in application code.

Quick answer

How to read a leak report

Every leak report is a list of allocation stacks. That is the first thing to get straight, because the stack tells you where the memory was born, not where the bug lives. Valgrind can show where a leaked block was allocated, but not when, how, or why the pointer was lost. You still have to decide ownership from the program logic.

Use this workflow to narrow the line to change:

• Ignore runtime allocator frames such as malloc inside libasan, liblsan, or Valgrind.
• Find the first useful frame in your own code. That is the allocation site.
• Walk upward through callers to determine which function owns that allocation and should eventually release it. Sometimes the allocation site itself is wrong; other times the missing release belongs to a caller higher on the stack.

Valgrind sorts unfreed memory into kinds, and the kind changes what you should do about it.

LeakSanitizer uses a shorter vocabulary. A direct leak matches definitely lost, an indirect leak matches indirectly lost, and it says nothing at all about still reachable memory. That difference matters later, because some of the hardest production leaks live in that blind spot.

Build a sample program that leaks four different ways

A one-line leak teaches nothing about reading reports, so this program leaks in four shapes at once. Create memleak.c in a working directory such as ~/memleak-lab/:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

struct row {
    char *key;
    char *value;
};

struct table {
    struct row *rows;
    size_t count;
};

static char *dup_field(const char *text)
{
    char *copy = malloc(strlen(text) + 1);
    strcpy(copy, text);
    return copy;
}

static struct table *build_table(size_t count)
{
    struct table *t = malloc(sizeof *t);

    t->rows = malloc(count * sizeof *t->rows);
    t->count = count;

    for (size_t i = 0; i < count; i++) {
        char key[32];
        snprintf(key, sizeof key, "key%zu", i);
        t->rows[i].key = dup_field(key);
        t->rows[i].value = dup_field("value");
    }
    return t;
}

static char *cache;
static char *interior;

int main(void)
{
    struct table *t = build_table(3);
    printf("loaded %zu rows\n", t->count);

    cache = dup_field("still referenced at exit");

    char *buf = malloc(64);
    interior = buf + 8;

    /* t is never freed */
    return 0;
}

Four things happen here, and each one produces a different line in the reports below. The struct table from line 24 is dropped without a free, so it is a real leak. The rows array and the six strings behind it were only reachable through that table, so they are collateral damage. The cache pointer at line 46 is never freed either, but a global still points at it. The block from line 48 survives only as buf + 8, which is a pointer into the middle of the allocation.

Compile a debug binary you will reuse in later sections:

gcc -g -O0 -o memleak memleak.c

The -g flag keeps the line numbers that make every report readable, and -O0 stops the compiler from inlining build_table into main or discarding allocations it considers dead. Both matter more than they sound; an optimized build is the most common reason a report points at the wrong function.

Run it once to confirm it behaves normally:

./memleak

loaded 3 rows

The program prints one line and exits with status 0. Nothing about the exit tells you that 97 bytes went missing, which is the whole problem with leaks.

Find the leaking line with LeakSanitizer

LeakSanitizer ships with GCC and Clang as part of AddressSanitizer. When I control the build this is the first tool I reach for, because it costs roughly 2× runtime instead of Valgrind's 10× to 50×, and it prints the source line directly.

Install the compiler and the sanitizer runtime library:

sudo dnf install gcc libasan -y

On Debian and Ubuntu the runtime comes from libasan8 (or the version matching your GCC), and Clang users get it from compiler-rt. Build the same source with the sanitizer enabled:

gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -o memleak-asan memleak.c

The frame pointer flag is not cosmetic. LeakSanitizer unwinds allocation stacks cheaply by walking frame pointers, so omitting them costs you the caller frames that identify the owner. Run the instrumented binary:

./memleak-asan

=================================================================
==449002==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f4323cfc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x40129d in build_table /root/memleak-lab/memleak.c:24
    #2 0x4014ef in main /root/memleak-lab/memleak.c:43
    #3 0x7f4323a5158d in __libc_start_call_main (/lib64/libc.so.6+0x2a58d)

Indirect leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f4323cfc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4012b7 in build_table /root/memleak-lab/memleak.c:26
    #2 0x4014ef in main /root/memleak-lab/memleak.c:43

Indirect leak of 18 byte(s) in 3 object(s) allocated from:
    #0 0x7f4323cfc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4011f9 in dup_field /root/memleak-lab/memleak.c:17
    #2 0x40141e in build_table /root/memleak-lab/memleak.c:33
    #3 0x4014ef in main /root/memleak-lab/memleak.c:43

Indirect leak of 15 byte(s) in 3 object(s) allocated from:
    #0 0x7f4323cfc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4011f9 in dup_field /root/memleak-lab/memleak.c:17
    #2 0x4013b4 in build_table /root/memleak-lab/memleak.c:32
    #3 0x4014ef in main /root/memleak-lab/memleak.c:43

SUMMARY: AddressSanitizer: 97 byte(s) leaked in 8 allocation(s).

Four records for one bug. The single direct leak is the entry point: build_table at line 24 allocated the struct table, main at line 43 received it, and nobody freed it. Everything else in the report was owned by that table, which is why the three indirect records point at the rows array and at dup_field.

Fix the direct leak, not the biggest one

The instinct is to jump on dup_field, because it appears in two records and looks like the sloppy function. That would be wrong. dup_field did its job correctly and returned ownership to its caller, and adding a free there would hand back memory the table still uses.

Read the records in this order and the choice becomes mechanical:

• Direct leaks first, because each one is a genuine missing release.
• Indirect leaks second, and usually only to understand what the direct leak was holding.
• Anything left after the direct leaks are fixed, because leaks hidden behind a parent are promoted to direct on the next run.

In this program that means one place to change: whoever called build_table must release the table. One fix retires all four records and all 97 bytes.

Options that make the sanitizer report say more

The sanitizer reads its settings from two environment variables at startup, so you can change behaviour without rebuilding. ASAN_OPTIONS covers the whole sanitizer and LSAN_OPTIONS covers the leak checker.

The context size is the one people set backwards, so it is worth seeing. Shorten the stacks to two frames and run the same binary:

ASAN_OPTIONS=malloc_context_size=2 ./memleak-asan

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f1cd5afc6c0 in malloc (/lib64/libasan.so.8+0xfc6c0)
    #1 0x40129d in build_table /root/memleak-lab/memleak.c:24

Indirect leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7f1cd5afc6c0 in malloc (/lib64/libasan.so.8+0xfc6c0)
    #1 0x4012b7 in build_table /root/memleak-lab/memleak.c:26

Indirect leak of 33 byte(s) in 6 object(s) allocated from:
    #0 0x7f1cd5afc6c0 in malloc (/lib64/libasan.so.8+0xfc6c0)
    #1 0x4011f9 in dup_field /root/memleak-lab/memleak.c:17

SUMMARY: AddressSanitizer: 97 byte(s) leaked in 8 allocation(s).

Four records collapsed into three. The two dup_field records merged into one 33-byte entry because the frame that told them apart, the caller line inside build_table, is gone. In a real codebase that is how a shared allocator ends up as one giant meaningless record: the leak is reported, but the caller who leaked is invisible.

Leaked object addresses are useful when you want to inspect the contents rather than the stack. Ask the leak checker to list them:

LSAN_OPTIONS=report_objects=1 ./memleak-asan

Indirect leak of 18 byte(s) in 3 object(s) allocated from:
    #0 0x7fd62c4fc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4011f9 in dup_field /root/memleak-lab/memleak.c:17
    #2 0x40141e in build_table /root/memleak-lab/memleak.c:33
Objects leaked above:
0x502000000050 (6 bytes)
0x502000000090 (6 bytes)
0x5020000000d0 (6 bytes)

Three separate 6-byte objects, each one a leaked "value" string. Those addresses can be dumped in a debugger when the record alone does not tell you which data was abandoned.

Why an optimized build points at the wrong function

Sanitizer output at -O2 reads differently, and it trips people who only test release builds. Compile with optimization and no frame pointers, then run it:

gcc -g -O2 -fsanitize=address -fomit-frame-pointer -o memleak-asan-opt memleak.c

The build itself prints nothing. Run the optimized binary and compare the stack against the -O0 report above:

./memleak-asan-opt

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7f7058efc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x401152 in build_table /root/memleak-lab/memleak.c:24
    #2 0x401152 in main /root/memleak-lab/memleak.c:43

Both frames report the same address, 0x401152, because GCC inlined build_table into main. The line numbers still work here, but in heavier code inlining merges several call sites into one and the frame you want disappears. When a report looks impossible, rebuild the suspect file at -O0 or -O1 with -fno-inline before you doubt the tool.

The full unwinder recovers depth that the frame-pointer walk misses. Run the same optimized binary through it:

ASAN_OPTIONS=fast_unwind_on_malloc=0 ./memleak-asan-opt

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7facbf0fc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x401152 in build_table /root/memleak-lab/memleak.c:24
    #2 0x401152 in main /root/memleak-lab/memleak.c:43
    #3 0x7facbee5158d in __libc_start_call_main (/lib64/libc.so.6+0x2a58d)
    #4 0x7facbee51648 in __libc_start_main_alias_2 (/lib64/libc.so.6+0x2a648)
    #5 0x401364 in _start (/root/memleak-lab/memleak-asan-opt+0x401364)

The stack now runs all the way to _start. Allocation-heavy programs slow down noticeably with this setting, so turn it on for one diagnostic run rather than leaving it in a test harness.

Hide known third-party leaks with a suppression file

Libraries leak on purpose. A one-time global cache inside a library will be reported on every run, and after the third time it stops being information and starts hiding your bug. Write a suppression file that matches the function by name:

printf 'leak:dup_field\n' > lsan.supp

The pattern is a substring match against function names in the allocation stack, so leak:dup_field covers every record that passes through it. Point the leak checker at the file and ask it to account for what it hid:

LSAN_OPTIONS=suppressions=./lsan.supp:print_suppressions=1 ./memleak-asan

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7fe9884fc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x40129d in build_table /root/memleak-lab/memleak.c:24
    #2 0x4014ef in main /root/memleak-lab/memleak.c:43

Indirect leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7fe9884fc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4012b7 in build_table /root/memleak-lab/memleak.c:26
    #2 0x4014ef in main /root/memleak-lab/memleak.c:43

-----------------------------------------------------
Suppressions used:
  count      bytes template
      6         33 dup_field
-----------------------------------------------------

SUMMARY: AddressSanitizer: 64 byte(s) leaked in 2 allocation(s).

Six blocks and 33 bytes were hidden, and the summary dropped from 97 to 64 bytes. Always keep print_suppressions=1 on in shared test suites. A silent suppression list is how a genuine regression stays invisible for months.

Build a leak-only binary when full ASan is too heavy

AddressSanitizer also instruments every load and store to catch overflows and use-after-free, which is why it needs so much memory. If you only want leak detection, link the standalone runtime instead:

sudo dnf install liblsan -y

The package provides liblsan.so, which -fsanitize=leak links against. Build the leak-only binary:

gcc -g -O0 -fsanitize=leak -o memleak-lsan memleak.c

There is no shadow memory and no red zones in this build, so it runs close to normal speed. Run it and the report format is identical:

./memleak-lsan

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x7ff801817cf2 in malloc (/lib64/liblsan.so.0+0x17cf2)
    #1 0x4011bd in build_table /root/memleak-lab/memleak.c:24
    #2 0x401281 in main /root/memleak-lab/memleak.c:43

Same record, same line, from liblsan instead of libasan. This is the build to use when a service is too memory hungry for full AddressSanitizer but you still want leak checking in a staging run.

Ask for a leak report in the middle of a run

A server that never exits never reaches the exit-time leak check. The sanitizer exposes an entry point you can call yourself, which turns leak detection into something you trigger at a chosen moment. Save this as leak-probe.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sanitizer/lsan_interface.h>

static char *stash;

static void *work(unsigned n)
{
    char *buf = malloc(4096);
    snprintf(buf, 4096, "request %u", n);
    if (n % 3 == 0) {
        stash = buf;
    }
    return buf;
}

int main(void)
{
    for (unsigned n = 0; n < 6; n++) {
        work(n);
        if (n == 2) {
            printf("mid-run leak check:\n");
            fflush(stdout);
            __lsan_do_recoverable_leak_check();
        }
    }
    return 0;
}

The recoverable variant performs a leak check and returns so the process can continue. Use it when you want repeated snapshots during a long-running process, for example from an admin or debug endpoint or another controlled code path. Do not call it directly from an async signal handler; sanitizer internals are not something you should assume are async-signal-safe. If you want signal-triggered checks, have the handler set a flag and let normal application code invoke the leak check later. Build it with the sanitizer:

gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -o leak-probe leak-probe.c

Nothing prints from the build. Run it and watch for two reports instead of one:

./leak-probe

mid-run leak check:

=================================================================
==450252==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8192 byte(s) in 2 object(s) allocated from:
    #0 0x7ff4cacfc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4011ca in work /root/memleak-lab/leak-probe.c:11
    #2 0x40123e in main /root/memleak-lab/leak-probe.c:22

SUMMARY: AddressSanitizer: 8192 byte(s) leaked in 2 allocation(s).

=================================================================
==450252==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 20480 byte(s) in 5 object(s) allocated from:
    #0 0x7ff4cacfc667 in malloc (/lib64/libasan.so.8+0xfc667)
    #1 0x4011ca in work /root/memleak-lab/leak-probe.c:11
    #2 0x40123e in main /root/memleak-lab/leak-probe.c:22

SUMMARY: AddressSanitizer: 20480 byte(s) leaked in 5 allocation(s).

The mid-run check found 8192 bytes and the exit check found 20480 from the same line. That growth between two snapshots is the signal you want from a long-running process, because a single number proves nothing while a rising number at a fixed allocation site proves a great deal.

Find the leaking line with Valgrind Memcheck

Valgrind needs no special build. It runs the program under an instrumentation layer, intercepts every allocation, and scans memory at exit to see what is still referenced. That costs 10× to 50× runtime, and buys you two things the sanitizer cannot give: it works on a binary you cannot rebuild, and it classifies unfreed memory into kinds instead of only reporting what is lost.

Install it from your distribution repositories:

sudo dnf install valgrind -y

The package also brings Massif and DHAT, which the growth sections use later. Run Memcheck against the plain debug binary and ask for every kind of record:

valgrind --leak-check=full --show-leak-kinds=all ./memleak

==449206== HEAP SUMMARY:
==449206==     in use at exit: 186 bytes in 10 blocks
==449206==   total heap usage: 11 allocs, 1 frees, 4,282 bytes allocated
==449206== 
==449206== 15 bytes in 3 blocks are indirectly lost in loss record 1 of 6
==449206==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449206==    by 0x401189: dup_field (memleak.c:17)
==449206==    by 0x401232: build_table (memleak.c:32)
==449206==    by 0x401281: main (memleak.c:43)
==449206== 
==449206== 25 bytes in 1 blocks are still reachable in loss record 3 of 6
==449206==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449206==    by 0x401189: dup_field (memleak.c:17)
==449206==    by 0x4012A9: main (memleak.c:46)
==449206== 
==449206== 48 bytes in 1 blocks are indirectly lost in loss record 4 of 6
==449206==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449206==    by 0x4011D1: build_table (memleak.c:26)
==449206==    by 0x401281: main (memleak.c:43)
==449206== 
==449206== 64 bytes in 1 blocks are possibly lost in loss record 5 of 6
==449206==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449206==    by 0x4012BA: main (memleak.c:48)
==449206== 
==449206== 97 (16 direct, 81 indirect) bytes in 1 blocks are definitely lost in loss record 6 of 6
==449206==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449206==    by 0x4011BD: build_table (memleak.c:24)
==449206==    by 0x401281: main (memleak.c:43)
==449206== 
==449206== LEAK SUMMARY:
==449206==    definitely lost: 16 bytes in 1 blocks
==449206==    indirectly lost: 81 bytes in 7 blocks
==449206==      possibly lost: 64 bytes in 1 blocks
==449206==    still reachable: 25 bytes in 1 blocks
==449206==         suppressed: 0 bytes in 0 blocks

Read the definitely lost records first

Valgrind loss records are not presented in any notable order, and record numbers are not inherently meaningful. Start with definitely lost records and inspect their allocation stacks. In this run, loss record 6 carries the diagnosis in one line: 97 (16 direct, 81 indirect) bytes in 1 blocks are definitely lost.

The parenthesis is the part worth learning. A block of 16 bytes was lost directly, and losing it stranded another 81 bytes that were only reachable through it. Freeing that one block at build_table (memleak.c:24) accounts for 97 of the 186 unfreed bytes. A record with a large indirect component often represents an owning or container allocation whose loss stranded child allocations, but record order itself has no significance.

The other three records describe the rest of the program honestly. The 25 bytes at main (memleak.c:46) are still reachable, because the global cache pointer keeps them. The 64 bytes at line 48 are possibly lost, which is Valgrind telling you it only found a pointer into the middle of that block, matching the buf + 8 arithmetic. The two indirect records are the strings the table owned.

By default Memcheck also hides most of this. Without --show-leak-kinds=all you see only definite and possible records, and indirect blocks get no stack trace at all, on the assumption that you will fix the parent. That assumption is usually right, and it is also why a default run can look thinner than the leak actually is.

Memcheck options that add detail

Memcheck has far more leak-related switches than the two everyone uses. These are the ones that change what you can conclude.

Stack depth is the first thing to raise on real software, and the effect is easiest to see by shrinking it. Cut the stacks to two callers:

valgrind --leak-check=full --show-leak-kinds=all --num-callers=2 ./memleak

==449352== 33 bytes in 6 blocks are indirectly lost in loss record 2 of 5
==449352==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449352==    by 0x401189: dup_field (memleak.c:17)
==449352== 
==449352== 97 (16 direct, 81 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5
==449352==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449352==    by 0x4011BD: build_table (memleak.c:24)

Six records became five and the caller chain is gone. Everything allocated through dup_field is now one record with no hint of who asked for it. Framework code that wraps allocation in three or four layers hits this at the default of 12, so raise it to 30 or 40 when records look suspiciously generic.

Grouping is the other knob. Memcheck merges blocks into one record when their stacks look the same, and how many frames it compares is your choice:

valgrind --leak-check=full --show-leak-kinds=all --leak-resolution=low ./memleak

==449270== 33 bytes in 6 blocks are indirectly lost in loss record 2 of 5
==449270==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449270==    by 0x401189: dup_field (memleak.c:17)
==449270==    by 0x401232: build_table (memleak.c:32)
==449270==    by 0x401281: main (memleak.c:43)

The low setting compares only two frames, so the key strings from line 32 and the value strings from line 33 were merged into one 33-byte record even though the stack still prints four frames. Use low when a report is a wall of near-identical records and you want totals per subsystem; keep the default high when you need to tell two call sites apart.

Rank leaked bytes per function with an xtree report

Everything so far has been per allocation site. On a large codebase the more useful question is which file or function is responsible for the most leaked memory, and Memcheck can emit that as an execution tree:

valgrind --leak-check=full --xtree-leak=yes --xtree-leak-file=xtleak.kcg ./memleak

The run prints the usual summary and writes the tree to xtleak.kcg. That file uses the Callgrind format, so annotate it with the tool that ships alongside Valgrind:

callgrind_annotate --show=DB,DIB,IB --inclusive=yes --threshold=100 xtleak.kcg

DB          DIB         IB           file:function
--------------------------------------------------------------------------------
97 (100.0%) 81 (100.0%) 81 (100.0%)  vg_replace_malloc.c:malloc
97 (100.0%) 81 (100.0%) 81 (100.0%)  memleak.c:main
 0           0          33 (40.74%)  memleak.c:dup_field
97 (100.0%) 81 (100.0%) 81 (100.0%)  memleak.c:build_table

--------------------------------------------------------------------------------
-- Auto-annotated source: memleak.c
--------------------------------------------------------------------------------
 .           .           .           static char *dup_field(const char *text)
 .           .           .           {
 0           0          33 (40.74%)      char *copy = malloc(strlen(text) + 1);

The columns are leaked byte counts: DB is definitely lost, DIB is direct plus indirect, and IB is indirectly lost. Because the report is inclusive, build_table and main carry the full 97 bytes while dup_field carries only the 33 bytes it allocated itself. Then the annotated source repeats the same numbers next to the exact lines. On a codebase where nobody can name the leaking module, this ranking is the fastest way to find the file to open, and kcachegrind will load the same file as a browsable call tree.

Generate a suppression instead of writing one

Valgrind suppression syntax is fiddly and there is no reason to hand-write it. Ask Valgrind to print the block for you:

valgrind --leak-check=full --gen-suppressions=all ./memleak

==450419== 97 (16 direct, 81 indirect) bytes in 1 blocks are definitely lost in loss record 6 of 6
==450419==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==450419==    by 0x4011BD: build_table (memleak.c:24)
==450419==    by 0x401281: main (memleak.c:43)
==450419== 
{
   <insert_a_suppression_name_here>
   Memcheck:Leak
   match-leak-kinds: definite
   fun:malloc
   fun:build_table
   fun:main
}

Each record is followed by a copy-ready block. Give it a name, drop it in a file, and pass --suppressions=FILE on later runs. Suppress library noise this way, never your own code; the match-leak-kinds line is what lets you hide reachable one-time allocations while still failing on definite leaks.

When both tools report nothing but memory keeps growing

This is the case that sends people in circles. A service climbs steadily for days, the team runs both tools on the same build, and both come back clean. Nothing is broken about the tools. The memory is still reachable, and a leak checker cannot call it lost while a live pointer exists.

Save this as leak-service.c. It models the shape that causes most production leaks, a session table that keeps every entry:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define MAX_SESSIONS 4096
#define PAYLOAD_SIZE (256 * 1024)

struct session {
    char *payload;
    size_t len;
};

static struct session *sessions[MAX_SESSIONS];
static size_t session_count;

static struct session *open_session(size_t len)
{
    struct session *s = malloc(sizeof *s);

    s->payload = malloc(len);
    memset(s->payload, 'x', len);
    s->len = len;
    return s;
}

static void close_session(struct session *s)
{
    free(s->payload);
    free(s);
}

static void handle_request(unsigned n)
{
    struct session *s = open_session(PAYLOAD_SIZE);

    sessions[session_count++] = s;

    if (n % 4 == 0) {
        close_session(sessions[--session_count]);
    }
}

int main(int argc, char **argv)
{
    unsigned requests = (argc > 1) ? (unsigned)atoi(argv[1]) : 40;
    unsigned delay = (argc > 2) ? (unsigned)atoi(argv[2]) : 0;

    for (unsigned n = 0; n < requests && session_count < MAX_SESSIONS; n++) {
        handle_request(n);
        if (delay) {
            sleep(delay);
        }
    }

    printf("served %u requests, %zu sessions still open\n", requests, session_count);
    return 0;
}

Every request allocates a 256 KB payload, and only one request in four is ever closed. There is no missing free anywhere in the code; close_session is correct and gets called. The bug is that the global sessions array holds the rest forever, which is a design leak rather than a coding slip. Build a sanitizer version of it:

gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -o leak-service-asan leak-service.c

Now the interesting part. Run 40 requests, which retains about 7.5 MB, and watch what the leak checker says:

./leak-service-asan 40

served 40 requests, 30 sessions still open

Not a single leak reported, and the exit status is 0. Thirty sessions are alive in the global array, so from the sanitizer's point of view the program is perfectly well behaved. A CI gate built only on this check would pass a service that grows until the kernel kills it.

Memcheck is more forthcoming, because it reports reachable memory when you ask:

valgrind --leak-check=full --show-leak-kinds=all ./leak-service 40

==449490== HEAP SUMMARY:
==449490==     in use at exit: 7,864,800 bytes in 60 blocks
==449490==   total heap usage: 81 allocs, 21 frees, 10,490,496 bytes allocated
==449490== 
==449490== 480 bytes in 30 blocks are still reachable in loss record 1 of 2
==449490==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449490==    by 0x40118B: open_session (leak-service.c:19)
==449490==    by 0x40120E: handle_request (leak-service.c:35)
==449490==    by 0x4012CE: main (leak-service.c:50)
==449490== 
==449490== 7,864,320 bytes in 30 blocks are still reachable in loss record 2 of 2
==449490==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==449490==    by 0x40119B: open_session (leak-service.c:21)
==449490==    by 0x40120E: handle_request (leak-service.c:35)
==449490==    by 0x4012CE: main (leak-service.c:50)
==449490== 
==449490== LEAK SUMMARY:
==449490==    definitely lost: 0 bytes in 0 blocks
==449490==    indirectly lost: 0 bytes in 0 blocks
==449490==      possibly lost: 0 bytes in 0 blocks
==449490==    still reachable: 7,864,800 bytes in 60 blocks
==449490==         suppressed: 0 bytes in 0 blocks

Zero errors, and 7.8 MB in 60 reachable blocks with the allocating call path spelled out. Reading still reachable as harmless is the mistake to avoid: 81 allocations and 21 frees means three out of four payloads were never released, and the fix is a retention policy rather than a free call. What tells the two situations apart is not this single report but the trend, which is where a heap profiler earns its keep.

Attribute the growth to a call path with Massif

Massif samples the heap over time and records which call paths hold the bytes. That is exactly the question a reachable-growth leak raises. Run the same workload under it:

valgrind --tool=massif --massif-out-file=massif.out ./leak-service 40

The run prints only the program's own output and writes snapshots to massif.out. Render them with the reader that ships with Valgrind:

ms_print massif.out

MB
7.505^                                                                       #
     |                                                                    :::#
     |                                                                @@@:: :#
     |                                                      :::::@ : :@ @ : :#
     |                                        :::::@ :: : @ :: : @ : :@ @ : :#
     |                         ::::::@: : : @ :: : @ :: : @ :: : @ : :@ @ : :#
     |                  ::::::@: : : @: : : @ :: : @ :: : @ :: : @ : :@ @ : :#
     |    :::::@ : : :@ : : : @: : : @: : : @ :: : @ :: : @ :: : @ : :@ @ : :#
   0 +----------------------------------------------------------------------->Mi
     0                                                                   10.16

Number of snapshots: 61
 Detailed snapshots: [2, 7, 12, 17, 22, 27, 32, 37, 44, 51, 52, 60 (peak)]

A steadily rising heap profile suggests retained memory worth investigating. Healthy workloads often release substantial memory between work cycles, but caching and intentional growth can also produce a rising baseline. Now open the peak snapshot, which carries the call tree:

ms_print massif.out | sed -n '/^ 60 /,+6p'

60     10,651,355        7,869,384        7,868,896           488            0
99.99% (7,868,896B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.94% (7,864,320B) 0x40119B: open_session (leak-service.c:21)
| ->99.94% (7,864,320B) 0x40120E: handle_request (leak-service.c:35)
|   ->99.94% (7,864,320B) 0x4012CE: main (leak-service.c:50)
|     
->00.06% (4,576B) in 1+ places, all below ms_print's threshold (01.00%)

Almost the entire 7.5 MB peak sits under one path: open_session line 21, called from handle_request line 35. That is the same answer a leak report would have given, except Massif produced it for memory that is not lost at all. A few options make the profile fit the workload:

• --time-unit=B puts allocated bytes on the x-axis instead of instructions, which makes runs comparable.
• --detailed-freq=N records a call tree every N snapshots when the default of every tenth is too coarse.
• --max-snapshots=N raises the 100-snapshot ceiling for long runs.
• --pages-as-heap=yes counts every mapped page, which catches growth from mmap that the heap profile misses.

Measure how much the heap still holds at the end with DHAT

DHAT answers a different question: of everything the program allocated, how much was still held when it finished, and which sites allocate the most. Run it over the same workload:

valgrind --tool=dhat --dhat-out-file=dhat.out ./leak-service 40

==449608== Total:     10,490,496 bytes in 81 blocks
==449608== At t-gmax: 7,868,896 bytes in 61 blocks
==449608== At t-end:  7,864,800 bytes in 60 blocks
==449608== Reads:     443 bytes
==449608== Writes:    10,486,443 bytes
==449608== 
==449608== To view the resulting profile, open
==449608==   file:///usr/libexec/valgrind/dh_view.html

Three numbers tell the story: 10.4 MB allocated in total, 7.8 MB held at peak, and 7.8 MB still held at the end. A program that releases what it borrows shows a small t-end relative to Total, so an ending figure that nearly equals the peak means the process is hoarding. Open dh_view.html in a browser and load dhat.out for the per-site breakdown, including average block lifetime, which is the number that exposes short-lived allocations that never get freed.

Investigate a live process without restarting it

Now the production case: a service is growing, you cannot rebuild it into a sanitizer binary, and restarting it destroys the evidence. Work outward from cheap symptoms to precise answers.

Confirm the growth with ps, pidstat, and smaps

Start the plain binary in the background so there is something to watch. It serves 300 requests one second apart:

setsid ./leak-service 300 1 > /dev/null 2>&1 &

The shell returns immediately and prints nothing useful, so look the PID up by process name and keep it in a variable:

PID=$(pgrep -x leak-service)

With the PID in hand, ask ps for the current resident size. Resident memory is what the process actually occupies in RAM:

ps -o pid,rss,vsz,comm -p "$PID"

PID   RSS    VSZ COMMAND
 450625  3028   4016 leak-service

RSS is in kilobytes, so this process holds about 3 MB a few seconds after startup. One reading proves nothing; the ps command only becomes evidence when you compare readings over time.

pidstat does the sampling for you. Take four samples five seconds apart:

pidstat -r -p "$PID" 5 4

07:28:35 PM   UID       PID  minflt/s  majflt/s     VSZ     RSS   %MEM  Command
07:28:40 PM     0    450625     38.17      0.00    5040    4052   0.05  leak-service
07:28:45 PM     0    450625     38.40      0.00    5808    4820   0.06  leak-service
07:28:50 PM     0    450625     49.21      0.00    6832    5716   0.07  leak-service
07:28:55 PM     0    450625     38.94      0.00    7600    6612   0.08  leak-service

RSS climbs from 4052 KB to 6612 KB across twenty seconds while minor faults hold near 40 per second. The steadily rising RSS together with ongoing minor faults shows that the process is bringing more pages into its resident set. It is a useful growth signal, but it does not identify the allocator or prove a leak by itself. If pidstat is missing, install sysstat; for longer windows, sar keeps the same counters across hours.

Proportional memory is the fairer number when a process shares pages with others. Read the rollup that the kernel computes for the whole address space:

grep -E "^(Rss|Pss)" /proc/$PID/smaps_rollup

Rss:                6612 kB
Pss:                5243 kB
Pss_Dirty:          5224 kB
Pss_Anon:           5224 kB
Pss_File:             19 kB
Pss_Shmem:              0 kB

Nearly all of the proportional memory is anonymous rather than file-backed or shared-memory pages. That makes application-private allocations a stronger suspect, but it still does not prove the growth is specifically the malloc heap. Per-process memory metrics covers how to keep watching the trend; Massif or Valgrind on a test build provides the call-path attribution. Everything so far confirms the symptom without naming a line, so switch tools.

Ask a running process for a leak report

Valgrind can accept commands while it runs. This is the technique to reach for when a leak only shows after hours of real traffic, and it is worth starting suspect services this way in staging. Launch the workload under Memcheck with the debugger gateway open:

setsid valgrind --vgdb=yes --leak-check=full --read-var-info=yes ./leak-service 300 1 > vg.log 2>&1 &

Two flags matter here. --vgdb=yes makes Valgrind listen for monitor commands, and --read-var-info=yes loads variable debug info, which is what lets a later command name a global by its source declaration. Valgrind writes its PID into the log banner:

head -4 vg.log

==451052== Memcheck, a memory error detector
==451052== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==451052== Using Valgrind-3.26.0 and LibVEX; rerun with -h for copyright info
==451052== Command: ./leak-service 300 1

Use that number, 451052, with vgdb. Ask for a full leak check including reachable blocks while the process keeps serving:

vgdb --pid=451052 leak_check full reachable any

sending command leak_check full reachable any to pid 451052
==451052== 1,048,576 bytes in 4 blocks are still reachable in loss record 2 of 2
==451052==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==451052==    by 0x40119B: open_session (leak-service.c:21)
==451052==    by 0x40120E: handle_request (leak-service.c:35)
==451052==    by 0x4012CE: main (leak-service.c:50)
==451052== 
==451052== LEAK SUMMARY:
==451052==    definitely lost: 0 bytes in 0 blocks
==451052==    indirectly lost: 0 bytes in 0 blocks
==451052==      possibly lost: 0 bytes in 0 blocks
==451052==    still reachable: 1,048,640 bytes in 8 blocks
==451052==         suppressed: 0 bytes in 0 blocks

One megabyte in eight blocks after a few seconds of traffic, with no restart and no rebuild. The real power comes from the second check. Wait half a minute, then ask only for what increased since last time:

vgdb --pid=451052 leak_check full reachable increased

sending command leak_check full reachable increased to pid 451052
==451052== 192 (+128) bytes in 12 (+8) blocks are still reachable in loss record 1 of 2
==451052==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==451052==    by 0x40118B: open_session (leak-service.c:19)
==451052==    by 0x40120E: handle_request (leak-service.c:35)
==451052==    by 0x4012CE: main (leak-service.c:50)
==451052== 
==451052== 3,145,728 (+2,097,152) bytes in 12 (+8) blocks are still reachable in loss record 2 of 2
==451052==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==451052==    by 0x40119B: open_session (leak-service.c:21)
==451052==    by 0x40120E: handle_request (leak-service.c:35)
==451052==    by 0x4012CE: main (leak-service.c:50)
==451052== 
==451052== LEAK SUMMARY:
==451052==    definitely lost: 0 bytes in 0 blocks
==451052==    indirectly lost: 0 bytes in 0 blocks
==451052==      possibly lost: 0 bytes in 0 blocks
==451052==    still reachable: 3,145,920 (+2,097,280) bytes in 24 (+16) blocks
==451052==         suppressed: 0 bytes in 0 blocks

The delta in parentheses shows which allocation site is growing between checks: plus 2,097,152 bytes in eight new blocks, all from open_session line 21. A busy server produces hundreds of reachable records, and this mode discards every one that stayed flat between the two checks. Two commands thirty seconds apart turn a noisy heap into a single suspect.

Find which variable still holds the memory

Knowing the allocation site is not the same as knowing why the memory survives. For a reachable leak, the question is which pointer keeps it alive, and Memcheck can walk that chain. Loss record 2 is the growing one, so list a couple of its blocks:

vgdb --pid=451052 block_list 2 limited 2

sending command block_list 2 limited 2 to pid 451052
==451052== 3,145,728 (+2,097,152) bytes in 12 (+8) blocks are still reachable in loss record 2 of 2
==451052==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==451052==    by 0x40119B: open_session (leak-service.c:21)
==451052== 0x4A84120[262144]
==451052== 0x4AC41B0[262144]

The last two lines are real block addresses with their sizes, each one a 256 KB payload. Record numbers refer to the most recent leak search, so run block_list right after a leak_check or it will refuse with an obsolete-list error. Pick the first address and ask what points at it:

vgdb --pid=451052 who_points_at 0x4A84120

sending command who_points_at 0x4A84120 to pid 451052
==451052== Searching for pointers to 0x4a84120
==451052== *0x4a840d0 points at 0x4a84120
 Address 0x4a840d0 is 0 bytes inside a block of size 16 alloc'd
==451052==    at 0x484387E: malloc (vg_replace_malloc.c:447)
==451052==    by 0x40118B: open_session (leak-service.c:19)

One level up the chain: the payload is held by a 16-byte heap block, which is the struct session allocated on line 19. That block must itself be held by something, so repeat the question one level higher:

vgdb --pid=451052 who_points_at 0x4a840d0

sending command who_points_at 0x4a840d0 to pid 451052
==451052== Searching for pointers to 0x4a840d0
==451052== *0x404060 points at 0x4a840d0
 Location 0x404060 is 0 bytes inside sessions[0],
 a global variable declared at leak-service.c:14

There is the root cause, named: sessions[0], a global variable declared at line 14. Not an allocation site, not a call path, but the variable that owns the memory and never releases it. That is the difference between "something leaks in open_session" and "the global session table has no eviction", and only the second sentence tells you what to change. This chain works on any reachable leak, which makes it the most valuable Memcheck feature that almost nobody uses.

Confirm the hypothesis with gdb

You can also sanity-check a growing container from outside Valgrind, on the plain binary, without stopping it for more than a moment. Attach gdb in batch mode and read the counter:

gdb -p "$PID" -batch -ex 'print session_count' -ex 'print sessions[0]->len'

0x00007f43a2fba847 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
$1 = 31
$2 = 262144

Thirty-one live sessions holding 262144 bytes each accounts for roughly 8 MB of heap, which is the growth those RSS samples were tracking. Attaching stops the process while gdb is connected, so keep batch commands short on anything serving traffic, and remember that this only works when the binary carries symbols for the variables you want.

Heap profilers you can attach to a PID

When the process was not started under Valgrind, a couple of profilers can attach to a live PID. Use them on a staging copy running the same build.

Runtime attach is intrusive. Heaptrack upstream marks PID attachment as unstable and warns that injection or detach can crash the target, which is reason enough to keep it away from a critical process. Debug symbols make these reports readable; without them you get addresses and library names instead of your own functions.

Fix the leak and prove it is gone

Each kind of record from the earlier reports needs a different repair. The direct leak needs an owner that releases the table, including the strings it holds. The reachable global needs releasing or a retention limit. The possibly lost block needs the base pointer kept alongside the offset. Save this complete fixed program as memleak-fixed.c:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

struct row {
    char *key;
    char *value;
};

struct table {
    struct row *rows;
    size_t count;
};

static char *dup_field(const char *text)
{
    char *copy = malloc(strlen(text) + 1);
    strcpy(copy, text);
    return copy;
}

static struct table *build_table(size_t count)
{
    struct table *t = malloc(sizeof *t);

    t->rows = malloc(count * sizeof *t->rows);
    t->count = count;

    for (size_t i = 0; i < count; i++) {
        char key[32];
        snprintf(key, sizeof key, "key%zu", i);
        t->rows[i].key = dup_field(key);
        t->rows[i].value = dup_field("value");
    }
    return t;
}

static void free_table(struct table *t)
{
    for (size_t i = 0; i < t->count; i++) {
        free(t->rows[i].key);
        free(t->rows[i].value);
    }
    free(t->rows);
    free(t);
}

static char *cache;
static char *base;
static size_t offset;

int main(void)
{
    struct table *t = build_table(3);
    printf("loaded %zu rows\n", t->count);

    cache = dup_field("still referenced at exit");

    base = malloc(64);
    offset = 8;

    free_table(t);
    free(cache);
    free(base);
    return 0;
}

free_table mirrors build_table exactly, releasing the strings before the array and the array before the struct. Keeping base and using an integer offset instead of storing buf + 8 removes the interior pointer that Memcheck flagged as possibly lost. Compile the corrected version:

gcc -g -O0 -o memleak-fixed memleak-fixed.c

The build is silent. Re-run the same Memcheck command that produced six loss records earlier:

valgrind --leak-check=full --show-leak-kinds=all ./memleak-fixed

==450305== HEAP SUMMARY:
==450305==     in use at exit: 0 bytes in 0 blocks
==450305==   total heap usage: 11 allocs, 11 frees, 4,282 bytes allocated
==450305== 
==450305== All heap blocks were freed -- no leaks are possible
==450305== 
==450305== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Eleven allocations and eleven frees, and the sentence you want to see. Compare that against the first run, which reported 11 allocs and 1 free, and the arithmetic is its own proof.

Confirm with the other tool as well, since the two disagree about reachable memory and a clean bill from both is worth having:

gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -o memleak-fixed-asan memleak-fixed.c && ./memleak-fixed-asan

loaded 3 rows

No report and an exit status of 0. The program prints its line and finishes with nothing held.

Common root causes behind each report shape

Reports repeat themselves once you have read a few hundred. These are the patterns behind the shapes in this guide.

Gate CI on the exit code

A leak fixed today comes back next quarter unless a test fails on it. Both tools can fail a build. Point Valgrind at the leaky binary and ask it to treat definite leaks as errors:

valgrind --leak-check=full --errors-for-leak-kinds=definite --error-exitcode=1 --log-file=vg-ci.log ./memleak

loaded 3 rows

The report went to the log file so it will not drown your test output, and the command exited 1. AddressSanitizer already exits non-zero when it reports leaks, which is why the earlier ./memleak-asan run returned 1 while the fixed build returned 0. Two practical notes for pipelines: use --errors-for-leak-kinds=definite so reachable one-time allocations do not fail every build, and keep ASAN_OPTIONS=detect_leaks=1 explicit in the job, since some platforms ship with it off.

Use a core dump when the leaking process also crashes

A core dump is not a leak detector. It captures process state at the moment of a crash, not a list of unfreed allocations. It still helps when a leaking service eventually aborts or gets killed, because the backtrace shows what the process was doing when memory ran out. On systemd distributions systemd-coredump stores the core, replacing the older abrt workflow.

List the cores recorded for one program by name:

coredumpctl list --no-pager leak-service

TIME                           PID UID GID SIG     COREFILE EXE                             SIZE
Fri 2026-08-21 19:42:13 IST 451781   0   0 SIGABRT present  /root/memleak-lab/leak-service 17.9K

One line per core, and the column that matters is COREFILE. A value of present means the core is on disk and can be extracted, while missing means only the metadata survived. Export this one with the PID from that listing:

coredumpctl dump 451781 -o /tmp/app.core

Message: Process 451781 (leak-service) of user 0 dumped core.
                
                Stack trace of thread 451781:
                #0  0x00007f5139913847 clock_nanosleep@GLIBC_2.2.5 (libc.so.6 + 0xd0847)
                #1  0x00007f513991f1c7 __nanosleep (libc.so.6 + 0xdc1c7)
                #2  0x00007f513993129c sleep (libc.so.6 + 0xee29c)
                #3  0x00000000004012df n/a (/root/memleak-lab/leak-service + 0x12df)
                #4  0x00007f513986d58e __libc_start_call_main (libc.so.6 + 0x2a58e)

The core is written to /tmp/app.core and the metadata already carries a rough stack. Frame 3 is only an address with n/a for the function, because coredumpctl has no access to the binary's debug info. Load the core in gdb together with the executable to fix that, and print every thread:

gdb -batch -ex 'thread apply all bt' ./leak-service /tmp/app.core

Core was generated by `./leak-service 300 1'.
Program terminated with signal SIGABRT, Aborted.

Thread 1 (Thread 0x7f5139840740 (LWP 451781)):
#0  0x00007f5139913847 in clock_nanosleep@GLIBC_2.2.5 () from /lib64/libc.so.6
#1  0x00007f513991f1c7 in nanosleep () from /lib64/libc.so.6
#2  0x00007f513993129c in sleep () from /lib64/libc.so.6
#3  0x00000000004012df in main (argc=3, argv=0x7ffd82226dd8) at leak-service.c:52

The same frame now resolves to main at leak-service.c:52, which is the sleep call in the request loop. This process was idle rather than allocating when it died, so the core tells you where it stopped and nothing about the leak. When a stack ends inside an allocator or in a failed allocation, the crash and the growth are related, and that narrows the search. The leak itself still needs a sanitizer or Valgrind run on a test build.

Troubleshooting

References

• AddressSanitizer and LeakSanitizer (Clang documentation)
• LeakSanitizer (Clang documentation)
• Valgrind Memcheck manual
• Massif heap profiler manual
• DHAT dynamic heap analysis tool
• Valgrind gdbserver and vgdb
• GDB documentation
• systemd-coredump(8)
• proc(5) man page

Summary

Finding a leak is a two-step job, and most guides stop after the first step. Detection is one command: build with -fsanitize=address when the source is yours, or run Valgrind Memcheck when all you have is a binary. Diagnosis is reading the report properly. Ignore runtime allocator frames, use the first useful frame in your own code to find the allocation site, then walk upward through callers to decide which function owns the memory. In Memcheck, start with definitely lost records. A line such as 97 (16 direct, 81 indirect) bytes often marks a container whose loss stranded child allocations behind it.

When a report tells you less than you need, the tools have more to give. --show-leak-kinds=all reveals the records Memcheck hides by default, --num-callers restores the caller chain that made a record generic, --xtree-leak=yes with callgrind_annotate ranks leaked bytes per function and per line across a whole codebase, and --gen-suppressions=all writes suppressions so you never hand-craft the syntax. On the sanitizer side, malloc_context_size controls how records are grouped, fast_unwind_on_malloc=0 fixes stacks in optimized builds, report_objects=1 lists the leaked addresses, and __lsan_do_recoverable_leak_check() gives a process that never exits something to report.

The harder leaks are the ones nothing calls a leak. Memory parked in a global session table, cache, or list is still reachable, so LeakSanitizer stays silent and Memcheck files it under still reachable with zero errors. Read the trend rather than one snapshot: a steadily rising Massif profile, a t-end figure from DHAT close to the peak, or a leak_check full reachable increased delta from vgdb that keeps naming the same allocation site. When the memory is reachable, block_list and who_points_at will chase the pointer chain until it names the global variable and the line it was declared on, which is as close to a root cause as tooling gets.

Then prove the fix rather than assuming it. Re-run and look for All heap blocks were freed, or matching allocation and free counts, and wire --error-exitcode=1 with --errors-for-leak-kinds=definite into CI so the next regression fails a build instead of a pager. For containerized services, check cgroup memory limits before you go leak hunting; more than one team has spent a week on a limit that was simply too low.