Inside the rsync Source: Three Ideas Worth Reading

October 22, 2026Michał Woliński9 min read

Almost everyone has typed rsync -avz and moved on. It is thirty years old, it is on every server, and it is one of those tools you trust precisely because you never have to think about it. So I did the thing you are not supposed to need to do: I read the source — the current tree at github.com/RsyncProject/rsync, wire protocol 33 — looking for the ideas behind the reflexes.

Three of them are worth your time, because each is a small masterclass in a different discipline: a delete that refuses to run is a lesson in failing safe, a checksum that rolls is a lesson in making the common case cheap, and a file list that learned to stream is a lesson in evolving a core without breaking a protocol millions of people depend on. Every snippet below is quoted from a real file and function; I have kept the line references so you can check me rather than trust me.

None of this is needed to use rsync well. All of it changes how you read your own systems afterwards.

The delete that refuses to run

Start with the flag everyone is a little afraid of: --delete. It makes the destination an exact mirror of the source, which means it removes files on the receiving side that are no longer on the sending side. The nightmare is obvious: if rsync misreads the source — a directory it cannot open, a transient I/O error — and concludes a file is gone, it will dutifully delete the perfectly good copy at the destination. A mirror of a half-read source is a shredder.

rsync does not take that risk. Inside the receiver's deletion walk, before it removes anything, there is this gate:

/* generator.c — delete_in_dir() */
if (io_error & IOERR_GENERAL && !ignore_errors) {
    if (already_warned)
        return;
    rprintf(FINFO,
        "IO error encountered -- skipping file deletion\n");
    already_warned = 1;
    return;
}

io_error is a global bitmask that any part of the run can raise. If a general error was recorded anywhere while reading the source tree, the deletion pass does not run — not "delete the files we are sure about", not "delete most of them". It skips all deletion and says so once. The only way past the gate is to assert, in writing, that you accept the risk: --ignore-errors.

The flag bit itself carries a fossil of the project's caution. In the header:

/* rsync.h */
#define IOERR_GENERAL   (1<<0) /* For backward compatibility, this must == 1 */
#define IOERR_VANISHED  (1<<1)
#define IOERR_DEL_LIMIT (1<<2)

The comment is the interesting part. IOERR_GENERAL cannot be renumbered because an older remote rsync, on the far end of the connection, still expects general errors to be the bit valued 1. The safety behaviour is wire-visible and frozen by compatibility — a theme that comes back twice more below. (IOERR_VANISHED handles the ordinary case of a file disappearing mid-run without blocking deletes; IOERR_DEL_LIMIT is how --max-delete reports that it stopped itself.)

The lesson is older than rsync and still routinely ignored: when a destructive action depends on a reading, a failed reading must block the action, not proceed on partial data. The default is "do nothing and tell me", and you have to opt in to "do it anyway". Most delete-by-diff code I have reviewed gets this exactly backwards — it treats an empty or errored listing as "nothing on the left, so remove everything on the right". That is the same mistake rsync spent a line of code refusing to make.

The receiver's delete pass as a gate: before removing any destination file not present in the source list, rsync checks whether a general I/O error was recorded while reading the source; if so it prints 'IO error encountered — skipping file deletion' and deletes nothing, unless the user passed --ignore-errors; only a clean read lets the delete proceed — The gate in front of every `--delete`. A read error does not degrade the delete into a partial one; it cancels it. `--ignore-errors` is the only key.

This is also why the rsync -avz in my own deploys deliberately has no --delete at all. Skipping deletion entirely is simply the most conservative point on the same axis: stale files may accumulate on the server, but a bad run can never remove a good file. rsync's authors drew the safe default one notch further along than most people's instinct, and then made you sign for every step past it.

A checksum that rolls, and a secret that makes it safe

The reason rsync can update a gigabyte file by sending a few kilobytes is the delta algorithm, and the reason the delta algorithm is fast enough to use by default is a single design decision: make the common answer cheap and the rare answer exact. The common answer, millions of times per file, is "does the block at this byte offset match anything?" — and the honest answer is almost always no.

So rsync asks a cheap question first. It computes a weak 32-bit checksum, a variant of Adler-32, unrolled to chew four bytes at a time:

uint32 get_checksum1(char *buf1, int32 len)
{
    int32 i;
    uint32 s1, s2;
    schar *buf = (schar *)buf1;

    s1 = s2 = 0;
    for (i = 0; i < (len-4); i+=4) {
        s2 += 4*(s1 + buf[i]) + 3*buf[i+1] + 2*buf[i+2] + buf[i+3] + 10*CHAR_OFFSET;
        s1 += (buf[i+0] + buf[i+1] + buf[i+2] + buf[i+3] + 4*CHAR_OFFSET);
    }
    for (; i < len; i++) {
        s1 += (buf[i]+CHAR_OFFSET); s2 += s1;
    }
    return (s1 & 0xffff) + (s2 << 16);
}

Two running sums, s1 and s2, packed into one word. It is deliberately weak — collisions are easy — because its only job is to reject non-matches quickly. And it has a property an MD5 does not: it rolls. When the search moves one byte forward, rsync does not recompute the sum over the whole window. It subtracts the byte that just left and adds the byte that just entered:

/* match.c — roll the window forward one byte, O(1) */
s1 -= map[0] + CHAR_OFFSET;        /* the byte leaving the window */
s2 -= k * (map[0]+CHAR_OFFSET);
if (more) {                        /* the byte entering the window */
    s1 += map[k] + CHAR_OFFSET;
    s2 += s1;
}

That is the whole trick. Testing every byte offset in the target file would be hopeless if each test cost a full checksum; because the weak sum rolls in constant time, rsync can afford to slide one byte at a time across the entire file, asking its cheap question at every position. Only when a hash of the weak sum — its two halves added and masked to 16 bits, (s1 + s2) & 0xffff, or taken modulo a larger table that rsync grows for a big basis file — hits an entry in a hash table does it spend real money, and even then in two steps: first it compares the full 32-bit weak sum, and only on that match does it compute the strong checksum (MD5, or xxhash on modern builds). Weak-and-rolling filters the ocean; the full sum narrows the catch; strong-and-exact confirms the drops.

A two-stage funnel over a file's bytes: a window slides one byte at a time, and at each offset the weak rolling checksum is updated in constant time by subtracting the outgoing byte and adding the incoming byte; a sixteen-bit hash of the weak sum, its two halves added, indexes a hash table; a miss just slides on cheaply, while a hit triggers the expensive strong checksum, seeded per session, which either confirms a block match or is discarded as a false alarm — Cheap question at every offset, expensive question almost never. The rolling update is what makes "test every byte" affordable.

There is one more thing in the strong checksum that most people never notice, and it is a quiet piece of security engineering. The strong sum is not computed over the block alone; it is salted with a per-session seed. The server coins the seed and sends it down the wire before the transfer starts:

/* compat.c — the server coins a per-session seed and sends it */
if (!checksum_seed)
    checksum_seed = time(NULL) ^ (getpid() << 6);
write_int(f_out, checksum_seed);
/* ...the other end simply reads it: */
checksum_seed = read_int(f_in);

Because the seed changes every run, the strong block checksums are different every run. Without it, an attacker who could feed data into one side could precompute two different blocks that share both the weak and the strong checksum, and quietly corrupt a transfer — a block-collision attack against a fixed hash. The seed turns a fixed target into a moving one: whatever you precomputed is worthless against the next session's salt. It is time(NULL) ^ (getpid() << 6) — not cryptographic randomness, but enough to be unpredictable per connection, which is all the defence needs.

And a small joke hiding in plain sight across all of this: CHAR_OFFSET, added to every single byte in every sum. Look it up and it is #define CHAR_OFFSET 0. It multiplies out to nothing. It is a vestigial knob from an old portability concern, left in because it is free and removing it would churn the checksum for no gain. Thirty years of transfers have added zero to every byte, faithfully.

The file list that outgrew memory

The first two ideas are about a single transfer. The third is about how rsync changed its own heart without anyone having to notice — the kind of migration most systems never manage.

For most of its life, rsync did something that seems reasonable until your trees get big: before sending a single byte, it walked the entire source, built the complete list of every file to be considered, and held it all in memory. On a tree of a few thousand files, fine. On a backup of millions of files, that list is enormous, the receiver sits idle while it is built, and both sides pay the memory cost up front. People hit it as "rsync just sits there for minutes doing nothing, then starts", and as plain out-of-memory on the largest trees.

The fix, landed at protocol 30, is incremental recursion: walk a directory, send its slice of the list, start transferring it, and walk the next directory while that transfer runs. Memory is bounded by a directory rather than by the whole tree, and the first byte moves almost immediately. What makes it a good story is that rsync could not simply switch — an old rsync on the other end has no idea what a streamed list is. So the capability is negotiated:

/* compat.c — incremental recursion is a negotiated capability */
compat_flags = allow_inc_recurse ? CF_INC_RECURSE : 0;
/* ...later, both ends agree on the result: */
inc_recurse = compat_flags & CF_INC_RECURSE ? 1 : 0;

Two modern rsyncs light up the CF_INC_RECURSE bit and stream; pair a new one with an ancient one and the bit stays dark and both fall back to building the whole list, exactly as before. The same wire, two behaviours, chosen by handshake — the same compatibility discipline that froze IOERR_GENERAL at bit 1.

The part I find most honest is where rsync refuses to stream, because some options genuinely need the whole list in hand first:

/* compat.c — set_allow_inc_recurse(): when streaming is refused */
if (!recurse || use_qsort)
    allow_inc_recurse = 0;
else if (!am_sender
 && (delete_before || delete_after
  || delay_updates || delay_symlinks || prune_empty_dirs))
    allow_inc_recurse = 0;

Read that second branch and you can see the seams between features. --delete-before and --delete-after have to know the complete picture of what will exist before they can decide what to remove, so they force the old, whole-list mode. --prune-empty-dirs cannot know a directory is empty until it has seen all of it. Each of these is a feature quietly paying the memory-and-latency cost that incremental recursion otherwise saves — which is also why the modern default, when you ask for --delete, is --delete-during: it was designed to be the one that keeps the streaming the other two give up.

Two timelines compared. The old mode, protocol 29 and earlier: scan the entire tree into a complete file list held in memory, and only then begin transferring, so the first byte is late and memory grows with the whole tree. The new mode, incremental recursion at protocol 30 and later: scan one directory, send its slice, transfer it while scanning the next directory, so the first byte is early and memory is bounded by a single directory — Before: build everything, then move. After: move while you build. The behaviour is chosen by a negotiated bit, and a handful of options opt back out of it.

What the three have in common

Lift out of the C and the three ideas rhyme.

Make the safe outcome the default, and charge for the dangerous one. rsync will not delete after a read error unless you pass --ignore-errors; the risky path exists, but it has a name and you have to say it. Your own destructive jobs — the cleanup cron, the reconciliation that removes orphans — should ask the same question: if my input looks wrong, is my default to proceed or to stop? If it is to proceed, you have built a shredder and not noticed.

Make the common case cheap and the rare case exact. The rolling weak checksum answers "no" millions of times for almost nothing; the strong checksum, the expensive and correct one, runs only where the cheap filter already said "maybe". Any hot path that validates, matches, or dedupes wants this shape: a fast, approximate, false-positive-only filter in front of the slow, authoritative check. Bloom filters, Git's pack index, database bitmap scans — the rolling checksum is the same instinct, written in 1996.

Evolve the core, but never break the wire. Incremental recursion replaced rsync's most fundamental data structure, and IOERR_GENERAL is pinned to bit 1, for the same reason: somewhere there is an old rsync on the other end of a pipe, and it must still understand you. Every capability that could have forked the ecosystem is instead a negotiated bit, dark by default, lit only when both ends agree. That is how a thirty-year-old protocol gets to grow new organs without a flag day.

You can use rsync for another decade without any of this. But the next time one of your own systems has to delete on a diff, match on a checksum, or change a format two parties already speak, there is a good answer sitting in generator.c, checksum.c and compat.c — written by people who assumed, correctly, that the other end of the line might be older, slower, or wrong, and built for it anyway.