Skip to content

Replace copies with hardlinks

Have you recently switched to a setup that supports hardlinks and Instant Moves (Atomic-Moves) and would like to replace duplicated files with hardlinks?

If your operating system supports it, you can make use of jdupes.

The latest version's binaries are available for Windows at the link above. You can use a package manager, such as homebrew, apt, or pacman, to install the latest available version for Mac or your flavor of Linux.

Usage

We won't cover every command ‼

If you want to know what else jdupes can do, please read the usage manual.

Tip

This process can put a significantly large load on your system resources for an extended time, depending on the size of your library. As duplicates are discovered and hardlinks are made, the process becomes more efficient - as linked and different-sized files are not repeatedly checked against each other. This leads to subsequent runs potentially finishing faster.

You can leverage a hash database to speed this process up significantly. This will store information about the files, including their signatures, across runs of jdupes - vastly increasing the speed at which runs are finished.

Simply use the following additional option BEFORE YOUR DIRECTORIES with a path that is always available and persistent:

-y "/mnt/user/appdata/scripts/media_hash.db"
  • We don't suggest using jdupes on a cloud-based setup.

The following is a basic usage template:

jdupes [options] DIR1 DIR2

The example below will do a dry run and summarize at the end.

Folder paths should be adjusted to match your directory structure.

jdupes -rMX onlyext:mp4,mkv,avi "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"
jdupes -rMX onlyext:mp4,mkv,avi -y "/mnt/user/appdata/scripts/media_hash.db" "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

The example below will hardlink all duplicate files without prompting.

Folder paths should be adjusted to match your directory structure.

jdupes -rLX onlyext:mp4,mkv,avi "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"
jdupes -rLX onlyext:mp4,mkv,avi -y "/mnt/user/appdata/scripts/media_hash.db" "/mnt/user/data/torrents/movies/" "/mnt/user/data/media/movies"

Windows only allows a maximum of 1023 hardlinks per file.

The -Q or --quick option only reads each file once, hashes it, and performs comparisons based solely on the hashes. There is a small but significant risk of a hash collision which is the purpose of the failsafe byte-for-byte comparison that this option explicitly bypasses. Please do not use it on ANY data set for which any data loss is unacceptable. You have been warned!


Questions or Suggestions?

If you have questions or suggestions, click the button below to join our Discord server.

Click For Support
Discord chat