In this article we will look at Java\\’s We will use Guava\\’s file hashing method to calculate a hash of every and the resulting output_dir could look like this:<\/p>\n If you use a build tool such as Maven or Gradle you can easily download Java\\’s new file I\/O package, Note that the first two methods above have as parameters a Path and a So with these things in mind, here is an example of a SimpleFileVisitor By itself, this class does nothing. We need to instantiate it and supply Here is a more complete example, which wraps up the FileHasherVisitor The example above could be bundled up into a JAR with its dependencies Picocli is an elegant and well-documented little library that uses Java There are many examples on the picocli website, so we\\’ll only give a The The There we have it, a working application! Now to build it. Below is the Build it with Gradle by running:<\/p>\n Note that in your project directory, Gradle will put the JAR in The program should instruct you on how to supply the correct options and Sierra, Bates, and Robson, OCP Java SE 8 Programmer II Exam Guide<\/em>, Oracle Java API: https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/<\/a><\/p>\n Google Guava: https:\/\/github.com\/google\/guava<\/a><\/p>\n
\nSimpleFileVisitor<\/a>
\nas a tool to solve this problem: how to consolidate multiple backups of
\ndata, photos, and videos, from multiple devices such as phones and
\nlaptops. Suppose you (or your not-so-tech-savvy friends) use multiple
\ndevices and copy files between them, and edit different versions of
\nthose files but don\\’t maintain a single \\"source of truth\\". How do you
\nget a set of unique files without manually checking each one? You could
\nwrite a shell script but for this example, we will work in Java, and
\nwrite a small program that achieves this and shows off SimpleFileVisitor
\nand a little of two excellent Java libraries: Google
\nGuava<\/a>, and
\npicocli<\/a> (ie \\"pico CLI\\").<\/p>\n
\nfile the program encounters as it traverses the file system, and use
\nthat as an approximation for uniqueness, assuming that any two files
\nwith the same hash are identical files. We will use picocli to make an
\nelegant command-line front-end for our code. The program will read in
\nfrom the command line a list of directories to traverse, and it will
\ntraverse those directories in the supplied order. It will copy only the
\nfirst file for each hash code calculated, ignoring subsequent files with
\nthe same hash code, to the user-supplied output directory (make sure
\nthis directory is empty or non-existent before executing the program).
\nWhen the program is finished you will be able to call it like this (in a
\nBash shell):<\/p>\n$ java -jar unique-merge-0.0.1-SNAPSHOT.jar \\\n> -o output_dir \\\n> first_src_dir \\\n> second_src_dir \\\n> third_src_dir<\/code><\/pre>\noutput_dir\n|-first_dir\n|-second_dir\n|-third_dir<\/code><\/pre>\nDependencies<\/h2>\n
\nthe dependencies from MVN Repository:<\/p>\n\n
\nFor this example I used Gradle 6.6.1, Guava 29, and picocli 4.5.1, so my
\nGradle dependencies section looks as follows:<\/li>\n<\/ul>\ndependencies {\n compile 'com.google.guava:guava:29.0-jre'\n compile 'info.picocli:picocli:4.5.1'\n}<\/code><\/pre>\nWalking the File System with SimpleFileVisitor<\/h2>\n
\njava.nio.file<\/a>,
\nprovides new classes and interfaces for dealing with files and
\ndirectories, including
\nPaths<\/a>,
\nPath<\/a>,
\nFiles<\/a>
\n(but not
\nFile<\/a> –
\nthat belongs to the old I\/O package), and
\nSimpleFileVisitor<\/a>.
\nThe SimpleFileVisitor provides a way to recurse through a directory and
\nits subdirectories without needing to implement a recursive method
\nyourself. In our example, we will walk through each directory that the
\ncaller supplies to our program, and use Guava to calculate the hash of
\neach file under that directory, including sub-directories. To use the
\nSimpleFileVisitor, extend it, and override those methods which represent
\nthe events you wish to intercept:<\/p>\npublic class MySimpleFileVisitor extends SimpleFileVisitor<Path> {\n @Override\n public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException { ... }\n\n @Override\n public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException { ... }\n\n @Override\n public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException { ... }\n\n @Override\n public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException { ... }\n}<\/code><\/pre>\n
\nBasicFileAttributes<\/a>,
\nwhile the last two accept a Path and an IOException, in case there was
\nan error while traversing the file system tree that you want to handle
\nbefore continuing. BasicFileAttributes are useful for learning about the
\nfile\\’s owner and permissions, but a proper treatment of that class is
\noutside of the scope of this article. Each of the methods above returns
\nan instance of the
\nFileVisitResult<\/a>
\nenum, which determine what happens next during file system traversal.
\nYour options are:<\/p>\n\n\n
\n \nEnum<\/th>\n Action<\/th>\n<\/tr>\n<\/thead>\n \n FileVisitResult.CONTINUE<\/code><\/td>\nkeep traversing the file system.<\/td>\n<\/tr>\n \n FileVisitResult.SKIP_SIBLINGS<\/code><\/td>\ndon\\’t examine any other files and directories within the parent of the current path object.<\/td>\n<\/tr>\n \n FileVisitResult.SKIP_SUBTREE<\/code><\/td>\ndon\\’t examine any other files or directories under the current Path object.<\/td>\n<\/tr>\n \n FileVisitResult.TERMINATE<\/code><\/td>\nstop traversing the file system.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\nthat uses Guava to calculate the hash code of every file it traverses,
\nand store the hash and the file\\’s path in a map. If the hash code
\nalready exists, then the file is assumed to be a duplicate, and the
\nfile\\’s path is appended to the list of paths for that hash code,
\nalthough at present we will not use it. Later, as an exercise, you might
\nwant to make the program print out a list of duplicate files to the
\nconsole. Using the hash code as an approximation for uniqueness is valid
\nfor most personal use cases, whether it is suitable for industrial scale
\ndata mining I am not qualified to comment.<\/p>\npackage com.cheerfulprogramming.edwhiting.uniquemerge;\n\nimport com.google.common.hash.HashCode;\nimport com.google.common.hash.Hashing;\n\nimport java.io.IOException;\nimport java.nio.file.FileVisitResult;\nimport java.nio.file.Path;\nimport java.nio.file.SimpleFileVisitor;\nimport java.nio.file.attribute.BasicFileAttributes;\nimport java.util.Deque;\nimport java.util.LinkedList;\nimport java.util.Map;\nimport java.util.Optional;\n\npublic class FileHasherVisitor extends SimpleFileVisitor<Path> {\n private Map<HashCode, Deque<Path>> fileHashes;\n\n public FileHasherVisitor(Map<HashCode, Deque<Path>> fileHashes) {\n this.fileHashes = fileHashes;\n }\n\n @Override\n public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {\n HashCode hash = com.google.common.io.Files.asByteSource(file.toFile()).hash(Hashing.sha512());\n Deque<Path> fileHash = Optional.ofNullable(fileHashes.get(hash)).orElseGet(() -> new LinkedList<>());\n fileHash.addLast(file);\n fileHashes.put(hash, fileHash);\n return FileVisitResult.CONTINUE;\n }\n}<\/code><\/pre>\n
\nit to a method that can traverse the file system. Fortunately, Java
\nprovides such a method, Files.walkFileTree(Path, SimpleFileVisitor), as
\nshown here:<\/p>\nfinal Map<HashCode, Deque<Path>> fileHashes = new HashMap<>();\nfinal FileHasherVisitor visitor = new FileHasherVisitor(fileHashes);\nfinal Path path = Paths.get("path\/to\/your\/directory");\ntry {\n Files.walkFileTree(path, visitor);\n} catch (IOException e) {\n e.printStackTrace();\n}<\/code><\/pre>\nResolving Paths<\/h2>\n
\ninvocation in a private method called walkFileTrees, resolves relative
\nfile system paths into absolute ones, copies only the first file for
\neach hash code from its source to the output directory, and exposes the
\npublic method mergePaths, whose parameters resemble the command line
\ninterface we are planning to build.<\/p>\npackage com.cheerfulprogramming.edwhiting.uniquemerge;\n\nimport com.google.common.hash.HashCode;\n\nimport java.io.IOException;\nimport java.nio.file.Files;\nimport java.nio.file.Path;\nimport java.nio.file.Paths;\nimport java.nio.file.StandardCopyOption;\nimport java.util.*;\nimport java.util.stream.Collectors;\n\npublic class UniqueMerger {\n\n public void mergePaths(Path outputDir, Path... srcDirs) {\n \/* Get the directory from where the user called the program.*\/\n final Path currentDir = Paths.get(System.getProperty("user.dir"));\n \/* \n * Resolve the output directory and source directories to absolute \n * file system paths.\n *\/\n final Path absOutputDir = currentDir.resolve(outputDir);\n final List<Path> absSrcDrs = Arrays.stream(srcDirs)\n .map(srcDir -> currentDir.resolve(srcDir))\n .collect(Collectors.toList());\n mergeAbsPaths(absOutputDir, absSrcDrs);\n }\n\n private void mergeAbsPaths(final Path absOutputDir, final List<Path> absSrcDrs) {\n \/* Calculate hash codes for all files under the supplied source directories. *\/\n final Map<HashCode, Deque<Path>> fileHashes = this.walkFileTrees(absSrcDrs);\n \/* \n * For each hash code, copy only the first file that has it, \n * ignoring the duplicates.\n *\/\n fileHashes.values().stream().forEach(fileHash -> {\n final Path src = fileHash.peekFirst();\n absSrcDrs.stream()\n .filter(path -> src.startsWith(path))\n .forEach(path -> {\n final Path dest = absOutputDir.resolve(path.getParent().relativize(src));\n try {\n Files.createDirectories(dest.getParent());\n Files.copy(src, dest, StandardCopyOption.COPY_ATTRIBUTES);\n } catch (IOException e) {\n e.printStackTrace();\n }\n });\n });\n }\n\n private Map<HashCode, Deque<Path>> walkFileTrees(List<Path> absSrcDrs) {\n final Map<HashCode, Deque<Path>> fileHashes = new HashMap<>();\n final FileHasherVisitor visitor = new FileHasherVisitor(fileHashes);\n \/* \n * Look at all the files under each of the supplied\n * source directories, and calculate their hash codes.\n *\/\n absSrcDrs.stream().forEach(path -> {\n try {\n Files.walkFileTree(path, visitor);\n } catch (IOException e) {\n \/* Don't stop for any bad files! *\/\n e.printStackTrace();\n }\n });\n return fileHashes;\n }\n}<\/code><\/pre>\n
\nand made into a library, although it might not be very useful in that
\nform. In the next section, we will use picocli to add a command-line
\ninterface to our program so that we can call it from a shell.<\/p>\nMake it Executable: Add a Command-Line Interface<\/h2>\n
\nannotations to make writing standard-compliant command line Java
\nprograms easy.<\/p>\n
\nbrief introduction here. To start, create a class that will hold your
\nmain method, and make it implement Runnable. Annotate it with picocli\\’s
\n\\@CommandLine.Command. Add non-static properties to your class to hold
\nthe values of the arguments to the application. In this example, we need
\na Path[] array to hold the names of the source directories, and a Path
\nobject to hold the output directory. Picocli takes care of instantiation
\nfor us, provided we annotate these class members correctly. See the full
\nexample below, followed with some remarks:<\/p>\npackage com.cheerfulprogramming.edwhiting.uniquemerge;\n\nimport picocli.CommandLine;\n\nimport java.nio.file.Path;\n\n@CommandLine.Command(\n name = "unique-merge",\n mixinStandardHelpOptions = true,\n version = "Unique Merge v1.0",\n description = "Recursively copies supplied directories into [outputDir] " +\n "directory so that only unique files get transferred, and identical duplicates " +\n "do not get transferred, even if they have different names."\n)\npublic class UniqueMergeApplication implements Runnable {\n\n @CommandLine.Parameters(\n index = "0..*",\n arity = "1..*",\n description = "Directories from which to copy files recursively.")\n private Path[] directories;\n\n @CommandLine.Option(\n names = { "-o", "--output-dir"},\n required = true,\n description = "Output directory where copied directories will be placed, with only unique files.")\n private Path outputDir;\n\n public static void main(String[] args) {\n final int exitCode = new CommandLine(new UniqueMergeApplication()).execute(args);\n System.exit(exitCode);\n }\n\n @Override\n public void run() {\n UniqueMerger u = new UniqueMerger();\n u.mergePaths(this.outputDir, this.directories);\n }\n}<\/code><\/pre>\n@CommandLine.Parameters<\/code> picocli annotation accepts many arguments
\nbut I have only listed in the example the ones needed for this
\napplication to work. The index argument tells picocli what position it
\nshould expect the relevant command line arguments to be. First position
\nis zero, and the argument also accepts a range, eg m..n<\/em> or m..*<\/em> for
\nany position after m<\/em>. The arity argument tells picocl how many values
\nto expect in the argument, which allows the user to pass in a list up to
\na size specified by you, or of an unlimited size if you specify an open
\nrange such as 1..*. In our example, the source directories are an array
\nargument of unlimited size, starting at the 0^th^ position.<\/p>\n@CommandLine.Option<\/code> picocli annotation allows you to specify
\nswitches that modify the behaviour of your program, or let the user pass
\nin named arguments. In our example, the output directory is given as a
\nmandatory switch, since the source directories are an unlimited array.
\nAlthough we could have designed our application to treat the last
\ndirectory in the array of directories as the output directory, much like
\nthe way the Unix cp program works, this way makes handling the arguments
\nsimpler, and makes it harder for your users to do something regrettable
\naccidentally.<\/p>\nBuilding a JAR<\/h2>\n
\nGradle build file. Make sure you have OpenJDK11 or higher installed, and
\nGradle 6 with the Gradle wrapper, gradlew.<\/p>\nplugins {\n id 'java'\n}\n\ngroup = 'com.cheerfulprogramming.edwhiting'\nversion = '1.0.0-SNAPSHOT'\nsourceCompatibility = '11'\n\nrepositories {\n mavenCentral()\n}\n\ndependencies {\n \/\/ https:\/\/mvnrepository.com\/artifact\/com.google.guava\/guava\n compile 'com.google.guava:guava:29.0-jre'\n \/\/ https:\/\/mvnrepository.com\/artifact\/info.picocli\/picocli\n compile 'info.picocli:picocli:4.5.1'\n testImplementation(platform('org.junit:junit-bom:5.7.0'))\n testImplementation('org.junit.jupiter:junit-jupiter')\n}\n\ncompileJava {\n options.compilerArgs << "-Xlint:unchecked"\n}\n\njar {\n manifest {\n attributes "Main-class": "com.cheerfulprogramming.edwhiting.uniquemerge.UniqueMergeApplication"\n }\n from {\n configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }\n }\n}\n\ntest {\n useJUnitPlatform()\n}<\/code><\/pre>\n$ .\/gradlew clean jar<\/code><\/pre>\n
\nbuild\/libs. Execute your application like this:<\/p>\n$ java -jar build\/libs\/unique-merge-1.0.0-SNAPSHOT.jar<\/code><\/pre>\n
\narguments, so you should then run it on the folders that you want to
\nmerge, like this:<\/p>\n$ java -jar build\/libs\/unique-merge-1.0.0-SNAPSHOT.jar -o output_dir first_src_dir second_src_dir third_src_dir<\/code><\/pre>\nFurther Reading<\/h2>\n
\nOracle Press, 2018, pp.268, 271, 277-279, 289-292<\/p>\n