Question: Comparing drives/folders for files

[CarlRCraw]

I have two drives/paths/folders (for example (a) C:\A-folder and (b) F:\B-folder) - each with multiple different sub-folders and files.

I need to verify that all the files of (a) exist somewhere on (b) without regard to path/folder.

My main purpose is to verify that the files of (a), which are proposed for deletion, exist somewhere on (b), or that I agree that the file is not necessary on (b) and can be deleted from (a) without impact.

Is there some way with Everything that I can do this analysis in an automated (or semi-automated) fashion?

Thanks for your help, Carl

[NotNull]

On first thought, that is not possible "out of the box", but should be do-able with some extra effort.

But first, an important question: when do you consider files to be the same? When they have the same name? Same name + same size? Same name + same size + content bit-exactly the same?

[CarlRCraw]

For my purpose probably just the name, but if it is not difficult to verify size/date as well then that might be preferable.

I want to verify that files still exist in the destination after a backup operation of a renamed folder. Many backup programs have to delete the old destination files and folder and than rebuild a new folder with the files rather than just renaming the destination backup folder to match the origin folder. My backup program saves all deleted files in a separate directory that allows me to verify the deletions. Doing this verification manually is very time consuming. Can't seem to find any software that will do this, so hoping Everything could help.

[NotNull]

Thanks for clarification. I do not have time today to write something, but tomorrow should be manageable.
What do you want to do with the found duplicates? Delete them? / List them (and what details)? / ... ?

Can't seem to find any software that will do this,

You might want to take a look at the free version of Easy Duplicate Finder.
It has a scan mode: Folder comparison that does what (I think that) you want.

[CarlRCraw]

Thank You.

I simply want to visually see the files and their path for all (a) files and if it finds a (b) file, it's path. If it does not find a (b) file, then simply list the (a) file and path.

Similar to what the program lists now for files that it finds (listing them all together with their paths), but limiting/filtering the search/listing to only those files in folder/path (a) (rather than the whole disk drive).

Thanks for your help.

Carl

[vanisk]

Try WinMerge (File and Folder comparision utility) which compares 2 different location and lists out identical files and "Location A only" files and "Location B only" files.

2019-01-11 23_29_27-WinMerge.jpg (23.33 KiB) Viewed 19040 times

2019-01-11 23_21_32-WinMerge.jpg (21.93 KiB) Viewed 19044 times

http://winmerge.org/

or if u prefer portable release
https://portableapps.com/apps/utilities ... e_portable

[CarlRCraw]

Hi,

Thanks for WinMerge idea. Tested it. It doesn't do what I need done. It only looks at current folder directories and files; it doesn't drill down into all folders for differences.

This is what I am looking for:
(1) For each file in the origin folder [A], search all folders/subfolders in the destination for its existence, displaying the path for each file found.
(2) Repeat this process for all files in all folders/sub-folders in [A].

Carl

[NotNull]

This is what I am looking for:
(1) For each file in the origin folder [A], search all folders/subfolders in the destination for its existence, displaying the path for each file found.
(2) Repeat this process for all files in all folders/sub-folders in [A].

Did you take a look at Easy Duplicate Finder? It seems to do what you want (also in portable version):

2019-01-14 22_41_07-Window.png (47.84 KiB) Viewed 19016 times

2019-01-14 22_52_11-Window.png (47.3 KiB) Viewed 19016 times

(I did not have any time last weekend to write something, btw. Will do when Easy Duplicate Finder turns out to be not what you are looking for.)

[therube]

Do a search on your two wanted paths:

Code: Select all

X: | M:

File | Export -> music-m+x.efu

File | Open File List -> music-m+x.efu

Do a search to find duplicates:

Code: Select all

dupe:

Or unique's:

Code: Select all

!dupe:

Now there can be some gotchas with that.
It will turn up two files with the same name, within the same directory tree.
(And probably some other things I'm not thinking of off the top of my head.)

Code: Select all

M:\___fromC\The Cinematic Orchestra\The Cinematic Orchestra - Ma Fleur (2007)\02 - Familiar Ground.mp3
X:\_MUSIC\___fromC\The Cinematic Orchestra\The Cinematic Orchestra - Ma Fleur (2007)\02 - Familiar Ground.mp3
X:\_MUSIC\jazz\cinematic orchestra\Ma Fleur\02 - Familiar Ground.mp3

[therube]

Duplicate Cleaner: Verify files between two drives/paths

You can also set DC to find duplicates between 2 directory trees (ignoring any dupes that may happen to be within the same tree).
(Ditto with AllDup, "Compare only files between different source folders".)

[NotNull]

Tonight I had some time to write something (just for fun; I still think the aforementioned tools are much better equipped for the task).
It's a basic CMD script that requires ES.exe (the command-line interface for Everything):

2019-01-17 00_03_42-C__Windows_System32_cmd.exe.png (17.35 KiB) Viewed 18995 times

And that generates 2 outputfiles: Output.txt and CanBeDeleted.txt.
It is really fast as it uses Everything (although my code could be optimized ...)

Output.txt

Code: Select all

[ "T:\DELETEST\NOTFOUND\AutoHotkeyU64.exe" ] 
C:\develop\JumpToFolder\AutoHotkeyU64.exe
C:\develop\JumpToFolder\GUI menu\AutoHotkeyU64.exe
C:\develop\JumpToFolder\GUI menu\IconTest\AutoHotkeyU64.exe
C:\develop\Tweaks\AutoHotkeyU64.exe
C:\Tools\AutoHotkey_1.1.27.07\AutoHotkeyU64.exe
[ "T:\DELETEST\Everything.chm" ] 
C:\Tools\Everything\Everything.chm
[ "T:\DELETEST\Everything.exe" ] 
C:\develop\Everything Installer\Everything-1.4.1.895.x64-Setup\$PLUGINSDIR\Everything\Everything.exe
C:\Tools\Everything\Everything.exe
[ "T:\DELETEST\NOTFOUND\Unique1 - Copy.txt" ] 
[ "T:\DELETEST\NOTFOUND\Unique1.txt" ] 

CanBeDeleted.txt

Code: Select all

"T:\DELETEST\NOTFOUND\AutoHotkeyU64.exe" 
"T:\DELETEST\Everything.chm" 
"T:\DELETEST\Everything.exe" 

[NotNull]

Duplicate Cleaner: Verify files between two drives/paths

You can also set DC to find duplicates between 2 directory trees (ignoring any dupes that may happen to be within the same tree).
(Ditto with AllDup, "Compare only files between different source folders".)

Was followingv your links and in the first one the question was asked by someone with the same username as "our" CarlRCraw

[vanisk]

It's a basic CMD script that requires ES.exe (the command-line interface for Everything):

Thank you & Waiting for the upload to try it .

[NotNull]

Thank you & Waiting for the upload to try it .

And I was waiting for feedback from @CarlRCraw ..
But now there is more interest, will post it tonight (European timezone).

[therube]

(Heh. Nope, I didn't catch that.)

-------

Another possibility to find unique files between two trees:

Code: Select all

dir  x:  m:  /b /s  |  sed "s/.*\\//"  |   sort   |  uniq -d  >  list_uniq

[NotNull]

This is tailor-made for @CarlRCraw, but can be made quite easily in something more universal
Files are considered the same when they have equal name, date modified AND size

Instructions

• If you haven't done so already, download ES.exe from here
• Save this script in some folder on your system.
  I named it DELTEST.cmd, but name it whatever you feel like (DualDirectoryDuplicateDetector.cmd ? )
• In the script, change the path to ES.exe to match your situation
• Check if Everything is running. If not: start it.
• Double-click your 'Whatever.cmd' in Explorer
• Fill in the path to your back-ups by (*) dragging the folder to this window, (*) typing the folder name or (*) use Tab-completion (type c:\p and <Tab> till you reach "Program Files").
• Same procedure for the other folder
  
  When done, you can view the results in Output.txt and CanBeDeleted.txt
  
  ( Build and (quickly) tested with Everything 1.4.1.895 x64 /ES 1.1.0.9 @Win10 )
  
  DELTEST.cmd

  Code: Select all

  @echo off
  setlocal
  
  ::_______________________________________________
  ::
  ::	SETTINGS
  ::_______________________________________________
  ::
  
  ::	PAth to ES.exe
  	set EXE=c:\Tools\Everything\ES.exe
  
  ::	Location outputfile
  	set OUTPUT=.\Output.txt
  ::	Location list of deletable files
  	set DELETE=.\CanBeDeleted.txt
  	
  
  ::_______________________________________________
  ::
  ::	INIT
  ::_______________________________________________
  ::
  
  	pushd "%~dp0"
  	set TEMPFILE=.\zzzFolder1.txt
  	del "%DELETE%" 2>nul
  	del "%OUTPUT%" 2>nul
  
  
  ::_______________________________________________
  ::
  ::	ACTION!
  ::_______________________________________________
  ::
  
  ::	Get backup folder
  	echo.
  	echo.   Type the foldername or
  	echo.	drag/drop the folder where your BACKUPS are
  	echo.   from Explorer to this window and press [Enter]
  	echo.
  	set /p FOLDER1=Backup folder : 
  
  ::	Get original folder
  	echo.
  	echo.   Type the foldername or
  	echo.   drag/drop the folder where your ORIGINAL FILES are
  	echo.   from Explorer to this window and press [Enter]
  	echo.
  	set /p FOLDER2=Original folder : 
    
  	cls
  	echo.
  	echo.___________________________________________________
  	echo.
  	echo.	Backup folder     : %FOLDER1%
  	echo.	Original files in : %FOLDER2%
  	echo.
  	echo.
  	echo.	Remember: Everything must be running!
  	echo.___________________________________________________
  	pause
  
  
  	
  ::	Strip "" from foldernames and 
  ::	remove possible trailing \ in foldername
  
  
  	for /f "useback delims=" %%x in ('%FOLDER1%\.') DO set FOLDER1=%%~dpnxx
  	for /f "useback delims=" %%x in ('%FOLDER2%\.') DO set FOLDER2=%%~dpnxx
  
  ::	Escape special chars & and ^ (according to ES help)
  	rem seems not needed :-s. Probably because those are ""
  	
  ::	"Index" files in Backup folder
  	"%EXE%" -full-path-and-name -dm -size -size-format 1 "%FOLDER1%\" file: -export-csv "%TEMPFILE%"
  
  ::	Compare each file with "Original folder"
  	for /F "skip=1 usebackq tokens=1-3 delims=," %%x  in ("%TEMPFILE%") DO call :PARSEFILE %%x "%%y" "%%z"  
  	del "%TEMPFILE%"
  	pause
  	
  goto :EOF
  
  
  
  :PARSEFILE
  ::	Skip empty lines
  	if [%1]==[] goto :EOF
  
  ::	The CALL command doubles a "^" (bug). Undo this (= THISFILE)
  	set THISFILE=%1
  	set THISFILE=%THISFILE:^^=^%
  
  ::	Strip path from filemae (= THISNAME)
  	for /f "delims=" %%a in (%THISFILE%) do set THISNAME="%%~nxa"
  
  ::	Log THISFILE
  	echo [ %THISFILE% ] >> "%OUTPUT%"
  
  ::	Find THISNAME in FOLDER2
  ::	If found: add to logfile  and to DELETE file
  	"%EXE%" "%FOLDER2%\" file: %THISNAME% dm:%2 size:%3 -size-format 1 | findstr "\\" >> "%OUTPUT%" && echo %THISFILE%>> "%DELETE%"
  
  ::	Done. Next file from FOLDER1 ...
  
  goto :EOF
  
  

[vanisk]

Code: Select all

dir  x:  m:  /b /s  |  sed "s/.*\\//"  |   sort   |  uniq -d  >  list_uniq

Thanks for the code @therube (Learning something new everyday)

DELTEST.cmd

Thanks @NotNull for sharing the script. Tested it and working fine.

[therube]

dir x: m: /b /s | sed "s/.*\\//" | sort | uniq -d > list_uniq

Revised:

Code: Select all

dir  x:  /b /s  |  sed "s/.*\\//"  |   sort -u    >  list_tmp
dir  m:  /b /s  |  sed "s/.*\\//"  |   sort -u   >>  list_tmp
uniq -d  <  list_tmp  > list_uniq

The sort -u eliminates duplicates found in a particular directory tree (so either X: or M:)
& then the uniq eliminates duplicates between both trees (so X: and M:).

Can you please tell me, where to get sed & uniq exe files.

GnuWin
GnuWin Packages
I've never "installed" (so can't comment on that).
I download the Zip binaries (extract the .exe's) & stick them in a place in my path (C:\BIN\UNIX\).
Some "packages" have "dependencies", so you might need those also.
Some programs may have stand-alone links, like sed & will contain "sed".
Some programs package a bunch of small stand-alone utilities together, like CoreUtils, which include lots, including uniq.


If you needed (even older) 16-bit DOS programs, there is DJGPP.

[NotNull]

GnuWin
GnuWin Packages
I've never "installed" (so can't comment on that).
I download the Zip binaries (extract the .exe's) & stick them in a place in my path (C:\BIN\UNIX\).

I switched to Cygwin as GnuWin seemed a bit abandoned and Cygwin had a 64-bit version. For the rest:same difference ...

[NotNull]

Thanks @NotNull for sharing the script. Tested it and working fine.

Thanks for testing!

It still has a couple of issues, which I will iron out later:
- problems with folders with spaces in their name (the default issue in any script/program )
- If you use folder c:\abc, but also have a folder like c:\abcd or "c:\abc something else", it will scan those too.
- I need to do some extra testing with special characters "&" and "^" (special meaning in ES)
- No idea what will happen with (for example) Cyrillic characters in filenames

Update will follow.

[NotNull]

Update will follow.

Updated.
Still no idea what will happen with (for example) Cyrillic characters in filename, though.

[Stamimail]

Another tool might help here - the new version of ToDownload by ovg:
viewtopic.php?f=7&t=5998#p19575