Duplicate file searching
CASTalk.com Forum Index CASTalk.com
Discussion of DSP, FPGA, storage and embedded system.
 
 FAQFAQ   MemberlistMemberlist     RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
 
Google
 
Web castalk.com
Duplicate file searching

 
Post new topic   Reply to topic    CASTalk.com Forum Index -> Storage System
Author Message
m0rk
Guest
Posted: Sat Sep 17, 2005 10:20 pm    Post subject: Duplicate file searching Reply with quote
Can anyone recommend software for searching for true duplicate files
over the network ... were running everything with ms w2k and the users
are copying sets of files all over the place, much of them duplicates
but doing it by hand would be an impossible task.
Back to top
Guest
Posted: Mon Oct 03, 2005 12:15 am    Post subject: Re: Duplicate file searching Reply with quote
m0rk :

Quote:
Can anyone recommend software for searching for true duplicate files
over the network ... were running everything with ms w2k and the users
are copying sets of files all over the place, much of them duplicates
but doing it by hand would be an impossible task.

Generate a list of the files to be checked and run a checksum over
them, like sha1 or md5. Sort on the checksum and duplicates should be
listed in adjacent positions. A shortish Perl script could be used as
well.

--

Joerg Lenneis

email: lenneis@wu-wien.ac.at
Back to top
RPR
Guest
Posted: Sun Oct 09, 2005 12:00 am    Post subject: Re: Duplicate file searching Reply with quote
#! /usr/bin/perl -w
# finddups.pl
# Lists duplicates in MD5 sums
# Use with find something -type f -print0 | xargs -i -0 md5sum "{}"
use strict;
$|=1;
my %h;
while(<>)
{ chomp $_;
# print STDERR substr($_,0,70),qq( \r);
my @a=split / /,$_,2;
push @{$h{$a[0]}},$a[1] if @a==2;
};
foreach(keys %h)
{ print join qq(\n),'',@{$h{$_}},'' if @{$h{$_}}>1;
}
Back to top
 
Post new topic   Reply to topic    CASTalk.com Forum Index -> Storage System All times are GMT
Page 1 of 1

 
 You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




VoIP Electronics Powered by phpBB