dcdev mailing list

What are the current thoughts on file hashing? I mean, I don't think
anyone has missed what advantages it would have to be able to hand out
MD5 or SHA1 sums on all shared files, but on the other hand it would
take unreasonably long to hash 100 GBs every time the client is
started.

So are there any plans in this area?

Fredrik Tolf

At 01:36 AM 11/30/2003 +0100, you wrote:

What are the current thoughts on file hashing? I mean, I don't think
anyone has missed what advantages it would have to be able to hand out
MD5 or SHA1 sums on all shared files, but on the other hand it would
take unreasonably long to hash 100 GBs every time the client is
started.

So are there any plans in this area?

I've implemented it in my DC++ mod (BCDC++).

It stores hashes of files it's hashed, so it needs only recalculate them when they change.

-cologic

On Saturday, November 29, 2003, at 07:57  PM, blackclaw@parsoma.net wrote:

I've implemented it in my DC++ mod (BCDC++).

It stores hashes of files it's hashed, so it needs only recalculate them when they change.

Just for information use, for those who can't go to Sedulus' website and download the source (which admittedly I have not yet done either), BCDC uses Merkle hash trees and the Tiger algorithm to hash files.  This the primary scheme that Shareaza uses.  A little more information can be found in the THEX 0.3 draft:

http://www.open-content.net/specs/draft-jchapweske-thex-02.html

On the DC++ forum and in the public/private hubs we've talked more about the subject, including vague musings over the possibility of hashing files according to other schemes, to perhaps benefit from Sharereactor, etc.  There's a bit of history in the area of thinking/talking about hashes.

Fredrik, what exactly did you want to know? ;))

(and what do you work on, if you don't mind me asking)

 - Todd
DC++ Lackey

John Bäckstrand writes:
> >Doesn't that still take pretty long both the first time the client
> >starts and when you add, say, a movie that takes a GB or so? Are
> >you by any chance using a much faster algorithm than eg. MD5?
> > With my 1400Mhz hashing seems to be taking a few seconds for mp3s,
> and minutes for 600MB-files. This is 1) Much faster than my
> download speed 2) Fast enough to make any latency between a
> finished download and the hash being available tolerable, assuming
> no other hashes are being calculcated.
> > In short, hashing times are not really a problem. There is no need
> to hash the entire share in 10 minutes.

But how about the first time you start the client, or when you move
something into your share from somewhere else?

I have been thinking that hashes might not have to be made available
immediately, but can be calculated while the client is running. Is
that the approach you have taken?

Fredrik Tolf

I have been thinking that hashes might not have to be made available immediately, but can be calculated while the client is running. Is that the approach you have taken?

Yes, just as shareaza, emule etc does.

---
John Bäckstrand

blackclaw@parsoma.net writes:
> At 01:36 AM 11/30/2003 +0100, you wrote:
> >What are the current thoughts on file hashing? I mean, I don't think
> >anyone has missed what advantages it would have to be able to hand out
> >MD5 or SHA1 sums on all shared files, but on the other hand it would
> >take unreasonably long to hash 100 GBs every time the client is
> >started.
> >
> >So are there any plans in this area?
> > I've implemented it in my DC++ mod (BCDC++).
> > It stores hashes of files it's hashed, so it needs only recalculate them > when they change.

Doesn't that still take pretty long both the first time the client
starts and when you add, say, a movie that takes a GB or so? Are you
by any chance using a much faster algorithm than eg. MD5?

Fredrik Tolf

Todd Pederzani writes:
> > On Saturday, November 29, 2003, at 07:57  PM, blackclaw@parsoma.net > wrote:
> > > I've implemented it in my DC++ mod (BCDC++).
> >
> > It stores hashes of files it's hashed, so it needs only recalculate > > them when they change.
> >
> > Just for information use, for those who can't go to Sedulus' website > and download the source (which admittedly I have not yet done either), > BCDC uses Merkle hash trees and the Tiger algorithm to hash files.  > This the primary scheme that Shareaza uses.  A little more information > can be found in the THEX 0.3 draft:
> > http://www.open-content.net/specs/draft-jchapweske-thex-02.html

Thanks for the link. I'll read it soon (must ... go ... to ... bed).

> On the DC++ forum and in the public/private hubs we've talked more > about the subject, including vague musings over the possibility of > hashing files according to other schemes, to perhaps benefit from > Sharereactor, etc.  There's a bit of history in the area of > thinking/talking about hashes.
> > Fredrik, what exactly did you want to know? ;))

Well, I just wanted to know the thoughts on making it faster.

If you want to know my thoughts, the only solution I've managed to
think of is hashing, say, the first 64 kBs or so of each
file. Admittedly, that is far from an optimal solution, but combined
with file size comparison, I have thought that it just might be better
than having to wait a week or so for my share to be hashed... ;-) (in
fact, with my HDD transfer speed of 30MBs/s it would "only" take just
a little more than an hour)
I've also played a bit with the idea that the whole file could be
hashed "on-demand" in the client<->client protocol. I've almost
decided to reject that idea, though, since it would still take a
minute or so to hash a fairly large file, especially on low-end
computers. It would be good in the way that the result of that
operation could be cached, though.

I'll check out the DC++ forums, though.

> (and what do you work on, if you don't mind me asking)

I was writing on dcpro (http://sf.net/projects/dcprod), but I've given
it up for a better implementation (I didn't really like the name
either...) that isn't public yet.

Fredrik Tolf

Doesn't that still take pretty long both the first time the client starts and when
you add, say, a movie that takes a GB or
so? Are you by any chance using a much
faster algorithm than eg. MD5?

With my 1400Mhz hashing seems to be taking a few seconds for mp3s, and
minutes for 600MB-files. This is 1) Much faster than my download speed 2)
Fast enough to make any latency between a finished download and the hash
being available tolerable, assuming no other hashes are being calculcated.

In short, hashing times are not really a problem. There is no need to hash
the entire share in 10 minutes.
---
John Bäckstrand

On Sunday, November 30, 2003, 15:33:55, John Bäckstrand wrote:

Yes, just as shareaza, emule etc does.

ed2k clients don't share files which haven't been hashed yet, because file
hashes are the only way to identify files there...

-- 
Jernej Simoncic, ender@ena.si
http://www2.arnes.si/~sopjsimo/
http://deepthought.ena.si/

History doesn't repeat itself -- historians merely repeat each other.
      -- First Rule of History

On the DC++ forum and in the public/private hubs we've talked more
about the subject, including vague musings over the possibility of
hashing files according to other schemes, to perhaps benefit from
Sharereactor, etc.

Yes, using hash algorithm like the one used in edonkey will be great because
we can use all the ed2k links found on the web....

IIRC edonkey uses MD4, while bittorrent uses SHA1

At 08:04 PM 11/30/2003, you wrote:

> On the DC++ forum and in the public/private hubs we've talked more
> about the subject, including vague musings over the possibility of
> hashing files according to other schemes, to perhaps benefit from
> Sharereactor, etc.

Yes, using hash algorithm like the one used in edonkey will be great because
we can use all the ed2k links found on the web....