dcdev mailing list

http://www.dolda2000.com/~fredrik/doldaconnect/protocol.html

write. However, it would be good if people would start looking through
the sections that I've written about the semantics of the protocol and
give me some early feedback.

First of all, do we really need *persistent* GUIDs? They won't help tracking malicious users, since anyone can always force his client to generate a new GUID when he likes. They can surely be used to differentiate two users sharing the same nick on different time frames, but:
1) even if GUIDs are not persistent, i.e. they are generated each time a client is started, the probability of a 128-bit UID clash is remote;
2) in the case that happens, rollback and TTH are more than enought to protect against file corruptions.
All in all I don't see any real need to make GUIDs persistent, while I don't like the idea of making all too easy the life of people willing to track my online presence.

In the "Semantic->Roles" section I find the following open question:

However, should this protocol define a hub-to-hub protocol as well, or should that be left to the hub implementor?

I think the current hub model is showing many limits; simply put it doesn't scale at all. Things could get better by making two or more servers work togheter (thus the need of a server-to-server protocol spec). I don't know if your draft is the right place, but I'd definitely like to explore the topic.

-- 
    Anakim Border
    http://pydc.sf.net

All in all I don't see any real need to make GUIDs persistent, while I
don't like the idea of making all too easy the life of people willing to
track my online presence.

IMHO, a persistent GUID is required. 1) to have a faster connection establishment with hub (see my previous mail with the modified "GUID ID" command. 2) to allow client to filter their upload. For example, a user can reject an upload request from a GUID not seen on the hub (Note: this must be improved because with such a simple idea, anyone can get GUID of a user connected to a hub, disconnect from the hub and do a direct download attempt).
3) to speed up dead connection deletion. Some people have dynamic IP. If the IP changes when they are connected to the hub, this create 2 things: a dead connection on the hub (takes ~15 minutes to disappear) and an "enumerated" user if the user comes back before the dead connection deletion. If a user has an unique GUID, when he comes back, its dead connection can be deleted immediately (Note: same problem as the previous one with "stolen" GUID).

To avoid the "stolen" GUID problem, a solution is perhaps to use a challenge based login.

I think the current hub model is showing many limits; simply put it doesn't
scale at all. Things could get better by making two or more servers work
togheter (thus the need of a server-to-server protocol spec). I don't know
if your draft is the right place, but I'd definitely like to explore the
topic.

Hub clustering works well (dchub does this) but I don't think this draft should discuss about this because IMHO, communication between hubs of a cluster is heavily based on the internal structures of a hub and also because I think that people building such configuration will always use the same hub program.

Eric

All in all I don't see any real need to make GUIDs persistent, while I
don't like the idea of making all too easy the life of people willing to
track my online presence.

IMHO, a persistent GUID is required. 1) to have a faster connection establishment with hub (see my previous mail with the modified "GUID ID" command. 2) to allow client to filter their upload. For example, a user can reject an upload request from a GUID not seen on the hub (Note: this must be improved because with such a simple idea, anyone can get GUID of a user connected to a hub, disconnect from the hub and do a direct download attempt).
3) to speed up dead connection deletion. Some people have dynamic IP. If the IP changes when they are connected to the hub, this create 2 things: a dead connection on the hub (takes ~15 minutes to disappear) and an "enumerated" user if the user comes back before the dead connection deletion. If a user has an unique GUID, when he comes back, its dead connection can be deleted immediately (Note: same problem as the previous one with "stolen" GUID).

To avoid the "stolen" GUID problem, a solution is perhaps to use a challenge based login.

I think the current hub model is showing many limits; simply put it doesn't
scale at all. Things could get better by making two or more servers work
togheter (thus the need of a server-to-server protocol spec). I don't know
if your draft is the right place, but I'd definitely like to explore the
topic.

Hub clustering works well (dchub does this) but I don't think this draft should discuss about this because IMHO, communication between hubs of a cluster is heavily based on the internal structures of a hub and also because I think that people building such configuration will always use the same hub program.

Eric

Anakim Border writes:
> > http://www.dolda2000.com/~fredrik/doldaconnect/protocol.html
> > > > write. However, it would be good if people would start looking through
> > the sections that I've written about the semantics of the protocol and
> > give me some early feedback.
> > First of all, do we really need *persistent* GUIDs? They won't help
> tracking malicious users, since anyone can always force his client
> to generate a new GUID when he likes. They can surely be used to
> differentiate two users sharing the same nick on different time
> frames, but: 1) even if GUIDs are not persistent, i.e. they are
> generated each time a client is started, the probability of a
> 128-bit UID clash is remote; 2) in the case that happens, rollback
> and TTH are more than enought to protect against file corruptions.
> All in all I don't see any real need to make GUIDs persistent,
> while I don't like the idea of making all too easy the life of
> people willing to track my online presence.

No, of course we don't _need_ persistent GUIDs. I was thinking that it
would be good, though. While noone should rely on the GUID for banning
purposes in that way (that's obvious), I was thinking that users that
want to identify themselves to hubs (aka registered users), could
likely find it useful, in the same way that a user who wants to find a
specific other user could do so, even if that user has changed his
nick.

I see your point about privacy, though, but in that case, I think that
client implementors could just as well implement an option to make the
GUID non-persistent. That's my thought.

> In the "Semantic->Roles" section I find the following open question:
> > >However, should this protocol define a hub-to-hub protocol as
> >well, or should that be left to the hub implementor?
> > I think the current hub model is showing many limits; simply put it
> doesn't scale at all. Things could get better by making two or more
> servers work togheter (thus the need of a server-to-server protocol
> spec). I don't know if your draft is the right place, but I'd
> definitely like to explore the topic.

Precisely my thought. However, I think that this is probably not the
time to discuss that. The first and foremost priority in my mind is to
make the normal functionality work as it should. I also hope that if
hub implementors start developing hub-to-hub protocols before we
include it in any draft, they would hopefully find it natural to use
the same protocol parser that they already have for the hub-client
protocol, and submit their specs to us for inclusion.

Speaking of x-to-x protocols; Eric had an idea that I found
interesting, namely to make to client-to-client protocol more-or-less
pure HTTP. After all, HTTP does almost exactly what the
client-to-client protocol is supposed to do. I would like people to
voice their opinions about that. In my mind, it is a very good
idea. Of course, it would have some DC-specific headers, but that
doesn't prevent anything at all. It would also make HTTPS quite
natural for encrypted transfers.

Fredrik Tolf

Speaking of x-to-x protocols; Eric had an idea that I found
interesting, namely to make to client-to-client protocol more-or-less
pure HTTP. After all, HTTP does almost exactly what the
client-to-client protocol is supposed to do. I would like people to
voice their opinions about that. In my mind, it is a very good
idea. Of course, it would have some DC-specific headers, but that
doesn't prevent anything at all. It would also make HTTPS quite
natural for encrypted transfers.

Yes, I totally agree with my own idea ;) and we clearly share the same point of view about specific headers (in both transfer query and reply).

Passive mode xfer will be the hardest problem to deal (using PUT instead of GET ?).

Eric

Nouanda writes:
> In the actual protocol all commands are fixed. We can't add a parameter or
> change order. We can perhaps use a more http like format (or DC++ tag) for
> all parameters. In fact, it's just to identify clearly all fields.
> > For example, your desc command can look like this:
> > desc nick="test" share=100000 tslots=3 fslots=3 hubs=1 addr=x.x.x.x

I agree that it is in a way an interesting thought. However, I would
guess that hub owners wouldn't like the bandwidth that's wasted on
sending redundant information like that.

Also, there is no problem with adding a parameter. If a command needs
to be extended, we could add an optional parameter, and if it is
omitted, the command will display the older behavior.

Changing the order of parameters seems rather unnecessary, IMHO. Could
you give an example of where that would be useful?

Fredrik Tolf

Anakim Border wrote:

First of all, do we really need *persistent* GUIDs? They won't help tracking malicious users, since anyone can always force his client to generate a new GUID when he likes.

I agree about your privacy concerns... I think a "good" client will offer the user a choice of both per session and global GUIDs.  If someone has queued files from you (and keeps your GUID as an identifer in their queue), a couple of hash matches (since you can request by hash, eliminating the pitfalls of you having reorganized your share in the meantime) should be enough for their client to learn your new GUID and re-establish you as a source for all those files.

- Todd

Fredrik Tolf wrote:

I agree that it is in a way an interesting thought. However, I would
guess that hub owners wouldn't like the bandwidth that's wasted on
sending redundant information like that.

If it allows the hub to do partial MyINFO-type broadcasts, the trade-off of a little more incoming bandwidth is well worth not having to broadcast a typical MyINFO sized packet out to all the users.

- Todd

Todd Pederzani writes:
> Fredrik Tolf wrote:
> > >I agree that it is in a way an interesting thought. However, I would
> >guess that hub owners wouldn't like the bandwidth that's wasted on
> >sending redundant information like that.
> >  > >
> > If it allows the hub to do partial MyINFO-type broadcasts, the trade-off > of a little more incoming bandwidth is well worth not having to > broadcast a typical MyINFO sized packet out to all the users.

That, however, can be solved in a much nicer way, in that case. I
would imagine it to be better in that case to leave the fields that
should not be updated being the empty string, like this:

desc "new nick" "" "" "" 4 ""

Isn't that even better?

Fredrik Tolf wrote:

That, however, can be solved in a much nicer way, in that case. I
would imagine it to be better in that case to leave the fields that
should not be updated being the empty string, like this:

desc "new nick" "" "" "" 4 ""

Isn't that even better?

No, not particularly - especially if you think of it in terms of XML and attributes.  Or if you want to make it extensible.

- Todd

desc "new nick" "" "" "" 4 ""

Little question. How to distinguish an empty string ("") and a string not to update (also coded "" here).

No, not particularly - especially if you think of it in terms of XML and
attributes.  Or if you want to make it extensible.

I though we have decided not to use XML because it is too heavy to carry and process.

Eric

Yeah, it allow client to send only the modified part of its
information...and hub broadcast. (Actually, we need to send the whole MyINFO
each time we connect/disconnect to a hub for example)

For the order part, it was just an example of what we can't do. We need to
remember the order of all arguments, and imho, it's not very important (and
little annoying)
Exactly the same thing for the order of commands sent from client to hub (or
client to client). Order is not important, important thing is at least,
destination have all information needed.

/Fabrice
-----Message d'origine-----
De : dcdev-bounces@dcportal.net [mailto:dcdev-bounces@dcportal.net] De la
part de Todd Pederzani
Envoyé : samedi 17 janvier 2004 15:59
À : Direct Connect developers
Objet : Re: [dcdev] Thoughts about Fredrik's draft

Fredrik Tolf wrote:

I agree that it is in a way an interesting thought. However, I would
guess that hub owners wouldn't like the bandwidth that's wasted on
sending redundant information like that.

If it allows the hub to do partial MyINFO-type broadcasts, the trade-off of a little more incoming bandwidth is well worth not having to broadcast a typical MyINFO sized packet out to all the users.

- Todd

eric writes:
> >desc "new nick" "" "" "" 4 ""
> > Little question. How to distinguish an empty string ("") and a
> string not to update (also coded "" here).

Well, that's quite simple. Since none of the attributes can be empty
(nick, share, slots, hub and address - wouldn't really be practical to
have an empty value), the empty string simply means use the old value.

Fredrik

Nouanda writes:
> Yeah, it allow client to send only the modified part of its
> information...and hub broadcast. (Actually, we need to send the whole MyINFO
> each time we connect/disconnect to a hub for example)

Well, that I have already described how to do.

> For the order part, it was just an example of what we can't do. We
> need to remember the order of all arguments, and imho, it's not
> very important (and little annoying)

While the order admittedly has no value in itself, it is the most
bandwidth-saving way to designate the role of the arguments, since it
takes no extra bandwidth at all. The extra penalty of having to
remember the order of the arguments isn't really that bad anyway, if I
may have my say (it's not really that hard... ;-) ).

Also, if this reordering part only is important for the description, I
could very well consider changing the syntax of that command instead,
such as having arguments that are word-pairs instead, the first
describing what the second is, like this:

desc nick "My nick" tslots 3 fslots 3 share 0...

In fact, that might actually be better than the current syntax, since
it's also more extensible. It's also very much like the syntax that
you described first, only that it only applies to the `desc' command.

The fact remains, that most commands in the DC protocol doesn't need
to be very extensible, and then it is my opinion that it might be
wiser to save the bandwidth that is required by a more complex
protocol. I'm fairly sure that hub owners can agree with me.

> Exactly the same thing for the order of commands sent from client
> to hub (or client to client). Order is not important, important
> thing is at least, destination have all information needed.

I agree about completely. I'm very much against enforcing a strict
order in which commands must be sent. That is something that I have
emphasized very much in my latest mails.

On Saturday, January 17, 2004, 19:50:29, Fredrik Tolf wrote:

Well, that's quite simple. Since none of the attributes can be empty
(nick, share, slots, hub and address - wouldn't really be practical to
have an empty value), the empty string simply means use the old value.

Why is it necessary to send your address in the info command at all? The hub
knows your IP (we're not doing an UDP-based protocol, are we?), so it would
be enough if the client just reported the port it was listening on (although
even this isn't really necessary, if the connection requests will still go
through the hub).

-- 
begin  .sig
< Jernej Simoncic >< http://deepthought.ena.si/ >

You're not drunk if you can lie on the floor without holding on.
      -- Dean Martin's Definition of Drunkenness
end

Jernej Simon?i? writes:
> Why is it necessary to send your address in the info command at all? The hub
> knows your IP (we're not doing an UDP-based protocol, are we?), so it would
> be enough if the client just reported the port it was listening on (although
> even this isn't really necessary, if the connection requests will still go
> through the hub).

There can actually be two reasons:

1. Say if you are on a network where you do have your own IP address,
  but the firewall doesn't allow incoming SYN packets. Then you might
  want to use a SOCKS server that is on another IP address for people
  to connect to.

2. You might want to connect to the hub using IPv6, while still
  listening on an IPv4 socket, so that people unable to use IPv6 stil
  can connect to you.

Also, since you're saving a maximum of 16 bytes, and only once when
you connect, by not sending the IP address, then really, why _not_
send the IP address? Or did you have another reason in mind?

Fredrik Tolf

On Saturday, January 17, 2004, 21:39:34, Fredrik Tolf wrote:

Also, since you're saving a maximum of 16 bytes, and only once when
you connect, by not sending the IP address, then really, why _not_
send the IP address? Or did you have another reason in mind?

How will the hub know that this address is real?

-- 
begin  .sig
< Jernej Simoncic >< http://deepthought.ena.si/ >

When things are going well, someone will inevitably experiment detrimentally.
      -- Boyle's Second Law
end

In the actual protocol all commands are fixed. We can't add a parameter or
change order. We can perhaps use a more http like format (or DC++ tag) for
all parameters. In fact, it's just to identify clearly all fields.

For example, your desc command can look like this:

desc nick="test" share=100000 tslots=3 fslots=3 hubs=1 addr=x.x.x.x

/Fabrice

Todd Pederzani writes:
> Fredrik Tolf wrote:
> > >Really? I would have thought that it would take to much hub resources
> >to do that. Do you really think that it's feasible?
>
> Well, if it's unfeasible, I'm not sure how eDonkey servers pull it off > with a much higher user count than DC hubs. ;)
> I'm a fan of this, because it directly benefits me: too many users > cannot configure active mode correctly.  An automated method to inform > them they're misconfigured and force them to passive mode is a good > thing.  And it can be armored against protection by the hubs (test only > the IP they connect from, test only once every 5 minutes unless the port > changes, not test if machine load is above xxx, don't test if user pings > drop [saturated upstream], etc.)

Indeed. I would have liked the idea; it's just I was so sure that
connecting would require to much of a hub's resources. However, now
that you inform me that it's a real alternative, I'm all go for
it. Shell we settle on that? (Of course, a warning system can still be
good for other reasons, not least fakesharing)

> If a user is behind a router, hasn't entered his external IP into the > settings, and goes to download a list from someone - they will send a > CTM with their private network IP.  The remote user, of course, cannot > connect to them using that.  After a short while, DC++ will stop waiting > for an incoming connection and display "Connection timeout."  For all > intents and purposes, the misconfigured user "cannot connect" to other > user.  This probably will only happen with users new to the DC network, > which means it will happen on a regularly annoying basis.

Oh yes, but that requires original DC semantics - sorry for not making
myself clear on that, but I was planning on having the downloaders
connect (by default) in this protocol. That's why I want to publish
the address in the client description. This way, noone will have to
waste the hub's bandwidth to connect to someone who is in active
mode. No more CTMs unless the other user is in passive mode.

> I think the idea of an automagical warning system is novel... but it's > really quite unworkable - ultimately, it requires OP intervention, and > their judgment about whether or not it's a real problem.

Does it really? Of course, to try very advanced techniques, it might
require OP intervention, but I think lots of things can be tested
automagically, such as connecting to a user, downloading the file
list, comparing the total size of all files in the list against the
indicated share, try to download a few large files at random from the
list to see if they exist, etc. Aren't these already things that many
hubs already implement through scripting?

Fredrik

Jernej Simon?i? writes:
> On Saturday, January 17, 2004, 21:39:34, Fredrik Tolf wrote:
> > > Also, since you're saving a maximum of 16 bytes, and only once when
> > you connect, by not sending the IP address, then really, why _not_
> > send the IP address? Or did you have another reason in mind?
> > How will the hub know that this address is real?

Ah, I see your point. Very good point, in fact. I guess there are two
ways of solving that. Either make the hub connect to each user to try
him (bad idea, I admit), or do something which I very much think that
we should implement anyway - build a "warning system":

If a user detects anything weird about another user (not being able to
connect to him, an error in a file request indicating fake sharers,
etc.), that user could send a warning to the hub, and thus the hub
would be able to connect to the original user and see so that he isn't
faking in any way.

It's a shame that there are such users, but apparently and undeniably,
some users just don't want to share. Therefore, I think we should have
this warning system in any case, don't you agree?

Also, if the hub disconnects anyone for this reason, it should also be
able to send a notification to all other clients to terminate their
uploads to that user.

Fredrik Tolf

Fredrik Tolf wrote:

Ah, I see your point. Very good point, in fact. I guess there are two
ways of solving that. Either make the hub connect to each user to try
him (bad idea, I admit)

eDonkey servers do this - they connect to a user's listening port and if it's unreachable, it puts them in their equivalent of passive mode (lowID).  eDonkey servers also host a lot more users than the biggest DC hubs as of yet do (the biggest I saw listed was 205,628 users - though the median size seems closer to 20-30k).

If a user detects anything weird about another user (not being able to
connect to him,

From my experience on the DC++ forums, if you implement a "warning system" that can be triggered on timeouts by the user, you will get a lot of misconfigured users using it.  (Many behind routers don't realize they have to put in their external IP to become active.)

- Todd

Todd Pederzani writes:
> Fredrik Tolf wrote:
> > >Ah, I see your point. Very good point, in fact. I guess there are
> >two ways of solving that. Either make the hub connect to each user
> >to try him (bad idea, I admit)
> >
> eDonkey servers do this - they connect to a user's listening port
> and if it's unreachable, it puts them in their equivalent of
> passive mode (lowID).  eDonkey servers also host a lot more users
> than the biggest DC hubs as of yet do (the biggest I saw listed was
> 205,628 users - though the median size seems closer to 20-30k).

Really? I would have thought that it would take to much hub resources
to do that. Do you really think that it's feasible?

> >If a user detects anything weird about another user (not being
> >able to connect to him,
> >
>  From my experience on the DC++ forums, if you implement a "warning
> system" that can be triggered on timeouts by the user, you will get
> a lot of misconfigured users using it.  (Many behind routers don't
> realize they have to put in their external IP to become active.)

If I'm not incorrect, none of the things it will detect are very prone
to misconfiguration. Connecting to a user won't be affected by many
(or any?) factors that I can think of. Likewise, fakeshared files
don't have any apparent misconfiguration factors that I can think of
either.

In any case, though, to avoid that users would increase a hubs load by
misusing it, I was thinking that, if the hub investigates a warning
and detects it to be bogus, it could ignore further warnings from that
same client for five minutes or so.

Fredrik

Fredrik Tolf wrote:

Really? I would have thought that it would take to much hub resources
to do that. Do you really think that it's feasible?

Well, if it's unfeasible, I'm not sure how eDonkey servers pull it off with a much higher user count than DC hubs. ;)
I'm a fan of this, because it directly benefits me: too many users cannot configure active mode correctly.  An automated method to inform them they're misconfigured and force them to passive mode is a good thing.  And it can be armored against protection by the hubs (test only the IP they connect from, test only once every 5 minutes unless the port changes, not test if machine load is above xxx, don't test if user pings drop [saturated upstream], etc.)

If I'm not incorrect, none of the things it will detect are very prone
to misconfiguration. Connecting to a user won't be affected by many
(or any?) factors that I can think of. Likewise, fakeshared files
don't have any apparent misconfiguration factors that I can think of
either.

If a user is behind a router, hasn't entered his external IP into the settings, and goes to download a list from someone - they will send a CTM with their private network IP.  The remote user, of course, cannot connect to them using that.  After a short while, DC++ will stop waiting for an incoming connection and display "Connection timeout."  For all intents and purposes, the misconfigured user "cannot connect" to other user.  This probably will only happen with users new to the DC network, which means it will happen on a regularly annoying basis.

You're correct about misconfigurations not affecting users who download fake-files.  I didn't quote that part of your original mail on purpose. =)

In any case, though, to avoid that users would increase a hubs load by
misusing it, I was thinking that, if the hub investigates a warning
and detects it to be bogus, it could ignore further warnings from that
same client for five minutes or so.

I think the idea of an automagical warning system is novel... but it's really quite unworkable - ultimately, it requires OP intervention, and their judgment about whether or not it's a real problem.  Essentially, you're making a new feature that's the same as PMing OPs.  Though it might be simpler, isn't educating users that they should report odd behavior by telling OPs a much better long-term solution?  How would any automated system establish the two-way communication you need to figure out what is really going on?

- Todd

Fredrik Tolf wrote:

Jernej Simon?i? writes:
 > On Saturday, January 17, 2004, 21:39:34, Fredrik Tolf wrote:
 >  > > Also, since you're saving a maximum of 16 bytes, and only once when
 > > you connect, by not sending the IP address, then really, why _not_
 > > send the IP address? Or did you have another reason in mind?
 >  > How will the hub know that this address is real?

Ah, I see your point. Very good point, in fact. I guess there are two
ways of solving that. Either make the hub connect to each user to try
him (bad idea, I admit), or do something which I very much think that
we should implement anyway - build a "warning system":

Ident servers? if to connect to something, why not fetch some (could be) useful info at the same time?

 > eDonkey servers do this - they connect to a user's listening port
 > and if it's unreachable, it puts them in their equivalent of
 > passive mode (lowID).  eDonkey servers also host a lot more users
 > than the biggest DC hubs as of yet do (the biggest I saw listed was
 > 205,628 users - though the median size seems closer to 20-30k).

Really? I would have thought that it would take to much hub resources
to do that. Do you really think that it's feasible?

I think it is possible for only one reason: the user list. ed2k does not have one but DC has and sending a big user list probably uses more bandwidth than checking a user port.

 > >If a user detects anything weird about another user (not being
 > >able to connect to him,
 >
 >  From my experience on the DC++ forums, if you implement a "warning
 > system" that can be triggered on timeouts by the user, you will get
 > a lot of misconfigured users using it.  (Many behind routers don't
 > realize they have to put in their external IP to become active.)

I have another problem with IP usage. Let's say a user has a computer with 2 network connection (like me for example :) ). How can I use the 2 connections (=2 IP) ? Opening 2 clients is a waste of bandwidth. I think a client should be able to have more than one address. I also think a client should send its IP(s) to the hub and the hub should check if commands containing IP and sent by a client has an IP matching the previously sent one (but this is not really a protocol problem, it is a hub problem).

About transfer port verification performed by the hub, I think it will be useless because some clients uses more than one port. Client verification seems to be more reliable.

Eric

Fredrik Tolf wrote:

1. Say if you are on a network where you do have your own IP address,
  but the firewall doesn't allow incoming SYN packets. Then you might
  want to use a SOCKS server that is on another IP address for people
  to connect to.

Isn't the non-validation of CTM what has allowed DC clients, in past events, to be used as a DDoS tool?

- Todd

On Sunday, January 18, 2004, 0:26:01, Todd Pederzani wrote:

Isn't the non-validation of CTM what has allowed DC clients, in past events, to be used as a DDoS tool?

$Search some.random.ip F?F?0?1?$

Guess what happens.

-- 
begin  .sig
< Jernej Simoncic >< http://deepthought.ena.si/ >

You can observe a lot just by watching.
      -- First Law of Science
end