What is the minimal semantic required to build a valid node?

[anon2076149]

Hi everyone,

I’m uncertain about many details regarding the specs:
Should I consider the protocol.md to be the generic protocol ? (abstraction that all protocols - WS2P/BMA/GVA needs to implements) it describes all the fields to be used for signing and hashing documents so I suppose this is the very minimum I need to implement to make a node.

However the description is not always very strict and therefore raises many questions.

For example : if we take output conditions,
matching something like : (SIG(pubkey) && (CLTV(number) || XHX(hash)))
the specs doesnt say if :

• SIG( pubkey ) (with inner white-space(s)) is valid ?
• SIG (pubkey) (with ws before parameter’s definition) is valid ?
• ( CLTV(number) || XHX(hash) ) (with ws arounds parenthesis of and/or condition)
• Are parenthesis surrounding AND / OR required ?
• AND / OR are restricted to 2 operands ? (is “x AND y AND z” valid ? )
• should any extra ws be authorized, is it taken into acount during hashing ?

Most of that seems trivial from a parsing point of view but is not since the hashing / signature is produced based on the raw Transaction string… if the protocol accepts infinite number of valid strings, it accepts infinite number of hashes for a TX/ block. Which, I believe is a design flow but as always I may just need to be convinced otherwise.

[elois]

Hi,

@anon2076149

I’m uncertain about many details regarding the specs:
Should I consider the protocol.md to be the generic protocol ? (abstraction that all protocols - WS2P/BMA/GVA needs to implements) it describes all the fields to be used for signing and hashing documents so I suppose this is the very minimum I need to implement to make a node.

In fact we dissociate the blockchain protocol (DUBP) and the network protocol (DUNP).
the protocol.md document describes the blockchain protocol.
The network protocol is when described by WS2P.

BMA and GVA are only Client APIs, they are not part of the protocol.
Each node is free to provide or not an API that can be used by clients.
In practice, we try to agree on a future common Client API that will be called GVA.

@anon2076149

However the description is not always very strict and therefore raises many questions.

I also find, and I’m having trouble coding DURS.
That is why I hope that the protocol will be fully clarified quickly and that all ambiguities will be removed.

The current blockchain protocol is only the 1st production version, and was made by very few people, they couldn’t think of everything.
To us who arrive after the battle to take over

In the meantime, what happens is the behavior of the Duniter Typescript code, so I’m forced to go through it when I encounter ambiguity !

[anon2076149]

@elois

In fact we dissociate the blockchain protocol (DUBP) and the network protocol (DUNP).

Where does one begins and ends ?

@elois

In the meantime, what happens is the behavior of the Duniter Typescript code, so I’m forced to go through it when I encounter ambiguity !

So I have to Read The F*** Code ? JS/TS/Rust is like chinese to me.

Can anyone answer this ambiguity then ?
should any extra whitespaces be accepted by the parser, is it taken into acount during hashing ?

I think we should first have a formal grammar for parsing DUBP I’ll draft one in perl6 perhaps ANTLR then But I feel little to no confidence without it. unless we all translate each other’s code exactly, which serves no purpose

[elois]

@anon2076149

Where does one begins and ends ?

In terms of development you have to start with the blockchain protocol, only then can you implement the WS2P network layer on top.
And in a 3rd time the client API. It’s in any case in this order that I proceed for DURS

@anon2076149

Can anyone answer this ambiguity then ?
should any extra whitespaces be accepted by the parser, is it taken into acount during hashing ?

To know this, tests would have to be done by submitting incorrect documents to the test currency. What some have done and it seems that Duniter Typescript is a great error tolerance, and takes these errors into account in the calculation of hash
On the other hand the sources of money thus created are not unthinkable!
In fact, being consistent with Duniter Typescript’s current behavior so as not to cause a fork is extremely difficult and can only be done on a trial and error basis.
And redefining the protocol is useless if @cgeek does not integrate these new spec. So as long as you redefine this part you might as well do it when binarizing documents, otherwise it will do twice as much work for @cgeek.
It is also possible that he doesn’t want to do either and that we are therefore condemned to try to blindly remain compliant until the Duniter nodes become minority on the Ğ1 network.
In any case, on hard I intend to binarize as soon as possible the blockchain protocol and I will propose an RFC on this subject when I have finished WS2Pv2 and g1-monitv2 (so not before 2019).

@anon2076149

I think we should first have a formal grammar for parsing DUBP I’ll draft one in perl6 perhaps ANTLR then But I feel little to no confidence without it.

As explained above it doesn’t suit me to do this because it will do me twice as much work (and a cgeek too), because defining a grammar will become obsolete when all documents are in binary format.

That’s also why I told you in private that the timing is really bad to launch you into this project because we are in full mutation of blockchain and network protocols. In addition, cgeek is withdrawing from the Typescrit implementation, which nobody planned to take over, and on my side I already have so much work planned on DURS for the next 3 years that I can’t add intermediate changes that would add more work to me.

[Mententon_03]

@elois

It is also possible that he doesn’t want to do either and that we are therefore condemned to try to blindly remain compliant until the Duniter nodes become minority on the Ğ1 network.

Dans tous les cas, j’espère que ton code sera plus élégant que ta façon de tuer le maître.

[elois]

@Mententon_03

Dans tous les cas, j’espère que ton code sera plus élégant que ta façon de tuer le maître.

Je ne “tue” personne, Bertrand a raison la doc actuelle est ambiguë, @nanocryk avait déjà relevé ce problème quand il était arrivé et avait suggéré comme solution que l’on binarise tout les documents.

Il est tout a fait normal que la 1ère version de prod du protocole pose quelques soucis, c’est impossible de pondre un protocole nickel dés le début, surtout quand on est peu nombreux
Donc je ne jette la pierre a personne, si @cgeek avait attendu suffisamment de contributeurs et de temps pour faire un protocole très robuste et non ambiguë dés le départ ben on attendrais encore et la ğ1 n’existerai pas…
Il vaut mieux faire en avancer même si c’est pas parfait, plutôt que de ne rien faire du tout, d’autant que quoi qu’on fasse ce ne sera jamais parfais.
Bien sur cela m’a causer des soucis pour développer durs et m’en causera encore, mais je ne me suis pas plains sur le moment, j’en parle ici uniquement pour répondre a Bertrand

[anon2076149]

@elois

BMA and GVA are only Client APIs, they are not part of the protocol.

Meaning fork resolution / blocks calculation occurs only through WS2P ?
BMA seems to accept POST requests though!
Why would the DUBP not use various networking utilities?
Because you dont like REST no one should be allowed to use it ? comon …

@elois

On the other hand the sources of money thus created are not unthinkable!

This confirms my suspicions! Where are the maths showing it? I mean, if you consider its fine that anyone knows how to cheat but not others, it becomes seriously difficult to trust anyone in here. Especially when most participants here spend there time correcting each other vocabulary. perhaps would be nice to cleanup our own backyard first !

@elois

And redefining the protocol is useless if @cgeek does not integrate these new spec.

We seriously disagree on the method here. I dont shoot first ask question later… I think a clear spec written out might impulse the changes!

@elois

And redefining the protocol is useless if @cgeek does not integrate these new spec. So as long as you redefine this part you might as well do it when binarizing documents, otherwise it will do twice as much work for @cgeek

You’re suggesting, only @cgeek can impact the development of the Protocol ? be serious please
You’re suggesting, we have to binarize documents and fix it all. At the same time we make it all unreadable for humans. What a bad long term idea! think twice friend, think twice!

@elois

In any case, on hard I intend to binarize as soon as possible the blockchain protocol and I will propose an RFC

Propose, Yes propose but really… binarizing a protocol, that is proven to be broken and having a single implementation? It would persists errors without the need for it yet (1 tx every 10 blocks avg, 1250 members stagnating).

@elois

As explained above it doesn’t suit me to do this because it will do me twice as much work (and a cgeek too), because defining a grammar will become obsolete when all documents are in binary format.

• I said “I will draft”, wasn’t asking you to do it.
• It doesnt suit me to install a node I cant read. nor does it suit me to work using a forum held by people so proud they threaten to kick anyone who disagree with them ? DO I HAVE A CHOICE THOUGH!?
• You are totally missing the point! I’ll explain it in a later post but in short: the grammar is there to specify formally the protocol changes
• again with your binary format… dude… like the binarization happens magically, without parser and lexers. Let me make it simple: if you send a message, the receiver will have to read it, and therefore know the set of rules required to do so… you may melt the parsing in your tools and programming stuff the way you want, it doesnt change the FACT that if your node intend to communicate, it’ll need to specify a language others can read and write. unless of course, your plan is a pure and simple technological take over.

@elois

I already have so much work planned on DURS for the next 3 years that I can’t add intermediate changes that would add more work to me.

If I am to participate further in this project, it will not be by waiting for a hero to save us all with a 3 years plan. I mean seriously, you are about to work for years on a piece of code but taking couple hours on specifications is too much? This is so inconsistent with your public presentations that I can’t help but to question your intentions.

I dont have any use for RFCs telling me “HOW I DID IT”, just write a blog post for it.
What I need is a basic consensus on how will nodes communicate, sync. How will the protocol change (unless the Version tag is just a joke) And I need this answer AGNOSTIC to any implementation: the MINIMUM to allow multiple implentations. Just as humans developped languages hundreds thousand years ago, Duniter’s ecosystem will do the same if it is meant to evolve. else its just a brotherhood of nodes talking to each others, like brothers and sisters having incestuous relations, one missing coma somewhere and the whole network is threaten.

Im not on a political quest here, Im on a literate one, those takes more time and are based on knowledge transfer rather than tough assertions “My node is just as fast as bitcoin YOUHA. Wait, why nobody uses it ?”

Exponentials takes a lot of time to become visible, but once they do, they reduce time! right ? Then, lets take our time! solve the missing pieces rather than rush the one true and godly implementation.

[nanocryk]

@anon2076149

You’re suggesting, we have to binarize documents and fix it all. At the same time we make it all unreadable for humans. What a bad long term idea! think twice friend, think twice!

Binarizing the document don’t prevent to have tools to provide readability. Making data taking less place is however a good long term idea; and a better idea than having a big human readable format used practicaly exclusively by the node program.

@anon2076149

binarizing a protocol, that is proven to be broken and having a single implementation?

Where is it proven to be broken ? It’s also normal to have at first one implementation, then followed by others if it gains traction.

@anon2076149

It would persists errors without the need for it yet

It removes many problems induces by a textual format. What errors do you speak about ?
It’s not because we don’t need it yet that we must not prepare. Having a protocol ready to handle a greater traffic is better that patching a protocol currently in use by a great amount of users.

[anon2076149]

@nanocryk

Where is it proven to be broken ?

• ( CLTV(number) || XHX(hash) ) (with ws arounds parenthesis of and/or condition) produces multiple hashs and junk bytes for the same transaction.
• I can create a transaction with a thousands of UTXO and dramatically increase the volume nodes would have to store…
  I only begin to implement and already notice some flaws. I dont blame anyone for the unfinished job, I have to much respect for the planted seed. But it seems pointless to productionize something so young.

@nanocryk

Binarizing the document don’t prevent to have tools to provide readability.

Sure, though, I’d like to emphasize that the persistance layer is purely related to an implementation and usage of that implentation. I wont design the persistance layer of a servicing node which may be used by clients the same way I would design the persistance of a computing/forking node…
Finally from an economic/energetic point of view, you should probably benchmark your solution because Im not at all convinced that a small storage necessarily means more cost efficient / faster node !!!
take the example of many big data system. that do it faster writing buffers on disk than loading them in RAM. not every usages would benefit from smaller datastruct: If I were to serve a million client a day through BMA, Id probably better save the raw JSON on disk and never load them in ram.

Disk is cheaper than ram, disk is more stable than ram. Disk can be extremely efficient on read access and saves RAM, thus avoiding the need for load balancing. And if I were to binarize I’d probably have a set of tools to do it that are not yet available on many language, parquet, avro, … Binarization on persistence layer is the implementation’s business only!!!

If you speak of binarization on the transport/communication layer, I challenge you to do better than the standard GZIP compression already available on any http server by setting “accept-encoding: deflate” or something like that!!! which often use your network card for the job… thus reducing load on CPU

Usually the smaller the data, the faster the processing… but its not true if you have load of preprocessing to unpack your data.

Assuming you do find a better datamodel, smaller than what gzip get you for free, why not. but we’d still need to formalize it so that other implementation use it… unless, again, thats just a take over.

@nanocryk

It’s also normal to have at first one implementation, then followed by others if it gains traction.

Absolutely !

@nanocryk

It removes many problems induces by a textual format

Not if specified properly

@nanocryk

It’s not because we don’t need it yet that we must not prepare

Sure but the way I see it, we’ll be all dead by the time its mainstream so …

What really pisses me off is the idea that because mister Elois said “don’t worry, in 3 years I’ll have written a better implementation” we should stop to improve the current state and expect every used protocol to be deprecated then. I dont see how a new channel binarized if you like it would require to stop another channel. This is just as stupid as having a single implementation. the day Net providers decides to block WS or any specific packets we’re all fucked. Diversity of network protocols and diversity of implementations is pure resilience which is what I expect from the network. Stop trying to deprecate the previous version ASAP. you wanna do better, fine, do it. try to at least. But please dont deprecate features others use… even if we want to improve the DUBP we need capability to handle at least 2 versions at once. And for what I see, there is no solution, no specification of how that would happen.

I’m not trying to implement it in java to see the first hero decide we change it all for an implementation in a langage only one dude knows. je préfère encore la chasse aux UNL que la dictature numérique!

[cgeek]

@anon2076149

I can create a transaction with a thousands of UTXO and dramatically increase the volume nodes would have to store…

You need to have the equivalent thousands of Ğ1. Otherwise the UTXOs, being less than 1 Ğ1, get garbaged and removed from the database.

Also, you should consider that we already create 1200 UTXO per day through the UD. For your attack to represent as much as one year of UDs, given those 1200 UTXOs per day over one year, you need more than 365*1200 = 438.000 Ğ1. We currently have 3.000.000 Ğ1 of monetary mass, your attack require approximately 1/6 of M. Just to be notable.

Currently, this is a non-problem. On long term, probably. That’s why I think Ğ1 should implement @nanocryk’s idea of off-chain UTXO storage.

[anon2076149]

Thanks

@cgeek

You need to have the equivalent thousands of Ğ1. Otherwise the UTXOs, being less than 1 Ğ1, get garbaged and removed from the database.

Is it possible to have a garbage rule saying < max(1, 0.01% of a UD) for instance and have it scale through time? or would that raise another issue ?
Is it less than 1 Ğ1 : 100 cents ? Does it mean the rest of this transaction is dumped ?
https://juniter.bnimajneb.online:8443/graphviz/svg/tx/3FD63BE1AFB8F2338B04DB14BD43DBC9CBB3FD2AFDC51B6FE9D323020761AA7E

[nanocryk]

I speak of binarisation in all layers : transport and storage. A data format close to the memory layout will always be better than a parsed text format.

I’m not working anymore on this format since I’m working on my own much lower level protocol which will indeed define a normal data format. Maybe it will serve for a future version of Duniter, or maybe not and I’ll start my own libre currency.

[anon2076149]

Grammars are not for parsing text

@nanocryk

parsed text format

Reading is only one use of language

@nanocryk

normal data format

what is normal?

@nanocryk

lower level protocol

This is not the UTXO thingy is it ?

[nanocryk]

@anon2076149

Grammars are not for parsing text

I never said that.

@anon2076149

Reading is only one use of language

And ?

@anon2076149

what is normal?

Normal = only one way to write it ?

@anon2076149

This is not the UTXO thingy is it ?

No, but a system I though of after thinking of the UTXO merkle tree structure.

[elois]

@anon2076149

Meaning fork resolution / blocks calculation occurs only through WS2P ?

Exactly. In addition to the local blockchain, only WS2P is required for a functional mirror node. Everything else is optional.

@anon2076149

Why would the DUBP not use various networking utilities?

the notion of network does not exist in the blockchain protocol, it only lists all the rules allowing to ensure the validity of a block.

@anon2076149

Because you dont like REST no one should be allowed to use it ? comon …

Excuse me? Who said we didn’t like REST first? Besides, there is no “we”.
I think you’re too aggressive, it’s just not pleasant. I feel it would be fraud for the project to fit you… it’s up to you to fit what IS.

@anon2076149

Where are the maths showing it?

Don’t exist.

@anon2076149

perhaps would be nice to cleanup our own backyard first !

In this case this problem comes only from the code of @cgeek, I only reproduced the behavior of its code to avoid producing fork.
I do not think he will correct that because he has many other priorities. Especially since the binarization of documents will correct this problem, it is counterproductive to try to correct the parsing of a format that will be abandoned.

@anon2076149

We seriously disagree on the method here. I dont shoot first ask question later… I think a clear spec written out might impulse the changes!

I would like to point out that I am the one who goes the most in this direction to specify everything before coding a single line!
But before writing specifications we must already agree on the destination. What we have done here:

Évolutions du protocole vers v11, v12, etc

Nos dernie…

@anon2076149

You’re suggesting, only @cgeek can impact the development of the Protocol ? be serious please

I can’t be more serious.
Currently only the cgeek implementation works, and it is it that makes Ğ work. Until I finish DURS, only @cgeek can implement blockchain protocol changes. Unless someone takes back control of his code to the point of being able to modify the blockchain protocol, which I highly doubt because this is the most critical part. (I might be able to do it but I won’t do it because I don’t want to take on the maintenance that this is responsible in case of technical problems in production. So personally I will not touch the blockchain protocol part of the Duniter Typescript code.)

@anon2076149

It doesnt suit me to install a node I cant read. nor does it suit me to work using a forum held by people so proud they threaten to kick anyone who disagree with them ? DO I HAVE A CHOICE THOUGH!?

Well, that’s life, man.
It’s the same for me, when I arrived there are many aspects that didn’t suit me in the project, but I don’t have the choice I have to do with what is or do nothing.

@anon2076149

again with your binary format… dude… like the binarization happens magically, without parser and lexers. Let me make it simple: if you send a message, the receiver will have to read it, and therefore know the set of rules required to do so… you may melt the parsing in your tools and programming stuff the way you want, it doesnt change the FACT that if your node intend to communicate, it’ll need to specify a language others can read and write. unless of course, your plan is a pure and simple technological take over.

Please stop thinking I’m stupid. It’s very offensive. I know all this perfectly well and that’s what I’ve already done for the network protocol. The precise and unambiguous language is detailed in this RFC :

rfc/0006_ws2p_v2.md · ws2p_v2 · nodes / common / doc · GitLab

Bon je ne vais plus répondre car ça me soule trop. Je sais que tu lis le français et je te signale @anon2076149 que je rédige justement toujours des spec AVANT de coder, donc arrête de m’accuser a tord et a travers comme ça, tu me manque trop de respect.

Je crois que tu est surtout victime d’une série de quiproquo et incompréhensions, quand je serait calmé il faudrait qu’on échange en privé en français pour démêler tout ces malentendus et clarifier tout ça !

[elois]

@anon2076149 concernant la binarisation du protocole on avait déjà fait un gros travail de défrichage il y a 6 mois :

rfc/0005 Duniter Protocol Rework.md · b31f2e4658ad0c4929c477d3f22c96621c90f5e4...

Au commit d’après @nanocryk a tout supprimer pour partir sur un tout autre concept qui donna finalement lieu a un projet a part (FYgg). Mais la base est toujours là et on la ressortira le moment venu.
Donc c’est un peu fort de café de nous faire la morale alors qu’on a planché des mois sur des specs détaillées…

[cgeek]

@anon2076149

BMA seems to accept POST requests though!

Yes, BMA allows to receive new blocks. But BMA is deprecated now.

@anon2076149

This confirms my suspicions! Where are the maths showing it? I mean, if you consider its fine that anyone knows how to cheat but not others, it becomes seriously difficult to trust anyone in here.

There is absolutely no cheating at all. Nor any flaw about this. It’s just that you can put spaces in your conditions. Yes. That’s all.

@elois

I do not think he will correct that because he has many other priorities.

Well I won’t correct a bug if there is no bug. The fact that we can put spaces in the conditions is consistent with the defined BNF grammar of the protocol: doc/Protocol.md · dev · nodes / typescript / duniter · GitLab

It exists libraries to handle a BNF grammar. So guys, can you tell me what is wrong here? I don’t get it.

Also, I don’t exclude to continue maintaining Duniter and keeping it up with Durs’ Protocol. I just don’t want to participate to the elaboration of this protocol, but I could try to make Duniter following it. And I agree with you and nanocryk that using a binarized format simplifies the grammar, precisely. Notably for conditions. And it will speed up processing as well as optimize memory storage.

[elois]

@cgeek

Well I won’t correct a bug if there is no bug. The fact that we can put spaces in the conditions is consistent with the defined BNF grammar of the protocol: doc/Protocol.md · dev · nodes / typescript / duniter · GitLab

It exists libraries to handle a BNF grammar. So guys, can you tell me what is wrong here? I don’t get it.

I was talking about the bug detected by @max. The fact that Duniter accepts outputs for which parentheses are missing, but that these outputs are never expendable afterwards. This caused me a lot of concern for Durs synchro and I had to create a special “non consumable source” status.