Simplified Verification API


#1

Introduction

Since we want to deprecate BMA, it’s time to think about what should replace it. I’m writing this draft to suggest a first API. Please let me know what you think, what is missing, or what could be done better. Obviously, it has been written from my point of view and my experience in the development of Sakia, and we need the experience of everyone to write the future of BMA.

Duniter Ecosystem

As you know, there are two types of clients in Duniter ecosystem :

  • Naive clients : These clients trust a given node and only one node. It trusts the administrator of the node. In case of cheating or in case of a bug, the client can be mistaken. The advantage of this behaviour is that the client is really smooth and fast. It doesn’t consume too much bandwith, and is compatible with small ressources endpoints.
  • Simplified Verification clients : These clients trust the data they get because they request many nodes. They requests nodes identified by an identity, and as such, they are not sybillizable. The advantage of this behaviour is safety : the client request any data against multiple nodes. If one is cheating or buggy, its data wont match the data of other nodes. In such a case, this data will be ignored

The problem with current API (Basic Merkled API) is that it’s not really merklized :slight_smile: When you require data, you can only get every things, to its smallest details. The consequence is that when you need to verify data, you can only request it again and again against multiple nodes, resulting in endless loading like below :

Requirements for a new API

This new API would make a full usage of Merkle Trees to verify data. As such, every data needs to have its associated merkled tree, to simplify verification against multiple nodes.

Identities data

These datas are requested for a given triplet of [uid/pubkey/blockstamp].
The data we need when we request Identity informations are :

  • Its signature
  • Its blockchain state : written or not-written
  • Its revokation state : revoked or not revoked
    • If revoked :
      • at which block
      • at which timestamp
  • Its distance state : outdistanced or not
  • Its sentry state : sentry or not
  • Its membership state : if a document was sent, which one (IN or OUT)
    • If IN document
      • at which block
      • at which timestamp

Certifications received data

These datas are requested for a given triplet of [uid/pubkey/blockstamp].
The data we need when we request Certifications informations are :

  • Its issuer data :
    • UID
    • Pubkey
    • Blockstamp
    • Signature
    • Member or not
  • Its issuance state :
  • at which block
  • at which timestamp
  • Its written state : written or not
    • if written :
      • at which block
      • at which timestamp

Certifications sent data

These datas are requested for a given triplet of [uid/pubkey/blockstamp].
The data we need when we request Certifications informations are :

  • Its receiver data :
    • UID
    • Pubkey
    • Blockstamp
    • Signature
  • Its issuance state :
  • at which block
  • at which timestamp
  • Its written state : written or not
    • if written :
      • at which block
      • at which timestamp

Transactions received data

These datas are requested for a given [pubkey]. It should be possible to use it with a given [address] too.
The data we need when we request Transactions history are :

  • The raw document
  • The signature
  • The written state : written or not
  • if written :
    • at which block
    • at which timestamp

Transactions sent data

These datas are requested for a given [pubkey].
The data we need when we request Transactions history are :

  • The raw document
  • The signature
  • The written state : written or not
  • if written :
    • at which block
    • at which timestamp

UDTXO (Unspent Dividend or Transaction Outputs Data)

These datas are requested for a given [pubkey]. It should be possible to use it with a given [address] too.
The data we need when we request Avalaible outputs are :

  • The type of output (Dividend or Transaction)
  • The index of the output (Block number or Index in TX document)
  • The identifier of the output (Pubkey or TxHash)
  • The amount of the output (int)
  • The base of the output (int)

Blockchain data

These datas are requested for a given block range [from]/[to]. The data we need when we request the blockchain are :

  • The block number
  • The block hash
  • The block raw document
  • The block signature

Network data

With the arrival of WS2P, Network data is not necessarily handled by this API.

Requests and answers proposal

Requests process

  1. The client sends a “Data Request” to a given node. It gets its answer, formed as a tree of data.
  2. The client builds the Merkled Tree corresponding to this tree
  3. The client send a “Verification request” to N given nodes. The verification request returns the root of the merkle tree.
  4. The client compares the Root received by the verification nodes to the one he built, and validate or unvalidate the data

These requests can be done in parallel. They should be sent to nodes that are on the same blockchain HEAD.

Verification only concerns written state of data. Validity of the data can be insured thanks to hashes and signatures.

General form of the requests

  • The requests should be done using JSON HTTP
  • The URI should distinguish :
    • The type of data requested
    • The source of data : from the pools or from the blockchain. Data from the pools cannot be verified using a merkle tree : there are not guarantees that they are the same on every nodes.
    • Filters (range, uid, pubkey, blockstamp, txhash…)
    • The type of request (Data request or Verification)

API proposal

State requests

These requests have a merkle tree associated. They should be used to verify the state of a given data.

They can be used on data in the blockchain or in the pool, as below :

  • If the data is found in the blockchain, its written field has a value and the hash is computed depending on this value
  • If the data is found in the pool, its written field is empty. The hash is the hash of an empty string : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
  • If the data is not found, it returns a 404

Transaction state

  • /transaction/[data|verify]/[txhash] : Returns the transaction associated to a given tx hash.

Available parameters :

raw: True or False. If True, returns the transaction document and its signature in the results.

Data :

{
  data: {
    written: {
        blockstamp: [Blockstamp],
        timestamp: [Timestamp]
    }
  },
  raw: {
    document: [String],
    signature: [String]
  }
}

Verification :

{
  result: Hash([BlockstampHash],[TimestampHash])
}

Certification state

  • /certification/[data|verify]/[certHash] : Returns the certification associated to a given hash

Available parameters :

raw: True or False. If True, returns the transaction document and its signature in the results.
state: True or False. If True, returns the issuing identity state in the answer
issuance: True or False. If True, returns the certification issuance date in the answer

Data :

{
  data: {
    identity: {
      uid: [str],
      pubkey: [str],
      blockstamp: [str],
      signature: [str],
    },
    state: {
      member: [bool]
    }
    issuance: {
      blockstamp: [Blockstamp],
      timestamp: [int]
    },
    written: {
        blockstamp: [Blockstamp],
        timestamp: [int]
    },
  },
  raw: {
    document: [String],
    signature: [String]
  }
}

Verification :

{
  result: Hash(
      Hash(Hash(member))?,
      Hash(Hash(issuance.blockstamp, issuance.timestamp))?,
      Hash(Hash(written.blockStamp, written.timestamp))
  )
}

Identity state

  • /identity/[data|verify]/[uid]/[pubkey]/[blockstamp] : Returns the identity data

Available parameters :

raw: True or False. If True, returns the identity document and its signature in the results.
state: True or False. If True, returns the full identity state details the answer
issuance: True or False. If True, returns the certification issuance date in the answer

Data :

{
  data: {
    state: {
      member: [bool],
     "revocation": {
        "blockstamp": [Blockstamp],
        "timestamp": [int],
        "signature": [str]
        "written": {
            "blockstamp": [Blockstamp],
            "timestamp": [int],
        }
     }
      "expired": [bool],
      "outdistanced": [bool],
      "sentry": [bool]
    }
    issuance: {
      blockstamp: [Blockstamp],
      timestamp: [int]
    },
    written: {
        blockstamp: [Blockstamp],
        timestamp: [int]
    },
  },
  raw: {
    document: [String],
    signature: [String]
  }
}

Verification :

{
  result: Hash(
      Hash(Hash(member),
                Hash(Hash(revocation.blockstamp),
                    Hash(revocation.timestamp),
                    Hash(revocation.signature)
                    Hash(Hash(revocation.written.blockstamp),
                            Hash(revocation.written.timestamp))?
                    )?,
                Hash(expired),
                Hash(outdistanced),
                Hash(sentry)),
            Hash(Hash(issuance.blockstamp),Hash(issuance.timestamp)),
            Hash(Hash(written.blockstamp),Hash(written.timestamp))
    )
}

Data list requesting

UDTXO list

  • /udtxo/[data|verify]/[pubkey|address] : Returns the UDTXO for a given pubkey or address. Filters the data using from and to parameters.

Available parameters :

pool: True or False. If True, returns also the UDTXO available in the pool of the node

Data :

{
  data: [
    {
      "type": [D or T],
      "noffset": [int],
      "identifier": [identifier],
      "amount": [int],
      "conditions": [str],
      "base": [int]
    },
    {
      ...
    },
  ]

Verification :

{
  result: Hash(UDTXO (Pool excluded))
}

Transactions lists

  • /txhistory/[data|verify]/[pubkey|address] : Returns the transaction history for a given pubkey or address. Filters the data using from and to parameters.

Blockchain lists

  • /blockchain/[data|verify]/[from]/[to] : Returns the blocks for a given range between [from] and [to].

Available parameters :

raw: True or False. If True, returns also the raw block documents

Data :

{
  data: [
    {
       number: [int],
       hash: [str],
       issuer: [str],
       signature: [str]
       raw: [str]
    }
  ]

Verification :

{
  result: BlockHash([to])
}

Articles explicatifs & techniques sur Duniter
#2

At least two other kind of APIs which would be useful to ease the work of clients :

  • An API to get informations about the money, such as :

    • UD reeavalutions history (members, mass, result of computation)
    • Members variation between two UD
  • An API to get :

    • UD issued by a given key (Its broken in latest BMA version and needs to be computed, it takes a long time to do. Cesium can only display it if Cesium+ is activated)
    • Money sources destruction. At the moment, its really hard to compute (it takes a lot of times in Sakia). I’m not sure how we could do it without indexing on the nodes the destructed sources.

#3

Is it possible to keep all functions already available?


#4

The idea is to provide a new API. BMA is deprecated in the last version, and will surelly be removed in future releases.


#5

Peharps but in the new API it is possible to keep the functions already available. It’s not antinomic.


#6

The old API is still there, so you can use “functions already available”. The new API will surely allow to do all the old API can do, only in a different way.

@Inso Je viens de regarder rapidement ton protocole, il m’a l’air bien organisé et ressemble beaucoup plus à un arbre de Merkle. Je regarderais plus en détails plus tard et je te dirais ce que j’en penses (ou aux RML si tu y vas)


#7

Je souhaiterai que cette nouvelle API fournis un statut de chaque nœud/peer, si celui-ci est UP ou DOWN.


#8

Je ne sais pas si ça sera possible, ça dépend surtout de WS2P maintenant. Mais on peut envisager l’ajouter à cette API “orientée clients”.


About the API : I wanted a method to get sources destructions. But it is rather complicated :

  • Sources destruction were implemented to limit the size of the database
  • So we cannot store the destructions which occured in the past (else the database does not have a size limit)

If we don’t want them to trust a particular database, they have to detect the sources destructions on their own… And its really slow (such a client have to parse the whole blockchain).

I’m opened to any idea. My first thought right now is that since we store transactions history, why don’t we store sources destruction history too ? What about the database size ? Can we have two distinct databases such as :

  • One database addressing general network state (finished size, built by reading the stream of blocks, only contains current network state)
  • One database adressing history and data useful to clients (not size-limited, but optional to nodes implementing SVAPI)

#9

Je souhaiterai que mon viewer du réseau (http://www.ğ1.com/ en bas à droite) puisse subsister dans le temps et ne pas faire directement des requêtes sur la base, comme on me le suggère fortement.


#10

Duniter already stores all the unconsumed sources, the api could provide this list directly to clients, this information being lighter than the consumed sources.


#11

I’m talking about sources destroyed because of account being < 1.00


#12

I don’t understand why you need to store these sources there, as long as you have all the unconsumed sources of an account you know its exact balance suddenly on.

Destroyed sources are not included in the index of unused sources so there is no risk of error :slight_smile:


#13

Yes but we have to make them appear in the user history so that he understands what happened :wink:


#14

Small fix about UTXO : it does not need any change in Duniter protocol. Actually, its already possible to do with current protocol (For details, see https://bitcoin.stackexchange.com/questions/37397/where-is-the-utxo-data-stored )


#15

Je soutiens les propositions ci-dessus :

  • Vraies vérifications par Arbre de Merckle
  • Notion d’état des données.

Je propose à nouveau d’utiliser un standard pour l’API.
Je propose GraphQL en HTTP ou Websocket.

Du pour : https://blog.risingstack.com/graphql-overview-getting-started-with-graphql-and-nodejs/
Du contre : https://blog.hitchhq.com/graphql-3-reasons-not-to-use-it-7715f60cb934


#16

La plupart des critiques à graphql ne s’appliquent pas vraiment à Duniter. La blockchain de Duniter ne bouge pas beaucoup, le type du contenu stocké non plus. Ce n’est pas comme une API d’un service web type twitter/github/etc qui change plusieurs fois par mois en fonction des différentes évolutions de service. La blockchain évolue lentement, de part son protocole basé sur un consensus notamment.

Une autre contrainte est d’avoir une API implémentable sur d’autres noeuds que les noeuds Nodejs à terme. GraphQL existe en rust c’est une bonne nouvelle : https://github.com/graphql-rust/juniper

Sur le principe, le format hiérarchique me parait particulièrement adapté effectivement. La requete en mode “merklized” ou “données” peut être gérée à partir des Directive a priori : http://graphql.org/learn/queries/#directives

Erf, il faudrait réécrire toute cette page. Je pense que c’est l’occasion de s’intégrer au process habituel, via une merge request sur : https://git.duniter.org/nodes/common/doc

Tu serais motivé pour réaliser ça @vtexier ?


#17

GraphQL à l’air très sympa, surtout s’il y a une implémentation pour NodeJS et Rust.

Sinon merci de proposer de faire la doc dans le dépot nodes/common/doc, personne ne s’en servait à par moi pour les RFC ^^


#18

J’ai commencé à rédiger une RFC en anglais (accrochez vous !).

Pour la numérotation des RFC, j’ai pris le numéro 003.

rfc/0003 RFC GraphQL API for Duniter Client.md

J’ai proposé le nom GraphQL Verification API, mais c’est à discuter…

Pas les droits pour créer une branche sur le dépôt GitLab…


#19

maintenant si :slight_smile:


#20

Voici le lien vers le document en cours de rédaction.

Pour profiter des exemples de code, je conseil d’aller dans vos settings/preferences et de choisir la couleur de coloration de la syntaxe en “dark” :

https://git.duniter.org/profile/preferences

Enjoy !