Add balances to squid accounts

Nodes Indexers
1

In v1 using GVA we can get the transactions and the balance in a single graphql query. You know well @poka:

But now in v2 we have to query an endpoint to get the free balance, do these DU calculations (thanks BTW), and also get the transaction history. IMHO, this is a source of problems for instance if things are out of sync.

Currently I have a bug in my development environment and I get zero for all the v2 balance queries, while transactions history are working.

Or I can imagine querying two different servers (duniter endpoint and indexer), with data not totally in sync and getting a different balance compared with transactions.

Why can we not get the total balance in the indexers too without doing the query to polkadot on-chain?

So in one query we can get the history and the balance (including DUs, etc).

Because visiting other users’ profiles (or if you have multiple wallets) and the need to do so many calculations and queries to get the history and balance, it seems too much computational work for clients and also a source of discrepancy. It sounds to me like something reasonable to do in a indexer.

I don’t know if this was discussed already. Sorry for the newbie question.

2 Likes
2

We can add balances to account in squid.

The only problem for me is that we do not have event triggered when balance changed which announce new balance value, as far as I know.

So the only way would be to compute balance on each transactions, and add UDs when they are created also, for each member accounts.

I understand the practical side to use same API for balance en tx, but this is not exactly the scope of the indexer. It should not be used for mutable values.

I’ve created an MR that adds the balance of accounts, but it remains to be seen whether all the balances are correct in all cases, and whether it’s really relevant to add this kind of mutable data to the indexer.

image
image1782Ă—368 36.9 KB

3 Likes
3

Sans surprise, une fois mon indexer local synchronisé, aucun solde n’est correct.

{
  "data": {
    "account": [
      {
        "id": "5Dq8xjvkmbz7q4g2LbZgyExD26VSCutfEc6n4W4AfQeVHZqz",
        "balance": 5602021717886,
        "identity": {
          "name": "HugoTrentesaux"
        }
      }
    ]
  }
}

image
image252Ă—139 11.7 KB

BC: 56020217440.77
SQ: 56020217178.86

Différence: 261.91

La première explication est que les frais ne sont pas pris en compte dans la somme des transferts. Mais ça n’explique pas toute la différence.

Donc mon approche n’est pas fiable, je pense qu’il faudrait ajouter un event de balanceChange à duniter qui diffuse la nouvelle valeur balance.free à chaque changement d’état. L’indexer n’aurait donc qu’a écouter cet evenement, mettre à jour la nouvelle valeur pour le compte en y ajoutant simplement les DU non réclamés.

Problème: Ces nouveaux events participeront à augmenter encore la taille de la blockchain.
Et je maintiens que ce n’est pas au rôle de l’indexer de gérer des données mutables.

Dans notre squid je vois des events concernant les balances, mais je ne les ai jamais observé pour de vrai en blockchain:

Balances.Endowed : An account was created with some free balance.
Balances.DustLost : An account was removed whose balance was non-zero but below ExistentialDeposit, resulting in an outright loss.
Balances.Transfer : Transfer succeeded.
Balances.BalanceSet : A balance was set by root.
Balances.Reserved : Some balance was reserved (moved from free to reserved).
Balances.Unreserved : Some balance was unreserved (moved from reserved to free).
Balances.ReserveRepatriated : Some balance was moved from the reserve of the first account to the second account. Final argument indicates the destination balance type.
Balances.Deposit : Some amount was deposited (e.g. for transaction fees).
Balances.Withdraw : Some amount was withdrawn from the account (e.g. for transaction fees).
Balances.Slashed : Some amount was removed from the account (e.g. for misbehavior).
Balances.Minted : Some amount was minted into an account.
Balances.Burned : Some amount was burned from an account.
Balances.Suspended : Some amount was suspended from an account (it can be restored later).
Balances.Restored : Some amount was restored into an account.
Balances.Upgraded : An account was upgraded.
Balances.Issued : Total issuance was increased by amount, creating a credit to be balanced.
Balances.Rescinded : Total issuance was decreased by amount, creating a debt to be balanced.
Balances.Locked : Some balance was locked.
Balances.Unlocked : Some balance was unlocked.
Balances.Frozen : Some balance was frozen.
Balances.Thawed : Some balance was thawed.
Balances.TotalIssuanceForced : The TotalIssuance was forcefully changed.

Aucun d’entre eux ne sont traités (hormis le Balance.tranfer) pour le moment dans notre squid.

Mais typiquement quand je fais un transfer, je ne vois que l’event de transfer (celui que j’utilise donc actuellement pour sommer le montant aux comptes), rien d’autre:

image
image870Ă—327 19 KB

ok pour la science, je relance une sync en ayant ajouté la gestion de ces events, pour les frais et les réserves, où je somme avec les balances existantes, on verra ce que ça donne, mais bon…

balances.deposit
balances.withdraw
balances.reserved
balances.unreserved
balances.endowed
balances.dustLost
balances.slashed

3 Likes
4

I started a developer documentation (Duniter | Developer guide) to explain these kind of things, but I have less time (Je ne suis plus à plein temps!), so it’s moving slowly.

To get the free balance, you have to use storage.system.account and not storage.balances.account. This does not account for unclaimed UDs.

It is the goal to be able to get balance from indexer (#11) but that’s not an easy task. So for now, it makes sense to ask directly to Duniter and compute unclaimed UDs.

Because it is not done yet. And it is not done yet because it is not easy. From indexer point of view, to get the balance, you have to handle for every kind of balance change event or directly listen on storage. And if you want to have unclaimed UDs you have to update as many accounts as there are identities at every new UD.

This “unclaimed UD” optimization was necessary (Duniter side) and discussed here:

If only all “newbies” were developing a full Duniter client :joy_cat:

Due to the nature of blockchain, we have to keep things quite minimal, especially what we want to happen at a high rate (money transfers). That’s the very reason of the UD optimization.

Yes it is. The indexer role is to make app development as easy as possible and to counterbalance the “minimalistic” nature of blockchain by providing as rich features as affordable. See #36 for indexer optimizations.

read more on pallet balances

For people who want to read more:

You have a bit more info on the events in the doc https://paritytech.github.io/polkadot-sdk/master/pallet_balances/pallet/enum.Event.html.

If you want examples:

> https://squid.gdev.coinduf.eu/v1/graphql
query Events($name: String!) {
  event(where: {name: {_eq: $name}}, limit: 1) {
    id
  }
}
{"name": "Endowed"}

And replace name by the event you want to test.

TODO check why UniversalDividend.UdsClaimed does not come with Balances.Issued but Balances.Deposit.

I’ll have a look at your MR !27. I hope to see unclaimed UD indexing ^^.

3 Likes
5

Yes this is done and work well. Pretty optimize (I think) and very simple thanks to squid event parsing and storage architecture (with iterative cache).

But still little difference between real balance and compute balance, have to investigate.

6

Un premier problème est que les clamUd sont diffusés dans les events Deposit.
Donc ça fait doublon avec les UD créés. Il faut donc soit:

  • Ne pas considĂ©rer le event Deposit. On risque peut ĂŞtre de louper des choses non comptabilisĂ©s, je ne sais pas
  • Trouver un moyen d’identifier les Deposit de claimUd spĂ©cifiquement pour les ignorer. Pas trouvĂ© comment faire
  • Ne pas comptabiliser les crĂ©ation de DU, et dans ce cas on comptabilise les DU rĂ©clamĂ©s via Deposit + calcul Ă  la volĂ©e des DU non rĂ©clamĂ©s restant. NĂ©cessite de faire des appels storage dans l’indexer, chiant et pas opti. (ou bien plutĂ´t d’indexer les lastEligibleUd pour que identity et le currendUdIndex, j’imagine qu’il y a un event pour chacune de ces valeurs)
query Events($name: [String!]!) {
  event(
    where: {name: {_in: $name}}
    limit: 10
    orderBy: {block: {timestamp: DESC}}
  ) {
    args
    name
    block {
      timestamp
    }
  }
}

{
  "name": [
    "Endowed",
    "DustLost",
    "Transfer",
    "BalanceSet",
    "Reserved",
    "Unreserved",
    "ReserveRepatriated",
    "Deposit",
    "Withdraw",
    "Slashed",
    "Minted",
    "Burned",
    "Suspended",
    "Restored",
    "Upgraded",
    "Issued",
    "Rescinded",
    "Locked",
    "Unlocked",
    "Frozen",
    "Thawed",
    "TotalIssuanceForced"
  ]
}

{
  "data": {
    "event": [
      {
        "args": {
          "who": "0x0ed0734a282c8d3551694d74e12f6ec9a568770ad351a67303994908b638071b",
          "amount": "3297995595286"
        },
        "name": "Deposit",
        "block": {
          "timestamp": "2024-12-18T18:52:18.002+00:00"
        }
      },
      {
        "args": {
          "who": "0x592b0961c7331c31edaa90d43f2a5659ec91e7cff0324ca04211050d3f43153c",
          "amount": "6825372531980"
        },
        "name": "Deposit",
        "block": {
          "timestamp": "2024-12-18T17:33:42.001+00:00"
        }
      },
      {
        "args": {
          "who": "0x18ed9f9bfbc28cd415d854a9210f16e948b0cdc21028389db4b03f0bf22b8f6f",
          "amount": "6762288716151"
        },
        "name": "Deposit",
        "block": {
          "timestamp": "2024-12-18T16:38:18.001+00:00"
        }
      },
      ...
    ]
  }
}

0x0ed0734a282c8d3551694d74e12f6ec9a568770ad351a67303994908b638071b c’est moi 5CQ8T4qpbYJq7uVsxGPQ5q2df7x3Wa4aRY6HUWMBYjfLZhnn qui vient de faire un clamUd par exemple.

image
image842Ă—231 15.8 KB

1 Like
7

I read quickly your MR and saw this:

      // Get all member identities
      const memberIdentities = await ctx.store.find(Identity, {
        relations: { account: true },
        where: { isMember: true }
      });

      // Distribute UD to each member
      for (const identity of memberIdentities) {
        await this.updateAccountBalance(ctx, identity.account!.id, amount); // we are sure that a non-removed identity has non-null account
      }

This will not work because of this:

I think the right way to do is to do like Duniter: index the “last claimed UD” as well as all balance events and provide the total balance through a computed field. An other option would be to have the balance split into two field: one the actual balance and the other the virtual difference that would be adjusted:

  • on each UD claim (to avoid accounting for the same currency multiple times)
  • on each UD creation (to account for newly created UD)

Not sure however if the optimization is worth the additional complexity.

1 Like
8

C’est donc cette solution qui a été retenu:

{
  "data": {
    "account": [
      {
        "id": "5Dq8xjvkmbz7q4g2LbZgyExD26VSCutfEc6n4W4AfQeVHZqz",
        "balance": 19958,
        "totalBalance": 75108,
        "identity": {
          "name": "HugoTrentesaux",
          "isMember": true,
          "firstEligibleUd": 23
        }
      }
    ]
  }
}

A voir si j’ai bien traité tous les cas de changement de firstEligibleUd correctement.

Ca nous fait une jolie nouvelle petite requête SQL associé au computed field Hasura: assets/sql/totalBalanceFunction_up.sql · account-balance · nodes / duniter-squid · GitLab

1 Like
9

J’ai enfin une version de l’indexer fonctionnelle avec les bonnes balances totales :slight_smile:

J’ai dû faire un peu de refac sur la manière dont on index les comptes et les identités (exit le hack de circular dependencies entre account et linked identity).

Testable sur ce endpoint: https://gdev-squid.axiom-team.fr/v1/graphql

J’ai dû pousser les images docker sur mon repo perso car je n’ai pas les droits sur le repo principal: https://hub.docker.com/r/poka/duniter-squid et https://hub.docker.com/r/poka/duniter-squid-hasura.

Je vous laisse vérifier que les totalBalance sont exactement identique avec ce que vous affiche Ğecko ou duniter portal.

Exemple:

query {
  account(where: {id: {_eq: "5Dq8xjvkmbz7q4g2LbZgyExD26VSCutfEc6n4W4AfQeVHZqz"}}) {
    id
    balance
    totalBalance
    identity {
      name
      firstEligibleUd
    }
  }
}

{
  "data": {
    "account": [
      {
        "id": "5Dq8xjvkmbz7q4g2LbZgyExD26VSCutfEc6n4W4AfQeVHZqz",
        "balance": 2969654943045,
        "totalBalance": 8813573540437,
        "identity": {
          "name": "HugoTrentesaux",
          "firstEligibleUd": 1823
        }
      }
    ]
  }
}

J’aurais bien fait un script de test unitaire standalone pour tester ça en masse, mais je crois que gcli ne compute pas encore les balance totales…

Quelques précisions:

  • Je ne comprends pas la nĂ©cessitĂ© de ce commit: add balance to accounts (!27) · Merge requests · nodes / duniter-squid · GitLab
    Je dois ajouter artificiellement 1 DU de dernière génération pour atteindre les montants correspondants à mes implémentations Dart et TypeScript.
    Donc soit il y a un bug de toutes mes implémentation qui ajoute un DU aux balances, soit il y a un comportement que je n’explique qui fait qu’en SQL, malgré un algo strictement identique, il manque un DU sans cet ajout.
  • On pourrait refac le schema de donnĂ©e pour faire plutĂ´t un field balance avec des sous fields:
    "balance": {
  "free": 42,
  "total": 4242,
  "reserved": 0,
  ect...
}
  • Le total est calculĂ© Ă  la volĂ©e lorsqu’on fait la requĂŞte. Ça a pour consĂ©quence d’avoir une très faible empreinte en Ă©criture pendant l’indexation, mais demande un peu de ressource serveur pendant la requĂŞte.
    Pour requêter un compte on est sûr ~14ms pour le totalBalance contre ~13ms pour le balance, donc totalement négligeable.
    Par contre, pour orderBy ou filtrer sur totalBalance, comme il doit computer pour tous les 40 000 accounts, on est sur ~4.5s contre ~75ms pour un orderBy balance.
    Ca reste acceptable étant donnée que ce n’est pas une requête qu’on va faire souvent, mais à garder en tête si un jour on veut optimiser ce genre de requête de masse sur les totalBalance, il faudra probablement refac pour préférer stocker et indexer les totalBalance lors de l’indexation en remplacement du computed field.
5 Likes
10

Bravo et Merci !

1 Like
11

I finally did the review of your MR!! add balance to accounts (!27) · Merge requests · nodes / duniter-squid · GitLab. Small things to change but after that, ready to go to prod ^^

3 Likes