Why we need a full featured indexer?

Nodes Indexers
1

After the discussion on Point d'étape sur la Ğ1v2, it appears that the urgent need of a full featured indexer is not obvious. I will detail here:

  • TypeScript: this will be of great help to use typescript everywhere in the indexer code. It will make future developments easier and code reuse more efficient.
  • smith-related events: with the goal to welcome more smith in ĞDev network, we have to build tools making easier to read the state of smith subwot and smith statuses. Some info can be found in a very tedious way in polkadotjs, some other can not be found at all (like the cert history of smiths, the history of declared metadata (endpoints, session keys)… This is necessary for @guenoel’s internship.
  • technical-committee-related events: while polkadotjs has good support for current votes, we need to get an history of the ĞDev votes who were submitted, expired, were accepted or rejected, and succeeded or failed. For example last vote to update technical committee failed with bad origin because we did not wrap it in upgrade origin, and this is very difficult to find this information back. Indexing these events is a limiting factor to migrate to Ğ1v2 and is urgent to explain the role of the technical committee and discuss its scope.

Here is a non-exhaustive list of events with checks when implemented. Please, precise if it is in your whishlist with a :pray:.

  • duniter
    • authorities :pray:
      • IncomingAuthorities
      • OutgoingAuthorities
      • MemberGoOffline
      • MemberGoOnline
      • MemberRemoved
    • cert
      • NewCert
      • RemovedCert :pray:
      • RenewedCert :pray:
    • duniter-account
      • ForceDestroy
      • RandomIdAssigned
    • identity
      • IdtyCreated
      • IdtyConfirmed
      • IdtyValidated
      • IdtyChangedOwnerKey
      • IdtyRemoved
    • membership
      • MembershipAcquired
      • MembershipExpired
      • MembershipRenewed
      • MembershipRequested
      • MembershipRevoked
      • PendingMembershipExpired
    • oneshot account
      • OneshotAccountCreated
      • OneshotAccountConsumed
      • Withdraw
    • provide randomness
      • FilledRandomness
      • RequestedRandomness
    • universal dividend
      • NewUdCreated
      • UdReevalued
      • UdsAutoPaidAtRemoval
      • UdsClaimed
    • upgrade origin
      • DispatchedAsRoot
  • substrate
    • frame-system
      • ExtrinsicSuccess :pray:
      • ExtrinsicFailed :pray:
      • NewAccount
      • KilledAccount :pray:
      • Remarked
    • pallet-atomic-swap
    • pallet-authority-discovery
    • pallet-authorship
    • pallet-babe
    • pallet-balances
      • Endowed
      • DustLost
      • Transfer
      • BalanceSet
      • Reserved
      • Unreserved
      • ReserveRepatriated
      • Deposit
      • Withdraw
      • Slashed
    • pallet-collective :pray:
      • Proposed
      • Voted
      • Approved
      • Disapproved
      • Executed
      • MemberExecuted
      • Closed
    • pallet-grandpa
      • NewAuthorities
      • Paused
      • Resumed
    • pallet-im-online
      • HeartbeatReceived
      • AllGood
      • SomeOffline
    • pallet-offences :pray:
      • Offence
    • pallet-multisig
      • NewMultisig
      • MultisigApproval
      • MultisigExecuted
      • MultisigCancelled
    • pallet-preimage
      • Noted
      • Requested
      • Cleared
    • pallet-proxy
      • ProxyExecuted
      • PureCreated
      • Announced
      • ProxyAdded
      • ProxyRemoved
    • pallet-scheduler
      • Scheduled
      • Canceled
      • Dispatched
      • CallUnavailable
      • PeriodicFailed
      • PermanentlyOverweight
    • pallet-session
      • NewSession :pray:
    • pallet-sudo :pray:
      • Sudid
      • KeyChanged
      • SudoAsDone
    • pallet-timestamp
    • pallet-transaction-payment
      • TransactionFeePaid :pray:
    • pallet-treasury :pray:
      • Proposed
      • Spending
      • Awarded
      • Rejected
      • Burnt
      • Rollover
      • Deposit
      • SpendApproved
      • UpdatedInactive
    • pallet-utility :pray:
      • BatchInterrupted
      • BatchCompleted
      • BatchCompletedWithErrors
      • ItemCompleted
      • ItemFailed
      • DispatchedAs

Because implementing all of this manually is a great effort, I think we should combine two strategies: a custom indexer with manual implementation and a generic indexer like Elois did setup with hasura to work with for specific applications that do not require manual implementation.

If somebody wants to work on the generic approach:

5 Likes
2

Maybe we should organize a visio videoconferencing to discuss about indexer features, and make a choice between implement all theses features in v2s-indexer (homemade) or use Subsquid (maybe a better choice for common substrate features) ?

Maybe is there a way to merge these 2 API for the clients ?
It’s a bit of strange to maintain 2 indexers kind which have almost same scope.

1 Like
3

Not really: for generic indexer, you can be satisfied with raw json output, but with custom indexer, you would be more confortable with shortcuts.
It’s a shame I’m incapable of setting up what elois did with hasura (Subsquid (ancienne Hydra): indexer de blockchain substrate exposant des API GraphQL). I’m too bad at docker :cry:
Did you try it before it went down?

4

SHAME ON YOUUUUU !!! :grin:

I’m confused with the prometheus/grafana stack which is absolutely hellish and which I have never been able to make work properly with substrate.

No I don’t think I’ve ever tested to launch Hydra, I’ll try it within the week.

1 Like
5

My question was: did you see the result of hydra? All the events and all the extrinsics indexed in a generic manner. It was super helpful.

6

No, I admit that I ran headlong into it on the realization of Magic Manu.

You’re right, we have to dig Hydra for sure.

7

Request:

{
  batch(
    limit: 10
    includeAllBlocks: false
    events: [{ name: "SmithsMembership.MembershipRequested" }]
  ) {
    header {
      height
      hash
    }    
  }
}

Response:

{
  "data": {
    "batch": [
      {
        "header": {
          "height": 632386,
          "hash": "0x2865b04a69d7da834c271d1e7fff1fb98eaac57305010840e7cb78efb5a0f59c"
        }
      },
      {
        "header": {
          "height": 917080,
          "hash": "0xd97803a1787f93951370a1973470b7e7865164e3af6785f28e7ed868525ba1e4"
        }
      },
      {
        "header": {
          "height": 917151,
          "hash": "0x61067bb524f647fe4c9f0dde3038a2e6a8ae32cd04624859fdd2ae98aebe11f8"
        }
      }
    ]
  }
}

You can look at blocks

and find the p2pEndpoint declared in the metadata:

  • /dns/gdev-smith.pini.fr/tcp/443/wss/p2p/12D3KooWCEVBrLK9g8unsHLj8wWzobbwArDZMMwg5LeCbiWhB3ug
  • gdev.trentesaux.fr
  • _

We have to rethink the usage of these fields and @guenoel it’s something we have to integrate in your app (read + write).

2 Likes