Squid pour Duniter, épisode 2

Je discute avec lui sur Telegram du coup.

Hormis le fait qu’il essai de me vendre leur solution payante de cloud ArrowSquid, voici une partie de ses réponses qui semblent confirmer les propos de @vit . Il semble aussi dire qu’on peut tout à fait remplacer le serveur graphql de squid qui est optionnel, par hasura.

The ArrowSquid solution is a decentralized data lake, where the whole historical data is compressed and distributed across a swarm of workers. This is what makes the indexing part so fast, as you don’t ingest this data from the RPC but rather from a data lake

As it only can be deployed to a decentralized network, there’s no option to self-host this part

Yet, you can use the ArrowSquid sdk using only the RPC, but it’s slow

the sdk is open-sourced

in this repo this seems to be exactly the case — RPC-only setup, which indeed you can fully self-host without integrating to the data lake (which would enable the archive: endpoint)

as for the FTS queries — yes, you can certainly do that, either by using the built-in PG capabilities or additionally pushing the data to ElasticSearch or smth else from your processor

you can also ditch our built-in graphql server and go for hasura if you want, it’s 100% optional

Sound squid is quite flexible. thank you for this confirmation.
I don’t understand why arrowsquid swarm should offer faster solution than simple RPC setup. Do you have a technical documentation about the way ArrowSquid works ?

the data lake enables batch-queries, so you extract tens of thousands of events from hundreds of blocks in a single call, then batch-transform it. So it normally takes just a couple of minutes vs hours/days from the rpc

this is API it exposes: https://docs.subsquid.io/archives/substrate/

the rpc essentially goes through each and every block so the longer is the chain the more requests you’d need to backfill

thank you for this informations, we will discuss about all of this :slight_smile:
I think we will try to replace you graphql server by Hasura, but that mean we need to dig how squid is build.

why? you can just spin it on top of the db, and you have full control over the schema (it’s in the migrations folder)

so integrating hasura should be very straightforward

In fact I didn’t work on this part yet, just my friends, so yes maybe it’s just simple

let me know if you need any help!

thank you :slight_smile: