Mon noeud forgeron a été offline (Hugo) → faute d'espace disque

HugoTrentesaux · 12 March 2025 22:53

Sur un serveur auquel je ne touche jamais (même pendant un déménagement ), mon noeud forgeron v2 semble avoir été offline. Investigation.

Offence

Je regarde les dernières offences “à la main” avec squid :

query Offences {
  event(where: {pallet: {_eq: "Offences"}}, limit: 20, orderBy: {block: {height: DESC}}) {
    block { height}
    pallet
    name
    args
  }
}

Parmi les blocs avec des offences, le dernier qui me concerne est le numéro 5063508, qui date du 4 février.

Logs système

Le serveur est bien en marche depuis longtemps, mais il est arrivé qqc au nœud il y a 12 jours, soit le 28 février.

$ history
# rien d'intéressant récemment, je ne touche pas à cette machine
$ uptime
 21:23:22 up 493 days, 20:28,  1 user,  load average: 0.15, 0.31, 0.32
$ docker compose ps
NAME                                 IMAGE                                 COMMAND               SERVICE         CREATED        STATUS       PORTS
duniter-gdev-smith-duniter-smith-1   duniter/duniter-v2s-gdev-800:latest   "docker-entrypoint"   duniter-smith   3 months ago   Up 12 days   127.0.0.1:9615->9615/tcp, 127.0.0.1:9944->9944/tcp, 0.0.0.0:30333->30333/tcp, :::30333->30333/tcp
$ sudo zless daemon.log.2.gz
Feb 28 06:20:52 trentesaux systemd[1]: Starting Daily apt upgrade and clean activities...
Feb 28 06:21:08 trentesaux systemd[1]: docker-92653af5cf0d558b72a9c56e481383d2725644dffcad8d8e38d93e3f415fce8a.scope: Succeeded.
Feb 28 06:21:08 trentesaux systemd[1]: docker-92653af5cf0d558b72a9c56e481383d2725644dffcad8d8e38d93e3f415fce8a.scope: Consumed 3d 9h 59min 57.603s CPU time.
Feb 28 06:21:10 trentesaux systemd[1]: run-docker-netns-977c5d45acbc.mount: Succeeded.
Feb 28 06:21:10 trentesaux systemd-udevd[3654697]: veth5f8eaa1: Failed to query device driver: No such device
Feb 28 06:21:10 trentesaux systemd-udevd[3654697]: veth5f8eaa1: Failed to get link config: No such device
$ dpkg -l | grep unattended
ii  unattended-upgrades            2.8                                all          automatic installation of security upgrades
$ journalctl -x
Jan 09 14:06:23 trentesaux systemd[1763857]: pam_unix(systemd-user:session): session closed for user hugo
Mar 12 21:13:05 trentesaux systemd[3859294]: Queued start job for default target Main User Target.
# (on voit un gros trou dans  les logs)

Des mises à jour automatiques peuvent redémarrer docker avec un downtime relativement petit, ce qui reste acceptable pour Duniter. Il faut aller chercher ce qui a causé une plus longue interruption. Mais ma config debian ne garde qu’un mois de logs, donc je ne peux pas remonter jusque là dans les logs système.

Logs Duniter

Je vais autour du 4 février :

duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:44 💤 Idle (39 peers), best: #5062740 (0x434b…6b15), finalized #5062738 (0xfb81…dda1), ⬇ 68.7kiB/s ⬆ 32.2kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:48 🏆 Imported #5062741 (0x434b…6b15 → 0xb2fd…7fa0)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:48 ♻️  Reorg on #5062741,0xb2fd…7fa0 to #5062741,0x5cb1…9c61, common ancestor #5062740,0x434b…6b15    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:48 🏆 Imported #5062741 (0x434b…6b15 → 0x5cb1…9c61)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:49 💤 Idle (39 peers), best: #5062741 (0x5cb1…9c61), finalized #5062738 (0xfb81…dda1), ⬇ 71.3kiB/s ⬆ 30.6kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:54 💤 Idle (39 peers), best: #5062741 (0x5cb1…9c61), finalized #5062739 (0x07f8…5db4), ⬇ 91.3kiB/s ⬆ 27.7kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:54 🏆 Imported #5062742 (0x5cb1…9c61 → 0xf2e5…864e)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:57:59 💤 Idle (39 peers), best: #5062742 (0xf2e5…864e), finalized #5062740 (0x434b…6b15), ⬇ 68.7kiB/s ⬆ 25.0kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:00 🏆 Imported #5062743 (0xf2e5…864e → 0x1de7…3c42)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:00 ♻️  Reorg on #5062743,0x1de7…3c42 to #5062743,0x00e3…66f4, common ancestor #5062742,0xf2e5…864e    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:00 🏆 Imported #5062743 (0xf2e5…864e → 0x00e3…66f4)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:04 💤 Idle (39 peers), best: #5062743 (0x00e3…66f4), finalized #5062741 (0x5cb1…9c61), ⬇ 76.4kiB/s ⬆ 28.0kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:06 🙌 Starting consensus session on top of parent 0x00e3c3a7593a296b3e94d08b2da11373c1b3bcc9fa5593d4eb72a8d21e9a66f4 (#5062743)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:06 🎁 Prepared block for proposing at 5062744 (2 ms) [hash: 0xada5a4bee826350849ee359313b31d1d87aa7ccff6ad3f001b874064d85ffd35; parent_hash: 0x00e3…66f4; ex
trinsics (1): [0x0cd1…7d8d]    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:06 🔖 Pre-sealed block for proposal at 5062744. Hash now 0x9c40ed5eca07932ea6e599752b3a5b80489d6cec40514535a0057ae605bb65ec, previously 0xada5a4bee826350849
ee359313b31d1d87aa7ccff6ad3f001b874064d85ffd35.    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:06 🏆 Imported #5062744 (0x00e3…66f4 → 0x9c40…65ec)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:06 🆕 Imported #5062744 (0x00e3…66f4 → 0x699a…c588)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:09 💤 Idle (39 peers), best: #5062744 (0x9c40…65ec), finalized #5062742 (0xf2e5…864e), ⬇ 77.1kiB/s ⬆ 32.7kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:12 🏆 Imported #5062745 (0x9c40…65ec → 0xab16…e18c)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:14 💤 Idle (39 peers), best: #5062745 (0xab16…e18c), finalized #5062743 (0x00e3…66f4), ⬇ 79.9kiB/s ⬆ 32.2kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:18 🏆 Imported #5062746 (0xab16…e18c → 0x853a…ad44)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:19 💤 Idle (39 peers), best: #5062746 (0x853a…ad44), finalized #5062744 (0x9c40…65ec), ⬇ 77.6kiB/s ⬆ 31.0kiB/s    
duniter-gdev-smith-duniter-smith-1  | Node key file '/var/lib/duniter/node.key' exists.
duniter-gdev-smith-duniter-smith-1  | Node peer ID is '12D3KooWBUofnBCndckssxLfMgygqioJTGWdfjkmjb8bDtjezWFE'.
duniter-gdev-smith-duniter-smith-1  | Starting duniter with parameters: --name HugoTrentesaux-smith --node-key-file /var/lib/duniter/node.key --public-addr /dns/gdev.trentesaux.fr/tcp/30333 --listen-addr /ip4/0.
0.0.0/tcp/30333 --rpc-cors all --rpc-methods Unsafe --validator --blocks-pruning 14400 --chain gdev -d /var/lib/duniter --unsafe-rpc-external
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 Duniter    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 ✌️  version 0.9.1-unknown    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 ❤️  by librelois <c@elo.tf>:tuxmain <tuxmain@zettascript.org>:c-geek <https://forum.duniter.org/u/cgeek>:HugoTrentesaux <https://trentesaux.fr>:bgallois <
benjamin@gallois.cc>:Duniter Developers <https://duniter.org>:Axiom-Team Developers <https://axiom-team.fr>, 2021-2025    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 📋 Chain specification: ĞDev    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 🏷  Node name: HugoTrentesaux-smith    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 👤 Role: AUTHORITY    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 00:58:23 💾 Database: ParityDb at /var/lib/duniter/chains/gdev/paritydb/full    
duniter-gdev-smith-duniter-smith-1  | Node key file '/var/lib/duniter/node.key' exists.
duniter-gdev-smith-duniter-smith-1  | Node peer ID is '12D3KooWBUofnBCndckssxLfMgygqioJTGWdfjkmjb8bDtjezWFE'.
duniter-gdev-smith-duniter-smith-1  | Starting duniter with parameters: --name HugoTrentesaux-smith --node-key-file /var/lib/duniter/node.key --public-addr /dns/gdev.trentesaux.fr/tcp/30333 --listen-addr /ip4/0.
0.0.0/tcp/30333 --rpc-cors all --rpc-methods Unsafe --validator --blocks-pruning 14400 --chain gdev -d /var/lib/duniter --unsafe-rpc-external

Après un fonctionnement normal, le nœud se met à redémarrer en boucle. La fin du redémarrage en boucle se situe quatre heure plus tard :

duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:19:57 🏷  Node name: HugoTrentesaux-smith    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:19:57 👤 Role: AUTHORITY    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:19:57 💾 Database: ParityDb at /var/lib/duniter/chains/gdev/paritydb/full    
duniter-gdev-smith-duniter-smith-1  | Node key file '/var/lib/duniter/node.key' exists.
duniter-gdev-smith-duniter-smith-1  | Node peer ID is '12D3KooWBUofnBCndckssxLfMgygqioJTGWdfjkmjb8bDtjezWFE'.
duniter-gdev-smith-duniter-smith-1  | Starting duniter with parameters: --name HugoTrentesaux-smith --node-key-file /var/lib/duniter/node.key --public-addr /dns/gdev.trentesaux.fr/tcp/30333 --listen-addr /ip4/0.
0.0.0/tcp/30333 --rpc-cors all --rpc-methods Unsafe --validator --blocks-pruning 14400 --chain gdev -d /var/lib/duniter --unsafe-rpc-external
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 Duniter    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 ✌️  version 0.9.1-unknown    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 ❤️  by librelois <c@elo.tf>:tuxmain <tuxmain@zettascript.org>:c-geek <https://forum.duniter.org/u/cgeek>:HugoTrentesaux <https://trentesaux.fr>:bgallois <
benjamin@gallois.cc>:Duniter Developers <https://duniter.org>:Axiom-Team Developers <https://axiom-team.fr>, 2021-2025    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 📋 Chain specification: ĞDev    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 🏷  Node name: HugoTrentesaux-smith    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 👤 Role: AUTHORITY    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:20:58 💾 Database: ParityDb at /var/lib/duniter/chains/gdev/paritydb/full    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 Local node identity is: 12D3KooWBUofnBCndckssxLfMgygqioJTGWdfjkmjb8bDtjezWFE    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 Running litep2p network backend    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 👶 Starting BABE Authorship worker    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 Operating system: linux    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 CPU architecture: x86_64    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 Target environment: gnu    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 CPU cores: 2    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 Memory: 2067MB    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 Kernel: 5.10.0-26-amd64    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 Linux distribution: Debian GNU/Linux 11 (bullseye)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 💻 Virtual machine: yes    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 📦 Highest known block at #5062746    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 〽️ Prometheus exporter started at 127.0.0.1:9615    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 Running JSON-RPC server: addr=0.0.0.0:9944,[::]:45377    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:04 ***** Duniter has fully started *****    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:05 🔍 Discovered new external address for our node: /ip4/45.147.99.153/tcp/30333    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:09 ⚙️  Preparing, target=#5065268 (8 peers), best: #5063385 (0xdeba…8ebd), finalized #5063168 (0x93b8…21e8), ⬇ 175.4kiB/s ⬆ 4.2kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:14 🔍 Discovered new external address for our node: /dns/gdev.trentesaux.fr/tcp/30333/p2p/12D3KooWBUofnBCndckssxLfMgygqioJTGWdfjkmjb8bDtjezWFE    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:14 ⚙️  Preparing 133.4 bps, target=#5065269 (9 peers), best: #5064052 (0xb1eb…a9ad), finalized #5063680 (0x72d3…6784), ⬇ 2.0kiB/s ⬆ 2.5kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:19 ⚙️  Preparing 130.3 bps, target=#5065270 (9 peers), best: #5064704 (0x18ea…3275), finalized #5064192 (0xbd5f…7451), ⬇ 14.4kiB/s ⬆ 2.3kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:24 💤 Idle (10 peers), best: #5065271 (0x76db…98a1), finalized #5065268 (0xd1ed…a932), ⬇ 13.6kiB/s ⬆ 3.6kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:29 💤 Idle (11 peers), best: #5065271 (0x76db…98a1), finalized #5065269 (0xe177…d969), ⬇ 25.9kiB/s ⬆ 23.0kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:30 🏆 Imported #5065272 (0x76db…98a1 → 0xcd04…02f2)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:34 💤 Idle (11 peers), best: #5065272 (0xcd04…02f2), finalized #5065270 (0xa0aa…9a4d), ⬇ 23.1kiB/s ⬆ 17.9kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:36 🏆 Imported #5065273 (0xcd04…02f2 → 0xc645…3e02)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:39 💤 Idle (12 peers), best: #5065273 (0xc645…3e02), finalized #5065271 (0x76db…98a1), ⬇ 21.1kiB/s ⬆ 19.2kiB/s    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:42 🏆 Imported #5065274 (0xc645…3e02 → 0x9f2a…b143)    
duniter-gdev-smith-duniter-smith-1  | 2025-02-04 05:21:44 💤 Idle (12 peers), best: #5065274 (0x9f2a…b143), finalized #5065272 (0xcd04…02f2), ⬇ 26.3kiB/s ⬆ 18.4kiB/s

Au final, le nœud a été offline entre

2025-02-04 00:58:06
2025-02-04 05:21:24

ce qui justifie l’exclusion.

Reste à comprendre l’origine du redémarrage en boucle.

$ df -h
/dev/sda1        30G   28G  238M 100% /

On comprend pourquoi.

Conclusion, il faut que je liste ce que benjamin a fait !! Profile disk use (#267) · Issues · nodes / rust / Duniter v2S · GitLab

poka · 12 March 2025 23:52

This is a major issue that requires extensive investigation on how to reduce Duniter’s disk footprint. (Ping @elois?)

G1v1 is actually 24GB.

elois · 14 March 2025 10:25

Since I was mentioned, here is my perspective on the disk usage topic:

Design Goals: It was never intended for duniter v2 to have a disk footprint similar to duniter v1.
Protocol Differences: In duniter v1, there was no need to store old on-chain states because it used two separate protocols—one for applying a block and another for reverting it. Maintaining two coherent protocols proved extremely challenging and led to difficult-to-resolve bugs. In contrast, Substrate (like Ethereum and most other blockchains) employs a single protocol for applying blocks; to revert a block, previous states must be stored.
Archive Node Requirement: The additional storage is mainly necessary for those who wish to host an archive node. This isn’t a requirement for being a smith (i.e., creating blocks).
Smiths don’t need archive: If you want to operate as a smith while keeping your disk usage low, simply avoid running an archive node or an indexer. For example, my node “gdev” currently uses only about 5GB and remains fully synchronized.
No Critical Bug Here: I don’t see any bug—critical or otherwise—in this behavior. Personally, I don’t plan to invest time in reducing Substrate’s disk usage.

poka · 14 March 2025 11:06

The thing is, I think we are talking about pruning nodes here.
I can understand the footprint of archive node, but even mirror seems to have huge footprint.

So maybe @HugoTrentesaux misconfigured something?

Do you have a custom pruning configuration?

tuxmain · 14 March 2025 17:01

My smith node is 18GB, with --state-pruning 1024.

Frederic_Renault · 17 March 2025 00:31

J’ai lancé un noeud mirroir gdev lors du G1 Camps et déjà 4,8 Go sont utilisés…

du -h mirror-gdev_data-mirror/
4,0K	mirror-gdev_data-mirror/_data/chains/gdev/keystore
4,0K	mirror-gdev_data-mirror/_data/chains/gdev/network
4,8G	mirror-gdev_data-mirror/_data/chains/gdev/paritydb/full
4,8G	mirror-gdev_data-mirror/_data/chains/gdev/paritydb
4,8G	mirror-gdev_data-mirror/_data/chains/gdev
4,8G	mirror-gdev_data-mirror/_data/chains
4,8G	mirror-gdev_data-mirror/_data
4,8G	mirror-gdev_data-mirror/

Ca fait beaucoup !!
Cela sera-t-il différent avec la gtest et la g1v2 ?
Quel espace disque faudra-t-il prévoir?

Nicolas80 · 17 March 2025 12:14

I understand that there is no need to have an Archive node (or Indexed/Squid) to be able to forge blocks.

But don’t we require those “Archive” nodes and Indexer/Squid for a lot of things done in the duniter Clients ?

I guess an “Archive” node is also necessary to be able to sync a new duniter node ?

What happens if we have too few of those Archive nodes and Indexer/Squid servers due to the amount of disk space requirement for them ?

cgeek · 17 March 2025 12:26

Currently we only need an archive node for the Squid indexer because archive node acts as a big store for the events of each block, more particulary the blocks which are out of the “pruning window”.

It is just a convenient way to do it for the short term, but these events could be stored somewhere else.

HugoTrentesaux · 18 March 2025 18:13

This issue mostly applies to mirror and archive nodes. Because

I agree with that, and this is the reason I ran my smith node on a machine with low disk space (30 GB), thinking the 1-day pruning would do it (--blocks-pruning 14400) :

HugoTrentesaux:

Starting duniter with parameters: --name HugoTrentesaux-smith --node-key-file /var/lib/duniter/node.key --public-addr /dns/gdev.trentesaux.fr/tcp/30333 --listen-addr /ip4/0.
0.0.0/tcp/30333 --rpc-cors all --rpc-methods Unsafe --validator --blocks-pruning 14400 --chain gdev -d /var/lib/duniter --unsafe-rpc-external

State pruning is nice for non-archive nodes, but for smith nodes, we recommand using light for DUNITER_PRUNING_PROFILE (docker ref Duniter | Configure your node (Docker)) : Duniter | Run a smith node.

My node has about the same size however (no monitoring):

$ docker system df -v
duniter-gdev-smith_data-smith   2         18.95GB

So the issue here is to guess what is the maximum size a smith node can reach during high network load and ask the smiths to have at least that available plus a safety margin. 20 GB seems like a big requirement to run a smith node, we have to inspect more carefully where does this requirement come from.