Nodes stuck to a block

Support Duniter-v2
1

My validator node and my RPC node are both stuck on the same block.

Validator logs look like this :

The logs get messages about syncing, but sometimes errors : 
2023-02-18 18:59:00 error sending packet on iface 10.0.0.90: Operation not permitted (os error 1)    
2023-02-18 18:59:00 error sending packet on iface 10.0.0.90: Operation not permitted (os error 1)    
2023-02-18 18:59:03 ⚙️  Syncing  0.0 bps, target=#953015 (6 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 0.4kiB/s ⬆ 0.2kiB/s    
2023-02-18 18:59:08 ⚙️  Syncing  0.0 bps, target=#953016 (6 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 1.1kiB/s ⬆ 0.4kiB/s    
2023-02-18 18:59:13 ⚙️  Syncing  0.0 bps, target=#953017 (6 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 0.8kiB/s ⬆ 0.3kiB/s    
2023-02-18 18:59:18 ⚙️  Syncing  0.0 bps, target=#953018 (6 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 0.9kiB/s ⬆ 0.4kiB/s

My RPC node is stuck also with these kind of logs :

2023-02-18 19:04:00 error sending packet on iface 10.0.0.88: Operation not permitted (os error 1)    
2023-02-18 19:04:03 ⚙️  Syncing  0.0 bps, target=#953065 (10 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 3.0kiB/s ⬆ 0.6kiB/s    
2023-02-18 19:04:08 ⚙️  Syncing  0.0 bps, target=#953066 (10 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 0.9kiB/s ⬆ 0.4kiB/s    
2023-02-18 19:04:13 ⚙️  Syncing  0.0 bps, target=#953067 (10 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 1.1kiB/s ⬆ 0.7kiB/s    
2023-02-18 19:04:18 ⚙️  Syncing  0.0 bps, target=#953068 (10 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 1.0kiB/s ⬆ 0.7kiB/s    
2023-02-18 19:04:23 ⚙️  Syncing  0.0 bps, target=#953068 (10 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 0.4kiB/s ⬆ 0.4kiB/s    
2023-02-18 19:04:28 ⚙️  Syncing  0.0 bps, target=#953069 (10 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 1.1kiB/s ⬆ 0.7kiB/s

I keep those node stuck if somebody wants to investigate… Or give me tools to do so :wink: .

[Edit]
A clue ? I loose internet access sometimes. Can disturb the sync, may be.
Docker compose (with ansible variables) :

  # ===== RPC =====
  duniter_rpc:
    image: "{{ duniter_v2s_rpc_image }}"
    restart: unless-stopped
    ports:
      # telemetry
      - 9615:9615
      # rpc
#      - 9933:9933
      # rpc-ws
      - 9944:9944
      # p2p
      - 30333:30333
    volumes:
      - "{{ duniter_v2s_rpc_data_path }}:/var/lib/duniter"
    environment:
      - DUNITER_CHAIN_NAME=gdev
    command:
      - "--node-key-file=/var/lib/duniter/node.key"
      - "--public-addr"
      - "/dns/{{ domain_name }}/tcp/30333/p2p/{{ duniter_v2s_rpc_peer_id }}"
#      - "--ws-external"
      - "--rpc-cors=all"
      - "--pruning=14400"
      - "--name"
      - "vit-rpc"

  # ===== VALIDATOR =====
  duniter_validator:
    image: "{{ duniter_v2s_validator_image }}"
    restart: unless-stopped
    ports:
      # telemetry
      - 9616:9615
      # rpc
      #- 9934:9933
      # rpc-ws
      - 9945:9944
      # p2p
      - 30334:30333
    volumes:
      - "{{ duniter_v2s_validator_data_path }}:/var/lib/duniter"
    environment:
      - DUNITER_CHAIN_NAME=gdev
    command:
      - "--node-key-file=/var/lib/duniter/node.key"
      - "--rpc-cors=all"
      - "--rpc-methods=Unsafe"
      - "--validator"
      - "--pruning=14400"
      - "--name"
      - "vit-validator"
2

Very strange error. I used to get this at start, and had to change permissions on the volume volder. Which docker image do you use ? (or do you use custom build ?)

3

No custom build. Only the Duniter V2S docker image for gdev.

[
    {
        "Id": "sha256:ca98dc1918e806d4b62e5a58ec3cdc61e5be74625865f439b7a6faff2f5fdee6",
        "RepoTags": [
            "duniter/duniter-v2s:sha-f442e6eb"
        ],
        "RepoDigests": [
            "duniter/duniter-v2s@sha256:64508c0d650e4e1a3f0f986835c3979a58479e7bac8d85efed84b964f146e88c"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2022-12-15T17:38:10.804524013Z",
        "Container": "",
4

Ok, tu peux essayer avec sha-latest ou 3189d769f70ec1f6c05282211e4d6ab4c2de6361 ? @Pini a fait un super boulot entre temps, et c’est beaucoup plus simple à utiliser et moins buggé.

5

L’image que @HugoTrentesaux et moi utilisons est pinidh/duniter-v2s:sha-latest. Elle n’a été uploadée que sur mon compte pour l’instant.

EDIT : d’ailleurs il serait probablement temps d’uploader en duniter/duniter-v2s:latest pour simplifier les choses.

6

Bien vu, j’avais oublié que c’était image: pinidh/duniter-v2s:sha-latest. Mais si je me souviens bien, j’avais mis une branche nommée “release/*” sur ce commit pour que GitLab génère une image. Mais peut-être la pipeline a échoué, je sais plus, j’ai trop de choses en tête.

7

J’ai édité mon premier billet avec le docker compose. @Pini si tu vois une bizarrerie…

8

Tu as mis ton template ansible, dans ce cas mais aussi ton playbook :smiley:
(sinon on n’a pas le numéro de version)

La nouvelle doc est ici, tu vas pouvoir simplifier grandement ton docker-compose.yml Duniter | Run a mirror node

9

Il y en a. Regarde la doc, c’est beaucoup plus simple maintenant. En particulier plus besoin de récupérer le peer ID.

10

Pas possible, c’est un role Ansible complet pour mon serveur. Je mets l’image que vous voulez moi, je suis pas regardant. Donc pini sha latest quoi. ;).

Bon je regarde la doc asap.
[edit] ben merde on cause en french maintenant :joy:

11

Sorry, I was the first starting to speak french :slight_smile:

12

My personal strategy for choosing the language:

personal, not intended to be useful for future people → best common language with immediate interlocutors (e.g. specific or temporary debug, PR discussion)
else → English

13

Node restarted with a docker-compose from

Image : pinidh/duniter-v2s:sha-latest

There is again an error on an network interface, but the sync goes well…

Node key file '/var/lib/duniter/node.key' exists.
Node peer ID is '12D3KooWDuzVbBcnnEEKh32R6MUKvQvLENyzLHHUfg4kTyUQq7hp'.
Starting duniter with parameters: --name vit-rpc --node-key-file /var/lib/duniter/node.key --rpc-cors all --chain gdev -d /var/lib/duniter --unsafe-rpc-external --unsafe-ws-external
2023-02-20 15:49:40 Duniter    
2023-02-20 15:49:40 ✌️  version 0.3.0-3189d769f70    
2023-02-20 15:49:40 ❤️  by Axiom-Team Developers <https://axiom-team.fr>, 2021-2023    
2023-02-20 15:49:40 📋 Chain specification: Ğdev    
2023-02-20 15:49:40 🏷  Node name: vit-rpc    
2023-02-20 15:49:40 👤 Role: FULL    
2023-02-20 15:49:40 💾 Database: ParityDb at /var/lib/duniter/chains/gdev/paritydb/full    
2023-02-20 15:49:40 ⛓  Native runtime: gdev-400 (duniter-gdev-1.tx1.au1)    
2023-02-20 15:53:29 🏷  Local node identity is: 12D3KooWDuzVbBcnnEEKh32R6MUKvQvLENyzLHHUfg4kTyUQq7hp    
2023-02-20 15:53:29 💻 Operating system: linux    
2023-02-20 15:53:29 💻 CPU architecture: x86_64    
2023-02-20 15:53:29 💻 Target environment: gnu    
2023-02-20 15:53:29 💻 CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz    
2023-02-20 15:53:29 💻 CPU cores: 4    
2023-02-20 15:53:29 💻 Memory: 7888MB    
2023-02-20 15:53:29 💻 Kernel: 4.15.0-142-generic    
2023-02-20 15:53:29 💻 Linux distribution: Debian GNU/Linux 10 (buster)    
2023-02-20 15:53:29 💻 Virtual machine: no    
2023-02-20 15:53:29 📦 Highest known block at #945735    
2023-02-20 15:53:29 〽️ Prometheus exporter started at 127.0.0.1:9615    
2023-02-20 15:53:29 Running JSON-RPC HTTP server: addr=0.0.0.0:9933, allowed origins=None    
2023-02-20 15:53:29 Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=None    
2023-02-20 15:53:29 ***** Duniter has fully started *****    
2023-02-20 15:53:39 creating instance on iface 10.0.0.94    
2023-02-20 15:53:39 creating instance on iface 172.18.0.3    
2023-02-20 15:53:39 creating instance on iface 10.0.2.45    
2023-02-20 15:53:39 error sending packet on iface 10.0.0.94: Operation not permitted (os error 1)    
2023-02-20 15:53:39 error sending packet on iface 10.0.0.94: Operation not permitted (os error 1)    
2023-02-20 15:53:39 💤 Idle (0 peers), best: #945735 (0x30a3…1fe2), finalized #917920 (0x2758…6af6), ⬇ 0 ⬆ 0    
2023-02-20 15:53:40 🔍 Discovered new external address for our node: /ip4/80.67.176.219/tcp/30333/ws/p2p/12D3KooWDuzVbBcnnEEKh32R6MUKvQvLENyzLHHUfg4kTyUQq7hp    
2023-02-20 15:53:44 ⚙️  Syncing 14.1 bps, target=#979879 (9 peers), best: #945806 (0x5208…1c56), finalized #917920 (0x2758…6af6), ⬇ 159.4kiB/s ⬆ 9.6kiB/s    
2023-02-20 15:53:49 ⚙️  Syncing  8.7 bps, target=#979880 (9 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 1.6kiB/s ⬆ 0.2kiB/s    
2023-02-20 15:53:54 ⚙️  Syncing  0.0 bps, target=#979881 (9 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 2.1kiB/s ⬆ 0.9kiB/s    
2023-02-20 15:53:59 ⚙️  Syncing  0.0 bps, target=#979881 (9 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 2.4kiB/s ⬆ 0.8kiB/s    
2023-02-20 15:54:04 ⚙️  Syncing  0.0 bps, target=#979882 (9 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 1.3kiB/s ⬆ 0.7kiB/s    
2023-02-20 15:54:09 ⚙️  Syncing  0.0 bps, target=#979883 (9 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 0.6kiB/s ⬆ 46 B/s    
2023-02-20 15:54:14 ⚙️  Syncing  0.0 bps, target=#979884 (9 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 3.2kiB/s ⬆ 1.6kiB/s    
2023-02-20 15:54:16 discovered: 12D3KooWB3zoZgS9GmzDMJSkgRXd3eiGft5HQAB84GEBj9GpZY9V /ip4/10.0.2.46/tcp/30333    
2023-02-20 15:54:16 discovered: 12D3KooWB3zoZgS9GmzDMJSkgRXd3eiGft5HQAB84GEBj9GpZY9V /ip4/127.0.0.1/tcp/30333    
2023-02-20 15:54:16 discovered: 12D3KooWB3zoZgS9GmzDMJSkgRXd3eiGft5HQAB84GEBj9GpZY9V /ip4/10.0.0.95/tcp/30333    
2023-02-20 15:54:16 discovered: 12D3KooWB3zoZgS9GmzDMJSkgRXd3eiGft5HQAB84GEBj9GpZY9V /ip4/172.18.0.7/tcp/30333    
2023-02-20 15:54:16 error sending packet on iface 10.0.0.94: Operation not permitted (os error 1)    
2023-02-20 15:54:19 ⚙️  Syncing  0.0 bps, target=#979885 (10 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 1.4kiB/s ⬆ 1.3kiB/s    
2023-02-20 15:54:24 ⚙️  Syncing  0.0 bps, target=#979886 (10 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 1.3kiB/s ⬆ 0.8kiB/s    
2023-02-20 15:54:30 ⚙️  Syncing  0.0 bps, target=#979886 (10 peers), best: #945850 (0x4ba1…4470), finalized #917920 (0x2758…6af6), ⬇ 0.5kiB/s ⬆ 0.5kiB/s
14

Your instance seems to have 3 network interfaces:

2023-02-20 15:53:39 creating instance on iface 10.0.0.94    
2023-02-20 15:53:39 creating instance on iface 172.18.0.3    
2023-02-20 15:53:39 creating instance on iface 10.0.2.45

Only one of them generates this error:

2023-02-20 15:53:39 error sending packet on iface 10.0.0.94: Operation not permitted (os error 1)
15

Very strange. I did not see 3 network interfaces on the host…

If you give me tool names to investigate further why the docker container mount 3 network interfaces, I will be happy to verify this.

vit@z68-gen3:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether bc:5f:f4:19:ba:ce brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.105/24 brd 192.168.1.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::8da7:f302:d13a:234/64 scope link 
       valid_lft forever preferred_lft forever
3: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:52:32:3e:4d brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:52ff:fe32:3e4d/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:93:c3:b4:b6 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

[EDIT]
Mystery solved for the error message: my docker server is of type swarm. It adds the stack network (group of containers declared in the docker-compose file). As well as a network called Ingress, which is the one that is causing the problem. But this doesn’t seem to bother Duniter v2s.

16

After several weeks of flawless work my RPC node is now stuck as well, at block #981318. Nothing in the log:

2023-02-20 18:17:19 ✨ Imported #981315 (0xe20d…e5e2)                                                                                                                                                              
2023-02-20 18:17:22 💤 Idle (6 peers), best: #981315 (0xe20d…e5e2), finalized #917920 (0x2758…6af6), ⬇ 2.8kiB/s ⬆ 0.6kiB/s                                                                                         
2023-02-20 18:17:22 💤 Idle (6 peers), best: #981315 (0xe20d…e5e2), finalized #917920 (0x2758…6af6), ⬇ 2.8kiB/s ⬆ 0.6kiB/s    
2023-02-20 18:17:24 ✨ Imported #981316 (0x4cf7…58ed)    
2023-02-20 18:17:27 💤 Idle (6 peers), best: #981316 (0x4cf7…58ed), finalized #917920 (0x2758…6af6), ⬇ 2.6kiB/s ⬆ 0.4kiB/s                                                                                         
2023-02-20 18:17:30 ✨ Imported #981317 (0x8d14…9cff)                                                                                                                                                              
2023-02-20 18:17:32 💤 Idle (6 peers), best: #981317 (0x8d14…9cff), finalized #917920 (0x2758…6af6), ⬇ 2.1kiB/s ⬆ 0.4kiB/s                                                                                         
2023-02-20 18:17:37 ✨ Imported #981318 (0x63bc…bf47)                                                                                                                                                              
2023-02-20 18:17:37 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 2.0kiB/s ⬆ 0.3kiB/s                                                                                         
2023-02-20 18:17:42 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 2.7kiB/s ⬆ 2.9kiB/s    
2023-02-20 18:17:47 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 1.4kiB/s ⬆ 0.2kiB/s                                                                                         
2023-02-20 18:17:52 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 1.8kiB/s ⬆ 0.3kiB/s    
2023-02-20 18:17:57 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 2.3kiB/s ⬆ 0.2kiB/s                                                                                         
2023-02-20 18:18:02 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 2.0kiB/s ⬆ 0.2kiB/s    
2023-02-20 18:18:07 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 2.6kiB/s ⬆ 0.5kiB/s                                                                                         
2023-02-20 18:18:12 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 1.4kiB/s ⬆ 0.2kiB/s    
2023-02-20 18:18:17 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 1.7kiB/s ⬆ 0.3kiB/s                                                                                         
2023-02-20 18:18:22 💤 Idle (6 peers), best: #981318 (0x63bc…bf47), finalized #917920 (0x2758…6af6), ⬇ 1.8kiB/s ⬆ 0.2kiB/s

Restarting it to see if its state improves…

Nope. It catches up with the other nodes then stalls:

2023-02-20 18:34:53 Running JSON-RPC HTTP server: addr=0.0.0.0:9933, allowed origins=None    
2023-02-20 18:34:53 Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=None    
2023-02-20 18:34:53 ***** Duniter has fully started *****    
2023-02-20 18:34:53 discovered: 12D3KooWCEVBrLK9g8unsHLj8wWzobbwArDZMMwg5LeCbiWhB3ug /ip4/172.18.0.16/tcp/30333/ws    
2023-02-20 18:34:53 🔍 Discovered new external address for our node: /ip4/5.135.160.24/tcp/30333/ws/p2p/12D3KooWH8fb9TsHvvXfD9cWLJtNuiMGGBTZ3a7gkxV4aXcsfDxG    
2023-02-20 18:34:58 💤 Idle (6 peers), best: #981329 (0xa628…9f65), finalized #917920 (0x2758…6af6), ⬇ 19.5kiB/s ⬆ 5.1kiB/s    
2023-02-20 18:35:03 💤 Idle (6 peers), best: #981352 (0xbad8…1082), finalized #917920 (0x2758…6af6), ⬇ 3.5kiB/s ⬆ 0.3kiB/s    
2023-02-20 18:35:08 💤 Idle (6 peers), best: #981361 (0xc82f…a1d8), finalized #917920 (0x2758…6af6), ⬇ 2.4kiB/s ⬆ 0.2kiB/s    
2023-02-20 18:35:12 Accepting new connection 1/100
2023-02-20 18:35:13 💤 Idle (6 peers), best: #981386 (0xfbe4…cfd9), finalized #917920 (0x2758…6af6), ⬇ 2.8kiB/s ⬆ 0.5kiB/s    
2023-02-20 18:35:18 💤 Idle (6 peers), best: #981396 (0x9edf…5267), finalized #917920 (0x2758…6af6), ⬇ 2.1kiB/s ⬆ 0.4kiB/s    
2023-02-20 18:35:23 💤 Idle (6 peers), best: #981419 (0x5de7…3872), finalized #917920 (0x2758…6af6), ⬇ 2.1kiB/s ⬆ 68 B/s    
2023-02-20 18:35:28 💤 Idle (6 peers), best: #981446 (0xa81a…fca0), finalized #917920 (0x2758…6af6), ⬇ 2.3kiB/s ⬆ 0.4kiB/s    
2023-02-20 18:35:33 💤 Idle (6 peers), best: #981468 (0x3590…f6cd), finalized #917920 (0x2758…6af6), ⬇ 2.2kiB/s ⬆ 0.2kiB/s    
2023-02-20 18:35:37 ✨ Imported #981493 (0xd5ff…8496)    
2023-02-20 18:35:38 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.0kiB/s ⬆ 0.1kiB/s    
2023-02-20 18:35:43 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.4kiB/s ⬆ 0.6kiB/s    
2023-02-20 18:35:48 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 1.3kiB/s ⬆ 0.1kiB/s    
2023-02-20 18:35:53 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 0.7kiB/s ⬆ 0    
2023-02-20 18:35:58 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 4.0kiB/s ⬆ 0.8kiB/s    
2023-02-20 18:36:03 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 1.5kiB/s ⬆ 0.2kiB/s    
2023-02-20 18:36:08 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.2kiB/s ⬆ 0.2kiB/s    
2023-02-20 18:36:13 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.0kiB/s ⬆ 0.3kiB/s    
2023-02-20 18:36:18 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.1kiB/s ⬆ 0.5kiB/s

Any idea?

EDIT: It has suddenly caught up again, reliably this time. Maybe I should have waited a bit before restarting it:

2023-02-20 18:40:03 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.5kiB/s ⬆ 0.3kiB/s    
2023-02-20 18:40:08 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.7kiB/s ⬆ 0.1kiB/s    
2023-02-20 18:40:13 ⚙️  Syncing  0.0 bps, target=#981541 (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 1.6kiB/s ⬆ 0.4kiB/s    
2023-02-20 18:40:18 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 2.5kiB/s ⬆ 0.4kiB/s    
2023-02-20 18:40:23 💤 Idle (6 peers), best: #981493 (0xd5ff…8496), finalized #917920 (0x2758…6af6), ⬇ 1.4kiB/s ⬆ 0           
2023-02-20 18:40:27 ✨ Imported #981497 (0xa226…d5ac)    
2023-02-20 18:40:27 ✨ Imported #981498 (0xad26…61e6)                                                                                                                                                              
2023-02-20 18:40:27 ✨ Imported #981498 (0xde9d…269c)    
2023-02-20 18:40:28 💤 Idle (6 peers), best: #981499 (0x3781…0bc0), finalized #981498 (0xad26…61e6), ⬇ 2.2kiB/s ⬆ 0.7kiB/s    
2023-02-20 18:40:28 ✨ Imported #981501 (0x9479…3b81)    
2023-02-20 18:40:29 ✨ Imported #981503 (0x9404…97fe)                                                                                                                                                              
2023-02-20 18:40:30 ✨ Imported #981510 (0x934e…c951)    
...
2023-02-20 18:42:23 💤 Idle (6 peers), best: #981562 (0x93ba…2f23), finalized #981560 (0x6139…0114), ⬇ 5.0kiB/s ⬆ 6.2kiB/s    
2023-02-20 18:42:24 ✨ Imported #981563 (0x000f…3c30)    
2023-02-20 18:42:24 ✨ Imported #981563 (0x9095…819f)    
2023-02-20 18:42:28 💤 Idle (6 peers), best: #981563 (0x000f…3c30), finalized #981561 (0x838b…f35a), ⬇ 4.9kiB/s ⬆ 5.7kiB/s    
2023-02-20 18:42:30 ✨ Imported #981564 (0x85bf…b2de)    
2023-02-20 18:42:33 💤 Idle (6 peers), best: #981564 (0x85bf…b2de), finalized #981562 (0x93ba…2f23), ⬇ 3.6kiB/s ⬆ 5.9kiB/s    
2023-02-20 18:42:36 ✨ Imported #981565 (0xd09b…42df)    
2023-02-20 18:42:38 💤 Idle (6 peers), best: #981565 (0xd09b…42df), finalized #981563 (0x000f…3c30), ⬇ 3.9kiB/s ⬆ 5.3kiB/s    
2023-02-20 18:42:42 ✨ Imported #981566 (0x5cbd…de6a)    
2023-02-20 18:42:43 💤 Idle (6 peers), best: #981566 (0x5cbd…de6a), finalized #981563 (0x000f…3c30), ⬇ 4.8kiB/s ⬆ 6.1kiB/s    
2023-02-20 18:42:48 ✨ Imported #981567 (0x110c…a2cf)    
2023-02-20 18:42:48 ♻️  Reorg on #981567,0x110c…a2cf to #981567,0x6ed0…cad9, common ancestor #981566,0x5cbd…de6a    
2023-02-20 18:42:48 ✨ Imported #981567 (0x6ed0…cad9)    
2023-02-20 18:42:48 💤 Idle (6 peers), best: #981567 (0x6ed0…cad9), finalized #981564 (0x85bf…b2de), ⬇ 6.1kiB/s ⬆ 5.8kiB/s

EDIT2: And I notice at the same time that finalization has resumed! Wooooot :tada:

17

As we can see on telemetry page, only the vit-rpc node has sync well and is following the blocks. The vit-validator node is stuck.

18

My Validator node is still stuck on an old block.
It doesn’t seem to connect to any peer, and the log display this repeatedly :

home_duniter_validator.1.flto5wdr5vpz@z68-gen3    | 2023-05-14 10:36:40 💤 Idle (0 peers), best: #1985800 (0x1ef4…52d5), finalized #1985512 (0xe7dc…1bf6), ⬇ 0 ⬆ 0    
home_duniter_validator.1.flto5wdr5vpz@z68-gen3    | 2023-05-14 10:36:45 💤 Idle (0 peers), best: #1985800 (0x1ef4…52d5), finalized #1985512 (0xe7dc…1bf6), ⬇ 0 ⬆ 0    
home_duniter_validator.1.flto5wdr5vpz@z68-gen3    | 2023-05-14 10:36:50 💤 Idle (0 peers), best: #1985800 (0x1ef4…52d5), finalized #1985512 (0xe7dc…1bf6), ⬇ 0 ⬆ 0    
home_duniter_validator.1.flto5wdr5vpz@z68-gen3    | 2023-05-14 10:36:55 💤 Idle (0 peers), best: #1985800 (0x1ef4…52d5), finalized #1985512 (0xe7dc…1bf6), ⬇ 0 ⬆ 0    
home_duniter_validator.1.flto5wdr5vpz@z68-gen3    | 2023-05-14 10:37:00 💤 Idle (0 peers), best: #1985800 (0x1ef4…52d5), finalized #1985512 (0xe7dc…1bf6), ⬇ 0 ⬆ 0    
home_duniter_validator.1.flto5wdr5vpz@z68-gen3    | 2023-05-14 10:37:05 💤 Idle (0 peers), best: #1985800 (0x1ef4…52d5), finalized #1985512 (0xe7dc…1bf6), ⬇ 0 ⬆ 0

Docker image has been updated today, but node restarted is still stuck:

                "ContainerSpec": {
                    "Image": "pinidh/duniter-v2s:sha-latest@sha256:f6645154991c67a2dfd7665b9a4f01eecfe2d5045e5cac16acef702bf9644a9d",
                    "Labels": {
                        "com.docker.stack.namespace": "home"
                    },
                    "Env": [
                        "DUNITER_CHAIN_NAME=gdev",
                        "DUNITER_NODE_NAME=vit-validator",
                        "DUNITER_VALIDATOR=true"
                    ],
19

Some more investigation on my validator node with no peers:

docker run -v /home/vit/duniter_validator:/var/lib/duniter -p 9616:9615 -p 127.0.0.1:9945:9944 -p 30334:30333 -e DUNITER_CHAIN_NAME=gdev -e DUNITER_VALIDATOR=true -e DUNITER_DE_NAME=vit-validator -e DUNITER_PUBLIC_ADDR=/dns/vit.fdn.org/tcp/30334 -e DUNITER_LISTEN_ADDR=/ip4/0.0.0.0/tcp/30333 pinidh/duniter-v2s:sha-latest --log sub-libp2p

Now the logs are more verbose:

2023-06-19 13:09:44.142 TRACE tokio-runtime-worker sub-libp2p: Addresses of PeerId("12D3KooWMYJzk1FfBZjEAuEvwUnH2Luj5Bq4ouLX1tgZBPpFegaB"): ["/dns/gdev.p2p.legal/tcp/30334", "/dns/gdev.p2p.legal/tcp/30334"]    
2023-06-19 13:09:44.143 TRACE tokio-runtime-worker sub-libp2p: Libp2p => Dialing(PeerId("12D3KooWMYJzk1FfBZjEAuEvwUnH2Luj5Bq4ouLX1tgZBPpFegaB"))    
2023-06-19 13:09:44.143 TRACE tokio-runtime-worker sub-libp2p: Addresses of PeerId("12D3KooWL1SQXYUWSHLczucVwk3nX4Hk3N8xo5ZR5w8RtuFKCYnW"): ["/dns/gdev.trentesaux.fr/tcp/30334", "/dns/gdev.trentesaux.fr/tcp/30334"]

There is a problem here, the port I use on the host and as external public port (30334), is used to connect to other nodes as well ??!! :sweat_smile:

[EDIT]
After some tests, it seems that the only two bootstrap nodes are really configured with port 30334 somewhere, but are unreachable…