Need to fix GitLab CI for Duniter-v2s

Ok, now that the CI is running, it fails for a reason of password: deploy_docker_release_sha (#76660) · Jobs · nodes / rust / Duniter v2S · GitLab
Does someone know how to fix it?

This §Variables is only available on § Protected branches or tags. The branch should be named release/* for the variable to be available in the CI.

1 Like

I’m not confortable with these kind of limitations. As @poka told me, the CI should be here to make the developer’s work easier, but as elois implemented it, they actually make it a nightmare (see Bootstraper une ĞDev and Difficultés avec l'outillage for examples of what an undocumented CI can make me feel link). If nobody is here to manage this part, I’m in favor of removing all the limitations so that @poka and I can publish docker images for smith to use and for developers to test (like duniter-indexer, gecko…).

1 Like

Without this feature, Docker Hub can be displayed in the CI and stolen by anyone having the right to push on a branch on this repository. I would not take the risk to remove this security feature.

One option could be to replace Docker Hub publication with GitLab container registry. The latter does not require a CI/CD variable to be protected, since the authentication is integrated.

I am not the one to take this decision. Publishing on Docker Hub was made on purpose to make the images available on the main platform. This is part of the vision as having duniter-v2s repository on GitHub for more visibility.

2 Likes

Ok, thanks for the explanations. So @poka only needs to rename his branch release/* for the pipeline to pass?

2 Likes

It looks like I arrived after the battle, but if you still need help I’m around.

I confirm that release/ and master are the only branches that can access to the dockerhub’s password, and those branches are reserved to “high privileged” (“maintainer”) users for good reason, since we don’t want to limit contribution from random persons but we don’t want them to do an “echo $PASSWORD” either.

2 Likes

It seems the current CI is highly inefficient: each step installs Rust, downloads all the dependencies, rebuilds them from scratch…

I think the cache should be kept between non-concurrent pipelines. Cargo is smart enough to rebuild what needs to be rebuilt, and Substrate is already long enough to build.

Is this possible?

Edit: La CI est très bien et très utile, c’est juste le côté “pour voir ce que ça donne si on déplace l’interrupteur du 15e étage, on va tout détruire et reconstruire l’immeuble” qui me gêne, avec la latence et le gâchis d’énergie et d’occupation et d’usure du serveur qui vont avec.

3 Likes

I gave it a try with these changes, but it seems to take more time because of the size of the cache (around 4GB) to be uploaded and then downloaded. I had jobs taken up to one hour and more.

1 Like

I do not know a lot about CI in general and for Rust in particular, but I wonder the relevance of the virtual machine approach to run this CI because:

  • building Duniter from scratch is very long (~30 min - 1h)
  • building Rust is very CPU-intensive, one powerful container is preferable than multiple limited ones
  • the target folder gets huge with debug build (20 Go - 30 Go)
  • we do not really need to run multiple CI in parallel
  • Rust should not be that bad for reproducible compilation and can manage cache himself

So, is it possible to set up one container to run all Duniter pipelines, keep everything between each run (except output files of end2end tests for instance), and only run one pipeline at a time?

Duniter CI still has an issue: Pipeline · nodes / rust / Duniter v2S · GitLab

This step failed: deploy_docker_release_sha (#78591) · Jobs · nodes / rust / Duniter v2S · GitLab

with the error

Error: Cannot perform an interactive login from a non TTY device

Can somebody work to fix it as it prevents publishing a new official Docker image which is the easiest option available to new smith?

Same as above

Ok, but this time, the branch is named release/* so it should have access to protected variables.

Branch name is release/poka-chainspec-gdev5-pini-docker

These variables REPO_DOCKER_{USER,PASS} are simply not defined:
https://git.duniter.org/nodes/rust/duniter-v2s/-/settings/ci_cd

This commit can be dropped in the meantime.

1 Like

Ok, thanks for finding out! I think @Pini did set these variables in his own repo to be able to upload to his dockerhub space, but the variables are not defined in the main repo.

I thing it is ok to keep these variables, but we have to document the CI. @Pini can you submit a MR documenting all the variables that have to be set in order to be able to use the CI? @Moul it would also be interesting to have a small paragraph summarizing the security rules of the pipeline (like release CI requirements).

I will add the missing variables REPO_DOCKER_USER and REPO_DOCKER_PASS in the main docker to be able tu publish an official release docker image for the new smiths to join.

GitLab documentation already contains it:

I let you add it to Duniter-v2s documentation.

1 Like

Sorry, all this is my fault: a while ago I accidentally pushed to this branch some commits I did to test the CI on my own fork. I didn’t clean up since I thought I was the only user of this now deprecated branch.

You don’t need to. These variables are not the ones used by the CI. Please see the .gitlab-ci.yml file on the master branch. To release you’ll only need the protected variable DUNITERTEAM_PASSWORD already defined in the repository settings.

1 Like

Ok, thank you both. I appreciate your answers, it allows me to understand without having to dig.

The documentation I imagine would be something like:


duniter-v2s repo has protected tags and branches (see here). To see how this work see

The protected tags are: runtime-*, v*
The protected branches are: master and release/*

These branches and tags give access to DUNITERTEAM_PASSWD variable allowing to push images to dockerhub.


That’s enough, but that’s what was missing for me when I published (Difficultés avec l'outillage). I had no clue about docker, gitlab, ci… That does not prevent me from contributing to Duniter because I know Rust and Substrate, but that makes everything difficult. I have no whish to learn that, I only want to learn Duniter as fast as I can. So thanks again for being here ^^

After adding the required variables, the CI fails with:

Error parsing reference: "/duniter-v2s:sha-3189d769" is not a valid repository/tag: invalid reference format

It seems @Pini that there is still some problems in this version of the CI.
Since the goal is to provide future smith an official docker image with the good chainspec, maybe we can create a new branch based on the newer Duniter client (and with all the docker improvements), with the chainspec commits, test if it connects well to the ĞDev network, and then propose a runtime upgrade on this network with the recent updates.

So instead of fixing CI for this old branch, only cherry-pick chainspec commits on an other branch.

I did launch again this job with the good variables, and now, it stops at a more interesting-looking error:

ERROR: Job failed (system failure): Error response from daemon: Cannot link to a non running container: /runner-yx9a3lzn-project-520-concurrent-0-7096a3feb2e9539c-docker-0 AS /runner-yx9a3lzn-project-520-concurrent-0-7096a3feb2e9539c-build-3/docker (exec.go:78:0s)

A bug with the gitlab-runner?

3 posts were split to a new topic: Release new image with ĞDev5 chainspecs