Syncing new node - questions about the sync

Nodes
1

Following my post on installing a new node I’ve some newbee question about the syncing process.

I issued a sync command from my fresh node:

ucoind --version
0.11.13

ucoind sync metab.ucoin.io 9201

Download: [||||||||||||||||||||] 100 %
Apply:    [||||                ] 22 %

Status: Remaining 16 hours

After few hours, it still in progress.

Can I safely interrupt the sync? And may be move this VM somewhere else. Or will it fail to restart?

Cgeek has said that the sync process will be easier with ucoind-v0.12. Does the version upgrade take care of previous work or should it be restarted from scratch.

As far as I know the protocol of syncing a node must slowdown newcomer node to stabilize the network of node of the community. Avoiding alien node to multiply and takeover a community may be. Which seems perfectly reasonable to me. But not if it can’t be interrupted at any time and restarted later.

What the node is doing during the download/Apply?
Can it do something else during the period?
Can we see where it is in the apply process?

If I can stop the node, can I slow down it’s CPU usage during the sync.
Can I upgrade the node with 0.12 code and make it use downloaded or even computed data of the version0.11?

Regards,
Sylvain.

2

Hi @Sylvain,

I have update documentation to install a node.

Yep, you can interrupt sync.
You should use uCoin 0.12 witch have a faster syncing method and you should use the --nocautious arguments when syncing which doesn’t verify blockchain

The verifying part consume lot of CPU.

You should be able to sync in less than 20 min.

2 Likes
3

You could have a little word about --autoconf option too, but what you wrote is OK. Thank you :slight_smile:

4

Cool I will try it.

Where do I find stored blockchain to save and restart using different parameter?

Where do I find log files?

5

Data and blockchain are stored in ~/.config/ucoin/ucoin_default/.
Actually, uCoin doesn’t store logs in a file.
But, you could store logs in a file using --log pm2 parameter.

6

Ah, OK. nice!

I see the CoreFS structure, a lot of inodes here. 4k each for the default ext4 partitioning. I read it was a choice to not use a alternative database for storing data, is there a discussion about that choice?

Where do the call like:

          logger.info('Sync started.');

are written?

var logger           = require('./logger')('sync');

this correspond to this code:

/usr/local/lib/node_modules/ucoin/node_modules/log4js/lib/logger.js

probably more this one first:
/usr/local/lib/node_modules/ucoin/app/lib/logger.js

right?

I stop this VM, I will dig more next time, thanks for the link. I will try to read the difference in the sync algorithm from v0.11 and v0.12.

Sylvain.

7

This is the logging module, it produces loggers. So the following code:

var logger           = require('./logger')('sync');

procudes a logger that can be used for logging messages with logger.info for example.

Every logging message stepped through this logging module in a first place. This is a logging abstraction, actually.

Essentially, the algorithm changed in commit 37c25b9d.

To sum it up: previously, I was using the “normal” process for adding blocks (line 125 of sync.js: method “applyGivenBlock”) which uses the same process as a node running and receiving a new block from HTTP interface. But now, I use a shortcut method (line 139 of sync.js: method “saveBlocksInMainBranch”) which avoid a lot of useless treatments. That’s why it is faster: no rules checking + less treatments accessing the database.

Well I do not speak that much alone, so you won’t find such a discussion :slight_smile: however in substance:

  • I want uCoin to run on almost any computer
  • I do not want users to have to install extra server databases like MySQL, MongoDB, …

So the simplest approach for me in a first place was to use simple files. It works everywhere, and FS is able to index files through directories.

However with data growing, uCoin became slower and slower so I had to look for another solution. That’s why 0.12 is now using LokiJS, an in-memory JavaScript database. It regularly persists its data on the file system too, but works exclusively in memory. This is a huge improvement in terms of performances and does not require an extra server database.

Still, almost all blocks are directly stored on the file system for archiving. Most of the time we only work with 10% of the blocks (I don’t know the exact number, we could calculate it).

You can see it & test it using DEV version (master branch).

8

Hi,

I didn’t read the logger api yet. How can I log to a file for the syncing process and see those logger messages?

[quote=“cgeek, post:7, topic:563, full:true”]That’s why it is faster: no rules checking + less treatments accessing the database.
[/quote]

Yes, I see, thanks for the direct link it will save me some time.

I’m not agree with this reasoning, but I don’t know yet enough to suggest anything better. I had to eat more code to see what you need and why you do it that way.

FS + file are consuming a lot of disk space and introduce a lot of redundancy in the records. I don’t know the data structure yet but I had a look at the blocks sub folder, I have for 41 000 records = 140 Mb of disk space, and a .tgz only 9Mb, approx values. But you get fs natural tools: grep, find, git, and more…

Not really, it take a bunch of disk space consuming inodes, and you have to take care of folder limitation on the FS and using “fragmentation” to store your blocks in many sub-folders to avoid folder limitation in number of file. For example…

Of course now you have programmed what database engine are performing naturally: in memory live data + persistence, transaction, replication, and more…

Once again, I need to read more code to understand your reflexion path.

Thank you, for sharing.
Sylvain.

9

Just to be clear for people reading us, about my position for uCoin:

  • I do not pretend to make a “one-shot” software that would just work out-of-the-box for thousand years
  • I do not pretend to master all the technologies I deal with, I am young and still have a lot to learn
  • I did not particularly took care about optimisations in general, either that would be about:
    • disk space usage
    • memory usage
    • CPU consumption

All I want to produce is a first software that could be installed easily and allowing a first freedom money to kick off and live hundred of years. But the software does not need to be the same all the way along :slight_smile: I do hope it will evolve!

2 Likes