Node syncronization time

post by cusma on Dec 13, 2019

Hi there,

what are the factors that influence the synchronization time of a node with the Mainnet?

post by Algodaddy on Dec 16, 2019

Cusma, talking to the guys on Telegram. It looks like it takes about 3 days to sync now. Mostly just depends on your internet speed and how many active connections you have to other nodes.

post by rmb on Jul 24, 2020

I’m currently synchronising a node that has a 10 year old AMD Athlon 64 X2 QL-54 processor and it’s being throttled by the CPU and not the network connectivity. Whilst both cores are running close to 100% utilsation it is averaging less that 0.3 MiB/s of a 60Mbps internet connection.

The question @cusma asks raises an interesting point that will become more of an issue in the future…

At the time of this post, the mainnet currently consists of a bit over 8 million blocks. As the blockchain continues to grow, how long will it take to synchronise a new node in 1-year, 5-year, 10-year’s time - and how much data will traverse the network just to get a new node to the point of synchronisation?

post by tsachi on Jul 25, 2020

@rmb,

The time to perform a catchup with the current implementation would be linear with the number of blocks need to be downloaded & validated. ( i.e. the downloaded is equivalent to network bandwidth and the validation is equivalent to CPU time ).

Reviewing the GitHub repository, it seems that there is some work there which attempts to address some of these concerns, but we’ll have to wait for the official release to find out.

Also, if all you want is a non-archival node, you can grab a sandbox and copy the data directory from there onto your local machine. I haven’t attempted to do that in person, but I believe it would be a viable alternative to a long wait.

post by rmb on Jul 27, 2020

Hi @tsachi, I noticed the fast catchup in the GitHub repository and look forward to hearing more about it.

My node is currently catching up at an averaging of around 61K block per hour (~17 blocks per second) and is not limited by network bandwidth. The mainnet is currently a little over 8.1 million blocks which give my machine an estimated time to catch up of nearly 133 hours (5 and a half days).

Granted that my machine is a relatively low spec and 10 years old so according to Moore’s law I might expect a similar machine purchased today to be processing at a rate of around 29 times faster. (assuming 40% annual growth rate). It is currently consuming around 300 KiB/s so the increased processing rate would take it up to around 9 MiB/s (which is well within the upper limit of my network connection).

Assuming that it will not be bound by network bandwidth, then could I really expect my node to take just 4.6 hours to synchronise if I were to deploy onto a recently purchased mid-spec laptop?

If so then why is this comment in the Docker Sandbox section “This method is recommended if you […] but can’t wait days for your node to catchup”

I am making a few assumptions here. E.g.

• The code is multi-threaded and the rate of block processing increases linearly dependent on the number of cores
• The requests for blocks from peers are appropriately pipelined so that the inter-block receive rate is not a bottleneck

What experience has anyone had synchronising a server-grade, high-bandwidth/cloud-hosted node? What is a realistic number of blocks per hour it can process while catching up?

post by tsachi on Jul 27, 2020

@rmb,

Moore’s law ( or its Intel interpretation of it… ) saying the compute power would double itself every so often. It doesn’t say anything about a particular application being capable of utilizing that growth of compute power. ( for instance, newer intel cpus have better vector-oriented instructions. If you were to compare a previous processors to newer ones, you would find that by using the newer instructions you can achieve better performance. but you need to have your binary optimized for that processor ).

Realistically, while a newer computer would be able to (usually) verify blocks faster, having more and more blocks to verify would make it a loosing battle. I.e. I don’t expect that the next-year processors would be fast enough to double the block verification rate, regardless of what optimization we will make to the blocks verification.

If an Algorand node of 10 years from now would want to catch up, it would either need a 10 times more compute power, or to use a different algorithm to achieve that.

post by rmb on Jul 27, 2020

@tsachi, I agree with you.

I’m in the early phase of planning implementation, BCP and DR strategies. Several hours (let alone several days) down-time to resync a node could be quite costly for some organisations. This needs to be weighed against the cost of operating multiple nodes and /or using an API service as a contingency.

post by fabrice on Jul 27, 2020

One option is to take regular snapshots of the Algorand data folder. If you trust where you store snapshots, you can start from these snapshots. This makes catchup very fast.

What I am not sure about is whether you can take the snapshot while algod is running or whether it is best to stop algod when taking the snapshot. (Using copy-on-write filesystems, the interruption can be very brief, but still not completely negligible depending on your load.)@tsachi Do you know whether snapshotting can be done while algod is running?

post by tsachi on Jul 27, 2020

@rmb,

It’s hard to advocate for any “good” numbers. As you’ve experienced, both bandwidth and compute power are factors in the practical time it takes to catch up. Generally speaking, using a computer ( host ) that is running in a high-bandwidth datacenter could accelerate the process dramatically.

post by rmb on Jul 28, 2020

@tsachi Hopefully someone will spot this thread in the near future and share their experience of starting up a node in such an environment.

post by rmb on Aug 2, 2020

@tsachi,

I see you have been active in the sandbox development. Might it be possible to extend the sandbox concept so that it could be used to create production-ready nodes?

post by tsachi on Aug 2, 2020

@rmb
I have not considered the backup on live system as a required system feature. Could you please share more information regarding which nodes you would like to back up ( archival nodes vs non-archival nodes ), and for what purpose.

My goal here is to see if backing up is really the right solution or not.

Keep in mind, that if you’ll be having 3 different nodes, each one of them would have a different blocks database since the votes each one of these would see might arrive at a different order. ( the blocks database stores the firsts votes for every block that reaches the vote threshold; other votes are omitted ).

The blocks paysets ( transactions ) themselves are naturally identical, and the balances for any particular round are also identical.

post by rmb on Aug 2, 2020

@tsachi,

Sorry for the delay - I guess you know what it is like! Anyway it’s always wise to step back from something and critique whether the right route is being considered, so I appreciate your response.

The scenario…

Consider a consortium requiring a private blockchain implementation (private for reasons we will not explore here). The implementation may probably be a hybrid with some transactions ending up on the mainnet, but the important point here is that there may be one or more private chains.

A minimum viable environment will require a minimum number of nodes…
Perhaps the minimum number of relays nodes (for the private chain) might be defined as 5, split across a minimum of 3 participating organisations. With such a small number of relay nodes then the need to get one back on its feet quickly after a failure/corruption/attack is quite important to the overall resilience of the implementation.

Now turning our attention to archival nodes. It is likely that all major players within the consortium will wish to have archival nodes for the private chain as it will contain data that has business value. It is possible that the players will wish to run business-critical analytics derived from data on the private chain. The loss of an archival node due to failure/corruption/attack could have significant operational implications and therefore a mechanism to recover it quickly would be highly desirable.

As for the participation nodes: For the purpose of this argument I will consider their role as application gateways (irrespective of whether they are also actively participating in consensus).

post by tsachi on Aug 2, 2020

@rmb
Thank you for the elaborated answer. This really help me to try and provide the most suitable solution for your needs.

One of the challenges in backing a node ( regardless of its function ) is that it’s ever changing. The node data keep changing very rapidly to reflect the latest state of the network. This has few implications:

1. No backup of the data would ever be current. By the time the backup is complete, it’s already stale.
2. When restoring the data, there will be a period of time which would be required to “catch-up”, and sync up with the network.
3. To make the restore operation feasible, we want to have a short sync time. That means that we would benefit from frequent backup intervals.

Given that we know all the above, we can go ahead and design a backup strategy that would meet all these conditions.