Node continues normally despite fast catchup

post by imikim on Jan 15, 2021

My normal node was synced to block 1422735.

Then I ran fast catchup to 11460000#UULY5I7LDPBZM4TX24BFPHXNAXGDCGIMKEPPPA4WYWYH3EYWGINQ

Fast catchup successfully downloads and verifies all the accounts and then downloads 1000 blocks.

After that, the node continues syncing from 1422735 again.

I thought that the node would only need to get blocks after the catchup point and I would be synced soon.

This is on Debian, using apt-get and algod and goal.

post by tsachi on Jan 15, 2021

@imikim ,

The fact that your node was keeping catching up from 1422735 instead of 11m, suggests that the fast catchup was not as successful as you’ve implied. Could you try to look in the log file for warning or error log entries?

post by imikim on Jan 15, 2021

I am running fast catchup again, whilst looking at node.log and will report if I see an error after it is ‘done’.

Strangely, it is stuck at ‘Catchpoint downloaded blocks: 962’ for a while now (maybe 10 minutes).

It was increasing all the time before that.

There was also nothing happening for some time after ‘Catchpoint accounts processed’ finished before ‘Catchpoint accounts verified’ started. But it did eventually.

EDIT: Finally progressing

EDIT2: As before, process continues at 1.6m, not from catchpoint. Will look at node.log now.

post by imikim on Jan 15, 2021

cat node.log.bak | grep error gives:

{"file":"catchpointService.go","function":" github.com/algorand/go-algorand/catchup.(*CatchpointCatchupService.run","level":"warning","line":202,"msg":"catchpoint catchup stage error : processStageBlocksDownload failed after multiple blocks download attempts","name":"","time":"2021-01-15T13:32:45.771148Z"}

EDIT: This must have happened when it was stuck at 962 downloaded catchup blocks

post by fabrice on Jan 15, 2021

How is your Internet speed?

You may want to try to increase CatchupBlockDownloadRetryAttempts in config.json.

post by tsachi on Jan 15, 2021

I believe that you’re experiencing a small bug that was already addressed in master, but have yet to be released : bugfix: incorrect block download retry accounting during fast catchup… · algorand/go-algorand@c1a8cf4 · GitHub

The workaround is exactly as @fabrice suggested : increase the CatchupBlockDownloadRetryAttempts in the config.json to a value higher than 1000.