Algorand Node stop synching after upgrading it to version 3.15.1

post by Sowna on May 25, 2023

Hi @fabrice,

Algorand node status stuck in the same block height after upgrading it to 3.15.1

catchup-time is increasing constantly

{
“catchpoint”: “”,
“catchpoint-acquired-blocks”: 0,
“catchpoint-processed-accounts”: 0,
“catchpoint-processed-kvs”: 0,
“catchpoint-total-accounts”: 0,
“catchpoint-total-blocks”: 0,
“catchpoint-total-kvs”: 0,
“catchpoint-verified-accounts”: 0,
“catchpoint-verified-kvs”: 0,
“catchup-time”: 1602302324241,
“last-catchpoint”: “29110000#FI2DUXOJUME3GMA6C2TIQOV2KWT7BOKKJGQCHP7UF6JIO2XIYEKA”,
“last-round”: 29117368,
“last-version”: “ [GitHub - algorandfoundation/specs at 44fa607d6051730f5264526bf3c108d51f0eadb6](https://github.com/algorandfoundation/specs/tree/44fa607d6051730f5264526bf3c108d51f0eadb6)”,
“next-version”: “ [GitHub - algorandfoundation/specs at 44fa607d6051730f5264526bf3c108d51f0eadb6](https://github.com/algorandfoundation/specs/tree/44fa607d6051730f5264526bf3c108d51f0eadb6)”,
“next-version-round”: 29117369,
“next-version-supported”: true,
“stopped-at-unsupported-round”: false,
“time-since-last-round”: 0
}

node.log logs

{“file”:“logger.go”,“function”:“ [github.com/algorand/go-algorand/daemon/algod/api/server/lib/middlewares.(*LoggerMiddleware).handler.func1”,“level”:“info”,“line”:56,“msg”:“10.213.120.116:48600 - - [2023-05-25 17:05:29.979282361 +0000 UTC m=+111184.156884083] "GET /v2/blocks/20145543?format=msgpack HTTP/1.1" 200 175972 "Go-http-client/1.1" 370.318µs”,“time”:“2023-05-25T17:05:29.979672Z”}

post by fabrice on May 25, 2023

Did you try to stop/start the node?

Did you check you’re not running out of memory?

If this does not work, can you provide all the information indicated in Node Troubleshooting - Algorand Developer Portal?

(including full node.log file, because the part you provided only shows queries you made to the node, but not errors from algod)

post by Sowna on May 31, 2023

Yes I have stopped and started the node multiple times and server is not running out of memory

with config.json:

{
“Archival”: true,
“GossipFanout”: 4,
“BaseLoggerDebugLevel”: 1,
“NetAddress”: “”,
“EndpointAddress”: “0.0.0.0:8887”,
“LogSizeLimit”: 1073741824,
“EnableMetricReporting”: false,
“EnableOutgoingNetworkMessageFiltering”: true,
“EnableIncomingMessageFilter”: false,
“CatchupParallelBlocks”: 50,
“Version”: 12,
“DNSSecurityFlags”: 0
}

getting below error repeatedly in node.log

Logging Starting
Telemetry Disabled
++++++++++++++++++++++++++++++++++++++++
{“file”:“trackerdb.go”,“function”:“ [github.com/algorand/go-algorand/ledger.trackerDBInitializeImpl”,“level”:“warning”,“line”:123,“msg”:“trackerDBInitialize database schema version is 9, but algod supports only 6”,“name”:“”,“time”:“2023-05-30T12:43:00.435378Z”}
{“file”:“node.go”,“function”:“ [github.com/algorand/go-algorand/node.MakeFull”,“level”:“error”,“line”:210,“msg”:“[Stack] goroutine 1 [running]:\nruntime/debug.Stack(0xc000546690, 0xc0001180b8, 0xc0002aca10)\n\truntime/debug/stack.go:24 +0x9f\ngithub.com/algorand/go-algorand/logging.logger.Errorf(0xc000546690, 0xc0001180b8, 0x1334b28, 0x21, 0xc000117ac0, 0x2, 0x2)\n\tgithub.com/algorand/go-algorand/logging/log.go:229 +0x4a\ngithub.com/algorand/go-algorand/node.MakeFull(0x1c97800, 0xc000114f60, 0x7ffda74a0f6f, 0x16, 0x100000015, 0x4, 0x0, 0x0, 0xdf8475800, 0x0, …)\n\tgithub.com/algorand/go-algorand/daemon/algod.(*Server).Initialize(0xc00061db18, 0x100000015, 0x4, 0x0, 0x0, 0xdf8475800, 0x0, 0x0, 0x1e, 0x0, …)\n\tgithub.com/algorand/go-algorand/daemon/algod/server.go:162 +0x903\nmain.run(0x0)\n\tgithub.com/algorand/go-algorand/cmd/algod/main.go:295 +0x1243\nmain.main()\n\tgithub.com/algorand/go-algorand/cmd/algod/main.go:61 +0x7b\n”,“time”:“2023-05-30T12:43:00.438043Z”}

post by fabrice on May 31, 2023

Unknown fields usually mean the software is outdated.

Can you provide the following information:

• OS version
• machine specs: number of CPU, size of RAM, disk type (NVMe SSD, SATA SSD, …)
• actual use: memory available, disk available, …
• algod version: algod -v
• goal version: goal version -v
• goal status: goal node status
• content of config.json (in the data folder $ALGORAND_DATA)
• link to the files algod-out.log, algod-err.log, node.log (in the data folder $ALGORAND_DATA) uploaded in a Github gist or equivalent system

post by Sowna on Jun 1, 2023

Sure @fabrice ,

OS version: Ubuntu 18.04.5 LTS

CPU: 16

RAM: 64GB

Disk Type: SSD

Disk : 8TB

Available Disk : 1.2 TB

algod version: 12885884929 3.15.1.stable [rel/stable] (commit #5f2d5ede)

goal version: 3.15.1.stable

goal node status : root@*****:/# goal node status

Last committed block: 29117368

Time since last block: 0.0s

Sync Time: 207.7s

Last consensus protocol: GitHub - algorandfoundation/specs at 44fa607d6051730f5264526bf3c108d51f0eadb6

Next consensus protocol: GitHub - algorandfoundation/specs at 44fa607d6051730f5264526bf3c108d51f0eadb6

Round for next consensus protocol: 29117369

Next consensus protocol supported: true

Last Catchpoint: 29110000#FI2DUXOJUME3GMA6C2TIQOV2KWT7BOKKJGQCHP7UF6JIO2XIYEKA

Genesis ID: mainnet-v1.0

Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

Config.json:

node.log:

algod-err.log:

Cannot read genesis file /var/lib/algorand/genesis.json: open /var/lib/algorand/genesis.json: no such file or directory

algod-out.log is empty

Hope this helps in narrow down the issue.

post by fabrice on Jun 1, 2023

It looks like you are using the user root, which is a bit dangerous and may create issues with permissions.

It is best to run the node under a non-privileged user.

If you install using the package manager, see the notes here: Install a node - Algorand Developer Portal

My guess is still that there are multiple installations of algod or weird permission issues.