Ipfs attempts to consume all memory, ETH node can’t sync

I’m having trouble keeping up with attestations since the merge. Server has plenty of power, disk and mem, but just cannot keep up. I removed & reinstalled geth completely and it was fine for a couple days. In grafana, I saw ipfs consuming almost all available memory. Could this be the issue? I restarted ipfs and it cut it back, but seems to want to consume as much as possible if extra mem is found (ipfs was using 16Gb ram before I restarted it). Any other suggestions on what to look into? Core i7 w/ 32gb ram, 4tb ssds. Ive turned off all other projects - it’s only geth, prysm and web3signer now - still can’t sync the geth or prysm chains well.

Thanks :pray:

1 Like

Here’s some pics of grafana. After restarting ipfs, mem usage went down, but it has a slow rise looks like to infinity and beyond. Server was able to catch up and sync both chains and is attesting now. How about switching ipfs to lowpower in config?

looking at the logs in geth, everything goes along just fine until I get this… the goroutines keep going until it looks like geth restarts… anyone know what it means? thx

fatal error: index out of range

runtime stack:
runtime.throw({0x14c10fe?, 0x2091360?})
	runtime/panic.go:992 +0x71
runtime.panicCheck1(0x0?, {0x14c10fe, 0x12})
	runtime/panic.go:35 +0x87
runtime.goPanicIndexU(0xf03028, 0x400000)
	runtime/panic.go:92 +0x34
runtime.heapBitsForAddr(...)
	runtime/mbitmap.go:321
runtime.scanobject(0xc000057238?, 0xc000057238?)
	runtime/mgcmark.go:1246 +0x325
runtime.gcDrain(0xc000057238, 0x7)
	runtime/mgcmark.go:1081 +0x1d4
runtime.gcBgMarkWorker.func2()
	runtime/mgc.go:1295 +0x6e
runtime.systemstack()
	runtime/asm_amd64.s:469 +0x49

goroutine 51 [GC worker (idle)]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:436 fp=0xc00041e758 sp=0xc00041e750 pc=0x468700
runtime.gcBgMarkWorker()
	runtime/mgc.go:1263 +0x1b1 fp=0xc00041e7e0 sp=0xc00041e758 pc=0x41cc51
runtime.goexit()
	runtime/asm_amd64.s:1571 +0x1 fp=0xc00041e7e8 sp=0xc00041e7e0 pc=0x46a921
created by runtime.gcBgMarkStartWorkers
	runtime/mgc.go:1131 +0x25

goroutine 1 [chan receive, 170 minutes]:
github.com/ethereum/go-ethereum/node.(*Node).Wait(...)
	github.com/ethereum/go-ethereum/node/node.go:534
main.geth(0xc00074c040)
	github.com/ethereum/go-ethereum/cmd/geth/main.go:354 +0x15e
github.com/urfave/cli/v2.(*App).RunContext(0xc000102b60, {0x17cc0e8?, 0xc000040060}, {0xc0000b8340, 0x19, 0x1a})
	github.com/urfave/cli/v2@v2.10.2/app.go:350 +0x9d3
github.com/urfave/cli/v2.(*App).Run(...)
	github.com/urfave/cli/v2@v2.10.2/app.go:247
main.main()
	github.com/ethereum/go-ethereum/cmd/geth/main.go:267 +0x47

.........>

goroutine 3971609 [select]:
github.com/ethereum/go-ethereum/core/state.(*subfetcher).loop(0xc046c79930)
	github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:307 +0x4d8
created by github.com/ethereum/go-ethereum/core/state.newSubfetcher
	github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:236 +0x1cb
INFO [09-21|17:33:47.897] Starting Geth on Ethereum mainnet... 
INFO [09-21|17:33:47.898] Bumping default cache on mainnet         provided=1024 updated=4096