Validator not working

Essohara · 3 December 2020 19:10

Hi Everyone,
I have tried all that I can but my validator wont work.
I generated with the eth-deposit-cli for a Mac
I imported the keystore json to prysm and all seemed ok but after 3 days the validator continues to show this:
time=“2020-12-03 17:38:47” level=warning msg=“Running on ETH2 Mainnet” prefix=flags
time=“2020-12-03 17:38:47” level=warning msg=“Enabled full web mode, wallet password will be written to disk at the wallet directory upon completing web onboarding.” prefix=flags
time=“2020-12-03 17:38:47” level=info msg=“Enabling web portal to manage the validator client” prefix=node
time=“2020-12-03 17:38:47” level=info msg=“Opened validator wallet” keymanager-kind=direct prefix=node wallet="/root/.eth2validators/direct"
time=“2020-12-03 17:38:47” level=info msg=“Checking DB” databasePath="/root/.eth2validators/direct" prefix=node
time=“2020-12-03 17:38:47” level=warning msg=“You are using a non-default web host. Web traffic is served by HTTP, so be wary of changing this parameter if you are exposing this host to the Internet!” prefix=node web-host=0.0.0.0
time=“2020-12-03 17:38:47” level=info msg=“Starting Prysm web UI on address, open in browser to access” address=“http://0.0.0.0:80” prefix=node
time=“2020-12-03 17:38:47” level=info msg=“Starting validator node” prefix=node version=“Prysm/v1.0.3/fc7c6776f66c675dfc0e26a9f16141720bf9f2df. Built at: 2020-12-01 14:01:51+00:00”
time=“2020-12-03 17:38:47” level=warning msg=“You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc” prefix=validator
time=“2020-12-03 17:38:47” level=info msg=“Starting gRPC gateway” address=“0.0.0.0:80” prefix=gateway
time=“2020-12-03 17:38:47” level=info msg=“gRPC server listening on address” address=“0.0.0.0:7000” prefix=rpc
time=“2020-12-03 17:38:47” level=info msg=“Waiting for beacon chain start log from the ETH 1.0 deposit contract” prefix=validator
time=“2020-12-03 17:38:47” level=info msg=“Beacon chain started” genesisTime=2020-12-01 12:00:23 +0000 UTC prefix=validator

I have restarted Prysm and the beacon comes up fine each time. All the ports are open but still no-go with the validator.
In Prysm web I get this message: showing a http failure response
unknown/eth/v1alpha1/beacon/chainhead: 0 Unknown Error
Also I cannot see any wallet or account information.
Hoping that someone can help me out.
Many Thanks

Support info: * dappmanager.dnp.dappnode.eth: 0.2.37, commit: 2b96e582

core.dnp.dappnode.eth: 0.2.41
wifi.dnp.dappnode.eth: 0.2.5
ipfs.dnp.dappnode.eth: 0.2.12
vpn.dnp.dappnode.eth: 0.2.7, commit: 12ae223f
bind.dnp.dappnode.eth: 0.2.6

System info

docker version: Docker version 18.09.8-ce, build 0dd43dd87fd530113bf44c9bba9ad8b20ce4637f
docker compose version: docker-compose version 1.25.5, build unknown
platform: linux, x64, 5.9.0-3-amd64
Disk usage: 332430468710

ruvenni · 4 December 2020 10:26

Hi @Essohara and welcome to our community forum!

I will help if you make it clear if this is happening in Prysm Pyrmont or Prysm Mainnet along with some screenshots of the UI

Essohara · 4 December 2020 12:10

Hi Ruvenni,

Thanks very much for your reply. Its nice to know that someone is out there
As far as I can tell its mainnet. However now that you mention it- I used the eth2deposit-cli-ed5a6d3-darwin-amd64. which is pyrmont.
The eth2 transaction is shown on mainnet. My screenshots are shown here:

If I run eth2deposit-cli-9310de0-darwin-amd64/deposit ; I see these options:

Usage: deposit [OPTIONS] COMMAND [ARGS]…

Options:
–help Show this message and exit.

Commands:
existing-mnemonic Generate (or recover) keys from an existing mnemonic
new-mnemonic Generate a new mnemonic and keys

[Process completed]
Do you know which command is appropriate? of indeed if either will work?

Thanks
Seamus

ruvenni · 4 December 2020 15:59

Maybe I’m wrong but I have the feeling that you are messing Pyrmont Testnet and Eth2.0 Mainnet.

They have different launchpads, and different client validators. One is for testing and uses Goerli and the other (mainnet) uses real ETH you may loose if you are not really careful on what you do.

I’d recommend you to a look at our guide on how to stake and use the search and join our discord to get some help for TESTING in Pyrmont before going real with staking.

Essohara · 4 December 2020 17:56

Thanks ruvenni,
Yes, I followed that guide but I think i still managed to select the incorrect Pyrmont key generator. DOH!
Still, There may be some hope- I sent real ETH and I see that the transaction is associated with a validator as shown below.
Is it possible that it is still queued?

Seamus

ruvenni · 10 December 2020 16:58

Hi Seamus

Have you been able to make any progress? Did you joined Discord or tried to keep on testing in Pyrmont before anything?

Essohara · 10 December 2020 19:37

Hi Ruvenni,
Thanks for checking in. At the first posting it was already too late to do testing. I only posted after I had deposited 32 ETH to mainnet and found that the validator was not working. So far I can see that 27601 is activated but not making attestations.
I joined the Prysm discord and enquired there and, there is a concern that ports 13000 and 12000 are not open. I Asked my ISP to change my home modem/router to IPV4 so I could make the necessary port forwarding. The changed the router config from IPv6 to IPv4 and the the configuration seems to be simple and correct ( i’ve done it successfully before for things like NAS, in the pre cloud era) but when I check the port status with Gibson research grc.com it indicates that ports 12000 and 13000 are closed. All the other ports necessary were opened automatically with UPnP. I have a service request with the ISP to check my router again and see if they can solve the problem with the ports. I find the ports issue interesting because as far as I can see ports 12000 and 13000 should affect the beacon chain and not the validator. However the beacon chain seems to run fine. I’ve added the logs so you can see.

All times in are in UTC and I have verified with the timeis server.

According to Dappnode logs. GETH looks good

INFO [12-10|19:00:09.072] Deep froze chain segment blocks=5 elapsed=82.178ms number=11336954 hash=“034201…948757”
INFO [12-10|19:00:29.678] Imported new chain segment blocks=1 txs=256 mgas=12.530 elapsed=174.112ms mgasps=71.963 number=11426955 hash=“ab86b0…f51af6” dirty=1022.78MiB
INFO [12-10|19:00:41.713] Imported new chain segment blocks=1 txs=316 mgas=12.523 elapsed=129.826ms mgasps=96.462 number=11426956 hash=“385f71…b7d6b6” dirty=1022.52MiB
INFO [12-10|19:00:47.312] Imported new chain segment blocks=1 txs=219 mgas=12.535 elapsed=138.628ms mgasps=90.424 number=11426957 hash=“952886…d7ca03” dirty=1022.43MiB

Beacon-chain looks good

time=“2020-12-10 19:01:35” level=info msg=“Finished applying state transition” attestations=7 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0
time=“2020-12-10 19:01:47” level=info msg=“Synced new block” block=0x5e76c7cc… epoch=2090 finalizedEpoch=2088 finalizedRoot=0x7260a1dc… prefix=blockchain slot=66907 slotInEpoch=27
time=“2020-12-10 19:01:47” level=info msg=“Finished applying state transition” attestations=21 attesterSlashings=0 deposits=0 prefix=blockchain proposerSlashings=0 voluntaryExits=0

Validator is stuck. I restarted it again this evening.

When I restarted the Validator the Beacon chain logs showed:
time=“2020-12-10 19:03:57” level=info msg=“New gRPC client connected to beacon node” addr=“172.33.0.7:37964” prefix=rpc
time=“2020-12-10 19:03:59” level=info msg=“Synced new block” block=0x9ef5b0c5… epoch=2091 finalizedEpoch=2089 finalizedRoot=0x989cced8… prefix=blockchain slot=66918 slotInEpoch=

And the validator logs show this:

time=“2020-12-10 19:03:56” level=warning msg=“Running on ETH2 Mainnet” prefix=flags
time=“2020-12-10 19:03:56” level=warning msg=“Enabled full web mode, wallet password will be written to disk at the wallet directory upon completing web onboarding.” prefix=flags
time=“2020-12-10 19:03:56” level=info msg=“Enabling web portal to manage the validator client” prefix=node
time=“2020-12-10 19:03:56” level=info msg=“Opened validator wallet” keymanager-kind=direct prefix=node wallet="/root/.eth2validators/direct"
time=“2020-12-10 19:03:56” level=info msg=“Checking DB” databasePath="/root/.eth2validators/direct" prefix=node
time=“2020-12-10 19:03:57” level=warning msg=“You are using a non-default web host. Web traffic is served by HTTP, so be wary of changing this parameter if you are exposing this host to the Internet!” prefix=node web-host=0.0.0.0
time=“2020-12-10 19:03:57” level=info msg=“Starting Prysm web UI on address, open in browser to access” address=“http://0.0.0.0:80” prefix=node
time=“2020-12-10 19:03:57” level=info msg=“Starting validator node” prefix=node version=“Prysm/v1.0.4/b4437e6cecd78257d2699f42073a90a940baf716. Built at: 2020-12-07 16:42:42+00:00”
time=“2020-12-10 19:03:57” level=warning msg=“You are using an insecure gRPC connection. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc” prefix=validator
time=“2020-12-10 19:03:57” level=info msg=“Starting gRPC gateway” address=“0.0.0.0:80” prefix=gateway
time=“2020-12-10 19:03:57” level=info msg=“gRPC server listening on address” address=“0.0.0.0:7000” prefix=rpc
time=“2020-12-10 19:03:57” level=info msg=“Waiting for beacon chain start log from the ETH 1.0 deposit contract” prefix=validator
time=“2020-12-10 19:03:57” level=info msg=“Beacon chain started” genesisTime=2020-12-01 12:00:23 +0000 UTC prefix=validator

It never gets beyond this stage.
Is there some way to reinstall the Prysm validator and restart the wallet? I have the mnemonic and the password.
By the way, I checked, and the actual cli tool I used was eth2deposit-cli-ed5a6d3-darwin-amd64- Thats Pyrmont. That may be a problem. Is there a substantial difference between that and the Cli for Mainnet ? after all, they both generates the same end products. deposit-data XXXX.json and Keystore XXX.json
I aslo have the mainnet cli eth2deposit-cli-9310de0-darwin-amd64 available but I dont know if I can employ it to retrieve my wallet.
Maybe I should rebuild from scratch? I have the Dappnode image installed on an i5 NUC.

Ultimately I suppose within a few days I will need to decide if I can get this functioning or if I should exit to avoid loosing ETH and leave the ETH where it is for a couple of years. (I have no intention to sell anyway)
As always your support and insight is greatly appreciated.

Thanks

ruvenni · 16 December 2020 17:48

Although I have only tried to validate on Pyrmont, I’d be careful to strictly follow https://pyrmont.launchpad.ethereum.org/ for testing, and https://launchpad.ethereum.org/ and never mix any of the steps or tools required to be usen in any of the two processes.

It does not seem you need to rush to exit just because you are a few days offline

The protocol will not “slash” inactive validators, merely subtract any incremental earnings as incremental penalties. This means that as long as your validator is online more than 2/3 of the time, you will still turn a profit.

I hope find this message helpful or you have already overcome your issues!

Essohara · 24 December 2020 12:07

Hi Ruvenni,

I made some progress with prysm.

I deleted existing wallets and created a new imported wallet

That has worked and now I have an imported wallet with 1 validator account

Wallet password:

(keymanager kind) imported wallet

Showing 1 validator account

View the eth1 deposit transaction data for your accounts by running `validator accounts list --show-deposit-data

Account 0 | one-two-three

[validating public key] 0xa40598b3a0a93734208eeff41fbcd3d3eb255a3811470dd99b3d7f93a3476367e8679d8f8e76b659623df066955b1fcd

when I run the command

validator accounts list --show-deposit-data

I get this response

Wallet password:

(keymanager kind) imported wallet

Showing 1 validator account

View the eth1 deposit transaction data for your accounts by running `validator accounts list --show-deposit-data

Account 0 | one-two-three

[validating public key] 0xa40598b3a0a93734208eeff41fbcd3d3eb255a3811470dd99b3d7f93a3476367e8679d8f8e76b659623df066955b1fcd

If you imported your account coming from the eth2 launchpad, you will find your deposit_data.json in the eth2.0-deposit-cli’s validator_keys folder

I have found the deposit_data.json file but can you tell me where I need to put it ?

Note:

the validator accounts import gave this result

validator accounts import keystore-xxxxxxx.json

[2020-12-24 11:36:10] WARN flags: Running on ETH2 Mainnet

[2020-12-24 11:36:10] INFO prompt: (wallet path) /root/.eth2validators/prysm-wallet-v2

the prysm webui tells me that the accounts keystore file should be at /root/.eth2validators/direct/all-accounts.keystore.json

and that the wallet directory should be /root/.eth2validators

thanks for your support

Essohara · 27 December 2020 14:51

Its Alive !!
Ultimately the solution was in the onboarding.
I used the Dappnode Admin console and selected Packages and then Prysm.
In the Prysm package I selected Volume and then I found Validator data.
The size was around 33.6 kB. I clicked to remove the Validator data and the size only reduced to around 33k B
This allowed me to go to the top of the page and click on ;
Go to prysm.dappnode to import your keystores and start validating.
I worked through the onboarding process and imported only the keystores .json and entered the passwords and it started working.
I note that now the Validator data size is 313 kB.
I hope this information is helpful to others who may experience difficulties.
Thanks for your help Ruvenni !

system · 3 January 2021 14:51

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.