Verbaltech2 - BitShares Witness Report - Monthly Update

I have been extremely busy in May, upgrading my nodes several times with new versions for both the testnet and main net as well as for PeerPlays nodes. The primary focus was completing the rollout of the docker based deployment system for all nodes, including the testnets. There are few nodes that do not yet have a functional web interface, such as the testnet for reasons I have yet to discover. They are fully functional aside from that.

I stopped feeding BTWTY after resuming it briefly, due to problems with non-BTWTY assest deposits into the BTWTY account that substantially increase log volume. Also, since black swan recovery has been possible for several months now and not seeing anyone stepping up to recover BTWTY, and, the creators of BTWTY have been focusing on a new index fund with the same top 20 coins but with different pricing point, and that very few other witnesses are feeding BTWTY I felt the time was right to discontinue BTWTY feeds. I am open to feeding the new asset when more information is available to do so.

Failover protection has saved my nodes on numerous occasions in the last several months. With the increase in missed blocks on the rise throughout the BitShares ecosytem and several witnesses experiencing hundreds of missed blocks within a 7 day period, I see more attention being drawn to this problem. An issue has been raised and discussed in github by abitmore concerning this back in November but is still open. It has overlap with other performance issues related to the maintenance window which are being addressed as well. I am optimistic we'll see improvements for this rolled out in a release, hopefully soon.

Lastly I will apologize for a significant problem witness verbaltech2 was responsible for that led to the witness double producing blocks. Double producing blocks is an major problem and steps must be taken to insure it never happens, such as making sure each potential witness node has a unique signing key and there are monitors in place to detect when such things occur. With the increased workload I had on my plate last month and all the node consolidation and juggling I was doing I failed to remove an old node from the list of potential witness nodes available to be activated when blocks are missed. I completed the addition of a new node around 2AM and tested it was working, and put it into production. It was signing blocks when I went to bed. However, I failed to remove an old node from the missed block monitor and as soon as the new node reached the 3 missed block limit the failover monitor switched in the signing key for the next available witness node, which happened to be the same key for the old node I failed to remove. This caused 2 witness nodes to generate blocks, which causes forks in the chain. Although the resiliency of the blockchain can handle that, it is something to be avoided at all costs. Moreover, the email server I use to monitor my nodes was clogged by oversized log files and was not performing, causing a delay of several hours to receive the double signing messages. As soon as I was aware of the issue I worked quickly to discover the cause and resolve it, but the period of double producing didn't end until just after 9AM, several hours from when it started.

To help guard against that in the future I have changed my procedures and all new nodes are initially deployed with a never to be activated signing key to prevent them being activated as a block producer (BP) until I have verified any old node to be deactivated has indeed been deactivated. To be frank, this measure would not have prevented the mishap, and I have always been very careful to avoid double producing, a real concern when you run multiple nodes for reliability and redundancy. This is the first time in my 3 year history as a witness this has occurred on my nodes. This was a mistake that cost me votes and I will be even more cautious from now on to avoid it from occurring again. It will not happen again!

There was an increase in the number of witnesses from 21 to the current 25, a roughly 25% reduction of witness pay but a corresponding increase in decentralization which is well worth the tradeoff. Moreover the increase allows new witnesses to come onboard. I am in favor of that, but it still concerns me there is little emphasis placed on qualifying new candidates by seeing them first operate feeds and a witness on the testnet.

Here is the market summary since last month (all prices via CMC): BTC=$7,423.70, BTS=$0.215, STEEM=$2.26, PPY=$3.76.

---

Please note that I have upgraded 3 of the 5 nodes, increasing their RAM and eliminating the lat of the shared VPS systems that were in use. For now the geolocation diversity has been reduced, but still fairly well distributed. I will resume reporting on the status of the testnet after I resolve the problem with the web interface on it.

Amsterdam

Gemany, node 1

Germany node 2

Singapore

And the testnet server for BTS:

Still active, but temporarily reporting is offline

Your vote for witness
Verbaltech2
is greatly appreciated! Thanks for your time and attention