Another blockchain puzzle: Last irreversible blocks
Background
There's a behavior that I've noticed for a year or more.
Pixabay License from TungArt7
As a general rule, when I'm reading content from the "latest" block on the blockchain, I use the field last_irreversible_block_num from the get_dynamic_global_properties call.
I do this as an effort to guarantee that the data won't be changed by blockchain reorganizations. (back in the proof of work mining days, I would frequently see blocks awarded and then taken away by blockchain reorgs. I suppose it's not very common now, with dpos only, but you never know). Further, according to this, virtual operations are not available until last_irreversible_block_num.
Virtual operations (curation rewards, etc) are derived from blockchain activity, but aren’t actually stored as operations themselves. They happen based on consensus from the blockchain based on other user initiated operations. These virtual operations are NOT available on the head block, so a 100% live feed of this information would not be possible. In order then to follow these operations you would have to stream the last_irreversible_block.
I don't remember my source, so perhaps I'm mistaken, but my understanding is that the last_irreversible_block_num should be set when it has been validated by 2/3 + 1 of the 21 witnesses in a validation round - i.e. 15.
However, what I have seen is that the last irreversible block changes somewhat arbitrarily, and I don't understand why.
Example
For example, look at the irregular block updates that I recorded when running SAM yesterday. This sort of behavior is consistently observable, and changing the api endpoint doesn't seem to make a difference.
Screen capture
Text capture
I was finally curious enough about this behavior that I slapped together a script to check whether the last irreversible block had changed whenever a new head block was observed. I threw it together in a hurry (it was a holiday, after all 😉), so mistakes are possible, but the results seem plausible.
At the same time that I was recording SAM, I used that script in another window to collect information from 40 get_dynamic_global_properties calls where the head_block_number changed. We can see that even though the head_block_number changed, the last_irreversible_block_num was unchanged in 18 of them - almost 50%.
Here's the script:
$ awk '{print " "$0}' chkBlockProgress.sh
#!/bin/bash
COUNT=0
chgCount=0
noChange=0
blockLib=0
blockHead=0
MAXBLOCKS=${1:-40}
apiEndpoint=https://api.steemit.com
echo "Time Head Last Irreversible LIB Status Unchanged Changed Total Witness Distance"
blockInfo=$(curl -s --data '{"jsonrpc":"2.0", "method":"database_api.get_dynamic_global_properties", "id":1}' ${apiEndpoint} | jq -Src '.result | [.time,.head_block_number, .last_irreversible_block_num] | @csv' | sed 's/\"//g')
while [ ${COUNT} -lt ${MAXBLOCKS} ]
do
COUNT=$(( ${COUNT} + 1 ))
blockTime=$(echo ${blockInfo} | awk -F, '{print $1}')
oldLib=${blockLib}
blockLib=$(echo ${blockInfo} | awk -F, '{print $3}')
Witness=$(curl -s --data '{"jsonrpc":"2.0", "method":"condenser_api.get_block_header", "params":['${blockHead}'], "id":1}' ${apiEndpoint} | jq -Src .result.witness)
echo -n ${blockTime} ${blockHead} ${blockLib}
if [ ${oldLib} -eq ${blockLib} ]
then
noChange=$(( ${noChange} + 1 ))
echo " << --- No change: ${noChange} ${chgCount} $(( ${noChange} + ${chgCount})) ${Witness} $(( ${blockHead} - ${blockLib} ))"
else
chgCount=$(( ${chgCount} + 1 ))
echo " << --- Changed: ${noChange} ${chgCount} $(( ${noChange} + ${chgCount})) ${Witness} $(( ${blockHead} - ${blockLib} ))"
fi
oldHead=${blockHead}
while [ ${oldHead} -eq ${blockHead} ]
do
sleep 1
blockInfo=$(curl -s --data '{"jsonrpc":"2.0", "method":"database_api.get_dynamic_global_properties", "id":1}' ${apiEndpoint} | jq -Src '.result | [.time,.head_block_number, .last_irreversible_block_num] | @csv' | sed 's/\"//g')
blockHead=$(echo ${blockInfo} | awk -F, '{print $2}')
done
done
And here's the output:
Time Head Last Irreversible LIB Status Unchanged Changed Total Witness Distance
2024-12-25T16:51:09 0 91544443 << --- Changed: 0 1 1 null -91544443
2024-12-25T16:51:09 91544458 91544443 << --- No change: 1 1 2 future.witness 15
2024-12-25T16:51:15 91544460 91544445 << --- Changed: 1 2 3 dhaka.witness 15
2024-12-25T16:51:18 91544461 91544445 << --- No change: 2 2 4 xpilar.witness 16
2024-12-25T16:51:21 91544462 91544445 << --- No change: 3 2 5 symbionts 17
2024-12-25T16:51:24 91544463 91544445 << --- No change: 4 2 6 steemchiller 18
2024-12-25T16:51:30 91544465 91544446 << --- Changed: 4 3 7 steem.history 19
2024-12-25T16:51:33 91544466 91544449 << --- Changed: 4 4 8 parse 17
2024-12-25T16:51:36 91544467 91544449 << --- No change: 5 4 9 boylikegirl.wit 18
2024-12-25T16:51:42 91544469 91544449 << --- No change: 6 4 10 future.witness 20
2024-12-25T16:51:45 91544470 91544450 << --- Changed: 6 5 11 etainclub 20
2024-12-25T16:51:48 91544471 91544451 << --- Changed: 6 6 12 ety001 20
2024-12-25T16:51:51 91544472 91544452 << --- Changed: 6 7 13 dev.supporters 20
2024-12-25T16:51:54 91544473 91544457 << --- Changed: 6 8 14 rnt1 16
2024-12-25T16:52:00 91544475 91544459 << --- Changed: 6 9 15 dhaka.witness 16
2024-12-25T16:52:03 91544476 91544461 << --- Changed: 6 10 16 upvu.witness 15
2024-12-25T16:52:06 91544477 91544462 << --- Changed: 6 11 17 faisalamin 15
2024-12-25T16:52:09 91544478 91544463 << --- Changed: 6 12 18 stmpak.wit 15
2024-12-25T16:52:15 91544480 91544465 << --- Changed: 6 13 19 h4lab.witness 15
2024-12-25T16:52:18 91544481 91544466 << --- Changed: 6 14 20 justyy 15
2024-12-25T16:52:21 91544482 91544466 << --- No change: 7 14 21 bangla.witness 16
2024-12-25T16:52:24 91544483 91544466 << --- No change: 8 14 22 faisalamin 17
2024-12-25T16:52:30 91544485 91544467 << --- Changed: 8 15 23 rnt1 18
2024-12-25T16:52:33 91544486 91544467 << --- No change: 9 15 24 future.witness 19
2024-12-25T16:52:36 91544487 91544467 << --- No change: 10 15 25 h4lab.witness 20
2024-12-25T16:52:39 91544488 91544467 << --- No change: 11 15 26 justyy 21
2024-12-25T16:52:45 91544490 91544470 << --- Changed: 11 16 27 roadofrich 20
2024-12-25T16:52:48 91544491 91544470 << --- No change: 12 16 28 upvu.witness 21
2024-12-25T16:52:51 91544492 91544470 << --- No change: 13 16 29 ety001 22
2024-12-25T16:52:54 91544493 91544472 << --- Changed: 13 17 30 etainclub 21
2024-12-25T16:53:00 91544495 91544472 << --- No change: 14 17 31 dlike 23
2024-12-25T16:53:03 91544496 91544475 << --- Changed: 14 18 32 boylikegirl.wit 21
2024-12-25T16:53:06 91544497 91544482 << --- Changed: 14 19 33 seven.wit 15
2024-12-25T16:53:12 91544499 91544484 << --- Changed: 14 20 34 steem.history 15
2024-12-25T16:53:15 91544500 91544485 << --- Changed: 14 21 35 steem-agora 15
2024-12-25T16:53:21 91544502 91544487 << --- Changed: 14 22 36 dev.supporters 15
2024-12-25T16:53:24 91544503 91544487 << --- No change: 15 22 37 symbionts 16
2024-12-25T16:53:27 91544504 91544487 << --- No change: 16 22 38 faisalamin 17
2024-12-25T16:53:30 91544505 91544487 << --- No change: 17 22 39 dlike 18
2024-12-25T16:53:36 91544507 91544487 << --- No change: 18 22 40 steem-agora 20
The head block advanced by 49, but the last irreversible block only advanced by 44.
I also ran the same script through 1000 blocks, and observed an overall set of 364 blocks where the head_block_number changed, but the last_irreversible_block_num didn't. This is about 36% of blocks during a stretch of ~50 minutes.
The Questions
So, my theoretical understanding is that if no blockchain reorganizations are taking place, the last_irreversible_block_num should always be 15 less than the head_block_number, and it should be updated every time the head_block_number updates.
In practice, that's not even close to what I am seeing.
When I first noticed this, I was chalking it up to the fact that we were seeing decent numbers of missed blocks from top-tier witnesses. However, that problem seems to have been addressed, and I don't think we still have enough missed blocks to account for this discrepancy.
So, finally, my questions are, why is this happening? Am I misunderstanding something? And, assuming my understanding of how it "should" work is correct, is this irregular update rhythm introducing latency into the Steem user experience?
Update:
In PowerBi, I downloaded the 500 historical global_dynamic_properties records from SteemDB, and observed the same phenomenon. It would appear that from June until now, only about 26% of blocks were "settled" in 15 validation intervals.
Thank you for your time and attention.
As a general rule, I up-vote comments that demonstrate "proof of reading".
Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.
Pixabay license, source
Reminder
Visit the /promoted page and #burnsteem25 to support the inflation-fighters who are helping to enable decentralized regulation of Steem token supply growth.
Very interesting topic. Maybe we can work on it together. I would have a look at the blockchain code later.
As a first approach, I would consider the witness shuffle...
0.00 SBD,
0.87 STEEM,
0.87 SP
I was curious so I decided to take a quick look at the code. I didn't absorb enough to fully understand what was going on, but in the database::update_last_irreversible_block() function I see it using this:
where STEEM_IRREVERSIBLE_THRESHOLD is (75 * STEEM_1_PERCENT), so I think the calculation is probably supposed to be 75% of 21, which I think works out to the same 15 as your 66%+1.
0.00 SBD,
0.00 STEEM,
0.84 SP
Thanks. That matches observations, too. 15 definitely seems to be the lower boundary. (According to the AIs, that's the block producer plus 14 confirmers😉.)
0.00 SBD,
0.02 STEEM,
0.02 SP