SteemData Indexer Update
SteemData now has new, multi-threaded blockchain indexers. This is an long-overdue update, bringing better scalability, performance (one order of magnitude increase) and consistency guarantees.
Further, SteemData has been re-synced from scratch to address past inconsistencies.
Syncing Status
- Operations (100%)
- Accounts (100%)
- Account Operations (99.9%)
- Posts and Comments (93%)
The database is currently @ manageable 150GB. You can see the status of the indexers and other basic stats here.
Real-Time Updates
The last open concern is state change propagation in near-real time.
For example, a single vote operation should trigger cascading updates of all affected Account, AccountOperations and Posts/Comments entries. This is because the vote affects the voters state (voting power), voters history, and the post/comment the vote was cast on.
As the Steem Blockchain usage grows, the amount of dependent state that needs to be updated with each block grows exponentially. Scaling SteemData infrastructure to support this naive model might be tricky in the future. I am currently evaluating 2 options: Using an indexer, which is more efficient and offers better consistency guarantees, vs using a distributed task queue which offers higher capacity (scalability). I hope to have some time this weekend to reach closure on the matter.
hey man,
pardon the off-topic comment…
I made a post today regarding a large-scale idea to advance Steem’s development, and it was recommended to share with the witnesses to help get this in front of the audience who’d be in the position to do something with/about it:
The $1 Billion Steem Development Fund: How Steemit Inc.'s Stake Could Be Best Allocated To Grow A Thriving Network Of Applications And Users...
would be cool if you could have a read, and IF you feel it’d be a great idea that’d serve the community, forward to anyone in particular you know who might be in a position of influence to advance the discussion.
either way, I thank you for your continued service to this community. 🙏
cheers,
Rok
Thanks for giving your service to this community! I truly appreciate that you do this, and I'm happy to support you in any way I can. Even though I don't understand it at all I know MANY depend on it, and that ultimately enhances my own steeming experience. Thanks for doing what you do so that I can come here and play!
Maybe this post can clarify you some questions. https://steemit.com/steem/@lightproject/recap-why-steemdata-is-so-important-for-steem
Yes! Thanks for bringing that to my awareness, @lightproject! This community is so helpful and resourceful!!!
Good job!
Excuse me @furion, accounts Collection seems to be not 100% synched. Been checking "lastpost" timestamps and are not updated. Also found couple of account that are not indexed. For instance, cannot find in Account collection account "vcelier".
Interesting...I'm looking into it.
Just for the record. Accounts that I found missing seem to be indexed now. Timestamps tho, seem not to be synched yet. Thanks a lot in advance for the efforts @furion.
Sounds great and great project!!!
I have just tried to make some Queries. if you get the account "cljg918" ( {name: "cljg918"} )
It says that it have 0 (zero) comments in "comment_count" but if I get comments by this author there is 700 and if I do the same query in steemDB it returns 1386? Do you know why?
HI, your web-api ( https://webapi.steemdata.com )
returns: 503 Service Temporarily Unavailable
Thanks for this update...SteemData very useful great work..
The Charts app is very insightful as well!
The charts app will be restored next week. Also, the Notify app is due for overhaul, and I have 1 unreleased app pending from the Steemfest 2 hackaton.
You are the best writer @furion
oh my god
your post is so amzing and really helped me
thank you very much
please help me to vote my post :) :)
good pos
I need all people post plenty