Gridcoin SQL Database
Hi, everyone!
@sc-steemit made a fantastic job doing gridcoinstats. But, you know, sometimes it's never enough! I was looking for some stats that are not available on gridcoinstats. I will not detail what I wanted to build on this post because I'm still developing it! Maybe I'll post something if I find it interesting.
First, I was playing with the daemon, but it's not that easy to fetch information on the blockchain if you only count on the client. RPC commands are clearly not made to query a lot of information and it takes ages to fetch it.
So, I was looking for a database already made or something to store the data and fetch information. After digging out the internet for some times, I found the Haskell script of @cm-steem. I had never worked with Haskell before. After looking at the syntax I told myself that I would spend hours debugging if something went wrong with this code. Sorry @cm-steem, but Haskell is unreadable for me! ;)
Anyway, I decided to build my own database with Python3 and SQLite3. SQLite is not the fastest database of the world, but at least it's quick to build, quite fast to access and you have all the power given by SQL to query the data you need.
So after spending a few days making the script and letting it suck all the blockchain block by block. I finally have my database to work on!
SQL Database Released
After building this database I decided to release it. So, if you want to make fancy stats about the Gridcoin blockchain, you can start with that!
The script that I used to build the database is available here. The database can be used without the daemon but if you want to sync your database, it's mandatory to have the daemon running.
Since it takes an incredible amount of time to sync a million blocks. I've uploaded an archive of the database available here, download it if you want to reduce the sync time. The database size is about 1Go and 370Mo for the archive.
Disclaimer:
- It's not an official database.
- Use this database at your own risk.
- It's your responsibility to check if this database is not corrupt/incorrect/inaccurate.
- You can tell me if something goes wrong.
- You can fix it if something goes wrong.
EDIT
I may have shouted victory too quickly ...
After playing some time on my database, I found a bug that i'm solving! The database is still in development. I'll let you know when I have news!
Feel free to 
!
Great work @chronosamoht!
I've invited you to the gridcoin-community org on github :)
oh yeah !
I just forked my script in the gridcoin hub ! ;)
This is awesome, thanks for the work. I too tried to use the Haskell script @cm-steem (as suggested by @sc-steemit) - after spending hours downloading and installing the software, dependencies, etc. I still kept getting lots of compiler and run-time errors - likely because I have no idea how to use Haskell.
I'm really exciting for your project because I've been looking for this for months. Thank you for sharing and providing, I look forward to playing around with it.
Did I get you right that you created a database for the blockchain of gridcoin ? Wouldn't it be possible to sync this against the chain you need to have for the wallet ? Hmm interesting idea should be fun to play around with that info. I doubt though that sqlite will have the proper power I you want to do a lot on it... But I can be wrong :) anyways nice project !
I created a sql database with it, yes.
I'm not sure of the format used in the blockchain. Maybe something exotic using merkle tree. To be honnest, I didn't check the datastructure of the blockchain itself.
Actually sqlite works better that I expected! Now, I think that the file system or/and the disk speed are limiting me and some complex queries takes more than a minute to run. But it's enough for what I want to do !
Hey @chronosampht - Join - #teamsweden !
[ #teamsweden ] Uniting the Swedish Community | Get Your Own Custom Footer! - Original art @jnart
Hey @chronosamoht, you forgot to put your GRC wallet address in the post ; ) = )
Thank you for this. It will be used.
Woah, I never asked for this because Steemit pay me well for that!
There it is : S5u3bTUxmfraYoYJ367dP7hhhJvR7jR274
And You can find it in the Readme at the end!
Wow that sounds really interesting, will be looking in the coming days!
Nice work. I'm sure your tool will be a great contribution for some new and interesting details.
I will definitely check it out when I am home this weekend. Thanks a lot for this! :)
Excellent. Glad to see more tools being released for our Gridcoin community 🙂
Hi, can you update the readme with instructions on how to use the script, I placed it in the gridcoin directory but Im having problems, Im guessing I dont quite understand the requirements to use the script.
I've updated the readme. You can DM me on slack if it's still not working.
Cheers I will take a look
OK Running on Linux (Windows gridcoinresearchd broken for now), but I had to add a shebang, put issue on your github. Building DB now, exciting stuff!
PS: Python 2.7 didnt run the script, syntax error row 67 def insertToBDD(item: tuple)