[WTF is JSON series] Ingestron -> Steemjson: Making STEEM a universal JSON-storage blockchain step by step

faddat (66)in #beyondbitcoin • 8 years ago (edited)

JSON & STEEM

Hey all you non-tech-people, today's article is for you. I'm going to explain JSON's role in today's internet and tell you why it matters to STEEM (the currency), STEEM (the blockchain daemon), steemit.com (the community) and you (presumably a holder of STEEM, STEEM Power, or Steem Backed Dollars. So, with no further ado, here's some JSON:

This is the 2,000,000th block of the STEEM blockchain:

{
  "id": 5,
  "result": {
    "previous": "001e847f77b2d0bc1c29caf02b1a98d79aefb7ad",
    "timestamp": "2016-06-02T23:58:45",
    "witness": "steempty",
    "transaction_merkle_root": "3335e6efe04f09aac61ad1fcc241ada1e1e8fc62",
    "extensions": [],
    "witness_signature": "1f26706cb7da8528a303f55c7e260b8b43ba2aaddb2970d01563f5b1d1dc1d8e0342e4afe22e95277d37b4e7a429df499771f8db064e64aa964a0ba4a17a18fb2b",
    "transactions": [{
      "ref_block_num": 33918,
      "ref_block_prefix": 2329120500,
      "expiration": "2016-06-02T23:58:54",
      "operations": [
        ["vote", {
          "voter": "proctologic",
          "author": "pal",
          "permlink": "re-dantheman-re-pal-httpssteemit-comsteempalsniper-whale-vote-bot-strategy-20160602t162811551z",
          "weight": 10000
        }]
      ],
      "extensions": [],
      "signatures": ["1f0ad8680045212314210892e338f14bc4fd34b2573e6217591b036be6222c5d980dbc2d3547429924389330e876ab650a1dd9548284d8a855c96b58c542d0a499"]
    }, {
      "ref_block_num": 33919,
      "ref_block_prefix": 3167793783,
      "expiration": "2016-06-02T23:58:57",
      "operations": [
        ["vote", {
          "voter": "proctologic",
          "author": "oholiab",
          "permlink": "re-dantheman-re-streemian-re-re-dantheman-lessons-learned-from-curation-rewards-discussion-20160602t150813-20160602t161538485z",
          "weight": 10000
        }]
      ],
      "extensions": [],
      "signatures": ["1f45456b18a8df371cb1890f15f4e9a7b59b9759d9982d3e96429343c2899fcda668a295bfd6eaa630785ed807aa93485301723870d6f34769bc25a6fbc91d0253"]
    }, {
      "ref_block_num": 33919,
      "ref_block_prefix": 3167793783,
      "expiration": "2016-06-02T23:58:57",
      "operations": [
        ["vote", {
          "voter": "proctologic",
          "author": "abit",
          "permlink": "re-dantheman-lessons-learned-from-curation-rewards-discussion-20160602t160856942z",
          "weight": 10000
        }]
      ],
      "extensions": [],
      "signatures": ["1f5fd52ca5c91b118c8ac2f2f6f6df28bc10122baaabbd3f510c37b3201c86a4845419a0d6a17cf267c73d939f1fa7230c7f9f85e60b784e54cf041091d0cbb41f"]
    }]
  }
}

It contains all the information that's needed to use STEEM's 2,000,000th block. Posts, upvotes, everything. Now, the reverse is true for writes onto the steem blockchain. Every time you click upvote, your upvote is written into one of the blocks in the STEEM blockchain. This works by having your browser send JSON to the blockchain over something called a websocket, which allows your browser to interact with the servers running the STEEM blockchain in real-time.

One of the cool things about that block of JSON that is the 2,000,000th block is that it will ALWAYS make sense to this:

type Block struct {
    ID     int `json:"id"`
    Result struct {
        Extensions            []interface{} `json:"extensions"`
        Previous              string        `json:"previous"`
        Timestamp             string        `json:"timestamp"`
        TransactionMerkleRoot string        `json:"transaction_merkle_root"`
        Transactions          []struct {
            Expiration     string          `json:"expiration"`
            Extensions     []interface{}   `json:"extensions"`
            Operations     [][]interface{} `json:"operations"`
            RefBlockNum    int             `json:"ref_block_num"`
            RefBlockPrefix int             `json:"ref_block_prefix"`
            Signatures     []string        `json:"signatures"`
        } `json:"transactions"`
        Witness          string `json:"witness"`
        WitnessSignature string `json:"witness_signature"`
    } `json:"result"`
}

That there is what is called a struct, or structure, is capable of parsing, or "reading and understanding" ANY block of STEEM's blockchain, and that lets us programmers use it actively. Generally if you see me spouting off that we need a "JSON RPC API" or a "SWAGGER REST API," I'm basically just looking to see these things be done once, instead of by each and every developer out there. We do have a pretty big duplication of effort problem going on in the STEEM community. Usually, we call classifiers "keys" and thieir corresponding "value" values. So for example:

username:faddat
mood:bitchy
favoriteprogramminglanguage:go
hatesbutistryingtohateless:react
tags:geek,startrek,alien,crabbytoday,futurist

Now, note "tags". The "value" field is comma-delimited there. Effectively the commas have allowed me to enter multiple values in a single value field, and the structure value1,value2,value3,value4,value5 allows a computer to easily read and understand those values. key:value pairs make are a fundamental part of today's data storage systems. They seem primitive but can create quite sophisticated results. Hey, remember that struct that can read any block's JSON? Well, Here's something else we can do with it:

Now that struct is set up with notation for a graph database. Graph databases analyize the relationships between items. As in: @faddat, what are his relationships to @kloudery, and poof all the connections will become clear. Well it's not exactly that easy, but let's just say that we're going to see some hotness around here :).

Source code: https://github.com/faddat?tab=repositories
Cayley forums post: https://discourse.cayley.io/t/cayley-and-blockchain-question-on-twitter/359/18

Questions: faddat@gmail.com preferably via hangouts

Research Sponsor: @officialfuzzy of #beyondbitcoin

This post IS about the money!

Now, I want to inform people, too-- it surely serves two purposes, but right now the situation is that @officialfuzzy sponsors my research work for the STEEM blockchain (and this is all the work I do). Now, I'd not be in this position if this wasn't what I wanted to do-- it absolutely is. The issue is money to keep the work afloat. Voluntarily, I cost around $3000 per month. My expenses tend to be $1500 per month, and I've got a three month stretch coming up soon where expenses will be ~$3000/mo.

I could take a much-higher paying position

....I just don't want to. And I sure as hell don't want to NEED to.

I like what I'm doing very much right now and I want to get to my goal of ensuring that STEEM can be used as a storage system for any kind of information that can be structured in JSON. Your upvotes and what have you go directly to my expenses. Fuzzy's got a wife and kid, and he can't pay much more than he's paying now. I get that, and so before going elsewhere for money, I'm taking advantage of the money-making opportunities baked into the blockchain I <3.

#steem #json #blockchains #cayley

8 years ago in #beyondbitcoin by faddat (66)

$724.14

Sort:

Trending

[-]

officialfuzzy (75) 8 years ago (edited)

It is important to note that approximately 6k has gotten us pretty far in terms of something faddat is building as an open source project. This funding has come directly from the SBD earned from my work on the platform. I keep the steempower for my work, which is meant to show everyone I intend to care about the value of steem past just sending it to poloniex.

I also have currently slotted approximately 5k of the SBD earned and held thus far in my account for jake's future work and the work we need others to do to create the #beyondbitcoin site as a template for others to build and improve upon.

Future payouts are not guaranteed for my work so I have asked faddat to begin posting here to give some transparency and sense of the value for what his work provides steem longterm. In return I will be asking others to upvote this real life proof of work via updates.

This doesnt mean I am going to stop giving back of course an paying him from some of what ive been blessed with. It simply means Im asking for help to make sure we can see this through even the hardest times and even if my own payouts dwindle significantly in the future.

$0.23

7 votes

[-]

cryptomancer (63) 8 years ago

I enjoy learning how Steemit works under the covers. Until reading this I had no idea that blockchain data is represented in JSON format, but it seems like a very natural choice. It's such a universally useful format no matter what the exact application is. These days I see it even more than XML, and I personally find it more readable than XML as well. JSON is one of the fundamental formats used to represent data in financial systems too. In my job as a software engineer for a big bank, I use JSON to represent the properties of things like stocks, bonds, swaps, and other financial products. That format is also used to store data about said products in a database (MondoDB is built around the concept of storing data as JSON "documents").

$0.07

2 votes

[-]

faddat (66) 8 years ago

I think this is the true genius behind STEEM. Technically speaking, STEEM can do anything that can be done with JSON, and I think you're catching the drift quite well:

If steem can do anything that can be done with JSON, then steem can do (almost) anything, because that's what is done with JSON these days: (almost) everything.

$0.00

1 vote

[-]

bendjmiller222 (64) 8 years ago

I have a limited knowledge in coding, but find it fascinating. It really is a bit like a foreign language when you start, but once you get the hang of it, it becomes far easier and with the internet quite literally in your back pocket, the answers to your coding questions are a few clicks away.

I sometimes wish there was a little CSS or JavaScript allowed in posts, but am happy that steemit does not want to become the cascading glitter orgy that MySpace devolved into.

Hopefully as I gain more knowledge in coding I will be able to further appreciate all aspects of this post.

$0.00

[-]

faddat (66) 8 years ago (edited)

If you've any questions please don't hesitate to ask :). I tried to make the post as accessible as possible and maybe I failed a bit, so please do let me know what I could have done better: Explaining these concepts is an artform in and of itself and I need to hone my craft in that dept.

$0.00

[-]

bendjmiller222 (64) 8 years ago

No you did a great job explaining. It's extremely difficult to explain coding to a non techie.

You may have to break it down even further into something about what Json is stands for and break it down like you did with examples.

I learn pretty quickly from repetition and memorization, but it took me quite a while to grasp exactly how JavaScript objects work and interact.

Maybe even breaking down a simple app or how one would go about using some of the info and the API to build something would be great.

I know (myself included) there are a lot of people with some great ideas for applications that have no idea how to implement them.

Usually the answer has been "it's quite tricky and I don't know if the way I've learned to do it would be the best way to teach you."

I feel someone taking the time to educate creative non coders in the basics of programming for a blockchain would be invaluable. There is a shortage of people who can code some of the great ideas that are proposed. Maybe even a feeler post like a "how many people would be interested in tutorial x" or take suggestions. Whether that turns out to be html and CSS or JavaScript or another language, I think the monetary support would be there.

I'd love to do it, but my "coding" is mostly limited to basic JavaScript and HTML that allows me to easily format my blog posts (something that should be made far easier so great writers can format their content as easily as Microsoft word and drag and drop pictures from a file sharing site).

Not sure if you have the time, but maybe you could partner up with a few coders and together give some pointers or break down how some of the current apps on steemit were made.

Probably a huge undertaking, but I see extreme value for someone willing to undertake and answer questions like this.

$0.00

1 vote

[-]

faddat (66) 8 years ago

You're entirely right about two things:

The way that the editor should work (I think more like google docs than like word, but yeah-- these posts should be able to hit 1000% sexy.
Teaching people how to code for blockchains is important (I'm going to add some corollaries):
- Providing resources to current coders to transition onto blockchains is important
- Doing this is the best possible thing for the financial health of our community. The easier we make development for STEEM, the more we all stand to gain.

I really appreciate your detailed reply, it was worth much more than the $.02 I was able to give it :).

$0.00

[-]

bendjmiller222 (64) 8 years ago

I got your 2 cents :) actually I feel that commenting has dropped off since people aren't being as heavily rewarded for comments as they once were. I see the value from having people value my input and follow me just as much as a large payout on a single whale vote, but I don't see as much engagement as there was a while back.

To me, comments are sometimes more valuable than the actual blog post. Especially if that post is asking for input from comments.

If the target votes per day were set at ten (or a low number) if I were a whale, there is no question that I would curate far more voting on a blog post than a comment so if I am limited to a certain number of 100% upvoted 9/10 would likely be blog posts.

I'm not sure how this can be addressed, but if comments become less important, some creative minds will cease tospend time crafting well thought out responses and opt for simple "bot like" responses.

A simple comment of "great post" really does very little for either the original poster or the commenter. If a bot that cannot read the article can make the same comment, I have to say it might as well be left unsaid.

I try to make my comments unique and meaningful as often as possible. Occasionally I'll make a brief comment on a picture or give someone a short encouragement, but I see how useless most comments are on things like twitter and YouTube, that I would hate to see the comment section cease to be useful.

Maybe some sort of "Comment Curie" curation team could be made to give rewards to the best ten comments or so of the week.

But that being said, 2 cents is more than I made on any other social media site, so the fact that you are able to share that makes everything worth it.

The more people realize that working together is the best way to keep steemit the better.

Steemfest is something that I believe will be an absolute game changer as some of the most dedicated creative minds will all be in one place. Nobody who isn't serious about steemit will attend so it will be a great experience.

Hope you can make it. I understand people have families, jobs, bills and other things preventing them from taking a 3 day trip to Amsterdam, but I'm quite excited to network and meet the faces behind many that I know solely by their username.

While the feature will most likely be added to change a profile picture, it's kind of nice to not see or judge anyone based on appearance unless they wish to be quite public.

Enough of my comment rambling. Glad you appreciated my comment.

$0.00

2 votes

[-]