Steem Documentation: Using the Rake Tasks

First of all, I'm very pleased that all of my pull requests for devportal have reached the Steemit, Inc. official site, as reported yesterday. My goal was to try to keep my fork and the origin fork as 1-to-1 as possible. There's no reason for me to have unique documentation on my fork.

It looks like it took them only a couple days to fully merge my changes that spanned over a month. Great job, Steemit, Inc! I believe moving forward, this should be much easier now that the relative URL structure has been addressed.

Here, I'd like to go over the various rake tasks that have also been merged to the official repository.

Some of these tasks are shown in the @steemitblog post from yesterday. But here's more detail on what they're all about.

Repository

https://github.com/inertia186/devportal

Rake (rake) is a command-line tool that means "Ruby Make." The command implements tasks defined in a Rakefile, usually in a cloned git project.

---

In the devportal Steem documentation project, the rake tasks are as follows:

$ rake -vT
rake ops_dump[vops,appbase]          # Dump all operation types
rake production:deploy               # Deploy current master to GH Pages
rake production:rollback             # Rollback GH Pages
rake scrape:api_defs                 # Scrape API Definitions
rake scrape:javascript               # Scrape steemjs docs
rake scrape:python                   # Scrape pysteem docs
rake scrape:tutorials                # Scrape all known tutorial repositories
rake scrape:tutorials:js[num,force]  # Scrape JS-Tutorials
rake scrape:tutorials:py[num,force]  # Scrape PY-Tutorials
rake scrape:tutorials:rb[num,force]  # Scrape RB-Tutorials
rake test:curl[apis]                 # Tests the curl examples of api definitions
rake test:proof                      # Want some work to do?  Run this report and get busy

I use many of these tasks to help with documentation. Here's a rundown of the most useful ones ...

Scrape API Definitions

For example, rake scrape:api_defs is especially useful. As it says, it scrapes the API. But what does that even mean?

When it scrapes the API, it's asking the (full) node what methods are available and what those method signatures are. Then, it saves the scraped information to a .yml file so that jekyll can make pages to browse. For example, rc_api is based on scraped data saved to rc_api.yml.

I don't just scrape the methods, though. Once scraped, I add details. Also, sometimes the scrape is missing some of them method signatures, so I add those too.

You might think this is only useful for catching up to methods that already exist, but it's also possible to use it for upcoming methods. All I need is to point the task at a node running on a future release. Often, new methods show up on api.steemitdev.com, so this will scrape from there:

TEST_NODE=https://api.steemitdev.com bundle exec rake scrape:api_defs

In fact, I could scrape from a testnet just as easily:

TEST_NODE=https://testnet.steemitdev.com bundle exec rake scrape:api_defs

It also works just as well to run a tintoy testnet if I'm looking for bleeding edge new methods:

docker run -d -p 8090:8090 inertia/tintoy:latest
# need to wait for it to fully deploy
sleep 300
TEST_NODE=http://localhost:8090 bundle exec rake scrape:api_defs

And here's the result when I ran it just now:

Definitions for: account_by_key_api, methods: 1
Definitions for: account_history_api, methods: 4
    Adding: enum_virtual_ops
    Changed: get_account_history
    Changed: get_ops_in_block
    Changed: get_transaction
    4 methods added/updated in account_history_api
Definitions for: block_api, methods: 2
Definitions for: condenser_api, methods: 84
    Dropped method: get_account_bandwidth (recommend removal from _data/apidefinitions/condenser_api.yml)
Definitions for: database_api, methods: 52
    Changed: find_accounts
    Changed: find_comments
    Changed: find_escrows
    Adding: find_smt_token_emissions
    Adding: find_smt_tokens
    Changed: find_vesting_delegation_expirations
    Changed: find_votes
    Changed: find_witnesses
    Changed: get_active_witnesses
    Changed: get_config
    Changed: get_dynamic_global_properties
    Adding: get_nai_pool
    Changed: get_transaction_hex
    Changed: get_witness_schedule
    Changed: list_accounts
    Changed: list_comments
    Changed: list_escrows
    Changed: list_limit_orders
    Changed: list_owner_histories
    Changed: list_savings_withdrawals
    Changed: list_sbd_conversion_requests
    Adding: list_smt_token_emissions
    Adding: list_smt_tokens
    Changed: list_vesting_delegation_expirations
    Changed: list_vesting_delegations
    Changed: list_votes
    Changed: list_withdraw_vesting_routes
    Changed: list_witness_votes
    Changed: list_witnesses
    29 methods added/updated in database_api
Definitions for: follow_api, methods: 10
Definitions for: jsonrpc, methods: 2
Definitions for: market_history_api, methods: 7
Definitions for: network_broadcast_api, methods: 2
Definitions for: rc_api, methods: 3
    Changed: find_rc_accounts
    Changed: get_resource_params
    Changed: get_resource_pool
    3 methods added/updated in rc_api

This is how I can be ahead of any changes that might roll out. In this example, there seems to be a lot of changes, but most of them are just method signatures with keys in different orders. The most interesting results, to me, are things like Adding: get_nai_pool.

If we focus only on that result, here's what we see newly added to database_api.yml:

  - api_method: database_api.get_nai_pool
    purpose: 
    parameter_json: "{}"
    expected_response_json: '{"nai_pool":[]}'

What I usually do at this point is search for the new method in the steemd repo. In this case, that leads to database_api.cpp and pull request #3058. I keep working backwards until I find the reason the method was added. Actually, this one is a little tricky. Looks like I need to investigate issue #2886. But I'll do that later.

Test Curl

The rake test:curl[apis] task is like a smoketest for the curl examples in the API Definitions. The rationale is that if we assume these curl examples are correct, then testing them periodically should show us problems in the API and/or node configuration. Conversely, it could also tell is if our documentation is out of date. Either way, running this rake task will give us a hint on how out of date documentation is.

The tests also accept node options, so we can test against pre-release/testnets and get ahead of any changes that might happen before they go to production.

Ops Dump

As mentioned in the @steemitblog post from yesterday, we have rake ops_dump. When we run it, we get:

account_create
account_create_with_delegation
account_update
account_witness_proxy
account_witness_vote
cancel_transfer_from_savings
challenge_authority
change_recovery_account
claim_account
claim_reward_balance
comment
comment_options
convert
create_claimed_account
custom
custom_binary
custom_json
decline_voting_rights
delegate_vesting_shares
delete_comment
escrow_approve
escrow_dispute
escrow_release
escrow_transfer
feed_publish
limit_order_cancel
limit_order_create
limit_order_create2
pow
pow2
price
prove_authority
recover_account
report_over_production
request_account_recovery
reset_account
set_reset_account
set_withdraw_vesting_route
transfer
transfer_from_savings
transfer_to_savings
transfer_to_vesting
vote
withdraw_vesting
witness_set_properties
witness_update

These are the broadcast ops that the devportal currently knows about, not including virtual ops. Broadcast ops are typically more important to app developers than virtual ops because they represent actions that applications can perform, whereas virtual ops are actions that the blockchain performs, typically delayed actions based on timestamps and other internal criteria.

Once we know about a new op, we can add it to the broadcast_ops.yml file which Jekyll uses to generate:

https://inertia186.github.io/devportal/apidefinitions/broadcast-ops

Test Proof

The rake test:proof task has Jekyll build the site, then run basically a lint check on the site. This allows us to find problems with the site. I also use Site Sucker to further validate the results.

Other Tasks

The other tasks, like rake production:deploy and rake production:rollback are just used to build the site for use on production environments or switch back to the previous version. We need to do this externally instead of relying on Github Pages because there are various non-standard libraries required to build the site and Github doesn't like running non-standard libraries.

We also have scrape tasks that will pull tutorials in from other repositories.

Resources

• To get a local copy of the devportal and try these tasks yourself: https://github.com/inertia186/devportal/blob/master/readme.md

Previous Posts of this Series

• More Current Steem Documentation (original fork announcement)
• Documenting Reputation and Database API Definitions
• Full Node Quickstart

Github

https://github.com/inertia186