SAFE Network Dev Update - July 4, 2019

Summary

Here are some of the main things to highlight this week:

• We’re beginning the design process for the Perpetual Web milestone with the simplest set of user stories—centred around the read-only mode of use. You can head on over to the GitHub repo, and follow along with the progress as the designs unfold.
• We're expecting the current milestone of the SAFE CLI, which brings the files put, files sync and cat commands, to be ready early next week.
• We've been working on the implementation of our own encoding for the XOR-URLs. We have just implemented our own encoding which incorporates not only the XOR name and content type in the XOR-URL string but also the type tag.
• The SAFE Client Libs team has begun adapting and implementing the FFI layer for the new data types starting from Mutable Data. Which means that the day for the non-Rust folks out there to interface with the new data types is arriving soon.

Marketing

This week, the marketing team have been exploring in more detail the 4 milestones @ JimCollinson shared last week. The first one we’ve dug into is Perpetual Web (here is the Medium link and forum one). This is really exciting and opens up a new world where public data lives on indefinitely. It cannot be deleted, removed, or tampered with. A network of truthfulness. Also, who loves the video from 2007? It's great, isn’t it?! But seriously, it just shows how long these ideas have been swimming around our heads and how we’re determined to follow through on our original ambitions in building the SAFE Network. We’ve got another three of these posts exploring the other milestones Take Control of Your Data, Private Communications and the New Digital Economy to come over the coming weeks so stay tuned...

@ SarahPentland has also published our monthly round-up, so for those who want a neat summary of all we’ve achieved in June, go check it out.

And good news for those who reside in the Glasgow area - we have another meetup 🎉. We’re just sorting out speakers but for those who attended the last time, it was absolutely fantastic meeting the community and we’re looking forward to seeing you all again, and hopefully even more of you! It's on the 22nd July but head over to meetup for all the details.

A quick update on exchanges in case you've missed it. We’ve taken the decision to remove all of our assets from the Bitker Exchange. Instead of clogging up the weekly dev update, you can read about our reasons on this forum post.

Finally, a piece of team news to wrap up with. We're happy to announce that with the recent move by @ dugcampbell to COO, @ Cgray has now been promoted to Head of Marketing. Congratulations Ceilidh! :tada:

User Experience

Following on from last week's UX roadmap announcements, we’re losing no time in getting stuck right into the making.

So, here we go, let’s start taking the wraps off the first piece of the Perpetual Web. Check out the GitHub repo, and follow along with the progress as the designs unfold, but for the non-git-hubbers amongst us, we've got all the details over on the main forum.

SAFE CLI

Project plan

The CLI has seen some interesting progress this week. The implementation of the files put and files sync command has been finalised and is now going through our internal testing/QA process. The cat command has also been implemented to support both published ImmutableData and FilesContainer XOR-URLs, i.e. files and folders can be uploaded with files put and files sync, obtaining the resulting XOR-URLs which can then be used to fetch the data with the cat command.

The cat command even supports providing a path with a XOR-URL, e.g. a file published in a FilesContainer with path/filename "/myfolder/myfile.txt" can be fetched using the XOR-URL of the FilesContainer with the file's path appended to it:

$ cargo run --features scl-mock -- cat safe://hbyw8kkqr3tcwfqiiqh4qeaehzr1e9boiuyfw5bqqx1adyh9sawdhboj5w/myfolder/myfile.txt
This is a test to create a text file with some content to work with files put and cat.

Another aspect that has been worked on this past week was the implementation of our own encoding for the XOR-URLs. Those who were involved in the XOR-URL RFC discussion should remember the proposal to use multiformats CID for encoding the XOR name, content type, etc. We have just implemented our own encoding which incorporates not only the XOR name and content type in the XOR-URL string but also the type tag. This means that it's not necessary to put the type tag as a port number in the URL as all the information is included in the XOR-URL string. This is implemented in an attempt to find out what the best format/encoding for the XOR-URL is while we move forward with the implementation of the CLI. The current encoding scheme is as follows (variable length from 35 to 42 bytes, base32z base encoding by default):

• 1 byte for version
• 1 byte for content type (ImmutableData, FilesContainer, Wallet, etc.)
• 32 bytes for the content's XOR name
• and up to 8 bytes for type tag, only if it's applicable

We resumed efforts this week, a bit more aggressively this time, to integrate the CLI with safe_client_libs. The plan is to continue with this for the next few days and it will be also one of the main goals of our next milestone, which will be outlined next week.

We were originally aiming to finish the current milestone by end of this week, but we decided to invest a few more days to make sure the main bugs are fixed, and some edge case scenarios are well-tested, before encouraging you to start using the new commands. We believe this should be finalised early next week, when we'll notify everyone so that you can enjoy the new commands available, and hopefully provide some feedback.

SAFE Browser

There is little to update you all on when it comes to the Browser this week. After our release last week, we were simply analysing the next steps for the Browser and the overall priorities across different fronts.

We plan to work on a new release that fixes some very high priority bugs only, while investing more of the front-end resources on some of the other projects we've prioritised currently (like the efforts related to the Perpetual Web deliverables). We'll be sharing more details about our plans for the next Browser release in the next few days though, as before, by simply sharing a link to the corresponding project board.

Mock interface

Project plan

SAFE Clients Libs has had its own share of action this week. To begin with, FFI are in the works! Which means that the day for the non-Rust folks out there to interface with the new data types is arriving soon.

@ marcin has begun adapting and implementing the FFI layer for the new data types starting from Mutable Data. Secondly, we have been focusing, reiterating and improving on how Wallet/Account creation and handling could be done in the SAFE Network. Multiple methods are being brainstormed to overcome any disadvantages and inefficiencies. RPCs for the same are being defined and are soon to be integrated with the mock interface so that you can try them out as soon as they are ready.

In addition to that, the next major milestone of Adapting SAFE Client Libs to the new data types is progressing well and we are halfway through. Some of the above tasks are also part of this milestone as well. This project requires the rest of the components in SCL such as safe_app and safe_authenticator to make use of the new data types and slowly replace the legacy ones. As this work proceeds, we'll simultaneously be working on refactoring the data types as per the needs and optimizations. We're also fixing pesky bugs and working on correcting the semantics of the data types as we stumble upon them during the milestone tasks. This lets us deliver an issue-free mock-interface once all the hustle ends.

Subtle changes have been made to the data types RFCs (Published/Unpublished Data & Immutable Data) to fix issues and semantic errors we came across during the development. These RFCs are still open for receiving your opinions as well. Please do check them out to see what the changes are, and do give us a nudge if you see something that doesn't fit well or something that might fit better :smile:

Vaults

Project plan

This week has seen plenty of progress in vaults. We now have four team members working in this area (@ adam, @ fraser, @ jon, and @ nbaksalyar) and to an extent in safe-nd, since it's tightly coupled to the vault crate (it defines the types which are common to vaults and SAFE Client Libs (SCL) - so mainly the RPC's and chunk types).

We've completed most of the work for the first couple of RPCs to support basic coin operations, Immutable Data and accounts data. With these in place, implementing the remaining RPC handlers should be somewhat quicker, as we have fewer unknowns and more concrete ideas about how to implement them.

We also now have a test framework in place which will let us build some more confidence in the code, although we're expecting to be able to run the SCL test suite against an instance of a vault very soon (hopefully within a week or two). Since that test suite is more mature and fairly extensive, it'll be a great way to check that the vault code is doing what is expected of it.

Integration of quic-p2p into Routing

Project plan

This week, we've been continuing the work on proper resending of messages on failures, which bore fruit today in the form of a PR. This is important for Reliable Message Delivery, as sometimes a node will try to pass a message to another node and fail (for example, because the other node disconnected in the meantime). If the node didn't retry, the message would get lost. Since we dropped the routes and acks mechanism when introducing Reliable Message Delivery, we had no way to resend attempts when sending failed - this now changes with this PR.

Reliable Message Delivery

Project plan

We decided that the last issue still opened in RMD fits the scope of Secure Message Delivery better than RMD. It concerned signature gathering so we moved it to the other project. This completes the RMD project! :tada:

BLS cryptography

The BLS RFC now moved to the implementation phase and we have a brand new Project plan:

Project plan

As we have been highlighting over the last few weeks, we started by working on the feature that is least likely to be altered, so the initial task is already closed. This week we completed the follow-up task: a user of PARSEC can now start a BLS Distributed Key Generation with all PARSEC voters. When complete, each peer can retrieve the group threshold public key, and that peer's secret key. This will be essential to keep secure message between Disjoint sections lightweight. With this scheme, a group can sign messages with a single signature.

Community contributions

Finally, we have been seeing some regular community PRs in quic-p2p which is great for us to see as it shows that the library is viewed as interesting and that people are spending time reading the code. PR #59, PR #56 and PR #57 are examples. Thanks go to @ d1vyank! Plus a big thanks go to @ mav who has once again carried out some fascinating research on chunk distribution within sections. Great work!