dsteem Modernization Plan -Phase 4

blaze.apps (71)verified developerin Steem Dev • yesterday (edited)

Day 4: this is the highest-risk single change in the whole modernization. I ripped out [email protected] — the only package in our entire dependency tree with a known high-severity CVE — and replaced it with @noble/curves + @noble/hashes, the same pure-JS crypto stack that ethers, viem, and most of the modern Ethereum ecosystem trust. Public API stays 100% backward compatible. But here's the twist: the new signatures are byte-different from the old ones — and that's actually correct. The golden-vector test from Phase 0 was specifically designed to catch this nuance.

Day 1 — Phase 0: dsteem Modernization Plan
Day 2 — Phase 1: CI Plumbing on GitHub Actions
Day 3 — Phase 2: Retiring tslint, Adopting ESLint 9
Day 4 — Phase 3: TypeScript 3.1 → 5.6 + Strict Mode
Today — Phase 4: The @noble Crypto Swap (highest risk — you are here)
Tomorrow — Phase 5: Build modernization with tsup + dual ESM/CJS
Day 7 — Phase 6: Mocha 11, c8 coverage, Playwright browser tests
Day 8 — Phase 7: typedoc 0.13 → 0.28
Next week — Phase 8: Cleanup, README refresh, and the v0.12.0 release

The Vulnerability We're Killing

[email protected] (the version v0.11.3 pinned) has a documented high-severity advisory on the GitHub Advisory Database. Beyond the CVE itself, the v3.x line is also:

A native C++ module — requires node-gyp, Python, a C++ compiler chain, and the right Node headers to build from source
A prebuild attack surface — when prebuilt binaries don't match your Node version, you fall back to source compilation; when they do, you're running someone else's compiled .node file with no easy way to verify it
Unmaintained on the v3 line — fixes only go to v4+ which has a completely different API

Replacing it isn't just about the CVE. It's about deleting an entire class of supply-chain and build-system risk.

Why `@noble/curves`?

I went with @noble/curves (which bundles secp256k1 + several other curves) and @noble/hashes for SHA256 / RIPEMD160. The reasons:

Property	What it means here
Pure JavaScript	No native build, no `node-gyp`, no prebuilt binaries
Audited	Multiple paid audits including Cure53 and Trail of Bits
Battle-tested	Used by `ethers`, `viem`, `web3.js`, and dozens of L2 SDKs
Tiny	secp256k1 implementation is ~30KB minified
Browser-native	Works in Workers, Deno, Bun, edge runtimes — no Node-specific assumptions
Maintainer	Paul Miller (`@paulmillr`) — one of the most respected JS crypto authors

@noble/curves@^1 was the right pin. v2 (released earlier this year) tightened its export map to require explicit .js subpath extensions; older bundlers and ts-node don't handle that cleanly yet. v1.7 is the sweet spot.

The API Mapping

The legacy secp256k1 API (v3.x):

secp256k1.publicKeyVerify(key: Buffer): boolean
secp256k1.privateKeyVerify(key: Buffer): boolean
secp256k1.publicKeyCreate(priv: Buffer): Buffer
secp256k1.sign(msg, priv, {data: extraEntropy}): {signature: Buffer, recovery: number}
secp256k1.verify(msg, sig, pub): boolean
secp256k1.recover(msg, sig, recovery): Buffer

The new @noble/curves API:

import {secp256k1 as nobleSecp} from '@noble/curves/secp256k1'

nobleSecp.utils.isValidPrivateKey(key)
nobleSecp.getPublicKey(priv, true)                      // returns Uint8Array (compressed)
nobleSecp.sign(msg, priv, {extraEntropy, lowS: false})  // returns Signature object
nobleSecp.verify(sig, msg, pub, {lowS: false})          // NOTE: arg order swap!
nobleSecp.Signature.fromCompact(sig).addRecoveryBit(rec).recoverPublicKey(msg)

I wrapped each one in an internal helper so the public PublicKey, PrivateKey, Signature class signatures stay byte-identical:

function publicKeyVerify(key: Buffer): boolean {
    try { nobleSecp.ProjectivePoint.fromHex(key); return true } catch { return false }
}
function privateKeyVerify(key: Buffer): boolean {
    return key.length === 32 && nobleSecp.utils.isValidPrivateKey(key)
}
function publicKeyCreate(priv: Buffer): Buffer {
    return Buffer.from(nobleSecp.getPublicKey(priv, true))
}
function ecSign(message: Buffer, priv: Buffer, extraEntropy: Buffer) {
    const sig = nobleSecp.sign(message, priv, {extraEntropy, lowS: false})
    return {signature: Buffer.from(sig.toCompactRawBytes()), recovery: sig.recovery!}
}
function ecVerify(message: Buffer, sigBuf: Buffer, pub: Buffer): boolean {
    return nobleSecp.verify(sigBuf, message, pub, {lowS: false})
}
function ecRecover(message: Buffer, sigBuf: Buffer, recovery: number): Buffer {
    const sig = nobleSecp.Signature.fromCompact(sigBuf).addRecoveryBit(recovery)
    return Buffer.from(sig.recoverPublicKey(message).toRawBytes(true))
}

And for hashes:

import {sha256 as nobleSha256} from '@noble/hashes/sha256'
import {ripemd160 as nobleRipemd160} from '@noble/hashes/ripemd160'

function sha256(input: Buffer | string): Buffer {
    const data = typeof input === 'string' ? Buffer.from(input) : input
    return Buffer.from(nobleSha256(data))
}
function ripemd160(input: Buffer | string): Buffer {
    const data = typeof input === 'string' ? Buffer.from(input) : input
    return Buffer.from(nobleRipemd160(data))
}

The Buffer.from(...) wrapping is because @noble/hashes returns Uint8Array — and while Buffer is a Uint8Array subclass in Node, the rest of the codebase calls Buffer-specific methods (.equals(), .slice() semantics, .readUInt8(), etc.) on these return values. Cheaper to wrap once at the boundary than to refactor 600 lines of Buffer usage.

One Critical Detail: `lowS: false`

This took me a while to get right. @noble defaults to "low-S normalization" — adjusting signatures to a canonical form that prevents transaction-malleability attacks on Bitcoin/Ethereum. Steem doesn't use that definition of canonical.

Steem's isCanonicalSignature is a byte-pattern check on the first byte of r and s:

function isCanonicalSignature(signature: Buffer): boolean {
    return (
        !(signature[0] & 0x80) &&
        !(signature[0] === 0 && !(signature[1] & 0x80)) &&
        !(signature[32] & 0x80) &&
        !(signature[32] === 0 && !(signature[33] & 0x80))
    )
}

The legacy code wraps secp256k1.sign in a retry loop, varying extraEntropy until the resulting signature happens to pass that check. If I'd let @noble do lowS: true (its default), it would force-normalize the signature — and the steemd canonical check would then fail in a different way, sending the retry loop into infinite retries.

So I pass lowS: false explicitly and keep the existing retry loop intact:

public sign(message: Buffer): Signature {
    let rv: {signature: Buffer, recovery: number}
    let attempts = 0
    do {
        const extraEntropy = sha256(Buffer.concat([message, Buffer.alloc(1, ++attempts)]))
        rv = ecSign(message, this.key, extraEntropy)
    } while (!isCanonicalSignature(rv.signature))
    return new Signature(rv.signature, rv.recovery)
}

This is the kind of detail that would never have been caught by a unit test that uses randomBytes() to generate messages. Both old and new code happily round-trip random data. The bug would have shown up months later as "some users can't broadcast transactions." Which brings us to…

The Golden Vector Test Catches a Surprise

Phase 0 froze 25 deterministic test vectors (hash, sign, transaction-digest) from the current [email protected] behavior. The Phase 4 acceptance gate is: re-run the same generator against @noble-backed code, get bit-identical output.

I ran it. Signatures failed.

- signatureHex: '1f52c3ba9f868dfcb8bf10e33312882864e477ff1c0980ed6f37aa2f8c7126b1ec...'  (secp256k1@3)
+ signatureHex: '2032ec4b991238c9d50028ab080866e322e30d2ad79227d44799baafa8c8d36f82...'  (@noble)

Different bytes. Different recovery bit. Different r, different s. Both for the same private key + message + extra entropy.

My first reaction: "Phase 4 is broken." My second reaction (after re-reading the @noble docs): "Wait."

ECDSA with RFC 6979 + extra entropy is deterministic for a given implementation — but different implementations are free to mix the extra entropy into the nonce derivation in different ways. Both produce valid signatures. Neither is wrong. They're just different valid signatures for the same message.

So I tested:

const oldSig = '1f52c3ba9f868dfcb8bf10e33312882864e477ff1c0980ed6f37aa2f8c7126b1ec...'
const newSig = '2032ec4b991238c9d50028ab080866e322e30d2ad79227d44799baafa8c8d36f82...'

// Recover the pubkey from each:
const oldRecPub = secp256k1.Signature.fromCompact(oldSig).addRecoveryBit(0).recoverPublicKey(msg).toRawBytes(true)
const newRecPub = secp256k1.Signature.fromCompact(newSig).addRecoveryBit(1).recoverPublicKey(msg).toRawBytes(true)

console.log(Buffer.from(oldRecPub).toString('hex'))  // 02f245ddbf442af5486e5b902e2b7037b2ad1c47d9a825fab4ddac994bb7042078
console.log(Buffer.from(newRecPub).toString('hex'))  // 02f245ddbf442af5486e5b902e2b7037b2ad1c47d9a825fab4ddac994bb7042078

Same public key. Both signatures are valid for the same key+message pair. Steem nodes will accept either one. The golden test's byte-equality assertion was too strict.

The Golden Test, Rewritten

The right contract isn't "same bytes." It's:

Deterministic things must be byte-identical — hashes (sha256, ripemd160, doubleSha256) and transaction digests (sha256 of serialized tx + chainId)
Signatures must be valid — fresh signatures must verify(), recover() to the signer's pubkey, and pass isCanonicalSignature
Historical signatures must still work — signatures produced by [email protected] (already on-chain in real Steem transactions) must still verify with the new @noble backend

That last one is the most important property. It proves we can still read existing on-chain history with the new code. Here's the rewritten test:

it('historical signatures (frozen v0.11.3 bytes) still verify + recover with new backend', () => {
    for (let i = 0; i < golden.signatures.length; i++) {
        const g = golden.signatures[i]
        const pub = PublicKey.fromString(g.pubKeyStr)
        const sig = Signature.fromString(g.signatureHex)
        const msg = Buffer.from(g.messageHex, 'hex')
        assert(pub.verify(msg, sig), `historical vec ${i}: must verify with new backend`)
        assert.strictEqual(sig.recover(msg).toString(), g.recoveredPubKeyStr,
            `historical vec ${i}: must recover correctly`)
    }
})

This is the load-bearing assertion in the whole modernization. If it passes, every signature that has ever been broadcast by a dsteem user is still verifiable by the new code. If it failed, dsteem v0.12.0 would be unable to read pre-existing on-chain history — a catastrophic backward-compatibility break that no test using randomBytes() would have caught.

The Phase 4 Test Result

$ npm test -- --grep golden

  crypto golden vectors
    √ matches version
    √ reproduces all hash vectors (byte-exact)
    √ signature vectors: fresh sigs are canonical + verify + recover correctly (39ms)
    √ historical signatures (frozen v0.11.3 bytes) still verify + recover with new backend (74ms)
    √ transaction-digest vectors: digests are byte-exact, signatures verify (75ms)

  5 passing (337ms)

$ npm test
  49 passing (440ms)

$ ls node_modules/secp256k1
ls: cannot access 'node_modules/secp256k1': No such file or directory

$ ls node_modules/@noble
curves  hashes

✅ All 5 golden vectors pass — hash + digest byte-exact, signatures verify, historical signatures still work
✅ Full constrained suite: 49 passing (added one new test, the "historical sig verifies" cross-check)
✅ Zero native modules: node_modules/secp256k1 is gone. No more node-gyp. No more prebuilds.
✅ Verified the same on Node 22 in Docker: 49 passing

Dependency Diff

- "secp256k1":          "^3.5.2"   ← HIGH-SEVERITY CVE, native build
- "@types/secp256k1":   "^4.0.7"   ← mismatched types anyway
+ "@noble/curves":      "^1.9.7"
+ "@noble/hashes":      "^1.8.0"

Two pure-JS audited libraries replacing one CVE-affected native module. Production dependency tree is now:

@noble/curves    ← secp256k1 ECDSA
@noble/hashes    ← SHA256, RIPEMD160
bs58             ← Base58 encoding for WIF
bytebuffer       ← binary serialization (latest stable, used by serializer)
verror           ← error wrapping

Five deps. All audited or stable. All maintainable on any Node version we care about going forward.

The Meta-Lesson

The most important property of a crypto migration isn't "do the new signatures match the old ones byte for byte." It's:

Will the network still accept my new signatures? (yes — canonical + valid)
Can my new code still read signatures the old code produced? (yes — historical cross-validation)
Are the cryptographic primitives (hashing, serialization) still byte-deterministic? (yes — these aren't ECDSA, they're pure functions)

A naive golden-vector test would have failed on #1's symptom (different bytes) and made me think the migration was broken. The realization that "different bytes, same semantics" is the correct state for ECDSA migrations took an evening of cross-validation work. The Phase 0 fixture file is now annotated with this reasoning, and the rewritten test asserts the right contract.

This is the whole reason Phase 0 existed. Without those frozen vectors, I'd be flying blind on whether the new code produces output that Steem nodes can ingest.

What's Next: Phase 5

Tomorrow is the biggest scope-by-line-count change in the migration: rip out the entire 2018 build toolchain (browserify + tsify + babelify + uglifyjs + dts-generator + watchify + derequire — the whole stack), replace it with tsup (esbuild under the hood), wire up a package.json exports map for proper dual ESM + CJS distribution, gut all the dead browser polyfills (core-js@2, regenerator-runtime, whatwg-fetch), and delete the Makefile.

Spoiler: 606 npm packages disappear from node_modules, the browser bundle drops from ~782KB to ~351KB, and the build goes from a Makefile + Unix sed requirements to plain npm run build on any platform including Windows.

How You Can Help

🔍 Watch the repo: blazeapps007/dsteem
🐛 Spot regressions: API compatibility with v0.11.x is non-negotiable
💬 Comment: questions about specific phase choices help shape the rest of the series

This work was developed with Claude AI assistance. All technical decisions reflect Steem ecosystem needs and the hard requirement of zero breaking .

Support Secure Steem Development

If you value proactive engineering, UX polish, and performance optimizations for the STEEM ecosystem, please consider supporting my witness: blaze.apps

🗳️ Vote Here:
Vote for blaze.apps Witness

Disclaimer: This post describes work in progress on the blazeapps007/dsteem fork. The release will be tagged v0.12.0 after Phase 8 ships next week.

#steem #steemdev #dsteem #opensource

yesterday in Steem Dev by blaze.apps (71)verified developer

$20.80

Sort:

Trending