Posts

Lightning Keysend is strange and how to send Keysend Payment in Lightning with the LND REST API via Python

avatar of @brianoflondon
25
Dr Brian of London ✅
@brianoflondon
·
0 views
·
6 min read

Image by Ron van den Berg from Pixabay

It took me 3 specific tries over 6 months to figure this out. It could have been easy but Lightning's docs are not what they should be and I just couldn't find any solid code out on the net that actually does what I was trying to do.

Jump to the end if you want to actually see Python code.

This wasn't a core part of what I'm building, mostly I needed it to TEST what I've built. The Podcasting 2.0 Value 4 Value system is built on something called Keysend Payments within Lightning. These were a bolt on cludge that was added in the last year and a half.

They are vital because they are the only thing which allows UNSOLICITED funds to be sent via the Lightning network.

Normal Lightning

The normal way of operating in Lightning (and this is counter to all your regular experience with other crypto payment systems) is this:

  1. Receiver's Node Creates an Invoice (with a unique key) ->
  2. Receiver's Node sends or shows invoice (QR code) ->
  3. Payer sees and pays invoice ->
  4. Payer's node tries to find a path to send sats via channels to the Receiver's node ->
  5. Path is found and time locked agreements made with all the channels in the path to transfer on an encrypted payment (sometimes taking fees along the way) ->
  6. Payment is sent and the Receiver unwraps it to verify that keys and hashes match and sats (payment) has moved to the Receiver's node in the channel it came in on. I've seen payments I send take 5 hops to get to a destination.

And we're done. But that involves generating an invoice. In fact steps 1 & 2 are what happens when you click on a link to send someone Lightning and get shown a QR Code like this link

Keysend is different

The difference with how Keysend works for us in streaming sats payments is that steps 1 to 3 don't happen.

Instead, the Payer's node comes up with a secret on its own. It wraps that up in a special payload section and then tries to find a path to the receiver's node.

  1. Payer's node creates a secret token
  2. Payer's node tries to find a payment path to the Receiver's node address
  3. If that path is found it sends the payment with a the secret hidden inside and a SHA256 Hash of the secret visible to all the nodes passing the payment.
  4. The Receiver's node gets the payment and can verify the hash and the secret match.

The practical upshot of this is that Lightning the payment system does actually resemble lightning the physical phenomena. ]Lightning strikes do feature a path of ionized air being formed milliseconds before the main bolt of electricity moves from the ground up](https://www.youtube.com/watch?v=qQKhIK4pvYo).

And I can see this when I receive payments: this is the log of watching for new invoices to come in when I get a Keysend payment:

2022-03-24T07:49:05+0000 : New invoice received: Value: 24 | Timer: 265.371702 
2022-03-24T07:49:05+0000 : Invoice has no htlcs:           | Timer: 265.372052 
2022-03-24T07:49:05+0000 : New invoice received: Value: 24 | Timer: 265.373830

As you can see, the same invoices is registered twice, my code ignores the first one (which is missing 'htlcs') because only when the second notification occurs does the useful information come in. This bit tells me which podcast was being listened to and which Hive address I should pass the payment on to.

As you can see 0.002s passes between these two points.

Mostly this is just an artefact of the way the LND software (which I'm using for my Lightning node) operates, but I thought it was interesting.

Python

I first got some help in this discussion thread on Github and I've now submitted the following text to the official documentation. I'm not sure they'll like my chatty style.

Sending a Keysend Payment in Python with lnd's REST API endpoint

This document is born out of some personal frustration: I found it particularlly hard to find working examples of using the LND Rest API (specifically on an Umbrel) from Python.

I will present here some working code examples to send a Keysend payment from Python. At first you'd think this was trivial considering how easy it is with lncli:

lncli sendpayment -d 0266ad2656c7a19a219d37e82b280046660f4d7f3ae0c00b64a1629de4ea567668 -a 1948 --keysend --data 818818=627269616e6f666c6f6e646f6e --json

That will send 1948 sats to the public key 0266ad2656c7a19a219d37e82b280046660f4d7f3ae0c00b64a1629de4ea567668 and add a special key of 818818 which passes the Hive address of brianoflondon as Hex: 627269616e6f666c6f6e646f6e

To find this Hex value in Python:

>>> a = "brianoflondon" 
>>> a.encode().hex() 
'627269616e6f666c6f6e646f6e'

How to do that with Python:

Actually getting this working and figuring out all the correct combinations of base64 and Hex encoding had me tearing my hair out. When it worked after I think I tried almost every possible combingation of .encode() hex() and base64.b64encode(plain_str.encode()).decode() left me feeling like the head of a team of monkeys which had just finished the last page typing out The Complete Works of William Shakespear.

I'm going to assume you've successfully managed to get a connection up and running to your LND API. If you haven't perhaps that needs to be better explained somewhere in these docs. Perhaps I can be persuaded.

I've documented this code and whilst its a bit different from what I'm actually using (I'm working on streaming value 4 value payments in podcasting so I'm sending quite a bit more information encoded in the dest_custom_records field but the method is exactly the same as I've shown here for the 818818 field). You can learn more about these Podcasting specific fields here.

 
import base64 
import codecs 
import json 
import os 
from hashlib import sha256 
from secrets import token_hex 
from typing import Tuple 
 
import httpx 
 
 
 
def get_lnd_headers_cert( 
    admin: bool = False, local: bool = False, node: str = None 
) -> Tuple[dict, str]: 
    """Return the headers and certificate for connecting, if macaroon passed as string 
    does not return a certificate (Voltage)""" 
    if not node: 
        node = Config.LOCAL_LND_NODE_ADDRESS 
 
    # maintain option to work with local macaroon and umbrel 
    macaroon_folder = ".macaroon" 
    if not admin: 
        macaroon_file = "invoices.macaroon" 
    else: 
        macaroon_file = "admin.macaroon" 
    macaroon_path = os.path.join(macaroon_folder, macaroon_file) 
    cert = os.path.join(macaroon_folder, "tls.cert") 
    macaroon = codecs.encode(open(macaroon_path, "rb").read(), "hex") 
    headers = {"Grpc-Metadata-macaroon": macaroon} 
    return headers, cert 
 
 
def b64_hex_transform(plain_str: str) -> str: 
    """Returns the b64 transformed version of a hex string""" 
    a_string = bytes.fromhex(plain_str) 
    return base64.b64encode(a_string).decode() 
 
 
def b64_transform(plain_str: str) -> str: 
    """Returns the b64 transformed version of a string""" 
    return base64.b64encode(plain_str.encode()).decode() 
 
 
def send_keysend( 
    amt: int, 
    dest_pubkey: str = "", 
    hive_accname: str = "brianoflondon", 
) -> dict: 
    """Pay a keysend invoice using the chosen node""" 
    node = "https://umbrel.local:8080/" 
    headers, cert = get_lnd_headers_cert(admin=True, node=node) 
    if not dest_pubkey: 
        dest_pubkey = my_voltage_public_key 
 
    # Base 64 encoded destination bytes 
    dest = b64_hex_transform(dest_pubkey) 
 
    # We generate a random 32 byte Hex pre_image here. 
    pre_image = token_hex(32) 
    # This is the hash of the pre-image 
    payment_hash = sha256(bytes.fromhex(pre_image)) 
 
    # The record 5482373484 is special: it carries the pre_image 
    # to the destination so it can be compared with the hash we 
    # pass via the payment_hash 
    dest_custom_records = { 
        5482373484: b64_hex_transform(pre_image), 
        818818: b64_transform(hive_accname), 
    } 
 
    url = f"{node}v1/channels/transactions" 
    data = { 
        "dest": dest, 
        "amt": amt, 
        "payment_hash": b64_hex_transform(payment_hash.hexdigest()), 
        "dest_custom_records": dest_custom_records, 
    } 
 
    response = httpx.post( 
        url=url, headers=headers, data=json.dumps(data), verify=cert 
        ) 
 
    print(json.dumps(response.json(), indent=2)) 
    return json.dumps(response.json(), indent=2)

This explanation of what is going on here is pretty useful too:

Since you're not paying an invoice, the receiver doesn't know the preimage for the given payment's hash. You need to add the preimage to your custom records. The number is 5482373484.

It also looks like you need to actually set the payment's hash (payment_hash=sha256(preimage)) of the payment.

See https://github.com/lightningnetwork/lnd/blob/master/cmd/lncli/cmd_payments.go#l358 on how it's done on the RPC level (what's behind the --keysend flag of lndcli sendpayment).

If you still have question, find me @brianoflondon pretty much anywhere online and I'll help if I can.

Support Proposal 201 on PeakD Support Proposal 201 with Hivesigner Support Proposal 201 on Ecency