How I set up my first open source Market Making bot on Raspberry Pi

LeoFinance
20 days ago
(edited)
16 Min Read
3267 Words

It's time I decided to test the waters in the market making area. Though I focus more on trading alone, I thought of giving it a try.

This article is all about Hummingbot; a great open source project for HFT market making and Arbitrage.

Disclaimer: I am not affiliated or sponsored by Hummingbot to write this article.
I (or hummingbot) is in no way responsible for any loss which may occur to you because of the wild market, and in case something terribly goes wrong, which is well beyond anyone's control (like markets dumping more than 40-50% in days and stuff like that)
I'd urge you to do some research about market making, understand how market making bots work, and then understand some advanced concepts to actually profit our of this market. If you are interested, please do some research before starting off with any market making bot.

image.png
(Image source: Command line interface of Hummingbot)


A few points to consider:

For consistent market making, make sure to go with this setup only if you have a high speed uninterrupted internet connection and uninterrupted power supply. I have no such issues, hence I proceeded with this setup.

  1. Windows installation is easy/hassle free
  2. Linux installation can be done via Docker as well
  3. You can Deploy the bot on cloud to run 24x7 (I have not tried that yet)
  4. Mac installation is also pretty easy and straight forward.

Note: https://hummingbot.io/download/

In my case, I'll be using my laptop which is having Windows installed in it, to set up the Pi.

A word or two about Raspberry pi: It is a cheap microcomputing device, having all the necessary hardware required to run a full fledged desktop computer; which costs a fraction of an actual computer (You can get is for less than $100. I got mine for around $75 using a deal from an online eCommerce site along with all the accessories.)

Check more about it the device here: https://www.raspberrypi.org/

My model is the latest Raspberry pi 4, but with a 4 gb RAM edition.
8 gb was out of stock for sometime in my region, so I settled for the 4 gb edition, but it works absolutely fine and you won't notice any difference unless you are running too may instances.

Before you proceed, please have a look at the original blogs by Hummingbot:
  1. https://hummingbot.io/blog/2020-07-deploying-hummingbot-on-a-raspberry-pi/
  2. https://docs.hummingbot.io/installation/source/raspberry-pi/

We will be following that only, but with a workaround of the issues which came up in my case.

In case you have any issues, feel free to check out their discord page: https://discord.hummingbot.io/
Their team is helpful and one person by the name Sigit took time to look into my issues to try and help me out. If you are reading this, thanks for your time :)


Setting the Raspberry Pi up via SSH:

Note: If you have access to an external Monitor, Mouse and Keyboard, you can skip connecting your Pi via the SSH. In that case, simply load the OS using Pi Imager on SD card, connect the HDMI with monitor, connect keyboard and mouse with the Pi, and you are good to go!

I'm not having any external monitor since I'm out of my hometown, so connecting it via SSH to be used with my laptop was my only option.

Software Pre-requisites:

  • (In my case I had windows on my laptop, but feel free to download the relevant packages if you are using Mac or Linux)
  1. Putty (for connecting to the SSH) (https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)
  2. Real VNC viewer on your laptop. (https://www.realvnc.com/en/connect/download/viewer/windows/)
  3. VNC server on your Raspberry pi (coming to that later)
  4. Pi Imager on your laptop (https://www.raspberrypi.org/documentation/installation/installing-images/)

Hardware Pre Requisites:

  1. Raspberry pi (of course you need that LOL)
  2. Memory card (high speed will be preferred)
  3. USB type C charger ( 3 Amperes recommended)
  4. Micro SD card reader (USB/SD card reader)
  5. Optional (recommended) : Get a box with a fan in case you want to run it 24x7, otherwise the Pi may get damaged because of heat.
  6. Optional: LAN cable (because I'll directly connect it with Wifi. I was not able to find the IP address of the Pi using LAN, hence I connected it with WiFi)

Step 1:

Download the 64 bit Raspbian OS (32 bit will have a lot of issues with installation, so make sure you download the 64 bit OS only. Currently the 64 bit version is in beta.)
Link: https://www.raspberrypi.org/forums/viewtopic.php?t=275370

Step 2:

Open the Raspberry pi imager on your laptop, and select the "Use Custom.img" and load up the img file you downloaded.
In my case it was: 2020-05-27-raspios-buster-arm64.img

image.png

Plug in the SD card and proceed with the "Write option"
image.png

Skip from step 3-8 if you are using external monitor with keyboard and mouse to control your pi

Step 3:

create a blank file (For windows user: open cmd, type in: echo>E:\ssh -> It will create a new blank file without any extension, with the name: ssh)

If your memory card is in some other drive, use that corresponding drive letter in place of E:

Or simply create a blank file without any extension, having the name: "ssh"; and save it in the Boot section of the memory card.

This will enable Raspberry Pi to open up SSH connections after boot.

Step 4:

If you want to connect SSH over WiFi (recommended)
(I connected via WiFi because I was unable to find the IP via LAN cable because of some unknown technical reasons)

Open Notepad++, create a config file <wpa_supplicant.conf> and save this in the Boot directory of the SD card:
(if it's E:/ , just save it under E:/)


ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=«your_ISO-3166-1_two-letter_country_code»

network={
ssid="Enter your Wifi name"
psk="Enter your password"
key_mgmt=WPA-PSK
}

Make sure you name it wpa_supplicant.conf (and don't save it as a text file else it won't work)


Step 5:

Remove the card and plug it in the Pi and power it on.

Raspberry pi will connect to wifi automatically if the above steps are correct
Now make sure the system from where you will connect is connected to the same wifi.

Step 6:

Open your router login page: (usually: http://192.168.1.1/), and I am assuming you are not using mobile hotspot, or any public Wifi in this case.
Note: If you are using mobile hotspot, try using an external third party IP Scanner tool and it should work fine and show you the Pi IP address.

Find your raspberry ip address.

In my case it was something like this:
image.png

Step 7:

Open Putty, type in: [email protected](raspberry pi ip address) in the host name and keep port as 22.
example:
image.png

login as "pi"
use default password: "raspberry"

At this time, the Pi should get connected to the SSH and you should see something like this:
Ignore the "inactive" status as it is coming after I connect my Pi to the VNC. May be some bug somewhere.

image.png

Step 8:

Note: 64 bit OS is still in beta stage, so the VNC server will probably display a blank screen when you connect. This is still an unknown bug, but there's a workaround for that

Follow the below steps:

sudo apt update
sudo apt install tightvncserver

This is required because 64 bit OS on Rpi is still on beta
and they don't natively support RealVNC server yet. (I was getting blank screen error, and dug up the possible fixes when I came to know about tightvnc)

tightvncserver -geometry 1280x720 (you can change your config if you want, but it will create new servers on a different port)
In my case I' using tightvncserver -geometry 1920x1080 instead.

(Set password, no need to set view only password.)

Now open Real VNC,

connect with: :1

image.png

And if it connects successfully, you can see the Raspberry Pi Desktop successfully.


Setting up Hummingbot, Part 2:

Step 1:

Open Terminal on your Pi, and check the following details:

check what version of python you are running by default:

python --version

Usually by default it will show as 2.7.

Set python 3 as default:

sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.7 2

image.png


Step 2:

sudo apt update
sudo apt upgrade

(and you can do this later on as well)

Step 3:

Download Miniforge

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh

Install Miniforge:

sh Miniforge3-Linux-aarch64.sh

Now if you already have python installed, skip installing python again.

Step 4:

Set Conda environment as Default:

sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1

Close and open the terminal again.

You should see something like this:

image.png

Step 5:

By Default, conda will install the latest version of python (3.8 currently). But don't do that. We need python 3.7 only, otherwise a lot of issues will come.

So create an environment with python 3.7 as the base:

conda create --name hummingbot python=3.7

then activate your environment:

conda activate hummingbot

check the version of python in your conda environment after activating the hummingbot environment"
It should be 3.7

Step 6:

Installing the dependencies. This is where you may face some errors.

Before proceeding with the below step:

pip install pandas cython cachetools aiohttp ruamel.yaml eth_account aiokafka sqlalchemy binance python-binance ujson
websockets signalr-client-aio web3 prompt_toolkit 0x-order-utils
0x-contract-wrappers eth_bloom pyperclip telegram python-telegram-bot pyjwt numpy mypy_extensions bitstring pyblake2 pysha3

While following the above command to install the dependencies I had a lot of issues; mostly because of Python versions mismatch.

1)Pip install pandas was taking forever, and I couldn't find any workaround, so I did a conda install pandas instead. (usually I have seen if you install anaconda, or a conda environment, using pip breaks come operations, so it is not recommended ideally. But since some packages are not available, we have to use the pip install.
you can also try with pip3 install if you get errors.
I personally did a few trial and errors before making everything work successfully :)
2)I saw eth_account was giving a lot of errors and would never install.
I tried reinstalling python, and even tried reinstalling the entire OS of the Raspberry pi, but then I figured out
that it was because of version mismatch.

For eth_account -> I ran into additional issues, it was not getting installed, because package was not found, rusty_rpl etc etc.

issues: https://nomodulenamed.com/m/eth_account.datastructures

So My recommendation:
Follow this:

conda install pandas
conda install numpy
pip install cython cachetools aiohttp ruamel.yaml

pip install eth_account gives the following error in my case, and I was almost losing hope when I found a workaround for the installation.
image.png

pip install "eth_account==0.4.0"

This solved the problem. Installing specific versions is very cumbersome in python.
I have got same issues on tensorflow, scipy, and many other modules. It bugs a lot.

pip install aiokafka sqlalchemy binance python-binance ujson
websockets signalr-client-aio web3 prompt_toolkit 0x-order-utils
0x-contract-wrappers eth_bloom pyperclip telegram python-telegram-bot pyjwt numpy mypy_extensions bitstring pyblake2 pysha3

At this stage everything should work fine.

Step 7:

Install, compile, and Run Hummingbot

Clone the Hummingbot repo from Github

git clone https://github.com/CoinAlpha/hummingbot.git

Compile and clean your Hummingbot directory

cd hummingbot && ./clean && ./compile

Run Hummingbot

bin/hummingbot.py

At this point Hummingbot should work absolutely fine without any issues.

If you want to open Hummingbot again after you restart, simply open terminal, and it should have the (base) environment activate.

Do the following:

conda activate hummingbot

Then navigate to the directory where the hummingbot.py is situated (should be in a bin folder)

Then:

bin/hummingbot.py

It should start up.

To learn about market making, check out the hummingbot docs, and also their YouTube channel:

https://www.youtube.com/c/HummingbotChannel/featured


In case you are still getting any issues, check the below steps.

Make sure the following details match with your system:

Type in:

lsb_release -a

and you should see something like this:

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Oct 6 12:17:11 2020
(base) [email protected]:~ $ python --version
Python 3.7.8
(base) [email protected]:~ $ lsb_release -a
No LSB modules are available.

Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster


Go ahead and type in:

conda -V

In my case it's Conda 4.8.5

pip -V

In my case, it's 20.2.3, from python 3.7

Now go ahead and type in:

conda list

you can also type in pip list and check.

Conda list should give the following output: (Check only if you are getting errors, and see the versions, otherwise ignore it :P

----- Start of the List:

packages in environment at /home/pi/miniforge3/envs/hummingbot:

Name Version Build Channel
0x-contract-addresses 3.0.0 pypi_0 pypi
0x-contract-artifacts 3.0.0 pypi_0 pypi
0x-contract-wrappers 2.0.0 pypi_0 pypi
0x-json-schemas 2.1.0 pypi_0 pypi
0x-order-utils 4.0.0 pypi_0 pypi
_openmp_mutex 4.5 1_gnu conda-forge
aiohttp 3.6.2 pypi_0 pypi
aiokafka 0.6.0 pypi_0 pypi
async-timeout 3.0.1 pypi_0 pypi
attrdict 2.0.1 pypi_0 pypi
attrs 20.2.0 pypi_0 pypi
autobahn 20.7.1 pypi_0 pypi
automat 20.2.0 pypi_0 pypi
base58 2.0.1 pypi_0 pypi
binance 0.3 pypi_0 pypi
bitarray 1.2.2 pypi_0 pypi
bitstring 3.1.7 pypi_0 pypi
ca-certificates 2020.6.20 hecda079_0 conda-forge
cachetools 4.1.1 py_0 conda-forge
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
cffi 1.14.3 pypi_0 pypi
chardet 3.0.4 pypi_0 pypi
constantly 15.1.0 pypi_0 pypi
cryptography 3.1.1 pypi_0 pypi
cython 0.29.21 py37h1c9c24e_0 conda-forge
cytoolz 0.11.0 pypi_0 pypi
dateparser 0.7.6 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
deprecated 1.2.10 pypi_0 pypi
eth-abi 2.1.1 pypi_0 pypi
eth-account 0.5.3 pypi_0 pypi
eth-bloom 1.0.3 pypi_0 pypi
eth-hash 0.2.0 pypi_0 pypi
eth-keyfile 0.5.1 pypi_0 pypi
eth-keys 0.2.4 pypi_0 pypi
eth-rlp 0.2.0 pypi_0 pypi
eth-typing 2.2.2 pypi_0 pypi
eth-utils 1.9.5 pypi_0 pypi
hexbytes 0.2.1 pypi_0 pypi
hyperlink 20.0.1 pypi_0 pypi
idna 2.10 pypi_0 pypi
importlib-metadata 2.0.0 pypi_0 pypi
incremental 17.5.0 pypi_0 pypi
ipfshttpclient 0.6.1 pypi_0 pypi
jsonschema 3.2.0 pypi_0 pypi
kafka-python 2.0.2 pypi_0 pypi
ld_impl_linux-aarch64 2.35 h71b2515_9 conda-forge
libblas 3.8.0 17_openblas conda-forge
libcblas 3.8.0 17_openblas conda-forge
libffi 3.2.1 h4c5d2ac_1007 conda-forge
libgcc-ng 9.3.0 h8ab15db_17 conda-forge
libgfortran-ng 7.5.0 h2b9752b_17 conda-forge

packages in environment at /home/pi/miniforge3/envs/hummingbot:

Name Version Build Channel
0x-contract-addresses 3.0.0 pypi_0 pypi
0x-contract-artifacts 3.0.0 pypi_0 pypi
0x-contract-wrappers 2.0.0 pypi_0 pypi
0x-json-schemas 2.1.0 pypi_0 pypi
0x-order-utils 4.0.0 pypi_0 pypi
_openmp_mutex 4.5 1_gnu conda-forge
aiohttp 3.6.2 pypi_0 pypi
aiokafka 0.6.0 pypi_0 pypi
async-timeout 3.0.1 pypi_0 pypi
attrdict 2.0.1 pypi_0 pypi
attrs 20.2.0 pypi_0 pypi
autobahn 20.7.1 pypi_0 pypi
automat 20.2.0 pypi_0 pypi
base58 2.0.1 pypi_0 pypi
binance 0.3 pypi_0 pypi
bitarray 1.2.2 pypi_0 pypi
bitstring 3.1.7 pypi_0 pypi
ca-certificates 2020.6.20 hecda079_0 conda-forge
cachetools 4.1.1 py_0 conda-forge
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
cffi 1.14.3 pypi_0 pypi
chardet 3.0.4 pypi_0 pypi
constantly 15.1.0 pypi_0 pypi
cryptography 3.1.1 pypi_0 pypi
cython 0.29.21 py37h1c9c24e_0 conda-forge
cytoolz 0.11.0 pypi_0 pypi
dateparser 0.7.6 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
deprecated 1.2.10 pypi_0 pypi
eth-abi 2.1.1 pypi_0 pypi
eth-account 0.5.3 pypi_0 pypi
eth-bloom 1.0.3 pypi_0 pypi
eth-hash 0.2.0 pypi_0 pypi
eth-keyfile 0.5.1 pypi_0 pypi
eth-keys 0.2.4 pypi_0 pypi
eth-rlp 0.2.0 pypi_0 pypi
eth-typing 2.2.2 pypi_0 pypi
eth-utils 1.9.5 pypi_0 pypi
hexbytes 0.2.1 pypi_0 pypi
hyperlink 20.0.1 pypi_0 pypi
idna 2.10 pypi_0 pypi
importlib-metadata 2.0.0 pypi_0 pypi
incremental 17.5.0 pypi_0 pypi
ipfshttpclient 0.6.1 pypi_0 pypi
jsonschema 3.2.0 pypi_0 pypi
kafka-python 2.0.2 pypi_0 pypi
ld_impl_linux-aarch64 2.35 h71b2515_9 conda-forge
libblas 3.8.0 17_openblas conda-forge
libcblas 3.8.0 17_openblas conda-forge
libffi 3.2.1 h4c5d2ac_1007 conda-forge
libgcc-ng 9.3.0 h8ab15db_17 conda-forge
libgfortran-ng 7.5.0 h2b9752b_17 conda-forge

packages in environment at /home/pi/miniforge3/envs/hummingbot:

Name Version Build Channel
0x-contract-addresses 3.0.0 pypi_0 pypi
0x-contract-artifacts 3.0.0 pypi_0 pypi
0x-contract-wrappers 2.0.0 pypi_0 pypi
0x-json-schemas 2.1.0 pypi_0 pypi
0x-order-utils 4.0.0 pypi_0 pypi
_openmp_mutex 4.5 1_gnu conda-forge
aiohttp 3.6.2 pypi_0 pypi
aiokafka 0.6.0 pypi_0 pypi
async-timeout 3.0.1 pypi_0 pypi
attrdict 2.0.1 pypi_0 pypi
attrs 20.2.0 pypi_0 pypi
autobahn 20.7.1 pypi_0 pypi
automat 20.2.0 pypi_0 pypi
base58 2.0.1 pypi_0 pypi
binance 0.3 pypi_0 pypi
bitarray 1.2.2 pypi_0 pypi
bitstring 3.1.7 pypi_0 pypi
ca-certificates 2020.6.20 hecda079_0 conda-forge
cachetools 4.1.1 py_0 conda-forge
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
cffi 1.14.3 pypi_0 pypi
chardet 3.0.4 pypi_0 pypi
constantly 15.1.0 pypi_0 pypi
cryptography 3.1.1 pypi_0 pypi
cython 0.29.21 py37h1c9c24e_0 conda-forge
cytoolz 0.11.0 pypi_0 pypi
dateparser 0.7.6 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
deprecated 1.2.10 pypi_0 pypi
eth-abi 2.1.1 pypi_0 pypi
eth-account 0.5.3 pypi_0 pypi
eth-bloom 1.0.3 pypi_0 pypi
eth-hash 0.2.0 pypi_0 pypi
eth-keyfile 0.5.1 pypi_0 pypi
eth-keys 0.2.4 pypi_0 pypi
eth-rlp 0.2.0 pypi_0 pypi
eth-typing 2.2.2 pypi_0 pypi
eth-utils 1.9.5 pypi_0 pypi
hexbytes 0.2.1 pypi_0 pypi
hyperlink 20.0.1 pypi_0 pypi
idna 2.10 pypi_0 pypi
importlib-metadata 2.0.0 pypi_0 pypi
incremental 17.5.0 pypi_0 pypi
ipfshttpclient 0.6.1 pypi_0 pypi
jsonschema 3.2.0 pypi_0 pypi
kafka-python 2.0.2 pypi_0 pypi
ld_impl_linux-aarch64 2.35 h71b2515_9 conda-forge
libblas 3.8.0 17_openblas conda-forge
libcblas 3.8.0 17_openblas conda-forge
libffi 3.2.1 h4c5d2ac_1007 conda-forge
libgcc-ng 9.3.0 h8ab15db_17 conda-forge
libgfortran-ng 7.5.0 h2b9752b_17 conda-forge

----- End of the List:

So just install the specific version in case you still run into issues, and make sure all the versions you installed are the same.

I installed them package by package, separately. to see whether I receive any further error.
Fortunately, there were no more errors, and all the packages got installed successfully.

and see whether the packages match with the below screenshot from my system. (don't bother much unless you are running into trouble)

Note: While installing web3, the Eth_account automatically got upgraded to a new version, but don't bother about that.
If it works fine, just let it be! :)

Even if you want to tinker with the code, make a backup and save the versions
just for a hassle free installation again in case things go wrong.

And, here is my final, minimalistic setup:

IMG_20201007_133818144[2957].jpg

IMG_20201007_133556749[2955].jpg

As you notice, I have connected wirelessly through SSH/VNC on my laptop. And the LAN cable was also not required. :)

IMG_20201007_133703777[2956].jpg


Thoughts: Installation was done successfully, and for running multiple instances of Hummingbot, use TMux
https://hummingbot.io/blog/2020-07-deploying-hummingbot-on-a-raspberry-pi/

Now you may want to start the actual market making:

And this is beyond the scope of this article for now, so I'm dropping the link to the Hummingbot Site:

https://docs.hummingbot.io/quickstart/

Check this out and let us know which Market making strategy you feel is the best ;)

I hope this will help you solve any issues you may face while installing the Hummingbot Market maker.

Feel free to explore further to see how you can mine liquidity as well:

https://miners.hummingbot.io/


Let us know in the comments what you think

If you liked the article, please this with your friends and circles, and also spare an upvote for me, so that I get motivated to keep sharing market insights and analysis :)


Click this to get onboarded on Hive :)

Leofinance.png

Optional: Want to support me?

In case you want to support my work, please consider using the referral links to sign up on any of the exchanges below using my affiliate link(s):
(You will also get benefit and save on fees)

FTX: https://ftx.com/#a=1846613
Binance: https://www.binance.com/en/register?ref=NL4008SG
Okex: https://www.okex.com/join/2172681 (Get $10 for free)
Kucoin: https://www.kucoin.com/ucenter/signup?rcode=E3t8Ao&lang=en_US
Bittrex: https://bittrex.com/Account/Register?referralCode=XWY-LBA-B5F
Pionex: https://www.pionex.com/en-US/sign?r=eOxX26Bq
Poloniex: https://poloniex.com/signup?c=M5NDJH48
Delta Exchange: https://www.delta.exchange/referral?code=CVIVPG
Bitbns: https://ref.bitbns.com/56721 (Get 50 Bns tokens)
Bitpolo: https://www.bitpolo.com/registerUser?ref=RAH25C

These are the exchanges I'm using right now. I'm not directly affiliated with any of the above exchanges.

You can also download one of the best Ad block/privacy Brave browser and try it out for a month to support my work: https://brave.com/bee435

(P.S. I'm a Brave verified creator, and not directly affiliated with them. If you use the link to download, I'll get some small amount as commission if you download Brave from my link)

Posted Using LeoFinance Beta