Bank integration tutorial (open banking)
Date | Responsible | Changes |
---|---|---|
December 23, 2022 | @Branko Durdevic | Initial version |
February 10, 2023 | @Branko Durdevic | Support for two phase commit protocol |
February 14, 2023 | @Branko Durdevic | Add persistence and simplify code examples |
February 27, 2023 | @Omar Monterrey | • Added version to Ledger URL (On SDK Initialization) • Refactored signer.schema -> signer.format , signature.schema → signature.method , meta.signatures → meta.proofs |
March 8, 2023 | @Branko Durdevic | Synced Tutorial with new Ledger version |
May 22, 2023 | @Omar Monterrey | Removed schema: rest from bridge summary when updating |
Introduction
This tutorial will show you how to connect a bank to a cloud based system built using the Minka Ledger. For this purpose we will be implementing a bridge component that connects the bank’s core systems with the cloud Ledger.
This tutorial shows how to implement a service used to explain the main steps involved in building a bridge, this is not a guide on how to build a production ready application.
When building a service that is going to be deployed to production, follow your usual best practices on application design, security etc. Basic knowledge of the Ledger v2 - docs is expected.
The code is not written as idiomatic JavaScript code, but in a way that makes it easy to understand even for developers not familiar with JavaScript.
Here is a diagram that will help you understand the role of the Bridge in the overall architecture of the system.
Setup
Prerequisites
We will need the following tools in order to develop and run the demo. You may already have some of them installed from the Making cross-ledger payments.
This tutorial is an extension of the Making cross-ledger payments . If you have not already, please complete it first to get familiar with basic Ledger concepts. The following steps assume that you already have a bank setup which is done as part of the previous tutorial.
Node.js and npm
https://nodejs.org/en/download/
Minka CLI tools
https://www.npmjs.com/package/@minka/cli
After installing Node.js and npm, you can install Minka CLI tools by running the following command.
Docker
https://docs.docker.com/get-docker/
Ngrok can be used to host and test the API service.
https://ngrok.com/docs/getting-started
If you are on a Mac and have brew installed, you can just run the command below. Otherwise, check out the getting started docs above.
Curl will be used to test the API, but you can also use Postman or another tool you prefer.
Repository
The accompanying git repository can be found at the following URL.
https://github.com/minkainc/demo-bridge
If at any point during the tutorial you get stuck or your code gets out of sync with the tutorial, you can simply check-out the appropriate commit and continue from there.
Sending money from account to account
Setting up the Bridge service
The simplest case of sending money between two banks is a direct account to account transfer.
This means that we know both the source and target account number and the corresponding banks. An account in this context represents any account whose balance is managed outside of the ledger also known as external accounts.
This includes checking and savings accounts as well as loan and credit card accounts. Of course, other types of products are also possible. You can find more details in the reference documentation.
We will now start implementing the Bridge component that we will use to communicate with the Ledger. For now, payment initiation is going to be performed using the CLI tool.
Setup
You will need to have the NodeJS
and npm
prerequisites installed locally.
First, we will create an empty nodeJS project with a src
directory.
Now that we have an empty node project, the next step is to install express
in order to expose several REST APIs to simulate common bank operations.
We can install express
using npm
.
It is important to note here that we will be using version 4
of express
, please be careful not to use version 5.
We will be using a newer import
syntax in our project, so we need to add a "type": "module"
option to our package.json
to enable this. We will also put our source files under src
so edit main to say "main": "src/index.js"
.
Now we can test if everything is setup correctly by creating a simple hello world route in src/index.js
.
To test our code we need to start the app.
If we open localhost:3001 in a browser or simply run the command below in a new terminal, we should see that our default route is being called and the message we return is there.
If you want to avoid having to restart the service after every change, you can install a tool like nodemon
that will monitor your project for changes and automatically restart the service.
To install nodemon
run:
Then you should run the application using:
Interfaces
We will build our bridge in this tutorial by adding functionality as we need it, step by step. Just to have a high level overview of what we will get in the end, here are all the endpoints that we will implement:
Endpoint | Request | Description |
---|---|---|
POST /v2/credits | prepare | Called during two-phase commit when Bridge needs to prepare a credit Entry. The Bridge should check if the account exists, is active and do other checks necessary to ensure the transfer intent can proceed. |
POST /v2/credits/:handle/commit | commit | Called during two-phase commit when Bridge needs to commit a credit Entry. This request must succeed and is considered successful even if there is an error while processing. In that case, the Bank needs to fix the issue manually. |
POST /v2/credits/:handle/abort | abort | Called during two-phase commit when Bridge needs to abort a credit Entry. This request gets called if there is an issue while processing the transfer. It can not fail just like commit and must completely reverse the transfer. |
POST /v2/debits | prepare | Called during two-phase commit when Bridge needs to prepare a debit Entry. Same as for credit, but needs to also hold or reserve the necessary funds on source account. |
POST /v2/debits/:handle/commit | commit | Called during two-phase commit when Bridge needs to commit a debit Entry. Same as for credit. |
POST /v2/debits/:handle/abort | abort | Called during two-phase commit when Bridge needs to abort a debit Entry. Same as for credit, but potentially also needs to release the funds. |
PUT /v2/intents/:handle | update | Called during two-phase commit when the Intent state updates. This can be used to update the Intent in local database so the Bank knows when it is fully processed. |
As you can see, the Bridge side of the Ledger - Bridge interface consists of 3 main routes and 7 endpoints in total. We will explore them one at a time in the next few sections.
Logging requests and errors
Let’s first just log the requests to see what they look like. To do that we will create a new file named middleware/logging.js
where we will implement our request logging middleware.
We start by creating a new directory.
Then we create the src/middleware/logging.js
file and add the logRequest
function.
We will also add simple error handling to make debugging easier, and we will keep the helper functions in src/middleware/errors.js
. You don’t need to concern yourself with the details of this, it is enough to know that we will be wrapping our handlers in asyncErrorWrapper
and that the handleErrors
function will get called for all errors that bubble up to the top of the execution stack. At the moment we will just be returning 500 Internal Server Error
for all errors to the client.
index.js
should now be edited to look like this.
Test endpoints
After restarting the bridge service, we can issue simple curl
commands in a separate window to check that our endpoints are accepting requests.
In the terminal where we have restarted the service we should see something like this.
Processing a credit Entry
Now that we are set up and receiving requests, we can create an intent and we should see some requests coming from the Ledger. Here we will use the signer, wallet and bridge that we created in the previous tutorial. We will use the same mint signer, wallet and bridge that you created in the previous tutorial and we will start by registering the Bridge with the Ledger. It should already exist so we will just update the URL to point to us. Before that, however, you will need a public URL that Ledger can target. If you don’t have one, you can use ngrok.
Start by opening a dedicated terminal for ngrok and running the following command.
If this is not your first time running ngrok, it is still possible that your URL changed so please check because you may need to update it anyway.
Leave ngrok running and continue the tutorial in a new terminal. You should now have three terminal windows open, one for the Bridge service, another one for ngrok and a new one for CLI commands.
Next, run the following command, substituting mint for your own bridge. When prompted, update the Server
field and set it to <bridge URL> from the command above.
We will now finally create an intent using the CLI tool like this.
We should see the first Entry record in Bridge logs.
Processing requests
We see that we got a request to the POST /v2/credits endpoint.
This request is part of the Ledger two-phase commit protocol, please read more about it in About Intents.
To process this request, we need to:
Save the request
We need this to properly handle idempotency and retries, but it is also valuable information we can use to debug any issues we encounter. We will just save it to a map for now, but will add a database later. Ledger may retry the request in case of issues like timeouts so we could get the same request multiple times.
Respond to Ledger with 202 Accepted
If we get the same request again, we should respond the same way. After this point the responsibility is on us to notify Ledger on how to proceed. If we don’t do it in time, transfer abort will be triggered.
Validate the request This means checking that the hash and proofs are correct and that the public key belongs to the Ledger.
Process the request Here we need to prepare for crediting the client account which involves things like checking that it exists, that it’s active and similar in the banking core.
To do this we will create a mock banking core which should be replaced by calls to the actual core in the real service.
In case of debit, we will also need to check that the client has enough funds and put a hold on those funds.
After saving the result to our database, we are no longer allowed to fail on commit. At that point the responsibility to commit or abort the transfer is passed on to Ledger, even though it’s still not notified of the result. This is important because we may get the same request multiple times and another request may have it’s response served from the DB sooner.
Notify Ledger We need to ensure that the Ledger is fully aware of the outcome of the processing and if it is possible to move forward with the Intent.
Credit Entry
We will first create a new directory for our request handlers.
Now we can create credits.js
where we will keep all of our credit handlers. Let’s start by adding the prepareCredit
function to the file and just responding with 202 Accepted
.
We will also need to register the POST /v2/credits route in index.js
. Notice where we placed the handler and that we wrapped it with asyncErrorWrapper
defined earlier. Don’t forget to update the imports too.
The already existing code will always be grayed out in this tutorial and unless otherwise specified, you just need to add the new code in red.
You can try the same request as earlier and you should still be receiving requests from Ledger. We will now proceed in bigger steps and explain the code in detail. Unfortunately, the biggest step is the first one because we have a lot of preparation to get through so bare with me. Let’s start with updating the prepareCredit
function according to the processing steps we wrote above. We will replace the entire file with the contents below.
As you can see there are a lot of things here that we haven’t yet implemented so let’s do that now. First of all, let’s create the src/handlers/common.js
file with the following content.
The code above is used to start and end Action processing. In addition, the saveIntent
action is used to save the Intent we get as part of a new Entry.
Saving requests
Next, let’s create src/persistence.js
. You will see that we are just using a map to implement get and create operations so our persistence is not very persistent at the moment, but we will come back to this later.
Notifying Ledger
In order to notify Ledger of the status of processing, we will use @minka/ledger-sdk
to sign the Intent with the state. To do that we will create ledger.js
with the notifyLedger
function which we will use to let Ledger know about processing results. You will need to populate the bankKeyPair
with your Signer, ledgerSigner
with Ledger public key and server
with the Ledger URL.
At this point, we will also need to install @minka/ledger-sdk
.
Validation
To validate requests, we will use a couple of functions which we will put into src/validators.js
.
Simulate banking core
Another thing we are missing at this point is a banking core. To simulate it we will create a simple module that will hold balances of client accounts.
It will have similar high-level concepts as a real ledger, but is rudimentary and just for demonstrative purposes.
In the real world this would be replaced by API calls to the actual core.
The internals of this module are not important so feel free to copy/paste it. The thing to note is that this is an in-memory mock ledger with some accounts set up in advance that will enable us to simulate different cases.
It exposes a few methods and will throw errors in case of eg. insufficient balance. The credit
and debit
methods credit and debit client accounts respectively. The reserve
method will reserve client balances so that they can’t be spent and release
will make them available for spending.
Now when we run Test 1
again, we should get:
If the output above is too verbose for you, you can comment out the second logging line in the logRequest
function.
Commit credit
We have successfully processed the prepare
request for a credit Entry and Ledger continued processing up to the point where we got POST /v2/credits/:handle/commit. Now we will need to implement the commit
request to continue.
We will use the same pattern as we did to process prepare so we will create the commitCredit
function accordingly and add processCommitCredit
to src/handlers/credits.js
.
We can notice a couple of differences, first of all we are using beginActionExisting
which expects the Entry to already exist.
Secondly, we only notify Ledger if we end up in the committed
state. As you know from About Intents, we are not permitted to fail the commit
request so if something happens during processing, we set the state to error
and this is something we will need to fix manually.
For all intents and purposes, the Entry is already committed
when we get the commit
, it was just not yet processed by the Bank. We will now add the beginActionExisting
function immediately below beginActionNew
to src/common.js
.
We will also need to import this in src/handlers/credits.js
.
And finally add an import and a route to src/index.js
.
We can now run Test 1
again and we should get something like this.
We can see that Ledger continued processing the intent and PUT /v2/intents/:handle was called. The last request gets called when the status of the intent has changed. We can use it to update the status of the Intent in our database if we decide to save it, but we will skip it for now.
Processing a debit Entry
Now we will do the same thing we did above, but for debit endpoints. To implement POST /v2/debits, we will create a file named src/handlers/debits.js
and add the following code to it.
You can notice that processing for the prepare
action is a little more complex than that of the credit counterpart because we have to hold the funds.
If we run into issues, we set the state to failed
and notify Ledger which will initiate an abort. If we instead notify Ledger that we are prepared
, we can no longer fail when we get the commit
action.
We also need to add an import and a route to src/index.js
.
Prepare debit test
To test the code above, we will create a new intent that will send money from account:1@mint
to account:73514@tesla
.
We should now be seeing something like this.
Commit debit
We will now implement the POST /v2/debits/:handle/commit endpoint by adding the following code to src/handlers/debits.js
We also need to add an import and a route to src/index.js
.
Again, you can see that the processing for commit
debit is a little more complex than for commit
credit because we first need to release the funds and then execute a debit operation. Depending on how transfer orders are implemented, it is possible that in an real core this may be a single operation.
After running Test 2
again, we should get the following.
Combining credits and debits
We are now ready to play with our new Bridge 🥳. To do that, we have 4 different accounts preconfigured in our core:
1 - no balance available because it’s a new account
2 - 70 USD available
3 - no available balance because everything is either spent or on hold
4 - inactive account
Let’s try sending some money from account 2 to account 1. In reality we would probably process this transfer internally since it is an intrabank transfer, but here for now it will be processed by Ledger.
We should now see both credit and debit requests in the logs like this.
Aborting a request
This may be a good time to show the state diagram for processing eg. credit Entries. Ledger sends prepare
, commit
or abort
requests and we respond with prepared
, failed
, committed
or aborted
, depending on the request and what happened. commit
and abort
requests can not fail and therefore end up in the error
state that will have to be fixed internally in the bank. The state diagram for debit Entries looks the same.
State | Description |
---|---|
processing-prepare | This is the first state that an Entry can be in. It is triggered by receiving a prepare request for credit or debit Entries. |
prepared | If a processing-prepare step is successful. We need to notify Ledger of this status. |
failed | If a processing-prepare step fails. We need to notify Ledger of this status. |
processing-commit | If all participants report prepared , Ledger will send commit debit and commit credit requests which will put us into this state. |
committed | If processing-commit step is successful. We need to notify Ledger of this status. |
processing-abort | If Ledger sends an abort request for credit or debit Entries. |
aborted | If processing-abort step is successful. We need to notify Ledger of this status. |
error | If processing-commit or processing-abort steps fail. This should not be possible so we have a special status for this. The step is considered successful even if there is an error so it is up to the Bank to fix all steps that end up in this status. |
It is important to note that the abort
request may arrive while we are in processing-prepare
or prepared
states, and will definitely happen if we are in the failed
state after we notify Ledger of the failure. If it happens while we are in prepared
or failed
states, we can process the request (we just need to take into account what state we were in before), but if the abort
request arrives while we are in processing-prepare
, we have set up our code to refuse the request and let Ledger send it again (by not returning 202 Accepted). We could also save this information, respond with 202 Accepted and process it after the current step finishes processing or even better (because we could do it faster) preempt the processing-prepare
step if possible.
Insufficient balance
To begin with, let’s try creating a request that should fail because there is not enough balance in the account.
We see that we get the error we expected: Insufficient available balance in account 3
and after notifying Ledger of the failure POST /v2/debits/:handle/abort was called in the Bridge.
We will now implement this endpoint the same way as the other ones. First we will add the following code to the end of src/handlers/debits.js
.
And like before we have to add the import and route handler to src/index.js
.
We can see that the processing here depends on the state the Entry was already in. If it was prepared
, we need to release the funds, otherwise we don’t have to do anything.
If we now rerun Test 4
, we will get the following logs which is exactly what we expected.
Account inactive
Finally, let’s create an intent that should fail because the target account is inactive.
We see that this time, because the error occurred on the credit side, we got two abort requests.
Let’s implement POST /v2/credit/:handle/abort the same way we did for debit. We will add the following code to src/handlers/credits.js
.
And of course add the following to src/index.js
.
We can again notice that the only state that we can end up in is aborted
. If something goes wrong, we will end up in the error
state and have to fix it internally.
And if we now retry Test 5
we see that it’s fully aborted.
We see that prepare
failed because Account 4 is inactive
so both the credit and debit Entries got the respective abort actions and were aborted.
Sending money to a phone number
Routing money to internal account
Create wallet
Another thing we can try is sending money to a phone number. To do this, we will first set up a phone wallet so that it redirects the money to account:1@mint
.
Create intent
Now let’s try sending some money to this account.
And we should see logs similar to the ones below.
You will notice that, because we have configured the phone wallet to forward the money to account:1@mint
, we now see two intents. Both of them belong to the same Thread.
Aborting account to phone transfer
We will finally do the ultimate test where we redirect money to an inactive internal account. We should see two requests from two different intents and both of them should be aborted.
Create wallet
First we need to create a phone account with the correct route.
Create intent
Next we create a transfer intent to send money to that wallet.
And we get the result we expected! 🍾
Updating Intents
When completing each of the examples above, we always got a PUT /v2/intents/:handle request in the end that allows us to update the Intent so let’s finally do that.
We’ll create a new file called src/handlers/intents.js
.
And add the following import and route to src/index.js
.
And that’s it, we have implemented the entire Ledger - Bridge interface! 😅
Database persistence
Introduction
Until now our Bridge was completely in-memory. This means that every time we restart it, we loose all data. This also makes it impossible for us to recover from a crash of our service or other situations. We also lose all debugging and reconciliation data. Besides, it is impossible to have more than one instance of the service. Of course, we need persistence, so let’s finally add it.
We will use PostgreSQL through Docker where we will save Bridge data. We will still lose our in-memory core data, but that is not the focus of this tutorial.
Setting up PostgreSQL using Docker
We will be using Docker to set everything up so we will first open a new terminal (yes, we are up to 4), create a docker
directory and create two files, src/docker/init.sql
and src/docker/docker-compose.yaml
.
To start the database we can run and leave the terminal open.
Ctrl+C will stop the database or you can run the following command.
Of course in a production system you would probably have a dedicated database server and migration scripts in the application, but for us this will do.
Saving to database
We will now replace the entire contents of src/persistence.js
with the following.
You will notice we are using the pg
library so let’s install it.
Let’s add calls to init
and shutdown
functions to index.js
.
The persistence functions we have defined above have the same signatures as the ones we replaced with one small difference. The first parameter in each one of them is client
. This is the pg
client that we will get from our connection pool and that we will pass to the function. Another thing you will notice is the transactionWrapper
function. It gets the client from the pool and passes it to the function it wraps. It will enable us to properly handle database transactions with multiple queries and rollbacks when something goes wrong.
The rest of the code will remain the same, the only thing we need to do is wrap all of our calls to persistence functions in transactionWrapper
and add client
to the arguments list.
And that’s it!
Error handling
Until now we have mostly ignored errors or just returned generic values. However, we do need to deal with errors at some point. To begin with, you can check out the Error reference.
For us there will be two main types of errors. Ones that we will respond with directly to all the Entry, Action and Intent requests (which includes HTTP status codes and Ledger error codes) and ones that we will respond with after processing (sending signatures).
There can be many different sources of errors for our application like validation errors, application errors or external errors when communicating with the Core or database (network issues, timeouts and similar).
Additionally, when communicating with the Core, we may get not just timeouts, but also HTTP status codes indicating issues or successful responses that themselves indicate an error.
You will need to cover all of these cases for your particular situation, but we will give some examples of various errors to serve as a guide.
Conclusion
With everything we have covered so far, we have built a real time payments system between bank accounts. You can now check out the payment initiation tutorial too.