RavenDB

Overview

RavenDB is our primary database, its a document database with integrated Lucene search indexes for powerful & fast querying capabilities

https://ravendb.net/

Rather than hosting RavenDB locally we have a hctr-dev cluster running in AWS London.

Each developer has their own database to allow them to work in isolation without impacting other developers or testing

Hctr-Dev Cluster Studio Access

RavenDB studio is available over HTTP, it is secured via an x509 certificate, you need to install the certificate in your local key chain in order to be able to access the studio from the cloud portal.

The certificate for the Hectare dev cluster is in the Engineering\RavenDB\Certificates folder on Google Drive, download the hctr-dev.hectare.client.certificate.with.password.pfx file to your local PC.

Follow these instructions to install the certificate on MacOS

Open KeyChain app
Click on Login keychain
In the top menu choose File > Import Items
Choose the .pfx file you downloaded from Google Drive
When prompted enter the password from the Backend Secrets secure note in LastPass under section RAVEN CERTIFICATE PASSWORDS (DEV/UAT password)
After the certificate has been imported click on the My Certificates tab and locate the hctr-dev.hectare certificate
Expand the certificate by clicking the arrow so you can see the private key
Right click the private key and choose "Get Info"
Click the access control tab and check the "Allow all applications to access this item"
Click "Save Changes"
Authenticate if required

Once the above steps have been completed the certificate will be available for your browser to use when accessing the RavenDB Studio

Click this link to access Raven Studio

You will be prompted to choose a certificate from a pop up, choose the hctr-dev.hectare certificate file, if you are prompted to authenticate again ensure you choose Allow Always so you don't have to keep doing this.

Certificate When Initialising Document Store

The APIs connect to RavenDB via a document store, we need to configure the client certificate when instantiating the document store.

This requires a few steps, we need to combine the private key and certificate, format it correctly and encrypt it. Once encrypted we can add it to the configuration file in the ravendb.certificate property

For the live certificate download it form the RavenDB server or from the Engineering > RavenDB folder in Google Drive.

Combine the certificate and private key as follows

-----BEGIN CERTIFICATE-----
MIIFIDCCAwigAwIBAgIUc....
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0B....
-----END PRIVATE KEY-----

This needs to be encrypted using the builder.spec tests in the configuration component

  it('Encrypt', () => {
    console.log(
      'DEV/UAT',
      encrypt(
        `-----BEGIN CERTIFICATE-----
MIIFIDCCAwigAw...
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkq...
-----END PRIVATE KEY-----`,
        '[KEY HERE]',
        '[SALT HERE]'
      )
    )
  })

Be careful to ensure there are no extra spaces in the string we're encrypting.

Baseline Database

We maintain a baseline dev database for use by developers in their local environments and in the Dev environment. This data can be added to at any time by following these steps.

Open overrides.json in your local environment and change the developer setting to Baseline
Start the APIs
Create or modify any data via the APIs or Client as necessary (or direct to the database if this is easier)
Revert the overrides.json

Once you have added more data to the baseline database, you need to create a new backup of the Baseline database. Backups are stored in an S3 bucket, create a new backup by following these steps.

Open the Platform_Baseline database in RavenDb
Open the Tasks menu and click Backups
You should see an existing backup called Baseline Database Backup, click the Info icon to the right of the backup
Click Backup Now

This will create a backup and upload it to S3, you can then restore the backup to your local database by running this script (make sure your developer setting is set to your name in overrides.json before restoring)

pnpm run db:local

RavenDB Node.js Client

The RavenDB Node.js client is open source and a very useful resource for understanding how to use the database from Node.js. There is a large suite of unit tests which cover the vast majority of the use cases we'll need. You can clone the repo from here

The documentation for the client also covers many of the common use cases

RavenDB Data Migrations

Each service has its own database and therefore its own migrations app for updating each database. We maintain a directory of migration scripts for each database, locate here...

./services/inventory/api/migrations/scripts
./services/system/api/migrations/scripts
./services/logistics/api/migrations/scripts

etc...

When you want to run migrations for a specific database simply run the appropriate command as follows

pnpm run db:customers
pnpm run db:inventory
pnpm run db:system
pnpm run db:trading

This will update the database on https://a.hctr-dev.hectare.ravendb.cloud that you are connected to via your configuration, the default is InventoryDev unless you have overridden the database through your local config overrides.json, see configuration for more info on local config overrides.

Schema and Index Changes

When you run migrations you are potentially updating search indexes and changing document structure, this can obviously cause breaking changes in the applications using the database being updated.

We should always approach schema and index changes as a 2 phase update.

When we identify a schema or index change that our under development feature relies upon the changes need to be deployed in a safe manner before we release the feature so that when we come to deploy the feature any database changes are already in place.

IDs

In RavenDB you IDs can be auto generated using a hilo algorithm or they can be manually specified.

We use HiLo IDs in general, this works by the RavenDB client requesting a set of IDs (usually 32) from the server and using up these IDs as when new entities are created, when the 32 have been used another batch is retrieved from the server.

HiLo IDs are in for format [collection]/[numeric_id]-[node] i.e. for an asset

assets/1-A

where assets is the collection name, 1 is the ID and A is the node where the document was created.

One issue with this format is it can intefere with URLs as the collection name will be interpreted as part of the URL path, therefore when returning IDs from an API we should strip the collection name from the ID and map it back when IDs are passed into APIs.

Overview​

Hctr-Dev Cluster Studio Access​

Certificate When Initialising Document Store​

Baseline Database​

RavenDB Node.js Client​

RavenDB Data Migrations​

Schema and Index Changes​

IDs​