Developer Experience

developer_experience.md

Getting Started

Node/Yarn Versions

To ease development between projects, you can choose to install a Node version manager (i.e.: nvm) and use it to switch between Node JS versions on-the-fly.

Then, link a suitable Node JS version:

nvm install 20 # only required the first time!
nvm use 20

Environment Setup

1. Install Localstack dependencies:

   pip install awscli-local
   pip install awscli
   pip install localstack

2. Install Docker

3. Copy .env.example --> .env and replace values accordingly.

   cp .env.example .env

4. Install dependencies using Yarn:

   yarn
   yarn bootstrap

Local Development

1. Start the infrastructure.

   Initialize Docker:

   yarn infra

   As Docker initializes, scan the logs for a ready signal from the Localstack service. Once Localstack is "up," we can bootstrap AWS infrastructure mocks. Execute the following command in a separate terminal window:

   yarn infra:init

2. Initialize the API Jobs and Frontend services. Create separate terminal windows for each of the following commands:

   yarn api:start:dev

   yarn jobs:start:dev

   yarn frontend:start:dev

Migrations

Migrations are managed in the persistence library. Migrations will automatically attempt to run when the api starts.

Generating Migrations

If you have made changes to Entities or added new Entities in the persistence library, you can generate new migrations for the changes by running:

yarn migration:generate NameOfMigration

Creating new Migrations

To create a new empty migration:

yarn migration:create NameOfMigration

Running Migrations

Migrations run automatically when the API is started via yarn api:start or api:start:dev but if migrations need to be run manually:

yarn migration:run

Using the Queue Coordinator to Trigger Jobs

Import the coordinator service and queue names. This class automatically infers the proper parameters to pass to the jobs.

import { QueueName, CoordinatorService } from '@lib/coordinator';

class MyInjectableClass {
   constructor(
      private readonly coordinator: CoordinatorService
   )

   ...
}

Triggering a Single Job

   const pendingJob = await this.coordinator.enqueueJob(QueueName.Email, 'send-email', {
      to: '',
      from: '',
      subject: 'File Uploaded',
      // template: someEmailTemplateWithData(data),
   });

Triggering Multiple Jobs at Once

Create a bulk coordinator and add the Job(s)

  const bulkEmails = this.coordinator.createBulkJobs(QueueName.Email);

  bulkEmails.addJob('send-email', {
      to: '',
      from: '',
      subject: '',
      template: '',
   });
  // ... add as many as needed

Enqueue the jobs to run once the worker is ready

   const pendingJobs = await this.coordinator.enqueueBulkJobs(bulkEmails);

Triggering Coordinated Job Flows

Job flows can be arbitrarily complex and as nested as neccessary. It's important to understand that the job flow runs as "breadth-first traversal" meaning the jobs will run from the deepest nested up to the top of the flow tree. and unpredictable results can occur with multiple leaves on the same level.

Create the flow

const flow = await this.coordinator.createFlowJob(QueueName.Email, {
   name: 'send-email',
   data: { to: '', from: '', subject: 'next email' },
   children: [
      {
      queueName: QueueName.Email,
      name: 'send-email',
      data: { to: '', from: '', subject: 'first email' },
      },
   ],
});

Enqueue the flow using the coordinators FlowProducer

const pendingFlow = await this.coordinator.flow.add(flow);

// or multiple flows at once
const pendingFlows = await this.coordinator.flow.addBulk([flow, ...]);

Creating New Workers (Job Handlers)

Creating new jobs, job workers, and flows requires the following steps:

1. Create the primary lib to store the service logic for the job e.g. libs/email-sender

2. Define the structure of the Job within the queue coordinator manifest at libs/queue-coordinator/coordiantor-manifest.ts

3. Create the worker to handle the job at: apps/jobs/queue/workers.

   Note: for now, the Worker must also be added manually to the @apps/jobs/queue/workers/index.ts

Creating New Cron Tasks

Creating cron jobs can be done by adding the job to apps/jobs/cron/tasks. No other configuration is necessary.

Testing

# unit tests
yarn test:unit

# e2e tests
yarn test:e2e

# test coverage
yarn test:cov

Deployment

# Local Deployment stand of env
yarn deploy {env}

# CI/CD (using Github Actions)
yarn release {stage|beta} [patch|minor|major]