Skip to main content

Getting Started

info

Synthetic was formerly named glhf.chat. You may still find references to glhf in the codebase.

Start the backends​

docker compose up --detach

First-time setup​

Currently, the canonical keys are stored in AWS Secrets Manager. Someone will need to grant you access.

Then install the AWS CLI, and run:

# (One time only) Configure AWS SSO login.
aws configure sso
# SSO session name: synthetic
# SSO start URL: https://syntheticlab.awsapps.com/start
# SSO region: us-east-1
# SSO registration scopes: sso:account:access
# Default client Region: us-east-1
# CLI default output format: json
# Profile name: default

# Login to AWS
aws sso login

# Install packages
npm install

# Set up your .env file. You can run this again in the future if new secrets or
# environment variables get added.
npm run sync-dotenv

First-time setup: Migrations​

npx tsx ./scripts/migrate.ts

First-time setup: Enable local HTTPS​

# MacOS
brew install mkcert

# Linux & other OS:
# https://github.com/FiloSottile/mkcert?tab=readme-ov-file#linux

# Add mkcert local Root CA to local trusted CA store.
mkcert -install

# Generate and sign certificates
cd certs
mkcert localhost
tip

If you are accessing the dev instance from a different URL, replace localhost with the correct hostname and either rename your certs to localhost*.pem or edit server.ts.

Run the dev server​

npm run dev:nobuild

# In another terminal session
# This enqueues jobs that run on a schedule. (eg. cleanup, usage credit auto-reload)
#   - A more apt name would be "scheduler". Autoscaler is a holdover from when this job
#   handled scaling on-demand GPUs up and down.
npx tsx autoscaler.ts

# In another terminal session
# This processes jobs from all queues (eg. the "chat" queue for responding to user messages)
npx tsx queue-worker.ts --work-all-queues

# In another terminal session
# This runs the python API which performs token counting via model tokenizers from HuggingFace.
# Setup: python-api/README.md
cd python-api
uv run fastapi dev --port 8080

Chat with LLMs​

  1. Open https://localhost:3000/ and create an account!

  2. Ask for access to the Stripe development sandbox if you don't have access already: https://dashboard.stripe.com/acct_1QFT7V2NOm0u2wFs/test/settings.

  3. (Optional) Set up the Stripe CLI to send webhooks to your development server. This will allow automatic syncing of Stripe state to your development server.

    This is useful for prod-parity validation of the billing system, but for first time setup, this can be skipped.

    Example invocation:

    stripe listen --forward-to https://localhost:3000/billing/stripe/webhook --skip-verify

  4. Find your development user and create a subscription for your user in the Stripe sandbox.

  5. If you did not set up the Stripe CLI above, visit https://localhost:3000/admin/subscriptions/sync and enter the subscription ID to sync your subscription from Stripe to your Next.js dev server.

  6. Verify the Subscription is active in https://localhost:3000/billing

  7. You're good to go!

Running a REPL​

Use the npm run repl command, which will automatically open up a TypeScript REPL for you with the correct env vars configured and tracing set up. This works both in development, and SSHed into production machines if necessary.

Queues & workers​

We have a small layer over Postgres and Redis for distributed job queues. Jobs are stored in Postgres, and we occasionally poll it for new work; however, we also attempt to instantly distribute work via Redis LPUSH/BLPOP, to avoid the latency of polling. The basic abstraction is a Queue (imported from @/app/queues), which takes:

  1. A name, and
  2. A mapping of job names to worker configurations

Worker configurations should generally be set up by first defining their max allowed attempts before failure, and then giving a Structural type for the job data and an async callback that runs the job. For example:

import { t } from "structural";
import { queue, maxAttempts } from "@/app/queues";

const worker = maxAttempts(5);
export const myQueue = queue("myQueue", {
  myJob: worker(t.str, async (someString) => {
    // ... business logic goes here
  }),
});


// To push a job:
await myQueue.push("myJob", "hello world");

Then, import all your queue files in @/queue-worker.ts, which is the main worker entrypoint. Your queue will automatically start running. Jobs will be delivered in-order; however, since jobs can be worked across multiple machines, there's no guarantee of them finishing in-order: only in-order delivery is guaranteed.

If a job fails too many times, it will get added to its queue's corresponding dead-letter queue, which is a capped-size list in Redis for holding permanently-failed jobs for debugging. You can check your queue's dead-letter queue with:

const deadLetters = await myQueue.deadLetters();

The dead-letter queue's size is fixed, so if you have too many job failures, there's no guarantee you'll be able to see all of them. The dead-letter queue isn't a durability mechanism: it's only a debugging tool.

DevOps​

Github Actions and Workflows​

The .github directory contains actions and workflows for Synthetic DevOps.

Testing​

Workflows can be tested locally without commits/pushes by using nektos/act

act push -s GITHUB_TOKEN="$(gh auth token)" --secret-file .env -v