PostgreSQL, Redis & Celery Configuration¶

Faraday Server v5.19.0 — Database, message broker, and background worker configuration reference.

Service Architecture¶

┌─────────────────┐     ┌─────────────┐     ┌───────────────────┐
│  Faraday Server │────►│ PostgreSQL   │     │ Redis / RabbitMQ  │
│  (Flask + API)  │     │ (Primary DB) │     │ (Message Broker)  │
└────────┬────────┘     └─────────────┘     └────────┬──────────┘
         │                                           │
         │           ┌───────────────────┐           │
         └──────────►│  Celery Workers   │◄──────────┘
                     │  (Background Jobs)│
                     └───────────────────┘

PostgreSQL Configuration¶

Version Requirements¶

Connection String¶

The PostgreSQL connection is configured in server.ini:

[database]
connection_string = postgresql+psycopg2://USERNAME:PASSWORD@HOST[:PORT]/DATABASE

Format components:

Local PostgreSQL Setup¶

For a Faraday Server installed on the same host as PostgreSQL:

# 1. Create the PostgreSQL user
sudo -u postgres createuser faraday_postgresql -P
# Enter password when prompted

# 2. Create the database
sudo -u postgres createdb -O faraday_postgresql faraday

# 3. Initialize with Faraday
faraday-manage initdb

The faraday-manage initdb command: - Creates the database user (uses FARADAY_DATABASE_USER env var, default faraday_postgresql). - Creates the database (uses FARADAY_DATABASE_NAME env var, default faraday). - Generates the connection string in server.ini. - Creates database tables. - Creates the initial admin user.

Remote PostgreSQL Setup¶

For a dedicated database server:

Step 1 — On the PostgreSQL server:

# Install PostgreSQL
sudo apt install postgresql-14

# Create user and database
sudo -u postgres createuser faraday_postgresql -P
sudo -u postgres createdb -O faraday_postgresql faraday

Step 2 — Configure remote access:

Edit pg_hba.conf (find location with sudo -u postgres psql -c "show hba_file"):

# Allow Faraday Server to connect
host    faraday    faraday_postgresql    192.168.1.0/24    md5

Edit postgresql.conf:

listen_addresses = '*'    # Or specific interface IP

Restart PostgreSQL:

sudo systemctl restart postgresql

Step 3 — On the Faraday Server:

Edit ~/.faraday/config/server.ini:

[database]
connection_string = postgresql+psycopg2://faraday_postgresql:SECRET@192.168.1.10/faraday

Step 4 — Create tables:

faraday-manage create-tables
faraday-manage migrate

Connection Pool Settings¶

The Faraday Server uses SQLAlchemy with a QueuePool connection pool:

Maximum simultaneous connections: 40 (pool_size + max_overflow).

PostgreSQL Tuning Recommendations¶

For production deployments with large datasets:

# postgresql.conf
shared_buffers = 256MB           # 25% of RAM (up to 1-2GB)
effective_cache_size = 768MB     # 50-75% of RAM
work_mem = 16MB                  # Per-query sort memory
maintenance_work_mem = 128MB     # For VACUUM, CREATE INDEX
max_connections = 150            # Faraday pool + workers + admin

Redis Configuration¶

Purpose¶

Redis serves as the default message broker and result backend for Celery background tasks. It is required when celery_enabled = true (the default).

Version Requirements¶

Installation¶

Debian/Ubuntu:

sudo apt install redis-server
sudo systemctl enable redis-server
sudo systemctl start redis-server

RHEL/CentOS:

sudo dnf install redis
sudo systemctl enable redis
sudo systemctl start redis

Configuration in server.ini¶

[faraday_server]
celery_broker_url = redis://127.0.0.1:6379/0
celery_backend_url = redis://127.0.0.1:6379/0

URL format: redis://[:PASSWORD@]HOST:PORT/DB_NUMBER

Shorthand: If you provide just a hostname (e.g., redis), Faraday automatically expands it to redis://HOSTNAME:6379/0.

Redis with TLS:

celery_broker_url = rediss://redis.internal:6380/0
celery_backend_url = rediss://redis.internal:6380/0

Redis with Authentication¶

celery_broker_url = redis://:mypassword@127.0.0.1:6379/0
celery_backend_url = redis://:mypassword@127.0.0.1:6379/0

Multi-Instance Deployments¶

When multiple Faraday instances share the same Redis server, use the celery_queue_prefix setting to isolate their queues:

# Instance 1
celery_queue_prefix = faraday_prod

# Instance 2
celery_queue_prefix = faraday_staging

This sets a global_keyprefix on Celery's Redis transport options.

RabbitMQ Configuration (Alternative Broker)¶

RabbitMQ can be used instead of Redis as the Celery message broker.

Installation¶

sudo apt install rabbitmq-server
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server

Configuration in server.ini¶

[faraday_server]
celery_broker_url = amqp://user:password@127.0.0.1:5672/vhost
celery_backend_url = redis://127.0.0.1:6379/0

RabbitMQ with TLS:

celery_broker_url = amqps://user:password@rabbitmq.internal:5671/vhost

Supported broker URL schemes:

Celery Worker Configuration¶

Worker Types¶

Faraday includes multiple worker entry points:

Worker Parameters¶

Workers are configured via command-line arguments or by starting them alongside the server:

# Start server with workers
faraday-server --with-workers

# Start server with gevent workers
faraday-server --with-workers-gevent

# Customize worker settings
faraday-server --with-workers \
  --workers-queue celery \
  --workers-concurrency 4 \
  --workers-loglevel INFO

Or run workers independently:

faraday-worker --queue celery --concurrency 4 --loglevel INFO

Worker Log File¶

Worker logs are written to ~/.faraday/logs/celery.log with 5 MB rotation and 5 backup files.

systemd Services¶

Bare-metal installations typically run three systemd services:

sudo systemctl enable faraday-server
sudo systemctl enable faraday-worker
sudo systemctl enable faraday-worker-reports   # For report processing

sudo systemctl start faraday-server
sudo systemctl start faraday-worker
sudo systemctl start faraday-worker-reports

Docker Service Configuration¶

In Docker Compose, all services are pre-configured:

services:
  db:
    image: postgres:12.7-alpine
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: faraday

  redis:
    image: redis:8.4-alpine

  faraday-server:
    image: faradaysec/faraday:latest
    environment:
      PGSQL_USER: postgres
      PGSQL_PASSWD: postgres
      PGSQL_HOST: db
      PGSQL_DBNAME: faraday
      REDIS_SERVER: redis

  faraday-default-worker:
    image: faradaysec/faraday:latest
    command: faraday-worker
    environment:
      PGSQL_USER: postgres
      PGSQL_PASSWD: postgres
      PGSQL_HOST: db
      PGSQL_DBNAME: faraday
      REDIS_SERVER: redis

See [[docker]] for the complete Docker deployment reference.

Health Checks¶

PostgreSQL¶

# From the Faraday server
faraday-manage sql-shell
# Or directly
psql -h HOST -U faraday_postgresql -d faraday -c "SELECT 1"

Redis¶

redis-cli -h HOST ping
# Expected response: PONG

Celery Workers¶

The Docker health check verifies the API is responsive:

curl -f http://localhost:5985/_api/config

Troubleshooting¶

"connection refused" to PostgreSQL¶

1. Verify PostgreSQL is running: systemctl status postgresql
2. Check pg_hba.conf allows the connection method and source IP.
3. Verify listen_addresses in postgresql.conf includes the server's IP.
4. Test connectivity: psql -h HOST -U USER -d faraday

"connection refused" to Redis¶

1. Verify Redis is running: systemctl status redis
2. Check Redis bind address: grep bind /etc/redis/redis.conf
3. Check if Redis requires a password: grep requirepass /etc/redis/redis.conf
4. Test connectivity: redis-cli -h HOST ping

Celery workers not starting¶

1. Check the broker URL in server.ini matches the running broker.
2. Ensure celery_enabled = true.
3. Check ~/.faraday/logs/celery.log for connection errors.
4. Verify the broker service is accessible from the worker host.

"QueuePool limit" / Database connection exhaustion¶

The server maintains up to 40 database connections. With multiple workers, this can be exceeded. Solutions: - Increase PostgreSQL max_connections. - Reduce worker --concurrency. - Consider connection pooling with PgBouncer for large deployments.

Docker — database connection refused on first start¶

The entrypoint script waits for PostgreSQL with nc -z, but the database may not be fully ready. The docker-compose.yaml uses a health check with retries (interval: 20s, retries: 10) to handle this. If you still see errors, increase the retry count or add a longer startup delay.