Add support for S3-backed image storage

README.md

@@ -50,7 +50,7 @@ or `PODMAN` to use another container command, for example `make PODMAN=docker`.
 5. Install `sqlx-cli`: `cargo install sqlx-cli`.
 6. Run migrations: `sqlx migrate run`.
 7. Set environment variables:
-    * `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
+    - `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
 8. Build: `cargo build --release`
 
 ### Configuring
@@ -58,41 +58,97 @@ or `PODMAN` to use another container command, for example `make PODMAN=docker`.
 Reminder Bot reads a number of environment variables. Some are essential, and others have hardcoded
 fallbacks. Environment variables can be loaded from a .env file in the working directory.
 
-__Required Variables__
+**Required Variables**
 
-* `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
-* `DISCORD_TOKEN` - your application's bot user's authorization token
+-   `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
+-   `DISCORD_TOKEN` - your application's bot user's authorization token
 
-__Other Variables__
+**Other Variables**
 
-* `MIN_INTERVAL` - default `600`, defines the shortest interval the bot should accept
-* `LOCAL_TIMEZONE` - default `UTC`, necessary for calculations in the natural language processor
-* `SUBSCRIPTION_ROLES` - default `None`, accepts a list of Discord role IDs that are given to
-  subscribed users
-* `CNC_GUILD` - default `None`, accepts a single Discord guild ID for the server that the
-  subscription roles belong to
-* `PYTHON_LOCATION` - default `/usr/bin/python3`. Can be changed if your Python executable is
-  located somewhere else
-* `THEME_COLOR` - default `8fb677`. Specifies the hex value of the color to use on info message
-  embeds
+-   `MIN_INTERVAL` - default `600`, defines the shortest interval the bot should accept
+-   `LOCAL_TIMEZONE` - default `UTC`, necessary for calculations in the natural language processor
+-   `SUBSCRIPTION_ROLES` - default `None`, accepts a list of Discord role IDs that are given to
+    subscribed users
+-   `CNC_GUILD` - default `None`, accepts a single Discord guild ID for the server that the
+    subscription roles belong to
+-   `PYTHON_LOCATION` - default `/usr/bin/python3`. Can be changed if your Python executable is
+    located somewhere else
+-   `THEME_COLOR` - default `8fb677`. Specifies the hex value of the color to use on info message
+    embeds
 
-Images can be uploaded directly when all of these settings are supplied:
+### Dashboard image uploads
 
- * `S3_BUCKET` - bucket
- * `S3_REGION` - bucket region
- * `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` - credentials
- * `S3_PUBLIC_URL` - public base URL. Objects must be publicly readable through this URL.
- * `NSFW_CHECK_ENABLED` - enables the use of a small model to block uploading
-   of NSFW images. Set it to `false` to skip NSFW moderation and model initialization.
+Image URL inputs offer an **Upload image** button when all of these settings are supplied:
+
+-   `S3_BUCKET` — destination bucket name.
+-   `S3_REGION` — bucket region, for example `eu-west-2`.
+-   `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` — credentials with `s3:PutObject` and
+    `s3:DeleteObject` permissions for the bucket's `uploads/` prefix.
+-   `S3_PUBLIC_URL` — public base URL for the bucket or CDN, for example
+    `https://my-bucket.s3.eu-west-2.amazonaws.com`. Objects must be publicly readable through
+    this URL; the application does not change bucket policies or set object ACLs.
+
+`NSFW_CHECK_ENABLED` defaults to `true`, and enables the use of a small model to block uploading
+of NSFW images. Set it to `false` to skip NSFW moderation and model initialization. Empty or
+unrecognized values keep checks enabled.
+
+The Debian package includes `reminder-rs-clean-images.timer`, an hourly cleanup job. Enable it
+after configuring S3 and starting the bot (which applies database migrations):
+
+```sh
+sudo systemctl enable --now reminder-rs-clean-images.timer
+```
+
+Each run records the first observed orphan time in `image_uploads.orphaned_at` (UTC). An upload
+is orphaned when neither a reminder (including archived reminders) nor a template uses its URL
+as an avatar, image, thumbnail, author icon or footer icon. Referenced uploads have this timestamp
+cleared. `IMAGE_UPLOAD_ORPHAN_RETENTION_SECONDS` controls the additional retention time after
+detection; it defaults to `604800` (seven days) and must be a positive integer. Unsaved uploads
+are also covered. Cleanup rechecks references before deleting and removes the tracking row only
+after S3 confirms deletion; failed deletions are retried on later runs. Each run attempts up to
+1,000 expired uploads from the configured public URL's `uploads/` prefix.
+
+For other deployments, schedule `/path/to/reminder-rs clean-images` hourly with cron, using the
+same working directory and configuration as the bot. The command requires no Discord token
+and skips NSFW model initialization. With no S3 settings it exits successfully without work;
+partial S3 settings or invalid retention values fail the job.
 
 ## Running with Docker
 
+### RustFS for local development
+
+Start the local object store and initialize the image upload bucket:
+
+```sh
+docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml up -d
+```
+
+Add the settings from `conf/rustfs.env` to your local `.env`, then restart the bot running on
+your host. RustFS serves the S3 API at `http://127.0.0.1:9000` and its console at
+`http://127.0.0.1:9001`. Log into the console with the access key and secret key from
+`conf/rustfs.env`. Both ports bind to localhost. The named data volume persists across restarts.
+
+The initializer creates `reminder-images` if needed and permits anonymous reads of its
+`uploads/` objects. Uploads, deletes and bucket listing require credentials. These settings
+enable dashboard image uploads locally; `NSFW_CHECK_ENABLED` keeps its configured value.
+`S3_ENDPOINT` uses path-style addressing for RustFS, and `S3_PUBLIC_URL` includes the bucket
+name because the browser fetches objects through that public URL.
+
+```sh
+docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml logs rustfs-init
+docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml down
+```
+
+`down` keeps the data volume. This development stack is separate from the bot/database stack
+below. The [RustFS Docker documentation](https://docs.rustfs.com/en/installation/container/docker)
+describes the upstream container configuration.
+
 A `compose.yml` file is provided to aid in running the bot agnostically using docker.
 
-* Populate a `.env` file as in `conf/default.env`
-* Add the additional variable `ROCKET_SECRET_KEY` with a key generated from
-  `head -c64 /dev/urandom | base64`
-* Run `docker compose up`
+-   Populate a `.env` file as in `conf/default.env`
+-   Add the additional variable `ROCKET_SECRET_KEY` with a key generated from
+    `head -c64 /dev/urandom | base64`
+-   Run `docker compose up`
 
 Please note that this is _not_ production-ready when run via compose. We do not offer a way for
 backing up of your data, or a way to run the dashboard securely via HTTPS, which is required for

compose.rustfs.yaml

@@ -0,0 +1,63 @@
+name: reminder-rustfs-dev
+
+services:
+    rustfs-permissions:
+        image: busybox:1.37
+        user: "0:0"
+        volumes:
+            - rustfs-data:/data
+            - rustfs-logs:/logs
+        command: ["sh", "-c", "chown 10001:10001 /data /logs"]
+        restart: "no"
+
+    rustfs:
+        image: rustfs/rustfs:latest
+        restart: unless-stopped
+        depends_on:
+            rustfs-permissions:
+                condition: service_completed_successfully
+        environment:
+            RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY_ID:-REMINDERDEV}
+            RUSTFS_SECRET_KEY: ${S3_SECRET_ACCESS_KEY:-reminder-rustfs-local-secret}
+            RUSTFS_REGION: ${S3_REGION:-us-east-1}
+            RUSTFS_VOLUMES: /data
+            RUSTFS_ADDRESS: "0.0.0.0:9000"
+            RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001"
+            RUSTFS_CONSOLE_ENABLE: "true"
+            RUSTFS_OBS_LOGGER_LEVEL: warn
+            RUSTFS_OBS_LOG_DIRECTORY: /logs
+            # Local single-volume development storage.
+            RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+        volumes:
+            - rustfs-data:/data
+            - rustfs-logs:/logs
+        ports:
+            - "127.0.0.1:9000:9000"
+            - "127.0.0.1:9001:9001"
+        healthcheck:
+            test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health"]
+            interval: 3s
+            timeout: 5s
+            retries: 20
+            start_period: 10s
+
+    rustfs-init:
+        image: amazon/aws-cli:latest
+        depends_on:
+            rustfs:
+                condition: service_healthy
+        environment:
+            AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:-REMINDERDEV}
+            AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:-reminder-rustfs-local-secret}
+            AWS_DEFAULT_REGION: ${S3_REGION:-us-east-1}
+            AWS_EC2_METADATA_DISABLED: "true"
+            AWS_PAGER: ""
+            S3_BUCKET: ${S3_BUCKET:-reminder-images}
+        volumes:
+            - ./scripts/init-rustfs.sh:/init-rustfs.sh:ro
+        entrypoint: ["/bin/sh", "/init-rustfs.sh"]
+        restart: "no"
+
+volumes:
+    rustfs-data:
+    rustfs-logs:

conf/rustfs.env

@@ -0,0 +1,7 @@
+# Local development only. These credentials match compose.rustfs.yaml defaults.
+S3_BUCKET=reminder-images
+S3_REGION=us-east-1
+S3_ACCESS_KEY_ID=REMINDERDEV
+S3_SECRET_ACCESS_KEY=reminder-rustfs-local-secret
+S3_ENDPOINT=http://127.0.0.1:9000
+S3_PUBLIC_URL=http://127.0.0.1:9000/reminder-images

scripts/init-rustfs.sh

@@ -0,0 +1,29 @@
+#!/bin/sh
+set -eu
+
+endpoint=http://rustfs:9000
+
+if ! aws --endpoint-url "$endpoint" s3api head-bucket --bucket "$S3_BUCKET" >/dev/null 2>&1; then
+    if [ "$AWS_DEFAULT_REGION" = us-east-1 ]; then
+        aws --endpoint-url "$endpoint" s3api create-bucket --bucket "$S3_BUCKET"
+    else
+        aws --endpoint-url "$endpoint" s3api create-bucket --bucket "$S3_BUCKET" \
+            --create-bucket-configuration "LocationConstraint=$AWS_DEFAULT_REGION"
+    fi
+fi
+
+# Dashboard images need public object reads. Listing and writes remain authenticated.
+cat > /tmp/public-images-policy.json <<EOF
+{
+    "Version": "2012-10-17",
+    "Statement": [{
+        "Effect": "Allow",
+        "Principal": "*",
+        "Action": "s3:GetObject",
+        "Resource": "arn:aws:s3:::$S3_BUCKET/uploads/*"
+    }]
+}
+EOF
+aws --endpoint-url "$endpoint" s3api put-bucket-policy --bucket "$S3_BUCKET" \
+    --policy file:///tmp/public-images-policy.json
+echo "RustFS development bucket ready: $S3_BUCKET"