Add support for S3-backed image storage

README.md

@@ -50,7 +50,7 @@ or `PODMAN` to use another container command, for example `make PODMAN=docker`.
 5. Install `sqlx-cli`: `cargo install sqlx-cli`.
 6. Run migrations: `sqlx migrate run`.
 7. Set environment variables:
-    - `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
+    * `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
 8. Build: `cargo build --release`
 
 ### Configuring
@@ -58,97 +58,41 @@ or `PODMAN` to use another container command, for example `make PODMAN=docker`.
 Reminder Bot reads a number of environment variables. Some are essential, and others have hardcoded
 fallbacks. Environment variables can be loaded from a .env file in the working directory.
 
-**Required Variables**
+__Required Variables__
 
--   `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
+* `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
--   `DISCORD_TOKEN` - your application's bot user's authorization token
+* `DISCORD_TOKEN` - your application's bot user's authorization token
 
-**Other Variables**
+__Other Variables__
 
--   `MIN_INTERVAL` - default `600`, defines the shortest interval the bot should accept
+* `MIN_INTERVAL` - default `600`, defines the shortest interval the bot should accept
--   `LOCAL_TIMEZONE` - default `UTC`, necessary for calculations in the natural language processor
+* `LOCAL_TIMEZONE` - default `UTC`, necessary for calculations in the natural language processor
--   `SUBSCRIPTION_ROLES` - default `None`, accepts a list of Discord role IDs that are given to
+* `SUBSCRIPTION_ROLES` - default `None`, accepts a list of Discord role IDs that are given to
-    subscribed users
+  subscribed users
--   `CNC_GUILD` - default `None`, accepts a single Discord guild ID for the server that the
+* `CNC_GUILD` - default `None`, accepts a single Discord guild ID for the server that the
-    subscription roles belong to
+  subscription roles belong to
--   `PYTHON_LOCATION` - default `/usr/bin/python3`. Can be changed if your Python executable is
+* `PYTHON_LOCATION` - default `/usr/bin/python3`. Can be changed if your Python executable is
-    located somewhere else
+  located somewhere else
--   `THEME_COLOR` - default `8fb677`. Specifies the hex value of the color to use on info message
+* `THEME_COLOR` - default `8fb677`. Specifies the hex value of the color to use on info message
-    embeds
+  embeds
 
-### Dashboard image uploads
+Images can be uploaded directly when all of these settings are supplied:
 
-Image URL inputs offer an **Upload image** button when all of these settings are supplied:
+ * `S3_BUCKET` - bucket
-
+ * `S3_REGION` - bucket region
--   `S3_BUCKET` — destination bucket name.
+ * `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` - credentials
--   `S3_REGION` — bucket region, for example `eu-west-2`.
+ * `S3_PUBLIC_URL` - public base URL. Objects must be publicly readable through this URL.
--   `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` — credentials with `s3:PutObject` and
+ * `NSFW_CHECK_ENABLED` - enables the use of a small model to block uploading
-    `s3:DeleteObject` permissions for the bucket's `uploads/` prefix.
+   of NSFW images. Set it to `false` to skip NSFW moderation and model initialization.
--   `S3_PUBLIC_URL` — public base URL for the bucket or CDN, for example
-    `https://my-bucket.s3.eu-west-2.amazonaws.com`. Objects must be publicly readable through
-    this URL; the application does not change bucket policies or set object ACLs.
-
-`NSFW_CHECK_ENABLED` defaults to `true`, and enables the use of a small model to block uploading
-of NSFW images. Set it to `false` to skip NSFW moderation and model initialization. Empty or
-unrecognized values keep checks enabled.
-
-The Debian package includes `reminder-rs-clean-images.timer`, an hourly cleanup job. Enable it
-after configuring S3 and starting the bot (which applies database migrations):
-
-```sh
-sudo systemctl enable --now reminder-rs-clean-images.timer
-```
-
-Each run records the first observed orphan time in `image_uploads.orphaned_at` (UTC). An upload
-is orphaned when neither a reminder (including archived reminders) nor a template uses its URL
-as an avatar, image, thumbnail, author icon or footer icon. Referenced uploads have this timestamp
-cleared. `IMAGE_UPLOAD_ORPHAN_RETENTION_SECONDS` controls the additional retention time after
-detection; it defaults to `604800` (seven days) and must be a positive integer. Unsaved uploads
-are also covered. Cleanup rechecks references before deleting and removes the tracking row only
-after S3 confirms deletion; failed deletions are retried on later runs. Each run attempts up to
-1,000 expired uploads from the configured public URL's `uploads/` prefix.
-
-For other deployments, schedule `/path/to/reminder-rs clean-images` hourly with cron, using the
-same working directory and configuration as the bot. The command requires no Discord token
-and skips NSFW model initialization. With no S3 settings it exits successfully without work;
-partial S3 settings or invalid retention values fail the job.
 
 ## Running with Docker
 
-### RustFS for local development
-
-Start the local object store and initialize the image upload bucket:
-
-```sh
-docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml up -d
-```
-
-Add the settings from `conf/rustfs.env` to your local `.env`, then restart the bot running on
-your host. RustFS serves the S3 API at `http://127.0.0.1:9000` and its console at
-`http://127.0.0.1:9001`. Log into the console with the access key and secret key from
-`conf/rustfs.env`. Both ports bind to localhost. The named data volume persists across restarts.
-
-The initializer creates `reminder-images` if needed and permits anonymous reads of its
-`uploads/` objects. Uploads, deletes and bucket listing require credentials. These settings
-enable dashboard image uploads locally; `NSFW_CHECK_ENABLED` keeps its configured value.
-`S3_ENDPOINT` uses path-style addressing for RustFS, and `S3_PUBLIC_URL` includes the bucket
-name because the browser fetches objects through that public URL.
-
-```sh
-docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml logs rustfs-init
-docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml down
-```
-
-`down` keeps the data volume. This development stack is separate from the bot/database stack
-below. The [RustFS Docker documentation](https://docs.rustfs.com/en/installation/container/docker)
-describes the upstream container configuration.
-
 A `compose.yml` file is provided to aid in running the bot agnostically using docker.
 
--   Populate a `.env` file as in `conf/default.env`
+* Populate a `.env` file as in `conf/default.env`
--   Add the additional variable `ROCKET_SECRET_KEY` with a key generated from
+* Add the additional variable `ROCKET_SECRET_KEY` with a key generated from
-    `head -c64 /dev/urandom | base64`
+  `head -c64 /dev/urandom | base64`
--   Run `docker compose up`
+* Run `docker compose up`
 
 Please note that this is _not_ production-ready when run via compose. We do not offer a way for
 backing up of your data, or a way to run the dashboard securely via HTTPS, which is required for

compose.rustfs.yaml

@@ -1,63 +0,0 @@
-name: reminder-rustfs-dev
-
-services:
-    rustfs-permissions:
-        image: busybox:1.37
-        user: "0:0"
-        volumes:
-            - rustfs-data:/data
-            - rustfs-logs:/logs
-        command: ["sh", "-c", "chown 10001:10001 /data /logs"]
-        restart: "no"
-
-    rustfs:
-        image: rustfs/rustfs:latest
-        restart: unless-stopped
-        depends_on:
-            rustfs-permissions:
-                condition: service_completed_successfully
-        environment:
-            RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY_ID:-REMINDERDEV}
-            RUSTFS_SECRET_KEY: ${S3_SECRET_ACCESS_KEY:-reminder-rustfs-local-secret}
-            RUSTFS_REGION: ${S3_REGION:-us-east-1}
-            RUSTFS_VOLUMES: /data
-            RUSTFS_ADDRESS: "0.0.0.0:9000"
-            RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001"
-            RUSTFS_CONSOLE_ENABLE: "true"
-            RUSTFS_OBS_LOGGER_LEVEL: warn
-            RUSTFS_OBS_LOG_DIRECTORY: /logs
-            # Local single-volume development storage.
-            RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
-        volumes:
-            - rustfs-data:/data
-            - rustfs-logs:/logs
-        ports:
-            - "127.0.0.1:9000:9000"
-            - "127.0.0.1:9001:9001"
-        healthcheck:
-            test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health"]
-            interval: 3s
-            timeout: 5s
-            retries: 20
-            start_period: 10s
-
-    rustfs-init:
-        image: amazon/aws-cli:latest
-        depends_on:
-            rustfs:
-                condition: service_healthy
-        environment:
-            AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:-REMINDERDEV}
-            AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:-reminder-rustfs-local-secret}
-            AWS_DEFAULT_REGION: ${S3_REGION:-us-east-1}
-            AWS_EC2_METADATA_DISABLED: "true"
-            AWS_PAGER: ""
-            S3_BUCKET: ${S3_BUCKET:-reminder-images}
-        volumes:
-            - ./scripts/init-rustfs.sh:/init-rustfs.sh:ro
-        entrypoint: ["/bin/sh", "/init-rustfs.sh"]
-        restart: "no"
-
-volumes:
-    rustfs-data:
-    rustfs-logs:

conf/rustfs.env

@@ -1,7 +0,0 @@
-# Local development only. These credentials match compose.rustfs.yaml defaults.
-S3_BUCKET=reminder-images
-S3_REGION=us-east-1
-S3_ACCESS_KEY_ID=REMINDERDEV
-S3_SECRET_ACCESS_KEY=reminder-rustfs-local-secret
-S3_ENDPOINT=http://127.0.0.1:9000
-S3_PUBLIC_URL=http://127.0.0.1:9000/reminder-images

scripts/init-rustfs.sh

@@ -1,29 +0,0 @@
-#!/bin/sh
-set -eu
-
-endpoint=http://rustfs:9000
-
-if ! aws --endpoint-url "$endpoint" s3api head-bucket --bucket "$S3_BUCKET" >/dev/null 2>&1; then
-    if [ "$AWS_DEFAULT_REGION" = us-east-1 ]; then
-        aws --endpoint-url "$endpoint" s3api create-bucket --bucket "$S3_BUCKET"
-    else
-        aws --endpoint-url "$endpoint" s3api create-bucket --bucket "$S3_BUCKET" \
-            --create-bucket-configuration "LocationConstraint=$AWS_DEFAULT_REGION"
-    fi
-fi
-
-# Dashboard images need public object reads. Listing and writes remain authenticated.
-cat > /tmp/public-images-policy.json <<EOF
-{
-    "Version": "2012-10-17",
-    "Statement": [{
-        "Effect": "Allow",
-        "Principal": "*",
-        "Action": "s3:GetObject",
-        "Resource": "arn:aws:s3:::$S3_BUCKET/uploads/*"
-    }]
-}
-EOF
-aws --endpoint-url "$endpoint" s3api put-bucket-policy --bucket "$S3_BUCKET" \
-    --policy file:///tmp/public-images-policy.json
-echo "RustFS development bucket ready: $S3_BUCKET"