Jude/s3 file storage #6
No files matched your search
@@ -1,6 +1,7 @@
|
|||||||
target
|
target
|
||||||
.env
|
.env
|
||||||
/venv
|
/venv
|
||||||
|
__pycache__/
|
||||||
.cargo
|
.cargo
|
||||||
.idea
|
.idea
|
||||||
static/index.html
|
static/index.html
|
||||||
|
|||||||
Generated
+1320
-81
File diff suppressed because it is too large.
Load diff
+10
-1
@@ -36,6 +36,9 @@ oauth2 = "4"
|
|||||||
csv = "1.2"
|
csv = "1.2"
|
||||||
sd-notify = "0.4.1"
|
sd-notify = "0.4.1"
|
||||||
cron-parser = "0.10"
|
cron-parser = "0.10"
|
||||||
|
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "gif", "webp"] }
|
||||||
|
nsfw = { version = "0.2", default-features = false }
|
||||||
|
rust-s3 = { version = "0.38", default-features = false, features = ["tokio-rustls-tls"] }
|
||||||
|
|
||||||
[dependencies.extract_derive]
|
[dependencies.extract_derive]
|
||||||
path = "extract_derive"
|
path = "extract_derive"
|
||||||
@@ -68,6 +71,12 @@ conf-files = [
|
|||||||
"/etc/reminder-rs/Rocket.toml",
|
"/etc/reminder-rs/Rocket.toml",
|
||||||
]
|
]
|
||||||
|
|
||||||
[package.metadata.deb.systemd-units]
|
[[package.metadata.deb.systemd-units]]
|
||||||
unit-scripts = "systemd"
|
unit-scripts = "systemd"
|
||||||
|
unit-name = "reminder-rs"
|
||||||
|
start = false
|
||||||
|
|
||||||
|
[[package.metadata.deb.systemd-units]]
|
||||||
|
unit-scripts = "systemd"
|
||||||
|
unit-name = "reminder-rs-clean-images"
|
||||||
start = false
|
start = false
|
||||||
@@ -50,7 +50,7 @@ or `PODMAN` to use another container command, for example `make PODMAN=docker`.
|
|||||||
5. Install `sqlx-cli`: `cargo install sqlx-cli`.
|
5. Install `sqlx-cli`: `cargo install sqlx-cli`.
|
||||||
6. Run migrations: `sqlx migrate run`.
|
6. Run migrations: `sqlx migrate run`.
|
||||||
7. Set environment variables:
|
7. Set environment variables:
|
||||||
* `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
|
- `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
|
||||||
8. Build: `cargo build --release`
|
8. Build: `cargo build --release`
|
||||||
|
|
||||||
### Configuring
|
### Configuring
|
||||||
@@ -58,32 +58,97 @@ or `PODMAN` to use another container command, for example `make PODMAN=docker`.
|
|||||||
Reminder Bot reads a number of environment variables. Some are essential, and others have hardcoded
|
Reminder Bot reads a number of environment variables. Some are essential, and others have hardcoded
|
||||||
fallbacks. Environment variables can be loaded from a .env file in the working directory.
|
fallbacks. Environment variables can be loaded from a .env file in the working directory.
|
||||||
|
|
||||||
__Required Variables__
|
**Required Variables**
|
||||||
|
|
||||||
* `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
|
- `DATABASE_URL` - the URL of your MySQL database (`mysql://user[:password]@domain/database`)
|
||||||
* `DISCORD_TOKEN` - your application's bot user's authorization token
|
- `DISCORD_TOKEN` - your application's bot user's authorization token
|
||||||
|
|
||||||
__Other Variables__
|
**Other Variables**
|
||||||
|
|
||||||
* `MIN_INTERVAL` - default `600`, defines the shortest interval the bot should accept
|
- `MIN_INTERVAL` - default `600`, defines the shortest interval the bot should accept
|
||||||
* `LOCAL_TIMEZONE` - default `UTC`, necessary for calculations in the natural language processor
|
- `LOCAL_TIMEZONE` - default `UTC`, necessary for calculations in the natural language processor
|
||||||
* `SUBSCRIPTION_ROLES` - default `None`, accepts a list of Discord role IDs that are given to
|
- `SUBSCRIPTION_ROLES` - default `None`, accepts a list of Discord role IDs that are given to
|
||||||
subscribed users
|
subscribed users
|
||||||
* `CNC_GUILD` - default `None`, accepts a single Discord guild ID for the server that the
|
- `CNC_GUILD` - default `None`, accepts a single Discord guild ID for the server that the
|
||||||
subscription roles belong to
|
subscription roles belong to
|
||||||
* `PYTHON_LOCATION` - default `/usr/bin/python3`. Can be changed if your Python executable is
|
- `PYTHON_LOCATION` - default `/usr/bin/python3`. Can be changed if your Python executable is
|
||||||
located somewhere else
|
located somewhere else
|
||||||
* `THEME_COLOR` - default `8fb677`. Specifies the hex value of the color to use on info message
|
- `THEME_COLOR` - default `8fb677`. Specifies the hex value of the color to use on info message
|
||||||
embeds
|
embeds
|
||||||
|
|
||||||
|
### Dashboard image uploads
|
||||||
|
|
||||||
|
Image URL inputs offer an **Upload image** button when all of these settings are supplied:
|
||||||
|
|
||||||
|
- `S3_BUCKET` — destination bucket name.
|
||||||
|
- `S3_REGION` — bucket region, for example `eu-west-2`.
|
||||||
|
- `S3_ACCESS_KEY_ID` and `S3_SECRET_ACCESS_KEY` — credentials with `s3:PutObject` and
|
||||||
|
`s3:DeleteObject` permissions for the bucket's `uploads/` prefix.
|
||||||
|
- `S3_PUBLIC_URL` — public base URL for the bucket or CDN, for example
|
||||||
|
`https://my-bucket.s3.eu-west-2.amazonaws.com`. Objects must be publicly readable through
|
||||||
|
this URL; the application does not change bucket policies or set object ACLs.
|
||||||
|
|
||||||
|
`NSFW_CHECK_ENABLED` defaults to `true`, and enables the use of a small model to block uploading
|
||||||
|
of NSFW images. Set it to `false` to skip NSFW moderation and model initialization. Empty or
|
||||||
|
unrecognized values keep checks enabled.
|
||||||
|
|
||||||
|
The Debian package includes `reminder-rs-clean-images.timer`, an hourly cleanup job. Enable it
|
||||||
|
after configuring S3 and starting the bot (which applies database migrations):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo systemctl enable --now reminder-rs-clean-images.timer
|
||||||
|
```
|
||||||
|
|
||||||
|
Each run records the first observed orphan time in `image_uploads.orphaned_at` (UTC). An upload
|
||||||
|
is orphaned when neither a reminder (including archived reminders) nor a template uses its URL
|
||||||
|
as an avatar, image, thumbnail, author icon or footer icon. Referenced uploads have this timestamp
|
||||||
|
cleared. `IMAGE_UPLOAD_ORPHAN_RETENTION_SECONDS` controls the additional retention time after
|
||||||
|
detection; it defaults to `604800` (seven days) and must be a positive integer. Unsaved uploads
|
||||||
|
are also covered. Cleanup rechecks references before deleting and removes the tracking row only
|
||||||
|
after S3 confirms deletion; failed deletions are retried on later runs. Each run attempts up to
|
||||||
|
1,000 expired uploads from the configured public URL's `uploads/` prefix.
|
||||||
|
|
||||||
|
For other deployments, schedule `/path/to/reminder-rs clean-images` hourly with cron, using the
|
||||||
|
same working directory and configuration as the bot. The command requires no Discord token
|
||||||
|
and skips NSFW model initialization. With no S3 settings it exits successfully without work;
|
||||||
|
partial S3 settings or invalid retention values fail the job.
|
||||||
|
|
||||||
## Running with Docker
|
## Running with Docker
|
||||||
|
|
||||||
|
### RustFS for local development
|
||||||
|
|
||||||
|
Start the local object store and initialize the image upload bucket:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
Add the settings from `conf/rustfs.env` to your local `.env`, then restart the bot running on
|
||||||
|
your host. RustFS serves the S3 API at `http://127.0.0.1:9000` and its console at
|
||||||
|
`http://127.0.0.1:9001`. Log into the console with the access key and secret key from
|
||||||
|
`conf/rustfs.env`. Both ports bind to localhost. The named data volume persists across restarts.
|
||||||
|
|
||||||
|
The initializer creates `reminder-images` if needed and permits anonymous reads of its
|
||||||
|
`uploads/` objects. Uploads, deletes and bucket listing require credentials. These settings
|
||||||
|
enable dashboard image uploads locally; `NSFW_CHECK_ENABLED` keeps its configured value.
|
||||||
|
`S3_ENDPOINT` uses path-style addressing for RustFS, and `S3_PUBLIC_URL` includes the bucket
|
||||||
|
name because the browser fetches objects through that public URL.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml logs rustfs-init
|
||||||
|
docker compose --env-file conf/rustfs.env -f compose.rustfs.yaml down
|
||||||
|
```
|
||||||
|
|
||||||
|
`down` keeps the data volume. This development stack is separate from the bot/database stack
|
||||||
|
below. The [RustFS Docker documentation](https://docs.rustfs.com/en/installation/container/docker)
|
||||||
|
describes the upstream container configuration.
|
||||||
|
|
||||||
A `compose.yml` file is provided to aid in running the bot agnostically using docker.
|
A `compose.yml` file is provided to aid in running the bot agnostically using docker.
|
||||||
|
|
||||||
* Populate a `.env` file as in `conf/default.env`
|
- Populate a `.env` file as in `conf/default.env`
|
||||||
* Add the additional variable `ROCKET_SECRET_KEY` with a key generated from
|
- Add the additional variable `ROCKET_SECRET_KEY` with a key generated from
|
||||||
`head -c64 /dev/urandom | base64`
|
`head -c64 /dev/urandom | base64`
|
||||||
* Run `docker compose up`
|
- Run `docker compose up`
|
||||||
|
|
||||||
Please note that this is _not_ production-ready when run via compose. We do not offer a way for
|
Please note that this is _not_ production-ready when run via compose. We do not offer a way for
|
||||||
backing up of your data, or a way to run the dashboard securely via HTTPS, which is required for
|
backing up of your data, or a way to run the dashboard securely via HTTPS, which is required for
|
||||||
|
|||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# Bundled NSFW model
|
||||||
|
|
||||||
|
`model.onnx` is the pretrained model distributed with
|
||||||
|
[Fyko/nsfw v0.2.0](https://github.com/fyko/nsfw/releases/tag/v0.2.0), based on
|
||||||
|
[GantMan/nsfw_model](https://github.com/GantMan/nsfw_model).
|
||||||
|
|
||||||
|
Source: https://github.com/fyko/nsfw/releases/download/v0.2.0/model.onnx
|
||||||
Binary file not shown.
@@ -0,0 +1,63 @@
|
|||||||
|
name: reminder-rustfs-dev
|
||||||
|
|
||||||
|
services:
|
||||||
|
rustfs-permissions:
|
||||||
|
image: busybox:1.37
|
||||||
|
user: "0:0"
|
||||||
|
volumes:
|
||||||
|
- rustfs-data:/data
|
||||||
|
- rustfs-logs:/logs
|
||||||
|
command: ["sh", "-c", "chown 10001:10001 /data /logs"]
|
||||||
|
restart: "no"
|
||||||
|
|
||||||
|
rustfs:
|
||||||
|
image: rustfs/rustfs:latest
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
rustfs-permissions:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
environment:
|
||||||
|
RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY_ID:-REMINDERDEV}
|
||||||
|
RUSTFS_SECRET_KEY: ${S3_SECRET_ACCESS_KEY:-reminder-rustfs-local-secret}
|
||||||
|
RUSTFS_REGION: ${S3_REGION:-us-east-1}
|
||||||
|
RUSTFS_VOLUMES: /data
|
||||||
|
RUSTFS_ADDRESS: "0.0.0.0:9000"
|
||||||
|
RUSTFS_CONSOLE_ADDRESS: "0.0.0.0:9001"
|
||||||
|
RUSTFS_CONSOLE_ENABLE: "true"
|
||||||
|
RUSTFS_OBS_LOGGER_LEVEL: warn
|
||||||
|
RUSTFS_OBS_LOG_DIRECTORY: /logs
|
||||||
|
# Local single-volume development storage.
|
||||||
|
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
|
||||||
|
volumes:
|
||||||
|
- rustfs-data:/data
|
||||||
|
- rustfs-logs:/logs
|
||||||
|
ports:
|
||||||
|
- "127.0.0.1:9000:9000"
|
||||||
|
- "127.0.0.1:9001:9001"
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health"]
|
||||||
|
interval: 3s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 20
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
|
rustfs-init:
|
||||||
|
image: amazon/aws-cli:latest
|
||||||
|
depends_on:
|
||||||
|
rustfs:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
AWS_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:-REMINDERDEV}
|
||||||
|
AWS_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:-reminder-rustfs-local-secret}
|
||||||
|
AWS_DEFAULT_REGION: ${S3_REGION:-us-east-1}
|
||||||
|
AWS_EC2_METADATA_DISABLED: "true"
|
||||||
|
AWS_PAGER: ""
|
||||||
|
S3_BUCKET: ${S3_BUCKET:-reminder-images}
|
||||||
|
volumes:
|
||||||
|
- ./scripts/init-rustfs.sh:/init-rustfs.sh:ro
|
||||||
|
entrypoint: ["/bin/sh", "/init-rustfs.sh"]
|
||||||
|
restart: "no"
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
rustfs-data:
|
||||||
|
rustfs-logs:
|
||||||
@@ -20,3 +20,18 @@ REPORT_EMAIL=
|
|||||||
SENT_CLEAN_AGE=
|
SENT_CLEAN_AGE=
|
||||||
# Delete all reminders after this many months
|
# Delete all reminders after this many months
|
||||||
TOTAL_CLEAN_AGE=
|
TOTAL_CLEAN_AGE=
|
||||||
|
|
||||||
|
# Optional dashboard image uploads. Leave these blank to use image URLs only.
|
||||||
|
# All five S3 settings are required. The bucket/CDN must serve objects publicly.
|
||||||
|
S3_BUCKET=
|
||||||
|
S3_REGION=
|
||||||
|
S3_ACCESS_KEY_ID=
|
||||||
|
S3_SECRET_ACCESS_KEY=
|
||||||
|
S3_PUBLIC_URL=
|
||||||
|
# Optional S3-compatible endpoint (uses path-style addressing).
|
||||||
|
S3_ENDPOINT=
|
||||||
|
# NSFW moderation is enabled by default. Set false (or 0/off) to disable it.
|
||||||
|
NSFW_CHECK_ENABLED=true
|
||||||
|
# Seconds to retain an upload after cleanup first detects that it is orphaned.
|
||||||
|
# Must be positive. Defaults to seven days when absent or blank.
|
||||||
|
IMAGE_UPLOAD_ORPHAN_RETENTION_SECONDS=604800
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# Local development only. These credentials match compose.rustfs.yaml defaults.
|
||||||
|
S3_BUCKET=reminder-images
|
||||||
|
S3_REGION=us-east-1
|
||||||
|
S3_ACCESS_KEY_ID=REMINDERDEV
|
||||||
|
S3_SECRET_ACCESS_KEY=reminder-rustfs-local-secret
|
||||||
|
S3_ENDPOINT=http://127.0.0.1:9000
|
||||||
|
S3_PUBLIC_URL=http://127.0.0.1:9000/reminder-images
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
CREATE TABLE image_uploads (
|
||||||
|
id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
-- Discord user ID from the authenticated session, including guild-only users.
|
||||||
|
uploader BIGINT UNSIGNED NOT NULL,
|
||||||
|
url VARCHAR(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL UNIQUE,
|
||||||
|
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
orphaned_at DATETIME DEFAULT NULL,
|
||||||
|
INDEX image_uploads_uploader (uploader),
|
||||||
|
INDEX image_uploads_orphaned_at (orphaned_at)
|
||||||
|
);
|
||||||
@@ -9,6 +9,7 @@ enum ColorScheme {
|
|||||||
type UserInfo = {
|
type UserInfo = {
|
||||||
name: string;
|
name: string;
|
||||||
patreon: boolean;
|
patreon: boolean;
|
||||||
|
image_uploads: boolean;
|
||||||
preferences: {
|
preferences: {
|
||||||
timezone: string | null;
|
timezone: string | null;
|
||||||
reset_inputs_on_create: boolean;
|
reset_inputs_on_create: boolean;
|
||||||
@@ -126,6 +127,13 @@ export const fetchUserInfo = () => ({
|
|||||||
staleTime: USER_INFO_STALE_TIME,
|
staleTime: USER_INFO_STALE_TIME,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
export const uploadImage = (file: File) =>
|
||||||
|
axios
|
||||||
|
.post<{ url: string }>("/dashboard/api/images", file, {
|
||||||
|
headers: { "Content-Type": "application/octet-stream" },
|
||||||
|
})
|
||||||
|
.then((response) => response.data.url);
|
||||||
|
|
||||||
export const patchUserInfo = () => ({
|
export const patchUserInfo = () => ({
|
||||||
mutationFn: (userInfo: UpdateUserInfo) => axios.patch(`/dashboard/api/user`, userInfo),
|
mutationFn: (userInfo: UpdateUserInfo) => axios.patch(`/dashboard/api/user`, userInfo),
|
||||||
});
|
});
|
||||||
@@ -236,7 +244,7 @@ export const fetchGuildTodos = (guild: string) => ({
|
|||||||
});
|
});
|
||||||
|
|
||||||
export const patchGuildTodo = (guild: string) => ({
|
export const patchGuildTodo = (guild: string) => ({
|
||||||
mutationFn: ({id, todo}) => axios.patch(`/dashboard/api/guild/${guild}/todos/${id}`, todo),
|
mutationFn: ({ id, todo }) => axios.patch(`/dashboard/api/guild/${guild}/todos/${id}`, todo),
|
||||||
});
|
});
|
||||||
|
|
||||||
export const postGuildTodo = (guild: string) => ({
|
export const postGuildTodo = (guild: string) => ({
|
||||||
|
|||||||
@@ -2,14 +2,22 @@ import { JSX } from "preact";
|
|||||||
import { createPortal } from "preact/compat";
|
import { createPortal } from "preact/compat";
|
||||||
|
|
||||||
type Props = {
|
type Props = {
|
||||||
setModalOpen: (open: boolean) => never;
|
setModalOpen: (open: boolean) => void;
|
||||||
title: string | JSX.Element;
|
title: string | JSX.Element;
|
||||||
onSubmitText?: string;
|
onSubmitText?: string;
|
||||||
onSubmit?: () => void;
|
onSubmit?: () => void;
|
||||||
|
onSubmitDisabled?: boolean;
|
||||||
children: string | JSX.Element | JSX.Element[] | (() => JSX.Element);
|
children: string | JSX.Element | JSX.Element[] | (() => JSX.Element);
|
||||||
};
|
};
|
||||||
|
|
||||||
export const Modal = ({ setModalOpen, title, onSubmit, onSubmitText, children }: Props) => {
|
export const Modal = ({
|
||||||
|
setModalOpen,
|
||||||
|
title,
|
||||||
|
onSubmit,
|
||||||
|
onSubmitText,
|
||||||
|
onSubmitDisabled,
|
||||||
|
children,
|
||||||
|
}: Props) => {
|
||||||
const body = document.querySelector("body");
|
const body = document.querySelector("body");
|
||||||
|
|
||||||
return createPortal(
|
return createPortal(
|
||||||
@@ -34,7 +42,11 @@ export const Modal = ({ setModalOpen, title, onSubmit, onSubmitText, children }:
|
|||||||
<section class="modal-card-body">{children}</section>
|
<section class="modal-card-body">{children}</section>
|
||||||
{onSubmit && (
|
{onSubmit && (
|
||||||
<footer class="modal-card-foot">
|
<footer class="modal-card-foot">
|
||||||
<button class="button is-success" onClick={onSubmit}>
|
<button
|
||||||
|
class="button is-success"
|
||||||
|
onClick={onSubmit}
|
||||||
|
disabled={onSubmitDisabled}
|
||||||
|
>
|
||||||
{onSubmitText || "Save"}
|
{onSubmitText || "Save"}
|
||||||
</button>
|
</button>
|
||||||
<button
|
<button
|
||||||
|
|||||||
@@ -1,13 +1,65 @@
|
|||||||
import { Modal } from "../Modal";
|
import { Modal } from "../Modal";
|
||||||
import { useState } from "preact/hooks";
|
import { useEffect, useRef, useState } from "preact/hooks";
|
||||||
|
import { useQuery } from "react-query";
|
||||||
|
import axios from "axios";
|
||||||
|
import { fetchUserInfo, uploadImage } from "../../api";
|
||||||
|
|
||||||
|
const useImageDrop = (enabled: boolean, onFiles: (files: File[]) => void, busy = false) => {
|
||||||
|
const [dragging, setDragging] = useState(false);
|
||||||
|
const dragDepth = useRef(0);
|
||||||
|
const acceptsDrag = (event: DragEvent) =>
|
||||||
|
enabled && event.dataTransfer?.types.includes("Files");
|
||||||
|
|
||||||
|
return {
|
||||||
|
dragging,
|
||||||
|
handlers: {
|
||||||
|
onDragEnter: (event: DragEvent) => {
|
||||||
|
if (!acceptsDrag(event)) return;
|
||||||
|
event.preventDefault();
|
||||||
|
event.stopPropagation();
|
||||||
|
dragDepth.current += 1;
|
||||||
|
if (!busy) setDragging(true);
|
||||||
|
},
|
||||||
|
onDragOver: (event: DragEvent) => {
|
||||||
|
if (!acceptsDrag(event)) return;
|
||||||
|
event.preventDefault();
|
||||||
|
event.stopPropagation();
|
||||||
|
event.dataTransfer.dropEffect = busy ? "none" : "copy";
|
||||||
|
},
|
||||||
|
onDragLeave: () => {
|
||||||
|
dragDepth.current = Math.max(0, dragDepth.current - 1);
|
||||||
|
if (dragDepth.current === 0) setDragging(false);
|
||||||
|
},
|
||||||
|
onDrop: (event: DragEvent) => {
|
||||||
|
if (!acceptsDrag(event)) return;
|
||||||
|
event.preventDefault();
|
||||||
|
event.stopPropagation();
|
||||||
|
dragDepth.current = 0;
|
||||||
|
setDragging(false);
|
||||||
|
if (!busy) onFiles(Array.from(event.dataTransfer.files));
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
export const ImagePicker = ({ alt, url, setImage, ...props }) => {
|
export const ImagePicker = ({ alt, url, setImage, ...props }) => {
|
||||||
const [modalOpen, setModalOpen] = useState(false);
|
const [modalOpen, setModalOpen] = useState(false);
|
||||||
|
const [droppedFiles, setDroppedFiles] = useState<File[] | null>(null);
|
||||||
|
const { data: userInfo } = useQuery(fetchUserInfo());
|
||||||
|
const uploadsEnabled = userInfo?.image_uploads === true;
|
||||||
|
const { dragging, handlers } = useImageDrop(uploadsEnabled, (files) => {
|
||||||
|
setDroppedFiles(files);
|
||||||
|
setModalOpen(true);
|
||||||
|
});
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<a
|
<a
|
||||||
|
{...handlers}
|
||||||
|
class={dragging ? "image-picker-dragging" : ""}
|
||||||
|
title={uploadsEnabled ? "Click to edit or drop an image to upload" : undefined}
|
||||||
onClick={() => {
|
onClick={() => {
|
||||||
|
setDroppedFiles(null);
|
||||||
setModalOpen(true);
|
setModalOpen(true);
|
||||||
}}
|
}}
|
||||||
role={"button"}
|
role={"button"}
|
||||||
@@ -18,14 +70,64 @@ export const ImagePicker = ({ alt, url, setImage, ...props }) => {
|
|||||||
<ImagePickerModal
|
<ImagePickerModal
|
||||||
setModalOpen={setModalOpen}
|
setModalOpen={setModalOpen}
|
||||||
setImage={setImage}
|
setImage={setImage}
|
||||||
|
url={url}
|
||||||
|
uploadsEnabled={uploadsEnabled}
|
||||||
|
droppedFiles={droppedFiles}
|
||||||
></ImagePickerModal>
|
></ImagePickerModal>
|
||||||
)}
|
)}
|
||||||
</>
|
</>
|
||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
const ImagePickerModal = ({ setModalOpen, setImage }) => {
|
const ImagePickerModal = ({ setModalOpen, setImage, url, uploadsEnabled, droppedFiles }) => {
|
||||||
const [value, setValue] = useState("");
|
const [value, setValue] = useState(url || "");
|
||||||
|
const [uploading, setUploading] = useState(false);
|
||||||
|
const [error, setError] = useState("");
|
||||||
|
const [blocked, setBlocked] = useState(false);
|
||||||
|
const fileInput = useRef<HTMLInputElement>(null);
|
||||||
|
const uploadInProgress = useRef(false);
|
||||||
|
|
||||||
|
const upload = async (files: File[]) => {
|
||||||
|
setBlocked(false);
|
||||||
|
if (!uploadsEnabled || uploadInProgress.current) return;
|
||||||
|
setError("");
|
||||||
|
if (files.length !== 1) {
|
||||||
|
setError("Please upload only one image.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const file = files[0];
|
||||||
|
if (file.size > 8 * 1024 * 1024) {
|
||||||
|
setError("Images must be no larger than 8 MiB.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
uploadInProgress.current = true;
|
||||||
|
setUploading(true);
|
||||||
|
try {
|
||||||
|
setValue(await uploadImage(file));
|
||||||
|
} catch (error) {
|
||||||
|
if (axios.isAxiosError(error) && error.response?.status === 451) {
|
||||||
|
setBlocked(true);
|
||||||
|
} else {
|
||||||
|
setError(
|
||||||
|
axios.isAxiosError(error) && typeof error.response?.data?.error === "string"
|
||||||
|
? error.response.data.error
|
||||||
|
: "Could not upload image. Please try again.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
uploadInProgress.current = false;
|
||||||
|
setUploading(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
const { dragging, handlers } = useImageDrop(
|
||||||
|
uploadsEnabled,
|
||||||
|
(files) => void upload(files),
|
||||||
|
uploading,
|
||||||
|
);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (droppedFiles) void upload(droppedFiles);
|
||||||
|
}, [droppedFiles]);
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<Modal
|
<Modal
|
||||||
@@ -36,6 +138,7 @@ const ImagePickerModal = ({ setModalOpen, setImage }) => {
|
|||||||
setModalOpen(false);
|
setModalOpen(false);
|
||||||
}}
|
}}
|
||||||
onSubmitText={"Save"}
|
onSubmitText={"Save"}
|
||||||
|
onSubmitDisabled={uploading}
|
||||||
>
|
>
|
||||||
<p>
|
<p>
|
||||||
Please note: if you attach an image directly from Discord, it will not be visible in
|
Please note: if you attach an image directly from Discord, it will not be visible in
|
||||||
@@ -46,10 +149,65 @@ const ImagePickerModal = ({ setModalOpen, setImage }) => {
|
|||||||
class="input"
|
class="input"
|
||||||
id="urlInput"
|
id="urlInput"
|
||||||
placeholder="Image URL..."
|
placeholder="Image URL..."
|
||||||
|
value={value}
|
||||||
|
disabled={uploading}
|
||||||
onInput={(ev) => {
|
onInput={(ev) => {
|
||||||
setValue(ev.currentTarget.value);
|
setValue(ev.currentTarget.value);
|
||||||
}}
|
}}
|
||||||
></input>
|
></input>
|
||||||
|
{uploadsEnabled && (
|
||||||
|
<div
|
||||||
|
{...handlers}
|
||||||
|
class={`mt-4 image-upload-dropzone ${dragging ? "image-picker-dragging" : ""}`}
|
||||||
|
>
|
||||||
|
<p>or…</p>
|
||||||
|
<input
|
||||||
|
ref={fileInput}
|
||||||
|
type="file"
|
||||||
|
accept="image/png,image/jpeg,image/gif,image/webp"
|
||||||
|
hidden
|
||||||
|
disabled={uploading}
|
||||||
|
aria-label="Choose an image to upload"
|
||||||
|
onChange={(event) => {
|
||||||
|
const file = event.currentTarget.files?.[0];
|
||||||
|
event.currentTarget.value = "";
|
||||||
|
if (file) void upload([file]);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
class={`button is-info ${uploading ? "is-loading" : ""}`}
|
||||||
|
disabled={uploading}
|
||||||
|
onClick={() => fileInput.current?.click()}
|
||||||
|
>
|
||||||
|
Upload image
|
||||||
|
</button>
|
||||||
|
<p class="help">Or drag and drop an image here.</p>
|
||||||
|
<p class="help">
|
||||||
|
Static PNG, JPEG, GIF or WebP, up to 8 MiB. All uploads are converted to
|
||||||
|
PNGs.
|
||||||
|
</p>
|
||||||
|
{blocked && (
|
||||||
|
<>
|
||||||
|
<p class="help" style="color: red;">
|
||||||
|
This image was flagged as potentially NSFW. Unfortunately, due to
|
||||||
|
the UK's Online Safety Act, we cannot handle explicit content.
|
||||||
|
Please defer to using Imgur or another image storage platform who
|
||||||
|
have the resources to deal with idiotic laws.
|
||||||
|
</p>
|
||||||
|
<p class="help" style="color: red;">
|
||||||
|
The image you attempted to upload has not been stored.
|
||||||
|
</p>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{uploading && <p role="status">Uploading image…</p>}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{error && (
|
||||||
|
<p class="help is-danger" role="alert">
|
||||||
|
{error}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
</Modal>
|
</Modal>
|
||||||
);
|
);
|
||||||
};
|
};
|
||||||
@@ -34,3 +34,24 @@ div.reminderContent {
|
|||||||
background-color: #f5f5f5;
|
background-color: #f5f5f5;
|
||||||
border-radius: 8px;
|
border-radius: 8px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.image-upload-dropzone {
|
||||||
|
border: 2px dashed #b5b5b5;
|
||||||
|
border-radius: 4px;
|
||||||
|
padding: 1rem;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.image-picker-dragging {
|
||||||
|
outline: 3px solid #3e8ed0;
|
||||||
|
outline-offset: 2px;
|
||||||
|
|
||||||
|
img {
|
||||||
|
opacity: 0.6;
|
||||||
|
}
|
||||||
|
|
||||||
|
&.image-upload-dropzone {
|
||||||
|
background-color: #eff5fb;
|
||||||
|
border-color: #3e8ed0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
endpoint=http://rustfs:9000
|
||||||
|
|
||||||
|
if ! aws --endpoint-url "$endpoint" s3api head-bucket --bucket "$S3_BUCKET" >/dev/null 2>&1; then
|
||||||
|
if [ "$AWS_DEFAULT_REGION" = us-east-1 ]; then
|
||||||
|
aws --endpoint-url "$endpoint" s3api create-bucket --bucket "$S3_BUCKET"
|
||||||
|
else
|
||||||
|
aws --endpoint-url "$endpoint" s3api create-bucket --bucket "$S3_BUCKET" \
|
||||||
|
--create-bucket-configuration "LocationConstraint=$AWS_DEFAULT_REGION"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Dashboard images need public object reads. Listing and writes remain authenticated.
|
||||||
|
cat > /tmp/public-images-policy.json <<EOF
|
||||||
|
{
|
||||||
|
"Version": "2012-10-17",
|
||||||
|
"Statement": [{
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Principal": "*",
|
||||||
|
"Action": "s3:GetObject",
|
||||||
|
"Resource": "arn:aws:s3:::$S3_BUCKET/uploads/*"
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
aws --endpoint-url "$endpoint" s3api put-bucket-policy --bucket "$S3_BUCKET" \
|
||||||
|
--policy file:///tmp/public-images-policy.json
|
||||||
|
echo "RustFS development bucket ready: $S3_BUCKET"
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Split an image into two lossless, random-looking PNG fixture shares."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from io import BytesIO
|
||||||
|
from pathlib import Path
|
||||||
|
import secrets
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
|
||||||
|
def split_image(input_path: Path, output_dir: Path, index: int | None = None) -> tuple[Path, Path]:
|
||||||
|
if index is None:
|
||||||
|
index = 1
|
||||||
|
while any((output_dir / f"image_{index}{part}.png").exists() for part in ("a", "b")):
|
||||||
|
index += 1
|
||||||
|
if index < 1:
|
||||||
|
raise ValueError("Image index must be a positive integer")
|
||||||
|
|
||||||
|
paths = (
|
||||||
|
output_dir / f"image_{index}a.png",
|
||||||
|
output_dir / f"image_{index}b.png",
|
||||||
|
)
|
||||||
|
for path in paths:
|
||||||
|
if path.exists():
|
||||||
|
raise FileExistsError(f"Refusing to overwrite {path}; choose another --index")
|
||||||
|
|
||||||
|
# RGBA shares preserve every decoded pixel, including transparency. For animated
|
||||||
|
# inputs, use the first frame, matching the dashboard's static image upload behavior.
|
||||||
|
with Image.open(input_path) as source:
|
||||||
|
image = source.convert("RGBA")
|
||||||
|
pixels = image.tobytes()
|
||||||
|
first = secrets.token_bytes(len(pixels))
|
||||||
|
second = bytes(pixel ^ mask for pixel, mask in zip(pixels, first))
|
||||||
|
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
for path, share in zip(paths, (first, second)):
|
||||||
|
# Exclusive creation also prevents overwriting a file created during splitting.
|
||||||
|
encoded = BytesIO()
|
||||||
|
Image.frombytes("RGBA", image.size, share).save(encoded, format="PNG")
|
||||||
|
with path.open("xb") as output:
|
||||||
|
output.write(encoded.getvalue())
|
||||||
|
return paths
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("input_image", type=Path)
|
||||||
|
parser.add_argument("--output-dir", type=Path, default=Path("."))
|
||||||
|
parser.add_argument("--index", type=int, help="Fixture number (default: first unused positive index)")
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
paths = split_image(args.input_image, args.output_dir, args.index)
|
||||||
|
except (OSError, ValueError) as error:
|
||||||
|
parser.exit(1, f"Could not split image: {error}\n")
|
||||||
|
for path in paths:
|
||||||
|
print(path)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,165 @@
|
|||||||
|
use std::env;
|
||||||
|
|
||||||
|
use log::{info, warn};
|
||||||
|
use sqlx::{Acquire, MySql, Pool};
|
||||||
|
|
||||||
|
use crate::{web::image_uploads::ImageUploadService, Error};
|
||||||
|
|
||||||
|
const DEFAULT_RETENTION_SECONDS: u32 = 7 * 24 * 60 * 60;
|
||||||
|
const BATCH_SIZE: u32 = 1000;
|
||||||
|
const IMAGE_COLUMNS: &str =
|
||||||
|
"avatar, embed_image_url, embed_thumbnail_url, embed_author_url, embed_footer_url";
|
||||||
|
|
||||||
|
fn binary_image_columns() -> String {
|
||||||
|
// Existing URL columns and image_uploads use different collations. S3 URLs must
|
||||||
|
// also match case-sensitively, so compare their bytes rather than text collation.
|
||||||
|
IMAGE_COLUMNS
|
||||||
|
.split(", ")
|
||||||
|
.map(|column| format!("BINARY {column}"))
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join(", ")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn retention_seconds(value: Option<&str>) -> Result<u32, Error> {
|
||||||
|
match value.map(str::trim).filter(|value| !value.is_empty()) {
|
||||||
|
None => Ok(DEFAULT_RETENTION_SECONDS),
|
||||||
|
Some(value) => value.parse::<u32>().ok().filter(|value| *value > 0).ok_or_else(|| {
|
||||||
|
"IMAGE_UPLOAD_ORPHAN_RETENTION_SECONDS must be a positive integer of seconds".into()
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only keys generated by the upload route in the configured bucket may be deleted.
|
||||||
|
fn object_key<'a>(public_url: &str, url: &'a str) -> Option<&'a str> {
|
||||||
|
let key = url.strip_prefix(public_url)?.strip_prefix('/')?;
|
||||||
|
let name = key.strip_prefix("uploads/")?.strip_suffix(".png")?;
|
||||||
|
(name.len() == 64
|
||||||
|
&& name.bytes().all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)))
|
||||||
|
.then_some(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn mark_orphans(pool: &Pool<MySql>) -> Result<(), Error> {
|
||||||
|
let columns = binary_image_columns();
|
||||||
|
sqlx::query(&format!(
|
||||||
|
"UPDATE image_uploads AS upload SET orphaned_at = CASE
|
||||||
|
WHEN NOT EXISTS (SELECT 1 FROM reminders
|
||||||
|
WHERE BINARY upload.url IN ({columns}))
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM reminder_template
|
||||||
|
WHERE BINARY upload.url IN ({columns}))
|
||||||
|
THEN COALESCE(upload.orphaned_at, UTC_TIMESTAMP(6)) ELSE NULL END"
|
||||||
|
))
|
||||||
|
.execute(pool)
|
||||||
|
.await?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) async fn run(pool: &Pool<MySql>) -> Result<(), Error> {
|
||||||
|
let retention =
|
||||||
|
retention_seconds(env::var("IMAGE_UPLOAD_ORPHAN_RETENTION_SECONDS").ok().as_deref())?;
|
||||||
|
let Some(storage) = ImageUploadService::for_cleanup()? else {
|
||||||
|
info!("Image cleanup skipped: S3 uploads are not configured");
|
||||||
|
return Ok(());
|
||||||
|
};
|
||||||
|
cleanup(pool, &storage, retention).await
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn cleanup(
|
||||||
|
pool: &Pool<MySql>,
|
||||||
|
storage: &ImageUploadService,
|
||||||
|
retention: u32,
|
||||||
|
) -> Result<(), Error> {
|
||||||
|
mark_orphans(pool).await?;
|
||||||
|
let candidates: Vec<u64> = sqlx::query_scalar(
|
||||||
|
"SELECT id FROM image_uploads
|
||||||
|
WHERE orphaned_at <= UTC_TIMESTAMP(6) - INTERVAL ? SECOND
|
||||||
|
ORDER BY orphaned_at, id LIMIT ?",
|
||||||
|
)
|
||||||
|
.bind(retention)
|
||||||
|
.bind(BATCH_SIZE)
|
||||||
|
.fetch_all(pool)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
let mut deleted = 0;
|
||||||
|
let mut failed = 0;
|
||||||
|
for id in candidates {
|
||||||
|
match delete_orphan(pool, storage, retention, id).await {
|
||||||
|
Ok(true) => deleted += 1,
|
||||||
|
Ok(false) => {}
|
||||||
|
Err(error) => {
|
||||||
|
failed += 1;
|
||||||
|
warn!("Could not clean up image upload {}: {}", id, error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
info!("Image cleanup deleted {} uploads; {} failed", deleted, failed);
|
||||||
|
if failed > 0 {
|
||||||
|
return Err(format!("Image cleanup failed for {failed} uploads; retained for retry").into());
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn delete_orphan(
|
||||||
|
pool: &Pool<MySql>,
|
||||||
|
storage: &ImageUploadService,
|
||||||
|
retention: u32,
|
||||||
|
id: u64,
|
||||||
|
) -> Result<bool, Error> {
|
||||||
|
let mut connection = pool.acquire().await?;
|
||||||
|
sqlx::query("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ")
|
||||||
|
.execute(&mut *connection)
|
||||||
|
.await?;
|
||||||
|
let mut transaction = connection.begin().await?;
|
||||||
|
// Serialize overlapping cleanup jobs and recheck the retention time under the row lock.
|
||||||
|
let url: Option<String> = sqlx::query_scalar(
|
||||||
|
"SELECT url FROM image_uploads
|
||||||
|
WHERE id = ? AND orphaned_at <= UTC_TIMESTAMP(6) - INTERVAL ? SECOND
|
||||||
|
FOR UPDATE",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.bind(retention)
|
||||||
|
.fetch_optional(&mut *transaction)
|
||||||
|
.await?;
|
||||||
|
let Some(url) = url else {
|
||||||
|
transaction.rollback().await?;
|
||||||
|
return Ok(false);
|
||||||
|
};
|
||||||
|
let key = object_key(&storage.public_url, &url)
|
||||||
|
.ok_or("Upload URL does not match a generated key in the configured S3 public URL")?;
|
||||||
|
|
||||||
|
for table in ["reminders", "reminder_template"] {
|
||||||
|
// Locking reads see current committed references, rather than a stale snapshot.
|
||||||
|
// InnoDB's REPEATABLE READ next-key locks also prevent a reference from
|
||||||
|
// being inserted while the object is being removed. Keep this transaction short:
|
||||||
|
// it covers one S3 request with the upload service's 30-second timeout.
|
||||||
|
let columns = binary_image_columns();
|
||||||
|
let referenced: Option<u32> = sqlx::query_scalar(&format!(
|
||||||
|
"SELECT id FROM {table} WHERE BINARY ? IN ({columns}) LIMIT 1 FOR UPDATE"
|
||||||
|
))
|
||||||
|
.bind(&url)
|
||||||
|
.fetch_optional(&mut *transaction)
|
||||||
|
.await?;
|
||||||
|
if referenced.is_some() {
|
||||||
|
sqlx::query("UPDATE image_uploads SET orphaned_at = NULL WHERE id = ?")
|
||||||
|
.bind(id)
|
||||||
|
.execute(&mut *transaction)
|
||||||
|
.await?;
|
||||||
|
transaction.commit().await?;
|
||||||
|
return Ok(false);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let response = storage.bucket.delete_object(key).await?;
|
||||||
|
// A missing object is already cleaned up, including a retry after a DB commit failure.
|
||||||
|
if !(200..300).contains(&response.status_code()) && response.status_code() != 404 {
|
||||||
|
return Err(format!("S3 deletion returned HTTP {}", response.status_code()).into());
|
||||||
|
}
|
||||||
|
sqlx::query("DELETE FROM image_uploads WHERE id = ?")
|
||||||
|
.bind(id)
|
||||||
|
.execute(&mut *transaction)
|
||||||
|
.await?;
|
||||||
|
transaction.commit().await?;
|
||||||
|
Ok(true)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests;
|
||||||
@@ -0,0 +1,258 @@
|
|||||||
|
use std::{sync::Arc, time::Duration};
|
||||||
|
|
||||||
|
use chrono::NaiveDateTime;
|
||||||
|
use s3::{creds::Credentials, Bucket, Region};
|
||||||
|
use tokio::{
|
||||||
|
io::{AsyncReadExt, AsyncWriteExt},
|
||||||
|
net::TcpListener,
|
||||||
|
sync::Semaphore,
|
||||||
|
task::JoinHandle,
|
||||||
|
};
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
const BASE_URL: &str = "https://images.example.com/bucket";
|
||||||
|
|
||||||
|
fn upload_url(number: u64) -> String {
|
||||||
|
format!("{BASE_URL}/uploads/{number:064x}.png")
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn upload(pool: &Pool<MySql>, number: u64) -> u64 {
|
||||||
|
sqlx::query("INSERT INTO image_uploads (uploader, url) VALUES (123, ?)")
|
||||||
|
.bind(upload_url(number))
|
||||||
|
.execute(pool)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.last_insert_id()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn orphaned_at(pool: &Pool<MySql>, id: u64) -> Option<NaiveDateTime> {
|
||||||
|
sqlx::query_scalar("SELECT orphaned_at FROM image_uploads WHERE id = ?")
|
||||||
|
.bind(id)
|
||||||
|
.fetch_one(pool)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn expire(pool: &Pool<MySql>, id: u64) {
|
||||||
|
sqlx::query(
|
||||||
|
"UPDATE image_uploads SET orphaned_at = UTC_TIMESTAMP(6) - INTERVAL 2 DAY WHERE id = ?",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.execute(pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn reference_tables(pool: &Pool<MySql>) -> (u64, u64) {
|
||||||
|
let guild = sqlx::query("INSERT INTO guilds (guild) VALUES (123)")
|
||||||
|
.execute(pool)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.last_insert_id();
|
||||||
|
let channel = sqlx::query("INSERT INTO channels (channel, guild_id) VALUES (123, ?)")
|
||||||
|
.bind(guild)
|
||||||
|
.execute(pool)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.last_insert_id();
|
||||||
|
(guild, channel)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn storage(status: u16) -> (ImageUploadService, JoinHandle<String>) {
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let endpoint = format!("http://{}", listener.local_addr().unwrap());
|
||||||
|
let task = tokio::spawn(async move {
|
||||||
|
let (mut socket, _) =
|
||||||
|
tokio::time::timeout(Duration::from_secs(5), listener.accept()).await.unwrap().unwrap();
|
||||||
|
let mut received = Vec::new();
|
||||||
|
let mut buffer = [0; 4096];
|
||||||
|
while !received.windows(4).any(|window| window == b"\r\n\r\n") {
|
||||||
|
let count = socket.read(&mut buffer).await.unwrap();
|
||||||
|
assert!(count > 0);
|
||||||
|
received.extend_from_slice(&buffer[..count]);
|
||||||
|
}
|
||||||
|
socket
|
||||||
|
.write_all(
|
||||||
|
format!("HTTP/1.1 {status} Test\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
|
||||||
|
.as_bytes(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
String::from_utf8(received).unwrap()
|
||||||
|
});
|
||||||
|
let credentials = Credentials::new(Some("test"), Some("test"), None, None, None).unwrap();
|
||||||
|
let mut bucket =
|
||||||
|
Bucket::new("images", Region::Custom { region: "local".into(), endpoint }, credentials)
|
||||||
|
.unwrap()
|
||||||
|
.with_path_style();
|
||||||
|
bucket.set_request_timeout(Some(Duration::from_secs(2)));
|
||||||
|
(
|
||||||
|
ImageUploadService {
|
||||||
|
bucket,
|
||||||
|
public_url: BASE_URL.into(),
|
||||||
|
model: None,
|
||||||
|
slots: Arc::new(Semaphore::new(2)),
|
||||||
|
},
|
||||||
|
task,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn retention_defaults_and_rejects_invalid_settings() {
|
||||||
|
for value in [None, Some(""), Some(" ")] {
|
||||||
|
assert_eq!(retention_seconds(value).unwrap(), 604800);
|
||||||
|
}
|
||||||
|
assert_eq!(retention_seconds(Some(" 3600 ")).unwrap(), 3600);
|
||||||
|
for value in ["0", "-1", "1.5", "forever", "4294967296"] {
|
||||||
|
assert!(retention_seconds(Some(value)).is_err());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn restricts_deletion_to_generated_keys_in_the_configured_url() {
|
||||||
|
assert_eq!(
|
||||||
|
object_key(BASE_URL, &upload_url(1)),
|
||||||
|
Some(format!("uploads/{:064x}.png", 1).as_str())
|
||||||
|
);
|
||||||
|
for url in [
|
||||||
|
upload_url(1).replace("images.example.com", "other.example.com"),
|
||||||
|
upload_url(1).replace("/bucket/", "/bucket-other/"),
|
||||||
|
upload_url(1).replace("/uploads/", "/private/"),
|
||||||
|
format!("{BASE_URL}/uploads/../private.png"),
|
||||||
|
format!("{BASE_URL}/uploads/{}.png", "g".repeat(64)),
|
||||||
|
format!("{}?download=true", upload_url(1)),
|
||||||
|
format!("{BASE_URL}/uploads/short.png"),
|
||||||
|
] {
|
||||||
|
assert!(object_key(BASE_URL, &url).is_none(), "{url}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn orphan_timestamp_survives_repeated_runs_and_resets_on_reuse(pool: Pool<MySql>) {
|
||||||
|
let id = upload(&pool, 1).await;
|
||||||
|
assert!(orphaned_at(&pool, id).await.is_none());
|
||||||
|
mark_orphans(&pool).await.unwrap();
|
||||||
|
let first = orphaned_at(&pool, id).await.unwrap();
|
||||||
|
mark_orphans(&pool).await.unwrap();
|
||||||
|
assert_eq!(orphaned_at(&pool, id).await, Some(first));
|
||||||
|
|
||||||
|
let (guild, _) = reference_tables(&pool).await;
|
||||||
|
let template =
|
||||||
|
sqlx::query("INSERT INTO reminder_template (guild_id, embed_image_url) VALUES (?, ?)")
|
||||||
|
.bind(guild)
|
||||||
|
.bind(upload_url(1))
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap()
|
||||||
|
.last_insert_id();
|
||||||
|
mark_orphans(&pool).await.unwrap();
|
||||||
|
assert!(orphaned_at(&pool, id).await.is_none());
|
||||||
|
sqlx::query("DELETE FROM reminder_template WHERE id = ?")
|
||||||
|
.bind(template)
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
mark_orphans(&pool).await.unwrap();
|
||||||
|
assert!(orphaned_at(&pool, id).await.unwrap() > first);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn protects_all_image_fields_in_reminders_and_templates(pool: Pool<MySql>) {
|
||||||
|
let (guild, channel) = reference_tables(&pool).await;
|
||||||
|
let mut ids = Vec::new();
|
||||||
|
for (index, column) in IMAGE_COLUMNS.split(", ").enumerate() {
|
||||||
|
for table in ["reminders", "reminder_template"] {
|
||||||
|
let number = ids.len() as u64 + 1;
|
||||||
|
ids.push(upload(&pool, number).await);
|
||||||
|
let query = if table == "reminders" {
|
||||||
|
// Archived/disabled reminders still retain their image URLs.
|
||||||
|
format!(
|
||||||
|
"INSERT INTO reminders (uid, channel_id, `utc_time`, status, enabled, {column})
|
||||||
|
VALUES ('field-{index}', {channel}, UTC_TIMESTAMP(), 'sent', 0, ?)"
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
format!("INSERT INTO reminder_template (guild_id, {column}) VALUES ({guild}, ?)")
|
||||||
|
};
|
||||||
|
sqlx::query(&query).bind(upload_url(number)).execute(&pool).await.unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let unused = upload(&pool, 100).await;
|
||||||
|
// Previously orphaned files are unmarked as soon as cleanup observes a reference.
|
||||||
|
for id in &ids {
|
||||||
|
expire(&pool, *id).await;
|
||||||
|
}
|
||||||
|
mark_orphans(&pool).await.unwrap();
|
||||||
|
for id in ids {
|
||||||
|
assert!(orphaned_at(&pool, id).await.is_none());
|
||||||
|
}
|
||||||
|
assert!(orphaned_at(&pool, unused).await.is_some());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn only_expired_orphans_are_deleted_and_rows_removed_after_success(pool: Pool<MySql>) {
|
||||||
|
let old = upload(&pool, 1).await;
|
||||||
|
let fresh = upload(&pool, 2).await;
|
||||||
|
expire(&pool, old).await;
|
||||||
|
let (storage, request) = storage(204).await;
|
||||||
|
cleanup(&pool, &storage, 86400).await.unwrap();
|
||||||
|
assert!(request.await.unwrap().starts_with(&format!("DELETE /images/uploads/{:064x}.png ", 1)));
|
||||||
|
let ids: Vec<u64> = sqlx::query_scalar("SELECT id FROM image_uploads ORDER BY id")
|
||||||
|
.fetch_all(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(ids, vec![fresh]);
|
||||||
|
assert!(orphaned_at(&pool, fresh).await.is_some());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn failed_deletions_keep_the_original_timestamp_and_retry_missing_objects(pool: Pool<MySql>) {
|
||||||
|
let id = upload(&pool, 1).await;
|
||||||
|
expire(&pool, id).await;
|
||||||
|
let original = orphaned_at(&pool, id).await;
|
||||||
|
let (failed_storage, failed_request) = storage(403).await;
|
||||||
|
assert!(cleanup(&pool, &failed_storage, 86400).await.is_err());
|
||||||
|
failed_request.await.unwrap();
|
||||||
|
assert_eq!(orphaned_at(&pool, id).await, original);
|
||||||
|
|
||||||
|
let (retry_storage, retry_request) = storage(404).await;
|
||||||
|
cleanup(&pool, &retry_storage, 86400).await.unwrap();
|
||||||
|
retry_request.await.unwrap();
|
||||||
|
let count: i64 =
|
||||||
|
sqlx::query_scalar("SELECT COUNT(*) FROM image_uploads").fetch_one(&pool).await.unwrap();
|
||||||
|
assert_eq!(count, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn rechecks_references_before_deleting_a_candidate(pool: Pool<MySql>) {
|
||||||
|
let id = upload(&pool, 1).await;
|
||||||
|
expire(&pool, id).await;
|
||||||
|
let (guild, _) = reference_tables(&pool).await;
|
||||||
|
sqlx::query("INSERT INTO reminder_template (guild_id, avatar) VALUES (?, ?)")
|
||||||
|
.bind(guild)
|
||||||
|
.bind(upload_url(1))
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let (storage, request) = storage(204).await;
|
||||||
|
assert!(!delete_orphan(&pool, &storage, 86400, id).await.unwrap());
|
||||||
|
assert!(orphaned_at(&pool, id).await.is_none());
|
||||||
|
request.abort();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn foreign_urls_are_retained_without_contacting_s3(pool: Pool<MySql>) {
|
||||||
|
let id = upload(&pool, 1).await;
|
||||||
|
sqlx::query(
|
||||||
|
"UPDATE image_uploads SET url = 'https://elsewhere.example.com/file.png' WHERE id = ?",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
expire(&pool, id).await;
|
||||||
|
let (storage, request) = storage(204).await;
|
||||||
|
assert!(cleanup(&pool, &storage, 86400).await.is_err());
|
||||||
|
assert!(orphaned_at(&pool, id).await.is_some());
|
||||||
|
request.abort();
|
||||||
|
}
|
||||||
@@ -9,6 +9,7 @@ mod consts;
|
|||||||
mod event_handlers;
|
mod event_handlers;
|
||||||
#[cfg(not(test))]
|
#[cfg(not(test))]
|
||||||
mod hooks;
|
mod hooks;
|
||||||
|
mod image_cleanup;
|
||||||
mod interval_parser;
|
mod interval_parser;
|
||||||
mod metrics;
|
mod metrics;
|
||||||
#[cfg(not(test))]
|
#[cfg(not(test))]
|
||||||
@@ -121,6 +122,7 @@ async fn _main(tx: Sender<()>) -> Result<(), Box<dyn StdError + Send + Sync>> {
|
|||||||
Pool::connect(&env::var("DATABASE_URL").expect("No database URL provided")).await.unwrap();
|
Pool::connect(&env::var("DATABASE_URL").expect("No database URL provided")).await.unwrap();
|
||||||
|
|
||||||
match cmd_word {
|
match cmd_word {
|
||||||
|
Some("clean-images") => image_cleanup::run(&database).await,
|
||||||
Some("clean") => {
|
Some("clean") => {
|
||||||
let sent_clean_age = env::var("SENT_CLEAN_AGE").expect("No SENT_CLEAN_AGE provided");
|
let sent_clean_age = env::var("SENT_CLEAN_AGE").expect("No SENT_CLEAN_AGE provided");
|
||||||
if sent_clean_age.is_empty() {
|
if sent_clean_age.is_empty() {
|
||||||
|
|||||||
@@ -0,0 +1,298 @@
|
|||||||
|
use std::{env, io::Cursor, sync::Arc};
|
||||||
|
|
||||||
|
use image::{ImageFormat, ImageReader, Limits};
|
||||||
|
use log::{info, warn};
|
||||||
|
use nsfw::model::{Classification, Metric};
|
||||||
|
use s3::{creds::Credentials, Bucket, Region};
|
||||||
|
use tokio::sync::Semaphore;
|
||||||
|
|
||||||
|
pub const MAX_UPLOAD_BYTES: u64 = 8 * 1024 * 1024;
|
||||||
|
const MAX_IMAGE_PIXELS: u64 = 16_000_000;
|
||||||
|
const NSFW_THRESHOLD: f32 = 0.5;
|
||||||
|
const MODEL_BYTES: &[u8] =
|
||||||
|
include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/assets/nsfw/model.onnx"));
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod fixture_tests;
|
||||||
|
|
||||||
|
// An empty configuration keeps the dashboard's existing URL-only behavior.
|
||||||
|
pub struct ImageUploads(pub Option<Arc<ImageUploadService>>);
|
||||||
|
|
||||||
|
pub struct ImageUploadService {
|
||||||
|
pub bucket: Box<Bucket>,
|
||||||
|
pub public_url: String,
|
||||||
|
pub model: Option<nsfw::Model>,
|
||||||
|
pub slots: Arc<Semaphore>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ImageUploads {
|
||||||
|
pub fn from_env() -> Self {
|
||||||
|
let names =
|
||||||
|
["S3_BUCKET", "S3_REGION", "S3_ACCESS_KEY_ID", "S3_SECRET_ACCESS_KEY", "S3_PUBLIC_URL"];
|
||||||
|
let values = names.map(|name| env::var(name).unwrap_or_default());
|
||||||
|
Self::from_settings(&values)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn from_settings(values: &[String; 5]) -> Self {
|
||||||
|
if values.iter().all(|value| value.trim().is_empty()) {
|
||||||
|
return Self(None);
|
||||||
|
}
|
||||||
|
if values.iter().any(|value| value.trim().is_empty()) {
|
||||||
|
warn!("Image uploads disabled: all S3 upload settings must be supplied");
|
||||||
|
return Self(None);
|
||||||
|
}
|
||||||
|
|
||||||
|
let check_nsfw = nsfw_check_enabled(env::var("NSFW_CHECK_ENABLED").ok().as_deref());
|
||||||
|
match ImageUploadService::new(values, check_nsfw) {
|
||||||
|
Ok(service) => {
|
||||||
|
info!("Dashboard image uploads enabled");
|
||||||
|
Self(Some(Arc::new(service)))
|
||||||
|
}
|
||||||
|
Err(error) => {
|
||||||
|
warn!("Image uploads disabled: {}", error);
|
||||||
|
Self(None)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ImageUploadService {
|
||||||
|
// Cleanup uses the same storage configuration without loading the moderation model.
|
||||||
|
pub(crate) fn for_cleanup() -> Result<Option<Self>, Box<dyn std::error::Error + Send + Sync>> {
|
||||||
|
let values =
|
||||||
|
["S3_BUCKET", "S3_REGION", "S3_ACCESS_KEY_ID", "S3_SECRET_ACCESS_KEY", "S3_PUBLIC_URL"]
|
||||||
|
.map(|name| env::var(name).unwrap_or_default());
|
||||||
|
if values.iter().all(|value| value.trim().is_empty()) {
|
||||||
|
return Ok(None);
|
||||||
|
}
|
||||||
|
if values.iter().any(|value| value.trim().is_empty()) {
|
||||||
|
return Err("Image cleanup requires all S3 upload settings".into());
|
||||||
|
}
|
||||||
|
Self::new(&values, false).map(Some).map_err(|error| error.to_string().into())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn new(values: &[String; 5], check_nsfw: bool) -> Result<Self, Box<dyn std::error::Error>> {
|
||||||
|
let url = reqwest::Url::parse(&values[4])?;
|
||||||
|
let public_url = url.as_str().trim_end_matches('/').to_string();
|
||||||
|
if !matches!(url.scheme(), "https" | "http")
|
||||||
|
|| url.host_str().is_none()
|
||||||
|
|| !url.username().is_empty()
|
||||||
|
|| url.password().is_some()
|
||||||
|
|| url.query().is_some()
|
||||||
|
|| url.fragment().is_some()
|
||||||
|
|| public_url.len() > 400
|
||||||
|
{
|
||||||
|
return Err("S3_PUBLIC_URL must be an HTTP(S) base URL of at most 400 bytes".into());
|
||||||
|
}
|
||||||
|
|
||||||
|
let endpoint = env::var("S3_ENDPOINT").ok().filter(|value| !value.trim().is_empty());
|
||||||
|
let region = match &endpoint {
|
||||||
|
Some(endpoint) => {
|
||||||
|
Region::Custom { region: values[1].clone(), endpoint: endpoint.clone() }
|
||||||
|
}
|
||||||
|
None => values[1].parse()?,
|
||||||
|
};
|
||||||
|
|
||||||
|
let credentials = Credentials::new(Some(&values[2]), Some(&values[3]), None, None, None)?;
|
||||||
|
let mut bucket = Bucket::new(&values[0], region, credentials)?;
|
||||||
|
if endpoint.is_some() {
|
||||||
|
bucket = bucket.with_path_style();
|
||||||
|
}
|
||||||
|
|
||||||
|
bucket.set_request_timeout(Some(std::time::Duration::from_secs(30)));
|
||||||
|
let model = if check_nsfw {
|
||||||
|
Some(nsfw::create_model(MODEL_BYTES)?)
|
||||||
|
} else {
|
||||||
|
info!("NSFW checks disabled for dashboard image uploads");
|
||||||
|
None
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(Self { bucket, public_url, model, slots: Arc::new(Semaphore::new(2)) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn nsfw_check_enabled(value: Option<&str>) -> bool {
|
||||||
|
match value.map(str::trim).map(str::to_ascii_lowercase).as_deref() {
|
||||||
|
None | Some("" | "true" | "1" | "on") => true,
|
||||||
|
Some("false" | "0" | "off") => false,
|
||||||
|
Some(_) => {
|
||||||
|
warn!("Unrecognized NSFW_CHECK_ENABLED value; keeping NSFW checks enabled");
|
||||||
|
true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, PartialEq)]
|
||||||
|
pub enum ImageError {
|
||||||
|
InvalidImage,
|
||||||
|
Nsfw,
|
||||||
|
ModerationFailed,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn probable_nsfw(predictions: &[Classification]) -> Result<bool, ImageError> {
|
||||||
|
if predictions.len() != 5
|
||||||
|
|| predictions.iter().any(|prediction| {
|
||||||
|
!prediction.score.is_finite() || !(0.0..=1.0).contains(&prediction.score)
|
||||||
|
})
|
||||||
|
{
|
||||||
|
return Err(ImageError::ModerationFailed);
|
||||||
|
}
|
||||||
|
let score: f32 = predictions
|
||||||
|
.iter()
|
||||||
|
.filter(|prediction| {
|
||||||
|
matches!(prediction.metric, Metric::Porn | Metric::Hentai | Metric::Sexy)
|
||||||
|
})
|
||||||
|
.map(|prediction| prediction.score)
|
||||||
|
.sum();
|
||||||
|
Ok(score >= NSFW_THRESHOLD)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Publish only the decoded pixels, which are moderated when NSFW checks are enabled.
|
||||||
|
// Static PNGs strip metadata and prevent unchecked animation frames from being published.
|
||||||
|
fn decode_image(bytes: &[u8]) -> Result<image::RgbaImage, ImageError> {
|
||||||
|
let mut reader = ImageReader::new(Cursor::new(bytes))
|
||||||
|
.with_guessed_format()
|
||||||
|
.map_err(|_| ImageError::InvalidImage)?;
|
||||||
|
if !matches!(
|
||||||
|
reader.format(),
|
||||||
|
Some(ImageFormat::Jpeg | ImageFormat::Png | ImageFormat::Gif | ImageFormat::WebP)
|
||||||
|
) {
|
||||||
|
return Err(ImageError::InvalidImage);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut limits = Limits::default();
|
||||||
|
limits.max_image_width = Some(8192);
|
||||||
|
limits.max_image_height = Some(8192);
|
||||||
|
limits.max_alloc = Some(128 * 1024 * 1024);
|
||||||
|
reader.limits(limits);
|
||||||
|
|
||||||
|
let decoder = reader.into_decoder().map_err(|_| ImageError::InvalidImage)?;
|
||||||
|
|
||||||
|
use image::ImageDecoder;
|
||||||
|
let (width, height) = decoder.dimensions();
|
||||||
|
if width == 0 || height == 0 || u64::from(width) * u64::from(height) > MAX_IMAGE_PIXELS {
|
||||||
|
return Err(ImageError::InvalidImage);
|
||||||
|
}
|
||||||
|
|
||||||
|
image::DynamicImage::from_decoder(decoder)
|
||||||
|
.map(|image| image.to_rgba8())
|
||||||
|
.map_err(|_| ImageError::InvalidImage)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn prepare_image(model: Option<&nsfw::Model>, bytes: &[u8]) -> Result<Vec<u8>, ImageError> {
|
||||||
|
let image = decode_image(bytes)?;
|
||||||
|
// Flatten transparency so the classifier sees exactly the published image.
|
||||||
|
let mut visible = image::RgbImage::new(image.width(), image.height());
|
||||||
|
|
||||||
|
for (target, source) in visible.pixels_mut().zip(image.pixels()) {
|
||||||
|
for channel in 0..3 {
|
||||||
|
let alpha = u32::from(source[3]);
|
||||||
|
target[channel] =
|
||||||
|
((u32::from(source[channel]) * alpha + 255 * (255 - alpha)) / 255) as u8;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let visible = image::DynamicImage::ImageRgb8(visible);
|
||||||
|
if let Some(model) = model {
|
||||||
|
let predictions = nsfw::examine(model, &visible.to_rgba8()).map_err(|error| {
|
||||||
|
warn!("Image moderation failed: {}", error);
|
||||||
|
ImageError::ModerationFailed
|
||||||
|
})?;
|
||||||
|
if probable_nsfw(&predictions)? {
|
||||||
|
return Err(ImageError::Nsfw);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut output = Cursor::new(Vec::new());
|
||||||
|
visible.write_to(&mut output, ImageFormat::Png).map_err(|_| ImageError::InvalidImage)?;
|
||||||
|
Ok(output.into_inner())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn predictions(scores: [f32; 5]) -> Vec<Classification> {
|
||||||
|
[Metric::Drawings, Metric::Hentai, Metric::Neutral, Metric::Porn, Metric::Sexy]
|
||||||
|
.into_iter()
|
||||||
|
.zip(scores)
|
||||||
|
.map(|(metric, score)| Classification { metric, score })
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn combines_unsafe_categories_and_rejects_invalid_scores() {
|
||||||
|
assert_eq!(probable_nsfw(&predictions([0.1, 0.2, 0.4, 0.2, 0.1])), Ok(true));
|
||||||
|
assert_eq!(probable_nsfw(&predictions([0.1, 0.1, 0.7, 0.05, 0.05])), Ok(false));
|
||||||
|
assert_eq!(
|
||||||
|
probable_nsfw(&predictions([0.0, 0.0, f32::NAN, 0.0, 0.0])),
|
||||||
|
Err(ImageError::ModerationFailed)
|
||||||
|
);
|
||||||
|
assert_eq!(probable_nsfw(&[]), Err(ImageError::ModerationFailed));
|
||||||
|
assert_eq!(
|
||||||
|
probable_nsfw(&predictions([0.0, -0.5, 1.0, 0.0, 0.0])),
|
||||||
|
Err(ImageError::ModerationFailed)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn absent_or_partial_configuration_disables_uploads() {
|
||||||
|
let mut settings = std::array::from_fn(|_| String::new());
|
||||||
|
assert!(ImageUploads::from_settings(&settings).0.is_none());
|
||||||
|
settings[0] = "images".into();
|
||||||
|
assert!(ImageUploads::from_settings(&settings).0.is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn disabling_nsfw_checks_skips_model_initialization() {
|
||||||
|
let settings = ["images", "eu-west-2", "test", "test", "https://images.example.com"]
|
||||||
|
.map(str::to_string);
|
||||||
|
assert!(ImageUploadService::new(&settings, false).unwrap().model.is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn disabled_moderation_allows_flagged_images_but_still_validates_files() {
|
||||||
|
let model = nsfw::create_model(
|
||||||
|
&include_bytes!("../../tests/fixtures/image-upload-unsafe.onnx")[..],
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let mut bytes = Cursor::new(Vec::new());
|
||||||
|
image::DynamicImage::new_rgb8(2, 2).write_to(&mut bytes, ImageFormat::Png).unwrap();
|
||||||
|
assert_eq!(prepare_image(Some(&model), bytes.get_ref()), Err(ImageError::Nsfw));
|
||||||
|
let published = prepare_image(None, bytes.get_ref()).unwrap();
|
||||||
|
assert_eq!(image::load_from_memory(&published).unwrap().width(), 2);
|
||||||
|
assert_eq!(prepare_image(None, b"not an image"), Err(ImageError::InvalidImage));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bundled_model_loads_and_classifies_an_image() {
|
||||||
|
let model = nsfw::create_model(MODEL_BYTES).unwrap();
|
||||||
|
let image = image::RgbaImage::from_pixel(224, 224, image::Rgba([255, 255, 255, 255]));
|
||||||
|
let predictions = nsfw::examine(&model, &image).unwrap();
|
||||||
|
assert!(probable_nsfw(&predictions).is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_non_images_and_truncated_images() {
|
||||||
|
assert_eq!(decode_image(b"<svg></svg>"), Err(ImageError::InvalidImage));
|
||||||
|
assert_eq!(decode_image(b"\x89PNG\r\n\x1a\n"), Err(ImageError::InvalidImage));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn decodes_real_images_without_trusting_the_filename() {
|
||||||
|
let image = image::DynamicImage::new_rgb8(2, 2);
|
||||||
|
for format in [ImageFormat::Png, ImageFormat::Jpeg, ImageFormat::Gif, ImageFormat::WebP] {
|
||||||
|
let mut bytes = Cursor::new(Vec::new());
|
||||||
|
image.write_to(&mut bytes, format).unwrap();
|
||||||
|
assert_eq!(decode_image(bytes.get_ref()).unwrap().dimensions(), (2, 2));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_excessive_image_dimensions() {
|
||||||
|
let image = image::DynamicImage::new_rgb8(8193, 1);
|
||||||
|
let mut bytes = Cursor::new(Vec::new());
|
||||||
|
image.write_to(&mut bytes, ImageFormat::Png).unwrap();
|
||||||
|
assert_eq!(decode_image(bytes.get_ref()), Err(ImageError::InvalidImage));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
use std::{
|
||||||
|
collections::BTreeSet,
|
||||||
|
fs,
|
||||||
|
io::Cursor,
|
||||||
|
path::{Path, PathBuf},
|
||||||
|
};
|
||||||
|
|
||||||
|
use image::{ImageFormat, RgbaImage};
|
||||||
|
|
||||||
|
use super::{prepare_image, ImageError, MODEL_BYTES};
|
||||||
|
|
||||||
|
const NSFW_IMAGES: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/nsfw");
|
||||||
|
const NON_NSFW_IMAGES: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/tests/fixtures/non_nsfw");
|
||||||
|
|
||||||
|
fn fixture_pairs(directory: &Path) -> Vec<(PathBuf, PathBuf)> {
|
||||||
|
let mut indices = BTreeSet::new();
|
||||||
|
for entry in fs::read_dir(directory).expect("Could not read image fixture directory") {
|
||||||
|
let entry = entry.expect("Could not read image fixture entry");
|
||||||
|
let path = entry.path();
|
||||||
|
if path.extension().and_then(|extension| extension.to_str()) != Some("png") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = path.file_name().unwrap().to_str().expect("Fixture name is not valid UTF-8");
|
||||||
|
let index = name
|
||||||
|
.strip_prefix("image_")
|
||||||
|
.and_then(|name| name.strip_suffix("a.png").or_else(|| name.strip_suffix("b.png")))
|
||||||
|
.filter(|index| index.parse::<u32>().is_ok_and(|index| index > 0))
|
||||||
|
.unwrap_or_else(|| {
|
||||||
|
panic!("Invalid fixture name {name}; expected image_1a.png/image_1b.png")
|
||||||
|
});
|
||||||
|
indices.insert(index.to_string());
|
||||||
|
}
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
!indices.is_empty(),
|
||||||
|
"No paired fixtures in {}. Add images with scripts/split_nsfw_test_image.py.",
|
||||||
|
directory.display()
|
||||||
|
);
|
||||||
|
indices
|
||||||
|
.into_iter()
|
||||||
|
.map(|index| {
|
||||||
|
let first = directory.join(format!("image_{index}a.png"));
|
||||||
|
let second = directory.join(format!("image_{index}b.png"));
|
||||||
|
assert!(
|
||||||
|
first.is_file() && second.is_file(),
|
||||||
|
"Both fixture parts must exist: {} and {}",
|
||||||
|
first.display(),
|
||||||
|
second.display()
|
||||||
|
);
|
||||||
|
(first, second)
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn recombine_parts(first: &RgbaImage, second: &RgbaImage) -> Result<RgbaImage, &'static str> {
|
||||||
|
if first.dimensions() != second.dimensions() {
|
||||||
|
return Err("Fixture parts must have identical dimensions");
|
||||||
|
}
|
||||||
|
let pixels = first.as_raw().iter().zip(second.as_raw()).map(|(a, b)| a ^ b).collect();
|
||||||
|
RgbaImage::from_raw(first.width(), first.height(), pixels).ok_or("Invalid fixture pixel data")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn check_directory(directory: &str, expected_nsfw: bool) {
|
||||||
|
let pairs = fixture_pairs(Path::new(directory));
|
||||||
|
let model = nsfw::create_model(MODEL_BYTES).expect("Could not load bundled NSFW model");
|
||||||
|
for (first, second) in pairs {
|
||||||
|
let a = image::open(&first).expect("Could not decode first fixture part").to_rgba8();
|
||||||
|
let b = image::open(&second).expect("Could not decode second fixture part").to_rgba8();
|
||||||
|
let reconstructed = recombine_parts(&a, &b).expect("Could not recombine fixture parts");
|
||||||
|
let mut bytes = Cursor::new(Vec::new());
|
||||||
|
reconstructed
|
||||||
|
.write_to(&mut bytes, ImageFormat::Png)
|
||||||
|
.expect("Could not encode reconstructed fixture");
|
||||||
|
|
||||||
|
// Use the production path, including flattening transparency and the same
|
||||||
|
// combined category threshold. Never write the reconstructed image to disk.
|
||||||
|
let result = prepare_image(Some(&model), bytes.get_ref());
|
||||||
|
if expected_nsfw {
|
||||||
|
assert_eq!(
|
||||||
|
result.as_ref().err(),
|
||||||
|
Some(&ImageError::Nsfw),
|
||||||
|
"Expected {} to be blocked as NSFW, got {:?}",
|
||||||
|
first.display(),
|
||||||
|
result.as_ref().err()
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
assert!(
|
||||||
|
result.is_ok(),
|
||||||
|
"Expected {} to be accepted, got {:?}",
|
||||||
|
first.display(),
|
||||||
|
result.as_ref().err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// The toggle must allow either category, while retaining image validation.
|
||||||
|
assert!(
|
||||||
|
prepare_image(None, bytes.get_ref()).is_ok(),
|
||||||
|
"Expected {} to be accepted with moderation disabled",
|
||||||
|
first.display()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
#[ignore = "Add local split fixtures to tests/fixtures/nsfw, then run with --ignored"]
|
||||||
|
fn nsfw_directory_images_are_blocked() {
|
||||||
|
check_directory(NSFW_IMAGES, true);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
#[ignore = "Add local split fixtures to tests/fixtures/non_nsfw, then run with --ignored"]
|
||||||
|
fn non_nsfw_directory_images_are_allowed() {
|
||||||
|
check_directory(NON_NSFW_IMAGES, false);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn recombining_parts_restores_every_pixel_and_alpha_channel() {
|
||||||
|
let original = [0, 17, 254, 255, 45, 99, 128, 0];
|
||||||
|
let random = vec![18, 201, 50, 34, 166, 76, 11, 220];
|
||||||
|
let masked = original.iter().zip(&random).map(|(pixel, mask)| pixel ^ mask).collect();
|
||||||
|
let first = RgbaImage::from_raw(2, 1, random).unwrap();
|
||||||
|
let second = RgbaImage::from_raw(2, 1, masked).unwrap();
|
||||||
|
assert_eq!(recombine_parts(&first, &second).unwrap().as_raw(), &original);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_parts_with_different_dimensions() {
|
||||||
|
assert_eq!(
|
||||||
|
recombine_parts(&RgbaImage::new(1, 1), &RgbaImage::new(2, 1)),
|
||||||
|
Err("Fixture parts must have identical dimensions")
|
||||||
|
);
|
||||||
|
}
|
||||||
+7
-4
@@ -4,6 +4,7 @@ mod macros;
|
|||||||
mod catchers;
|
mod catchers;
|
||||||
mod fairings;
|
mod fairings;
|
||||||
mod guards;
|
mod guards;
|
||||||
|
pub(crate) mod image_uploads;
|
||||||
mod ip_blocking;
|
mod ip_blocking;
|
||||||
mod routes;
|
mod routes;
|
||||||
|
|
||||||
@@ -124,7 +125,8 @@ pub async fn initialize(
|
|||||||
if Path::new("static").exists() { "static" } else { "/lib/reminder-rs/static" };
|
if Path::new("static").exists() { "static" } else { "/lib/reminder-rs/static" };
|
||||||
info!("Using static path: {}", static_path);
|
info!("Using static path: {}", static_path);
|
||||||
|
|
||||||
rocket::build()
|
let image_uploads = image_uploads::ImageUploads::from_env();
|
||||||
|
let rocket = rocket::build()
|
||||||
.attach(MetricProducer)
|
.attach(MetricProducer)
|
||||||
.attach(Template::fairing())
|
.attach(Template::fairing())
|
||||||
.register(
|
.register(
|
||||||
@@ -214,9 +216,10 @@ pub async fn initialize(
|
|||||||
routes::dashboard::export::todos::export,
|
routes::dashboard::export::todos::export,
|
||||||
routes::dashboard::export::todos::import,
|
routes::dashboard::export::todos::import,
|
||||||
],
|
],
|
||||||
)
|
);
|
||||||
.launch()
|
|
||||||
.await?;
|
let rocket = routes::dashboard::api::images::mount(rocket, image_uploads);
|
||||||
|
rocket.launch().await?;
|
||||||
|
|
||||||
warn!("Exiting rocket runtime");
|
warn!("Exiting rocket runtime");
|
||||||
// distribute kill signal
|
// distribute kill signal
|
||||||
|
|||||||
@@ -0,0 +1,319 @@
|
|||||||
|
use log::warn;
|
||||||
|
use rand::{rngs::OsRng, RngCore};
|
||||||
|
use rocket::{
|
||||||
|
data::{Data, ToByteUnit},
|
||||||
|
http::{CookieJar, Status},
|
||||||
|
post,
|
||||||
|
request::{FromRequest, Outcome},
|
||||||
|
serde::json::{json, Json, Value},
|
||||||
|
Request, State,
|
||||||
|
};
|
||||||
|
use sqlx::{MySql, Pool};
|
||||||
|
|
||||||
|
use crate::web::image_uploads::{prepare_image, ImageError, ImageUploads, MAX_UPLOAD_BYTES};
|
||||||
|
|
||||||
|
pub fn mount(
|
||||||
|
rocket: rocket::Rocket<rocket::Build>,
|
||||||
|
uploads: ImageUploads,
|
||||||
|
) -> rocket::Rocket<rocket::Build> {
|
||||||
|
let enabled = uploads.0.is_some();
|
||||||
|
let rocket = rocket.manage(uploads);
|
||||||
|
if enabled {
|
||||||
|
rocket.mount("/dashboard", rocket::routes![upload_image])
|
||||||
|
} else {
|
||||||
|
rocket
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct Uploader(u64);
|
||||||
|
|
||||||
|
#[rocket::async_trait]
|
||||||
|
impl<'r> FromRequest<'r> for Uploader {
|
||||||
|
type Error = ();
|
||||||
|
|
||||||
|
async fn from_request(request: &'r Request<'_>) -> Outcome<Self, Self::Error> {
|
||||||
|
match uploader_id(request.cookies()) {
|
||||||
|
Some(id) => Outcome::Success(Self(id)),
|
||||||
|
None => Outcome::Error((Status::Unauthorized, ())),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn uploader_id(cookies: &CookieJar<'_>) -> Option<u64> {
|
||||||
|
cookies.get_private("userid")?.value().parse().ok().filter(|id| *id != 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
type UploadResult = Result<Json<Value>, (Status, Json<Value>)>;
|
||||||
|
|
||||||
|
fn error(status: Status, message: &str) -> (Status, Json<Value>) {
|
||||||
|
(status, Json(json!({ "error": message })))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Requiring a non-form content type prevents cross-site HTML form submissions.
|
||||||
|
// Authorization is a request guard, so it runs before the body is read.
|
||||||
|
#[post("/api/images", format = "application/octet-stream", data = "<data>")]
|
||||||
|
pub async fn upload_image(
|
||||||
|
uploader: Uploader,
|
||||||
|
uploads: &State<ImageUploads>,
|
||||||
|
pool: &State<Pool<MySql>>,
|
||||||
|
data: Data<'_>,
|
||||||
|
) -> UploadResult {
|
||||||
|
let service =
|
||||||
|
uploads.0.as_ref().ok_or_else(|| error(Status::NotFound, "Image uploads disabled"))?;
|
||||||
|
let _permit = service
|
||||||
|
.slots
|
||||||
|
.clone()
|
||||||
|
.try_acquire_owned()
|
||||||
|
.map_err(|_| error(Status::TooManyRequests, "Image uploads busy. Please try again."))?;
|
||||||
|
let bytes = data
|
||||||
|
.open(MAX_UPLOAD_BYTES.bytes())
|
||||||
|
.into_bytes()
|
||||||
|
.await
|
||||||
|
.map_err(|_| error(Status::BadRequest, "Could not read image"))?;
|
||||||
|
if !bytes.is_complete() {
|
||||||
|
return Err(error(Status::PayloadTooLarge, "Images must be no larger than 8 MiB"));
|
||||||
|
}
|
||||||
|
let service_for_moderation = service.clone();
|
||||||
|
let moderated = tokio::task::spawn_blocking(move || {
|
||||||
|
prepare_image(service_for_moderation.model.as_ref(), &bytes.into_inner())
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.map_err(|err| {
|
||||||
|
warn!("Image moderation task failed: {}", err);
|
||||||
|
error(Status::ServiceUnavailable, "Image moderation unavailable. Please try again.")
|
||||||
|
})?;
|
||||||
|
let image = moderated.map_err(|err| match err {
|
||||||
|
ImageError::InvalidImage => error(
|
||||||
|
Status::UnprocessableEntity,
|
||||||
|
"Please upload a valid PNG, JPEG, GIF or WebP image (at most 16 million pixels)",
|
||||||
|
),
|
||||||
|
ImageError::Nsfw => error(
|
||||||
|
Status::UnavailableForLegalReasons,
|
||||||
|
"This image was flagged as probably NSFW and cannot be uploaded.",
|
||||||
|
),
|
||||||
|
ImageError::ModerationFailed => {
|
||||||
|
error(Status::ServiceUnavailable, "Image moderation unavailable. Please try again.")
|
||||||
|
}
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let mut random = [0u8; 32];
|
||||||
|
OsRng.fill_bytes(&mut random);
|
||||||
|
let key = format!(
|
||||||
|
"uploads/{}.png",
|
||||||
|
random.iter().map(|byte| format!("{:02x}", byte)).collect::<String>()
|
||||||
|
);
|
||||||
|
let url = format!("{}/{}", service.public_url, key);
|
||||||
|
let response =
|
||||||
|
service.bucket.put_object_with_content_type(&key, &image, "image/png").await.map_err(
|
||||||
|
|err| {
|
||||||
|
warn!("Could not upload image to S3: {}", err);
|
||||||
|
error(Status::BadGateway, "Could not store image. Please try again.")
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
if !(200..300).contains(&response.status_code()) {
|
||||||
|
warn!("S3 image upload returned {}", response.status_code());
|
||||||
|
return Err(error(Status::BadGateway, "Could not store image. Please try again."));
|
||||||
|
}
|
||||||
|
if let Err(err) = sqlx::query("INSERT INTO image_uploads (uploader, url) VALUES (?, ?)")
|
||||||
|
.bind(uploader.0)
|
||||||
|
.bind(&url)
|
||||||
|
.execute(pool.inner())
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
warn!("Could not record image upload: {}", err);
|
||||||
|
match service.bucket.delete_object(&key).await {
|
||||||
|
Ok(response) if (200..300).contains(&response.status_code()) => {}
|
||||||
|
_ => warn!("Could not clean up untracked image object {}", key),
|
||||||
|
}
|
||||||
|
return Err(error(
|
||||||
|
Status::InternalServerError,
|
||||||
|
"Could not record image upload. Please try again.",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(Json(json!({ "url": url })))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use std::{io::Cursor, sync::Arc};
|
||||||
|
|
||||||
|
use rocket::{http::ContentType, local::asynchronous::Client};
|
||||||
|
use s3::{creds::Credentials, Bucket, Region};
|
||||||
|
use tokio::sync::Semaphore;
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
use crate::web::image_uploads::ImageUploadService;
|
||||||
|
|
||||||
|
fn service(unsafe_image: bool, endpoint: &str) -> ImageUploads {
|
||||||
|
let model: &[u8] = if unsafe_image {
|
||||||
|
include_bytes!("../../../../../tests/fixtures/image-upload-unsafe.onnx")
|
||||||
|
} else {
|
||||||
|
include_bytes!("../../../../../tests/fixtures/image-upload-safe.onnx")
|
||||||
|
};
|
||||||
|
let bucket = Bucket::new(
|
||||||
|
"images",
|
||||||
|
Region::Custom { region: "eu-west-2".into(), endpoint: endpoint.into() },
|
||||||
|
Credentials::new(Some("test"), Some("test"), None, None, None).unwrap(),
|
||||||
|
)
|
||||||
|
.unwrap()
|
||||||
|
.with_path_style();
|
||||||
|
ImageUploads(Some(Arc::new(ImageUploadService {
|
||||||
|
bucket,
|
||||||
|
public_url: "https://images.example.com".into(),
|
||||||
|
model: Some(nsfw::create_model(model).unwrap()),
|
||||||
|
slots: Arc::new(Semaphore::new(2)),
|
||||||
|
})))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn image_bytes() -> Vec<u8> {
|
||||||
|
let mut bytes = Cursor::new(Vec::new());
|
||||||
|
image::DynamicImage::new_rgb8(2, 2).write_to(&mut bytes, image::ImageFormat::Png).unwrap();
|
||||||
|
bytes.into_inner()
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn client(uploads: ImageUploads) -> Client {
|
||||||
|
let pool = sqlx::mysql::MySqlPoolOptions::new()
|
||||||
|
.connect_lazy("mysql://unused:unused@127.0.0.1:1/unused")
|
||||||
|
.unwrap();
|
||||||
|
Client::tracked(mount(rocket::build().manage(pool), uploads)).await.unwrap()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[rocket::async_test]
|
||||||
|
async fn disabled_upload_route_is_not_available() {
|
||||||
|
let client = client(ImageUploads(None)).await;
|
||||||
|
assert_eq!(
|
||||||
|
client
|
||||||
|
.post("/dashboard/api/images")
|
||||||
|
.header(ContentType::Binary)
|
||||||
|
.body(image_bytes())
|
||||||
|
.dispatch()
|
||||||
|
.await
|
||||||
|
.status(),
|
||||||
|
Status::NotFound
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[rocket::async_test]
|
||||||
|
async fn upload_requires_an_authenticated_private_cookie() {
|
||||||
|
let client = client(service(false, "http://127.0.0.1:1")).await;
|
||||||
|
for cookie in [None, Some("not-an-id"), Some("0")] {
|
||||||
|
let mut request = client
|
||||||
|
.post("/dashboard/api/images")
|
||||||
|
.header(ContentType::Binary)
|
||||||
|
.body(image_bytes());
|
||||||
|
if let Some(cookie) = cookie {
|
||||||
|
request = request.private_cookie(("userid", cookie));
|
||||||
|
}
|
||||||
|
assert_eq!(request.dispatch().await.status(), Status::Unauthorized);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[rocket::async_test]
|
||||||
|
async fn unsafe_images_return_451_without_contacting_storage_or_database() {
|
||||||
|
let client = client(service(true, "http://127.0.0.1:1")).await;
|
||||||
|
let response = client
|
||||||
|
.post("/dashboard/api/images")
|
||||||
|
.header(ContentType::Binary)
|
||||||
|
.private_cookie(("userid", "123456789012345678"))
|
||||||
|
.body(image_bytes())
|
||||||
|
.dispatch()
|
||||||
|
.await;
|
||||||
|
assert_eq!(response.status(), Status::UnavailableForLegalReasons);
|
||||||
|
assert!(response.into_json::<Value>().await.unwrap()["error"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("NSFW"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[rocket::async_test]
|
||||||
|
async fn rejects_invalid_oversized_and_form_uploads() {
|
||||||
|
let client = client(service(false, "http://127.0.0.1:1")).await;
|
||||||
|
for (body, status) in [
|
||||||
|
(b"this is not an image".to_vec(), Status::UnprocessableEntity),
|
||||||
|
(vec![0; MAX_UPLOAD_BYTES as usize + 1], Status::PayloadTooLarge),
|
||||||
|
] {
|
||||||
|
let response = client
|
||||||
|
.post("/dashboard/api/images")
|
||||||
|
.header(ContentType::Binary)
|
||||||
|
.private_cookie(("userid", "123456789012345678"))
|
||||||
|
.body(body)
|
||||||
|
.dispatch()
|
||||||
|
.await;
|
||||||
|
assert_eq!(response.status(), status);
|
||||||
|
}
|
||||||
|
let response = client
|
||||||
|
.post("/dashboard/api/images")
|
||||||
|
.header(ContentType::Form)
|
||||||
|
.private_cookie(("userid", "123456789012345678"))
|
||||||
|
.body("file=anything")
|
||||||
|
.dispatch()
|
||||||
|
.await;
|
||||||
|
assert_eq!(response.status(), Status::NotFound);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn stores_moderated_image_and_tracks_uploader(pool: Pool<MySql>) {
|
||||||
|
use tokio::{
|
||||||
|
io::{AsyncReadExt, AsyncWriteExt},
|
||||||
|
net::TcpListener,
|
||||||
|
};
|
||||||
|
|
||||||
|
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||||
|
let endpoint = format!("http://{}", listener.local_addr().unwrap());
|
||||||
|
let storage = tokio::spawn(async move {
|
||||||
|
let (mut socket, _) = listener.accept().await.unwrap();
|
||||||
|
let mut received = Vec::new();
|
||||||
|
let mut buffer = [0; 4096];
|
||||||
|
let (headers, body) = loop {
|
||||||
|
let count = socket.read(&mut buffer).await.unwrap();
|
||||||
|
assert!(count > 0);
|
||||||
|
received.extend_from_slice(&buffer[..count]);
|
||||||
|
if let Some(end) = received.windows(4).position(|window| window == b"\r\n\r\n") {
|
||||||
|
let headers = String::from_utf8(received[..end].to_vec()).unwrap();
|
||||||
|
let length: usize = headers
|
||||||
|
.lines()
|
||||||
|
.find_map(|line| {
|
||||||
|
line.to_lowercase()
|
||||||
|
.strip_prefix("content-length:")
|
||||||
|
.map(|length| length.trim().parse().unwrap())
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
if received.len() >= end + 4 + length {
|
||||||
|
break (headers, received[end + 4..end + 4 + length].to_vec());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
socket
|
||||||
|
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
(headers, body)
|
||||||
|
});
|
||||||
|
let client =
|
||||||
|
Client::tracked(mount(rocket::build().manage(pool.clone()), service(false, &endpoint)))
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let response = client
|
||||||
|
.post("/dashboard/api/images")
|
||||||
|
.header(ContentType::Binary)
|
||||||
|
.private_cookie(("userid", "123456789012345678"))
|
||||||
|
.body(image_bytes())
|
||||||
|
.dispatch()
|
||||||
|
.await;
|
||||||
|
assert_eq!(response.status(), Status::Ok);
|
||||||
|
let response = response.into_json::<Value>().await.unwrap();
|
||||||
|
let url = response["url"].as_str().unwrap();
|
||||||
|
let row: (u64, String) = sqlx::query_as("SELECT uploader, url FROM image_uploads")
|
||||||
|
.fetch_one(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(row, (123456789012345678, url.to_string()));
|
||||||
|
let (headers, body) = storage.await.unwrap();
|
||||||
|
assert!(headers.starts_with("PUT /images/uploads/"));
|
||||||
|
assert!(headers.to_lowercase().contains("content-type: image/png"));
|
||||||
|
assert_eq!(image::load_from_memory(&body).unwrap().width(), 2);
|
||||||
|
assert!(url.starts_with("https://images.example.com/uploads/"));
|
||||||
|
assert!(url.ends_with(".png"));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
pub mod guild;
|
pub mod guild;
|
||||||
|
pub mod images;
|
||||||
pub mod user;
|
pub mod user;
|
||||||
|
|
||||||
use log::warn;
|
use log::warn;
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ use serenity::{
|
|||||||
use sqlx::{MySql, Pool};
|
use sqlx::{MySql, Pool};
|
||||||
|
|
||||||
use crate::web::guards::transaction::Transaction;
|
use crate::web::guards::transaction::Transaction;
|
||||||
|
use crate::web::image_uploads::ImageUploads;
|
||||||
|
|
||||||
#[derive(Serialize)]
|
#[derive(Serialize)]
|
||||||
struct UserPreferences {
|
struct UserPreferences {
|
||||||
@@ -36,6 +37,7 @@ struct UserPreferences {
|
|||||||
struct UserInfo {
|
struct UserInfo {
|
||||||
name: String,
|
name: String,
|
||||||
patreon: bool,
|
patreon: bool,
|
||||||
|
image_uploads: bool,
|
||||||
preferences: UserPreferences,
|
preferences: UserPreferences,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -52,10 +54,12 @@ pub async fn get_user_info(
|
|||||||
cookies: &CookieJar<'_>,
|
cookies: &CookieJar<'_>,
|
||||||
ctx: &State<Context>,
|
ctx: &State<Context>,
|
||||||
pool: &State<Pool<MySql>>,
|
pool: &State<Pool<MySql>>,
|
||||||
|
uploads: &State<ImageUploads>,
|
||||||
) -> JsonValue {
|
) -> JsonValue {
|
||||||
offline!(json!(UserInfo {
|
offline!(json!(UserInfo {
|
||||||
name: "Discord".to_string(),
|
name: "Discord".to_string(),
|
||||||
patreon: true,
|
patreon: true,
|
||||||
|
image_uploads: uploads.0.is_some(),
|
||||||
preferences: UserPreferences {
|
preferences: UserPreferences {
|
||||||
timezone: "UTC".to_string(),
|
timezone: "UTC".to_string(),
|
||||||
use_browser_timezone: true,
|
use_browser_timezone: true,
|
||||||
@@ -101,6 +105,7 @@ pub async fn get_user_info(
|
|||||||
);
|
);
|
||||||
|
|
||||||
let user_info = UserInfo {
|
let user_info = UserInfo {
|
||||||
|
image_uploads: uploads.0.is_some(),
|
||||||
name: cookies
|
name: cookies
|
||||||
.get_private("username")
|
.get_private("username")
|
||||||
.map_or("Discord User".to_string(), |c| c.value().to_string()),
|
.map_or("Discord User".to_string(), |c| c.value().to_string()),
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Clean orphaned image uploads from Reminder Bot
|
||||||
|
Wants=network-online.target
|
||||||
|
After=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
User=reminder
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/usr/bin/reminder-rs clean-images
|
||||||
|
WorkingDirectory=/etc/reminder-rs
|
||||||
|
Environment="RUST_LOG=reminder_rs=info"
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Clean orphaned Reminder Bot image uploads hourly
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnCalendar=hourly
|
||||||
|
Persistent=true
|
||||||
|
Unit=reminder-rs-clean-images.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
Vendored
+48
@@ -0,0 +1,48 @@
|
|||||||
|
The two tiny ONNX models are deterministic classifiers used by the image upload API tests.
|
||||||
|
They accept the same `[1, 224, 224, 3]` float input as the production nsfw model and return
|
||||||
|
`[Drawings, Hentai, Neutral, Porn, Sexy]` scores. One always returns Neutral = 1 and the
|
||||||
|
other always returns Porn = 1. They exercise the real nsfw inference path without requiring
|
||||||
|
image fixtures.
|
||||||
|
|
||||||
|
## Image fixtures for the bundled model
|
||||||
|
|
||||||
|
Add image pairs to these project directories:
|
||||||
|
|
||||||
|
* `tests/fixtures/nsfw/` — images expected to be rejected by moderation.
|
||||||
|
* `tests/fixtures/non_nsfw/` — images expected to be accepted by moderation.
|
||||||
|
|
||||||
|
Install the splitter's dependency with `python3 -m pip install Pillow`, then run from the
|
||||||
|
project root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
python3 scripts/split_nsfw_test_image.py /path/to/input.jpg --output-dir tests/fixtures/nsfw
|
||||||
|
python3 scripts/split_nsfw_test_image.py /path/to/safe.jpg --output-dir tests/fixtures/non_nsfw
|
||||||
|
```
|
||||||
|
|
||||||
|
To run the tests:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
SQLX_OFFLINE=true cargo test --locked directory_images -- --ignored
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deterministic model fixtures
|
||||||
|
|
||||||
|
To regenerate them with Python's `onnx` package:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from pathlib import Path
|
||||||
|
from onnx import TensorProto, checker, helper
|
||||||
|
|
||||||
|
for name, scores in [("safe", [0., 0., 1., 0., 0.]), ("unsafe", [0., 0., 0., 1., 0.])]:
|
||||||
|
tensor = helper.make_tensor("probabilities", TensorProto.FLOAT, [1, 5], scores)
|
||||||
|
graph = helper.make_graph(
|
||||||
|
[helper.make_node("Identity", ["probabilities"], ["output"])],
|
||||||
|
"constant classifier for upload endpoint tests",
|
||||||
|
[helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 224, 224, 3])],
|
||||||
|
[helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 5])],
|
||||||
|
[tensor],
|
||||||
|
)
|
||||||
|
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 13)], ir_version=8)
|
||||||
|
checker.check_model(model)
|
||||||
|
Path(f"tests/fixtures/image-upload-{name}.onnx").write_bytes(model.SerializeToString())
|
||||||
|
```
|
||||||
BIN
Binary file not shown.
BIN
Binary file not shown.
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 2.9 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 2.9 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 2.3 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 2.3 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 6.0 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 3.6 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 3.6 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 2.9 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 2.9 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 5.3 MiB |
Vendored
BIN
Binary file not shown.
|
After Width: | Height: | Size: 5.3 MiB |
Reference in new issue
Block a user