Add support for S3-backed image storage
This commit is contained in:
1 parent
7fcb2eb747
commit
8763a8e74a
55 files changed
+2611
-111
No files matched your search
Vendored
+67
@@ -0,0 +1,67 @@
|
||||
The two tiny ONNX models are deterministic classifiers used by the image upload API tests.
|
||||
They accept the same `[1, 224, 224, 3]` float input as the production nsfw model and return
|
||||
`[Drawings, Hentai, Neutral, Porn, Sexy]` scores. One always returns Neutral = 1 and the
|
||||
other always returns Porn = 1. They exercise the real nsfw inference path without requiring
|
||||
image fixtures.
|
||||
|
||||
## Image fixtures for the bundled model
|
||||
|
||||
Add image pairs to these project directories:
|
||||
|
||||
* `tests/fixtures/nsfw/` — images expected to be rejected by moderation.
|
||||
* `tests/fixtures/non_nsfw/` — images expected to be accepted by moderation.
|
||||
|
||||
Install the splitter's dependency with `python3 -m pip install Pillow`, then run from the
|
||||
project root:
|
||||
|
||||
```sh
|
||||
python3 scripts/split_nsfw_test_image.py /path/to/input.jpg --output-dir tests/fixtures/nsfw
|
||||
python3 scripts/split_nsfw_test_image.py /path/to/safe.jpg --output-dir tests/fixtures/non_nsfw
|
||||
```
|
||||
|
||||
Each command automatically selects the first unused positive index, starting with
|
||||
`image_1a.png` and `image_1b.png`. Indices with either part already present are skipped.
|
||||
You can override the selection with `--index 2`; existing parts are never overwritten.
|
||||
Each part contains random-looking
|
||||
RGBA pixels. The original decoded pixels, including alpha, are recovered by XORing the two
|
||||
parts. Only the first frame of animated input is used. Keep both files unchanged: resizing,
|
||||
re-encoding with loss or flattening their alpha channels prevents correct reconstruction.
|
||||
Both parts together reconstruct the original image, so splitting is not access control.
|
||||
|
||||
Run the splitter's round-trip and overwrite checks with
|
||||
`python3 -m unittest discover -s scripts -p 'test_*.py'`.
|
||||
|
||||
The directories initially contain no images. The two dataset tests are explicitly ignored
|
||||
until you provide fixtures. To run them after adding pairs to both directories:
|
||||
|
||||
```sh
|
||||
SQLX_OFFLINE=true cargo test --locked directory_images -- --ignored
|
||||
```
|
||||
|
||||
The tests load all numbered pairs, recombine them in memory and pass the reconstructed PNG
|
||||
through the production upload moderation function using the bundled model and its current
|
||||
threshold. They also check that disabling moderation accepts the reconstructed images.
|
||||
Empty directories, missing partners, mismatched dimensions, unexpected names and unexpected
|
||||
moderation results fail the tests. Recombined images are never written to disk.
|
||||
|
||||
## Deterministic model fixtures
|
||||
|
||||
To regenerate them with Python's `onnx` package:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from onnx import TensorProto, checker, helper
|
||||
|
||||
for name, scores in [("safe", [0., 0., 1., 0., 0.]), ("unsafe", [0., 0., 0., 1., 0.])]:
|
||||
tensor = helper.make_tensor("probabilities", TensorProto.FLOAT, [1, 5], scores)
|
||||
graph = helper.make_graph(
|
||||
[helper.make_node("Identity", ["probabilities"], ["output"])],
|
||||
"constant classifier for upload endpoint tests",
|
||||
[helper.make_tensor_value_info("input", TensorProto.FLOAT, [1, 224, 224, 3])],
|
||||
[helper.make_tensor_value_info("output", TensorProto.FLOAT, [1, 5])],
|
||||
[tensor],
|
||||
)
|
||||
model = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 13)], ir_version=8)
|
||||
checker.check_model(model)
|
||||
Path(f"tests/fixtures/image-upload-{name}.onnx").write_bytes(model.SerializeToString())
|
||||
```
|
||||
Reference in new issue
Block a user