mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-10-28 08:25:57 +08:00
wyuc
fccbf82c24
## Summary Large or unsupported images attached to vision requests can crash prompt building or be rejected by the model provider, and a single bad image fails the whole generation step (retries fail the same way). - **`@openmaic/generation` 0.3.19**: `buildVisionUserContent` no longer runs a regex over the whole data URL. A new exported `parseBase64DataUrl` splits header and payload with plain string ops, so multi-megabyte payloads are never scanned by the regex engine. - **Vision image cap (app)**: new `prepareVisionImages` (`lib/server/generation/vision-images.ts`) runs right before every vision call — outline, scene content, scene actions, and the PBL evaluator (which receives uploaded submissions). - Pass-through only for single-frame, 8-bit sRGB/grayscale png/jpeg/webp/gif whose **base64 payload** is within `MAX_VISION_IMAGE_BASE64_BYTES` (1 MiB — providers measure the encoded payload) and whose longest side is within `MAX_VISION_IMAGE_DIMENSION` (2048 px). Pixels are strictly decoded first so truncated data is caught. http(s) URLs are untouched. - Everything else (oversized, animated, CMYK, 16-bit, unsupported formats) is decoded once with sharp (first frame, EXIF-rotated, resized inside 2048, flattened on white, sRGB) and encoded to JPEG, stepping quality down until under the cap. - Images that still don't fit or can't be decoded are dropped with a warning; the step continues. - Preparation runs where vision images are resolved (outline and scene-content pre-resolution), so the prompt's "[see attached]" placeholders and asset-id mappings only describe images that are actually attached. The PBL evaluator logs when an image submission is evaluated without its image. - Processing is sequential to bound peak memory. Non-goals: no change to material upload/extraction, prompts, or generation logic. ## Test plan - [x] `pnpm check`, `pnpm lint`, `npx tsc --noEmit` - [x] `@openmaic/generation` build + typecheck + tests (incl. an 8 MiB payload, malformed and non-base64 data URLs, http passthrough) - [x] New `tests/generation/vision-images.test.ts` (small untouched, large downscaled under caps, unsupported format converted, undecodable dropped while others kept, http untouched); updated scene-content / retry-boundary / evaluator tests - [x] Full `pnpm test` (Node 22) - [x] `node scripts/check-package-version-bumps.mjs origin/main` 🤖 Generated with [Claude Code](https://claude.com/claude-code)