avfilter/vf_opencolorio: negotiate the output format properly

The filter picked its output format in config_props(), after format
negotiation had already settled the output link. The frames it emitted
could therefore disagree with the negotiated link format, for example
when a downstream zscale had negotiated something else.

Switch to FILTER_QUERY_FUNC2() and decide the output format during
negotiation instead:

- with format=, the output is constrained to that single format, which
  is now also validated in init() against the supported output list;
- otherwise, when no half-float candidate remains on the input, the
  output list is the input list, so the filter passes the format
  through and downstream constraints can still narrow the input;
- otherwise the output is restricted to float, and negotiation picks
  GBRPF32 or GBRAPF32 to match the input's alpha.

Half-float input has to be promoted because swscale cannot read or
write half-float in its new backend, and its legacy code clamps
half-float to [0,1], which would destroy values outside that range.
Half-float is therefore no longer offered as an output format.

The default output depends on the input format, so the query waits for
the input list to be merged with the upstream one, which it detects by
comparing it against the input link's other end. The merged list need
not be a singleton. Waiting for a singleton would reject valid graphs,
because a filter such as zscale offers half-float and integer formats
on its output independently of its input, so that list never narrows
to one of the two.

While the query is delayed it publishes the input color space, color
range, alpha mode and chroma location lists, because a converter
inserted on the input requires them, but the matching output lists are
only published together with the output pixel formats, as a converter
inserted on the output would otherwise find them without a format list
and crash.

Finally, supply opaque input alpha when the output format has alpha and
the input format does not. Prepend an alpha-initialization matrix to the
resolved OCIO processor, allowing OCIO to optimize it with the user's
transform. This avoids a separate per-frame alpha fill and ensures that
transforms which use alpha in RGB calculations or modify alpha receive
the opaque value before processing and retain their transformed alpha.
Filling output alpha after processing would instead leave alpha-dependent
RGB incorrect and overwrite alpha produced by the user's transform.

Note, this includes Fairys fairy/pr21799-replace patch (Thank-you Fairy!).

Signed-off-by: Sam Richards <Sam.Richards@taurich.org>
Assisted-by: Fairy

doc/filters.texi

@@ -19424,7 +19424,19 @@ When used in combination with display and view, this inverts the transform, so g
 Allows you to specify an external file-transform to use instead of the OCIO config file. This is useful for applying a single transform without needing a full OCIO config file.
 
 @item format
-Allow you to specify the output pix_fmt of the OCIO filter. This *has* to be a RGB colorspace, so you really are limited to rgb24, rgba, rgb48, rgba48, gbrp10, gbrp12, gbrpf32le, gbrapf32le, for most encoding we would recommend rgb48
+Allow you to specify the output pix_fmt of the OCIO filter. This *has* to be a
+RGB colorspace, so you are limited to rgb24, rgba, rgb48, rgba64, gbrp10,
+gbrap10, gbrp12, gbrap12, gbrpf32 and gbrapf32; for most encoding we would
+recommend rgb48.
+
+Half-float (gbrpf16, gbrapf16) is accepted as input but is not a valid output
+format, because swscale has no half-float write support and its legacy
+half-float read path clamps values to [0,1]. When this option is not set, the
+output format defaults to the input format, except that half-float input is
+promoted to the matching float format. Where negotiation cannot yet tell
+whether the input will be half-float, the output falls back to float, so a
+float output may be selected even when the input turns out to be an integer
+format.
 
 @item context_params
 Allow you to specify additional context parameters for the OCIO filter. This is a list of key=value pairs, separated by colons.

libavfilter/ocio_wrapper.cpp

@@ -219,9 +219,33 @@ int ocio_finalize_processor(AVFilterContext *ctx, OCIOHandle handle, int input_f
         if (!s || !s->processor)
             return -1;
 
-        s->cpu = s->processor->getOptimizedCPUProcessor(
-        get_ocio_depth(input_format), get_ocio_depth(output_format),
-        OCIO::OPTIMIZATION_DEFAULT);
+        OCIO::ConstProcessorRcPtr processor = s->processor;
+        const AVPixFmtDescriptor *in_desc =
+            av_pix_fmt_desc_get((AVPixelFormat)input_format);
+        const AVPixFmtDescriptor *out_desc =
+            av_pix_fmt_desc_get((AVPixelFormat)output_format);
+
+        if (!(in_desc->flags & AV_PIX_FMT_FLAG_ALPHA) &&
+             (out_desc->flags & AV_PIX_FMT_FLAG_ALPHA)) {
+            // Supply opaque input alpha before the user's transform so it
+            // can use alpha in RGB calculations and transform alpha itself.
+            const double matrix[16] = { 1, 0, 0, 0,
+                                       0, 1, 0, 0,
+                                       0, 0, 1, 0,
+                                       0, 0, 0, 0 };
+            const double offset[4] = { 0, 0, 0, 1 };
+            OCIO::MatrixTransformRcPtr opaque = OCIO::MatrixTransform::Create();
+            opaque->setMatrix(matrix);
+            opaque->setOffset(offset);
+            OCIO::GroupTransformRcPtr group = OCIO::GroupTransform::Create();
+            group->appendTransform(opaque);
+            group->appendTransform(s->processor->createGroupTransform());
+            processor = s->config->getProcessor(group);
+        }
+
+        s->cpu = processor->getOptimizedCPUProcessor(
+            get_ocio_depth(input_format), get_ocio_depth(output_format),
+            OCIO::OPTIMIZATION_DEFAULT);
 
         return 0;
     } catch (OCIO::Exception &e) {

libavfilter/vf_opencolorio.c

@@ -63,21 +63,117 @@ static int ocio_filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_
     return ocio_apply(ctx, s->ocio, in, out, slice_start, slice_h);
 }
 
-static int query_formats(AVFilterContext *ctx) {
-    static const enum AVPixelFormat pix_fmts[] = {
-      // 8-bit
-      AV_PIX_FMT_RGBA, AV_PIX_FMT_RGB24,
-      // 16-bit
-      AV_PIX_FMT_RGBA64, AV_PIX_FMT_RGB48,
-      // 10-bit
-      AV_PIX_FMT_GBRP10, AV_PIX_FMT_GBRAP10,
-      // 12-bit
-      AV_PIX_FMT_GBRP12, AV_PIX_FMT_GBRAP12,
-      // Half-float and float
-      AV_PIX_FMT_GBRPF16, AV_PIX_FMT_GBRAPF16,
-      // Float
-      AV_PIX_FMT_GBRPF32, AV_PIX_FMT_GBRAPF32, AV_PIX_FMT_NONE};
-    return ff_set_common_formats(ctx, ff_make_format_list(pix_fmts));
+// Formats supported for output (also accepted as input).
+static const enum AVPixelFormat output_pix_fmts[] = {
+    // 8-bit
+    AV_PIX_FMT_RGBA, AV_PIX_FMT_RGB24,
+    // 16-bit
+    AV_PIX_FMT_RGBA64, AV_PIX_FMT_RGB48,
+    // 10-bit
+    AV_PIX_FMT_GBRP10, AV_PIX_FMT_GBRAP10,
+    // 12-bit
+    AV_PIX_FMT_GBRP12, AV_PIX_FMT_GBRAP12,
+    // Float
+    AV_PIX_FMT_GBRPF32, AV_PIX_FMT_GBRAPF32,
+    AV_PIX_FMT_NONE
+};
+
+// All formats accepted as input, including half-float.
+// Half-float is input-only because swscale has no F16 write support.
+// We promote half-float to float on output, which is lossless. swscale's
+// legacy F16 read path also clamps to [0,1], and its new backend has no
+// half-float support yet.
+static const enum AVPixelFormat input_pix_fmts[] = {
+    // 8-bit
+    AV_PIX_FMT_RGBA, AV_PIX_FMT_RGB24,
+    // 16-bit
+    AV_PIX_FMT_RGBA64, AV_PIX_FMT_RGB48,
+    // 10-bit
+    AV_PIX_FMT_GBRP10, AV_PIX_FMT_GBRAP10,
+    // 12-bit
+    AV_PIX_FMT_GBRP12, AV_PIX_FMT_GBRAP12,
+    // Half-float (input only)
+    AV_PIX_FMT_GBRPF16, AV_PIX_FMT_GBRAPF16,
+    // Float
+    AV_PIX_FMT_GBRPF32, AV_PIX_FMT_GBRAPF32,
+    AV_PIX_FMT_NONE
+};
+
+
+static int query_formats(const AVFilterContext *ctx,
+                         AVFilterFormatsConfig **cfg_in,
+                         AVFilterFormatsConfig **cfg_out)
+{
+    static const enum AVPixelFormat float_fmts[] = {
+        AV_PIX_FMT_GBRPF32, AV_PIX_FMT_GBRAPF32, AV_PIX_FMT_NONE
+    };
+    const OCIOContext *s = ctx->priv;
+    const AVFilterFormats *in;
+    int has_f16 = 0;
+    int ret;
+
+    // Input accepts all supported formats including half-float
+    if (!cfg_in[0]->formats) {
+        ret = ff_formats_ref(ff_make_pixel_format_list(input_pix_fmts),
+                             &cfg_in[0]->formats);
+        if (ret < 0)
+            return ret;
+
+        // Returning AVERROR(EAGAIN) skips the generic defaults, but the
+        // graph may insert a converter on the input before this filter
+        // publishes anything, and that code expects these lists to exist.
+        if ((ret = ff_formats_ref(ff_all_color_spaces(), &cfg_in[0]->color_spaces)) < 0 ||
+            (ret = ff_formats_ref(ff_all_color_ranges(), &cfg_in[0]->color_ranges)) < 0 ||
+            (ret = ff_formats_ref(ff_all_alpha_modes(), &cfg_in[0]->alpha_modes)) < 0 ||
+            (ret = ff_formats_ref(ff_all_chroma_locations(), &cfg_in[0]->chroma_locations)) < 0)
+            return ret;
+    }
+
+    // The default output depends on the input, so wait until the input list
+    // has been merged with the upstream one. Until then it is still this
+    // filter's own list and says nothing about the actual input format.
+    // The merged list need not be a singleton. This inspects the link
+    // because the merge cannot be observed through cfg_in alone.
+    if (s->output_format == AV_PIX_FMT_NONE &&
+        ctx->inputs[0]->incfg.formats != cfg_in[0]->formats)
+        return AVERROR(EAGAIN);
+
+    // Only publish the output attributes together with the output pixel
+    // formats: a converter inserted on the output would otherwise find
+    // them set while the format list is still missing.
+    if ((ret = ff_formats_ref(cfg_in[0]->color_spaces, &cfg_out[0]->color_spaces)) < 0 ||
+        (ret = ff_formats_ref(cfg_in[0]->color_ranges, &cfg_out[0]->color_ranges)) < 0 ||
+        (ret = ff_formats_ref(cfg_in[0]->alpha_modes, &cfg_out[0]->alpha_modes)) < 0 ||
+        (ret = ff_formats_ref(cfg_in[0]->chroma_locations, &cfg_out[0]->chroma_locations)) < 0)
+        return ret;
+
+    // Output: if user specified a format, constrain to just that format
+    if (s->output_format != AV_PIX_FMT_NONE) {
+        const enum AVPixelFormat user_fmts[] = { s->output_format, AV_PIX_FMT_NONE };
+        return ff_formats_ref(ff_make_pixel_format_list(user_fmts),
+                              &cfg_out[0]->formats);
+    }
+
+    // No user-specified format: the default output is the input format,
+    // except that half-float input is promoted to the matching float
+    // format, because swscale clips F16 to [0,1] when converting it.
+    in = cfg_in[0]->formats;
+    for (unsigned i = 0; i < in->nb_formats; i++) {
+        const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(in->formats[i]);
+        if (desc->comp[0].depth == 16 && (desc->flags & AV_PIX_FMT_FLAG_FLOAT))
+            has_f16 = 1;
+    }
+
+    // No half-float candidate: the output is the input format, so share
+    // the list and let downstream constraints narrow the input as well.
+    if (!has_f16)
+        return ff_formats_ref(cfg_in[0]->formats, &cfg_out[0]->formats);
+
+    // The input may be half-float (possibly among other candidates, e.g.
+    // behind zscale): only offer float output, which is lossless for every
+    // input format. Negotiation picks the one matching the input's alpha.
+    return ff_formats_ref(ff_make_pixel_format_list(float_fmts),
+                          &cfg_out[0]->formats);
 }
 
 static int config_props(AVFilterLink *inlink)
@@ -91,19 +187,10 @@ static int config_props(AVFilterLink *inlink)
         return AVERROR(EINVAL);
     }
 
-    int is_half =
-        (desc->comp[0].depth == 16 && desc->flags & AV_PIX_FMT_FLAG_FLOAT);
-    if (s->output_format == AV_PIX_FMT_NONE) {
-        // Need to set the output format now, if not known.
-        if (is_half) {
-            // If its half-float, we output float, due to a bug in ffmpeg with
-            // half-float frames
-            s->output_format = AV_PIX_FMT_GBRAPF32;
-        } else {
-            // If output format not set, use same as input
-            s->output_format = inlink->format;
-        }
-    }
+    // The output format was already constrained during negotiation.
+    // Read it from the negotiated output link if not set by the user.
+    if (s->output_format == AV_PIX_FMT_NONE)
+        s->output_format = ctx->outputs[0]->format;
 
     s->channels = desc->nb_components; // 3 or 4
 
@@ -127,10 +214,29 @@ static int config_props(AVFilterLink *inlink)
 static av_cold int init(AVFilterContext *ctx)
 {
     OCIOContext *s = ctx->priv;
-    if (s->out_format_string != NULL)
+    if (s->out_format_string != NULL) {
         s->output_format = av_get_pix_fmt(s->out_format_string);
-    else
+        if (s->output_format == AV_PIX_FMT_NONE) {
+            av_log(ctx, AV_LOG_ERROR, "Unknown pixel format: '%s'\n",
+                   s->out_format_string);
+            return AVERROR(EINVAL);
+        }
+        // Verify the format is a supported output format
+        int valid = 0;
+        for (int i = 0; output_pix_fmts[i] != AV_PIX_FMT_NONE; i++) {
+            if (s->output_format == output_pix_fmts[i]) {
+                valid = 1;
+                break;
+            }
+        }
+        if (!valid) {
+            av_log(ctx, AV_LOG_ERROR,
+                   "Unsupported output format: '%s'\n", s->out_format_string);
+            return AVERROR(EINVAL);
+        }
+    } else {
         s->output_format = AV_PIX_FMT_NONE; // Default to same as input format (see later).
+    }
 
     if (s->filetransform && strlen(s->filetransform) > 0) {
         s->ocio = ocio_create_file_transform_processor(
@@ -287,4 +393,4 @@ const FFFilter ff_vf_ocio = {
     .uninit = uninit,
     FILTER_INPUTS(inputs),
     FILTER_OUTPUTS(outputs),
-    FILTER_QUERY_FUNC(query_formats)};
+    FILTER_QUERY_FUNC2(query_formats)};