avfilter/vf_vignette: keep the backward gain finite

In backward mode the gain is 1 / factor, but the factor is 0 for pixels
farther than dmax from the center (possible with x0/y0 or aspect), so
the gain is inf. inf * sample, or inf * 0 = NaN, is then converted to
int, which is undefined behaviour; on x86 these pixels come out black
instead of saturated.

For angles close to PI/2, the factor can also become small enough that
a finite backward gain produces sample values outside the range of int
before av_clip_uint8() is called. Bound the gain at 256 to keep these
conversions in range while preserving the saturated 8-bit output.

Signed-off-by: Martin Mirchev <mmirchev@comp.nus.edu.sg>

libavfilter/vf_vignette.c

@@ -175,8 +175,14 @@ static void update_context(VignetteContext *s, AVFilterLink *inlink, AVFrame *fr
 
     if (s->backward) {
         for (y = 0; y < inlink->h; y++) {
-            for (x = 0; x < inlink->w; x++)
-                dst[x] = 1. / get_natural_factor(s, x, y);
+            for (x = 0; x < inlink->w; x++) {
+                const double f = get_natural_factor(s, x, y);
+                /* f is 0 beyond dmax and can be tiny for an angle close to
+                 * PI/2: bound the gain to keep the scaled samples in the
+                 * range of int, 256 already saturates any non-zero 8-bit
+                 * sample */
+                dst[x] = f > 1. / 256 ? 1. / f : 256;
+            }
             dst += dst_linesize;
         }
     } else {