I have just finished my first version of a function that returns an array of video screenshots created with canvas. What I am looking for are ways to improve this speed by as much as possible. I want to almost be able to instantly display the results. Yes, this should be a backend thing, no I can't do it in the backend. It needs to be done in the frontend.

One of my biggest performance issues at the moment, or what I at least think it is, is the fact that everytime a new image is pushed inside the array I call the callback function. This is NOT the desired functionality, I wish to return the contents of the result only when the images array is the same length as the given amount parameter. However, this is outside of my ability because I simply do not know how this could be accomplished. If anyone knows a way to fix this, please elaborate.

One more issue I am having is the fact that my renderImage function keeps old images from previous videos. I have no idea why or how I can fix this. Help is appreciated but my biggest priority is to increase the render and display speed of the fake generated gif's (src loops in this case).

// Max Screenshots (Firefox) = 19

// Max Screenshots (Chrome) = Infinite



const generateVideoThumbnail = (

  event,

  callback,

  options = {

    amount: 15,

    step: 2

  },

) => {

  let {

    amount,

    step

  } = options;

  if (amount > 19) {

    amount = 19;

  }

  if (step > 4) {

    step = 4;

  }



  const fileReader = new FileReader();

  const response = {

    status: '',

    images: ,

    firstImage: null

  };

  const videoBlob = event.target.files[0];



  if (videoBlob.type.match('video')) {

    fileReader.onload = () => {

      const blob = new Blob([fileReader.result], {

        type: videoBlob.type

      });

      const url = URL.createObjectURL(blob);

      const video = document.createElement('video');



      const snapImage = () => {

        const canvas = document.createElement('canvas');

        canvas.width = video.videoWidth;

        canvas.height = video.videoHeight;

        canvas

          .getContext('2d')

          .drawImage(video, 0, 0, canvas.width, canvas.height);



        const image = canvas.toDataURL();

        const success = image.length > 100000;



        if (success) {

          response.status = 'Video successfully parsed.';

          response.images.push(image);



          if (!response.firstImage) {

            response.firstImage = image;

          }



          video.currentTime += step;

          URL.revokeObjectURL(url);

          callback(response);

        }



        return success;

      };



      const timeupdate = () => {

        snapImage();

        if (response.images.length < amount) {

          snapImage();

        } else {

          console.log(response);

          video.removeEventListener('timeupdate', timeupdate);

          video.pause();

        }

      };



      video.addEventListener('loadeddata', () => {

        snapImage();

      });



      video.addEventListener('timeupdate', timeupdate);

      video.preload = 'metadata';

      video.src = url;

      video.muted = true;

      video.playsInline = true;

      video.play();

    };

  } else {

    response.status = 'File is not a video.';

    response.images = null;

    return callback(response);

  }



  fileReader.readAsArrayBuffer(videoBlob);

};



const renderImage = (res, amount) => {

  const gif = document.getElementById('gif');



  let imageIndex = 0;

  let timer = undefined;



  const gifAnimation = () => {

    gif.src = res.images[imageIndex];

    imageIndex++;



    if (imageIndex === res.images.length - 1) {

      imageIndex = 0;

    }



    timer = setTimeout(gifAnimation, 225);

  };



  const mouseover = () => {

    gifAnimation();

  };



  const mouseleave = () => {

    clearTimeout(timer);

    gif.src = res.firstImage;

    imageIndex = 0;

  };



  gif.removeEventListener('mouseover', mouseover);

  gif.removeEventListener('mouseleave', mouseleave);



  if (res.images.length === amount) {

    gif.src = res.firstImage;



    gif.addEventListener('mouseover', mouseover);

    gif.addEventListener('mouseleave', mouseleave);

  }

};



document.getElementById('inputFile').addEventListener('change', event =>

  generateVideoThumbnail(event, res => renderImage(res, 15), {

    amount: 15,

    step: 2,

  }),

);

<!DOCTYPE html>

<html lang="en">



<head>

  <meta charset="UTF-8" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />

  <meta http-equiv="X-UA-Compatible" content="ie=edge" />

  <title>Demo 3</title>



  <style>

    body {

      margin: 0;

      padding: 0;

      text-align: center;

    }

    

    img {

      display: block;

      height: 33vh;

      object-fit: cover;

      transition: all .2s ease-in-out;

      width: auto;

    }

    

    #gif {

      margin: 5% auto;

    }

  </style>

</head>



<body>

  <img id="gif" />

  <input type="file" id="inputFile" accept=".mov, .mp4" />

  <script src="./index.js"></script>

</body>



</html>

Wrong approch

Data URL's are for transport only.

A data URL converts a binary stream to a format that can be transported over networks that handle text files as UTF-8. It does this by EXPANDING the data to fit a safe range of characters (64 of them) storing 6 bits in every byte (UTF-8). That means for every 3 bytes in the source the data URL needs 4 bytes to store.

This is compounded in javascript by the fact that JavaScript strings are 16bit UCS-2 characters. That means that for every 3 bytes in the source the JavaScript dataURL needs 8 bytes to store.

Video

Videos are highly compressed streams, not only is each frame compressed, but that compression is related to previous frames. This allows very high compression rates. For example I picked a random low res video 420 by 270 at 20Fps, running time of 1h 25m, and a file size of 380MB.

A random frame saved as png was 238kb in size.

The data throughput to display is about 10mb a second (and this is low res LOW quality video, full frame HD is about 0.5Gig a second)

That means that running your function on that video the memory needed to complete the function is about 380 * 1024 * 1024 + (1 * 60 * 60 * 20fps + 25 * 60 * 20fps) * 200kb * 1024 * ((8 / 3)dataUrl Storage overhead) which is around 56 gigabytes of RAM (Something only the very top range devices can handle).

If you compare the Video size to the image dataURLs used to store the frames you have expanded the video memory needs to almost 400 times the original size.

Even if you consider better options, store frames as Jpeg Blobs (near one to one of bin size) set at very low compression (eg jpeg quality 0.4) each frame requires ~12kb. The final RAM need to store the frames is (1 * 60 * 60 * 20fps + 25 * 60 * 20fps) * 12kb * 1024 = 1.2Gig (do-able on PC's and laptops if you don't count the IMG element overhead)

Though it will fit memory the processing will no less require a lot of time to render, compress, and store.

The Sad truth

Until browsers come with a way to use a disk based cache for large data needs, video editing like applications will not be able to provide fast random access frame editing like traditional native video editing apps.

What to do?

I did notice that there was a image element id named gif which hints that the application may be a video to gif like thing.

Again this will be slow, but it is do-able if you compress the video frames to gif as you go. Get video frame, quantize color, compress to GIF, delete frame. Use webWorkers to pipeline the quantize, and compress steps. Ensure that unsaved gif data size does not crash the page.

You should also consider options for resolution and frame rate reduction. Also add warnings for long (large) videos. It will be slow, there is nothing you can do, that is the nature of videos,

requestAnimationFrame could be substituted for setTimeout where necessary in the existing code.