Transforming an image to animating Particles with Web Audio

We're going to transform an image to particles and then animate those with values from Web Audio

9 min read

At my company Level30Wizards, we experiment with web techniques to learn how we can implement those for clients. Check it out on our website Disclaimer: This article contains a lot of WebGL shader code, which I, as of today, don't understand.

Tech stack

HTML
CSS
P5js to get audio values from audio clips
WebPack for compiling
ThreeJS / WebGL for creating and animating particles

Libraries

ControlKit - GUI
gsap - animation platform
glslify - module system for GLSL
stats.js - performance monitor
Three.js - WebGL library

Goal

We want to transform a photo into particles and then animate those particles based on values we get from audio.

Threejs Scene

First of all, we create a new scene with Three.js

1// scene
2this.scene = new THREE.Scene();
3
4// camera
5this.camera = new THREE.PerspectiveCamera(
6  100,
7  window.innerWidth / window.innerHeight,
8  1,
9  1000
10);
11this.camera.position.z = 300;
12
13// renderer
14this.renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
15
16// clock
17this.clock = new THREE.Clock(true);

Next to that we add some standard resize functions. Then, we initialize our particle function.

1initParticles() {
2  this.particles = new Particles(this);
3  this.scene.add(this.particles.container);
4}
5
6// Create a draw function
7draw() {
8  this.renderer.render(this.scene, this.camera);
9}
10
11// Create an update function
12update() {
13  const delta = this.clock.getDelta();
14
15  if (this.particles) this.particles.update(delta);
16}

Particles

Classes?! Well, this project is an experiment. So I combined work from other people into a new one. #FullStackOverflowDeveloper

1import * as THREE from "three";
2import { gsap } from "gsap";
3
4const glslify = require("glslify");
5
6export default class Particles {
7  constructor(webgl) {
8    this.webgl = webgl;
9    this.container = new THREE.Object3D();
10    this.mapBass = 0;
11    this.mapTremble = 0;
12    this.mapMid = 0;
13    this.mapLowMid = 0;
14    this.mapHighMid = 0;
15    this.root = document.documentElement;
16    this.playPauseButton = document.querySelector(
17      '[aria-label="toggle-music"]'
18    );
19  }

Initialize particles

This code will initialize the particle animation, load the audio file and gathers the audio values.

1init(src) {
2  gsap.fromTo(
3    ".loading",
4    {
5      autoAlpha: 1,
6    },
7    {
8      duration: 0.5,
9      autoAlpha: 0,
10    }
11  );
12
13  const loader = new THREE.TextureLoader();
14  loader.load(src, (texture) => {
15    this.texture = texture;
16    this.texture.minFilter = THREE.LinearFilter;
17    this.texture.magFilter = THREE.LinearFilter;
18    this.texture.format = THREE.RGBFormat;
19
20    this.width = texture.image.width;
21    this.height = texture.image.height;
22
23    this.initPoints(true);
24    this.resize();
25    this.hide();
26  });
27
28  const s = (p) => {
29    let audio, fft;
30
31    p.preload = () => {
32      audio = p.loadSound("/audio/music-file.mp3", () => {
33        this.playPauseButton.addEventListener("click", () => {
34          if (audio.isPlaying()) {
35            audio.pause();
36            gsap.to("#pause", {
37              autoAlpha: 0,
38              duration: 0.15,
39            });
40            gsap.to("#play", {
41              autoAlpha: 1,
42              duration: 0.15,
43              delay: 0.15,
44            });
45          } else {
46            audio.loop();
47            gsap.to("#play", {
48              autoAlpha: 0,
49              duration: 0.15,
50            });
51            gsap.to("#pause", {
52              autoAlpha: 1,
53              duration: 0.15,
54              delay: 0.15,
55            });
56          }
57        });
58      });
59    };
60    p.setup = () => {
61      fft = new p5.FFT();
62      const allowButton = document.querySelector("#allow-music");
63
64      allowButton.addEventListener("click", () => {
65        audio.loop();
66        this.show();
67        gsap.to([allowButton, ".container img"], {
68          autoAlpha: 0,
69          duration: 0.5,
70        });
71      });
72    };
73    p.draw = () => {
74      fft.analyze();
75
76      const bass = fft.getEnergy("bass");
77      const treble = fft.getEnergy("treble");
78      const lowMid = fft.getEnergy("lowMid");
79      const mid = fft.getEnergy("mid");
80      const highMid = fft.getEnergy("highMid");
81
82      this.mapBass = p.map(bass, 0, 255, 0, 2.0);
83      this.mapTremble = p.map(treble, 0, 255, 0, 2.0);
84      this.mapLowMid = p.map(lowMid, 0, 255, 0, 2.0);
85      this.mapMid = p.map(mid, 0, 255, 0, 2.0);
86      this.mapHighMid = p.map(highMid, 0, 255, 0, 2.0);
87    };
88  };
89  new p5(s);
90}

Image to particles

This code was written by Bruno Imbrizi

1initPoints(discard) {
2  this.numPoints = this.width * this.height;
3
4  let numVisible = this.numPoints;
5  let threshold = 0;
6  let originalColors;
7
8  if (discard) {
9    // discard pixels darker than threshold #22
10    numVisible = 0;
11    threshold = 34;
12
13    const img = this.texture.image;
14    const canvas = document.createElement("canvas");
15    const ctx = canvas.getContext("2d");
16
17    canvas.width = this.width;
18    canvas.height = this.height;
19    ctx.scale(1, -1);
20    ctx.drawImage(img, 0, 0, this.width, this.height * -1);
21
22    const imgData = ctx.getImageData(0, 0, canvas.width, canvas.height);
23    originalColors = Float32Array.from(imgData.data);
24
25    for (let i = 0; i < this.numPoints; i++) {
26      if (originalColors[i * 4 + 0] > threshold) numVisible++;
27    }
28  }
29
30  const uniforms = {
31    uTime: { value: 0 },
32    uRandom: { value: 1.0 },
33    uDepth: { value: 100.0 },
34    uSize: { value: 1.58 },
35    uTextureSize: { value: new THREE.Vector2(this.width, this.height) },
36    uTexture: { value: this.texture },
37    uTouch: { value: null },
38    // This variable was added to use audio values in the vertex shader
39    music: { value: null },
40  };
41
42  const material = new THREE.RawShaderMaterial({
43    uniforms,
44    vertexShader: glslify(require("../../../shaders/particle.vert")),
45    fragmentShader: glslify(require("../../../shaders/particle.frag")),
46    depthTest: false,
47    transparent: true,
48  });
49
50  const geometry = new THREE.InstancedBufferGeometry();
51
52  // positions
53  const positions = new THREE.BufferAttribute(new Float32Array(4 * 3), 3);
54  positions.setXYZ(0, -0.5, 0.5, 0.0);
55  positions.setXYZ(1, 0.5, 0.5, 0.0);
56  positions.setXYZ(2, -0.5, -0.5, 0.0);
57  positions.setXYZ(3, 0.5, -0.5, 0.0);
58  geometry.addAttribute("position", positions);
59
60  // uvs
61  const uvs = new THREE.BufferAttribute(new Float32Array(4 * 2), 2);
62  uvs.setXYZ(0, 0.0, 0.0);
63  uvs.setXYZ(1, 1.0, 0.0);
64  uvs.setXYZ(2, 0.0, 1.0);
65  uvs.setXYZ(3, 1.0, 1.0);
66  geometry.addAttribute("uv", uvs);
67
68  // index
69  geometry.setIndex(
70    new THREE.BufferAttribute(new Uint16Array([0, 2, 1, 2, 3, 1]), 1)
71  );
72
73  const indices = new Uint16Array(numVisible);
74  const offsets = new Float32Array(numVisible * 3);
75  const angles = new Float32Array(numVisible);
76
77  for (let i = 0, j = 0; i < this.numPoints; i++) {
78    if (discard && originalColors[i * 4 + 0] <= threshold) continue;
79
80    offsets[j * 3 + 0] = i % this.width;
81    offsets[j * 3 + 1] = Math.floor(i / this.width);
82
83    indices[j] = i;
84
85    angles[j] = Math.random() * Math.PI;
86
87    j++;
88  }
89
90  geometry.addAttribute(
91    "pindex",
92    new THREE.InstancedBufferAttribute(indices, 1, false)
93  );
94  geometry.addAttribute(
95    "offset",
96    new THREE.InstancedBufferAttribute(offsets, 3, false)
97  );
98  geometry.addAttribute(
99    "angle",
100    new THREE.InstancedBufferAttribute(angles, 1, false)
101  );
102
103  this.object3D = new THREE.Mesh(geometry, material);
104  this.container.add(this.object3D);
105}

Cool, so now the image will be transformed to particles.

Updating Vertex Shader variables with ThreeJS

Now, we need to update values in the Vertex Shader for all the particles to animate. We do this by utilizing a RequestAnimationFrame that updates the WebGL values based on your refresh rate.

1update(delta) {
2  if (!this.object3D) return;
3  if (this.touch) this.touch.update();
4  this.object3D.material.uniforms.music.value = Math.max(1, this.mapBass);
5
6  this.object3D.material.uniforms.uTime.value += delta;
7}

Vertex Shader

This is where it gets magical for me. I don't have a lot of experience with WebGL, Vertex shaders or Fragment shaders. But after tinkering a bit, I was able to use the music variable in the animation.

1// @author brunoimbrizi / http://brunoimbrizi.com
2
3precision highp float;
4
5attribute float pindex;
6attribute vec3 position;
7attribute vec3 offset;
8attribute vec2 uv;
9attribute float angle;
10
11uniform mat4 modelViewMatrix;
12uniform mat4 projectionMatrix;
13
14// Create music variable
15uniform float music;
16
17uniform float uTime;
18uniform float uRandom;
19uniform float uDepth;
20uniform float uSize;
21uniform vec2 uTextureSize;
22uniform sampler2D uTexture;
23uniform sampler2D uTouch;
24
25varying vec2 vPUv;
26varying vec2 vUv;
27
28#pragma glslify: snoise2 = require(glsl-noise/simplex/2d)
29
30float random(float n) {
31    return fract(sin(n) * 43758.5453123);
32}
33
34void main() {
35    vUv = uv;
36
37    // particle uv
38    vec2 puv = offset.xy / uTextureSize;
39    vPUv = puv;
40
41    // pixel color
42    vec4 colA = texture2D(uTexture, puv);
43    float grey = colA.r * 0.21 + colA.g * 0.71 + colA.b * 0.07;
44
45    // displacement
46    vec3 displaced = offset;
47    // randomise and utilize music variable
48    displaced.xy += vec2(random(pindex) - 0.25, random(offset.x + pindex) - 0.25) * uRandom * (music * 0.2);
49    float rndz = (random(pindex) + snoise_1_2(vec2(pindex * 0.1, music * 0.1)));
50    displaced.z += rndz * (random(pindex) * uDepth) * (music * 6.0);
51    // center
52    displaced.xy -= uTextureSize * 0.5;
53
54    // touch
55    float t = texture2D(uTouch, puv).r;
56    displaced.z += t * 20.0 * rndz;
57    displaced.x += cos(angle) * t * 20.0 * rndz;
58    displaced.y += sin(angle) * t * 20.0 * rndz;
59
60    // particle size
61    float psize = (snoise_1_2(vec2(uTime* music, pindex) * 0.5) + 2.0) * music * 0.5;
62    psize *= max(grey, 0.2);
63    psize *= uSize;
64
65    // final position
66    vec4 mvPosition = modelViewMatrix * vec4(displaced, 1.0);
67    mvPosition.xyz += position * psize;
68    vec4 finalPosition = projectionMatrix * mvPosition;
69
70    gl_Position = finalPosition;
71}

Fragment shader

To give some texture to the particles we add a fragment shader.

1// @author brunoimbrizi / http://brunoimbrizi.com
2
3precision highp float;
4
5uniform sampler2D uTexture;
6uniform float uTime;
7
8varying vec2 vPUv;
9varying vec2 vUv;
10
11void main() {
12    vec4 color = vec4(0.0);
13    vec2 uv = vUv;
14    vec2 puv = vPUv;
15
16    // pixel color
17    vec4 colA = texture2D(uTexture, puv);
18
19    // greyscale
20    float grey = colA.r * 0.21 + colA.g * 0.71 + colA.b * 0.07;
21    vec4 colB = vec4(grey, grey, grey, 1.0);
22
23    // circle
24    float border = 0.3;
25    float radius = 0.5;
26    float dist = radius - distance(uv, vec2(0.5));
27    float t = smoothstep(0.0, border, dist);
28
29    uv = 2.0 * uv - 1.0;
30
31    // Double the speed
32    float wave = sin(uTime * 2.0);
33
34    // Scale to make the circle bigger so it reaches the far edges
35    float circle = (uv.x * uv.x + uv.y * uv.y) * 0.2;
36    vec4 color1 = vec4(0.1, 0.2, 0.8, 1.0); // Red
37    vec4 color2 = vec4(0.2, 0.1, 0.8, 1.0); // Blue
38
39    color = mix(color1, color2, circle + wave);
40    // final color
41    // color = colB;
42    color.a = t;
43
44    gl_FragColor = color;
45}

Finalize

Ofcourse, more goes into the experiment than all this code. But I don't think anyone will learn from the rest of it. By now I learned how to use ThreeJS to update variables in shaders, in an inefficient way. I understand how transforming an image to particles works, how you update a vertex shader and how you combine audio values with web animation.

Demo

Have a look at the demo to see how it all comes together. GitHub Repo

Thanks for reading!

I hope someone somewhere learned something via this post! If you did, please consider sharing the article. If you have some web animation concept or library you want to learn about, let me know!