Transforming an image to animating Particles with Web Audio

We're going to transform an image to particles and then animate those with values from Web Audio

9 min read
by Mees Rutten | Wed Aug 19 2020

At my company Level30Wizards, we experiment with web techniques to learn how we can implement those for clients. Check it out on our website Disclaimer: This article contains a lot of WebGL shader code, which I, as of today, don't understand.

Tech stack

  • HTML
  • CSS
  • P5js to get audio values from audio clips
  • WebPack for compiling
  • ThreeJS / WebGL for creating and animating particles

Libraries

Goal

We want to transform a photo into particles and then animate those particles based on values we get from audio.

Threejs Scene

First of all, we create a new scene with Three.js

1// scene
2this.scene = new THREE.Scene();
3
4// camera
5this.camera = new THREE.PerspectiveCamera(
6 100,
7 window.innerWidth / window.innerHeight,
8 1,
9 1000
10);
11this.camera.position.z = 300;
12
13// renderer
14this.renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
15
16// clock
17this.clock = new THREE.Clock(true);

Next to that we add some standard resize functions. Then, we initialize our particle function.

1initParticles() {
2 this.particles = new Particles(this);
3 this.scene.add(this.particles.container);
4}
5
6// Create a draw function
7draw() {
8 this.renderer.render(this.scene, this.camera);
9}
10
11// Create an update function
12update() {
13 const delta = this.clock.getDelta();
14
15 if (this.particles) this.particles.update(delta);
16}

Particles

Classes?! Well, this project is an experiment. So I combined work from other people into a new one. #FullStackOverflowDeveloper

1import * as THREE from "three";
2import { gsap } from "gsap";
3
4const glslify = require("glslify");
5
6export default class Particles {
7 constructor(webgl) {
8 this.webgl = webgl;
9 this.container = new THREE.Object3D();
10 this.mapBass = 0;
11 this.mapTremble = 0;
12 this.mapMid = 0;
13 this.mapLowMid = 0;
14 this.mapHighMid = 0;
15 this.root = document.documentElement;
16 this.playPauseButton = document.querySelector(
17 '[aria-label="toggle-music"]'
18 );
19 }

Initialize particles

This code will initialize the particle animation, load the audio file and gathers the audio values.

1init(src) {
2 gsap.fromTo(
3 ".loading",
4 {
5 autoAlpha: 1,
6 },
7 {
8 duration: 0.5,
9 autoAlpha: 0,
10 }
11 );
12
13 const loader = new THREE.TextureLoader();
14 loader.load(src, (texture) => {
15 this.texture = texture;
16 this.texture.minFilter = THREE.LinearFilter;
17 this.texture.magFilter = THREE.LinearFilter;
18 this.texture.format = THREE.RGBFormat;
19
20 this.width = texture.image.width;
21 this.height = texture.image.height;
22
23 this.initPoints(true);
24 this.resize();
25 this.hide();
26 });
27
28 const s = (p) => {
29 let audio, fft;
30
31 p.preload = () => {
32 audio = p.loadSound("/audio/music-file.mp3", () => {
33 this.playPauseButton.addEventListener("click", () => {
34 if (audio.isPlaying()) {
35 audio.pause();
36 gsap.to("#pause", {
37 autoAlpha: 0,
38 duration: 0.15,
39 });
40 gsap.to("#play", {
41 autoAlpha: 1,
42 duration: 0.15,
43 delay: 0.15,
44 });
45 } else {
46 audio.loop();
47 gsap.to("#play", {
48 autoAlpha: 0,
49 duration: 0.15,
50 });
51 gsap.to("#pause", {
52 autoAlpha: 1,
53 duration: 0.15,
54 delay: 0.15,
55 });
56 }
57 });
58 });
59 };
60 p.setup = () => {
61 fft = new p5.FFT();
62 const allowButton = document.querySelector("#allow-music");
63
64 allowButton.addEventListener("click", () => {
65 audio.loop();
66 this.show();
67 gsap.to([allowButton, ".container img"], {
68 autoAlpha: 0,
69 duration: 0.5,
70 });
71 });
72 };
73 p.draw = () => {
74 fft.analyze();
75
76 const bass = fft.getEnergy("bass");
77 const treble = fft.getEnergy("treble");
78 const lowMid = fft.getEnergy("lowMid");
79 const mid = fft.getEnergy("mid");
80 const highMid = fft.getEnergy("highMid");
81
82 this.mapBass = p.map(bass, 0, 255, 0, 2.0);
83 this.mapTremble = p.map(treble, 0, 255, 0, 2.0);
84 this.mapLowMid = p.map(lowMid, 0, 255, 0, 2.0);
85 this.mapMid = p.map(mid, 0, 255, 0, 2.0);
86 this.mapHighMid = p.map(highMid, 0, 255, 0, 2.0);
87 };
88 };
89 new p5(s);
90}

Image to particles

This code was written by Bruno Imbrizi

1initPoints(discard) {
2 this.numPoints = this.width * this.height;
3
4 let numVisible = this.numPoints;
5 let threshold = 0;
6 let originalColors;
7
8 if (discard) {
9 // discard pixels darker than threshold #22
10 numVisible = 0;
11 threshold = 34;
12
13 const img = this.texture.image;
14 const canvas = document.createElement("canvas");
15 const ctx = canvas.getContext("2d");
16
17 canvas.width = this.width;
18 canvas.height = this.height;
19 ctx.scale(1, -1);
20 ctx.drawImage(img, 0, 0, this.width, this.height * -1);
21
22 const imgData = ctx.getImageData(0, 0, canvas.width, canvas.height);
23 originalColors = Float32Array.from(imgData.data);
24
25 for (let i = 0; i < this.numPoints; i++) {
26 if (originalColors[i * 4 + 0] > threshold) numVisible++;
27 }
28 }
29
30 const uniforms = {
31 uTime: { value: 0 },
32 uRandom: { value: 1.0 },
33 uDepth: { value: 100.0 },
34 uSize: { value: 1.58 },
35 uTextureSize: { value: new THREE.Vector2(this.width, this.height) },
36 uTexture: { value: this.texture },
37 uTouch: { value: null },
38 // This variable was added to use audio values in the vertex shader
39 music: { value: null },
40 };
41
42 const material = new THREE.RawShaderMaterial({
43 uniforms,
44 vertexShader: glslify(require("../../../shaders/particle.vert")),
45 fragmentShader: glslify(require("../../../shaders/particle.frag")),
46 depthTest: false,
47 transparent: true,
48 });
49
50 const geometry = new THREE.InstancedBufferGeometry();
51
52 // positions
53 const positions = new THREE.BufferAttribute(new Float32Array(4 * 3), 3);
54 positions.setXYZ(0, -0.5, 0.5, 0.0);
55 positions.setXYZ(1, 0.5, 0.5, 0.0);
56 positions.setXYZ(2, -0.5, -0.5, 0.0);
57 positions.setXYZ(3, 0.5, -0.5, 0.0);
58 geometry.addAttribute("position", positions);
59
60 // uvs
61 const uvs = new THREE.BufferAttribute(new Float32Array(4 * 2), 2);
62 uvs.setXYZ(0, 0.0, 0.0);
63 uvs.setXYZ(1, 1.0, 0.0);
64 uvs.setXYZ(2, 0.0, 1.0);
65 uvs.setXYZ(3, 1.0, 1.0);
66 geometry.addAttribute("uv", uvs);
67
68 // index
69 geometry.setIndex(
70 new THREE.BufferAttribute(new Uint16Array([0, 2, 1, 2, 3, 1]), 1)
71 );
72
73 const indices = new Uint16Array(numVisible);
74 const offsets = new Float32Array(numVisible * 3);
75 const angles = new Float32Array(numVisible);
76
77 for (let i = 0, j = 0; i < this.numPoints; i++) {
78 if (discard && originalColors[i * 4 + 0] <= threshold) continue;
79
80 offsets[j * 3 + 0] = i % this.width;
81 offsets[j * 3 + 1] = Math.floor(i / this.width);
82
83 indices[j] = i;
84
85 angles[j] = Math.random() * Math.PI;
86
87 j++;
88 }
89
90 geometry.addAttribute(
91 "pindex",
92 new THREE.InstancedBufferAttribute(indices, 1, false)
93 );
94 geometry.addAttribute(
95 "offset",
96 new THREE.InstancedBufferAttribute(offsets, 3, false)
97 );
98 geometry.addAttribute(
99 "angle",
100 new THREE.InstancedBufferAttribute(angles, 1, false)
101 );
102
103 this.object3D = new THREE.Mesh(geometry, material);
104 this.container.add(this.object3D);
105}

Cool, so now the image will be transformed to particles.

Updating Vertex Shader variables with ThreeJS

Now, we need to update values in the Vertex Shader for all the particles to animate. We do this by utilizing a RequestAnimationFrame that updates the WebGL values based on your refresh rate.

1update(delta) {
2 if (!this.object3D) return;
3 if (this.touch) this.touch.update();
4 this.object3D.material.uniforms.music.value = Math.max(1, this.mapBass);
5
6 this.object3D.material.uniforms.uTime.value += delta;
7}

Vertex Shader

This is where it gets magical for me. I don't have a lot of experience with WebGL, Vertex shaders or Fragment shaders. But after tinkering a bit, I was able to use the music variable in the animation.

1// @author brunoimbrizi / http://brunoimbrizi.com
2
3precision highp float;
4
5attribute float pindex;
6attribute vec3 position;
7attribute vec3 offset;
8attribute vec2 uv;
9attribute float angle;
10
11uniform mat4 modelViewMatrix;
12uniform mat4 projectionMatrix;
13
14// Create music variable
15uniform float music;
16
17uniform float uTime;
18uniform float uRandom;
19uniform float uDepth;
20uniform float uSize;
21uniform vec2 uTextureSize;
22uniform sampler2D uTexture;
23uniform sampler2D uTouch;
24
25varying vec2 vPUv;
26varying vec2 vUv;
27
28#pragma glslify: snoise2 = require(glsl-noise/simplex/2d)
29
30float random(float n) {
31 return fract(sin(n) * 43758.5453123);
32}
33
34void main() {
35 vUv = uv;
36
37 // particle uv
38 vec2 puv = offset.xy / uTextureSize;
39 vPUv = puv;
40
41 // pixel color
42 vec4 colA = texture2D(uTexture, puv);
43 float grey = colA.r * 0.21 + colA.g * 0.71 + colA.b * 0.07;
44
45 // displacement
46 vec3 displaced = offset;
47 // randomise and utilize music variable
48 displaced.xy += vec2(random(pindex) - 0.25, random(offset.x + pindex) - 0.25) * uRandom * (music * 0.2);
49 float rndz = (random(pindex) + snoise_1_2(vec2(pindex * 0.1, music * 0.1)));
50 displaced.z += rndz * (random(pindex) * uDepth) * (music * 6.0);
51 // center
52 displaced.xy -= uTextureSize * 0.5;
53
54 // touch
55 float t = texture2D(uTouch, puv).r;
56 displaced.z += t * 20.0 * rndz;
57 displaced.x += cos(angle) * t * 20.0 * rndz;
58 displaced.y += sin(angle) * t * 20.0 * rndz;
59
60 // particle size
61 float psize = (snoise_1_2(vec2(uTime* music, pindex) * 0.5) + 2.0) * music * 0.5;
62 psize *= max(grey, 0.2);
63 psize *= uSize;
64
65 // final position
66 vec4 mvPosition = modelViewMatrix * vec4(displaced, 1.0);
67 mvPosition.xyz += position * psize;
68 vec4 finalPosition = projectionMatrix * mvPosition;
69
70 gl_Position = finalPosition;
71}

Fragment shader

To give some texture to the particles we add a fragment shader.

1// @author brunoimbrizi / http://brunoimbrizi.com
2
3precision highp float;
4
5uniform sampler2D uTexture;
6uniform float uTime;
7
8varying vec2 vPUv;
9varying vec2 vUv;
10
11void main() {
12 vec4 color = vec4(0.0);
13 vec2 uv = vUv;
14 vec2 puv = vPUv;
15
16 // pixel color
17 vec4 colA = texture2D(uTexture, puv);
18
19 // greyscale
20 float grey = colA.r * 0.21 + colA.g * 0.71 + colA.b * 0.07;
21 vec4 colB = vec4(grey, grey, grey, 1.0);
22
23 // circle
24 float border = 0.3;
25 float radius = 0.5;
26 float dist = radius - distance(uv, vec2(0.5));
27 float t = smoothstep(0.0, border, dist);
28
29 uv = 2.0 * uv - 1.0;
30
31 // Double the speed
32 float wave = sin(uTime * 2.0);
33
34 // Scale to make the circle bigger so it reaches the far edges
35 float circle = (uv.x * uv.x + uv.y * uv.y) * 0.2;
36 vec4 color1 = vec4(0.1, 0.2, 0.8, 1.0); // Red
37 vec4 color2 = vec4(0.2, 0.1, 0.8, 1.0); // Blue
38
39 color = mix(color1, color2, circle + wave);
40 // final color
41 // color = colB;
42 color.a = t;
43
44 gl_FragColor = color;
45}

Finalize

Ofcourse, more goes into the experiment than all this code. But I don't think anyone will learn from the rest of it. By now I learned how to use ThreeJS to update variables in shaders, in an inefficient way. I understand how transforming an image to particles works, how you update a vertex shader and how you combine audio values with web animation.

Demo

Have a look at the demo to see how it all comes together. GitHub Repo


Thanks for reading!

I hope someone somewhere learned something via this post! If you did, please consider sharing the article. If you have some web animation concept or library you want to learn about, let me know!

Special thanks to @moniac for sharing this template!

by Mees Rutten @

All rights reserved