Shader to Produce Sound?

[Pryme8]

Are we able to do audio output off a shader like:
https://www.shadertoy.com/view/ldfSW2

Or do I need to rely on html5 audio api methods to procedurally generate noise?

Here is how they do it:
https://stackoverflow.com/questions/34859701/how-do-shadertoys-audio-shaders-work

Which means I doubt we have a like method, but from what this describes I bet we could pull it off decently easy.

[Nabroski]

I would start to send sound to shader that a bit easier to pull off from understanding. 

[Pryme8]

Yeah, but I want to generate the sound procedurally on the GPU.  Its part of my asset-less game project.

[Nabroski]

Understand that, but when you know how to send sound you only a step away from generate it. We can make this a community project if you like to.

I will use WegGL2 32bit Float Textures, later you can convert to WebGL1 equivalent. Hope that is ok. 

Do you have experience with HTML5 Audio??? i have only limited knowledge. Would be a good start, if someone show me how to get a 32FloatArray out of Audio context.  A 440Hz Sine would be also nice. hahah, im a noob. 

[Pryme8]

yes I do, I have programmed full synths.

https://youtu.be/E_HpCoOGMQ8?t=55s

here is a lame example from like years ago.

Ive got an idea. Hang tight.

[Pryme8]

I think there might be a possible bug in 

renderTargetTexture.readPixels();

1. buffer:[Exception: TypeError: Method get %TypedArray%.prototype.buffer called on incompatible receiver [object Object] at TypedArray.get buffer [as buffer] (<anonymous>) at TypedArray.remoteFunction (<anonymous>:2:14)]
2. byteLength:[Exception: TypeError: Method get TypedArray.prototype.byteLength called on incompatible receiver [object Object] at TypedArray.get byteLength [as byteLength] (<anonymous>) at TypedArray.remoteFunction (<anonymous>:2:14)]
3. byteOffset:[Exception: TypeError: Method get TypedArray.prototype.byteOffset called on incompatible receiver [object Object] at TypedArray.get byteOffset [as byteOffset] (<anonymous>) at TypedArray.remoteFunction (<anonymous>:2:14)

This is pretty odd.  Im pretty sure if I can get this byteArray to construct correct, we should hear a horrifying noise being played.
http://www.babylonjs-playground.com/#CL6JEL#4

Nevermind Im dumb, I got this.

Getting closer with this:
http://www.babylonjs-playground.com/#CL6JEL#6
Cause at least I'm starting to get the correct bucket size now.

[Nabroski]

@PeapBoy If you find time i need an example of gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGB32F, data, 1, 0, gl.RGB, gl.FLOAT, null); in Babylonjs. Thank you

[Nabroski]

@Pryme8 Don't rush, i need to test things, i keep this thread in mind. will be back asap.  

[Nabroski]

maybe its also easier in terms of babylonjs to work with an Unit8Array. (But then you have to pack it, and unpack it, and pack it ...)

[Nabroski]

ok got it.
i will continue tomorrow. out of time right now.

https://www.babylonjs-playground.com/#0GT2HX

[Pryme8]

awesome I will check out whats going on when I get home and have speakers!  Ill use your rtt method, and combine it with a shader to output a image that gets converted into a byteArray.

So the way that I was conceptualizing this is:
44100 sample rate
10 seconds of sample

Gives us 441000 bytes
So in an Image with our max width being 4k, we get a height of 110.25 which we ceil to 111;

so now we have a 4k by 111px image which if we wanted to be fancy we could do a square image by doing the sqrt of (441000) which equals approx 665.  So instead of a 4k by 111px lets do a 665x665px image that becomes our rtt.

Now in our sound generation shader we pass the uniforms bufferSize vec2(665.), duration float 10.0.  (we might have to do different ones later but this is a prototype)

Then we need to get our sampling space correct which can effectively be a float that each pxl needs assigned to it.
so something like 
vec2 timeSpace =  vec2(1.0/(vec2(duration)/bufferSize));
float timeCoords = (vUV.x*timeSpace.x)+(((vUV.y-1.0)*timeSpace.y)*timeSpace.x);

^^^^ that math is not correct I bet, but hopefully gets the idea across.
So from there each pxl effectively has a timeCoord which should be a unique float starting at 0 and hopefully ending at 10, every pixel after 10 could be ignored unless you wanted a overflow gutter for the sound (useless most of the time)

Now from there we pass the timeCoords through various generators that change each pxl accordingly.
red => LeftChannel Min
green => LeftChannel Max
blue => RightChannel Min
alpha => RightChannel Max

then whabam, grab the rtt internal texture when its ready and pass that to the audio context to spin up the buffer.
 

This is gonna be dope, if we do this right we could effectively generate whole songs off the GPU and create a whole niche for music/sound production.

[Nabroski]

here comes the sun

https://www.babylonjs-playground.com/#0GT2HX#1

[Pryme8]

Close, looks like you are doing a mono deployment though.
What you made and what I got sitting at the house, I think we can make this a real thing.

there is no reason to add the iTime in either, that will just mess up the sound generation once we are using functions to make the sounds we want.

https://www.babylonjs-playground.com/#0GT2HX#2
 

[Pryme8]

https://www.babylonjs-playground.com/#16HY5Y#3

This is more or less the structure I was hoping for.  Have not gotten this to play a sound yet, so maybe you can help with that part.

Trying to get it so the sound can be generated in its entirety in one pass and not have a uniform be updated constantly.  Basically one pass its done and its now a usable asset.

[Nabroski]

hello
ok, i will look into it. 
Yes, stereo is on my to do list. Their is actually a smarter way to to it with Float32 Textures, but i save this one for a weekend.
And meanwhile: this is the base WebGL (easy) : https://jsfiddle.net/nabr/xam09g6c/ 
You need somehow adapt it into you project. Not sure how it should work with Rendertarget textures. What i trying to say, i keep this thread in mind. Will be back asap. 

[Nabroski]

@Pryme8 Problem solved. Or do you have additional questions. Feel free to ask. 

[Pryme8]

Still trying to get a buffer to render out completely, I think we should leave this open for now.

The method that we have that "works" right now generates constantly and is not ideal for "precompiling" sounds procedurally and storing them in the ram for later use.

I have been doing a bunch of side reading on how the buffers are stored and decoded and think Ill have a more robust solution soon.

I just dont get to look at this really while at work, cause we dont have any speakers and only get to check it out when I get home and have free time.  So its not a major priority.

[Nabroski]

On 7/25/2018 at 2:09 AM, Pryme8 said:

there is no reason to add the iTime in either, that will just mess up the sound generation once we are using functions to make the sounds we want.

https://www.babylonjs-playground.com/#0GT2HX#2
 

iTime is in this context maybe not a proper name, but you need to find the location of a pixel in an array. so the formula 
location = x + y * width; is just right. Here is a good tutorial: https://processing.org/tutorials/pixels/
I chosen iTime becourse you usually want do more stuff inside main(). 
- RenderTarget Texture their is not way you can pull data ( buffer ) out of it. You need something with r e a d option. 

Ok. Good luck. Keep us posted.

[Pryme8]

I am already pulling the data buffers out of the rtt's, as an Uint8Array.
https://www.babylonjs-playground.com/#16HY5Y#3 <- you can see it in the console report.

Getting pixel data from rtt's is super easy, just need to convert it and then pass it to a Audio Node.

I think I have it figured out, I just have not had a chance to do anything with it for a day or two.

[Nabroski]

yes i think you almost their. you can experiment with rawtextures ( i think their are closer to metal )
https://www.babylonjs-playground.com/#16HY5Y#4

[Pryme8]

http://www.bitsnbites.eu/hello-world/

Does this mean i can just convert the buffer to a base64 and then strap the correct header to it?
Or maybe even as a  blob?

[Nabroski]

YES! Before WebAudio people done all kind of adventures, still the preferred technique in size coding and testing stuff. 

• https://gist.github.com/xem/670dec8e70815842eb95
• https://js1k.com/2012-love/details/1265