Texture 3D, is it real?

[jonathanlurie]

Hi all,
I am new here and new with BJS so I might ask for something obvious. Though, I looked well in the resources, how-tos and was digging pretty deeply into BJS codebase and couldn't find any answer. Also, in case of lack of satisfying answers, I'll also post my own solution to my problem (a hack).

I want to load a 3D texture of a brain (MRI) and display it on 3 orthogonal planes, then i can move my plane-set around, and even spin it to display oblique slices. On my GLSL code, i need to load my 3D texture as a sampler3D and lookup for some world coordinates to feed gl_FragColor with the colors (actually LUMINANCE) from the texture (Look at the screenshot). And it works, but with a hack, because I could not find how to build a raw 3D texture object because the `RawTexture` class, even though it inherits `Texture`, will only call `engine.createRawTexture()` and never `engine.createRawTexture3D()`.

At first, I thought I would just create my texture 3D object calling directly `engine.createRawTexture3D()` but this doesn't create an instance of the prototype/class `Texture`, only an `InternalTexture`, so when the texture validation time comes, calling methods like `myTexture.isReady()` would fail because `isReady()` (as well as other methods) are part of the `Texture` prototype/class.

What is missing here is the 3D counterpart of `RawTexture`, so I made it myself and called it `RawTexture3D`, that I declared somewhere in my code where I have access to the BABYLON module:

/*
(In ES6 syntax)

Here, 'data' is an Uint8Array and the rest is the usual things
*/
class RawTexture3D extends BABYLON.Texture {

    constructor(data, width, height, depth, format, scene,
                generateMipMaps = true, invertY = false,
                samplingMode = BABYLON.Texture.TRILINEAR_SAMPLINGMODE,
                type = BABYLON.Engine.TEXTURETYPE_UNSIGNED_INT)
    {
        super(null, scene, !generateMipMaps, invertY);
      
        this._texture = scene.getEngine().createRawTexture3D(
            data,
            width,
            height,
            depth,
            format,
            generateMipMaps,
            invertY,
            samplingMode
        )

        this.is3D = true;
    }
}

This works well but it would be much better if it was part of the core, especially for maintenance. Say tomorrow `Texture.is3D` gets renamed into `Texture._is3D`, I would probably find out the hard way...

If you have another solution to this hack, please share! We are not a lot having to work with 3D textures, so let's help each other! If you were looking for a hack, then this one is yours, and if you are one of the core dev of BabylonJS, then, could you add that to the core, pleeeeaaase? (or maybe i will do a PR at some point but I am not super familiar with TS) And also, thanks for this awesome lib! (I've started to use it yesterday so I don't even fully realize how awesome it is)

Cheers.

[Guest]

This is cool!

And you are right, we should expose a RawTexture3D. I will expose it for the next commit (in a couple of hours)

Please grab it and let me know if it works (I won't be against a cool sample with your data in the PG ;))

 

[jonathanlurie]

YEAH! it works perfectly! Thanks for adding that to the core!

For loading the MRI data I use Pixpipe , which is a lib I am developing at the Montreal Neurological Hospital. I will share this example in a git page because I am not too sure how to deal with dependencies and remote data in a PG, and also the MRI has to be loaded in advance and use a callback that only then creates the scene -- I'm not sure we can do that in the playground. I will update this thread when I have a cleaner source to share! 

But in the meantime, here is a youtube capture of it in action: 

 

Cheers!

 

[Pryme8]

You should make a custom shader for it and discard back.

[jonathanlurie]

Well, it's already a custom shader and by discarding back, do you mean discarding the black surrounding? If so, then no, researchers and clinicians are used to keep the surrounding because getting rid of it (with a simple threshold) could mean discarding important signal.

[Guest]

I would love to share a demo on babylonjs.com!

[jonathanlurie]

Hi @Deltakosh , I am currently cleaning up this example a bit to make it more robust because for now, a lot of things are dirty and hardcoded. As soon as it's cleaner (hopefully this week), I'll let you know!!

Thanks

[jonathanlurie]

I am making a cleaner example that I can share but there is a thing. My 3D texture comes from an MRI (NIfTI file) and is encoded in Float32. For the sake of simplicity, I was so far converting the buffer from Float32Array into a Uint8Array and used BABYLON.Engine.TEXTUREFORMAT_LUMINANCE as a format. I'd rather use a flag that let me use the original float buffer, it would same time, memory and would allow more operation once on the shader. I thought of doing:

let texture3D = new BABYLON.RawTexture3D(
  img3D._data, // <--- THIS IS a Float32Array
  dim0,
  dim1,
  dim2,
  BABYLON.Engine.TEXTUREFORMAT_LUMINANCE,
  scene
)

// THIS IS THE LINE THAT MATTERS
texture3D.textureType = BABYLON.Engine.TEXTURETYPE_FLOAT;

Unfortunately, this does not work and I have a warning message saying "texImage3D: type UNSIGNED_BYTE but ArrayBufferView not Uint8Array".

For the record, the float buffer I want to send has values from in [0, few thousands] (my point is, it's not a [0, 1] buffer).

Is it something possible?

[Guest]

It should be possible but not afterwards. You have to create your texture with the format you want. The format is then immutable

[jonathanlurie]

Hey @Deltakosh , I finally took the time to bake a more robust demo, it's here and the source is on Github and part of the list of Pixpipe's example. It's ok to share because all the scripts are on CDN and the volume data as well

 

[Guest]

Cool! Will update bjs website with it!