GUI markup

[brianzinn]

The thought experiment continues... some kind of GLML (graphics library markup language??) could be useful.  I spent quite a bit of time manually adding Rectangles and StackPanels with TextBlocks/Images and it got a bit difficult to keep track of sometimes!  My original idea was to make a JSON structure to load to 2D GUI something like:

[
  [{ type: 'rectangle' background: 'white'}]
  [{ type: 'stackPanel' isVertical:false onClick: () => { console.log('clicked')} children:[
      {type:'textBlock' text:'hi'},
      {type:'image' src:'/images/x.png'}
    ]
  }]
]

Anyway, that's how the idea started - that wouldn't take too much effort.  Then I wanted that if you changed the background property of the first rectangle from 'white' to 'blue' or whatever, that the rectangle would change colour in the scene automatically.  I could do that with something like ES6 Proxy and flow the properties.  A list in a stack panel - maybe could be an RxJS observable array or like a Knockout or MobX observable array - it would need to automatically call removeControl() or addControl() accordingly on list updates.  Something like that would be quite portable.

I ended up implementing in ReactJS as I had a project sort of on the go.  Essentially with markup like this, I have accomplished those goals:

<Scene id="sample-canvas" onMeshPicked={this.meshPicked}>
... camera, lights, buttons ...
{this.state.showModal === true &&
  <Plane name="dialog" width={1} height={dialogHeight/dialogWidth} onCreated={this.setPlane}>
    <AdvancedDynamicTexture createForParentMesh={true}>
      ... header
      <Rectangle height="60%" thickness={2} color="#EEEEEE">
        <StackPanel>
          <Text text={`You have clicked on '${this.state.clickedMeshName}' ...`} ... />
            {this.state.allowedMeshes.map(allowedMesh => (
                <Text key={...} text={'• ' + allowedMesh} color="black" fontSize={28} height="20%" />
            ))}
        </StackPanel>
      </Rectangle>
       ... footer
    </AdvancedDynamicTexture>
  </Plane>
}
</Scene>

If 'this.state.clickedMeshName' changes in my application, then it automatically updates the associated TextBlock.text in the scene without me explicitly writing code to do so.  I can wire this up to props instead and get changes from redux/websockets or anywhere and the GUI (ie: TextBlocks in a StackPanel) will automatically add/remove themselves to build lists dynamically - reflecting or observing the current state.
Here is a working example harnessing the built-in capabilities of the React framework for flowing state changes. Obviously to some extent just enough to get the example working.

Click on different boxes *while* the dialog is open.  Click a box to open the dialog (button is colour of selected box) and dynamic list of remaining boxes is shown.  Try in VR for best experience (it's not a full-screen GUI - it just follows the camera) 
https://brianzinn.github.io/create-react-app-babylonjs/with2DUI

It's a work in progress, so not perfect!

[Guest]

This is reallllly cool! Thanks @brianzinn