Skip to content

Commit

Permalink
reorganize map stimuli and display env article for clarity
Browse files Browse the repository at this point in the history
  • Loading branch information
@banchan86
banchan86 committed Jun 28, 2024
1 parent 8a17ff7 commit 87ca9f4
Show file tree
Hide file tree
Showing 6 changed files with 152 additions and 39 deletions.
6 changes: 2 additions & 4 deletions docs/articles/basic-workflow.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,11 @@ This is where you generate all the aspects of the visual environment.

### Map Stimuli (optional)

This is where the stimulus is rendered onto a a surface. This can be skipped for prototyping. Ideally one should use `SphereMapping`.
If `SphereMapping` is used, the Define display section next is necessary

This is where 2D stimuli are rendered onto a surface for display in 3D environments using the `SphereMapping` operator. More information can be found in the [Display Environments](display-environment-basics.md).

### Define Display (optional)

Here we define the parameters of the display system by adding a `ViewingWindow` (which are windows into the visual environment) and a `DrawViewport` (for configuring 1 or more monitors) node.
Here we define the parameters of the display system by adding a `ViewingWindow` (which are windows into the visual environment) and a `DrawViewport` (for configuring 1 or more monitors) operator.

:::workflow
![Sphere Mapping](../workflows/overview-map-stimuli-define-display.bonsai)
Expand Down
2 changes: 1 addition & 1 deletion docs/articles/define-display.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ the two views to two displays.
</div>


## (Optional) Post-processing Operators
## Post-processing Operators (optional)
Bonvision comes with post-processing operators to correct for various distortions in the final display of the image. These operators would be placed between the `ViewWindow` and `DrawViewport` operator.

### PerspectiveMapping and MeshMapping
Expand Down
27 changes: 25 additions & 2 deletions docs/articles/display-environment-basics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ At runtime, each screen becomes a window that looks out into that surrounding en

Once the cube mapping rendering pipeline is in place, we can very efficiently generate an arbitrary number of projections into the visual space for each display in the experiment. The remaining challenge is then how to generate the 6 faces of the cubemap in a way that accurately represents the visual field surrounding the subject.

## 3D Stimuli
### 3D Stimuli

For 3D scenes, there is a straightforward solution: render the visual scene once for each face of the cube, with a perspective 90º field of view, as seen from the observer. Each rendered perspective will fill exactly one face of the cube. For example:

Expand All @@ -31,7 +31,7 @@ Using the cube mapping pipeline, one can then point a viewing window at any dire
![CubemapEnvironment](~/images/DisplayLogic/CubemapEnvironment.webp){width=500}


## 2D Stimuli
### 2D Stimuli

For 2D scenes, such as those containing gabor patches, checkerboards, gratings, random dots and so on, we would like to specify our environments in an orthonormal space, where X and Y represent longitude and latitude, respectively, in degrees of visual field. For example, below is a checkerboard stimulus covering 100 degrees of visual field horizontally and vertically, where each square subtends 10 degrees:

Expand All @@ -55,3 +55,26 @@ The disadvantage is that the arrangement of the planar mapping is not orthonorma

[More details on spherical mapping from the point of view of modelling software](https://en.wikibooks.org/wiki/Blender_3D:_Noob_to_Pro/UV_Map_Basics)

## Virtual Reality vs Augmented Reality

- Shawn's note - This needs more clarity as they both seem to be the same (fixed window, objects/environment),

### Virtual Reality (VR)
VR can be easily defined as a situation where the eye, and the screens (windows) are fixed positions, while the all the objects (or VR environment) moves across the eye.

![VR](~/images/DisplayLogic/VRcartoon.png){width=500}

Example rendering to be added here

### For Augmented Reality (AR)

This is a scenario where, generally, the screens remain in a fixed position and the animal can move around. Since we have an eye-centric coordinate frame, the objects and the screen move around to generate an AR.

![AR](~/images/DisplayLogic/ARcartoon.jpg){width=500}

Example rendering to be added here





2 changes: 1 addition & 1 deletion docs/articles/drawing-region.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ In this example workflow, we use `CubemapView` to simulate the walls of a virtua
> `CubemapView` requires two additional nodes for a basic workflow, a `RenderCubemap` operator from the `Bonsai.Shaders` package (included as a dependency) that renders the cubemap and a `ViewWindow` that looks out into the virtual environment.
> [!Note]
> While the description of the operator suggests that you need to define all 6 faces of the cubemap, you can display an arbitary number of textures.
> While the description of the operator suggests that you need to define six textures for each face of the cubemap, you can display an arbitary number of textures (including less than six).
### Other nodes to document (under construction)
`HMDview`
43 changes: 12 additions & 31 deletions docs/articles/map-stimuli.md
View file
Original file line number Diff line number Diff line change
@@ -1,43 +1,24 @@
# Map Stimuli

## Under Construction
## Sphere Mapping

## A. Sphere Mapping
> [!NOTE]
> This is best for creating 2D stimuli.
> [!NOTE]
> Needs a Display object: preferably a ViewingWindow
When rendering 3D environments, in the case of 3D stimuli, `BonVision` applies by default a cubemap rendering approach and no additional transformation/operators are necessary.

[The details of the implementation of Sphere mapping](https://en.wikibooks.org/wiki/Blender_3D:_Noob_to_Pro/UV_Map_Basics)

In this case, stimuli are always rendered onto the inside of a sphere. This allows easy eye-centric definitions of stimuli in visual angle units. The displays are then windows that observe these rendered stimuli. We use Spherical coordinates to define all stimulus parameters in this case.
However, in the case of 2D stimuli, we use `SphereMapping` to render them onto the inside of a 3D sphere to display them in a 3D space.
More info can be found in the [Display Environments](display-environment-basics.md)

![SphericalCoord](~/images/DisplayLogic/SphericalCoord_resized.png){width=500}

## B. Cube Mapping
> [!NOTE]
> This is best for creating 3D stimuli, for Virtual or Augmented Reality Systems.
> [!NOTE]
> Needs a Display object: preferably a PerspectiveViewingWindow
[The details of Cube Mapping are explained here:](https://en.wikipedia.org/wiki/Cube_mapping)

In this case all the stimuli are created in eye-centric physical coordinates (centimeters for example) and rendered onto a unit Cube. The displays (Perspective cameras) are then windows into these rendered images.
*probably easiest with a whiteboard drawing here*
This workflow below draws a checkboard which is mapped to a sphere using `SphereMapping`.

### For Virtual Reality (VR)
VR can be easily defined as a situation where the eye, and the screens (windows) are fixed positions, while the all the objects (or VR environment) moves across the eye.
:::workflow
![Sphere Mapping](../workflows/overview-map-stimuli-define-display.bonsai)
:::

![VR](~/images/DisplayLogic/VRcartoon.png){width=500}

Example rendering to be added here

### For Augmented Reality (AR)

This is a scenario where, generally, the screens remain in a fixed position and the animal can move around. Since we have an eye-centric coordinate frame, the objects and the screen move around to generate an AR.

![AR](~/images/DisplayLogic/ARcartoon.jpg){width=500}
> [!NOTE]
> `SphereMapping` requires a `ViewWindow`.
Example rendering to be added here
> [!NOTE]
> When drawing stimuli, `SphereMapping` requires a `PublishSubject` and `SubscribeSubject` pair (see the [multiple stimuli](stimuli-multiple.md) for instructions on how to use `Subjects` and this [link](https://github.com/orgs/bonsai-rx/discussions/1835) for an explanation of the reasons).

111 changes: 111 additions & 0 deletions docs/workflows/map-stimuli-sphere-mapping.bonsai
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
<?xml version="1.0" encoding="utf-8"?>
<WorkflowBuilder Version="2.8.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:gl="clr-namespace:Bonsai.Shaders;assembly=Bonsai.Shaders"
xmlns:res="clr-namespace:Bonsai.Resources;assembly=Bonsai.System"
xmlns:rx="clr-namespace:Bonsai.Reactive;assembly=Bonsai.Core"
xmlns="https://bonsai-rx.org/2018/workflow">
<Workflow>
<Nodes>
<Expression xsi:type="Combinator">
<Combinator xsi:type="gl:CreateWindow">
<gl:Width>640</gl:Width>
<gl:Height>480</gl:Height>
<gl:VSync>On</gl:VSync>
<gl:SwapSync>false</gl:SwapSync>
<gl:ClearColor>Black</gl:ClearColor>
<gl:ClearMask>DepthBufferBit ColorBufferBit</gl:ClearMask>
<gl:CursorVisible>true</gl:CursorVisible>
<gl:Location xsi:nil="true" />
<gl:WindowBorder>Resizable</gl:WindowBorder>
<gl:WindowState>Normal</gl:WindowState>
<gl:DisplayDevice>Primary</gl:DisplayDevice>
<gl:TargetRenderFrequency>60</gl:TargetRenderFrequency>
<gl:TargetUpdateFrequency xsi:nil="true" />
<gl:RenderState />
<gl:GraphicsMode>
<gl:ColorFormat>
<gl:Red>8</gl:Red>
<gl:Green>8</gl:Green>
<gl:Blue>8</gl:Blue>
<gl:Alpha>8</gl:Alpha>
</gl:ColorFormat>
<gl:Depth>16</gl:Depth>
<gl:Stencil>0</gl:Stencil>
<gl:Samples>0</gl:Samples>
<gl:AccumulatorFormat>
<gl:Red>0</gl:Red>
<gl:Green>0</gl:Green>
<gl:Blue>0</gl:Blue>
<gl:Alpha>0</gl:Alpha>
</gl:AccumulatorFormat>
<gl:Buffers>2</gl:Buffers>
<gl:Stereo>false</gl:Stereo>
</gl:GraphicsMode>
</Combinator>
</Expression>
<Expression xsi:type="IncludeWorkflow" Path="BonVision:Primitives.BonVisionResources.bonsai" />
<Expression xsi:type="Combinator">
<Combinator xsi:type="res:LoadResources" />
</Expression>
<Expression xsi:type="Combinator">
<Combinator xsi:type="gl:RenderFrame" />
</Expression>
<Expression xsi:type="IncludeWorkflow" Path="BonVision:Environment.OrthographicView.bonsai">
<Left>-180</Left>
<Right>180</Right>
<Bottom>-90</Bottom>
<Top>90</Top>
</Expression>
<Expression xsi:type="rx:PublishSubject">
<Name>Draw</Name>
</Expression>
<Expression xsi:type="IncludeWorkflow" Path="BonVision:Environment.SphereMapping.bonsai">
<ClearColor>Gray</ClearColor>
<Width xsi:nil="true" />
<Height xsi:nil="true" />
<RotationZ>0</RotationZ>
<RotationY>0</RotationY>
<RotationX>0</RotationX>
<FaceSize xsi:nil="true" />
</Expression>
<Expression xsi:type="IncludeWorkflow" Path="BonVision:Environment.ViewWindow.bonsai">
<Width>20</Width>
<Height>15</Height>
<Rotation>
<X>0</X>
<Y>0</Y>
<Z>0</Z>
</Rotation>
<Translation>
<X>0</X>
<Y>0</Y>
<Z>-10</Z>
</Translation>
</Expression>
<Expression xsi:type="SubscribeSubject">
<Name>Draw</Name>
</Expression>
<Expression xsi:type="IncludeWorkflow" Path="BonVision:Primitives.DrawCheckerboard.bonsai">
<ExtentX>90</ExtentX>
<ExtentY>90</ExtentY>
<LocationX>0</LocationX>
<LocationY>0</LocationY>
<Layer>0</Layer>
<Angle>0</Angle>
<NumberColumns>5</NumberColumns>
<NumberRows>5</NumberRows>
<Phase>0</Phase>
</Expression>
</Nodes>
<Edges>
<Edge From="0" To="1" Label="Source1" />
<Edge From="1" To="2" Label="Source1" />
<Edge From="3" To="4" Label="Source1" />
<Edge From="4" To="5" Label="Source1" />
<Edge From="5" To="6" Label="Source1" />
<Edge From="6" To="7" Label="Source1" />
<Edge From="8" To="9" Label="Source1" />
</Edges>
</Workflow>
</WorkflowBuilder>

0 comments on commit 87ca9f4

Please sign in to comment.