02

02

02

SpatiaLLM

SpatiaLLM

SpatiaLLM

LLM-generated Human Interface for Co-presence Experience in Mixed Reality

LLM-generated Human Interface for Co-presence Experience in Mixed Reality

LLM-generated Human Interface for Co-presence Experience in Mixed Reality

Description

Description

SpatiaLLM is a multi-agent framework that seamlessly bridges Large Language Model (LLM) capabilities with spatial understanding, enabling intelligent and context-aware content placement in Mixed Reality environments.


SpatiaLLM is a multi-agent framework that seamlessly bridges Large Language Model (LLM) capabilities with spatial understanding, enabling intelligent and context-aware content placement in Mixed Reality environments.


Keywords

Keywords

Spatial Computing

Spatial Computing

Mixed Reality (MR)

Mixed Reality (MR)

Human–Computer Interaction (HCI)

Human–Computer Interaction (HCI)

Large Language Models

Adaptive Interfaces

Year

Year

2024

2024

Technical Details

Technical Details

SpatiaLLM combines advanced language-based reasoning with real-time environment sensing to deliver adaptive 3D layouts and immersive user experiences. By leveraging Large Language Models as orchestrators of semantic logic—and coupling them with sensor-driven agents that detect planes, boundaries, and objects—SpatiaLLM ensures that virtual elements remain anchored, coherent, and physically consistent within the user’s real-world surroundings. This dynamic integration of computational linguistics and spatial perception facilitates a more natural presentation of 3D information, transforming traditional slides into fully interactive and environment-aware storytelling. Originally developed as a “3D PPT” concept, SpatiaLLM now extends into educational, industrial, and entertainment settings, providing a robust framework for contextualized, cross-platform MR design.

SpatiaLLM combines advanced language-based reasoning with real-time environment sensing to deliver adaptive 3D layouts and immersive user experiences. By leveraging Large Language Models as orchestrators of semantic logic—and coupling them with sensor-driven agents that detect planes, boundaries, and objects—SpatiaLLM ensures that virtual elements remain anchored, coherent, and physically consistent within the user’s real-world surroundings. This dynamic integration of computational linguistics and spatial perception facilitates a more natural presentation of 3D information, transforming traditional slides into fully interactive and environment-aware storytelling. Originally developed as a “3D PPT” concept, SpatiaLLM now extends into educational, industrial, and entertainment settings, providing a robust framework for contextualized, cross-platform MR design.

Highlights

Highlights

  • Multi-Agent Architecture: Specialized agents handle perception, retrieval, layout optimization, and user interaction, unifying language queries with precise geometric checks.

  • Spatial-Temporal Cognitive Map (ST-CM): Ensures continuity and realistic object behavior across multiple scenes by blending sensor-based constraints with semantic anchors.

  • Adaptive Storytelling: Transitions from static slides to immersive scenarios where 3D content can scale, reposition, and adapt in real time based on the physical environment.

  • Broad Applicability: Designed for mixed reality teaching, product showcases, and engaging industrial simulations, bridging the gap between engineering prototypes and immersive user experiences.

  • Multi-Agent Architecture: Specialized agents handle perception, retrieval, layout optimization, and user interaction, unifying language queries with precise geometric checks.

  • Spatial-Temporal Cognitive Map (ST-CM): Ensures continuity and realistic object behavior across multiple scenes by blending sensor-based constraints with semantic anchors.

  • Adaptive Storytelling: Transitions from static slides to immersive scenarios where 3D content can scale, reposition, and adapt in real time based on the physical environment.

  • Broad Applicability: Designed for mixed reality teaching, product showcases, and engaging industrial simulations, bridging the gap between engineering prototypes and immersive user experiences.

Credits

Research

Yueze Zhang

Dev

Yueze Zhang, Shucun Zhao

Advisors

Prof. Dr. Martin Werner, Prof Tim Purdy

Appendix

Credits

Research

Yueze Zhang

Dev

Yueze Zhang, Shucun Zhao

Advisors

Prof. Dr. Martin Werner, Prof Tim Purdy

Appendix