02
02
02
SpatiaLLM
SpatiaLLM
SpatiaLLM
LLM-generated Human Interface for Co-presence Experience in Mixed Reality
LLM-generated Human Interface for Co-presence Experience in Mixed Reality
LLM-generated Human Interface for Co-presence Experience in Mixed Reality



Description
Description
SpatiaLLM is a multi-agent framework that seamlessly bridges Large Language Model (LLM) capabilities with spatial understanding, enabling intelligent and context-aware content placement in Mixed Reality environments.
SpatiaLLM is a multi-agent framework that seamlessly bridges Large Language Model (LLM) capabilities with spatial understanding, enabling intelligent and context-aware content placement in Mixed Reality environments.
Keywords
Keywords
Spatial Computing
Spatial Computing
Mixed Reality (MR)
Mixed Reality (MR)
Human–Computer Interaction (HCI)
Human–Computer Interaction (HCI)
Large Language Models
Adaptive Interfaces
Year
Year
2024
2024






Technical Details
Technical Details
SpatiaLLM combines advanced language-based reasoning with real-time environment sensing to deliver adaptive 3D layouts and immersive user experiences. By leveraging Large Language Models as orchestrators of semantic logic—and coupling them with sensor-driven agents that detect planes, boundaries, and objects—SpatiaLLM ensures that virtual elements remain anchored, coherent, and physically consistent within the user’s real-world surroundings. This dynamic integration of computational linguistics and spatial perception facilitates a more natural presentation of 3D information, transforming traditional slides into fully interactive and environment-aware storytelling. Originally developed as a “3D PPT” concept, SpatiaLLM now extends into educational, industrial, and entertainment settings, providing a robust framework for contextualized, cross-platform MR design.
SpatiaLLM combines advanced language-based reasoning with real-time environment sensing to deliver adaptive 3D layouts and immersive user experiences. By leveraging Large Language Models as orchestrators of semantic logic—and coupling them with sensor-driven agents that detect planes, boundaries, and objects—SpatiaLLM ensures that virtual elements remain anchored, coherent, and physically consistent within the user’s real-world surroundings. This dynamic integration of computational linguistics and spatial perception facilitates a more natural presentation of 3D information, transforming traditional slides into fully interactive and environment-aware storytelling. Originally developed as a “3D PPT” concept, SpatiaLLM now extends into educational, industrial, and entertainment settings, providing a robust framework for contextualized, cross-platform MR design.






Highlights
Highlights
Multi-Agent Architecture: Specialized agents handle perception, retrieval, layout optimization, and user interaction, unifying language queries with precise geometric checks.
Spatial-Temporal Cognitive Map (ST-CM): Ensures continuity and realistic object behavior across multiple scenes by blending sensor-based constraints with semantic anchors.
Adaptive Storytelling: Transitions from static slides to immersive scenarios where 3D content can scale, reposition, and adapt in real time based on the physical environment.
Broad Applicability: Designed for mixed reality teaching, product showcases, and engaging industrial simulations, bridging the gap between engineering prototypes and immersive user experiences.
Multi-Agent Architecture: Specialized agents handle perception, retrieval, layout optimization, and user interaction, unifying language queries with precise geometric checks.
Spatial-Temporal Cognitive Map (ST-CM): Ensures continuity and realistic object behavior across multiple scenes by blending sensor-based constraints with semantic anchors.
Adaptive Storytelling: Transitions from static slides to immersive scenarios where 3D content can scale, reposition, and adapt in real time based on the physical environment.
Broad Applicability: Designed for mixed reality teaching, product showcases, and engaging industrial simulations, bridging the gap between engineering prototypes and immersive user experiences.















Credits
Research
Yueze Zhang
Dev
Yueze Zhang, Shucun Zhao
Advisors
Prof. Dr. Martin Werner, Prof Tim Purdy
Appendix
Credits
Research
Yueze Zhang
Dev
Yueze Zhang, Shucun Zhao
Advisors
Prof. Dr. Martin Werner, Prof Tim Purdy
Appendix

