Context-Aware Fusion of AR, Vision Models, and LLMs for Safety Inspection

Primary supervisor

Jiazhou 'Joe' Liu

Co-supervisors

System Architecture

This project builds upon an existing safety inspection system framework, integrating augmented reality (AR), artificial intelligence (AI), and automated reporting. You will follow the framework to design, implement, and evaluate a fully functioning prototype system that supports safety inspections in real-world construction environments.

The proposed system leverages Apple Vision Pro to support AR-assisted inspection, allowing inspectors to track inspected areas using a virtual brush, capture live video streams and photos, and annotate findings through voice or text. These inputs are processed by an AI-assisted gap detection module, featuring image segmentation and gap classification models powered by Ultralytics YOLOv11. Finally, the inspection data is fed into a Python-based module that generates comprehensive inspection reports for safety analysis and documentation.

Through this project, you will work closely with real construction scenarios and contribute to the ongoing industry need for smarter, technology-driven safety solutions.

Student cohort

Double Semester

Aim/outline

  1. Prototype Development: To implement a functional prototype that integrates AR-based user interaction, real-time AI gap detection, and automatic report generation.

  2. System Integration: To ensure seamless interoperability between the AR-assisted inspection tool, AI detection module, and report generator.

  3. User-Centred Design: To refine interaction flows for safety inspectors using voice/text input and intuitive AR gestures within the Vision Pro environment.

  4. Model Evaluation: To assess the performance of segmentation and classification models (e.g., YOLOv11) in identifying safety gaps in visual data.

  5. Field Testing and Feedback (Optional): To conduct pilot evaluations with industry experts or simulated site conditions, collecting feedback to inform system refinement.

  6. Research Contribution: To explore the effectiveness of combining immersive and AI technologies for construction safety, contributing new insights to human-centred safety inspection systems.

URLs/references

Required knowledge

  • Strong programming skills (e.g., Swift/SwiftUI and Python)
  • Strong communication skills
  • Experience in iOS/visionOS development
  • Knowledge of deploying and applying AI models
  • Knowledge of distributed computing