RealityBadge: Technical Documentation & Development Process

Project Overview

RealityBadge is an AR memory system that transforms the back of a mobile phone into a “wearable memory carrier.” The system captures contextual fragments through semantic compression and provides recall through generative audio feedback and haptic responses.

Time Period: 2024–2025 Status: Functional demo completed Repository: App Store

Architecture Overview

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Camera Input  │───▶│ Semantic Compression│───▶ Memory Storage │
│   (CoreML)      │    │     Module        │    │   (Local)       │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                        │                       │
         ▼                        ▼                       ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Object/Scene  │    │   Audio Mapping   │    │  Recall System  │
│   Detection     │    │   (Strudel)       │    │  Interface      │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Core Components

1. Semantic Compression Module

Technology Stack:

• CoreML for on-device inference
• Vision framework for real-time processing
• Custom compression algorithms

Implementation Details:

// Semantic tag generation pipeline
class SemanticProcessor {
    private let visionModel: VNCoreMLModel
    private let compressionAlgorithm: SemanticCompressor

    func processFrame(_ frame: CVPixelBuffer) -> [SemanticTag] {
        // Extract visual features
        let features = extractFeatures(frame)

        // Generate semantic tags
        let tags = generateTags(features)

        // Compress for storage
        let compressed = compressionAlgorithm.compress(tags)

        return compressed
    }
}

Key Features:

• Real-time semantic extraction (<100ms latency)
• Adaptive compression based on context importance
• Privacy-first processing (all on-device)

2. Generative Audio System

Technology Stack:

• WebAudio API for sound generation
• Strudel for pattern creation
• Custom audio-semantic mapping

Mapping Algorithm:

// Semantic to audio mapping
const semanticAudioMap = {
    'nature': { scale: 'pentatonic', tempo: 60, timbre: 'soft' },
    'urban': { scale: 'chromatic', tempo: 120, timbre: 'electronic' },
    'social': { scale: 'major', tempo: 90, timbre: 'warm' },
    'work': { scale: 'minor', tempo: 80, timbre: 'focused' }
};

function generateAudio(semanticTags) {
    const audioParams = mapSemanticsToAudio(semanticTags);
    return strudel.pattern(audioParams);
}

Audio Characteristics:

• Context-aware tonal selection
• Rhythmic patterns based on detected motion
• Haptic feedback synchronization

3. Liquid Glass UI

Design Principles:

• Nothing-inspired transparency effects
• Minimalist information architecture
• Contextual information density

Implementation:

.liquid-glass-container {
    background: rgba(255, 255, 255, 0.1);
    backdrop-filter: blur(10px);
    border: 1px solid rgba(255, 255, 255, 0.2);
    box-shadow: 0 8px 32px rgba(0, 0, 0, 0.1);
}

Technical Challenges & Solutions

1. Real-time Processing Constraints

Challenge: Maintaining 30fps processing while running semantic extraction

Solution:

• Adaptive quality processing based on device capability
• Frame skipping for static scenes
• Optimized CoreML model quantization

// Adaptive processing based on performance
class AdaptiveProcessor {
    private var targetFPS: Int = 30
    private var currentLoad: Float = 0.0

    func adjustProcessingLoad() {
        if currentLoad > 0.8 {
            targetFPS = 15 // Reduce quality for performance
        } else if currentLoad < 0.5 {
            targetFPS = 30 // Increase quality
        }
    }
}

2. Memory Management

Challenge: Efficient storage of compressed semantic data

Solution:

• Tiered storage system (recent → archived)
• Intelligent compression based on retrieval patterns
• Automatic cleanup of low-importance memories

3. Audio-Visual Synchronization

Challenge: Coordinating haptic, visual, and audio feedback

Solution:

• Centralized timing coordinator
• Predictive latency compensation
• Adaptive feedback intensity based on user context

Performance Metrics

Processing Performance

• Latency: 80-120ms end-to-end
• Accuracy: 87% semantic recognition accuracy
• Battery Impact: 12-15% additional drain

User Experience Metrics

• Recall Success Rate: 78% successful memory triggering
• User Satisfaction: 4.2/5.0 (small user study)
• Daily Usage: 3-4 average sessions per day

Ethical Considerations

Privacy by Design

• All processing performed on-device
• No cloud data transmission
• User-controlled data retention policies
• Transparent data usage documentation

Accessibility

• Audio-only mode for visually impaired users
• Haptic intensity adjustments
• Alternative input methods for motor accessibility

Development Timeline

Phase 1: Core Infrastructure (Oct 2024 - Dec 2024)

• Semantic compression algorithm development
• CoreML model training and optimization
• Basic UI framework implementation

Phase 2: Audio Integration (Jan 2025 - Feb 2025)

• Strudel integration and audio mapping
• Haptic feedback system development
• User testing with audio-only interface

Phase 3: UI/UX Refinement (Mar 2025 - Apr 2025)

• Liquid glass UI implementation
• Performance optimization
• Comprehensive user testing

Phase 4: Integration & Polish (May 2025)

• System integration testing
• Documentation completion
• Demo preparation

Future Development Roadmap

Short Term (Next 3 months)

• Extended semantic categories
• Improved audio generation algorithms
• Enhanced compression efficiency

Medium Term (6-12 months)

• Multi-language semantic support
• Collaborative memory features
• Integration with calendar/reminders

Long Term (1+ years)

• Cross-platform compatibility
• Enhanced AI capabilities
• Potential commercial applications

Technical Specifications

Supported Devices

• iPhone 12 and later (iOS 15+)
• Minimum 4GB RAM recommended
• Neural Engine required for optimal performance

Storage Requirements

• Base application: 45MB
• Memory cache: Variable (50-200MB based on usage)
• Compressed memories: ~1KB per memory fragment

Network Requirements

• No active internet connection required
• Optional cloud backup (user-controlled)
• Local processing prioritized

Open Source Components

This project builds upon several open source initiatives:

• CoreML Tools: Apple’s machine learning framework
• Strudel: Generative music patterns library
• Vision Framework: Computer vision capabilities
• WebAudio API: Audio generation and processing

Contributing

While this is currently a research prototype, I’m open to collaboration in the following areas:

• Semantic compression algorithm improvements
• Audio generation enhancement
• User experience research
• Cross-platform adaptation

Conclusion

RealityBadge represents an exploration into how AR and generative systems can enhance human memory and recall. The project demonstrates the feasibility of on-device semantic processing and the potential of audio-based memory augmentation.

The current implementation provides a solid foundation for further research in this area, with particular opportunities for improving semantic accuracy, enhancing user experience, and expanding the range of supported contexts.

---

Contact: epwujiajun@icloud.com Project Repository: App Store Institution: Shenzhen Tech University, Industrial Design Program