CTR-WebCam/docs/design.md

Technical Design Document

Architecture Overview

The 3DS Webcam Bridge consists of two main components:

1. 3DS Homebrew Application: Captures camera frames, encodes to JPEG, and streams over TCP
2. PC Server: Receives frames, decodes, and exposes as virtual webcam(s)

┌─────────────────┐                      ┌──────────────────┐
│   Nintendo 3DS  │                      │    PC (Server)   │
│                 │                      │                  │
│  ┌───────────┐  │                      │  ┌────────────┐  │
│  │  Camera   │  │    TCP/IP (Wi-Fi)   │  │   Server   │  │
│  │  Capture  │  │───────────────────►  │  │  (Python)  │  │
│  └─────┬─────┘  │     Port 9000        │  └──────┬─────┘  │
│        │        │                      │         │        │
│  ┌─────▼─────┐  │                      │  ┌──────▼─────┐  │
│  │   JPEG    │  │                      │  │   JPEG     │  │
│  │  Encoder  │  │                      │  │  Decoder   │  │
│  └─────┬─────┘  │                      │  └──────┬─────┘  │
│        │        │                      │         │        │
│  ┌─────▼─────┐  │                      │  ┌──────▼─────┐  │
│  │ Network   │  │                      │  │  Virtual   │  │
│  │ Protocol  │  │                      │  │  Webcam    │  │
│  └───────────┘  │                      │  └────────────┘  │
└─────────────────┘                      └──────────────────┘

---

Protocol Specification

Wire Protocol

The protocol is designed to be backward-compatible and support both mono and stereo streams.

Header Format

All frames begin with a 2-byte header:

[FLAGS (1 byte)][VERSION (1 byte)]

FLAGS:

• Bit 0: Stereo flag (1 = stereo, 0 = mono)
• Bits 1-7: Reserved (must be 0)

VERSION: Protocol version (currently 0x01)

Mono Frame Format

┌─────────┬─────────┬────────────┬──────────────┐
│ FLAGS   │ VERSION │ LENGTH     │ JPEG DATA    │
│ (1 byte)│ (1 byte)│ (4 bytes)  │ (LENGTH)     │
│ 0x00    │ 0x01    │ Big-endian │              │
└─────────┴─────────┴────────────┴──────────────┘

Total header: 6 bytes

• FLAGS: 0x00 (no stereo)
• VERSION: 0x01
• LENGTH: 32-bit unsigned big-endian integer (JPEG size)

Stereo Frame Format

┌─────────┬─────────┬────────────────────────────────┬─────────────┬──────────────┬─────────────┬──────────────┐
│ FLAGS   │ VERSION │ DIMENSIONS                     │ LEFT_LEN    │ RIGHT_LEN    │ LEFT_JPEG   │ RIGHT_JPEG   │
│ (1 byte)│ (1 byte)│ (8 bytes)                      │ (4 bytes)   │ (4 bytes)    │ (LEFT_LEN)  │ (RIGHT_LEN)  │
│ 0x01    │ 0x01    │ wL,hL,wR,hR (16-bit BE each)   │ Big-endian  │ Big-endian   │             │              │
└─────────┴─────────┴────────────────────────────────┴─────────────┴──────────────┴─────────────┴──────────────┘

Total header: 18 bytes

• FLAGS: 0x01 (stereo enabled)
• VERSION: 0x01
• DIMENSIONS:
  • wL: Left width (16-bit BE)
  • hL: Left height (16-bit BE)
  • wR: Right width (16-bit BE)
  • hR: Right height (16-bit BE)
• LEFT_LEN: Left JPEG size (32-bit BE)
• RIGHT_LEN: Right JPEG size (32-bit BE)

Network Transport

• Protocol: TCP
• Port: 9000 (default, configurable)
• Connection: Client (3DS) → Server (PC)
• Reconnection: Automatic with exponential backoff (3s delay)
• Socket Options: TCP_NODELAY enabled for lower latency

---

3DS Implementation Details

Camera Subsystem

The 3DS has multiple camera configurations:

Camera	Hardware	Purpose	Stereo?
Inner (SELECT_IN1)	Single lens	Front-facing selfie camera	❌ No
Outer (SELECT_OUT1, SELECT_OUT2)	Dual lens	Rear stereo cameras	✅ Yes

Camera Configuration

CAMU_SetSize(select, SIZE_CTR_TOP_LCD, CONTEXT_A);          // 400×240
CAMU_SetOutputFormat(select, OUTPUT_YUV_422, CONTEXT_A);    // YUV422
CAMU_SetFrameRate(select, FRAME_RATE_15);                   // 15 FPS target
CAMU_SetNoiseFilter(select, true);                          // Reduce noise
CAMU_SetAutoExposure(select, true);                         // Auto exposure
CAMU_SetAutoWhiteBalance(select, true);                     // Auto WB

Capture Process

1. Select camera(s): SELECT_IN1 or SELECT_OUT1|SELECT_OUT2
2. Set transfer size: CAMU_GetMaxBytes() → CAMU_SetTransferBytes()
3. Clear buffer: CAMU_ClearBuffer()
4. Start capture: CAMU_StartCapture()
5. Set receiving: CAMU_SetReceiving() → returns handle
6. Wait for completion: svcWaitSynchronization(handle)
7. Stop capture: CAMU_StopCapture()

For stereo, both cameras are captured sequentially but as quickly as possible to minimize desynchronization.

Color Space Conversion

3DS outputs YUV422 format. We convert to RGB24 for JPEG encoding:

// YUV422 format: [Y0 U Y1 V] packed in 16-bit values
// Conversion matrix:
R = 1.164*(Y - 16) + 1.596*(V - 128)
G = 1.164*(Y - 16) - 0.391*(U - 128) - 0.813*(V - 128)
B = 1.164*(Y - 16) + 2.018*(U - 128)

Fixed-point integer math is used for performance:

R = (298*c + 409*e + 128) >> 8
G = (298*c - 100*d - 208*e + 128) >> 8
B = (298*c + 516*d + 128) >> 8

Where: c = Y-16, d = U-128, e = V-128

JPEG Encoding

We use stb_image_write.h (public domain, single-header library):

stbi_write_jpg_to_func(callback, context, width, height, 3, rgb_data, quality);

• Quality: 65 (configurable via JPEG_QUALITY define)
• Typical size: 10-20 KB per 400×240 frame
• Encoding time: ~30-50ms on 3DS hardware

Threading Model

Main Thread:
  - UI rendering (top/bottom screens)
  - Input handling
  - State management

Stream Thread:
  - Socket connection/reconnection
  - Camera capture loop
  - JPEG encoding
  - Network transmission
  - Statistics tracking

Synchronization via LightLock for shared state access.

Performance Characteristics

Metric	Mono	Stereo
Capture Time	~30ms	~60ms
Encode Time	~40ms	~80ms
Network Time	~20ms	~40ms
Total Latency	~90ms	~180ms
Target FPS	12-15	10-12
Bandwidth	~150 KB/s	~300 KB/s

Note: Actual performance varies based on scene complexity, network conditions, and JPEG quality settings.

---

PC Server Implementation

Backend Architecture

The server supports multiple virtual camera backends:

┌────────────────────────────────────┐
│        Server Core (server.py)     │
│  - Protocol parsing                │
│  - Frame decoding (JPEG → RGB)     │
│  - Stereo composition              │
└──────────────┬─────────────────────┘
               │
       ┌───────┴────────┐
       │                │
   ┌───▼───┐      ┌────▼────┐      ┌────────┐
   │ pyvir │      │ pyfake  │      │ ffmpeg │
   │tualcam│      │ webcam  │      │  pipe  │
   └───┬───┘      └────┬────┘      └────┬───┘
       │               │                 │
   ┌───▼────┐     ┌────▼────┐      ┌────▼───┐
   │Windows │     │ Linux   │      │ Linux  │
   │OBS Virt│     │v4l2loop │      │v4l2loop│
   └────────┘     └─────────┘      └────────┘

Stereo Modes

1. Off (Mono)

• Send left eye only
• Resolution: W × H
• Use case: Standard webcam applications

2. Side-by-Side (SBS)

• Compose horizontally: [LEFT | RIGHT]
• Resolution: 2W × H
• Use case: 3D displays, VR headsets
• Implementation:

  sbs = np.concatenate([left_arr, right_arr], axis=1)
  

3. Anaglyph (Red-Cyan)

• Red channel from left, green+blue from right
• Resolution: W × H
• Use case: Red-cyan 3D glasses
• Implementation:

  anaglyph[:,:,0] = left[:,:,0]   # Red from left
  anaglyph[:,:,1] = right[:,:,1]  # Green from right
  anaglyph[:,:,2] = right[:,:,2]  # Blue from right
  

4. Dual Devices

• Two separate virtual cameras
• Resolution: W × H each
• Use case: Advanced VR setups, custom stereo applications

Backend Selection Logic

User preference: --prefer-backend
    │
    ├─ pyvirtualcam?
    │   ├─ Available? → Use it
    │   └─ Not available → Fallback
    │
    ├─ pyfakewebcam? (Linux only)
    │   ├─ Available? → Use it
    │   └─ Not available → Fallback
    │
    └─ ffmpeg? (Linux only)
        ├─ Available? → Use it
        └─ Not available → Error

Frame Processing Pipeline

1. TCP Receive
   ├─ Read 2-byte header
   ├─ Parse FLAGS and VERSION
   └─ Determine mono/stereo

2. Decode
   ├─ Mono: Read 4-byte length → JPEG data
   └─ Stereo: Read 8-byte dims → 8-byte lengths → 2× JPEG data

3. PIL Decoding
   ├─ JPEG bytes → PIL Image
   └─ Convert to RGB mode

4. Composition (if stereo)
   ├─ Off: left only
   ├─ SBS: horizontal concat
   ├─ Anaglyph: color channel merge
   └─ Dual: process separately

5. Resize
   ├─ Target resolution (--width, --height)
   └─ BILINEAR interpolation

6. NumPy Conversion
   └─ RGB24 format (uint8, HxWx3)

7. Backend Send
   ├─ pyvirtualcam: camera.send(frame)
   ├─ pyfakewebcam: camera.schedule_frame(frame)
   └─ ffmpeg: stdin.write(frame.tobytes())

---

Platform-Specific Details

Windows: OBS Virtual Camera

• Provider: OBS Studio 26.0+
• Driver: OBS VirtualCam plugin (bundled)
• Location: Installed with OBS, typically C:\Program Files\obs-studio\
• Access: Via pyvirtualcam library
• Multiple devices: User must configure in OBS settings

First-time setup:

1. Launch OBS Studio
2. Tools → VirtualCam → Start
3. This initializes the driver
4. Close OBS (driver remains active)

Device naming:

• Default: "OBS Virtual Camera"
• Can be referenced by index: 0, 1, etc.

Linux: v4l2loopback

• Provider: Kernel module v4l2loopback
• Installation: sudo apt install v4l2loopback-dkms v4l2loopback-utils
• Device creation:

  sudo modprobe v4l2loopback devices=1 video_nr=0 card_label="3DS Webcam" exclusive_caps=1
  

Persistent setup (/etc/modprobe.d/v4l2loopback.conf):

options v4l2loopback devices=1 video_nr=0 card_label="3DS Webcam" exclusive_caps=1

Multiple devices:

sudo modprobe v4l2loopback devices=2 video_nr=0,1 \
  card_label="3DS Left","3DS Right" exclusive_caps=1,1

Verification:

v4l2-ctl --list-devices
ls -l /dev/video*

---

Performance Tuning

Bandwidth Optimization

JPEG Quality vs Size:

Quality	Avg Size (400×240)	Bandwidth @ 15 FPS
50	8 KB	120 KB/s
65	12 KB	180 KB/s
80	18 KB	270 KB/s
95	30 KB	450 KB/s

Recommendation: Quality 60-70 for good balance.

Resolution Options

Resolution	Capture Time	Bandwidth @ Q65
320×240	~25ms	~140 KB/s
400×240	~30ms	~180 KB/s
512×384*	N/A	(Unsupported)

Note: 3DS hardware maximum is 640×480, but we use 400×240 for performance.

Network Recommendations

• Topology: Wired PC + wireless 3DS ideal
• Wi-Fi: 2.4 GHz (3DS limitation), keep 3DS close to AP
• QoS: Prioritize port 9000 if router supports it
• Interference: Minimize other 2.4 GHz devices

CPU Usage

3DS:

• Mono: ~40% CPU (one ARM11 core)
• Stereo: ~70% CPU
• Leaves headroom for system processes

PC:

• Negligible (<5% on modern CPU)
• JPEG decode + resize are lightweight

---

Security Considerations

Threat Model

In Scope:

• Local network (trusted environment)
• Personal/educational use

Out of Scope:

• Internet-facing deployment
• Untrusted network actors

Current Security Posture

⚠️ No Encryption: Traffic is plaintext over TCP ⚠️ No Authentication: Server accepts any connection ⚠️ No Integrity Checks: Corrupted data may cause crashes

Recommendations for Production Use

If deploying in a less-trusted environment:

1. VPN Tunnel: Use Wireguard/OpenVPN between 3DS and PC
2. SSH Tunnel: ssh -L 9000:localhost:9000 user@server
3. Firewall Rules: Restrict to known 3DS IP only
4. Protocol Extension: Add HMAC for integrity checking

Example SSH tunnel:

# On PC
ssh -L 9000:localhost:9000 user@localhost -N

# On 3DS
# Connect to 127.0.0.1:9000 (tunneled)

---

Future Enhancements

Potential Improvements

1. H.264 Encoding: Lower bandwidth, but requires hardware encoder (limited on 3DS)
2. Adaptive Quality: Adjust JPEG quality based on network conditions
3. Frame Skipping: Drop frames under high load to maintain real-time
4. Audio Capture: Add microphone support (separate audio stream)
5. Recording: Save to file on PC side
6. Multi-Client: Support multiple 3DS devices simultaneously
7. WebRTC: Browser-based viewing without virtual camera

Known Limitations

• 3DS Hardware: Max resolution 640×480, limited CPU/RAM
• Network Latency: ~100-200ms typical, inherent in Wi-Fi
• Synchronization: Stereo frames not hardware-synced (~10ms difference)
• Frame Rate: Limited by capture/encode pipeline, not true 30 FPS

---

References

3DS Development

• libctru documentation: https://libctru.devkitpro.org/
• devkitPro: https://devkitpro.org/
• 3DS Homebrew: https://www.3dbrew.org/

Libraries

• stb_image_write: https://github.com/nothings/stb
• pyvirtualcam: https://github.com/letmaik/pyvirtualcam
• v4l2loopback: https://github.com/umlaeute/v4l2loopback

Standards

• YUV color space: ITU-R BT.601
• JPEG: ITU-T T.81 / ISO/IEC 10918-1
• TCP: RFC 793