Server Storage

JSON File Storage System

Storage Model

The server uses a simple JSON file-based storage system for chat persistence.

src/server/
├── chatSyncApi.js     # HTTP API server
├── chatStore.js       # Storage operations
└── chatStore.json     # Data file (created automatically)

File Structure

[
  {
    "id": "chat-123",
    "title": "Example Chat",
    "messages": [
      {
        "id": "msg-456",
        "role": "user",
        "content": "Hello",
        "timestamp": "2024-01-15T10:30:00.000Z"
      }
    ],
    "createdAt": "2024-01-15T10:00:00.000Z",
    "updatedAt": "2024-01-15T10:30:00.000Z",
    "lastModified": "2024-01-15T10:30:00.000Z",
    "model": "gpt-4",
    "provider": "openai",
    "isArchived": false
  }
]

Storage Operations

File I/O Implementation

Read Operations

function readChats() {
  if (!fs.existsSync(STORE_PATH)) return [];

  const raw = fs.readFileSync(STORE_PATH, 'utf-8');
  try {
    return JSON.parse(raw);
  } catch {
    // Corrupted JSON returns empty array
    return [];
  }
}

Write Operations

function writeChats(chats) {
  // Atomic write with pretty formatting
  fs.writeFileSync(STORE_PATH, JSON.stringify(chats, null, 2));
}

Atomic Operations

Read-Modify-Write: All operations follow atomic pattern
File Locking: Synchronous operations prevent concurrent access
Error Recovery: Corrupted files default to empty state

Core Storage Functions

getAllChats()

export function getAllChats() {
  return readChats();
}

Returns complete chat array
Handles missing file gracefully
Recovers from JSON corruption

addOrUpdateChats(newChats)

export function addOrUpdateChats(newChats) {
  const chats = readChats();
  const byId = Object.fromEntries(chats.map(c => [c.id, c]));

  // Merge new chats (overwrites existing by ID)
  for (const chat of newChats) {
    byId[chat.id] = chat;
  }

  writeChats(Object.values(byId));
}

Merge Strategy: ID-based deduplication
Conflict Resolution: Last-write-wins (no timestamp comparison)
Batch Operations: Processes multiple chats in single write

Data Consistency Model

Consistency Guarantees

Atomic Writes: Complete file replacement ensures consistency
Read-After-Write: Immediate consistency for single server
No Transactions: Simple model without ACID guarantees

Conflict Resolution

// Simple ID-based merge
const existing = chatsById[newChat.id];
const merged = newChat; // Always use incoming chat (last-write-wins)

Resolution Strategy:

Incoming chat completely replaces existing chat
No field-level merging or timestamp comparison
Client is authoritative for chat state

Concurrency Handling

Single Process: No concurrent access protection
Synchronous I/O: Blocking operations prevent race conditions
File System: OS-level file locking provides basic safety

File Management

File Path Resolution

import { fileURLToPath } from 'url';
import path from 'path';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const STORE_PATH = path.join(__dirname, 'chatStore.json');

File Creation

// File created automatically on first write
if (!fs.existsSync(STORE_PATH)) {
  writeChats([]); // Initialize with empty array
}

Error Handling

function readChats() {
  if (!fs.existsSync(STORE_PATH)) return [];

  try {
    const raw = fs.readFileSync(STORE_PATH, 'utf-8');
    return JSON.parse(raw);
  } catch (error) {
    console.error('Failed to read chat store:', error);
    return []; // Graceful degradation
  }
}

Performance Characteristics

Read Performance

Cold Read: Disk I/O + JSON parsing
File Size Impact: Linear with number of chats
Typical Latency: <10ms for small files (<1MB)
Memory Usage: Entire file loaded into memory

Write Performance

Full File Rewrite: Always writes complete JSON
No Incremental Updates: Simple but inefficient for large datasets
Formatting Overhead: Pretty-printed JSON (2-space indentation)
Atomic Safety: Prevents partial writes

Storage Efficiency

JSON Overhead: ~20-30% storage overhead vs binary
Pretty Printing: Additional ~15% for readability
Compression: Not implemented (could reduce by ~70%)

Scalability Limits

Current Limitations

File Size: Single file grows with chat count
Memory Usage: Entire dataset loaded for each operation
Concurrent Access: No multi-process support
Backup/Recovery: Manual file management required

Scaling Thresholds

Small Scale: <1,000 chats, <10MB file - Good performance
Medium Scale: <10,000 chats, <100MB file - Acceptable performance
Large Scale: >10,000 chats, >100MB file - Performance degradation

Performance Degradation

Chat Count  |  File Size  |  Read Time  |  Write Time
1,000       |  1MB        |  <10ms      |  <50ms
10,000      |  10MB       |  <100ms     |  <500ms
100,000     |  100MB      |  <1s        |  <5s

Migration Considerations

Database Migration Path

For production scaling, consider migration to:

SQL Database (PostgreSQL)

CREATE TABLE chats (
  id VARCHAR PRIMARY KEY,
  title TEXT NOT NULL,
  messages JSONB,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  last_modified TIMESTAMP,
  model VARCHAR,
  provider VARCHAR,
  is_archived BOOLEAN DEFAULT FALSE
);

CREATE INDEX idx_chats_last_modified ON chats(last_modified DESC);

NoSQL Database (MongoDB)

// Document-based storage maintains JSON structure
{
  _id: ObjectId(),
  id: "chat-123",
  title: "Example Chat",
  messages: [...],
  createdAt: ISODate(),
  updatedAt: ISODate(),
  lastModified: ISODate()
}

Key-Value Store (Redis)

// Hash-based storage for individual chats
HSET chat:123 title "Example Chat"
HSET chat:123 messages JSON.stringify(messages)
HSET chat:123 lastModified "2024-01-15T10:30:00.000Z"

Backup and Recovery

Manual Backup

# Simple file copy
cp chatStore.json chatStore.json.backup.$(date +%Y%m%d)

# Compressed backup
gzip -c chatStore.json > chatStore.json.$(date +%Y%m%d).gz

Automated Backup

// Backup before each write
function writeChats(chats) {
  // Create backup
  if (fs.existsSync(STORE_PATH)) {
    const backup = STORE_PATH + '.backup';
    fs.copyFileSync(STORE_PATH, backup);
  }

  // Write new data
  fs.writeFileSync(STORE_PATH, JSON.stringify(chats, null, 2));
}

Recovery Procedures

Corruption Recovery: Delete corrupted file, restart with empty array
Backup Restoration: Copy backup file to chatStore.json
Data Loss Prevention: Regular backups before major operations

Security Considerations

File System Security

File Permissions: Restrict access to server process user
Directory Security: Secure server directory permissions
Backup Security: Encrypt backup files for sensitive data

Data Protection

# Secure file permissions
chmod 600 chatStore.json          # Owner read/write only
chmod 700 /path/to/server/        # Owner access only

Access Control

No Authentication: Current implementation has no access control
File-Level Security: Relies on OS file permissions
Network Security: HTTPS recommended for data in transit

Monitoring and Observability

File System Monitoring

// Monitor file size growth
const stats = fs.statSync(STORE_PATH);
console.log(`Chat store size: ${stats.size} bytes`);

// Monitor disk space
const free = fs.statSync('.').free;
console.log(`Available disk space: ${free} bytes`);

Operation Logging

function writeChats(chats) {
  const startTime = Date.now();
  fs.writeFileSync(STORE_PATH, JSON.stringify(chats, null, 2));
  const duration = Date.now() - startTime;

  console.log(`Wrote ${chats.length} chats in ${duration}ms`);
}

Health Checks

File Existence: Verify store file exists and is readable
JSON Validity: Parse test to ensure file isn't corrupted
Disk Space: Monitor available storage space
Performance: Track read/write operation latency

PreviousClient Storage NextData Models

Last updated 16 days ago