S3 Storage Configuration

This guide covers production deployment of FOVEA with S3 storage for video files. For general S3 usage, see the S3 Storage User Guide.

Production Architecture

In production, FOVEA uses S3 for video storage with the following architecture:

┌─────────────┐         ┌──────────────┐         ┌──────────────┐
│   Browser   │────────▶│   Frontend   │────────▶│   Backend    │
└─────────────┘         │  (React App) │         │  (Fastify)   │
      │                 └──────────────┘         └──────────────┘
      │                                                  │
      │                                                  │
      │                                           ┌──────▼──────┐
      └──────────── Presigned URL ───────────────│   AWS S3    │
                                                  │   Bucket    │
                                                  └─────────────┘

Backend generates presigned URLs for video access
Browser streams video directly from S3
No video data passes through backend (efficient bandwidth usage)

AWS S3 Setup

1. Create S3 Bucket

# Using AWS CLI
aws s3 mb s3://fovea-production-videos --region us-east-1

# Configure bucket settings
aws s3api put-bucket-versioning \
  --bucket fovea-production-videos \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-encryption \
  --bucket fovea-production-videos \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }]
  }'

2. Configure CORS

Allow frontend to access S3 directly via presigned URLs:

aws s3api put-bucket-cors \
  --bucket fovea-production-videos \
  --cors-configuration file://cors.json

cors.json:

{
  "CORSRules": [
    {
      "AllowedOrigins": [
        "https://fovea.yourcompany.com"
      ],
      "AllowedMethods": ["GET", "HEAD"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3600
    }
  ]
}

3. Create IAM User

Create a dedicated IAM user for FOVEA:

aws iam create-user --user-name fovea-production

# Attach policy (see below)
aws iam put-user-policy \
  --user-name fovea-production \
  --policy-name FOVEAVideoAccess \
  --policy-document file://policy.json

# Create access key
aws iam create-access-key --user-name fovea-production

policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::fovea-production-videos",
        "arn:aws:s3:::fovea-production-videos/*"
      ]
    }
  ]
}

Save the access key ID and secret access key for environment configuration.

4. Configure Bucket Lifecycle (Optional)

Automatically transition old videos to cheaper storage:

aws s3api put-bucket-lifecycle-configuration \
  --bucket fovea-production-videos \
  --lifecycle-configuration file://lifecycle.json

lifecycle.json:

{
  "Rules": [
    {
      "Id": "ArchiveOldVideos",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 365,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Environment Configuration

Docker Deployment

Update docker-compose.prod.yml:

services:
  backend:
    environment:
      # S3 Storage
      - S3_ENABLED=true
      - S3_ENDPOINT=https://s3.amazonaws.com
      - S3_REGION=us-east-1
      - S3_BUCKET=fovea-production-videos
      - S3_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID}
      - S3_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY}
      - S3_MANIFEST_KEY=manifest.json

      # Security
      - NODE_ENV=production
      - FRONTEND_URL=https://fovea.yourcompany.com

EC2 with IAM Role (Recommended)

For deployments on AWS EC2, use IAM instance roles instead of access keys:

Create IAM Role:

aws iam create-role \
  --role-name FOVEA-EC2-S3-Access \
  --assume-role-policy-document file://trust-policy.json

trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "ec2.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }
  ]
}

Attach Policy:

aws iam attach-role-policy \
  --role-name FOVEA-EC2-S3-Access \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/FOVEAVideoAccess

Create Instance Profile:

aws iam create-instance-profile \
  --instance-profile-name FOVEA-EC2-Profile

aws iam add-role-to-instance-profile \
  --instance-profile-name FOVEA-EC2-Profile \
  --role-name FOVEA-EC2-S3-Access

Attach to EC2 Instance:

aws ec2 associate-iam-instance-profile \
  --instance-id i-1234567890abcdef0 \
  --iam-instance-profile Name=FOVEA-EC2-Profile

Update Docker Compose (no access keys needed):

services:
  backend:
    environment:
      - S3_ENABLED=true
      - S3_REGION=us-east-1
      - S3_BUCKET=fovea-production-videos
      # S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY not needed
      # AWS SDK will use IAM role automatically

CloudFront CDN (Optional)

For global deployment, use CloudFront as a CDN in front of S3:

1. Create CloudFront Distribution

aws cloudfront create-distribution \
  --distribution-config file://cloudfront-config.json

cloudfront-config.json:

{
  "CallerReference": "fovea-videos-cdn",
  "Comment": "FOVEA Video CDN",
  "Enabled": true,
  "Origins": {
    "Quantity": 1,
    "Items": [
      {
        "Id": "S3-fovea-production-videos",
        "DomainName": "fovea-production-videos.s3.amazonaws.com",
        "S3OriginConfig": {
          "OriginAccessIdentity": ""
        }
      }
    ]
  },
  "DefaultCacheBehavior": {
    "TargetOriginId": "S3-fovea-production-videos",
    "ViewerProtocolPolicy": "redirect-to-https",
    "AllowedMethods": {
      "Quantity": 2,
      "Items": ["GET", "HEAD"]
    },
    "ForwardedValues": {
      "QueryString": true,
      "Headers": {
        "Quantity": 0
      }
    },
    "MinTTL": 0,
    "DefaultTTL": 86400,
    "MaxTTL": 31536000
  }
}

2. Update Backend Configuration

services:
  backend:
    environment:
      - S3_ENABLED=true
      - S3_ENDPOINT=https://d1234567890abc.cloudfront.net
      - S3_BUCKET=fovea-production-videos
      # Other S3 vars...

Video Upload Pipeline

Automate video uploads with a CI/CD pipeline:

GitHub Actions Example

.github/workflows/upload-videos.yml:

name: Upload Videos to S3

on:
  push:
    paths:
      - 'videos/**'

jobs:
  upload:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Upload Videos
        run: |
          for video in videos/*.mp4; do
            aws s3 cp "$video" "s3://fovea-production-videos/"
            aws s3 cp "${video%.mp4}.info.json" "s3://fovea-production-videos/"
          done

      - name: Generate Manifest
        run: |
          python scripts/generate-manifest.py
          aws s3 cp manifest.json s3://fovea-production-videos/manifest.json

AWS Lambda Trigger (Advanced)

Automatically update manifest when videos are uploaded:

# lambda_function.py
import boto3
import json
from datetime import datetime

s3 = boto3.client('s3')
BUCKET = 'fovea-production-videos'

def lambda_handler(event, context):
    # List all MP4 files
    response = s3.list_objects_v2(Bucket=BUCKET)
    videos = [
        {
            'key': obj['Key'],
            'size': obj['Size'],
            'lastModified': obj['LastModified'].isoformat()
        }
        for obj in response.get('Contents', [])
        if obj['Key'].endswith('.mp4')
    ]

    # Update manifest
    manifest = {
        'videos': videos,
        'generatedAt': datetime.utcnow().isoformat() + 'Z'
    }

    s3.put_object(
        Bucket=BUCKET,
        Key='manifest.json',
        Body=json.dumps(manifest, indent=2),
        ContentType='application/json'
    )

    return {'statusCode': 200, 'body': f'{len(videos)} videos'}

Trigger this Lambda whenever a .mp4 file is uploaded to S3.

Monitoring and Logging

S3 Access Logging

Enable S3 access logs for audit and debugging:

aws s3api put-bucket-logging \
  --bucket fovea-production-videos \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "fovea-logs",
      "TargetPrefix": "s3-access/"
    }
  }'

CloudWatch Metrics

Monitor S3 usage with CloudWatch:

Requests: Track GET/HEAD request counts
Bandwidth: Monitor data transfer
Errors: Alert on 4xx/5xx errors

Backend Logs

Backend logs S3 operations:

# View S3-related logs
docker compose logs backend | grep S3

# Common log messages
# [INFO] S3 enabled, using bucket: fovea-production-videos
# [INFO] Generated presigned URL for video: video1.mp4
# [ERROR] S3 error: Access Denied

Cost Optimization

1. Use Intelligent Tiering

aws s3api put-bucket-intelligent-tiering-configuration \
  --bucket fovea-production-videos \
  --id AutoTiering \
  --intelligent-tiering-configuration '{
    "Id": "AutoTiering",
    "Status": "Enabled",
    "Tierings": [
      {
        "Days": 90,
        "AccessTier": "ARCHIVE_ACCESS"
      },
      {
        "Days": 180,
        "AccessTier": "DEEP_ARCHIVE_ACCESS"
      }
    ]
  }'

2. Compress Videos

Store videos in efficient codecs:

Codec: H.264/H.265
Container: MP4
Bitrate: Optimize for quality vs. size

3. Monitor Costs

# Get bucket size
aws s3 ls s3://fovea-production-videos --recursive --summarize

# Estimate monthly cost (assuming Standard storage at $0.023/GB)
# For 1TB = $23.50/month storage + data transfer costs

Disaster Recovery

Bucket Replication

Replicate to another region for disaster recovery:

aws s3api put-bucket-replication \
  --bucket fovea-production-videos \
  --replication-configuration file://replication.json

replication.json:

{
  "Role": "arn:aws:iam::ACCOUNT_ID:role/S3-Replication-Role",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {},
      "Destination": {
        "Bucket": "arn:aws:s3:::fovea-backup-videos",
        "ReplicationTime": {
          "Status": "Enabled",
          "Time": {"Minutes": 15}
        }
      }
    }
  ]
}

Backup Strategy

Versioning: Enable bucket versioning (prevents accidental deletes)
Replication: Cross-region replication for redundancy
Snapshots: Periodic manifests saved as backups
Testing: Regularly test restoration from backup bucket

Troubleshooting

Backend Can't Access S3

Check credentials and permissions:

# Test with AWS CLI
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
aws s3 ls s3://fovea-production-videos/

CORS Errors in Browser

Verify CORS configuration on bucket
Check frontend URL matches AllowedOrigins
Inspect browser console for specific CORS error

Presigned URL Expired

Presigned URLs expire after 1 hour. If playback fails mid-stream:

Backend generates fresh URLs automatically
Check for clock skew between client and server

High S3 Costs

Review bucket size: aws s3 ls s3://fovea-production-videos --recursive --summarize
Check request metrics in CloudWatch
Enable lifecycle policies to archive old videos
Consider CloudFront CDN to reduce S3 requests

Production Architecture​

AWS S3 Setup​

1. Create S3 Bucket​

2. Configure CORS​

3. Create IAM User​

4. Configure Bucket Lifecycle (Optional)​

Environment Configuration​

Docker Deployment​

EC2 with IAM Role (Recommended)​

CloudFront CDN (Optional)​

1. Create CloudFront Distribution​

2. Update Backend Configuration​

Video Upload Pipeline​

GitHub Actions Example​

AWS Lambda Trigger (Advanced)​

Monitoring and Logging​

S3 Access Logging​

CloudWatch Metrics​

Backend Logs​

Cost Optimization​

1. Use Intelligent Tiering​

2. Compress Videos​

3. Monitor Costs​

Disaster Recovery​

Bucket Replication​

Backup Strategy​

Troubleshooting​

Backend Can't Access S3​

CORS Errors in Browser​

Presigned URL Expired​

High S3 Costs​

See Also​