AWS End-of-Day Trading Data Storage

The Problem

Compliance required collecting end-of-day trading data from 15 external trading platforms for internal trade spoofing surveillance, books and records retention, and external audits (including FRB reviews).

The data needed to be immutable, encrypted, auditable, and cost-effective for permanent storage. Legacy on-prem solutions were expensive and operationally rigid.

This was the team's first cloud deployment — no existing playbook.

What I Built

I owned the architecture decision (S3 vs SingleStore vs on-prem) and implemented the entire solution:

Secure Data Ingestion

• SFTP pipelines from 15 external platforms
• Dual authentication: username/password + RSA key-based
• Retry mechanisms, alerting, and support escalation for vendor outages
• Multipart uploads for reliable S3 ingestion

Compliant Storage

• AWS S3 with ARN-controlled access policies (least privilege)
• Immutable storage guarantees — no modification of historical data
• Encryption in transit (SFTP) and at rest (S3)
• Audit logging for compliance reviews and external auditors

Cost Optimization

• Lifecycle policies to transition data to Glacier after 15 days
• 35% permanent cost reduction vs on-prem

Infrastructure as Code

• Deployed entire solution via IaC: S3 buckets, lifecycle rules, access policies, SFTP config, encryption, monitoring

Impact

Why This Was Hard

• First Cloud Deployment — Required multiple PoCs, architecture reviews, security sign-offs
• Compliance Constraints — Immutability, encryption, auditability all mandatory
• Cross-Team Coordination — Worked with compliance, infrastructure, and 15 external vendors
• Architecture Decision — Evaluated S3 vs SingleStore vs on-prem on cost, scalability, and compliance

My Role

Full ownership. I made the architecture decision, implemented all S3 policies, lifecycle rules, SFTP setup, and IaC. Coordinated with compliance, infra teams, and external vendors.