01 / Flagship / Local + AWS verified
NASA Earth Observation Event Intelligence Platform
A governed NASA FIRMS platform built to preserve scientific source truth while testing deterministic replay, distributed processing, recovery, geospatial serving, and bounded managed-cloud execution.
Problem / Users / Guarantees
Scale the processing—not the scientific claim.
NASA FIRMS observations become more useful when identity, provenance, quality, geospatial grain, and processing history remain explicit. The platform serves Earth-observation analysts, geospatial data teams, downstream analytical applications, and platform operators with reproducible data products and bounded read-only contracts.
The core design separates an event message from its underlying NASA detection and replay lineage. Five million test messages therefore remain 10,000 source detections—not five million new observations.
The platform does not declare wildfire incidents, infer perimeters, estimate burned area or severity, or predict future risk. FIRMS rows remain active-fire or thermal-anomaly detections at their source-defined grain.
- 01 / MessageProcessing eventA replay message is an execution unit—not a new scientific observation.
- 02 / DetectionSource identity10,000 NASA FIRMS detections retain their source-defined grain.
- 03 / LineageReplay identityThe deterministic replay multiplier remains traceable without changing source truth.
- 04 / GoldReplay-neutral outputAnalytical products reconcile to detections rather than replay-message volume.
System / Designed and implemented locally
From governed source to spatial evidence.
NASA FIRMS observations enter a controlled Python ingestion path, retain source identity and lineage, and become deterministic replay events for Kafka and Spark. Governed Silver and replay-neutral Gold Parquet form the rebuild boundary for PostgreSQL/PostGIS; eight read-only FastAPI routes serve an API-only Streamlit dashboard. Airflow coordinates the proven batch workflow and safe-rerun contract.
InputNASA FIRMS · Python ingestion · checksums
Observations enter a controlled Python ingestion path and retain source identity and lineage.
TransformationDeterministic replay · Kafka · Spark batch and Structured Streaming
Deterministic replay events exercise both batch and streaming processing without changing the underlying detection count.
IntelligenceSilver/Gold Parquet · lineage · validation · PostGIS
Governed Silver and replay-neutral Gold Parquet form the rebuild boundary for PostgreSQL/PostGIS.
OutputRead-only FastAPI · bounded cache · Streamlit
Eight read-only FastAPI routes serve an API-only Streamlit dashboard through bounded contracts.
Python · SQL · Apache Kafka · Apache Spark · Airflow · Parquet · PostgreSQL/PostGIS · FastAPI · Streamlit · Docker
Evidence / Verified and measured
Two environments. Explicitly different claims.
- Underlying detections
- 10,000 NASA FIRMS detections selected for controlled replay
- Complete local pipeline
- 1,000,000 Replay events, not one million original observations
- Spark batch
- 149.502 s 4 CPUs · 4 GiB container · 32 shuffle partitions
- Measured throughput
- 6,688.88/s Spark batch events per second
- PostgreSQL/PostGIS
- 1M passed Counts, geometry, identity, lineage, and aggregates reconciled
- Portable suite
- 117 passed 126 collected · 9 expected environment-dependent skips · 0 failed
- AWS managed batch
- 5,000,000 Replay messages processed and independently verified
- AWS throughput
- 8,734.80/s 572.423-second governed application interval
- Physical verification
- 82 matched Every Silver/Gold data-object size and SHA-256
AWS managed batch / Verified · bounded · torn down
Cloud execution with temporary authority.
A private CloudFormation-managed foundation in us-east-1 used KMS-encrypted S3, a dedicated VPC with two private subnets and six endpoints, least-privilege IAM, EMR Serverless 7.10, CloudWatch observability, and Budget/SNS cost controls. It was a bounded batch-verification deployment—not a continuously hosted service.
Input10,000 detections replayed 500 times · encrypted S3 Bronze
The controlled replay produced five million messages while preserving 10,000 distinct source detections.
ProcessEMR Serverless 7.10 · Spark 3.5.5 · Python 3.11
The managed batch ran inside the private CloudFormation-managed foundation in us-east-1.
VerifyIndependent EMR verifier · counts, lineage, science, hashes
Independent verification reconciled counts, identity, lineage, science, and every Silver/Gold object hash.
CloseExecution permission removed · evidence preserved · foundation torn down
Temporary workload authority was removed and the foundation was deleted after evidence preservation.
- Accepted / rejected / duplicate
- 5M / 0 / 0
- Replay-neutral science
- 10K / 5,281 Detections / daily cells
- Estimated EMR compute
- $0.186997 Ancillary and foundation charges excluded
Authorization boundary: temporary workload permission was deleted after verification and permanent StartJobRun returned to implicit denial. The foundation and retained buckets were deleted after evidence preservation.
AWS EMR Serverless · S3 · KMS · IAM · CloudFormation · VPC endpoints · CloudWatch · Budget/SNS
Reliability / Passed
Stable identities make reruns inspectable.
Kafka offset reconciliation, Spark checkpoints, staged database loads, stable run identities, and conflict detection prevent silent duplication or overwrite. A checkpoint restart consumed no new Kafka records; an idempotent PostgreSQL rerun inserted zero rows and recognized all one million existing events.
Kafka and PostgreSQL restart tests preserved committed offsets and serving truth. A deliberate database identity conflict rolled back without partial mutation. Airflow verified ordering, bounded retries, failure propagation, and safe rerun behavior on its integration profile.
Capacity boundary / Passed + failed
The 10M result diverges.
Generation and read-back
Exactly 10,000,000 replay events were generated from 10,000 underlying detections and independently reconciled. Two deterministic outputs produced the same SHA-256 checksum.
Spark processing
The 10M Spark attempt did not complete. It reached a verified local Java heap exhaustion boundary after approximately 629 seconds; no Spark output or admitted manifest was produced.
Meaning: 10M deterministic local generation and verification passed. Local 10M Spark did not. AWS 10M was not run because the verified 5M managed gate had already answered the bounded cloud-execution question.
Limitations / Explicit
What this evidence does not claim.
- Complete local platform validation stops at one million replay events.
- Local 10M generation is not local 10M Spark or full-platform validation.
- AWS processed five million replay messages; AWS 10M was not executed.
- The AWS result is bounded batch verification, not a hosted API/dashboard, availability, concurrency, or production-SLA claim.
- Local Kafka uses one KRaft broker and does not demonstrate broker failover or high availability.
- Recorded local timings are sequential measurements, not multi-user load tests.
- The EMR figure is an estimated compute charge and excludes ancillary/foundation charges while billing ingestion settles.
- FIRMS detections are not incidents, perimeters, burned area, severity, or predictive risk.