Storage systems and comparisons
View All Tags
As organizations move workloads to the cloud, one of the most common questions is: should I use HDFS or Amazon S3 as my Hadoop storage layer? Both are valid choices, but they have very different performance profiles and operational characteristics.
The s3a:// filesystem connector in Hadoop lets you use Amazon S3 as a drop-in replacement for HDFS storage. It's the foundation for cost-effective data lake architectures where compute and storage are decoupled. This guide covers configuration, performance tuning, and production best practices.
Three SQL engines dominate the Hadoop data lake landscape: Apache Hive, Presto, and Trino (Presto's open-source fork). Each evolved to solve different problems. Picking the wrong one leads to either unbearably slow interactive queries or over-engineered infrastructure for simple batch ETL. Here's how they compare.