Estimated Time:less than a minute

Spark interactions with Postgres

Both spark and sqoop allow nativelly to ingest/load relational databases such postgresql. After being disapointed by both, I have finally developped an optimized library to better bridge spark and postgres: reading from postgres, writing to postgres from spark.

The spark-postgres library

Comparison with Sqoop

Automation with yaml