Data Pipelines and Docker

docker intro

• data pipeline - stream data from different sources -> processing engines -> redis
• fast iterations
• resilient system over easy deploy
• use case news articles

docker-compose

think micro services, add containers for a new component

• just install docker on remote
• docker images can be pushed to remote repo
• make instances scale in any cloud IaaS
• instance down -> new instance, pull docker images, start containers