Part 2 - Tips and Advice
Try to keep your Docker images small (e.g. base them on busybox, alpine, or debian base images)
Don't use Docker containers like full blown VMs - Run a single process/service in each Docker container
Use Docker Networks as an alternative to legacy links for communicating between containers
Consider using OS service scripts * (e.g. Upstart, Systemd) to manage your containers (e.g. pass environment variables, keep them running, start on
boot and/or in a specific order)
(*) We mean outside the container (on the host); avoid running an init system inside the container.
Part 3
We will look at advanced concepts that are needed when deploying your Dockerized applications to production
You will learn about:
Docker networking
$ docker network ls
$ docker network [create|rm|inspect|connect|disconnect] [NETWORK]
$ docker run --network=[NETWORK]
Docker default networks
None: Disable container networking
Host: Containers IP config == Host IP config, public/direct network exposure
Bridge(docker0): Virtual ethernet bridge that connects to containers eth0 interfaces
https://docs.docker.com/v1.10/engine/userguide/networking/ https://www.oreilly.com/learning/what-is-docker-networking
Multi host networks
Container-to-container communication across multiple hosts
Grouping containers by subnet
Better network isolation of containers
Relies on overlay traffic encapsulation (tunneling)
docker network create --subnet=10.0.1.0/24
Examples: Flannel for Kubernetes, Swarm mode / overlay driver
https://www.singlestoneconsulting.com/~/media/files/whitepapers/dockernetworking2.pdf Advanced concepts: https://blog.docker.com/2016/08/docker-online-meetup-41-deep-dive-docker-1-12-networking/
Overlay Networks
Overlay networks are part of User-Defined Networks
Supports multi-host networking across Docker machine clusters
Service discovery via embedded DNS server
Relies on traffic encapsulation via tunnelling
Require an external key-value store like Consul, Etcd, and ZooKeeper
Container Network Model
Clustering
For production
A real-world production environment does not usually consist of a single host running the application and services
We want to:
Reduce single points of failure in the system
Increase availability of our applications and services
Horizontally scale our applications and services
We can do this by:
Running multiple instances of each application/service behind load-balancers
Distributing our instances across multiple hosts
Cluster concepts
Dynamic load balancing
Service discovery
Desired state
Self healing
Auto scaling
Automated host provisioning
Load balancing & Service discovery
Application instances and locations are not static
Desired state
Based on the reconciliation model
Concept similar to CM tools: Chef/Puppet/Ansible
Configuration described the desired state of infrastructure
Cluster manager guarantees the desired state
For example:
Desired state:
"I want at least 2 instances of my frontend service to be running"
If one of the frontend containers goes down, the cluster manager creates a new one
Self healing
Design for failure
Service health
Failure detection and automated remediation based on service healthchecks
Cluster manager checks if each service is healthy
If yes, nothing to be done
If not, different healing mechanisms: Service relocation / restart / replication / auto-scaling
Host health
Auto scaling
Host level
Need experimentation to adapt it to your applications
Auto scaling
Container level
Hosted Container-As-A-Service Offerings
Automated Container Provisioning
Kubernetes - Pod definition / Replication controller
apiVersion: v1
kind: ReplicationController
metadata:
name: busmeme-rc
labels:
name: web
spec:
replicas: 2 # tells deployment to run 2 pods matching the template
selector:
name: web
template: # create pods using pod definition in this template
metadata:
labels:
name: web
spec:
containers:
- image: mongo
name: mongo
ports:
- name: mongo
containerPort: 27017
hostPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- image: minillinim/busmemegenerator
name: web
ports:
- containerPort: 3000
env:
- name: NODE_ENV
value: "production"
- name: PORT
value: "3000"
- name: BM_MONGODB_URI
value: "mongodb://localhost/app-toto"
volumes:
- name: mongo-persistent-storage
hostPath:
path: /data/db
Docker Swarm mode
Cluster manager and workers
https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/
Docker Swarm mode
Docker EE for AWS
https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/
Docker Swarm mode
Services and Tasks
https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/
Docker Compose V3
Docker now supports multi-host deployments with Docker Compose files via the stack concept
Simple, consistent laptop (single host) to production workflow (Swarm mode cluster) experience
Imperative to declarative scripts
Exercise: Clustering concepts
In this exercise we create a Docker Swarm-mode cluster running with 3 Docker hosts
Complete the exercise: 03-advanced-clustering
Managing data in a container
Managing data in a container
Data is coupled with the lifetime of the container
Data cannot be persisted
Same data set cannot be shared between containers
Data Volumes
Directories/Files outside of the Union File System
Normal directories on the host file system
I/O performance on volume is exactly same as I/O on host
Volume content is not included in the docker image
Any change to volume content (via RUN) is not part of image
Can be shared and reused across containers
Persists even if the container itself is deleted
Data persistence across Docker hosts
Distributed filesystems
Data can be shared and reused across hosts
NFS / GlusterFS / Ceph / Flocker
Volumes mounted from host
Volumes mounted from other containers (data-only containers)
Volumes mounted from other containers (data-only containers)
Volumes mounted from host vs data-only containers
Possible mount points as Data Volumes
docker run -v ~/.a-file:/destination
Host directory
ocker run -v /src/:/destination
Shared-storage volume
docker run --volume-driver=flocker -v a-named-volume:/destination
Mount from data volume container
docker create -v /some-data --name store
docker run --volumes-from store
Security aspects
Containers reduce attack surfaces and isolate applications to only the required components, interfaces, libraries and network connections
Aaron Grattafiori, NCC Group
Docker security efforts
Docker is part of the Vendor Security Alliance (VSA )
Members include: Docker, Uber, Dropbox, Palantir, Twitter, Square, Atlassian, Godaddy and Airbnb
Docker Security Bench
Docker security benchmark
Security levels
Docker daemon configuration
Docker registry
Images
Container-to-host security
Container-to-container security
Container security
Secrets management
Docker daemon security
Mitigate container breakouts
Docker registry security
Do not use insecure registries
Use TLS/SSL communication with Docker clients
Enterprise-class authentication: AD/LDAP support
Role based access control: Who is able to pull or push images?
Tools on top of Docker registry:
Image security
Build your own images if possible
Use only trusted base images from the Docker Hub / Store
Use SHASUM check, Docker Trusted Registry, or setup Notary
Rebuild the images to include security patches
USER instruction to set UID to use when running the image
Avoid large distributions (Ubuntu, CentOS) / prefer minimal distributions: Alpine, BusyBox, SCRATCH
Reduces attack surface
Less patching requirements
Download time decreased
Docker vulnerability scanner
Source Docker Hub (Must be logged in)
Container-to-host security
Since Docker 1.11, number of active processes running inside a container can be limited to prevent fork bombs
This requires a linux kernel >= 4.3 with CGROUP_PIDS=y to be in the kernel configuration
$ docker run --pids-limit=64
Container-to-container security
Least access strategy
Restrict incoming traffic to the right host interface
$ docker run --detach --publish [ip]:[source-port]:[dest-port] [image]
Mount root file-system or volumes as read-only
$ docker run -d --read-only [image]
$ docker run -d -v /src/webapp:/webapp:ro [image]
Container security
Mandatory Access Control (MAC)
Reduce kernel capabilities
Use non privileged containers
Secrets management
Do not store secrets (Tokens, Keys, Passwords) in Dockerfiles and/or images
Do not store secrets in environment variables because they are shared by the system/docker inspect: use KV stores (Vault)
Use secret management Key-Value stores (e.g. Vault , AWS KMS )
Monitor process / resources usage
Some useful tools to monitor containers:
$ docker top
$ docker stats
Control groups & Namespaces
Control groups
Resources isolation with cgroups: CPU, Memory, Block I/O, Network, Devices
Namespaces
Processes isolation with namespaces
Resource isolation - control groups
Namespaces
Limit the resource visibility
Processes with their own view of the system
pid: Each PID namespace has its own numbering (parent PID 1)
net: Define network interfaces / rules
mnt: Provide its own rootfs (~chroot)
user: Map processes GID/UID to host GID/UID
uts: Hostname and NIS domain name
ipc: System V IPC, POSIX message queues
Process isolation - namespaces
Exercise: Monitoring
In this exercise we monitor the Voting App container with cAdvisor
Complete the exercise: 03-advanced-monitoring
Logging
Why is it important?
Ephemeral nature of containers implies loss of context: where/for how long/which containers are running?
Log correlation between containers
Need for live analysis
Retention of information in case of catastrophic events
Centralised logging
Analyze cluster-wide issues
Docker daemon logs
Centralized and remote containers logging
Containers lifecycle / Exit signals
Health check / Restart policies
Container logs
Log correlation / log content is important
Timestamp with time zone (Ideally UTC)
Docker host FQDN/IP
Container ID
Image name and tag
Logging strategies
Depends on OS / existing log strategy / centralized logging solution
Default logging driver: json-file
Other options: journald, fluentd, splunk, awslogs
Log collectors as containers and host log forwarding: logspout, sumologic
/!\ Volumes of logs /!\ Log format
Exercise: Logging
In this exercise we use Fluentd to collect and store Docker containers' logs
Complete the exercise: 03-advanced-logging
Continuous Integration
Automated build
Build images on git repository events: push, tags
CI: Matrix testing
Refactoring / upgrades made easier
Use Docker for regression testing or to test new OS/versions
Run tests in parallel
For example, when you want to run tests like this:
CI: Dockerfile/image quality checks
Lint your Dockerfile
Good images respect simple but important principles
Hadolint -- Dockerfile linter
Useful rules:
DL3002: Do not switch to root USER
DL3007: Using latest is prone to errors if the image will ever update
DL3020: Use COPY instead of ADD for files and folders
CI: other considerations
Create a separate partition to extend disks
Frequently run docker-gc to remove old containers/images/volumes
Part 3 - Tips and Advice
Volumes != Persistance
Volumes are not garbage collected
It is ok to use same images for stateful (database) container and its corresponding data-only container
Use load balancers (like HAProxy) to scale your services
Put containers with sensitive data on their own host
Ensure your services provide healthcheck endpoints so that load balancers can stop routing traffic to unhealthy instances
Cache images whenever possible, download takes a long time
Part 3 - Wrap-up
Learnt about:
Cluster concepts
Networking, Storage, Security
Resource management / Monitoring
Logging
CI/CD, Testing
Which platform should I chose?
It depends!
Docker CE
Community edition
ASsembled from upstream components in Moby project
Docker EE
Enterprise version of CE
Commercial support
Certified components, containers and plugins
Docker Trusted Registry (DTR), RBAC, LDAP/AD integration
Docker Security Scanning, Multi-tenancy
Docker Cloud
Cloud based continuous delivery
Automated builds, testing, security scanning
Teams and organisations
Hosted registry (Docker Hub)
Provision and manage swarms -- Swarm mode
Docker for Mac/Windows
More native experience that than Docker Toolbox
Integrates with Docker Cloud (Swarms)
More efficient use of resources (xhyve, Hyper-V) - no need for Virtualbox and Docker Machine
Docker for AWS/Azure/GCP
Fully-baked, opinionated, tested setup for cloud providers
Deploy via templates for CloudFormation, Resource Manager, Deployment Manager
Kubernetes
The most widely adopted container orchestration platform
Designed around Google’s internal experience of operating containers at scale
Helm – Kubernetes Package Manager
Cloud offerings
IBM BlueMix Container Service
Google Container Engine (GKE)
Azure Container Service
Rancher (with Kubernetes backend)
Deis Workflow (Heroku-12 factor-like PaaS built on top of Kubernetes)
Kubernetes
AWS options
AWS EC2 Container Service (ECS)
Managed container scheduling and orchestration
Uses AWS resources under the hood (EC2, ALB, IAM, etc.)
Only works on AWS
AWS Elastic Beanstalk
Simplified, opinionated infrastructure setup on AWS
Single container – EC2
Multi-container – ECS
Docker CE/EE for AWS (discussed earlier)
Part 5
Next steps, future trends and Q&A
Next steps after this tutorial
Congratulations! You've made it to the end :)
What now?
Go back over the tutorial material on your own:
Review scripts, configuration, source code for the app and services, etc.
Read through the linked references in the slides and READMEs
Try the suggested Homework in the READMEs
Experiment!
Adopting Docker In Your Own Organisation?
Think about Dev & Test first...
Then focus on getting one app or service into production!
Dev:
Reproducible dev environments (alternative to full VMs, e.g. vagrant + virtualbox) - docker machine can help here
Build containers, i.e. containers with build dependencies in them. E.g. node+npm, java, ruby. Get them off your local machine.
Test:
Build dependencies in Docker containers - keep CI boxes lean and similar
Prod:
Docker images as artefacts
Docker for Dev & Test
Consider using it as a way to create prod-like environments on developers' laptop
Use it in your CI process to build up and throw away instances of your app
Run your build tool chain from within containers too!
Your first app...
Once you're happy with Dev & Test, get an app into prod!
Start with something low risk, and ideally stateless, but that does have a good change frequency
Sit with it a bit on prod before you take the next steps
Other Advice
Get log aggregation and monitoring in place early
Only look at a cluster management once when you understand Docker and its edge-cases properly
You may not need advanced schedulers like Mesos and Kubernetes for your Enterprise apps just yet - but it's worth understanding what they provide to know if you need them or can just get away with swarm-mode and your own scripts
Embrace the Docker mindset!
Really make sure the Docker mindset is understood - otherwise you may end up fighting the tech, so...
One app/service per container
Images as build artifacts
Immutable servers!
Cluster management, scheduling and orchestration
Hosted Container-As-A-Service (CaaS) Offerings
Self-hosted Container-based CaaS/PaaS
Short-lived Containers!
Spin up containers on demand, perhaps for specific users
Google do this all the time!
Can be a great way to reduce resources required to run your system...
...and keep process/data isolated...
...and improve security...
...but you'll have to do some work yourselves to make this work!
Unless you can use something for this purpose, like...
...AWS Lambda (or Azure Functions ,
Google Cloud Functions )
...aka "Serverless " computing
Storage
"Storage is hard" [1] ...but, still a lot going on in this area.
Selection of Volume Plugins
Flocker (ClusterHQ):Volume portability, supports many backends
netappdvp (NetApp): iSCSI/NFS on NetApp storage
blockbridge-docker-volume (Blockbridge): Vendor-specific
Netshare (ContainX): NFS, AWS EFS & Samba/CIFS
REX-Ray (EMC): AWS EBS, GCE, Cinder, EMC, VirtualBox
Contiv Storage [2] (Cisco): Ceph, NFS
Convoy (Rancher): Volume backup/restore, snapshot, migration
Open Storage : Stateful services in multi-host Docker environment
[1] Brian Goff, Core Engineer, Docker
[2] Contiv Storage is still in alpha phase
Networking
Overlays
Weave Net : Network plugin supporting VXLAN, Service Discovery (DNS), Encrypted traffic (IPSec), Load Balancing
libnetwork : Core Docker library supporting driver / plugin model (built-in Overlay needs external KV-store)
Swarm Mode Overlay : New in Docker Engine 1.12
Alternatives
Calico : Can be deployed without encapsulation or overlays to provide high performance at massive scales
Contiv Networking : Pluggable networking alternative to built-in Docker, Kubernetes, Mesos, and Nomad ecosystems
Unikernels!
Allows you to build specific purpose kernels for a given process
Promise to be even leaner than containers...
And provide better isolation, as the kernels are separate
Could provide the speed/lightweight nature of containers, with the improved separation of VMs
But tooling in this space is where it was for containers several years ago, although...
Docker Ecosystem
WARNING : While the core Docker system is mature, the wider ecosystem is still undergoing lots of change. The options around scheduling tools, platforms, etc., based on top of Docker are changing rapidly!