Continuous delivery

The French Way

LesFurets.com

• 22 software engineers
• 3 Devops (automation & operations)
• 1 Architect

LesFurets.com

• 1 website, 6 product lines
• 400k lines of code, 30k Unit tests
• 200 Selenium tests
• 10+ Servers
• 1 code base

In 2012

11 releases

• Mind set:
• Plan/Estimate/Code/Test/Release monthly

In 2014

143 releases

• Mind set:
• Ship what is READY the next day

In 2012

• Scrum sprints ended by a release
• Subversion : trunk + 1 maintenance branch
• Build : 15min, Selenium in a grid: 200 in 1h
• Testing phase : 4-6 days, Deployment in 2h
• Commit to Prod : from 5d to 4 weeks

In 2014

• Lean Kanban
• Daily releases
• GIT : master + feature branches
• Build : 3' (was 15') + Selenium : 10' (was 1h)
• Release : 5min + 30min monitoring
• Commit to Prod : coding time + 1 day

A few things we read

Agile Manifesto

Principle 3

"Our highest priority is to satisfy the customer through early and continuous delivery of valuable software."

Product Development Flow

Don Reinertsen

• Work with a flow
• Manage Queues of work
• Lead Time more than work duration
• Reduce per release costs

Continuous Delivery

Jez Humble, David Farley

1. Fast builds
2. Pre-Commit check (all tests ever ok)
3. Simple and fast automated deployments
4. Production monitoring and alerting
5. Root Cause Analysis

Deliver Early, Deliver Often

Source: http://paulhammant.com/2013/03/13/facebook-tbd-take-2/

Kanban : Evolutionary Change

David Anderson

• Visualize the work
• Limit the Work in Process
• Explicit Rules
• Continuous Improvement
• Leadership

How Google Test Software

James Whitaker

• NoQA : Quality Engineering
• SET : build test tools for devs
• Dev QA their own changes

GIT FLOW

Vincent Driessen

• A successful git branching model
• http://nvie.com/posts/a-successful-git-branching-model/

Delivery

Automate

"Infrastructure as code"

Deployment

0 downtime

Blue/green

• Doubled production environment
• Loadbalancer

Validation

• Manual tests
• Regression Testing : Zeno
• Selenium
• Code reviews
• Showcase/functional validation

Zeno

Visual Regression

Selenium

Selenium

• Page Object Pattern
• Fluent API (FluentLenium like)
• Selenium in the sky with LXC
• 200 Selenium tests in <10min

Continuous integration

impacts the codebase

Continuous integration

Branching model

• Trunk + Release branch
• Trunk based (all commits to master + toggle)
• Feature Branching + Merge when ready

Continuous integration

Feature branching

Continuous integration

On every commit

• Compilation
• Automated tests
• Continuous Merge (Octopus)

Continuous Merge

Continuous Merge

• Build a test environment
• Detect conflicts
• Plug/unplug branches easily
• Git Octopus Merge

Managing conflicts

Avoid conflicts vs Resolve conflicts

Managing conflicts

A lot of options

• Remove the conflicting branch (put aside, comeback later)
• Find a way to make it merge
• Merge branches in conflict (resolve the conflict)
• Merge in master (+ merge everywhere)

Merge is the worst case (default in trunk based development)

30 branches at work !