Continuous Delivery with Jenkins

Our experience

& JenkinsPipelineUnit

Ozan GUNALP - Emmanuel QUINCEROT

LesFurets - Team structure

  • 4 feature teams
  • 22 developers
  • 3 devops
  • 7 business analysts

Continuous delivery

  • 200 releases a year
  • 30~ branch validations every day
  • 15 deployments a day

Continuous Merge

Feature Branching + Continuous Integration

  • 1 development = 1 branch
  • Staging developments = 1 merge branch
  • 1 release = 1 release branch

Our environments

  • Dev
  • Stage
  • Preprod
  • Prod
  • + 30 exposed dockers for insurer tests

Once upon a time...

Teamcity

  • CI
  • Flyway
  • Continuous Merge
  • Packaging
  • Release

Jenkins

  • Deployments
  • Seleniums
  • Tooling
  • Data Batches

Two systems: unnecessary complexity

Deployment to Prod

  • Create release branch
    • Packaging
    • CI
    • Nightly
  • Deploy to PreProd
  • Seleniums
  • Find packaging ID
  • Deploy to Production

Teamcity configuration

Our problem

  • Cost
  • Complexity
  • Lack of flexibility

Now in 2017

Jenkins only!

  • CI
  • Flyway
  • Continuous Merge
  • Packaging
  • Deployment
  • Seleniums
  • Data Batches
  • Release
  • Tooling

Why Jenkins?

Huge Open Source Community

1000+ plugins
120k+ installations

Pipeline-as-code

Code your Continuous Delivery pipeline

Check-in to your code base

Spoiler Alert : Test it!

You said pipeline ?

How to declare a job?

Basic pipeline

1 pipeline file → 1 job

Multibranch pipeline

1 branch → 1 job defined in Jenkinsfile


Groovy DSL

  • Organize pipeline into (parallel) stages
  • Call any step: bat, sh, retry, timeout...
  • Agent allocation with node
  • Extensible by new and existing plugins (e.g. junit)

Groovy-based DSL


node() {
    stage('Checkout') {
        checkout(scm)
        sh 'git clean -xdf'
    }
    stage('Build and test') {
        sh './gradlew build'
        junit 'build/test-results/test/*.xml'
    }
}
            

Master and slaves

Pipelines? Why?

  • Use the power of Groovy
  • Version your pipeline code in VCS
  • No damage caused by accidental click
  • Steps parallelization
  • Let developers maintain their Jenkins jobs
  • Better UX

... and they lived happily ever after

The real life

Jobs, Lots of Jobs...

  • 10 jobs
  • 30+ environments
  • 100 parallel developments




Pipelines are complex

Job Deploy

  • 14 files
  • 642 lines of code
  • 214 lines of comment

You said it's Groovy?

#alternativefacts

Continuation Passing Style

Groovy interpreter that runs Groovy code in the continuation passing style, so that the execution can be paused any moment and restarted without cooperation from the program being interpreted.

Be Serializable or you're dead


import groovy.json.JsonSlurper

@NonCPS
def parseJson() {
    return new JsonSlurper().parseText('{"aaa":3}')
}
                    
node() {
    def a
    stage('List') {
        for (def item : parseJson()) {
            a = item
        }
    }

    stage('print') { echo a }
}


            

java.io.NotSerializableException: java.util.TreeMap$Entry
            

This is serializable


import groovy.json.JsonSlurper

@NonCPS
def parseJson(String fieldName) {
    return new JsonSlurper().parseText('{"aaa":3}')?."$fieldName"
}

node() {
    String output
    stage('List') {
        output = parseJson('aaa')
    }

    stage('print') { echo output }
}
            

Multibranch pipelines and workspaces

  • One branch validated on many agents
  • One workspace per branch on each agent
  • 96 branches x 1.2GB

No space left on disk

Multibranch pipelines and workspaces

First try:
  • Clear workspace at the end of the job
  • Shallow clone for some tasks

Result:

  • Disk usage trouble solved
  • Clone from scratch, intensive network usage

Multibranch pipelines and workspaces

Next try:

  • Share the workspace among branches
  • Always start with git clean -xdf

Guidelines by LesFurets.com

  • The code simple, you'll keep.
  • One unique structure for every job, you'll follow.
  • The tricks with your team, you'll share.

There are still risks

The same code for many environments.

One change can cause regression.

Oops! The load test is triggered after deploying the production!

We need something to test and track the impact of our changes!

JenkinsPipelineUnit

pipeline-as-code

  • Cd pipelines are configured/described with code.
  • Pipeline code is versioned in our code base.
  • Unit tests to check what will be executed.

Open Sourced by LesFurets.com

Example Job


node() {
    stage('Checkout') {
        checkout(scm)
        sh 'git clean -xdf'
    }
    stage('Build and test') {
        sh './gradlew build'
        junit 'build/test-results/test/*.xml'
    }
}
                    

Unit Test

import com.lesfurets.jenkins.helpers.BasePipelineTest
class TestJenkinsfile extends BaseRegressionTestCPS {

    /*...*/

    @Test
    void testJenkinsFile() throws Exception {
        loadScript('Jenkinsfile')
        printCallStack()
    }
}
                    

Setup Mock Environment


    @Override
    @Before
    void setUp() throws Exception {
        super.setUp()
        helper.scriptRoots += 'src/main/groovy'
        helper.baseScriptRoot = ''
        binding.setVariable('scm', [
                $class                           : 'GitSCM',
                branches                         : [[name: 'AMX-12345_test']],
                doGenerateSubmoduleConfigurations: false,
                extensions                       : [],
                submoduleCfg                     : [],
                userRemoteConfigs                : [[ url          : "/var/git-repo" ]]
        ])
        helper.registerAllowedMethod('junit', [String.class], null)
    }
                    

Call Stack


   Jenkinsfile.run()
      Jenkinsfile.node(groovy.lang.Closure)
         Jenkinsfile.stage(Checkout, groovy.lang.Closure)
            Jenkinsfile.checkout({$class=GitSCM, branches=[{name=AMX-12345_test}], ...
            Jenkinsfile.sh(git clean -xdf)
         Jenkinsfile.stage(Build and test, groovy.lang.Closure)
            Jenkinsfile.sh(./gradlew build)
            Jenkinsfile.junit(build/test-results/test/*.xml)
                    

Non-regression Test


@Test
void testNonRegression() throws Exception {
    loadScript('Jenkinsfile')
    printCallStack()
    super.testNonRegression(false)
}
                    

How does it work?

Transforms and interprets Groovy code as in Jenkins

Intercepts method calls for stack tracing and mocking

JenkinsPipelineUnit

  • Mock your environment
    • Jenkins steps
    • Job environment
  • See the callstacks
  • Test your own functions
  • Check for regressions
  • Integrates with test frameworks like JUnit, TestNG, Spock

Conclusion

Done:

  • Most jobs migrated to pipelines
  • Each pipeline is unit tested

Continuous improvement:

  • More feedback (slack)
  • Better error management
  • More automation

Conclusion

Pros:

  • Flexibility Jobs and application evolve together
  • Safety Job changes are code-reviewable
  • Repeatability with the non regression tests
  • Reusability all environment deploy with the same code

Cons:

  • Missing documentation
  • Cost of early adoption
  • Some traps and imperfections

Questions?

Emmanuel Quincerot

Ozan Gunalp

Open Source @ LesFurets
https://github.com/lesfurets

JenkinsPipelineUnit
https://github.com/lesfurets/JenkinsPipelineUnit

Scripted vs. Declarative pipelines

Scripted


node('linux') {
  stage('build') {
    try {
      sh 'mvn clean install'
      sh 'notif_success.sh'
    }
    catch (e) { sh 'rollback.sh' }
  }
}
            

Declarative


pipeline {
  agent linux
  stages {
    stage('build') {
      sh 'mvn clean install'
    }
  postBuild {
    success {
      sh 'notif_success.sh'
    }
    failure {
      sh 'rollback.sh'
    }
  }
}
            

NonCPS calling CPS


node() {
  stage('List') {
    List<Long> list = createList()
  }
}
@NonCPS
long getLong() {
  return 2L
}

@NonCPS
List<Long> createList() {
  return [getLong()]
}
                

hudson.remoting.ProxyException:
org.codehaus.groovy.runtime.typehandling.GroovyCastException:
Cannot cast object '2' with class 'java.lang.Long' to class 'java.util.List'