Continuous Delivery with Jenkins

Our experience

& JenkinsPipelineUnit

Ozan GUNALP - Emmanuel QUINCEROT

LesFurets.com

  • ~70 people
  • 4 feature teams
  • 22 developers
  • 3 devops
  • 7 business analysts

Continuous delivery

  • 200 releases a year
  • 30~ branch validations every day
  • 15 deployments a day

Continuous Merge

Feature Branching + Continuous Integration

  • 1 development = 1 branch
  • Staging developments = 1 merge branch
  • 1 release = 1 release branch

Our environments

  • Dev
  • Stage
  • Preprod
  • Prod
  • + 30 exposed dockers for insurer tests

Once upon a time...

Teamcity

  • CI
  • Flyway
  • Continuous Merge
  • Packaging
  • Release

Jenkins

  • Deployments
  • Seleniums
  • Tooling
  • Data Batches

Two systems: unnecessary complexity

Deployment to Prod

  • Create release branch
    • Packaging
    • CI
    • Nightly
  • Deploy to PreProd
  • Seleniums
  • Find packaging ID
  • Deploy to Production

Teamcity configuration

Our problem

  • Cost
  • Complexity
  • Lack of flexibility

Now in 2017

Jenkins only!

  • CI
  • Flyway
  • Continuous Merge
  • Packaging
  • Deployment
  • Seleniums
  • Data Batches
  • Release
  • Tooling

The real life

You said it's Groovy?

Continuation Passing Style

Groovy interpreter that runs Groovy code in the continuation passing style, so that the execution can be paused any moment and restarted without cooperation from the program being interpreted.

Be Serializable or you're dead


@NonCPS
def parseJson() {
    return new JsonSlurper().parseText('{"tag":"jenkins"}')
}
                    
node() {
    def a
    stage('List') {
        for (def item : parseJson()) {
            a = item
        }
    }

    stage('Print') { echo a.tag }
}


            

java.io.NotSerializableException: java.util.TreeMap$Entry
            

This is serializable


import groovy.json.JsonSlurper

@NonCPS
def parseJson(String fieldName) {
    return new JsonSlurper().parseText('{"tag":"jenkins"}')?."$fieldName"
}

node() {
    String output
    stage('List') {
        output = parseJson('tag')
    }

    stage('print') { echo output }
}
            

NonCPS calling CPS


node() {
  stage('List') {
    List<Long> list = createList()
  }
}
@NonCPS
long getLong() {
  return 2L
}

@NonCPS
List<Long> createList() {
  return [getLong()]
}
                

hudson.remoting.ProxyException:
org.codehaus.groovy.runtime.typehandling.GroovyCastException:
Cannot cast object '2' with class 'java.lang.Long' to class 'java.util.List'
                    

Multibranch pipelines and workspaces

  • One branch validated on many agents
  • One workspace per branch on each agent
  • 100 branches x 1GB
  • No automatic cleanup

No space left on disk

Multibranch pipelines and workspaces

First try

  • Clear workspace at the end of the job
  • Shallow clone for some tasks

Result

  • Disk usage trouble solved
  • Clone from scratch, slow and intensive network usage

Multibranch pipelines and workspaces

Second try

  • Share the workspace among branches, with ws
  • Always start with git clean -xdf


Much closer to the TeamCity behaviour

Script Loader - why?

  • Many Jenkins scripts
  • Each script has its own structure
  • Duplicate code
  • Very long scripts

Script Loader - goals

  • Single entry point
  • Force a unique script structure
  • Reuse code
  • Optimize the loading

Script loader - Project structure


src/main/jenkins
├── job                         All Jenkins jobs declarations
│   ├── deploy.jenkins
│   ├── integration.jenkins
│   ├── seleniums.jenkins
│   └── sonar.jenkins
├── lib
│   ├── commons.jenkins         Common function declarations
│   └── scriptLoader.jenkins    Main loader for scripts
└── step
    ├── deploy                  Jenkins files for Deployment steps
    │   ├── flyway.jenkins
    │   └── tomcat.jenkins
    └── ...
                

Script Loader - Job structure


Map imports() {
    [   commons:   'lib/commons.jenkins',
        flyway:    'step/deploy/flyway.jenkins',
        tomcat:    'step/deploy/tomcat.jenkins',
    ]
}

void execute() {
    stage('flyway') {
        flyway.execute()
    }
    stage('tomcat') {
        tomcat.execute()
    }
}
return this
                    

Script loader - Config in Jenkins


String scriptToLoad = 'deploy.jenkins'
def loadedScript = node () {
   checkout(/* GIT CONFIGURATION */)
   def runner = load 'jenkins/src/main/jenkins/lib/scriptLoader.jenkins'
   return runner.configure(scriptToLoad)
}
loadedScript.execute()
                    

Script loader - Source


def configure(filename) {
    def runnable = load(filename)
    checkIsScript(runnable, filename)
    importAll(runnable)
    return runnable
}

private void importAll(runnable) {
    List l = createList(runnable.imports().keySet())
    for (int i = 0; i < l.size(); i++) {
        String scriptFile = runnable.imports()[l.get(i)]
        def script = load(scriptFile)
        checkIsScript(script, scriptFile)
        runnable."${l.get(i)}" = script
    }
}
return this
                    

There are still risks

The same code for many environments.

One change can cause regression.

Oops! The load test is triggered after deploying the production!

We need something to test and track the impact of our changes!

JenkinsPipelineUnit

pipeline-as-code

  • Cd pipelines are configured/described with code.
  • Pipeline code is versioned in our code base.
  • Unit tests to check what will be executed.

Open Sourced by LesFurets.com

Example Job


node() {
    stage('Checkout') {
        checkout(scm)
        sh 'git clean -xdf'
    }
    stage('Build and test') {
        sh './gradlew build'
        junit 'build/test-results/test/*.xml'
    }
}
                    

Let's test it!

import com.lesfurets.jenkins.helpers.BasePipelineTest
class TestJenkinsfile extends BaseRegressionTestCPS {

    /*...*/

    @Test
    void testJenkinsFile() throws Exception {
        loadScript('Jenkinsfile')
        printCallStack()
    }
}
                    

How does it work?

Transforms and interprets Groovy code as in Jenkins

Intercepts method calls for stack tracing and mocking

JenkinsPipelineUnit

  • Mock your environment
    • Jenkins steps
    • Job environment
  • See the callstacks
  • Test your own functions
  • Check for regressions
  • Integrates with test frameworks like JUnit, TestNG, Spock

Conclusion

Pros

  • Pipeline as code
  • Readable change history
  • Code is reusable
  • Code is testable
  • Isolation with feature branching
  • May evolve with the application

Cons

  • Cost of early adoption
  • Some traps and imperfections

That rocks!

Questions?

Emmanuel Quincerot

Ozan Gunalp

Open Source @ LesFurets
https://github.com/lesfurets

JenkinsPipelineUnit
https://github.com/lesfurets/JenkinsPipelineUnit

Scripted vs. Declarative pipelines

Scripted


node('linux') {
  stage('build') {
    try {
      sh 'mvn clean install'
      sh 'notif_success.sh'
    }
    catch (e) { sh 'rollback.sh' }
  }
}
            

Declarative


pipeline {
  agent linux
  stages {
    stage('build') {
      sh 'mvn clean install'
    }
  postBuild {
    success {
      sh 'notif_success.sh'
    }
    failure {
      sh 'rollback.sh'
    }
  }
}