14

If you ever been bitten by a plugin upgrade that broke some functionality you must have given some thought about this problem: What should be the Jenkins plugins upgrade policy? How do you test changes before deploying them?

Has anyone gone as far as having a test instance running dummy jobs to test the new versions or do you just pray that upgrading the versions won't break anything?

Improve this question
5
  • Do you mean Jeankins' team policy or your (organisation's) policy?
    – Dan Cornilescu
    Commented Mar 5, 2017 at 4:22
  • I would take a snapshot of the Jenkins node before upgrading and just test it out. In my experience Jenkins has never been a mission critical component. If it's "down" for 15 minutes because some plugin upgrade broke it, it's usually not blocking production in any way, so manual intervention is acceptable. Of course, if that's not the case for you (and Jenkins must be 100% HA), it's not the right approach.
    – Assaf Lavie
    Commented Mar 5, 2017 at 8:24
  • @DanCornilescu my organisation policy as this is for our internal Jenkins server
    – Michael Pereira
    Commented Mar 6, 2017 at 19:58
  • @AssafLavie That's highly dependent on the way Jenkins is running: Standalone server, VM, docker container, kubernetes pods (our case). It may not be easy to take a snapshot of the current state to restore it as is. In our case we can clone the EBS volume holding the Jenkins data but it's a manual and time-consuming process to restore both the container and data volume to a specific state.
    – Michael Pereira
    Commented Mar 6, 2017 at 20:01
  • Hi @MichaelPereira if either of the two answers below have solved your question please consider accepting it by clicking the check-mark. This indicates to the wider community that you've found a solution and gives some reputation to both the answerer and yourself. There is no obligation to do this. If you don't feel that your question has been answered please do feel free to engage with the authors in the comments.
    – Richard Slater
    Commented Apr 4, 2017 at 14:15

3 Answers 3

Reset to default
4

According to the policies of the company where I work we have dev, preprod and prod environments (on some service dev can be missing). And the path of new version preprod->tests->validation->prod.

In our case jobs in preprod are heavy and complex enough to be sure we do not need to pray when implement in prod :)

Note: We use svn to maintain and deliver configuration. We do not make changes inplace.

Improve this answer
3
  • How do you maintain the configuration of the different Jenkins servers? Manually?
    – Michael Pereira
    Commented Feb 28, 2017 at 17:26
  • We use svn to maintain and deliver configuration. We do not make changes inplace
    – Romeo Ninov
    Commented Feb 28, 2017 at 17:27
  • I feel like this doesn't quite answer the question wholly. This answer describes how you deploy changes, but not how you test the changes through the deploy pipeline.
    – jayhendren
    Commented Feb 19, 2018 at 20:12
2

We needed a 100% HA Jenkins environment. we often upgrade plugins / Jenkins itself.

This causes a big headache if the build breaks after the upgrade.

The safest way to sort this is really getting a Demo Jenkins setup. Maybe on the same machine using multiple Tomcat application, you can achieve this cheaper.

What we did is create a separate (Demo)VM and replicated the prod set-up on the Demo VM. Before changing / upgrading anything we would take a snapshot of both VMs. Then we would test the upgrades on Demo VM. If it works good, change it on Prod.

I guess you can look up the community (like SE/SO) if anyone has faced any problems with the plugin you are planning for.

Improve this answer
0

I'd always manually trigger a re-run or two on at least one recent green (or almost green) label on every relevant project/branch that uses the respective plugin and check that I get the same results. Just to be on the safe side.

Any result discrepancy would need to be investigated to determine it they're caused by the plugin update or not. Maybe a few more re-runs with both the old and new plugins?

Improve this answer
2
  • Sure, no problem.
    – Dan Cornilescu
    Commented Mar 15, 2017 at 20:08
  • From past experience my opinions often aren't that popular, so in general I tend to avoid the spotlight if possible :) I'm also unfamiliar with the mod tools. But I don't mind helping, especially if it is needed - I have high hopes for this site.
    – Dan Cornilescu
    Commented Mar 16, 2017 at 0:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.