Packaging perl (LPW2010)

Packaging Perl...

Paulo Castro
London Perl Workshop 2010 Perl Developer
Packaging Perl and dependencies Net-A-Porter

Introduction

• Paulo Edgar Castro aka (G@SP@R)
• Portuguese
• System Administrator
• Perl Developer ( writing baby perl since 1999 )
• Arrived in London in August 2007
• Came to learn and to be closer to technology

• No perl jobs in Porto, a lot of proprietary tech

• MobileStreams, RedBeeMedia, Net-A-Porter

Brief Summary

• Where we were / Who we were
• The problems of such approach
• The decision
• Investigation and Initial steps
• Questions
• Tools (part I)
• Problems and pitfalls
• Tools (part II)
• Getting closer
• More problems

Brief Summary

• Evil Hacks
• Wrapping it up
• Pros/Cons
• Final thoughts

Where we were / Who we were

• 64 bit perl 5.8.8 compiled for CentOS 5.2
• GIT repository of perl binaries and CPAN modules
checked out into standard dir (prefix=/opt/zt)
• export PATH=/opt/zt/zt-perl/bin:$PATH
• Developers using different OS'es Arch's ( Ubuntu 32,
Ubuntu 64, Fedora, OSX, Gentoo, Windows ?!?!?
QUACK !!! )
• cpan My::Evil::Module; git commit -am "Install/Update
My::Evil::Module"
• Deployment consisted of App tarball + GIT checkout tarball

The problems of such approach

• Very tricky for developers to update modules which build
.so libs
• .so linked against developer libs which were more recent
than the libs present in CentOS 5.2
• Only a rough idea of the modules we were currently using
• Production/Staging/Testing boxes had to be manually pre-
installed with all the dependencies
(libjpg,libpng,libgd,libdb,...)
• Trial and error deployment into new boxes ( "Oh, we forgot
to install libpng, that's why the bar-codes aren't being
generated!" )
• Only ran tests for modules being installed/updated

The decision

• We needed a way to describe all these related
dependencies
• I suggested RPM since I was very familiar with it ( My first
job in London consisted of packaging up their perl app in a
nice RPM package )
• Our current production system was RedHat
• We didn't seem to really have any other viable options to
accomplish this
• Our ultimate goal was, to be able to deploy our apps into a
minimal installed box and have the whole process figure
out things for us rather than the other way around.
LET'S GO!

Investigation and Initial steps

• We knew more or less what we wanted
• Not entirely sure what the end result was going to be
• We needed to know what we had (CPAN modules
inventory)
• 2 Apps, 2 Makefile.PL with deps, merged together yielded
235 CPAN packages
• Grand total of 618 CPAN modules to satisfy the deps of the
235.
• Dug through CPAN, PerlMonks, Google for answers ....
• cpanspec ?
• cpan2rpm ?

Questions

• 1 perl bin rpm package + n perl modules packages ?
• (this is overkill, who's going to maintain 618 rpm builds)

• 1 perl bin rpm package + 1 perl modules package ?
• 1 perl bin + modules package ?

• How will we handle our 618 sources ?
• CPAN mirror ?

• Customized CPAN repo of the modules we use ?

• How are we going to manage the unattended install of 618
packages ?!?!? 'Oh dear... :( Frack, let me check
jobs.perl.org'

Tools (part I)

• pmtools, to generate us an inventory of what we had
Text::CSV_XS (0.76) - comma-separated values manipulation routines
Compress::Bzip2 (2.09) - Interface to Bzip2 compression library
PPI (1.213) - Parse, Analyze and Manipulate Perl (without perl)
XML::Parser::Expat (2.19) - Lowlevel access to James Clark's expat XML parser
XML::Parser (2.19) - A perl module for parsing XML documents

• A couple of bash scripts to do some simple greping and
parsing of the results ( I know this is a perl workshop, but
bash one-liners seemed good enough for the job )
• Mark Overmeer's CPAN::Site, quote "The cpansite script is
used to create your own CPAN server." - GREAT STUFF!
• How the frack are we going to create an unattended install
of the CPAN deps ? 618 of them ?

Tools (part I)
• I remembered how we were in some cases installing new
modules... ( a way I'd used before I confess ... )
• Add the modules to the project Makefile.PL which is based in
Module::Install
• perl Makefile.PL; make; echo “Module installed”
• What about we assume perl is a project with a Makefile.PL full of
dependencies ?

Tools (part I)

Typical Makefile.PL layout
use strict; use warnings; use inc::Module::Install 0.98;
use Module::Install::AuthorRequires; use Module::Install::ExtraTests; use 5.008003;

name 'Moose';
perl_version '5.008003';
all_from 'lib/Moose.pm';
license 'perl';

requires 'Carp';
requires 'Class::MOP' => '1.11';
requires 'Data::OptList' => '0';
requires 'List::MoreUtils' => '0.12';
requires 'Package::DeprecationManager' => '0.10';
requires 'Params::Util' => '1.00';
requires 'Scalar::Util' => '1.19';
requires 'Sub::Exporter' => '0.980';
requires 'Sub::Name' => '0';
requires 'Task::Weaken' => '0';
requires 'Try::Tiny' => '0.02';

test_requires 'Test::More' => '0.88';
test_requires 'Test::Fatal' => '0.001';
test_requires 'Test::Requires' => '0.05';
......
.....

Problems and pitfalls
• Getting the sources for the modules
• wget ${Module}-${Version}.tar.gz from cpan
• if (!$found) { wget ${Module}-${Version}.tar.gz from backpan }
else { echo "Damn it, GRRRR, $#$%#$%/#$%#$ " }
• Some module versions just couldn't be found anywhere.
• Solution: Install the closest higher version and hope that
everything will be OK
• Other modules had been tweaked in house and their version
bumped manually to say (0.78_02) not matching anything
found in the wild
• This meant, the module was actually a couple versions below
(0.78) and it had a in-house patch applied.

Problems and pitfalls
• Some perl modules had deps on third-party libraries like
mysql-devel, db4-devel, openssl-devel, postgresql-devel,
libxml2-devel, etc...
• Others, Math::Pari, depended on specific tarballs to be
installed
• Some modules didn't pass some tests
• Others didn't pass any test at all....
• How the frack did they got installed in the first place ???

“Oh Lordy!!! Is it time to
check jobserve this time ?”

Tools (part II)
• RPM spec files
• They're just a clever wrapper around a tarball of stuff built in
a temporary directory
• With this wrapper we can
• Specify build requirements
• Specify lib dependencies
• Provide information about our package
• Describe a recipe for the build/installation process
• Changelog of the actions
• Just before creating the final package, foreach existing file,
rpmbuild lists:
• What the file is providing/requiring

Tools (part II)

Provides Example:

Byte.so()(64bit) ByteLoader.so()(64bit) Bzip2.so()(64bit) C.so()(64bit)
Opcode.so()(64bit) POSIX.so()(64bit) mysql.so()(64bit) perl(Algorithm::C3) =
0.07 perl(Algorithm::Diff) = 1.19 perl(Algorithm::Diff::_impl) perl(Catalyst::Log)
perl(Catalyst::Log::Log4perl) = 1.00 perl(Catalyst::Model)
perl(Catalyst::Model::ActiveMQ) perl(Catalyst::Model::DBIC::Schema) = 0.29
perl(Catalyst::Model::DBIC::Schema::Types) perl(Catalyst::Model::File) = 0.08
perl(DBD::mysql) = 4.007 perl(DBD::mysql::GetInfo) perl(DBD::mysql::db)
perl(Moose) >= 0.82 perl(PPI::Token::Pod) perl(PPI::Token::Prototype) .....

Requires Example:

libc.so.6(GLIBC_2.4)(64bit) libcrypt.so.1()(64bit) libgd.so.2()(64bit) libgmp.so.3()
(64bit) libjpeg.so.62()(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit)
libmysqlclient.so.16()(64bit) libnsl.so.1()(64bit) libpng12.so.0()(64bit) .....

Tools (part II)
Example spec file
Summary: A searchable xml/html based mailinglist %install
archiver %makeinstall
Name: lurker install -d -m0755 $RPM_BUILD_ROOT/%
Version: 1.1 {_sysconfdir}
Release: 1.evo.2 install -m0755 lurker.conf $RPM_BUILD_ROOT/%
Group: Applications/Internet {_sysconfdir}/lurker.conf
License: GPL install -d -m2775 $RPM_BUILD_ROOT/%
URL: http://lurker.sourceforge.net/ {_localstatedir}/www/lurkdb
Packager: Jim Perrin <perrin@ohio.edu> %clean
Source0: lurker-%{version}.tar.gz rm -rf $RPM_BUILD_ROOT
Source1: mimelib-3.1.1.tar.gz %files
BuildRequires: zlib-devel %defattr(-,root,root)
Requires: libxslt %doc AUTHORS ChangeLog COPYING FAQ NEWS
BuildRoot: %{_tmppath}/%{name}-%{version}-root README INSTALL
%description %{_mandir}/man1/*
Lurker is not just another mailing list archiver. Blá blá %{_bindir}/*
blá %{_libdir}/*
%prep %config(noreplace) %{_sysconfdir}/lurker.conf
%setup -a 1 %defattr(-,apache,apache)
%build %{_localstatedir}/www/lurker/
./configure --with-mimelib-local %{_localstatedir}/www/lurkdb/
--prefix=$RPM_BUILD_ROOT %changelog
--localstatedir=$RPM_BUILD_ROOT/make %{? * Fri Jan 2 2004 Jim Perrin <perrin@ohio.edu>
_smp_mflags} - Changed the spec file to better use macros

Tools (part II)
• CPAN Distroprefs (Configuration for individual distributions)
• A distropref file per badly behaved package
• Define which tests to run, excluding the ones that fail and
with which we're ok
• Pass additional arguments to one of the four commands
• Set environment variables
• Instantiate an Expect object that reads from the console,
waits for some regular expressions and enters some answers
• Temporarily override assorted CPAN.pm configuration
variables
• Specify dependencies the original maintainer forgot
• Disable the installation of an object altogether
• Examples:

Tools (part II)

DB_File.yml
---
match:
distribution: "/DB_File-d"
pl:
env:
DB_FILE_INCLUDE: "/usr/include"
DB_FILE_LIB: "/usr/lib64"

Convert-PEM-0.07.yml
---
comment: "skip test t/01-readwrite.t, i got fed up of it failing randomly
sometimes without any noticeable explanation."

match:
distribution: "/Convert-PEM-d"

test:
args:
- TEST_FILES="t/00-compile.t t/02-encode.t t/03-ede3.t"

Tools (part II)
Alien-ActiveMQ-0.000003.yml
---
comment: Module doesn't explicitly require the dependency below, so it fails
miserably when it tried to require it.
match:
distribution: "/Alien-ActiveMQ-0.00003"
depends:
requires:
File::Copy::Recursive: 0
patches:
- "MYCOMPANY/patches/Alien-ActiveMQ.patch"

Term-ReadLine-Gnu.yml
---
comment: "Skip the entire test suite - hudson doesn't run with a real /dev/tty"
match:
distribution: "/Term-ReadLine-Gnu-1"
test:
args:
- TEST_FILES=""
expect:
- "Skip the test suite?"
- "Yesn"

Getting Closer
• Merge projects Makefile.PL creating a single Makefile.PL
with everything
• Create our CPAN site with our packages, patches, and our
distroprefs files
• Create a rigged CPAN/Config.pm based on sane defaults
• 'urllist' => [q[file:///home/pecastro/build/CPANSITE]],
• 'make_arg' => q[-j 3]
• 'connect_to_internet_ok' => q[0]

Info for the next slides....
• %new_perl is an RPM variable that means
• %{new_perl_flags} $RPM_BUILD_ROOT/%{perl_bin_dir}/perl
• %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE
• %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

Getting Closer
• Main recipe so far:
• Extract perl sources
• %perl_base_dir = /opt/zt/zt-perl
• Configure -Dprefix=%perl_base_dir; make;
• make install DESTDIR=~/rpmbuild/opt/zt/zt-perl ( this will
install us new perl )
• export PERL5LIB,PERLLIB,PATH to match temporary
installation dir "~/rpmbuild/opt/zt/zt-perl/{bin,lib}"
• Copy our rigged CPAN/Config.pm into the right place
• %new_perl -MCPAN -e 'install Bundle::CPAN' ( so we have a
sane CPAN client that can do our bidding )
• %new_perl -MCPAN -e 'install Module::Install' ( so we can
use our auto magical unattended installation process
outsourcing dependency sorting to CPAN )

Getting Closer

• export PERL_MM_USE_DEFAULT=1 ( This prevents
automated processes from blocking on user input. )
• copy Makefile.PL into cwd
• %new_perl Makefile.PL --defaultdeps
• make

• Installs all the modules defined in Makefile.PL and their
dependencies as well auto-magically....

Getting Closer

This is excellent.
All we need now is to wrap up all of this in the SPEC
file and presto....
perl RPM with everything we need ! ROCK ON...

More problems
• When deployed, our custom perl, will live in /opt/zt/zt-
perl/bin
• What happens when we're building a perl binary & friends in
a system which already has a custom perl installed ?
• Kaboom !!! But Why Kaboom ?
• our ~/rpmbuild/opt/zt/zt-perl/bin/perl is being built to live
in /opt/zt/zt-perl/bin/perl and even though it's being run from
a temporary location, has it's INC paths pointing to it's final
location /opt/zt/zt-perl/lib ... hardcoded in the perl binary.
• When we say:
• %new_perl Makefile.PL --defaultdeps && make ;
• It replies: Oh, I can see that I already have everything
installed. Nothing else to do..... :(

Evil Hacks
• After having built perl and before starting auto magical
modules installation
• Temporarily hack Config.pm and Config_heavy.pl
%define archlib $RPM_BUILD_ROOT/%{perl_lib_dir}/%version/%
{perl_archname}

%new_perl -pi -e "
s|%perl_base_dir|$RPM_BUILD_ROOT%perl_base_dir|g"
$RPM_BUILD_ROOT/%archlib/Config.pm
$RPM_BUILD_ROOT/%archlib/Config_heavy.pl

And now...........The most EVIL HACK of all hacks......

Evil Hacks
%perl_base_dir = /opt/zt/zt-perl

garbled_base_dir=`
%new_perl -e '$_=”%perl_base_dir”;s|/|%|g;print'
`

%new_perl -pi -e "
s|%perl_base_dir|$mess_base_dir|g"
$RPM_BUILD_ROOT%{perl_bin_dir}/perl

REMEMBER.....
●%new_perl is an RPM variable that means

● %{new_perl_flags} $RPM_BUILD_ROOT%{perl_bin_dir}/perl

● %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE
● %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

Evil Hacks
Characteristics of this binary (from libperl):
Compile-time options: PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT
USE_LARGE_FILES USE_PERLIO
Built under linux
Compiled at Dec 1 2010 20:52:24
%ENV:
PERL5LIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/5.8.8:/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/site_perl/5.8.8"
PERLLIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/5.8.8"
@INC:
/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/5.8.8/x86_64-linux
/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8
perl/lib/site_perl/5.8.8/x86_64-linux
perl/lib/site_perl/5.8.8
%opt%zt%zt-perl/lib/5.8.8/x86_64-linux
%opt%zt%zt-perl/lib/5.8.8
%opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux
%opt%zt%zt-perl/lib/site_perl/5.8.8
%opt%zt%zt-perl/lib/site_perl
.
[pecastro@brutalix perl-zap-5.8.8]$ new_perl -VT
Can't locate Config.pm in @INC (@INC contains: %opt%zt%zt-perl/lib/5.8.8/x86_64-linux %opt
%zt%zt-perl/lib/5.8.8 %opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux %opt%zt%zt-
perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/site_perl).
BEGIN failed--compilation aborted

Wrapping it up
• RPM specifics
• BuildRequires: postgresql-devel expat-devel openssl-devel gd-
devel graphviz jdk , etc .... to specify all the third-party
dependencies
• %prep stage used to do custom validation and basic unpacking of
sources
• Example: mysql -utest -ptest -e 'use test' || echo
"echo "GRANT ALL PRIVILEGES ON test.* to test@localhost
identified by 'test'" |mysql -uroot" && exit 1
• %build stage to configure and build perl
• %install stage to install it and do all the magical Makefile.PL bits
• Make sure you undo your Evil (Config.pm, Config_heavy.pl, perl)
• Amend our rigged CPAN/Config.pm
• o conf build_dir /tmp/.cpan/build ;
• o conf make_install_make_command 'sudo make'
• o conf urllist http://mirrorservice.org/sites/ftp.cpan.org/pub/CPAN

Pros/Cons
• Pros
• Easily deployed package with dependency management
• Easy to spot missing packages ( perspective of dependent
package )
• Fast spotting of badly behaved modules
• Module install/test failure means no build
• Enabled us to replicate environments like if they were
mushrooms
• Cons
• Module updates not as straightforward as before
• Tricky CPAN module dependency
• Build time tied to CPU (84 min @ Virtual Double-Core Xeon
2.33GHz/4GB mem)

Final Thoughts

• I'd like to thank my colleagues for their input and feedback
• Recommended experiment
• Most likely to be possible to replicate the same approach in
other Linux/Unix flavors (Debian,Gentoo,...)
• The initial process is tedious but worth going through
• Once everything is streamlined there's no thinking just
building on top of...
• I've heard of other strategies to do deployment but
honestly.....
• This is currently the one that makes more sense to me

END {
say “QUESTIONS ?”;
say “Paulo Castro”;
say “<pauloedgarcastro@gmail.com>”;
close $SLIDE_FILE;
}

Packaging perl (LPW2010)

Related slideshows

More Related Content

Packaging perl (LPW2010)