Skip navigation

Monthly Archives: May 2009

Dealing with perl binary compatibility issues as a user of XS modules.

chromatic discusses binary compatibility with the perl executable. I’m someone who only uses XS modules and hopes to never have to write one. Yet, unlike what seems to be the prevailing fear, I don’t worry about binary compatibility. I don’t worry about upgrades breaking things. I don’t worry that I’m going to have mismatches in executables/libraries.

Why? Continuous integration.

Don’t just grab modules from the CPAN and install them somewhere never to be upgraded until you’re forced to at a huge amount of time and expense. Regularly rebuild everything you use, preferably daily or hourly.

  1. Install a perl binary and then lock it down. Don’t use site dirs. Have the perl core be inviolate.
  2. Instead, install to some other directory and point PERL5LIB at it. Both CPAN and CPANPLUS allow you to configure what you want to be passed to the configuration program. Pass a value for INSTALL_BASE to Makefile.PL and install_base to Build.PL that points to your directory.
  3. Set up a build server to use CPAN or CPANPLUS to “install” and build everything. Since your build machine will be rebuilding everything regularly, when you upgrade your perl binary or an XS module, any problems will be quickly identified. Everything being rebuilt in one go means that all binary code will match up and you don’t have to worry about them getting out of sync. Because installing a distribution causes all of that distribution’s tests to be run, you’ll know that that distribution works in your environment.

(If you’re using taint mode and thus cannot use PERL5LIB, have your build machine build a custom perl executable along with everything else.)

Details for doing this belong in another post. But the point is, if you regularly rebuild all of the CPAN code you use, you don’t need to worry about binary issues.

DarkPAN setup

Gabor is looking at setting up corporate CPANs. While it doesn’t cover the web interfaces that he wants, brian d foy has been working on MyCPAN::Indexer and MyCPAN::App::DPAN.

The former has been in progress for a while as part of brian’s BackPAN indexing work. The latter is a more recent effort at creating a tool for managing CPAN-like repositories. It can copy archives into a repository and update the standard index files. It’s being used on a daily basis on both Windows and Linux to maintain a single DarkPAN at present, and, once some more work on automated building of Perl distributions under TeamCity is done, will be used for maintaining a lot more private repositories.

The way that a repository is updated is to have a copy of it checked out of Subversion, put the distribution archive in place, run dpan on it to update the indexes, and check in the result. CPAN.pm can be pointed at the Subversion HTTP URL and use it just as if it were one of the public CPAN mirrors. (Though of course, the DarkPAN repository has a restricted set of distributions in it, not even a mini-CPAN.)

The problem with Modern::Perl.

Modern::Perl has laudable goals. I agree with chromatic. This is why I use 5.10.0 at work. This is why I will be upgrading to 5.10.1 almost as soon as it is released. I even plan on testing our code with the 5.10.1 release candidates. The first significant line in all our modules is “use 5.010;“. (The second is “use utf8;“.) As I said in a prior post, I upgrade all the modules that we use from the CPAN every three months.

However, as a practical matter, Modern::Perl is broken. What happens when perl 5.12 comes out? Does chromatic upgrade the module to add “use feature qw/ :5.12 /;“? Are all of the modules that use Modern::Perl suddenly broken? Someone requested that TestingAndDebugging::RequireUseStrict include Modern::Perl in the default set of modules that are treated as equivalent to “use strict;“. (You can add to that list yourself via the “equivalent_modules” option.) I don’t want to do that; I want to discourage the use of Modern::Perl. Maybe if there was a Modern::Perl_5_10 I’d think differently.

pmtools

pmtools is a set of little utilities for dealing with your installed modules. (Unfortunately, if you want to install these using CPAN.pm or CPANPLUS, you need to tell them to install “Devel::Loaded”.)

The ones that I use on a daily basis are pmpath, pmvers, and pminst.

I use pminst and pmvers to maintain a couple of DarkPANs, a.k.a. internal CPANs. I want to manage the versions of modules we use so that we don’t get surprises, like they won’t compile in our environment or they make changes that we can’t deal with right away. So, we have repositories in the standard CPAN layout and point CPAN(PLUS)? to those instead of the public CPAN.

Since the install tools are pointed at an internal repository, when I want to use a module that we haven’t used before, I need to download the distribution myself. It’s not as simple as just grabbing the distribution archive. I need to get the module’s dependencencies. In order to find out which of those I’ve already gets installed, I use pminst.

If I run into a problem with a module, I need to figure out which version I’ve got in order to look at whether there’s an updated version or to properly report a bug. Thus, pmvers.

I regularly need to figure out what path I’m getting a module from, for example whether it’s in the core perl install or in my installed modules path. (In order to avoid cross-project module version entanglements, we don’t use sitelibs; we use separate install directories. The core perl installation is readonly and nothing ever gets added to it.) pmpath probably gets the most usage of the three.

The most common of my use of pmpath isn’t actually direct. When I get an exception in some module, I want to bring it up in the editor to see the code. I’ve created a script that I call pmedit:

 1 #!/usr/bin/env bash
 2
 3 if [[ $# -le 0 ]]
 4 then
 5     echo "usage: $0 module […]" >&2
 6     exit 1
 7 fi
 8
 9 declare -a FILES
10
11 for MODULE in "$@"
12 do
13     FILE=`pmpath "${MODULE}" 2>/dev/null`
14     if [[ -z "${FILE}" ]]
15     then
16         echo "Could not find ${MODULE}." >&2
17     else
18         FILES[${#FILES[@]}]="${FILE}"
19     fi
20 done
21
22 if [[ ${#FILES[@]} -gt 0 ]]
23 then
24     gvim "${FILES[@]}"
25 fi