Thursday, December 19, 2013

Cordova 3.3 adds Ubuntu

Upstream Cordova 3.3.0 is released just in time for the holidays with a gift we can all appreciate: built-in Ubuntu support!

Cordova: multi-platform HTML5 apps

Apache Cordova is a framework for HTML5 app development that simplifies building and distributing HTML5 apps across multiple platforms, like Android and iOS. With Cordova 3.3.0, Ubuntu is an official platform!

The cool idea Cordova starts with is a single www/ app source directory tree that is built to different platforms for distribution. Behind the scenes, the app is built as needed for each target platform. You can develop your HTML5 app once and build it for many mobile platforms, with a single command.

With Cordova 3.3.0, one simply adds the Ubuntu platform, builds the app, and runs the Ubuntu app. This is done for Ubuntu with the same Cordova commands as for other platforms. Yes, it is as simple as:

$ cordova create myapp REVERSEDOMAINNAME.myapp myapp
$ cd myapp
(Optionally modify www/*)
$ cordova build [ ubuntu ]
$ cordova run ubuntu


Cordova is a lot more than an HTML5 cross-platform web framework though.
It provides JavaScript APIs that enable HTML5 apps to use of platform specific back-end code to access a common set of devices and capabilities. For example, you can access device Events (battery status, physical button clicks, and etc.), Gelocation, and a lot more. This is the Cordova "plugin" feature.

You can add Cordova standard plugins to an app easily with commands like this:

$ cordova plugin add org.apache.cordova.battery-status
(Optionally modify www/* to listen to the batterystatus event )
$ cordova build [ ubuntu ]
$ cordova run ubuntu

Keep an eye out for news about how Ubuntu click package cross compilation capabilities will soon weave together with Cordova to enable deployment of plugins that are compiled to specified target architecture, like the armhf architecture used in Ubuntu touch images (for phones, tablets and etc.).


As a side note, I'm happy to note that my documentation of initial Ubuntu platform support has landed and has been published at Cordova 3.3.0 docs.

Friday, November 8, 2013

Ubuntu HTML5 API docs

HTML5 API docs published

I'm pleased to note that the Ubuntu HTML5 API docs I wrote are now done and published on These cover the complete set of JavaScript objects that are involved in the UbuntuUI framework for HTML5 apps (at this time). For each object, the docs show how the corresponding HTML is declared and, of course, all public methods are documented.

A couple notes:
  • I wrote an html5APIexerciser app that implements every available public method in the framework. This was helpful to ensure that what I wrote matched reality ;) It may be useful to folks exploring development of  Ubuntu HTML5 apps. The app can be run directly in a browser by opening its index.html, but it is also an Ubuntu SDK project, so it can be opened and run from the Ubuntu SDK, locally and on an attached device.
  • The html5APIexerciser app does not demonstrate the full set of Ubuntu CSS styles available. For example, the styles provide gorgeous toggle buttons and progress spinnners, but since they have no JavaScript objects and methods they are not included in the API docs. So be sure to explore the Gallery by installing the ubuntu-html5-theme-examples package and then checking out /usr/share/ubuntu-html5-theme/0.1/examples/
  • I decided to use yuidoc as the framework for adding source code comments as the basis for auto generated web docs.  After you install yuidoc using npm you can build the docs from source as follows:
  1. Get the ubuntu-html5-theme branch: bzr branch lp:ubuntu-html5-theme
  2. Move to the JavaScript directory: cd ubuntu-html5-theme/0.1/ambiance/js/
  3. Build the docs: yuidoc -c yuidoc.json . This creates the ./build directory.
  4. Launch the docs by opening build/index.html in your browser. They should look something like this 
Thanks to +Adnane Belmadiaf for some theme work and his always helpful consultation, to +Daniel Beck for his initial writeup of the Ubuntu HTML5 framework, and of course to the team for their always awesome work!

Friday, March 1, 2013

Apport-Valgrind 2.9 supports unpackaged executables

Apport-valgrind 2.9 picks up support for unpackaged [1] executables (thanks Martin Pitt for working with me to land my changes in trunk).

Here's what this means.
  • Previously if you ran apport-valgrind EXE, where EXE was not installed by a debian package, the process quit with a warning because it could only obtain debug symbol packages through debian dependencies
  • As of release 2.9, execution succeeds, and it obtains debug symbols packages based on  the shared object (.so) files that the EXE is linked to

Digging deeper into how it works

As explained here, apport-valgrind is a wrapper for valgrind, the venerable memory leak finder (among other things). 

Apport-valgrind first creates a sandbox directory that contains debug symbol files related to the executable you are checking for memory leaks. It then launches valgrind and points it also at the new sandbox directory. Valgrind creates memory leak stack traces, looking in the standard system directories for debug symbol files, but also in the generated sandbox directory. It uses the debug symbol files to create the most useful stack traces it can: stack traces that display source file names (instead of installed library file names) and function names (instead of "???", which is an unresolved symbol). All this is done without installing debug symbol packages onto your system: they are unpacked into the temporary sandbox directory, which is automatically deleted after use (unless an optional persistent sandbox is used). 

Release 2.9 uses a new technique for populating the sandbox directory that extends support for executables that are not installed by a package. (More on why that may be useful below.)

Flow for packaged EXE

Previously, the sandbox was populated only using debian dependencies of the debian package that installed the executable, like so:
  • Find the package that installed the executable
  • Find that package's full set of dependencies (all the way down, that is, recursively)
  • For each, get the best debug symbol package available (this depends on your system's apt configuration, for example whether you have added "", as described here)
  • Unpack the debug symbol packages into the sandbox
This code path still exists and is used for packaged executables.

Flow for unpackaged EXE

With 2.9, the sandbox is populated with the debug packages related to the executable's linked .so files, as reported by ldd, like so:
  • Use ldd to determine the executable's linked shared libraries (.so files)
  • Find the package that installed each linked shared library
  • For each, get the best debug symbol package available
  • Unpack the debug symbol packages into the sandbox

Use case

One use case I see is that one can now write a C program specifically to memory check a library with apport-valgrind simply by including and using the library. You do not need to rely only on existing debian packages to find executables that use the libraries in order to memory check them. And, you can target your C program to precisely target what you want to memory check, and no more.

For example: libappindicator.

As an exercise, I wanted to find memory leaks in application indicators (the icons on the top right like power, network, etc.), which means I needed to launch an app indicator at the command line with apport-valgrind (apport-valgrind EXE). So I tried to kill those processes, and they relaunched automatically. 

So I thought: write my own app indicator -- which turned out to be unnecessary: I found some sample C code that demonstrates how to write a simple application indicator here on (Scroll down to "Typical usage (C version)".)

So I grabbed it. It includes a libappindicator header file:

#include <libappindicator/app-indicator.h>

I compiled and linked this code, like so:

$ gcc myappindicator.c $(pkg-config --cflags --libs appindicator3-0.1) -o myappindicator.o 

That produces the unpackaged executable myappindicator.o

Running this at the command line (./myappindicator.o) creates a new icon in the appindicator area that has a menu and pops up a dialog with a text editing area. The indicator icon's menu has a Quit item, which works as expected, quitting the application and removing its indicator, so all is well.

Then, I ran it under apport-valgrind [2], like so:
$ apport-valgrind ./myappindicator.o 

And voila, the valgrind log is generated: ./valgrind.log

This sample C code could be simplified to more precisely target (memory check) libappindicator functions.

Quality of the unpackaged stack traces?

OK, so it works. But does it work well? That is, can one expect the same number of unresolved symbols for an unpackaged executable as for its identical packaged version, even though the code paths to create the sandbox are quite different?

Easy to find out by running apport-valgrind twice, once on a normal packaged executable found on the current PATH (for example apport-valgrind notify-send), and then again on an instance of notify-send simply copied to the current directory (which makes it unpackaged, since dpkg cannot find any package that owns it in this location, and which invokes the new ldd-based code path instead of the dependency based code path). Then counting the number of unresolved symbols with grep, like so:

$ apport-valgrind notify-send 
$ grep -c \?\?\? valgrind.log

$ cp $(which notify-send) .
$ apport-valgrind ./notify-send 
$ grep -c \?\?\? valgrind.log

Success! The same number of unresolved symbols in both cases means that the unpackaged stack traces are (here, at least) just as good as the packaged stack traces.


[1] You can tell if an executable is packaged with dpkg -S $(which EXE). For example, to find the package that installed ls:

$ dpkg -S $(which ls)
coreutils: /bin/ls

The package is coreutils.

[2] Always upgrade your system (apt-get update and apt-get dist-upgrade) before running apport-valgrind in order to increase the quality of the stack traces by ensuring the installed libraries are the most recent and therefore will match with the most recent debug symbol packages.

Monday, February 4, 2013

dbg versus dbgsym

Will the real debug package please stand up

While mucking around with stack traces, valgrind, and apport-valgrind, I stumbled across two types of debug symbol [1] packages:
  • dbg packages, for example: empathy-dbg
  • dbgsym packages, for example: empathy-dbgsym
Being rather new to this area, I was not sure which to use. But, I wanted to valgrind empathy, so I looked further.

The package Description fields [2] shed no light on which to use:

Description-en: GNOME multi-protocol chat and call client (debug symbols) Instant messaging program supporting text, voice, video, file transfers and inter-application communication over many different protocols, including: AIM, MSN, Google Talk (Jabber/XMPP), Facebook, Yahoo!, Salut, Gadu-Gadu, Groupwise, ICQ and QQ. This package contains the Empathy IM application and account manager.
Description: debug symbols for package empathy Instant messaging program supporting text, voice, video, file transfers and inter-application communication over many different protocols, including: AIM, MSN, Google Talk (Jabber/XMPP), Facebook, Yahoo!, Salut, Gadu-Gadu, Groupwise, ICQ and QQ. This package contains the Empathy IM application and account manager.
Turning to the internet, I found many others had asked the same question, and they usually got answers like this:

You can use either the -dbg or the -dbgsym packages, but you can't use both.
Still, no guidance as to which to use. And anyway, I wanted to understand what was going on at a deeper level. Investigating further, I found that dbg packages are hosted in the standard archive (, whereas dbgsym packages are hosted in a special archive:

It also turns out that the dbgsym packages on com do not appear to be strictly normal debian packages:
  • Normal debian packages end in ".deb" (for example: empathy-dbg_3.6.0.3-0ubuntu1_amd64.deb)
  • The packages on end in ".ddeb" (empathy-dbgsym_3.6.3-0ubuntu2_amd64.ddeb)
This ".ddeb" extension reminded me of ".udeb" packages, which I have encountered in the context of live-build installations. Are these dbgsym/ddeb packages also intended for some narrowly defined use case?

This warranted further poking around. Here are my findings.

dbg packages

dbg packages are the traditional debian method for providing debug symbols separately from the normally installed binary packages.

dbg packages are created by upstream packagers and flow naturally into the standard Ubuntu archives. They are normal debian packages (that end in ".deb").

There is usually only one binary dbg package per source package. That is, there are usually not separate dbg packages for individual binary packages, there's just one named after the source package, like empathy-dbg.

They are pretty easy to make these days. The normal way is described here.

So, thanks to a lot of upstream work, the Ubuntu standard archive already has lots of these dbg packages. You can see a list of them with something like this:

$ apt-cache search dbg | less

These dbg packages work as expected: if you install, for example, empathy-dbg, and then generate an empathy stack trace (with valgrind or gdb, for example):
  • The symbols it contains should be converted to useful function names (instead of being question marks)
  • The source file names should be displayed (instead of paths to shared object (.so) and other binary files)
So let's take a moment to appreciate the work upstream maintainers have done!

But, not every package in Ubuntu has a dbg package. And, even those that do may have created them in different ways, and possibly with different naming conventions, which makes them difficult (or impossible) to find mechanistically.

This is where dbgsym packages come in.

dbgsym packages

Consider the Ubuntu archive: thousands of binary packages (more than 7000 in the Ubuntu 12.10 Quantal main archive component alone).

That's a lot of different people in teams maintaining a lot of packages.

Inevitably, some packages that arguably should provide binary debug symbol packages do not. Or they do, but in a slightly different way, for example, the package name may not follow the standard pattern.

This wonderful diversity adds up to a concrete problem: the complete set of debug symbols required for debugging and creating useful stack traces of any arbitrary packaged C executable (ELF file) are not reliably present in standard Ubuntu archives.

Suppose I need to check something for memory leaks using apport-valgrind or debug it with gdb and I hit one of these missing bits?

Automation \o/

Fortunately, there is a systematic (and automatic) solution (which is almost fully working).

The idea is to automate the generation of debug symbols at package build time (whether or not the package maintainers took steps to create dbg packages). For each binary package that installs an executable for which symbols may be helpful, automatically create a new BINARYPACKAGE-dbgysm package, and let it install the appropriate set of debug symbol binaries. Make it conflict with the dbg package to avoid file collisions. Then, publish them in

That's two steps:
  1. Creating the dbgsym binary packages (this part seems to be fully implemented)
  2. Publishing them to (this part may not be fully implemented)

Binary package oriented

The dbgsym approach has a finer granularity than the usual dbg approach: it is binary package oriented. That is, a dbgysm package is created for each appropriate binary package (each that installs an ELF), whereas with the traditional, manual dbg approach, there is usually one dbg package created for all binary packages.

So there are often many dbgsym packages per source package with this approach, which results in many more dbgsym packages than dbg packages, a good thing: you only need to download and install the parts you want.

Conflicts - a closer look

These automatically generated binary dbgsym packages install some of the same files [3] as the corresponding dbg package, by design. But, without additional packaging steps, this would amount to a packaging error (since no single file may be installed by two binary packages without special steps to resolve this collision).

This is automatically handled by setting the dbgsym package to conflict with the dbg package(s). (Conflicting packages cannot both be fully installed at the same time.) This explains why I found so many statements saying that "dbg and dbgsym packages cannot both be installed at the same time." It is literally not possible due to these intentional conflicts settings.

How dbgsym packages are created

dbgsym packages are automatically created during a normal source package binary build by the pkg-create-dbgsym package, if it is installed. That is, if you install pkg-create-dbgsym, whenever a debian source package is built (for example with debuild in the source tree), pkg-create-dbgsym scripts create the dbgysm binary packages and set them to conflict as appropriate. (Thanks to Martin Pitt and other debian developers.) These steps are in addition to the normal activities of a build, so you also generate non-debug binary packages.

For example, after installing pkg-create-dbgsym, I built the empathy source package, and here are the binary packages that were created [4]. Lots of dbgsym ddebs!

ddeb: Huh? What?

These dbgsym binary package files end in ".ddeb", not ".deb", so what's up with that?

As far as I can tell, this is merely a convenience for archive maintenance. That is: a .ddeb is exactly the same as a .deb, except that its extension is different. This serves the purpose of allowing archive management tools to find ddeb binary packages and treat them differently, in this case:
  • Posting them on 
  • Not posting them on

Enough already: which to use?

So, after this exploration, I vote for dbgsym packages over dbg packages for the following reasons
  • The trend is towards automatic generation of dbgsym packages and automatic posting of them to separate ddebs archive like
  • Such ddeb archives should contain more debug symbol packages than the traditional dbg packages because they are automatically created for all appropriate packages (instead of relying on maintainers to add a dbg package) so you get more
  • Such ddeb archives certainly should contain equivalents for the existing dbg packages as well, so you lose nothing
  • If you want, you can just get the debug symbols you need, since dbgsym packages are more granular 
This all means that if you use dbgsym packages, your stack traces and debugging experiences are more likely to provide you the data you need to get your work done, with less bloat.

Also, manually created dbg packages may disappear over time. Why? No need for package maintainers to think about debug packages. And the normal archives get smaller as debug symbol packages disappear.

Using dbgsym

As noted, the dbgsym packages are published on Ubuntu systems do not know this archive by default.

You can easily add it to your "software sources". For example, the following line adds for Quantal, main to your /etc/apt/sources.list file:

deb quantal main

Then run sudo apt-get update, and you should be all set to go.

For example, check whether empathy-dbgsym is available for installation:

$ apt-cache policy empathy-dbgsym
  Version table:
 *** 0
        500 quantal/main amd64 Packages
        500 quantal/main amd64 Packages
        100 /var/lib/dpkg/status

There it is! (I happen to have this one installed.)

How about the dbgsym version of empathy's binary package account-plugin-aim?

$ apt-cache policy account-plugin-aim-dbgsym
  Installed: (none)
  Version table: 0
        500 quantal/main amd64 Packages

There it is!

A caveat

It appears that not all dbgsym binary packages that one might expect are actually present on

For example, when I built empathy on a Quantal amd64 system with pgk-create-dbgsym installed, the expected dbgsym binary packages were created, including, for example, one for the account-plugin-irc binary package, and here it is:


But, I don't see a dbgsym package for account-plugin-irc published in the Ubuntu ddebs quantal main archive. The same is true for some (but by no means all) other binary packages deriving from empathy, like account-plugin-icq-dbgsym

This would seem to be a hiccup in the publishing part of the automation, not the dbgsym generation part.


[1] Debug symbols fill in stack traces to make them more human readable by substituting C function names and C source code file name for question marks and library file names. See apport-valgrind for details.

[2] You can display a package Description for any package your system's apt knows about with apt-cache show PACKAGE, and look for the Description field.

[3] Conflicting with the dbg package(s), an example using account-plugin-aim as an example:

The empathy source package has the accounts-plugin-aim binary package. It installs the following file: /usr/lib/libaccount-plugin-1.0/providers/ (as shown by dpkg -S  /usr/lib/libaccount-plugin-1.0/providers/

The empathy-dbg package installs a debug symbol version of this file in a debug directory: /usr/lib/debug/usr/lib/libaccount-plugin-1.0/providers/ There is no conflict yet, since the paths are different.

The automatically generated dbgsym packages for empathy include account-plugin-aim-dbgsym. This package installs the same debug .so file as we saw for the empathy-dbg package:  /usr/lib/debug/usr/lib/libaccount-plugin-1.0/providers/

That's a conflict. Two packages (empathy-dbg and account-plugin-aim-dbgsym) install the same file.

This is expected and handled by making the dbgysm binary packages conflict with the dbg package (if any), which it does:

$ apt-cache show account-plugin-aim-dbgsym | grep Conflicts
Conflicts: empathy-dbg 

[4] With pkg-create-dbgsym installed, building empathy creates all these debs and dbgsym/ddebs:


Saturday, January 26, 2013

Apport-Valgrind: a case of apples and oranges?

Who doesn't love a good fruit salad?

As noted a previous post, Ubuntu 13.04 (Raring) has the new apport-valgrind binary package [1]. 

But apport is a crash reporting system, whereas valgrind is memory leak detector (among other things). So it may make you wonder: what's the connection? Is this a case of apples-and-oranges?

Let's try to answer this, starting with a look at valgrind.


Valgrind is a workhorse of looking at C programs at run time. It has many capabilities. The one we are interested in here is its 'memcheck' tool, which is used to find memory leaks [2]. 

For example, you can run valgrind --tool=memcheck /usr/bin/foo [3] and a report is generated:
  • The report shows memory leaks created while running /usr/bin/foo. This includes leaks in foo's code and leaks in any code executed by foo, for example called functions that exist in external shared libraries. 
  • For each leak, the log contains a stack trace that shows the sequence of function calls that created the memory leak. You can follow the stack trace by hand and find the code errors that lead to the memory leak, except for one issue: raw stack traces are like swiss cheese with lots of missing bits.

Raw stack traces are like swiss cheese

The problem with a raw stack trace is that it is full of holes.

What's missing? 
  • In a  raw valgrind stack trace, one does not see function names or source code file names
  • Instead one sees question marks for function names and executable file names (frequently shared object library files) instead of source file names
Fortunately, valgrind can display the function names and source code file names if the debug symbols are present (more on this below). 

Let's take a look at some real-life valgrind output and compare how it looks before and after debug symbols are present.

Here's a line from a raw valgrind memory leak stack trace:

==2874==    by 0x51E727D: ??? (in /usr/lib/x86_64-linux-gnu/

And here's how the same line looks with debug symbols for libgtk-3.0:

==3643==    by 0x51E727D: gtk_menu_item_class_intern_init (gtkmenuitem.c:427)

In the raw example: 
  • The function name is unknown: ???
  • The shared object file name in which the function lives is listed: /usr/lib/x86_64-linux-gnu/
In the debug symbols example:
  • The function name is shown:  gtk_menu_item_class_intern_init
  • The source code file name is shown: gtk_menu_item_class_intern_init:427 (It even knows the line number: 427)
Clearly, the second stack trace is much more useful for a human being who wants find and fix a memory leak. The question therefore is: how can we easily obtain the debug symbols that will make our valgrind stack traces useful? That's where apport comes in.

The apport connection

Apport is a collection of packages and tools that provide automatic crash reporting. Among them is apport-retrace. This little gem provides the debug symbol capabilities for valgrind through its apport-retrace script.

It has the ability to find, download and extract available debug packages for all dependencies of a packaged [4] executable. (This was used for purposes unrelated to finding memory leaks with valgrind, but  as of Raring, it is used for valgrind purposes too.)

Let's unpack this a bit. Suppose the executable is nm-applet. Apport-retrace could:
  • Find the tree of packages that nm-applet needs to run (that is, the package that owns nm-applet and that package's direct and indirect dependencies)
  • Find the available debug symbol packages for them and make their debug symbol files available in a temporary directory (in /tmp, unless you specify otherwise) called the 'sandbox'
  • The debug symbols packages are not installed, but extracted into the sandbox directory
  • The sandbox directory is deleted after use (again, unless you specified your own sandbox directory) 
This automatic discovery of the complete set of available debug symbol files related to an executable would be very useful for the valgrind use case, as explained above. And as of Raring, we have it, as explained below. (Thanks to Martin Pitt for his invaluable assistance with this work.)

But first, a nice thing to note: extracting (rather than installing) the debug packages has a smaller impact on the the system. Notably: the extract directory (the sandbox) can simply be deleted after use to regain the disk space. No debug packages are actually installed, so no package removal needs to be done either. If you intend to valgrind the same executable many times, you can reuse a persistent sandbox, or simply install the debug symbol packages.

Pointing valgrind at the sandbox

We have seen that if debug symbol files are available, valgrind uses them to make much more useful stack traces. And we have seen that apport can download and extract into a temporary sandbox all available debug symbols for the packaged executable for which we want to find memory leaks. 

This leaves one more piece of the puzzle: telling valgrind to also look in the sandbox directory for debug symbols. (Valgrind always looks in the normal system directories for debug symbols.)

In Ubuntu 13.04 (Raring), the valgrind package now has this ability thanks to a patch from Alex Chiang. You simply add the information to the valgrind command line using the --extra-debuginfo-path=DIR argument.

With these two functions in place, the table is set for apport-valgrind, as described next.

Apport and valgrind: two peas in a pod

In Ubuntu 13.04 (Raring), the apport-valgrind package is introduced. The apport code that creates the sandbox was moved from apport-retrace into a new apport python module: apport/, which makes it available for new code, such as apport-valgrind.

So here's what apport-valgrind does:
  • It uses apport/sandboxutils to obtain and extract all available debug symbol packages into the sandbox directory for your (packaged) executable
  • It calls valgrind and tells it to also look in the sandbox directory (with the new --extra-debuginfo-path argument)
  • When valgrind is done, the sandbox directory is deleted (there are options for persistent sandboxes -- useful when you want to use valgrind repeatedly)
  • The valgrind log file is created in the current directory: ./vagrind.log
So with a single command like this: apport-valgrind nm-applet, you can generate a valgrind memory check log of stack traces with all available debug symbols used, with no new debug packages installed on the system and all debug  files (the sandbox) automatically deleted after execution.

So, apport and valgrind, a case of two peas in a pod :)

All of these strained metaphors about food make me hungry: time for lunch.


[1] Install it with sudo apt-get install apport-valgrind. You will also want to install valgrind and valgrind-dbgsym: sudo apt-get install valgrind valgrind-dbgsym

[2] A memory leak occurs in C when you allocate memory on the heap and fail to deallocate ('free') the memory. Such memory is not available for use by any process until the the code's main process terminates. If the process does not terminate (perhaps it is a daemon, or a piece of code that is persistent, such as a part of the desktop GUI) the memory is essentially lost forever.

[3] One picks the valgrind tool with --tool=TOOLNAME. The memcheck tool is the default, so there is no actual need to specify it.

[4] If the executable is not in a debian package, apport does not know how to find and get all appropriate debug packages. There are other approaches to automate this. See future posts here.

Friday, January 25, 2013

Apport-valgrind in Ubuntu Raring

Apport-valgrind lands in Raring

On Jan 8, 2013, my apport-valgrind landed in apport 2.8 for Ubuntu Raring (2.8-0ubuntu1).

This allows you to run valgrind for an executable (to find memory leaks) while first populating a temporary directory with available debug symbols for the executable, which makes the resulting valgrind log file more complete and useful (because more symbols names are filled in, making the stack traces clearer).

Thanks to Alex Chiang for the initial work on the script and for adding support for the additional debug symbol directory to valgrind, and to Martin Pitt for his assistance in landing this in apport upstream.