tag:blogger.com,1999:blog-129889814985929882024-03-13T14:15:39.917-07:00Kyle Nitzsche UbuntuKyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-12988981498592988.post-52273326966142521432019-08-29T13:38:00.001-07:002019-08-29T13:38:41.296-07:00Snapping ROS2 - that was quick!<br />
<div>
Hearing that snapcraft now supports the ROS2 colcon build system, I figured I'd give it go. I'm a good victim, since I'm pretty new to ROS. Without much fuss or bother, I snapped two C++ ROS2 package, a publisher and subscriber, built with the snapcraft colcon plugin (beta in snapcraft 3.2).<br />
<br />
Here's the amd64 <a href="https://snapcraft.io/kyle-ros2-colcon-pub-sub" target="_blank">snap</a> in the store.</div>
<div>
<br />
It has two commands, one to publish, one to subscribe:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kyle-ros2-colcon-pub-sub.publish</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ kyle-ros2-colcon-pub-sub.subscribe</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
Here's the github <a href="https://github.com/knitzsche/ros2-2" target="_blank">repo</a> with a snapcraft.yaml file and the two ROS2 C++ packages.<br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
See sample output below.<br />
<br />
<div>
<h3>
Goals</h3>
</div>
<div>
<ul>
<li>Build a ROS2 snap using the snapcraft colcon plugin.</li>
<li>Include two ROS2 C++ packages (a publisher and a subscriber).</li>
<li>Provide snap commands for each package node.</li>
<li>The snap commands should use ROS2's node launch system.</li>
<li>The snap should run with strict confinement.</li>
<li>The packages's source should follow the normal ROS2 workspace structure: WORKSPACE/src/PACKAGES to allow me to develop the packages first (without thinking about snapping them) and then easily snap them (without changing the tree structure).</li>
<li>Each ROS2 package should be a separate <a href="https://snapcraft.io/docs/snapcraft-parts-metadata" target="_blank">snapcraft part</a>, so I can independently build them.</li>
</ul>
</div>
<br />
<h3>
Getting started</h3>
<div>
<br /></div>
<div>
<h4>
Learning</h4>
</div>
</div>
<div>
<br />
<ul>
<li>To get started, I read Kyle Fazzari's ROS2 Colcon Snap <a href="https://snapcraft.io/blog/building-ros2-snaps-with-colcon" target="_blank">post</a>. </li>
<li>I then reviewed the state of ROS2 <a href="https://index.ros.org/doc/ros2/" target="_blank">here</a>.</li>
<li>I found that the <a href="https://index.ros.org/doc/ros2/Installation/" target="_blank">Crystal Clemmy</a> ROS2 release is current and that it is supported on Ubuntu 18.04 Bionic <a href="https://www.ros.org/reps/rep-2000.html#crystal-clemmys-december-2018-december-2019" target="_blank">here</a>. </li>
<li>I reviewed colcon <a href="https://index.ros.org/doc/ros2/Tutorials/Colcon-Tutorial/" target="_blank">here</a>.</li>
<li>I learned about the new ROS2 launch system <a href="https://index.ros.org/doc/ros2/Tutorials/Launch-system/" target="_blank">here</a>.</li>
</ul>
</div>
<div>
<br />
<h4>
Bionic LXD container for development</h4>
<br /></div>
I created a Bionic LXD container named <span style="font-family: Courier New, Courier, monospace;">ros2</span> to develop in (<span style="font-family: Courier New, Courier, monospace;">lxc launch ubuntu:18.04 ros2</span>). Then, <span style="font-family: Courier New, Courier, monospace;">lxc start ros2</span> and<span style="font-family: Courier New, Courier, monospace;"> lxc shell ros2</span> (which logs in as root user). When logged as root, I created another user in the container with my name and added the ssh key I use for github. I log in to that use with <span style="font-family: Courier New, Courier, monospace;">lxc exec ros2 -- su myusername</span>.<br />
<br />
I decided to build in the container directly instead of using the default (for core18 snaps) of multipass, which builds in a dedicated VM. The LXD container is dedicated to building just this ROS2 snap, so I don't mind if debian packages needed to build the snap get installed. So, I always used <span style="font-family: Courier New, Courier, monospace;">--destructive-mode</span> flag on snapcraft to not use multipass.<br />
<br />
<h3>
Create and modify the packages</h3>
<div>
<br /></div>
<div>
I got working C++ example ROS2 source package code <a href="https://github.com/ros2/examples" target="_blank">here</a>, crystal branch. </div>
<div>
<br /></div>
<div>
<ul>
<li><span style="font-family: Courier New, Courier, monospace;">examples/rclcpp/minimal_publisher/</span></li>
<li><span style="font-family: Courier New, Courier, monospace;">examples/rclcpp/minimal_subscriber/</span></li>
</ul>
</div>
<div>
<br /></div>
<div>
I modified the <span style="font-family: Courier New, Courier, monospace;">package.xml</span> files and few other minor items to use my own package and node names. </div>
<div>
<br /></div>
<div>
(I also added a bit of code to the publisher to publish messages consisting of a random adjective and a random noun, just for fun.)</div>
<div>
<br /></div>
<h3>
Source tree directories</h3>
<div>
<br /></div>
<div>
The following directory layout is structured as a ROS2 workspace, so I used this for my snap source tree structure:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">workspace/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">└── src</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> ├── kyle_publisher</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> │ └── launch</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> └── kyle_subscriber</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> └── launch</span></div>
</div>
<div>
<br /></div>
<div>
There is a <span style="font-family: "Courier New", Courier, monospace;">src </span>directory that contains two packages: <span style="font-family: "Courier New", Courier, monospace;">kyle_publisher</span> and <span style="font-family: "Courier New", Courier, monospace;">kyle_subscriber</span>.</div>
<div>
<br /></div>
<div>
Each package contains a <span style="font-family: "Courier New", Courier, monospace;">launch </span>directory that contains a ROS2-style python launch file:</div>
<div>
<ul>
<li>kyle_publisher launch <a href="https://github.com/knitzsche/ros2-2/blob/master/src/kyle_publisher/launch/launch_publisher.py" target="_blank">file</a></li>
<li>kyle_subscriber launch <a href="https://github.com/knitzsche/ros2-2/blob/master/src/kyle_subscriber/launch/launch_subscriber.py" target="_blank">file</a></li>
</ul>
</div>
<div>
I'll discuss snapping these launch assets below.</div>
<div>
<br /></div>
<h3>
Colcon-source-space keyword</h3>
<div>
<br /></div>
<div>
After a little experimentation, I realized I could create the two snapcraft parts (one for the publisher, one for the subscriber) with references to he same <span style="font-family: Courier New, Courier, monospace;">source: src</span><span style="font-family: "Courier New", Courier, monospace;"> </span>directory and use the <span style="font-family: "Courier New", Courier, monospace;">colcon-source-space </span>keyword to point the part at different package directories:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">parts:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> publisher:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> plugin: colcon</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> source: src/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> colcon-source-space: kyle_publisher</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">[. . .]</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> subscriber:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> plugin: colcon</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> source: src/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> colcon-source-space: kyle_subscriber</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">[. . .]</span></div>
<div>
<br /></div>
<div>
This allows each package to be a separate part that I could manage (pull, build, stage, prime) independently to save time during development. </div>
<div>
<br /></div>
<h4>
Issue: duplicate files in both parts</h4>
<div>
<br /></div>
<div>
My parts approach causes an issue of some duplicated files. That, is there are a set of files that have the same path and the same name that are installed by both parts, and that is not allowed (or even possible).</div>
<div>
<br /></div>
<div>
The snapcraft tool is kind enough to even list the duplicated files when trying to build the snap!</div>
<div>
<br /></div>
<h4>
Solution: suppress the files from one part</h4>
<div>
<br /></div>
<div>
The solution is to suppress these files from one of the parts. Folks who have snapped a few things know that the snap build system has stages. First each part is pulled (into <span style="font-family: Courier New, Courier, monospace;">parts/PART/src/</span>). Then it is built (into <span style="font-family: "Courier New", Courier, monospace;">parts/PART/build/</span>). And then the install directory is populated <span style="font-family: "Courier New", Courier, monospace;">parts/PART/install/</span>. Next, the install directories for all parts are merged into the single <span style="font-family: "Courier New", Courier, monospace;">stage/ </span>directory. And here is where any duplicated install files clash. </div>
<div>
<br /></div>
<div>
So, the solution is to suppress one of the set of duplicate files from being put into the <span style="font-family: "Courier New", Courier, monospace;">stage/</span>directory, which is easily done like so, in the <span style="font-family: Courier New, Courier, monospace;">subscriber</span> part. </div>
<div>
<br /></div>
<div>
<b>Note</b>: This is a yaml array, so the first hyphen indicates an array element. The second hyphen means suppress the following file, so all of these are suppressed from being placed into the stage/ directory </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">subscriber:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">[ . . . ]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> stage:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -usr/lib/python3/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -usr/lib/python3.6/</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/local_setup.bash</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/local_setup.ps1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/local_setup.sh</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/local_setup.zsh</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/setup.bash</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/setup.ps1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/setup.sh</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> - -opt/ros/snap/setup.zsh</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
With this in place, the two parts are separately buildable (if you use <span style="font-family: Courier New, Courier, monospace;">snapcraft build subscriber</span>, for example), and they combine without a problem into a final snap, with no missing and no duplicate files.</div>
<div>
<br /></div>
<h3>
Snap commands and launch files</h3>
<div>
I added two commands to the snap to launch the publisher and the subscriber from the ROS2 style python launch files in each package's launch directory.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">apps:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> publish:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> command: opt/ros/crystal/bin/ros2 launch kyle_publisher launch_publisher.py</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> plugs: [network, network-bind]</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> subscribe:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> command: opt/ros/crystal/bin/ros2 launch kyle_subscriber launch_subscriber.py</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> plugs: [network, network-bind]</span></div>
</div>
<div>
<br /></div>
<h4>
Issue: launch files are not installed in snap</h4>
<div>
<br /></div>
<div>
I found that the packages' launch directories (and the ROS2 python launch files they contain) were not automatically carried through to the final snap. Specifically, I found that even though each package had a <span style="font-family: Courier New, Courier, monospace;">launch/</span> subdirectory, it was <b>not populated</b> into the part's install directory here, for example for the publisher part:</div>
<div>
<br /></div>
<div>
<span style="font-family: "Courier New", Courier, monospace;">parts/</span><span style="font-family: Courier New, Courier, monospace;">publisher/install/opt/ros/snap/share/kyle_publisher/</span></div>
<div>
<br /></div>
<h4>
Solution: override each part's build stage </h4>
<div>
The solution is to override the snapcraft build stage for each of the parts. First in the override, the normal snapcraft build is executed (with <span style="font-family: Courier New, Courier, monospace;">snapcraftctl build</span>) , and then the <span style="font-family: "Courier New", Courier, monospace;">launch/</span> directory is manually copied from the part's <span style="font-family: Courier New, Courier, monospace;">src/</span> directory to its <span style="font-family: Courier New, Courier, monospace;">install/</span> directory, as follows:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">publisher:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">[. . .]</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> override-build: |</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> snapcraftctl build</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> cp -r ../src/kyle_publisher/launch $SNAPCRAFT_PART_INSTALL/opt/ros/snap/share/kyle_publisher/</span></div>
</div>
<div>
<br /></div>
<div>
This uses the <span style="font-family: Courier New, Courier, monospace;">override-build</span> keyword to take control of the snapcraft build for the part. After completing the snapcraft build (with <span style="font-family: "Courier New", Courier, monospace;">snapcraftctl build</span>), the directory is copied. The only tricky part is knowing that the <span style="font-family: Courier New, Courier, monospace;">cp</span> command executes in the part's <span style="font-family: Courier New, Courier, monospace;">build</span> directory. Since I need to copy from the part's <span style="font-family: Courier New, Courier, monospace;">src</span> directory, I need to know the relative path to it: <span style="font-family: Courier New, Courier, monospace;">../src</span> . Also note the use of the <span style="font-family: Courier New, Courier, monospace;">SNAPCRAFT_PART_INSTALL</span> variable to name the part's install directory without needing to know its location. </div>
<div>
<br /></div>
<h3>
That's it</h3>
Using the snapcraft colcon plugin couldn't have been much easier: it pretty much worked as I intuitively expected it to. As noted, launch asset handling required manual steps to copy them into the snap. But this was a simple fix that may not be unfamiliar to anyone who uses snapcraft. I also needed to suppress some redundant files, but this resulted from my decision to treat a single ROS2 workspace with two packages as separate snapcraft parts, which may have been naive on my part.<br />
<br />
<h3>
Try out the snap</h3>
<div>
<br />
Here's the amd64 <a href="https://snapcraft.io/kyle-ros2-colcon-pub-sub" target="_blank">snap</a> in the store.</div>
<div>
<br /></div>
<div>
Install with:</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">$ snap install kyle-ros2-colcon-pub-sub</span></div>
<div>
<br /></div>
<div>
Run the publisher like so: </div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">$ kyle-ros2-colcon-pub-sub.publish </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[INFO] [launch]: process[publisher_member_function-1]: started with pid [2508]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[INFO] [kyle_publisher]: Publishing: 'tiny television'</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[INFO] [kyle_publisher]: Publishing: 'sorry sympathy'</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[...]</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
Run the subscriber like so: </div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">$ kyle-ros2-colcon-pub-sub.subscribe </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[INFO] [launch]: process[subscriber_member_function-1]: started with pid [2218]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[INFO] [minimal_subscriber]: 'tiny television'</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[INFO] [minimal_subscriber]: 'sorry sympathy'</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[...]</span></div>
</div>
<div>
<br /></div>
KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com4tag:blogger.com,1999:blog-12988981498592988.post-70883551795315988472016-07-18T14:21:00.001-07:002016-07-20T10:09:17.501-07:00Running X Apps on Ubuntu Devices<html lang="en">
<title>Running X Apps on Ubuntu Devices</title>
<head>
</head>
<body>
You can install, launch, and use traditional debian-packaged X apps on Ubuntu devices. This may be unexpected given that Ubuntu devices do not seem to support user-installed debian packages, nor do they run the X Display Server. But it does work, courtesy of Mir/XMir and Libertine.<br/><br/>
So here’s a bit of background to get started.<br/><br/>
But first, please note that at this time, display and use of X apps on an external monitor is only available on the Pro5/M10 and on future devices. (BQ 4.5/E5 and Meizu MX4 do not support this feature.)<br /><br/>
<h3>Hello Mir (Goodbye X)</h3>
Traditionally, and still on the Ubuntu Classic desktop with Unity 7, Ubuntu runs an X Display Server. Apps are debian packaged. And, they are written for X:<br /><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-jLZrhWx6Yhs/V40w7zaiLoI/AAAAAAAADjM/ilX2ybM1CkcvRGJA1VZDmEXHxdnUljhBACLcB/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="156" src="https://2.bp.blogspot.com/-jLZrhWx6Yhs/V40w7zaiLoI/AAAAAAAADjM/ilX2ybM1CkcvRGJA1VZDmEXHxdnUljhBACLcB/s320/1.png" width="320" /></a></div>
Due in part to X’s inherent security shortcomings, the Mir display server is now used on Ubuntu Devices under Unity 8 (although not yet by default on the desktop). XMir bridges traditional X apps to Mir. That is, apps written for X can run fine in a Mir/XMir environment:<br /><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-9LIYU2iTkMw/V40xUl-GCDI/AAAAAAAADjQ/qkMUt-Jl9CQ52RmVtMqexzNfnVPmjWAGQCLcB/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="158" src="https://3.bp.blogspot.com/-9LIYU2iTkMw/V40xUl-GCDI/AAAAAAAADjQ/qkMUt-Jl9CQ52RmVtMqexzNfnVPmjWAGQCLcB/s320/2.png" width="320" /></a></div>
<h4>
Packages and the root file system</h4>
Ubuntu Classic has a root file system (rootfs) that is populated through installation of a carefully curated set of debian packages. At run time, users can install debian packages to add apps or modify their system.<br /><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-t7jCydUr58c/V40xZD1gk-I/AAAAAAAADjU/BF8cRMdL0fsFYIStT7iaeBjivey8F1lnACLcB/s1600/3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="139" src="https://3.bp.blogspot.com/-t7jCydUr58c/V40xZD1gk-I/AAAAAAAADjU/BF8cRMdL0fsFYIStT7iaeBjivey8F1lnACLcB/s320/3.png" width="320" /></a></div>
This approach raises security concerns because debian packages execute installation scripts with root level privileges and because debian packages can alter what the rootfs provides by modifying or replacing core system components.<br /><br/>
Ubuntu devices are designed for security and reliability. Ubuntu devices have a read-only rootfs that is small and tight, providing just what is needed and simplifying system updates. The rootfs is not modifiable by the user. Indeed it is mounted as a read-only partition. Users install apps through click packages that do not modify the rootfs.<br /><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://3.bp.blogspot.com/-Wd0Pess1lYs/V40xdecJLaI/AAAAAAAADjY/Uh6GUepeGrANbPPlNym-AdCCpu6FgelogCLcB/s1600/4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="233" src="https://3.bp.blogspot.com/-Wd0Pess1lYs/V40xdecJLaI/AAAAAAAADjY/Uh6GUepeGrANbPPlNym-AdCCpu6FgelogCLcB/s320/4.png" width="320" /></a></div>
Given all of this: how do users install debian packaged apps that use X on Ubuntu Devices? The answer is LIbertine with XMir.<br /><br/>
<h3>
Hello Libertine</h3>
Libertine is a system to manage app containers. It is specifically designed to support the many traditional X apps that are debian packaged. Each container is a separate Ubuntu rootfs populated through debian package installations. (Currently these containers are chroots: later, LXD contains will be supported. Also, currently the containers must be of the same Ubuntu series as the device: Vivid.)<br /><br/>
So, you can install or create a libertine container, install debian packaged X apps into it, and launch them using the XApps scope. The apps access to the user’s key directories: Documents, Downloads, Music, Pictures, and Videos. So data files created and saved by an app in one container are available to apps in any other container, and indeed outside of the containers.<br /><br/>
Let’s take a quick look at the XApps scope.<br /><br/>
<h4>
XApps Scope</h4>
This scope simply lists the containers and, for each container, it displays its apps. Here’s a look at a device with two containers. This system has two containers (Puritine and My Container). And each has a few apps:<br /><br/>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-Jzi0e6rNQWM/V400OpgsX8I/AAAAAAAADjs/xWAU3UwB4woTduusG2qzXwvVRe-Jngq1QCLcB/s1600/scope.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://1.bp.blogspot.com/-Jzi0e6rNQWM/V400OpgsX8I/AAAAAAAADjs/xWAU3UwB4woTduusG2qzXwvVRe-Jngq1QCLcB/s320/scope.png" width="192" /></a></div>
<ul>
<li>Tap an app to launch it.</li>
<li>long press an app to hide it.</li>
<li>If you have any hidden apps, see them from the search icon (magnifying glass) and tap Hidden X Apps. Long press a hidden app to unhide it.</li>
<li>Note that a container with no apps does not display in the scope.</li>
</ul>
So how does one create and delete containers, and add or remove apps from them?<br /><br/>
<h4>
Libertine Container Manager</h4>
libertine-container-manager is a command line tool you use on the device to create and manage containers. This includes installing debian packaged apps into them. (These containers are created in the phablet user’s home directory and are not a part of the read-only rootfs.)<br /><br/>
Note: libertine-container-manager currently cannot be run in the Terminal App. Instead please connect to your device from an Ubuntu system using phablet-shell.
<h5>
Listing Containers</h5>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager list<br />
puritine<br />
my-container
</div><br/>
The “puritine” container is pre-installed on many devices through the com.ubuntu.puritine click package (“Desktop Applications”):<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ click list | grep puritine<br />
com.ubuntu.puritine 0.11
</div><br/>
The second container (“my-container”) was created on the device with libertine-container-manager.<br /><br/>
Note: It is possible to pre-install customized containers through bespoke channels.<br />
<h5>
Creating a Libertine Container</h5>
You can create a new container on a device. The container needs a unique ID and (optionally) a name.<br /><br/>
Note: The container must be the same Ubuntu series as the device, currently: vivid.<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager create --id my-container --name "My Container" --distro vivid --type chroot<br /><br/>
I: Retrieving Release<br />
I: Retrieving Release.gpg <br />
I: Checking Release signature<br />
I: Valid Release signature (key id 790BC7277767219C42C86F933B4FE6ACC0B21F32)<br />
I: Retrieving Packages <br />
I: Validating Packages <br />
I: Resolving dependencies of required packages...<br />
[...]
</div>
<h5>Listing Apps in a Container</h5>
It’s easy to list the apps in a container. You just use the container’s id, as follows:<br /><br/>
Note: We add the optional --json argument here and show only lines with “name” for display convenience.<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager list-apps --id my-container --json | grep "\"name\""<br />
"name": "Panel Manager",<br />
"name": "Python (v3.4)",<br />
"name": "Python (v2.7)",<br />
"name": "gedit",<br />
"name": "Help",<br />
"name": "Notification Daemon",<br />
"name": "Terminal",
</div>
Also note that all apps that install a .desktop file are listed by this command, although many of them are not displayed in the XApps scope since they are not appropriate.<br />
<h5>Installing an app in a container</h5>
To install a debian package in a container, you just use install-package with the container id and the debian binary package name, as follows:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager install-package --id my-container --package terminator
</div><br/>
The package and all of its dependencies are installed in the container. After this, assuming the package installs a .desktop file, it displays in the XApps scope and is launchable with a tap as expected.<br />
<h5>
Installing an app from a specific Launchpad PPA</h5>
By default, available debian packages are installed from the standard Ubuntu archive the chroot’s apt configuration points to. You can add a launchpad PPA, as follows:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager configure --id my-container --archive ppa:USER/PPA-NAME
</div><br/>
(Currently, private PPAs are scheduled for an upcoming release.)<br /><br/>
After this, you can install packages into the container as usual, including from the PPA.<br />
<h5>
Removing apps from a container</h5>
Remove a debian package from a container with:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager remove-package --id my-container --package PACKAGE_NAME<br />
Libertine-container-manager help<br />
Use the --help for top level help.
</div>
<br />
You can see details on each subcommand, for example remove-package, as follows:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~$ libertine-container-manager remove-package --help<br />
usage: libertine-container-manager remove-package [-h] -p PACKAGE [-i ID] [-r]<br />
<br />
<br />
optional arguments:<br />
-h, --help show this help message and exit<br />
-p PACKAGE, --package PACKAGE<br />
Name of package to remove. Required.
-i ID, --id ID Container identifier. Default container is used if<br />
omitted.<br />
-r, --readline Readline mode. Use text-based frontend during debconf<br />
Interactions.
</div>
<h5>Updating a container</h5>
Want the debian packages in a container updated? Easy:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager update --id my-container<br />
Executing a Command in a Container<br />
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager exec --command "apt-get update" --id my-container<br />
Atteint http://ppa.launchpad.net vivid InRelease<br />
Atteint http://ports.ubuntu.com vivid InRelease<br />
Atteint http://ports.ubuntu.com vivid-updates InRelease<br />
Atteint http://ppa.launchpad.net vivid/main armhf Packages<br />
Atteint http://ppa.launchpad.net vivid/main Translation-en<br />
Atteint http://ports.ubuntu.com vivid/main armhf Packages
[...]
</div><br/>
Note: Running the apt-get update command in a container may be useful to update the container’s knowledge of newly available packages without installing/updating them all. You can then see whether a package is available, with:<br /><br />
<div class="monospace">
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager exec --command "apt-cache policy firefox" --id my-container<br />
firefox:<br />
Installé : (aucun)<br />
Candidat : 44.0+build3-0ubuntu0.15.04.1<br />
Table de version :<br />
44.0+build3-0ubuntu0.15.04.1 0<br />
500 http://ports.ubuntu.com/ubuntu-ports/ vivid-updates/main armhf Packages<br />
37.0+build2-0ubuntu1 0<br />
500 http://ports.ubuntu.com/ubuntu-ports/ vivid/main armhf Packages
</div>
<br />
<h4>
More about the Libertine Containers</h4>
As noted, the container is a directory containing an Ubuntu rootfs. Container directories are here:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~/.cache/libertine-container$ pwd<br />
/home/phablet/.cache/libertine-container<br />
phablet@ubuntu-phablet:~/.cache/libertine-container$ ls<br />
my-container puritine<br />
phablet@ubuntu-phablet:~/.cache/libertine-container$ cd my-container/<br />
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ ls<br />
rootfs
</div><br/>
You can get a bash shell into the container as follows:<br /><br/>
<div class="monospace">
phablet@ubuntu-phablet:~/.cache/libertine-container/my-container$ libertine-container-manager exec --command "/bin/bash" --id my-container<br />
groups: cannot find name for group ID 1001<br />
[...]<br />
root@ubuntu-phablet:/#
</div>
</body>
</html>KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com144tag:blogger.com,1999:blog-12988981498592988.post-89233378055445176322014-04-07T11:29:00.000-07:002014-04-07T11:29:13.098-07:00<h2 style="height: 0px;">
New Unity 8 Scopes Docs</h2>
<div>
<br /></div>
<div>
I have completed and published several documents on the Ubuntu development portal, scopes section. I hope these are useful when developing Unity 8 Scopes: </div>
<div>
<ul>
<li><a href="http://developer.ubuntu.com/scopes/guides/scopes-guide/">Scopes Guide</a>: Overview of Unity 8 Scopes</li>
<li><a href="http://developer.ubuntu.com/scopes/tutorials/scope-development-procedures/">Scopes Development Procedures</a>: Common procedures used during development</li>
<li><a href="http://developer.ubuntu.com/scopes/tutorials/unity-8-scope-example-openclipart/">Unity 8 Scope Example: Openclipart</a>: A sample scope that queries openclipart.com, with code discussion on key points for scope developers</li>
</ul>
The team also published these new docs:</div>
<div>
<ul>
<li><a href="http://developer.ubuntu.com/scopes/api/">Unity 8 Scope API reference docs</a></li>
<li>Which includes an <a href="http://developer.ubuntu.com/api/scopes/sdk-14.04/index/">In-Depth Scopes Developer Guide</a></li>
</ul>
</div>
KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com7tag:blogger.com,1999:blog-12988981498592988.post-49085197103332203832014-03-07T11:23:00.000-08:002014-03-07T11:23:15.490-08:00Ubuntu HTML5 App Dev SessionsNoting that this week I delivered three hour long sessions of interest to those interested in writing HTML5 apps for Ubuntu:<br />
<br />
<br />
<ul>
<li><a href="http://summit.ubuntu.com/appdevweek-1403/meeting/22148/html5-ubuntuui-components/">Tour of the Ubuntu HTML5 GUI Components</a></li>
<li><a href="http://summit.ubuntu.com/appdevweek-1403/meeting/22146/cordova-in-html5-apps/">Cordova in Ubuntu HTML5 Apps</a></li>
<li><a href="http://summit.ubuntu.com/appdevweek-1403/meeting/22149/html5-app-debugging/">Debugging Ubuntu HTML5 Apps</a></li>
</ul>
<br />
<br />KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com3tag:blogger.com,1999:blog-12988981498592988.post-90416126547604844812013-12-19T14:56:00.000-08:002013-12-19T14:56:12.483-08:00Cordova 3.3 adds UbuntuUpstream Cordova 3.3.0 is released just in time for the holidays with a gift we can all appreciate: built-in Ubuntu support!<br />
<h2>
<span style="color: #6fa8dc;">Cordova: multi-platform HTML5 apps</span></h2>
Apache Cordova is a framework for HTML5 app development that simplifies building and distributing HTML5 apps across multiple platforms, like Android and iOS. With Cordova 3.3.0, Ubuntu is an official platform!<br />
<br />
The cool idea Cordova starts with is a single <i>www/</i> app source directory tree that is built to different platforms for distribution. Behind the scenes, the app is built as needed for each target platform. You can develop your HTML5 app once and build it for many mobile platforms, with a single command.<br />
<br />
With Cordova 3.3.0, one simply adds the Ubuntu platform, builds the app, and runs the Ubuntu app. This is done for Ubuntu with the same Cordova commands as for other platforms. Yes, it is as simple as:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">$ cordova create myapp REVERSEDOMAINNAME.myapp myapp</span><br />
<span style="font-family: Courier New, Courier, monospace;">$ cd myapp</span><br />
<span style="font-family: Courier New, Courier, monospace;">(Optionally modify www/*)<br />$ cordova build [ ubuntu ]<br />$ cordova run ubuntu</span><br />
<h2>
<span style="color: #6fa8dc;">Plugins</span></h2>
Cordova is a lot more than an HTML5 cross-platform web framework though.<br />
It provides JavaScript APIs that enable HTML5 apps to use of platform specific back-end code to access a common set of devices and capabilities. For example, you can access device <a href="http://cordova.apache.org/docs/en/3.3.0/cordova_events_events.md.html#Events">Events</a> (battery status, physical button clicks, and etc.), <a href="http://cordova.apache.org/docs/en/3.3.0/cordova_geolocation_geolocation.md.html#Geolocation">Gelocation</a>, and a lot more. This is the Cordova "plugin" feature.<br />
<br />
You can add Cordova standard plugins to an app easily with commands like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">$ cordova plugin add org.apache.cordova.battery-status</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">(Optionally modify www/* to listen to the <a href="http://cordova.apache.org/docs/en/3.3.0/cordova_events_events.md.html#batterystatus">batterystatus</a> event )</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">$ cordova build [ ubuntu ]</span><br style="font-family: 'Courier New', Courier, monospace;" /><span style="font-family: 'Courier New', Courier, monospace;">$ cordova run ubuntu</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span>
Keep an eye out for news about how Ubuntu click package cross compilation capabilities will soon weave together with Cordova to enable deployment of plugins that are compiled to specified target architecture, like the armhf architecture used in Ubuntu touch images (for phones, tablets and etc.).<br />
<h2>
<span style="color: #6fa8dc;">Docs</span></h2>
As a side note, I'm happy to note that my documentation of initial Ubuntu platform support has landed and has been published at <a href="http://cordova.apache.org/docs/en/3.3.0/">Cordova 3.3.0 docs</a>.<br />
<br />
<br />KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com6tag:blogger.com,1999:blog-12988981498592988.post-4679507405838862292013-11-08T05:46:00.001-08:002013-11-08T06:11:10.992-08:00Ubuntu HTML5 API docs<h2>
HTML5 API docs published</h2>
<div>
I'm pleased to note that the Ubuntu HTML5 API docs I wrote are now done and published on <a href="http://developer.ubuntu.com/api/html5/sdk-1.0/UbuntuUI/">developer.ubuntu.com</a>. These cover the complete set of JavaScript objects that are involved in the UbuntuUI framework for HTML5 apps (at this time). For each object, the docs show how the corresponding HTML is declared and, of course, all public methods are documented.</div>
<div>
<br /></div>
<div>
A couple notes:</div>
<div>
<ul>
<li>I wrote an <a href="https://code.launchpad.net/~knitzsche/ubuntu-html5-theme/html5APIexerciserApp">html5APIexerciser</a> app that implements every available public method in the framework. This was helpful to ensure that what I wrote matched reality ;) It may be useful to folks exploring development of Ubuntu HTML5 apps. The app can be run directly in a browser by opening its <span style="font-family: Courier New, Courier, monospace;">index.html</span>, but it is also an Ubuntu SDK project, so it can be opened and run from the Ubuntu SDK, locally and on an attached device.</li>
<li>The html5APIexerciser app does not demonstrate the full set of Ubuntu CSS styles available. For example, the styles provide gorgeous toggle buttons and progress spinnners, but since they have no JavaScript objects and methods they are not included in the API docs. So be sure to explore the Gallery by installing the <b>ubuntu-html5-theme-examples</b> package and then checking out <span style="font-family: Courier New, Courier, monospace;">/usr/share/ubuntu-html5-theme/0.1/examples/</span></li>
<li>I decided to use <a href="http://yui.github.io/yuidoc/">yuidoc</a> as the framework for adding source code comments as the basis for auto generated web docs. After you install yuidoc using <b>npm</b> you can build the docs from source as follows:</li>
</ul>
<ol>
<li>Get the ubuntu-html5-theme branch: <span style="font-family: 'Courier New', Courier, monospace;">bzr branch lp:ubuntu-html5-theme</span></li>
<li>Move to the JavaScript directory: <span style="font-family: Courier New, Courier, monospace;">cd ubuntu-html5-theme/0.1/ambiance/js/</span></li>
<li><span style="font-family: inherit;">Build the docs: </span><span style="font-family: Courier New, Courier, monospace;">yuidoc -c yuidoc.json .</span><span style="font-family: inherit;"> This creates the </span><span style="font-family: Courier New, Courier, monospace;">./build</span><span style="font-family: inherit;"> directory.</span></li>
<li>Launch the docs by opening <span style="font-family: Courier New, Courier, monospace;">build/index.html</span><span style="font-family: inherit;"> in your browser. They should look something like <a href="http://people.canonical.com/~knitzsche/yuidoc-ubuntu-html5-theme/latest/">this</a> </span></li>
</ol>
</div>
<div>
<div>
Thanks to <span class="proflinkWrapper" style="background-color: white; color: #404040; font-family: arial, sans-serif; font-size: 12.800000190734863px; line-height: 14.399999618530273px;"><span class="proflinkPrefix" style="-webkit-transition: color 0.218s; color: #427fed; transition: color 0.218s;">+</span><a class="proflink" href="https://plus.google.com/101694416703170881163" oid="101694416703170881163" style="-webkit-transition: color 0.218s; color: #427fed; cursor: pointer; font-weight: bold; text-decoration: none; transition: color 0.218s;">Adnane Belmadiaf</a></span> for some theme work and his always helpful consultation, to <span class="proflinkPrefix" style="-webkit-transition: color 0.218s; background-color: white; color: #427fed; font-family: arial, sans-serif; font-size: 12.800000190734863px; line-height: 14.399999618530273px; transition: color 0.218s;">+</span><a class="proflink" href="https://plus.google.com/108263416239719671624" oid="108263416239719671624" style="-webkit-transition: color 0.218s; background-color: white; color: #427fed; cursor: pointer; font-family: arial, sans-serif; font-size: 12.800000190734863px; font-weight: bold; line-height: 14.399999618530273px; transition: color 0.218s;">Daniel Beck</a> for his initial writeup of the Ubuntu HTML5 framework, and of course to the developer.ubuntu.com team for their always awesome work!</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com0tag:blogger.com,1999:blog-12988981498592988.post-33032291426000483762013-03-01T13:26:00.002-08:002013-03-01T13:26:53.472-08:00Apport-Valgrind 2.9 supports unpackaged executables<a href="https://launchpad.net/apport/trunk/2.9">Apport-valgrind 2.9</a> picks up support for unpackaged [1] executables (thanks Martin Pitt for working with me to land my changes in trunk).<br />
<div>
<br /></div>
<div>
Here's what this means.</div>
<div>
<ul>
<li>Previously if you ran <span style="font-family: Courier New, Courier, monospace;">apport-valgrind EXE</span><span style="font-family: inherit;">, where EXE was not installed by a debian package, the process quit with a warning because it could only obtain debug symbol packages through debian dependencies</span></li>
<li><span style="font-family: inherit;">As of release 2.9, execution succeeds, and it obtains debug symbols packages based on the shared object (.so) files that the EXE is linked to</span></li>
</ul>
<div>
<br /></div>
</div>
<h1>
Digging deeper into how it works</h1>
<div>
As explained <a href="http://kylenubuntu.blogspot.com/2013_01_01_archive.html">here</a>, apport-valgrind is a wrapper for <b>valgrind</b>, the venerable memory leak finder (among other things). </div>
<div>
<br /></div>
<div>
Apport-valgrind first creates a sandbox directory that contains debug symbol files related to the executable you are checking for memory leaks. It then launches valgrind and points it also at the new sandbox directory. Valgrind creates memory leak stack traces, looking in the standard system directories for debug symbol files, but also in the generated sandbox directory. It uses the debug symbol files to create the most useful stack traces it can: stack traces that display source file names (instead of installed library file names) and function names (instead of "???", which is an unresolved symbol). All this is done without <i>installing</i> debug symbol packages onto your system: they are <i>unpacked</i> into the temporary sandbox directory, which is automatically deleted after use (unless an optional persistent sandbox is used). </div>
<div>
<br /></div>
<div>
Release 2.9 uses a new technique for populating the sandbox directory that extends support for executables that are not installed by a package. (More on why that may be useful below.)<br />
<br /></div>
<h1>
Flow for packaged EXE</h1>
<div>
Previously, the sandbox was populated only using <i>debian dependencies</i> of the debian package that installed the executable, like so:</div>
<div>
<ul>
<li>Find the package that installed the executable</li>
<li>Find that package's full set of dependencies (all the way down, that is, recursively)</li>
<li>For each, get the best debug symbol package available (this depends on your system's apt configuration, for example whether you have added "ddebs.ubuntu.com", as described <a href="http://kylenubuntu.blogspot.com/2013/02/dbg-versus-dbgsym.html">here</a>)</li>
<li>Unpack the debug symbol packages into the sandbox</li>
</ul>
<div>
This code path still exists and is used for packaged executables.<br />
<br /></div>
<div>
<h1>
Flow for <i>un</i>packaged EXE</h1>
</div>
</div>
<div>
With 2.9, the sandbox is populated with the debug packages related to the <i>executable's linked .so files</i>, as reported by <b>ldd</b>, like so:</div>
<div>
<ul>
<li>Use <b>ldd</b> to determine the executable's linked shared libraries (.so files)</li>
<li>Find the package that installed each linked shared library</li>
<li>For each, get the best debug symbol package available</li>
<li>Unpack the debug symbol packages into the sandbox</li>
</ul>
<div>
<br /></div>
<h1>
Use case</h1>
</div>
<div>
One use case I see is that one can now write a C program specifically to memory check a library with apport-valgrind simply by including and using the library. You do not need to rely only on existing debian packages to find executables that use the libraries in order to memory check them. And, you can target your C program to precisely target what you want to memory check, and no more.</div>
<div>
<br /></div>
<div>
For example: <b>libappindicator</b>.</div>
<div>
<br /></div>
<div>
As an exercise, I wanted to find memory leaks in application indicators (the icons on the top right like power, network, etc.), which means I needed to launch an app indicator at the command line with apport-valgrind (<span style="font-family: Courier New, Courier, monospace;">apport-valgrind EXE</span>). So I tried to kill those processes, and they relaunched automatically. </div>
<div>
<br /></div>
<div>
So I thought: write my own app indicator -- which turned out to be unnecessary: I found some sample C code that demonstrates how to write a simple application indicator <a href="http://developer.ubuntu.com/resources/technologies/application-indicators">here</a> on developer.ubuntu.com. (Scroll down to "Typical usage (C version)".)</div>
<div>
<br /></div>
<div>
So I grabbed it. It includes a libappindicator header file:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">#include <libappindicator/app-indicator.h></span></div>
<div>
<br /></div>
<div>
I compiled and linked this code, like so:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ gcc myappindicator.c $(pkg-config --cflags --libs appindicator3-0.1) -o myappindicator.o </span></div>
<div>
<br /></div>
<div>
That produces the <i>unpackaged</i> executable <span style="font-family: inherit;"><b>myappindicator.o</b></span></div>
<div>
<br /></div>
<div>
Running this at the command line (<span style="font-family: Courier New, Courier, monospace;">./myappindicator.o</span><span style="font-family: inherit;">) creates a new icon in the appindicator area that has a menu and pops up a dialog with a text editing area. The indicator icon's menu has a <b>Quit</b> item, which works as expected, quitting the application and removing its indicator, so all is well.</span></div>
<div>
<br /></div>
<div>
Then, I ran it under apport-valgrind [2], like so:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ apport-valgrind ./myappindicator.o </span></div>
</div>
<div>
<br /></div>
<div>
And voila, the valgrind log is generated: <span style="font-family: 'Courier New', Courier, monospace;">./valgrind.log</span><br />
<br />
<span style="font-family: inherit;">This sample C code could be simplified to more precisely target (memory check) libappindicator functions.</span><br />
<span style="font-family: inherit;"><br /></span></div>
<h1>
Quality of the unpackaged stack traces?</h1>
<div>
OK, so it <i>works</i>. But does it work well? That is, can one expect the same number of unresolved symbols for an <i>unpackaged</i> executable as for its <i>identical packaged</i> version, even though the code paths to create the sandbox are quite different?</div>
<div>
<br /></div>
<div>
Easy to find out by running apport-valgrind twice, once on a normal packaged executable found on the current PATH (for example <span style="font-family: 'Courier New', Courier, monospace;">apport-valgrind notify-send</span><span style="font-family: inherit;">), and then again on an instance of notify-send simply copied to the current directory (which makes it unpackaged, since dpkg cannot find any package that owns it in this location, and which invokes the new ldd-based code path instead of the dependency based code path). Then counting the number of unresolved symbols with <b>grep</b>, like so:</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Packaged:</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ apport-valgrind notify-send </span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">..</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ grep -c \?\?\? valgrind.log</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">89</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: inherit;">Unpackaged:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ cp $(which notify-send) .</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ apport-valgrind ./notify-send </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">..</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">$ grep -c \?\?\? valgrind.log</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">89</span></div>
</div>
<div>
<br />
Success! The same number of unresolved symbols in both cases means that the unpackaged stack traces are (here, at least) just as good as the packaged stack traces.<br />
<br />
<h1>
Footnotes</h1>
</div>
<div>
[1] You can tell if an executable is packaged with <span style="font-family: Courier New, Courier, monospace;">dpkg -S $(which EXE)</span><span style="font-family: inherit;"><span style="font-family: inherit;">.</span> For example, to find the package that installed <b>ls</b>:</span></div>
<div>
<br /></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace;">$ dpkg -S $(which ls)</span></div>
<span style="font-family: Courier New, Courier, monospace;">coreutils: /bin/ls</span>
<span style="font-family: Courier New, Courier, monospace;">
</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<span style="font-family: Courier New, Courier, monospace;">
</span>
<br />
<div>
The package is <b>coreutils</b>.</div>
<div>
<br /></div>
<div>
[2] Always upgrade your system (apt-get update and apt-get dist-upgrade) before running apport-valgrind in order to increase the quality of the stack traces by ensuring the installed libraries are the most recent and therefore will match with the most recent debug symbol packages.</div>
KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com0tag:blogger.com,1999:blog-12988981498592988.post-67684104451674698042013-02-04T14:51:00.000-08:002013-02-05T07:47:56.905-08:00dbg versus dbgsym<h1>
Will the real debug package please stand up</h1>
<br />
While mucking around with stack traces, valgrind, and <a href="http://kylenubuntu.blogspot.com/2013/01/apport-valgrind-case-of-apples-and.html">apport-valgrind</a>, I stumbled across two types of debug symbol <a href="http://kylenubuntu.blogspot.com/2013/02/dbg-versus-dbgsym.html#footnote1">[1]</a> packages:
<br />
<div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;"><b>dbg</b> packages, for example: empathy-dbg</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;"><b>dbgsym</b> packages, for example: empathy-dbgsym</span></li>
</ul>
</div>
Being rather new to this area, I was not sure which to use. But, I wanted to valgrind empathy, so I looked further.<br />
<br />
The package Description fields <a href="http://kylenubuntu.blogspot.com/2013/02/dbg-versus-dbgsym.html#footnote2">[2]</a> shed no light on which to use:<br />
<br />
<b>empathy-dbg</b>:<br />
<blockquote>
Description-en: GNOME multi-protocol chat and call client (debug symbols) Instant messaging program supporting text, voice, video, file transfers and inter-application communication over many different protocols, including: AIM, MSN, Google Talk (Jabber/XMPP), Facebook, Yahoo!, Salut, Gadu-Gadu, Groupwise, ICQ and QQ. This package contains the Empathy IM application and account manager.</blockquote>
<b>empathy-dbgsym</b>:<br />
<blockquote>
Description: debug symbols for package empathy Instant messaging program supporting text, voice, video, file transfers and inter-application communication over many different protocols, including: AIM, MSN, Google Talk (Jabber/XMPP), Facebook, Yahoo!, Salut, Gadu-Gadu, Groupwise, ICQ and QQ. This package contains the Empathy IM application and account manager.</blockquote>
Turning to the internet, I found many others had asked the same question, and they usually got answers like this:<br />
<br />
<blockquote>
You can use either the -dbg or the -dbgsym packages, but you can't use both.</blockquote>
Still, no guidance as to which to use. And anyway, I wanted to understand what was going on at a deeper level. Investigating further, I found that <b>dbg</b> packages are hosted in the standard archive (<a href="http://archive.ubuntu.com/ubuntu/dists/">archive.ubuntu.com</a>), whereas <b>dbgsym</b> packages are hosted in a special archive: <a href="http://ddebs.ubuntu.com/dists/">ddebs.ubuntu.com</a>.<br />
<br />
It also turns out that the <b>dbgsym</b> packages on ddebs.ubuntu.com com do not appear to be strictly normal debian packages:<br />
<ul>
<li>Normal debian packages end in ".deb" (for example: empathy-dbg_3.6.0.3-0ubuntu1_amd64.deb)</li>
<li>The packages on ddebs.ubuntu.com end in ".ddeb" (<span style="white-space: pre-wrap;">empathy-dbgsym_3.6.3-0ubuntu2_amd64.ddeb)</span></li>
</ul>
This ".ddeb" extension reminded me of ".udeb" packages, which I have encountered in the context of live-build installations. Are these <b>dbgsym</b>/ddeb packages also intended for some narrowly defined use case?<br />
<br />
<span style="font-family: inherit;">This</span><span style="font-family: inherit;"> warranted further poking around. </span>Here are my findings.<br />
<br />
<h1>
dbg packages</h1>
<br />
<b>dbg</b> packages are the traditional debian method for providing debug symbols separately from the normally installed binary packages.<br />
<b><br /></b>
<b>dbg</b> packages are created by upstream packagers and flow naturally into the standard Ubuntu archives. They are normal debian packages (that end in ".deb").<br />
<br />
There is usually only one binary <b>dbg</b> package per source package. That is, there are usually not separate <b>dbg</b> packages for individual binary packages, there's just one named after the source package, like empathy-dbg.<br />
<br />
They are pretty easy to make these days. The normal way is described <a href="http://wiki.debian.org/DebugPackage">here</a>.<br />
<br />
So, thanks to a lot of upstream work, the Ubuntu standard archive already has lots of these <b>dbg</b> packages. You can see a list of them with something like this:<br />
<br />
<code>$ apt-cache search dbg | less</code><br />
<br />
These <b>dbg</b> packages work as expected: if you install, for example, empathy-dbg, and then generate an empathy stack trace (with valgrind or gdb, for example):<br />
<ul>
<li>The symbols it contains should be converted to useful function names (instead of being question marks)</li>
<li>The source file names should be displayed (instead of paths to shared object (.so) and other binary files)</li>
</ul>
So let's take a moment to appreciate the work upstream maintainers have done!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://britscollegebound.files.wordpress.com/2010/10/thank-you-merci-gracias.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://britscollegebound.files.wordpress.com/2010/10/thank-you-merci-gracias.jpg" height="182" width="320" /></a></div>
<br />
But, not every package in Ubuntu has a <b>dbg</b> package. And, even those that do may have created them in different ways, and possibly with different naming conventions, which makes them difficult (or impossible) to find mechanistically.<br />
<br />
This is where <b>dbgsym</b> packages come in.<br />
<br />
<h1>
dbgsym packages</h1>
<br />
Consider the Ubuntu archive: thousands of binary packages (more than 7000 in the Ubuntu 12.10 Quantal main archive component alone).<br />
<br />
That's a lot of different people in teams maintaining a lot of packages.<br />
<br />
Inevitably, some packages that arguably should provide binary debug symbol packages do not. Or they do, but in a slightly different way, for example, the package name may not follow the standard pattern.<br />
<br />
This wonderful diversity adds up to a concrete problem: the <i>complete</i> set of debug symbols required for debugging and creating useful stack traces <i>of any arbitrary packaged C executable</i> (ELF file) are not reliably present in standard Ubuntu archives.<br />
<br />
Suppose I need to check something for memory leaks using apport-valgrind or debug it with gdb and I hit one of these missing bits?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-yaPJ0WT9oWk/TWchirXDYEI/AAAAAAAAAMw/Oxq5fiIMguc/s320/yikes.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-yaPJ0WT9oWk/TWchirXDYEI/AAAAAAAAAMw/Oxq5fiIMguc/s320/yikes.jpg" /></a></div>
<br />
<h1>
Automation \o/</h1>
<br />
Fortunately, there is a systematic (and automatic) solution (which is almost fully working).<br />
<br />
The idea is to automate the generation of debug symbols at package build time (whether or not the package maintainers took steps to create <b>dbg</b> packages). For each binary package that installs an executable for which symbols may be helpful, automatically create a new BINARYPACKAGE-<b>dbgysm</b> package, and let it install the appropriate set of debug symbol binaries. Make it <i>conflict</i> with the <b>dbg</b> package to avoid file collisions. Then, publish them in ddebs.ubuntu.com.<br />
<br />
That's two steps:<br />
<ol>
<li>Creating the <b>dbgsym</b> binary packages (this part seems to be fully implemented)</li>
<li>Publishing them to ddebs.ubuntu.com (this part may not be fully implemented)</li>
</ol>
<h1>
Binary package oriented</h1>
<br />
The <b>dbgsym</b> approach has a finer granularity than the usual <b>dbg</b> approach: it is <i>binary package oriented</i>. That is, a <b>dbgysm</b> package is created <i>for</i> <i>each</i> <i>appropriate</i> <i>binary package</i> (each that installs an ELF), whereas with the traditional, manual <b>dbg</b> approach, there is usually one <b>dbg</b> package created for all binary packages.<br />
<br />
So there are often many <b>dbgsym</b> packages per source package with this approach, which results in many more <b>dbgsym</b> packages than <b>dbg</b> packages, a <i>good</i> thing: you only need to download and install the parts you want.<br />
<br />
<h1>
Conflicts - a closer look</h1>
<br />
These automatically generated binary <b>dbgsym</b> packages install some of the <i>same files</i> <a href="http://kylenubuntu.blogspot.com/2013/02/dbg-versus-dbgsym.html#footnote3">[3]</a> as the corresponding <b>dbg</b> package, by design. But, without additional packaging steps, this would amount to a packaging error (since no single file may be installed by two binary packages without special steps to resolve this collision).<br />
<br />
This is automatically handled by setting the <b>dbgsym</b> package to <b><i>conflict</i></b> with the <b>dbg</b> package(s). (Conflicting packages cannot both be fully installed at the same time.) This explains why I found so many statements saying that "dbg and dbgsym packages cannot both be installed at the same time." It is literally <i>not possible</i> due to these intentional <i>conflicts</i> settings.<br />
<br />
<h1>
How dbgsym packages are created</h1>
<br />
<span style="font-family: inherit;"><b>dbgsym</b> packages are automatically created during a normal source package binary build by the <b>pkg-create-dbgsym</b> package, if it is installed. That is, if you install pkg-create-dbgsym, whenever a debian source package is built (for example with <code>debuild</code> in the source tree), pkg-create-dbgsym scripts create the <b>dbgysm</b> binary packages and set them to conflict as appropriate. (Thanks to Martin Pitt and other debian developers.) These steps are in addition to the normal activities of a build, so you also generate non-debug binary packages.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">For example, after installing pkg-create-dbgsym, I built the empathy source package, and here are the binary packages that were created <a href="http://kylenubuntu.blogspot.com/2013/02/dbg-versus-dbgsym.html#footnote4">[4]</a>. Lots of <b>dbgsym</b> ddebs!</span><br />
<br />
<h1>
ddeb: Huh? What?</h1>
<br />
These <b>dbgsym</b> binary package files end in ".ddeb", not ".deb", so what's up with that?<br />
<br />
As far as I can tell, this is merely a convenience for archive maintenance. That is: a .ddeb is exactly the same as a .deb, except that its extension is different. This serves the purpose of allowing archive management tools to find ddeb binary packages and treat them differently, in this case:<br />
<ul>
<li>Posting them on ddebs.ubuntu.com </li>
<li>Not posting them on archive.ubuntu.com</li>
</ul>
<h1>
Enough already: which to use?</h1>
<br />
So, after this exploration, I vote for <b>dbgsym</b> packages over <b>dbg</b> packages for the following reasons<br />
<ul>
<li>The trend is towards automatic generation of <b>dbgsym</b> packages and automatic posting of them to separate ddebs archive like http://ddebs.ubuntu.com</li>
<li>Such ddeb archives should contain more debug symbol packages than the traditional <b>dbg</b> packages because they are automatically created for all appropriate packages (instead of relying on maintainers to add a <b>dbg</b> package) so you get more</li>
<li>Such ddeb archives certainly should contain equivalents for the existing <b>dbg</b> packages as well, so you lose nothing</li>
<li>If you want, you can just get the debug symbols you need, since <b>dbgsym</b> packages are more granular </li>
</ul>
This all means that if you use <b>dbgsym</b> packages, your stack traces and debugging experiences are more likely to provide you the data you need to get your work done, with less bloat.<br />
<br />
Also, manually created <b>dbg</b> packages may disappear over time. Why? No need for package maintainers to think about debug packages. And the normal archives get smaller as debug symbol packages disappear.<br />
<br />
<h1>
Using dbgsym</h1>
<br />
As noted, the <b>dbgsym</b> packages are published on ddebs.ubuntu.com. Ubuntu systems do not know this archive by default.<br />
<br />
You can easily add it to your "software sources". For example, the following line adds ddebs.ubuntu.com for Quantal, main to your <span style="font-family: Courier New, Courier, monospace;">/etc/apt/sources.list</span> file:<br />
<br />
<code>deb http://ddebs.ubuntu.com quantal main</code><br />
<code><br /></code>
Then run <code>sudo apt-get update</code>, and you should be all set to go.<br />
<br />
For example, check whether empathy-dbgsym is available for installation:<br />
<br />
<code>$ apt-cache policy empathy-dbgsym</code><br />
<code>empathy-dbgsym:</code><br />
<code> Installed: 3.6.0.3-0ubuntu1</code><br />
<code> Candidate: 3.6.0.3-0ubuntu1</code><br />
<code> Version table:</code><br />
<code> *** 3.6.0.3-0ubuntu1 0</code><br />
<code> 500 http://ddebs.ubuntu.com/ quantal/main amd64 Packages</code><br />
<code> 500 http://ddebs.ubuntu.com/ quantal/main amd64 Packages</code><br />
<code> 100 /var/lib/dpkg/status</code><br />
<br />
There it is! (I happen to have this one installed.)<br />
<br />
How about the dbgsym version of empathy's binary package account-plugin-aim?<br />
<br />
<code>$ apt-cache policy account-plugin-aim-dbgsym</code><br />
<code>account-plugin-aim-dbgsym:</code><br />
<code> Installed: (none)</code><br />
<code> Candidate: 3.6.0.3-0ubuntu1</code><br />
<code> Version table:</code><br />
<code> 3.6.0.3-0ubuntu1 0</code><br />
<code> 500 http://ddebs.ubuntu.com/ quantal/main amd64 Packages</code><br />
<br />
There it is!<br />
<br />
<h1>
A caveat</h1>
<br />
It appears that not all <b>dbgsym</b> binary packages that one might expect are actually present on ddebs.ubuntu.com.<br />
<br />
For example, when I built empathy on a Quantal amd64 system with pgk-create-dbgsym installed, the expected <b>dbgsym</b> binary packages <i>were</i> created, including, for example, one for the account-plugin-irc binary package, and here it is:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">account-plugin-irc-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb</span><br />
<br />
But, I don't see a <b>dbgsym</b> package for account-plugin-irc published in the Ubuntu ddebs quantal main <a href="http://ddebs.ubuntu.com/dists/quantal/main/binary-amd64/Packages">archive</a>. The same is true for some (but by no means all) other binary packages deriving from empathy, like account-plugin-icq-dbgsym<br />
<br />
This would seem to be a hiccup in the <i>publishing</i> part of the automation, not the <b>dbgsym</b> <i>generation</i> part.<br />
<br />
<h1>
Footnotes</h1>
<br />
<a href="https://www.blogger.com/blogger.g?blogID=12988981498592988" name="footnote1"></a>[1] <b>Debug symbols</b> fill in stack traces to make them more human readable by substituting C function names and C source code file name for question marks and library file names. See <a href="http://kylenubuntu.blogspot.com/2013/01/apport-valgrind-case-of-apples-and.html">apport-valgrind</a> for details.
<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=12988981498592988" name="footnote2"></a>[2] You can display a <b>package Description</b> for any package your system's apt knows about with <code>apt-cache show PACKAGE</code>, and look for the Description field.<br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=12988981498592988" name="footnote3"></a>[3] <b>Conflicting</b> with the <b>dbg</b> package(s), an example using account-plugin-aim as an example:<br />
<br />
The empathy source package has the <b>accounts-plugin-aim </b>binary package. It installs the following file: /usr/lib/libaccount-plugin-1.0/providers/libaim.so (as shown by dpkg -S /usr/lib/libaccount-plugin-1.0/providers/libaim.so).<br />
<br />
The <b>empathy-dbg</b> package installs a debug symbol version of this file in a debug directory: /usr/lib/debug/usr/lib/libaccount-plugin-1.0/providers/libaim.so. There is no conflict yet, since the paths are different.<br />
<br />
The automatically generated <b>dbgsym</b> packages for empathy include <b>account-plugin-aim-dbgsym</b>. This package installs <i>the same</i> debug .so file as we saw for the <b>empathy-dbg</b> package: /usr/lib/debug/usr/lib/libaccount-plugin-1.0/providers/libaim.so.<br />
<br />
That's a <b>conflict</b>. Two packages (empathy-dbg and account-plugin-aim-dbgsym) install the same file.<br />
<br />
This is expected and handled by making the <b>dbgysm</b> binary packages conflict with the <b>dbg</b> package (if any), which it does:<br />
<br />
<code>$ apt-cache show account-plugin-aim-dbgsym | grep Conflicts<br />
Conflicts: empathy-dbg </code><br />
<br />
<a href="https://www.blogger.com/blogger.g?blogID=12988981498592988" name="footnote4"></a>[4] With <b>pkg-create-dbgsym installed</b>, building empathy creates all these debs and <b>dbgsym/ddebs</b>:<br />
<br />
account-plugin-zephyr_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-yahoojp_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-yahoo_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-sip_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-sametime_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-salut_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-myspace_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-mxit_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-jabber_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-irc_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-icq_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-groupwise_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-gadugadu_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-aim_3.6.0.3-0ubuntu1_amd64.deb<br />
mcp-account-manager-uoa_3.6.0.3-0ubuntu1_amd64.deb<br />
mcp-account-manager-goa_3.6.0.3-0ubuntu1_amd64.deb<br />
nautilus-sendto-empathy_3.6.0.3-0ubuntu1_amd64.deb<br />
empathy-dbg_3.6.0.3-0ubuntu1_amd64.deb<br />
empathy_3.6.0.3-0ubuntu1_amd64.deb<br />
account-plugin-zephyr-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-yahoojp-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-yahoo-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-sip-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-sametime-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-salut-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-myspace-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-mxit-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-jabber-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-irc-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-icq-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-groupwise-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-gadugadu-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
account-plugin-aim-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
mcp-account-manager-uoa-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
mcp-account-manager-goa-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
nautilus-sendto-empathy-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
empathy-dbgsym_3.6.0.3-0ubuntu1_amd64.ddeb<br />
empathy-common_3.6.0.3-0ubuntu1_all.deb<br />
<br />KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com3tag:blogger.com,1999:blog-12988981498592988.post-76979991358697917002013-01-26T08:35:00.001-08:002013-01-29T09:48:06.280-08:00Apport-Valgrind: a case of apples and oranges?<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<h2>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;">Who doesn't love a good fruit salad?</span></h2>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">As noted a previous <a href="http://kylenubuntu.blogspot.com/2013/01/on-jan-8-2013-my-apport-valgrind-landed.html">post</a>, Ubuntu 13.04 (Raring) has the new <b>apport-valgrind</b> binary package [1]. </span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: Arial, Helvetica, sans-serif;">But </span><span style="font-family: Arial, Helvetica, sans-serif;"><b>apport</b> is a crash reporting system, whereas <b>valgrind</b> is memory leak detector (among other things). So it may make you wonder: what's the connection? Is this</span><span style="font-family: Arial, Helvetica, sans-serif;"> a case of apples-and-oranges?</span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Nr5RH_w1Eco/UQK-zxysDLI/AAAAAAAAAJA/q3NxAw7A2x8/w497-h373/IMG_20130125_121428.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-Nr5RH_w1Eco/UQK-zxysDLI/AAAAAAAAAJA/q3NxAw7A2x8/w497-h373/IMG_20130125_121428.jpg" height="320" width="240" /></a></div>
<br />
<div class="separator" style="clear: both;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Arial, Helvetica, sans-serif;">Let's try to answer this, starting with a look at valgrind.</span></div>
<div>
<h2>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;">Valgrind</span></h2>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Valgrind is a workhorse of looking at C programs at run time. It has many capabilities. The one we are interested in here is its 'memcheck' tool, which is used to find memory leaks [2]. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">For example, you can run </span><span style="font-family: Courier New, Courier, monospace;">valgrind --tool=memcheck /usr/bin/foo</span><span style="font-family: Arial, Helvetica, sans-serif;"> [3] and a report is generated:</span></div>
<div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The report shows memory leaks created while running </span><span style="font-family: Courier New, Courier, monospace;">/usr/bin/foo</span><span style="font-family: Arial, Helvetica, sans-serif;">. This includes leaks in foo's code and leaks in any code executed by foo, for example called functions that exist in external shared libraries. </span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">For each leak, the log contains a <b>stack trace</b> that shows the sequence of function calls that created the memory leak. You can follow the stack trace by hand and find the code errors that lead to the memory leak, except for one issue: raw stack traces are like swiss cheese with lots of missing bits.</span></li>
</ul>
</div>
<h3>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Raw stack traces are like swiss cheese</span></h3>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">The problem with a raw stack trace is that it is full of holes.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-uTGwDPQxUeI/TwZS795vA4I/AAAAAAAABrk/RpD1asqHPOo/s1600/SwissCheese.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-uTGwDPQxUeI/TwZS795vA4I/AAAAAAAABrk/RpD1asqHPOo/s1600/SwissCheese.jpg" height="240" width="320" /></a></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">What's missing? </span></div>
<div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">In a raw valgrind stack trace, one does not see function names or source code file names</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Instead one sees question marks for function names and executable file names (frequently shared object library files) instead of source file names</span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Fortunately, valgrind can display the function names and source code file names if the <b>debug symbols</b> are present (more on this below). </span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Let's take a look at some real-life valgrind output and compare how it looks before and after debug symbols are present.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">
</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Here's a line from a <b>raw </b>valgrind memory leak stack trace:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">==2874== by 0x51E727D: ??? (in /usr/lib/x86_64-linux-gnu/libgtk-3.so.0.600.0)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">And here's how the same line looks with <b>debug symbols</b> for libgtk-3.0:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">==3643== by 0x51E727D: gtk_menu_item_class_intern_init (gtkmenuitem.c:427)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In the raw example: </span></div>
<div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The function name is unknown: </span><span style="font-family: 'Courier New', Courier, monospace;">???</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The shared object file name in which the function lives is listed: <span style="font-family: 'Courier New', Courier, monospace;">/usr/lib/x86_64-linux-gnu/libgtk-3.so.0.600.0</span></span></li>
</ul>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In the debug symbols example:</span></div>
<div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The function name is shown: </span><span style="font-family: 'Courier New', Courier, monospace;">gtk_menu_item_class_intern_init</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The source code file name is shown: </span><span style="font-family: 'Courier New', Courier, monospace;">gtk_menu_item_class_intern_init:427</span><span style="font-family: Arial, Helvetica, sans-serif;"> (It even knows the line number: 427)</span></li>
</ul>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Clearly, the second stack trace is much more useful for a human being who wants find and fix a memory leak. The question therefore is: how can we easily obtain the debug symbols that will make our valgrind stack traces useful? That's where apport comes in.</span></div>
<h2>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;">The apport connection</span></h2>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Apport is a collection of packages and tools that provide automatic crash reporting. Among them is </span><b style="font-family: 'Courier New', Courier, monospace;">apport-retrace</b><span style="font-family: Arial, Helvetica, sans-serif;">. This little gem provides the debug symbol capabilities for valgrind through its</span><span style="font-family: Arial, Helvetica, sans-serif;"> </span><span style="font-family: Courier New, Courier, monospace;">apport-retrace </span><span style="font-family: Arial, Helvetica, sans-serif;">script.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">It has the </span><span style="font-family: Arial, Helvetica, sans-serif;">ability to find, download and extract available debug packages for all dependencies of a packaged [4] executable. (This was used for purposes unrelated to finding memory leaks with valgrind, but as of Raring, it is used for valgrind purposes too.)</span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both;">
<span style="font-family: Arial, Helvetica, sans-serif;">Let's unpack this a bit. Suppose the executable is </span><span style="font-family: Courier New, Courier, monospace;">nm-applet.</span><span style="font-family: Arial, Helvetica, sans-serif;"> Apport-retrace could:</span></div>
<div class="separator" style="clear: both;">
</div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Find the tree of packages that </span><span style="font-family: Courier New, Courier, monospace;">nm-applet</span><span style="font-family: Arial, Helvetica, sans-serif;"> needs to run (that is, the package that owns <span style="font-family: 'Courier New', Courier, monospace;">nm-applet</span> and that package's direct and indirect dependencies)</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">Find the available debug symbol packages for them and make their debug symbol files available in a temporary directory (in /tmp, unless you specify otherwise) called the 'sandbox'</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The debug symbols packages are not installed, but extracted into the sandbox directory</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The sandbox directory is deleted after use (again, unless you specified your own sandbox directory) </span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">This automatic discovery of the complete set of available debug symbol files related to an executable would be very useful for the valgrind use case, as explained above. And as of Raring, we have it, as explained below. (Thanks to Martin Pitt for his invaluable assistance with this work.)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">But first, a nice thing to note: e</span><span style="font-family: Arial, Helvetica, sans-serif;">xtracting (rather than installing) the debug packages has a smaller impact on the the system. Notably: the extract directory (the sandbox) can simply be deleted after use to regain the disk space. No debug packages are actually installed, so no package removal needs to be done either. If you intend to valgrind the same executable many times, you can reuse a persistent sandbox, or simply install the debug symbol packages.</span></div>
<div>
<h2>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;">Pointing valgrind at the sandbox</span></h2>
</div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">We have seen that if debug symbol files are available, valgrind uses them to make much more useful stack traces. And we have seen that apport can download and extract into a temporary sandbox all available debug symbols for the packaged executable for which we want to find memory leaks. </span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">This leaves one more piece of the puzzle: telling valgrind to also look in the sandbox directory for debug symbols. (Valgrind always looks in the normal system directories for debug symbols.)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In Ubuntu 13.04 (Raring), the valgrind package now has this ability thanks to a patch from Alex Chiang. You simply add the information to the valgrind command line using the </span><span style="font-family: Courier New, Courier, monospace;">--extra-debuginfo-path=DIR</span><span style="font-family: Arial, Helvetica, sans-serif;"> argument.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">With these two functions in place, the table is set for <b>apport-valgrind</b>, as described next.</span></div>
<h2>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: x-large;">Apport and valgrind: two peas in a pod</span></h2>
<div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">In Ubuntu 13.04 (Raring), the <b>apport-valgrind</b> package is introduced. The apport code that creates the sandbox was moved from apport-retrace into a new apport python module: apport/sandboxutils.py, which makes it available for new code, such as apport-valgrind.</span></div>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">So here's what apport-valgrind does:</span></div>
<div>
<ul>
<li><span style="font-family: Arial, Helvetica, sans-serif;">It uses apport/sandboxutils to obtain and</span><span style="font-family: Arial, Helvetica, sans-serif;"> extract all available debug symbol packages into the sandbox directory for your (packaged) executable</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">It calls valgrind and tells it to also look in the sandbox directory (with the new <span style="font-family: 'Courier New', Courier, monospace;">--extra-debuginfo-path</span> argument)</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">When valgrind is done, the sandbox directory is deleted (there are options for persistent sandboxes -- useful when you want to use valgrind repeatedly)</span></li>
<li><span style="font-family: Arial, Helvetica, sans-serif;">The valgrind log file is created in the current directory: ./vagrind.log</span></li>
</ul>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">So with a single command like this: </span><span style="font-family: Courier New, Courier, monospace;">apport-valgrind nm-applet</span><span style="font-family: Arial, Helvetica, sans-serif;">, you can generate a valgrind memory check log of stack traces with all available debug symbols used, with no new debug packages installed on the system and all debug files (the sandbox) automatically deleted after execution.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;">So, apport and valgrind, a case of two peas in a pod :)</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-83YP437_z_g/Tqnz60V-4EI/AAAAAAAABdw/qrBk0ZBG-jo/s1600/Peas+Pod.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-83YP437_z_g/Tqnz60V-4EI/AAAAAAAABdw/qrBk0ZBG-jo/s1600/Peas+Pod.jpg" height="320" width="320" /></a></div>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">All of these strained metaphors about food make me hungry: time for lunch.</span></div>
<h4>
<span style="font-family: Arial, Helvetica, sans-serif;">Footnotes</span></h4>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">[1] Install it with </span><span style="font-family: Courier New, Courier, monospace;">sudo apt-get install apport-valgrind</span><span style="font-family: Arial, Helvetica, sans-serif;">. You will also want to install valgrind and valgrind-dbgsym: </span><span style="font-family: Courier New, Courier, monospace;">sudo apt-get install valgrind valgrind-dbgsym</span></div>
<div>
<br /></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">[2] A memory leak occurs in C when you allocate memory on the heap and fail to deallocate ('free') the memory. Such memory is not available for use by any process until the the code's main process terminates. If the process does not terminate (perhaps it is a daemon, or a piece of code that is persistent, such as a part of the desktop GUI) the memory is essentially lost forever.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">[3] One picks the valgrind tool with </span><span style="font-family: Courier New, Courier, monospace;">--tool=TOOLNAME</span><span style="font-family: Arial, Helvetica, sans-serif;">. The memcheck tool is the default, so there is no actual need to specify it.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">[4] If the executable is not in a debian package, apport does not know how to find and get all appropriate debug packages. There are other approaches to automate this. See future posts here.</span></div>
</div>
KyleNhttp://www.blogger.com/profile/05975055015232126866noreply@blogger.com0