<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	xmlns:georss="http://www.georss.org/georss" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:media="http://search.yahoo.com/mrss/"
	>

<channel>
	<title>bholley.work.blog</title>
	<atom:link href="http://bholley.wordpress.com/feed/" rel="self" type="application/rss+xml" />
	<link>http://bholley.wordpress.com</link>
	<description>Things I&#039;m Working On</description>
	<lastBuildDate>Thu, 21 Mar 2013 16:43:26 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.com/</generator>
<cloud domain='bholley.wordpress.com' port='80' path='/?rsscloud=notify' registerProcedure='' protocol='http-post' />
<image>
		<url>http://s2.wp.com/i/buttonw-com.png</url>
		<title>bholley.work.blog</title>
		<link>http://bholley.wordpress.com</link>
	</image>
	<atom:link rel="search" type="application/opensearchdescription+xml" href="http://bholley.wordpress.com/osd.xml" title="bholley.work.blog" />
	<atom:link rel='hub' href='http://bholley.wordpress.com/?pushpress=hub'/>
		<item>
		<title>At Long Last: Compartment-Per-Global</title>
		<link>http://bholley.wordpress.com/2012/05/04/at-long-last-compartment-per-global/</link>
		<comments>http://bholley.wordpress.com/2012/05/04/at-long-last-compartment-per-global/#comments</comments>
		<pubDate>Fri, 04 May 2012 15:28:44 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=192</guid>
		<description><![CDATA[Yesterday morning, I landed a very major change to our Javascript, XPConnect, and DOM code. This change is commonly known as Compartment-Per-Global. Less formally, it&#8217;s CPG. Even less formally, it&#8217;s the ****-ing holy grail that blocks everything important. I wanted to give a few details about what&#8217;s going on here, and what to expect. Before [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=192&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>Yesterday morning, I landed a very major change to our Javascript, XPConnect, and DOM code. This change is commonly known as Compartment-Per-Global. Less formally, it&#8217;s <em>CPG</em>. Even less formally, it&#8217;s <em>the ****-ing holy grail that blocks everything important</em>. I wanted to give a few details about what&#8217;s going on here, and what to expect.</p>
<p>Before getting deep into history, I want to leave a quick up-front warning for frontend/chrome developers. Every scope, including JSMs, now has its own compartment. So objects that formerly touched each other directly now interact via wrappers. This should be pretty transparent, since chrome-&gt;chrome wrappers just forward everything along. Here&#8217;s the rub though: when chrome accesses content objects, it gets a special Xray (formerly known as XPCNativeWrapper) view on the underlying object, which bypasses content expandos and gives chrome a fresh slate. This wrapper also has a special expando holder to store expandos that chrome wants to put on the object. Xray wrappers are per-compartment. So previously there would only be one Xray wrapper (and thus one holder object) for a given content object, but now there can be several. This means that <strong>different chrome scopes no longer share expandos</strong> for content objects. You have been warned.</p>
<p>Back to the story.</p>
<p>Those who were around for the Firefox 4 release remember the hair-raising months when some of Mozilla&#8217;s most seasoned hackers struggled to land one of the release&#8217;s flagship features: <strong>Compartments</strong>. Compartments weren&#8217;t a hugely sexy feature from a marketing standpoint, but JaegerMonkey wouldn&#8217;t work without them, and shipping Firefox 4 without the new fast Javascript Engine would quite possibly have been the end of Mozilla. Remember Brain Transplants? That was the final piece of Compartments. Needless to say, it was one of the most stressful times to be involved in this area of the platform. I was quite thankful to be watching from the sidelines while finishing up my degree.</p>
<div class="wp-caption alignnone" style="width: 521px"><a href="http://bholley.files.wordpress.com/2012/05/computing_stress.jpg"><img class="size-full wp-image" title="Compartments and Brain Transplants" src="http://bholley.files.wordpress.com/2012/05/computing_stress.jpg?w=497" alt="Compartments and Brain Transplants" width="511" height="504" /></a><p class="wp-caption-text">mrbkap, circa October 2010</p></div>
<p>Andreas has a nice blog post explaining <a href="http://andreasgal.com/2010/10/13/compartments/">what compartments are and why they&#8217;re important</a>. The details are complicated, but suffice it to say that all Javascript objects used to share a single heap, and Compartments gave us a separate heap for each origin. This was a win on a number of fronts, including decreased GC pauses and stronger security invariants.</p>
<p>Most importantly though, it made everything under the hood make much, <em>much</em> more sense. I use this phrase broadly to refer to both understandability (how easy is it for the programmer to grok what&#8217;s happening) and power (how well do the abstractions we use map to the things we want to do). The value of this can&#8217;t be overstated; It&#8217;s the difference between &#8220;we <em>could</em> fix that &#8211; I guess &#8211; It would probably take a month to implement and another month to track down all the regressions&#8221; and &#8220;sure, I can probably write and land that by the end of the day&#8221;. The more sense the code makes, the easier it is to diagnose and fix bugs, add new features, and make things fast. And as a rule, more compartments makes the code make more sense.</p>
<p>So having one compartment for every origin (whereas before, we effectively had one compartment for all of Firefox) was an amazing milestone. It helped us ship Firefox 4, paved the way for countless improvements and modernizations, and put us in a much better place with respect to things like security. But while having one compartment per origin was an amazing milestone, having one compartment per global object (think |window| in javascript) is <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=650353#c0">a never-ending stream of goodness</a>.</p>
<div class="wp-caption aligncenter" style="width: 250px"><a href="http://bholley.files.wordpress.com/2012/05/rainbow-stream.jpg"><img class="wp-image  " title="Rendition" src="http://bholley.files.wordpress.com/2012/05/rainbow-stream.jpg?w=240&#038;h=134" alt="Rendition" width="240" height="134" /></a><p class="wp-caption-text">Artist&#8217;s Rendition of Compartment-Per-Global<br />(by Demilion &#8211; <a href="http://fav.me/d3g2fue" rel="nofollow">http://fav.me/d3g2fue</a> )</p></div>
<p>So why didn&#8217;t we do Compartment-Per-Global from the beginning? There are a number of reasons, but two stand out. First, compartments at that time imposed a much larger memory overhead, and so we didn&#8217;t want to be be creating them willy-nilly. More importantly though, ensuring that same-origin code shared compartments was a good way to avoid having too much stuff break. Objects in different compartments have a membrane between them that can only be crossed by <strong>Cross-Compartment-Wrappers</strong>. These wrappers are mostly transparent, but in practice there are a few ways in which they change behavior in subtle ways. Fortunately for us at the time, the security model of the web places pretty strong barriers between code running from different origins. So in general, code wasn&#8217;t expecting to do intimate and subtle things with objects from other origins. Therefore, by introducing compartments on a per-origin basis, we could avoid breaking too much existing code. Privileged browser code (aka <strong>chrome</strong>) was a particularly big beneficiary of this strategy. Chrome code all runs with the same principal (<strong>System Principal</strong>), so most of it saw no change.</p>
<p>So Firefox 4 landed with a splash, and everyone was happy. For a while. As time went on, it became apparent that more and more stuff that we wanted to do required a one-to-one mapping between compartments and global objects. A lot of these dependency chains are pretty long. For example, IonMonkey won&#8217;t work until we get rid of JSStackFrames, but we can&#8217;t get rid of JSStackFrames until we change our security architecture to pull principals directly off object and context compartments, but we can&#8217;t do that until the compartment principal exactly matches that of the scope (i.e. until we have CPG). And in general, there are lots of operations that can be made much faster with the help of CPG.</p>
<p>So about a year ago, Luke started looking into what it would take to make this a reality. A number of dependent bugs had to be fixed first, and other work got in the way. By early January, he had patches that would at least launch the browser. But there were lots of things that were broken. So I took the reins for a while, and pounded away at XPConnect, the JS Engine, the DOM, and the frontend to get things green. Some fixes <a href="http://hg.mozilla.org/mozilla-central/rev/597edb1cfe21">were simple</a>. Others <a href="http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=a8115574617b">were harder</a>. Some had a <a href="http://hg.mozilla.org/mozilla-central/rev/850547d0b635">very high comment-to-code ratio</a>. But almost everything was landed as dependent fixes, meaning that the final patches that flipped the switch were quite manageable.</p>
<p>The final landing was small, but its effects rippled throughout the code. So every few days it was delayed, something new would be checked into the tree that broke it. This made the last stretch pretty strenuous. But thankfully, I wasn&#8217;t alone. Luke did a ton of work (and in fact holds authorship on eight of the nine patches that landed on Thursday). Mano, Mak, and zpao were hugely helpful in diagnosing browser-chrome issues. Blake was on the hook for reviewing all the really tricky stuff. Boris provided much-needed insight, and took some of my other work so that I could finish this stuff up. Johnny greased the wheels when they needed it. Kyle heckled. And Ed Morley helped out at the last minute to avert a backout. Go team.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/192/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/192/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=192&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2012/05/04/at-long-last-compartment-per-global/feed/</wfw:commentRss>
		<slash:comments>17</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2012/05/computing_stress.jpg?w=497" medium="image">
			<media:title type="html">Compartments and Brain Transplants</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2012/05/rainbow-stream.jpg?w=400" medium="image">
			<media:title type="html">Rendition</media:title>
		</media:content>
	</item>
		<item>
		<title>Typed Arrays Supported in XPConnect</title>
		<link>http://bholley.wordpress.com/2011/12/13/typed-arrays-supported-in-xpconnect/</link>
		<comments>http://bholley.wordpress.com/2011/12/13/typed-arrays-supported-in-xpconnect/#comments</comments>
		<pubDate>Tue, 13 Dec 2011 21:07:30 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=178</guid>
		<description><![CDATA[tl;dr &#8211; You can now pass typed arrays to XPCOM methods that expect XPIDL arrays. These days I work mostly on XPConnect, which serves as the bridge between Spidermonkey and the rest of Gecko. For those not steeped in Mozilla lore, XPConnect has a reputation for being one of the nastiest and most incomprehensible parts [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=178&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>tl;dr &#8211; You can now pass typed arrays to XPCOM methods that expect XPIDL arrays.</p>
<p>These days I work mostly on XPConnect, which serves as the bridge between <a href="https://developer.mozilla.org/en/SpiderMonkey">Spidermonkey</a> and the rest of Gecko. For those not steeped in Mozilla lore, XPConnect has a reputation for being one of the nastiest and most incomprehensible parts of the platform. Unfortunately, it&#8217;s also a central bottleneck through which most important things must pass.</p>
<div id="attachment_181" class="wp-caption alignnone" style="width: 470px"><a href="http://bholley.files.wordpress.com/2011/12/balrog-xpconnect.jpg"><img src="http://bholley.files.wordpress.com/2011/12/balrog-xpconnect.jpg?w=460&#038;h=580" alt="lord of the rings" title="balrog-xpconnect" width="460" height="580" class="size-full wp-image-181" /></a><p class="wp-caption-text">Fig 1: Technical diagram of a method call on an XPCWrappedNative.</p></div>
<p>So while many people desperately need features and bug fixes for XPConnect, very few people have the knowledge and fortitude to hack on the code. Those who do, folks like Blake Kaplan, Boris Zbarsky, and Peter Van Der Beken, tend to be very busy people. </p>
<p>I started working on XPConnect in large part to alleviate this problem. Unfortunately, when word got out that there was a fresh face in the cartel, I too quickly became a very busy person. Thus, when a <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=699156">bug was filed</a> to implement support for passing <a href="https://developer.mozilla.org/en/JavaScript_typed_arrays">typed arrays</a> to XPCOM methods, I expressed skepticism that I&#8217;d have the bandwidth for it any time in the near future.</p>
<p>So imagine my surprise and joy when one of the requesters (Antti Haapala, ztane on IRC) expressed interest in taking a crack at it. And imagine my utter disbelief when a working patch, with tests, appeared a day later:</p>
<p><a href="http://bholley.files.wordpress.com/2011/12/ztane-bugzilla.png"><img src="http://bholley.files.wordpress.com/2011/12/ztane-bugzilla.png?w=460&#038;h=61" alt="screenshot from bugzilla" title="ztane-bugzilla" width="460" height="61" class="alignnone size-full wp-image-179" /></a></p>
<p>A few review iterations later, the patch <a href="https://hg.mozilla.org/mozilla-central/rev/d662c4cfabae">landed on mozilla-central</a>.</p>
<p>The force is strong in this one &#8211; let&#8217;s hope he sticks around.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/178/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/178/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=178&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2011/12/13/typed-arrays-supported-in-xpconnect/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2011/12/balrog-xpconnect.jpg" medium="image">
			<media:title type="html">balrog-xpconnect</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2011/12/ztane-bugzilla.png" medium="image">
			<media:title type="html">ztane-bugzilla</media:title>
		</media:content>
	</item>
		<item>
		<title>Using Git With Mozilla</title>
		<link>http://bholley.wordpress.com/2010/10/23/using-git-with-mozilla/</link>
		<comments>http://bholley.wordpress.com/2010/10/23/using-git-with-mozilla/#comments</comments>
		<pubDate>Sat, 23 Oct 2010 05:41:14 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=91</guid>
		<description><![CDATA[This summer I went on a quest to improve my workflow. I wasn&#8217;t really happy with the standard Mercurial/mq approach used by most Mozilla developers. I spent a while experimenting with alternative ways of using Mercurial, and even did a fair amount of hacking on hg itself to fix some bugs and shortcomings. I wrote [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=91&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>This summer I went on a quest to improve my workflow. I wasn&#8217;t really happy with the standard Mercurial/mq approach used by most Mozilla developers. I spent a while experimenting with alternative ways of using Mercurial, and even did a fair amount of hacking on hg itself to fix some <a href="http://hg.mozilla.org/users/bobbyholley_stanford.edu/hgext/file/8723e06e57c8/peacepipe.py">bugs</a> and <a href="http://hg.mozilla.org/users/bobbyholley_stanford.edu/hgext/file/8723e06e57c8/frk.py">shortcomings</a>. I wrote quite a long blog post about all of this and almost published it, but in the end I decided that it still wasn&#8217;t as good as I&#8217;d like it to be.</p>
<p>To its credit, Mercurial&#8217;s extension model made all this very doable, and I probably could have continued to cobble together a workflow that did what I wanted. However, I was pretty sure that Git did exactly what I wanted out of the box. So I gave it a shot, and it works even better than I&#8217;d hoped.</p>
<h2>Brief Aside &#8211; A Ten-Second Introduction to Git</h2>
<p>Git and Mercurial are quite similar &#8211; both use SHA-1 hashes to identify commits. The primary user-facing difference between Git and Mercurial is that Git branches are extremely lightweight. Git is essentially a user-space filesystem, where each commit is represented as file named by its SHA-1 hash. A branch is nothing more than a smart alias for a hash identifier. So a Git repository consists of 3 primary things (this is a bit of an oversimplification, but it&#8217;s fine for our purposes):</p>
<ol>
<li>An <tt>objects</tt> directory, which contains a soup of commit files, bucketed into sub-directories.</li>
<li>A <tt>refs/heads</tt> directory, which contains one file for each named branch. So if I have a branch called <tt>collectunderpants</tt> whose latest commit is <tt>7bc99958bc164028b94ec47dbf1fb1ad9034c580</tt>, there&#8217;s a file called <tt>refs/heads/collectunderpants</tt> whose contents is simply <tt>7bc99958bc164028b94ec47dbf1fb1ad9034c580</tt>. That&#8217;s all git needs.</li>
<li>A file called <tt>HEAD</tt> containing the name of the current branch. This is important, because when I make a commit, Git needs to know which branch should be scooted forward to point to the new commit.</li>
</ol>
<h2>Getting Started</h2>
<p>Suspend your disbelief for the time being and assume that I have a git repository called <tt>/files/mozilla/link</tt> that contains an up-to-date mirror of mozilla-central in git form (I&#8217;ll explain how this is done later).</p>
<p><code><strong>$</strong> cd /files/mozilla<br />
<strong>$</strong> git clone link src<br />
</code></p>
<p>After a waiting a few moments, I now have a full git repository named <tt>src</tt>. The default branch is <tt>master</tt>, which I can see immediately because of a neat shell prompt trick (works best when put in <tt>~/.profile</tt>):</p>
<p><code><strong>$</strong> export PS1='\u@\h \w$(__git_ps1 " (%s)") $ '<br />
<strong>/files/mozilla/src (master) $</strong> echo w00t!<br />
</code></p>
<p>So I&#8217;m on <tt>master</tt>. Unfortunately, I check TBPL and it looks like the tree is burning as a result of another Jonas Sicking push-and-run. The last green commit was 5 changesets back, so I want to base my work off of that.</p>
<p><code><strong>(master) $</strong> git checkout -b master-stable master~5<br />
<strong>(master-stable) $</strong><br />
</code></p>
<p>This makes a new branch called <tt>master-stable</tt> based 5 commits back from the commit pointed to by <tt>master</tt>, and switches the working directory to it.</p>
<p>I make a <tt>.mozconfig</tt>, set the objdir to <tt>/files/mozilla/build/main</tt>, <tt>make -f client.mk</tt>, and go shoot some nerf darts at dolske. A short while later, I&#8217;ve got a full build waiting for me in <tt>/files/mozilla/build/main</tt>.</p>
<h2>The Basics</h2>
<p>Let&#8217;s run some simple diff queries:</p>
<p><code><strong>(master-stable) $</strong> git diff HEAD # Diffs against the current head<br />
<strong>(master-stable) $</strong> git diff master # Inverse of Sicking's bad push<br />
<strong>(master-stable) $</strong> git diff HEAD^^^ # workdir vs 3-commits-back<br />
<strong>(master-stable) $</strong> git diff HEAD~3 # Same as above<br />
<strong>(master-stable) $</strong> git diff master-stable~3 # Same as above<br />
</code></p>
<p>The ability to reference revisions symbolically (relative to either heads or branches) is really nice, and is something that I missed with Mercurial. <em>Edit: bz points out in the comments that this is actually possible with Mercurial.</em></p>
<p>Now suppose I get an idea for a quick one-off patch, and hack on a few files. To save this work (along with its ancestry), I create a branch off the current head:</p>
<p><code><strong>(master-stable) $</strong>...hack hack hack...<br />
<strong>(master-stable) $</strong> git checkout -b oneoff<br />
<strong>(oneoff) $</strong> git commit -a<br />
</code></p>
<p>The first command creates a new branch called <tt>oneoff</tt> that points to the same commit as <tt>master-stable</tt>. The second creates a new commit containing the changes in the working directory. The reason for the <tt>-a</tt> option has to do with a git feature called the &#8220;index&#8221;, which is a staging area between your working directory and full-blown commits. I don&#8217;t want to digress too much, but you should definitely <a href="http://book.git-scm.com/1_the_git_index.html">read more about it</a>.</p>
<p>Remember that branches are just aliases to SHA-1 identifiers, which in turn are used to locate the actual commit in the soup. So <tt>oneoff</tt> is an alias for a SHA-1 identifier which points to the new commit. That commit knows the hash of its parent, which is the same hash pointed to by <tt>master-stable</tt>. Git commits are immutable, since their names a are cryptographic function of their contents (so if a commit changes, it&#8217;s really just a new commit). Furthermore, git is garbage collected when you call <tt>git gc</tt>. So objects in git are just like immutable objects in a garbage-collected language. For example, suppose we want to modify that commit we just made:</p>
<p><code><strong>(oneoff) $</strong> ...more hacking...<br />
<strong>(oneoff) $</strong> git commit -a --amend<br />
</code></p>
<p>Normally <tt>git commit</tt> makes a new child of the previous commit. However the <tt>--amend</tt> option makes a new sibling that combines the previous commit with any working changes, and points the branch and head to it. The old commit is still there, but is now orphaned, and will be removed in the next call to <tt>git gc</tt>.</p>
<h2>Rebasing</h2>
<p>I use one branch per bug, and one commit per patch. This allows me to model my patches as a DAG, where patches are descendents of work they depend on. Contrast this with the MQ model, where a linear ordering is forced upon possibly unrelated patches.</p>
<p>Suppose I&#8217;m doing some architectural refactoring in a bug called <tt>substrate</tt>, and using the clean new architecture in a feature bug called <tt>bling</tt>. Initially, I start work on <tt>bling</tt> as follows:</p>
<p><code><strong>(substrate) $</strong> git checkout -b bling<br />
<strong>(bling) $</strong> ...hack commit hack commit hack....<br />
</code></p>
<p>But then I think of something else that would be useful for <tt>bling</tt> that should really go in <tt>substrate</tt>. So I <a href="http://www.kernel.org/pub/software/scm/git/docs/git-stash.html">stash</a> away my uncommitted changes, and go add another patch to substrate:</p>
<p><code><strong>(bling) $</strong> git stash<br />
<strong>(bling) $</strong> git checkout substrate<br />
<strong>(substrate) $</strong> ...hack hack...<br />
<strong>(substrate) $</strong> git commit -a<br />
<strong>(substrate) $</strong> git checkout bling<br />
<strong>(bling) $ </strong><br />
</code></p>
<p>At this point, I&#8217;d really like get back to working on <tt>bling</tt>, but unfortunately <tt>bling</tt> isn&#8217;t yet based on the latest patch in <tt>substrate</tt>. To fix this, we need to rebase:</p>
<p><code><strong>(bling) $</strong> git rebase --onto substrate bling~3..bling</code></p>
<p>This tells git to take all the changesets in the range <tt>(bling~3, bling]</tt> and apply them incrementally as commits on top of <tt>substrate</tt>. If there are conflicts, I&#8217;m given the opportunity to resolve them, or to abort the whole endeavor. Once the rebase is complete, the branch <tt>bling</tt> is updated to point to the new, rebased tip. Now I can reapply my work-in-progress and get back to business:</p>
<p><code><strong>(bling) $</strong> git stash pop</code></p>
<h2>Rewriting History</h2>
<p>My code is always perfect the first time I write it, but suppose for the sake of argument that Joe gets a bee in his bonnet and I have to alter patch 7 of 18 in <tt>bigbug</tt> to appease him. I could do it the long way:</p>
<p><code><strong>(bigbug) $</strong> git checkout -b _tmp HEAD~11<br />
<strong>(_tmp) $</strong> ...appease appease...<br />
<strong>(_tmp) $</strong> git commit -a --amend<br />
<strong>(_tmp) $</strong> git rebase --onto _tmp bigbug~11..bigbug<br />
<strong>(_tmp) $</strong> git checkout bigbug<br />
<strong>(bigbug) $</strong> git branch -d _tmp<br />
</code></p>
<p>This gets tedious after a while though. Thankfully, there&#8217;s a better way:</p>
<p><code><strong>(bigbug) $</strong> git rebase --interactive HEAD~12<br />
</code></p>
<p>This fires up an editor, which allows me to select which parts of the history I want to modify:</p>
<p><code> pick f901b35 patch 7<br />
pick 613cb9e patch 8<br />
pick db26bd3 patch 9<br />
....<br />
pick 678b170 patch 18<br />
<br />
# Commands:<br />
#  p, pick = use commit<br />
#  r, reword = use commit, but edit the commit message<br />
#  e, edit = use commit, but stop for amending<br />
#  s, squash = use commit, but meld into previous commit<br />
#  f, fixup = like "squash", but discard this commit's log message<br />
</code></p>
<p>So if I change the <tt>pick</tt> on the first line to <tt>edit</tt> (or just <tt>e</tt>), git brings me to that revision, lets me edit it, and does all the rebasing for me. Huzzah!</p>
<h2>Pushing to Bugzilla</h2>
<p>One nice bonus of git is an add-on developed by Owen Taylor called <a href="http://git.fishsoup.net/cgit/git-bz/">git-bz</a>. I&#8217;ve made some modifications to it to make it more mozilla-friendly, and haven&#8217;t yet found the time to make them upstreamable. So in the mean time, I&#8217;d recommend that you grab my fork, <a href="http://github.com/bholley/git-bz-moz">git-bz-moz</a>.</p>
<p>While it does a lot of things, my favorite part of git-bz is pushing to bugzilla. For credentials, git-bz uses login cookie of your most recently opened Firefox profile &#8211; so if you&#8217;re already logged into BMO things should work seamlessly. Let&#8217;s say I want to attach all 18 patches of <tt>bigbug</tt> to bug 513681. I run:</p>
<p><code><strong> (bigbug) $</strong> git bz attach --no-add-url -e 513681 HEAD~19..HEAD</code></p>
<p>And then I&#8217;m presented with a sequence of 18 files to edit in my editor, each of which looks like the following:</p>
<p><code># Attachment to Bug 513681 - Eliminate repeated code with image decoder superclass<br />
<br />
&lt;Summary&gt;<br />
<br />
&lt;Bug Comment&gt;<br />
<br />
Commit-Message: Bug 513681 - Eliminate repeated code with image decoder superclass.<br />
<br />
#Review: reviewer1@example.com<br />
#Review: reviewer2@example.com<br />
#Review: :me+<br />
#Super-Review: superreviewer@example.com<br />
<br />
#Super-Review: :me+<br />
#Obsoletes: 470931 - patch v2<br />
<br />
# Please edit the description (first line) and comment (other lines). Lines<br />
# starting with '#' will be ignored.  Delete everything to abort.<br />
# To obsolete existing patches, uncomment the appropriate lines.<br />
 </code></p>
<p>This pulls the relevant data from the bug, and let&#8217;s me do a lot in one edit. I can set the patch description, add a comment in the bug, edit the commit message (for facilitating <tt>hg qimport</tt>), obsolete other patches in the bug, flag for review, and grant self-review. I&#8217;ve found this to be a massive timesaver when working on many-part bugs.</p>
<p>When I want to push, I just qimportbz from the bug. This gives me an incentive to make sure that the patches committed are the ones on bugzilla.</p>
<p><em>Aside &#8211; I haven&#8217;t done much active development since the end of august, and git-bz just choked on the cookie database of a recent nightly when I tried it. A 3.6 profile still works fine though.</em><br />
<em>Edit &#8211; dwitte points out in the comments that this is due to a change in the sqlite database format, and should be fixed by upgrading to sqlite 3.7.x.</em></p>
<h2>Multiple Working Directories</h2>
<p>The ability to multitask is crucial to being productive in the Mozilla ecosystem. I can be waiting on tryserver results for one patch, guidance from bz on a second, review from Jeff on a third, and a dependent patch from Joe for a fourth. I need to be able to work on multiple patches at once, and context-switch quickly.</p>
<p>In theory, multitasking with git is quite simple: just do a <tt>git checkout</tt> of the branch you want to work on. However, some code changes require significant rebuilding. For example, if I have a patch that modifies <tt>nsDocument.h</tt>, context-switching between that patch and any other patch incurs a massive recompilation burden.</p>
<p>I&#8217;ve heard through the grape-vine that bz manages this problem by having 8 different mercurial repositories (each with its own object directory), and economizing on space via hardlinks. This eliminates the recompilation burden, but doesn&#8217;t allow work to be easily shared between repositories. For example, I might want to give both <tt>bling</tt> and <tt>substrate</tt> separate object directories, but still be able to rebase <tt>bling</tt> on top of new code in <tt>substrate</tt>.</p>
<p>Thankfully, git allows me to get the best of both worlds with <a href="http://git.kernel.org/?p=git/git.git;a=blob_plain;f=contrib/workdir/git-new-workdir">multiple working directories</a>.</p>
<p><code><strong>/files/mozilla/src (blah) $</strong> mkdir ../proj<br />
<strong>/files/mozilla/src (blah) $</strong> cd ../proj<br />
<strong>/files/mozilla/proj/ $</strong> git-new-workdir ../src a<br />
</code></p>
<p>This gives me a full working directory and a lightweight repository that is composed mostly of symlinks to files in <tt>../src/.git/</tt>. Everything is shared seamlessly between them, and just about the only thing private to the new repository is the HEAD file, which specifies the checked-out branch. I can then make a </tt>.mozconfig</tt> pointing to a new object directory in <tt>/files/mozilla/build/a</tt>, and build away.</p>
<h2>The Link</h2>
<p>Earlier in this post I promised to explain where <tt>/files/mozilla/link</tt> came from.</p>
<p>I initially started using git with a <a href="http://repo.or.cz/w/mozilla-central.git">mirror</a> maintained by Julien Rivaud. Unfortunately, there was some flakiness with the cron job, and the repository would often stop updating from mozilla-central. So I decided to generate my own mirror. <em>Edit: Julien mentions in the comments that the repository should be reliable now. Give it a shot!</em></p>
<p>Long-story short: don't use <a href="http://hg-git.github.com/">hg-git</a>. It chokes miserably on mozilla-central. Instead, use <a href="http://repo.or.cz/w/fast-export.git">hg-fast-export</a>. Let it run overnight, and it should be done in the morning. Incremental updates are also very fast (roughly linear in the number of new commits), so I don't ever find myself waiting for it.</p>
<h2>Other Thoughts</h2>
<ul>
<li>From a general zippiness standpoint, git seems about 5 times faster than Mercurial. Your mileage may vary.</li>
<li>Overall, I really like the garbage-collection model of git. With Mercurial, rewriting history involves stripping entries out of the repository, which can be very slow. With git, unwanted objects go away just by redirecting pointers, and they're still recoverable (with careful munging) until the next <tt>git gc</tt>.</li>
<li>I've found that I'm spending a lot less time dealing with merge conflicts than I did when I was using hg/mq. Git seems to be pretty smart about these things, and I think it uses 3 lines of context internally. In contrast, it's standard to use 8 lines of context for mq patches so they can be easily exported to bugzilla. I've modified <tt>git-bz</tt> to generate 8 lines of context when posting to bugzilla, which allows me to be more efficient locally while still sharing my work in the appropriate format.</li>
</ul>
<p>There's lots more to say about git, but I think that this is enough for now. Share your experiences in the comments!</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/91/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/91/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=91&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2010/10/23/using-git-with-mozilla/feed/</wfw:commentRss>
		<slash:comments>14</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>
	</item>
		<item>
		<title>Rogue Diplomacy</title>
		<link>http://bholley.wordpress.com/2009/08/17/rogue-diplomacy/</link>
		<comments>http://bholley.wordpress.com/2009/08/17/rogue-diplomacy/#comments</comments>
		<pubDate>Mon, 17 Aug 2009 09:02:05 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=55</guid>
		<description><![CDATA[This weekend I had the opportunity to give a talk about the Mozilla community to a pretty diverse audience. The focus of the event was on open/participatory models of cooperation, so I tried to tilt the talk in that direction. The response was pretty amazing. Despite having only one or two programmers in the audience, [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=55&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>This weekend I had the opportunity to give a talk about the Mozilla community to a pretty diverse audience. The focus of the event was on open/participatory models of cooperation, so I tried to tilt the talk in that direction. The response was pretty amazing. Despite having only one or two programmers in the audience, there was a ton of engagement, with lots of questions and discussion. I&#8217;d originally slated the talk to last about an hour, but it ended up taking 2 and a half hours with all of the audience participation. A lot was discussed about whether the &#8220;Mozilla Model&#8221; could be applied to other aspects of society, or whether the forkability and patchability of code (that is to say, the ease of experimenting with mutually exclusive solutions simultaneously) makes software a special case. In the end I think that everybody, including myself, learned a lot.</p>
<p>Note to self &#8211; don&#8217;t recommend Adblock when there might be web publishers in the audience.</p>
<p>I put together some slides (<a href="http://www.stanford.edu/~bh10/bholleymoz.pdf">pdf</a>, <a href="http://www.stanford.edu/~bh10/bholleymoz.key.zip">keynote</a>) for the occasion that I based on <a href="http://beltzner.ca/mike/">Mike Beltzner</a>&#8216;s 2009 intern brownbag slides (<a href="http://people.mozilla.org/~beltzner/presentations/2009-intern-brownbag.key.zip">keynote</a>). I was pretty happy with how they came out, and I&#8217;d encourage anyone interested in talking about Mozilla (including beltzner!) to make use of them.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/55/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/55/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=55&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2009/08/17/rogue-diplomacy/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>
	</item>
		<item>
		<title>All Hail QCMS!</title>
		<link>http://bholley.wordpress.com/2009/06/04/all-hail-qcms/</link>
		<comments>http://bholley.wordpress.com/2009/06/04/all-hail-qcms/#comments</comments>
		<pubDate>Thu, 04 Jun 2009 23:44:39 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=52</guid>
		<description><![CDATA[Sometime in early 2009, it became clear that lcms as a module wasn&#8217;t really working out for us, and we needed to write our own color management implementation (mostly) from scratch. Since I was a full-time student at the time, I didn&#8217;t have it in me to lead the effort and get things done in [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=52&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>Sometime in early 2009, it became clear that lcms as a module wasn&#8217;t really working out for us, and we needed to write our own color management implementation (mostly) from scratch. Since I was a full-time student at the time, I didn&#8217;t have it in me to lead the effort and get things done in time for 3.5. Thankfully, Jeff Muizelaar stepped in and took up the task, resulting in <a href="http://muizelaar.blogspot.com/2009/06/qcms-color-management-for-web.html">qcms</a>. I&#8217;d encourage everyone to go check out his post.</p>
<p>Also, worry not &#8211; most of the work I did on lcms (mainly the performance optimizations) are still there in qcms, so things should be as fast as ever, or faster.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/52/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/52/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=52&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2009/06/04/all-hail-qcms/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>
	</item>
		<item>
		<title>Did You Really Expect Anything Else?</title>
		<link>http://bholley.wordpress.com/2008/12/11/did-you-really-expect-anything-else/</link>
		<comments>http://bholley.wordpress.com/2008/12/11/did-you-really-expect-anything-else/#comments</comments>
		<pubDate>Thu, 11 Dec 2008 21:45:46 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=46</guid>
		<description><![CDATA[So I&#8217;ll just follow the trend here a la sdwilsh and post the wordle representation of my blog. I can&#8217;t say I&#8217;m surprised.<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=46&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>So I&#8217;ll just follow the trend here a la <a href="http://shawnwilsher.com/archives/190">sdwilsh</a> and post the <a href="http://www.wordle.net">wordle</a> representation of my blog.</p>
<div id="attachment_47" class="wp-caption alignnone" style="width: 510px"><img class="size-full wp-image-47" title="wordle-bholley" src="http://bholley.files.wordpress.com/2008/12/wordle-bholley.png?w=460" alt="wordle representation of bholley.wordpress.com"   /><p class="wp-caption-text">wordle representation of bholley.wordpress.com</p></div>
<p>I can&#8217;t say I&#8217;m surprised.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/46/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/46/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=46&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2008/12/11/did-you-really-expect-anything-else/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2008/12/wordle-bholley.png" medium="image">
			<media:title type="html">wordle-bholley</media:title>
		</media:content>
	</item>
		<item>
		<title>Paging Dr. Developer &#8211; The Tree is Burning</title>
		<link>http://bholley.wordpress.com/2008/12/08/paging-dr-developer-the-tree-is-burning/</link>
		<comments>http://bholley.wordpress.com/2008/12/08/paging-dr-developer-the-tree-is-burning/#comments</comments>
		<pubDate>Mon, 08 Dec 2008 03:45:27 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=41</guid>
		<description><![CDATA[Checking code into mozilla-central is a serious time commitment. Among other things, the person pushing the patch must watch the Tinderbox for several hours after committing to make sure that no unforeseen problems show up. This is enough of a pain that many of Mozilla&#8217;s most senior developers prefer to let other people push patches [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=41&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>Checking code into mozilla-central is a serious time commitment. Among other things, the person pushing the patch must watch the <a title="Tinderbox" href="http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox">Tinderbox</a> for several hours after committing to make sure that no unforeseen problems show up. This is enough of a pain that many of Mozilla&#8217;s most senior developers prefer to let other people push patches on their behalf.</p>
<p>It&#8217;s much easier to land a patch after hours, since there are fewer other people landing at the same time and so you&#8217;re less likely to get caught in someone else&#8217;s mess. However, I don&#8217;t always want to spend my evenings refreshing the tinderbox page. As such, I&#8217;ve always dreamed of having a convenient notification system that I could configure to send me an SMS message when the tree changed status. This would mean that I&#8217;d only need to be near a computer, rather than sitting at one, while I was watching the tree.</p>
<p>This evening I was talking to Wolf on IRC and had the idea of piggy-backing on <a title="firebot" href="http://www.psychoticwolf.net/projects/firebot/">firebot</a>&#8216;s tree notification system. Currently, firebot sits on #developers, checking the tree periodically, and speaks up any time the tree changes status. However, there was still quite a ways to go from the IRC channel to my cell phone. I use <a title="Colloquy" href="http://colloquy.info/">Colloquy</a> as my IRC client, which has nice support for the <a title="Growl" href="http://growl.info/">Growl</a> notification framework (all Mac only, sorry). Growl, in turn, has a notification option called &#8220;MailMe&#8221;, which uses your default Mail.app account to send a notification to the address of your choice. Finally, my cellular provider (AT&amp;T) provides a service by which you can email [phone number] AT txt.att.net, and the message will be delivered to that number. So by putting a Colloquy watch-word on &#8220;Success&#8221;, &#8220;Failed&#8221;, and &#8220;Burning&#8221; and setting the Growl notification event JVChatMentioned to use the MailMe plugin, I had firebot sending me text messages. (Side note &#8211;  Growl does in fact have an SMS plugin, but it requires having an account with a commercial service and seems to cost money.)</p>
<p>The MailMe plugin has a slight problem though: it appends a long footer string to every message. This is fine over email, but it caused all my SMS notifications to span multiple messages, which can get pretty expensive if all the boxes start burning at once. Furthermore, this message was hard-coded into the binary, so there was no easy configuration file to modify. As a result, I grabbed the source, hacked it to silence both the footer and the subject, and made a replacement plugin that you can download <a href="http://www.stanford.edu/~bh10/growl-mailme-lite.tar.bz2">here</a>. Just delete the existing MailMe.growlView in /Library/PreferencePanes/Growl.prefPane/Contents/Resources/GrowlHelperApp.app/Contents/PlugIns, drop this one in instead, and you should be good to go. I&#8217;m not sure how portable it is, but I can&#8217;t imagine any problems if you&#8217;re using up-to-date versions of Leapord and Growl.</p>
<p>If you decide to give this a try, let me know how it works out.</p>
<p>Edit: It looks like I forgot to include the source of the (minor) modifications to Growl. I&#8217;ve since deleted the directory I was working in, so I just re-did the changes as I remember them and posted them <a href="http://www.stanford.edu/~bh10/growl-mailme-lite.patch">here</a>. I haven&#8217;t tested the new patch beyond compilation, but it&#8217;s trivial enough that I didn&#8217;t feel like it.</p>
<br />  <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/41/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/41/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=41&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2008/12/08/paging-dr-developer-the-tree-is-burning/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>
	</item>
		<item>
		<title>So Many Colors</title>
		<link>http://bholley.wordpress.com/2008/09/12/so-many-colors/</link>
		<comments>http://bholley.wordpress.com/2008/09/12/so-many-colors/#comments</comments>
		<pubDate>Fri, 12 Sep 2008 04:24:14 +0000</pubDate>
		<dc:creator>bholley</dc:creator>
				<category><![CDATA[Mozilla]]></category>

		<guid isPermaLink="false">http://bholley.wordpress.com/?p=3</guid>
		<description><![CDATA[(A Belorussian translation of this post is available, courtesy of Designcontest) This summer, I&#8217;ve been working on getting Mozilla&#8217;s color management backend ready for the prime time. We&#8217;re finally turning it on in tonight&#8217;s nightly builds, so I thought I&#8217;d give a bit of background on the history of color management in Mozilla and on [...]<img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=3&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></description>
				<content:encoded><![CDATA[<p>(A <a href="http://www.designcontest.com/show/so-many-colors-be">Belorussian translation</a> of this post is available, courtesy of <a href="http://www.designcontest.com/">Designcontest</a>)</p>
<p>This summer, I&#8217;ve been working on getting Mozilla&#8217;s color management backend ready for the prime time. We&#8217;re finally turning it on in tonight&#8217;s nightly builds, so I thought I&#8217;d give a bit of background on the history of color management in Mozilla and on color management in general. You should all go download the latest <a href="http://www.mozilla.org/developer/">nightly</a> when it comes out and give it a try.</p>
<p>In a nutshell, color management involves using information about a computer&#8217;s display, and in some cases information about certain images, to render pages more accurately. Contrary to popular belief, an RGB value (such as #ffffff) alone does not represent a color. Rather, the values for the red, green, and blue channels can be thought of as coordinates in a color space that contains the actual colors. The color that an RGB tuple represents thus depends on how the color space is defined (see <a href="http://en.wikipedia.org/wiki/Color_space">Wikipedia</a> for more information).</p>
<p>This type information is generally stored in a file called an ICC color profile. Computer displays often ship with ICC profiles that specify exactly how various RGB values will look on the screen. Additionally, JPG and PNG images are often tagged with ICC profiles to describe the color space of the pixels. In theory, every image should come with a color profile specifying its color space. However, many images do not contain embedded profiles, nor would it make sense for a CSS stylesheet to embed a color profile for CSS colors. As a result, the W3C determined that untagged web content should be assumed to be in a standard color space called sRGB. So color management in Mozilla is really a two-part process: we map incoming RGB values out of their respective color spaces, and into the color space of your display (which we obtain by querying the operating system).</p>
<div align="center">
<div id="attachment_10" class="wp-caption aligncenter" style="width: 281px"><a href="http://bholley.files.wordpress.com/2008/09/ciexy1931_ciergb.png"><img class="size-medium wp-image-10" title="ciexy1931_ciergb" src="http://bholley.files.wordpress.com/2008/09/ciexy1931_ciergb.png?w=271&#038;h=300" alt="Colors are more complicated than you might think." width="271" height="300" align="center" /></a><p class="wp-caption-text">Colors are more complicated than you might think.</p></div>
</div>
<p>About a year ago, Mike Beltzner unveiled Mozilla&#8217;s color management support <a href="http://www.beltzner.ca/mike/archives/2007/08/do-you-like-the-pretty.html">in a blog post</a>. Unfortunately, there were some major issues that kept the color management mode set squarely to &#8216;off&#8217; for the release of Firefox 3.0.</p>
<p>First of all, there were quite a few rendering bugs filed for our color management implementation. I was able to confirm a good number of them, and discover a few more myself, and I&#8217;ve landed the necessary fixes. There were also quite a few issues with our Reftest rendering test suite that needed fixing so that the tests would pass with color management on. These issues have also been fixed. However, several people reported bizarre rendering behavior (whites showing up yellowish, blues showing up purple) that turned out to be the result of bogus display profiles on the reporter&#8217;s system (many thanks to <a href="http://www.colorremedies.com/">Chris Murphy</a> for lending his expertise on this issue). This is scary, because it raises the possibility that any number of people might see inexplicably weird colors in their browser once we turn on color management. As such, we&#8217;re hoping to get as many testers as possible in the nightlies and betas to see how widespread the problem is.</p>
<p>Another concern we had over color management had to do with plugins. Since plugins like Flash and Java do their own drawing, we don&#8217;t have any control over whether they use color management. While an extension to the plugin API to allow this is in the works, there&#8217;s currently no way for the browser and plugins to coordinate on color management. If the browser color corrects a color while the plugin doesn&#8217;t, they no longer look exactly the same side by side. At this point, it&#8217;s unclear exactly how much of a problem this is, but we&#8217;d appreciate feedback.</p>
<p>However, the biggest reason color management was disabled for Firefox 3.0 was that it regressed performance by about 20-30% (according to our <a href="http://quality.mozilla.org/projects/automation/talos">Talos tests</a>) when switched on. This was deemed unacceptable for a default setting. As such, I spent a great deal of time profiling code and then devising, implementing, and testing dozens of strategies to squeeze the performance overhead as close to zero as possible.</p>
<p>This turned out to be really hard. Color management is a per-pixel transform, meaning that we need to call a function for every pixel of an image (CSS colors are easier since repeated pixels are known to the rendering code). Such an embarrassingly parallel problem would be a perfect candidate for GPU acceleration, but we&#8217;re just beginning to draw up plans for OpenGL and Direct3D backends to Cairo, so it will be quite a while before we can just write a custom shader and make the entire problem disappear. As a result, I dug in my heels and worked on making the transformation of one pixel really, really fast. After some rearchitecting, various precaching strategies, and a whole lot of hand-tuned SSE2 assembly, the performance overhead of full-page color management is finally at about 4-5% on all platforms.</p>
<p>For the past few months, this seemed to be enough. Beltzner was receptive to the idea of turning it on, and nobody in either of the development meetings raised any gripes on the issue. Robert O&#8217;Callahan and David Baron had voiced some concerns about standards compliance and interoperability, but both were willing to turn it on for Firefox 3.1 Beta 1 in order to get feedback from the community. However, as Vlad and I were landing the final patch to turn color management on, we were beset by a flurry of naysayers who didn&#8217;t feel like the performance hit was worth it.</p>
<p>As a contingency, I had previously implemented a third mode for color management, one that color-corrects tagged images only, that has a performance hit of about 1%. Safari has been color-correcting tagged images for a while now, and this is certainly the dominant use case for color management. Given the pushback on full color management, it looks like we&#8217;ll have to settle for a default of &#8220;tagged only&#8221; for now. The capability for full correction is available in Firefox, and hopefully we&#8217;ll have an easy way to switch modes by the time 3.1 rolls out the door. For now, you can go to about:config and set gfx.color_management.mode to &#8217;0&#8242; for no color management, &#8217;1&#8242; for full color management, and &#8217;2&#8242; (the default) for tagged-only correction (for technical reasons, you&#8217;ll need to restart your browser for this setting to take effect). Deb Richardson put together a <a href="http://www.dria.org/wordpress/archives/2008/04/29/633/">great blog post</a> where she highlighted some of the differences between color-managing and not color-managing tagged images. I&#8217;d encourage you to check it out.</p>
<div align="center">
<div id="attachment_12" class="wp-caption aligncenter" style="width: 352px"><a href="http://bholley.files.wordpress.com/2008/09/deb_cms_comparison.png"><img class="size-full wp-image-12" title="deb_cms_comparison" src="http://bholley.files.wordpress.com/2008/09/deb_cms_comparison.png?w=460" alt="Color Management Off, Color Management On"   align="center" /></a><p class="wp-caption-text">Color Management Off, Color Management On   (Courtesy of Deb)</p></div>
</div>
<p>Personal disappointment aside, I think it would be a good idea to open up discussion on the subject of full color management on the web. In a world where every browser supported color management and nobody had bogus display profiles (idealistic, I know), a web designer could pick a color scheme for his or her web page and rest assured that it would actually look the same for each viewer. Whites would really be white, and that soothing hue of green would look just as good on your mother&#8217;s laptop as it does on your desktop. I&#8217;d encourage people to try browsing the web with gfx.color_management.mode=1. Is the performance impact noticeable? Do you run across any web pages that don&#8217;t look right? How many color-matched plugins do you see on a daily basis? Do you care about color correction? As always, Firefox is <em>your</em> browser, and we want to know what <em>you</em> want. So tell us.</p>
<br /><img alt="" border="0" src="http://feeds.wordpress.com/1.0/categories/bholley.wordpress.com/3/" /> <img alt="" border="0" src="http://feeds.wordpress.com/1.0/tags/bholley.wordpress.com/3/" /> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/bholley.wordpress.com/3/"><img alt="" border="0" src="http://feeds.wordpress.com/1.0/comments/bholley.wordpress.com/3/" /></a> <img alt="" border="0" src="http://stats.wordpress.com/b.gif?host=bholley.wordpress.com&#038;blog=4081394&#038;post=3&#038;subd=bholley&#038;ref=&#038;feed=1" width="1" height="1" />]]></content:encoded>
			<wfw:commentRss>http://bholley.wordpress.com/2008/09/12/so-many-colors/feed/</wfw:commentRss>
		<slash:comments>66</slash:comments>
	
		<media:content url="http://2.gravatar.com/avatar/5f42f7935fa8fd0a80c14bf5c8c38856?s=96&#38;d=http%3A%2F%2F2.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96" medium="image">
			<media:title type="html">bholley</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2008/09/ciexy1931_ciergb.png?w=271" medium="image">
			<media:title type="html">ciexy1931_ciergb</media:title>
		</media:content>

		<media:content url="http://bholley.files.wordpress.com/2008/09/deb_cms_comparison.png" medium="image">
			<media:title type="html">deb_cms_comparison</media:title>
		</media:content>
	</item>
	</channel>
</rss>
