[PNG icon] PNG Support in Mozilla

(Last update: 1999-05-06 )

Intro

This page currently contains Greg's (incomplete) notes on the basic flow of control through Mozilla's ImageLib and various front ends when decoding and displaying a PNG image, with special attention to alpha-channel handling. Much of it still applies to Classic Mozilla, but Greg is currently updating it for SeaMonkey/Gecko (a.k.a. proto-Navigator 5.0, basically rewritten from the ground up).

Classic Mozilla didn't do any compositing on its own; it left that to various platform-dependent API calls in the front ends, which is why it was limited to binary transparency. (For example, X does not provide any APIs for dealing with an alpha channel, only bitmasks.) SeaMonkey includes the new layout engine (at one time called Raptor) that supposedly allows an image-rendering component to figure out what's beneath the image frame and do proper alpha-blending. Certainly this appears to be the case in Demo 10 bundled with SeaMonkey, wherein a partially transparent animated GIF (that is, using both GIF binary transparency and CSS partial transparency) is displayed correctly on top of an animated background image.

[rewriting this section...pardon the mess]

Update 1999-03-06:   Change is in the air! Greg finally finished his book and is currently in the process of getting a working SeaMonkey build under control (Slackware Linux, kernel 2.0.36, libc 5.4.44, autoconf 2.13, automake 1.4, libtool 1.2, glib 1.2.0, gtk+ 1.2.0, gcc 2.7.2.3). Some big-time house-cleaning in PNGland is next (including better gamma support), followed by the long-awaited alpha support. Stay tuned (i.e., see Build Notes below).

A direct consequence is that progressive, replicating display of an interlaced PNG with alpha is impossible without a large overhead in memory usage, which seems to be highly frowned upon; sparse display will probably be necessary, at least as an initial approach. The new layout engine (originally called Raptor, then NGLayout; Netscape's early beta release was called Gecko) is now in place, and Lee Crocker says that retrieving the background pixels directly should be possible, eliminating the need for extra buffers. But Greg still has to figure out how all of the new stuff works (and compile it, for that matter).

Apparently there are Macintosh (CopyDeepMask()) and Windows (DirectDraw) APIs to handle full alpha, but in X it will probably be necessary to do it in the Mozilla code itself (e.g., XGetImage(), do software compositing, then XPutImage() as usual). Ideally this would be dealt with in device-independent RGB space, but at least the pre-Raptor X code looked to be severely device-dependent. Bleah.

In addition, Pamela Nunn (ImageLib owner) had the following comments on 17 April 1998:

There are a few things in the current code base that I should mention to you. They could be very confusing if you didn't know what I was planning. If you have a better solution to the problem, it would be great.

1) plain old bug: the image mask used for alpha channel is not resized properly for interlaced png's.

2) The need of a double buffer for interlaced png's was a bit of a problem. We have to cut down on memory use whenever possible.

I planned to work out another mask scheme so the correct portions of the progressive images would overlay the previously displayed image. I've not had time to implement this.

Also, Greg had a minor revelation while standing in line for the Future of Linux panel discussion on 14 July 1998: as a stopgap measure, why not simply replace the current alpha-to-bitmask conversion code with a ``smart'' version that uses Floyd-Steinburg dithering (or some other error-diffusion method)? The result is still a bitmask, but it's a bitmask that simulates an alpha channel halfway decently, at least allowing the general flavor of RGBA images to shine through. And since it involves absolutely no changes to the front ends, it's something that can possibly be integrated into the Communicator tree prior to 5.0. Greg will be looking into this as soon as the book is done.

See http://www.mozilla.org/ for pointers to the full Mozilla source code and documentation, particularly for ImageLib. All of the code is web-accessible from http://lxr.mozilla.org/; the core browser itself (SeaMonkey) is at http://lxr.mozilla.org/seamonkey/ . Also check the Mozilla owners list for contact information on PNG and related functions in Mozilla. Most PNG-related discussion is likely to occur on the png-implement mailing list, at least in the beginning; see the Miscellaneous Links page for subscription info. Note that the old LXR site (http://cvs-mirror.mozilla.org/webtools/lxr/) is no longer kept up to date, at least not at its top-level page.

Finally, here are some convenient links relevant to the new code, also courtesy of Pamela:

And here are some of the reported bugs that are more or less related to PNG support:

Update 1999-04-24:   Pamela's new ImageLib branch (IMGLIB_DECODERS_BRANCH) was checked into the main tree on 22 April, so you can now browse it directly via LXR:

Build Notes

System configuration:

Built and installed autoconf, automake, libtool in /usr/local/bin and /usr/local/lib. With previously installed libnspr21 (24 Dec 1998) in /usr/local/experimental, attempted to get glib 1.2.0 configuration to detect and use NSPR threads; only detected if patched configure (LDFLAGS and DEFS modified for /usr/local/experimental directory), but compilation failed anyway (GLOCK(GModule) macro -> "g__GModule_lock undeclared"). Reverted to stock configure, built without threads, and installed in /usr/local/experimental:

For gtk+ 1.2.0, did following:

For Mozilla, began with source archive from 4 March 1999 and basically followed Unix build instructions:

This got as far as the "libs" target in layout/html/forms/src before barfing on nsFormControlHelper.cpp. Apparently did the CVS checkout while someone else was busy checking into that directory; fixed by visiting Bonsai and Tinderbox and doing an updated checkout early on 7 March 1999. (Also archived and nuked all of the CVS '?' files except for depend.mk and *.OBJ directories.)

This time the compile (still "libs" target) barfed in editor/base on nsTextEditor.cpp. No clue what's wrong, but the editor is unnecessary, so reconfigured without it:

Yet another "libs" failure, now in xpfe/AppCores/src on nsBaseAppCore.cpp. Again, no clue (random C++ barfage apparently related to forward-referencing an incomplete class)...is XPFE really needed with a GTK-based front end? Ah, Sir Ramiro saves the day. Continued by doing the following:

This time the build succeeded. Wrapped dist/bin/viewer in a small shell script that sets LD_LIBRARY_PATH and tried it on a local HTML file; it popped up a gray window, said ``Going to create the event queue'' and segfaulted with an 11 MB core dump.

Update 1999-03-11:   Pulled updated sources this morning after last night's report by Andrew Volkov that the long-standing OBJECT bug is finally fixed (that is, supposedly Mozilla will now render OBJECT PNGs, JPEGs and GIFs directly without looking for a plug-in or messing with the contents of the OBJECT container). The tree was a lovely shade of green at the time, which seemed a good portent.

The build succeeded with no further tweaking (whew, about time). Running viewer resulted in the following output before dying with a 3 MB core dump due to a charset assertion failure in htmlparser/src/nsScanner.cpp: 

nsComponentManager: Creating Directory /home/roelofs/.mozilla
Going to create the event queue
font refcount: 3
font refcount: 4
font refcount: 5
Using '/util/www/mozilla/dist/bin' as the resource: base
font refcount: 6
Got thew event queue from the service
Calling gdk_input with event queue
Abort (core dumped)

While this was printing to the console, a window popped up and even displayed four menu items (File, Edit, Debug, Tools), two buttons (Forward, Back), and a text-entry window (resource:/res/samples/test0.html), but it vanished almost immediately.

Running apprunner, on the other hand, produced the following console output and then stayed running, albeit with a completely blank window: 

Argument: -progname, ****** Value: /util/www/mozilla/dist/bin/apprunner
iconic  state not set
width was not set
height was not set
Using '/util/www/mozilla/dist/bin' as the resource: base
Got thew event queue from the service
Calling gdk_input with event queue
Reading file...
DocLoaderFactory: Unable to create ContentViewer for content-type: text/xul
OnConnectionsComplete

Giving it an argument (the local URL to this file, which is about as vanilla as HTML gets) resulted in the following: 

apprunner file:///home/roelofs/WWW/pngmoz.html
Argument: -progname, ****** Value: /util/www/mozilla/dist/bin/apprunner
Argument: -url, ****** Value: file:///home/roelofs/WWW/pngmoz.html
URL to load is file:///home/roelofs/WWW/pngmoz.html
iconic  state not set
width was not set
height was not set
Got thew event queue from the service
Calling gdk_input with event queue
Using '/util/www/mozilla/dist/bin' as the resource: base
Reading file...
Abort (core dumped)

The same thing occurred with the remote copy of the page (http://www.cdrom.com/pub/png/pngmoz.html, presumably what you are reading right now) and with the Mozilla home page. No time for gdb on apprunner yet, but soon...

Update 1999-03-14:   Again pulled updated sources after noticing that core dump probably had to do with XPCOM, which has been under active development lately:

Fired up viewer with no arguments once more; by golly, it runs! Even better, so does apprunner, which is more or less equivalent to the ``Gecko'' browser Netscape distributed in December 1998. Here are some small screenshots of apprunner rendering the image-PNGs test page and two sections of the object-PNGs test page:


Click to see the full-sized versions.

Note that the doubled penguins (middle screenshot) and paired PNG/JPEG icons (righthand screenshot) are not correct; only the left image in each case should be visible. Greg reopened the original OBJECT bug.

But OBJECTs notwithstanding, a working browser means that Greg can finally get started on improving Mozilla's PNG support, you betcha!

Update 1999-04-06:   The OBJECT bug (1945, above, and also 3692 and 4441) reportedly is fixed. Greg will update his build tonight and see how it looks.

Update 1999-04-08:   After a big CVS pull (around an hour, give or take; modem died in the middle of the first try), an interesting gcc bug (4631), a failed ``make depend'' build, a ``make clobber'' and 94 minutes of non-mailnews rebuilding (222 MB of object files!), Greg finally got a working apprunner again. And lo and behold! The OBJECT bug is indeed eradicated at last--or at least until Greg figures out an even more twisted test case. Hallelujah, etc., etc., and big thanks to both Rick Gessner and Andrew Volkov.


Click to see the full-sized versions.

There is, however, a bug in apprunner's rendering of the nested lists on the PNG home page; this can be seen in the last image, where several lines suddenly unindent all the way to the left margin (bug 5460).

Update 1999-04-17:   Grabbed libIDL 0.6.5 (now required) from ftp://ftp.mozilla.org/pub/mozilla/libraries/source/, compiled and barfed on old version of makeinfo (1.63). Grabbed texinfo 3.12 from ftp://ftp.gnu.org/pub/gnu/texinfo/, compiled and installed, then recompiled libIDL and installed in /usr/local/experimental (as with glib, gtk+ and libnspr3).

More important, Greg finally tracked down source of the PNG interlace bug in Mozilla; it stems from ImageLib modifying a libpng row buffer in place (by stripping the alpha bytes out of an RGBA row--this happens in il_emit_row() in scale.cpp). That's bad, because libpng returns the same buffer for replicated lines in an interlaced image--for example, the first row is returned a total of 8 times in the first interlaced pass, and the 4:3 ``compression'' gets repeated 7 times even though the alpha values are gone after the first round. The fix is to allocate a couple of working row buffers in the PNG decoder (as is done for GIF), but that can wait until after the new ImageLib code gets checked in.

Update 1999-04-23:   Pulled a new tree, including ImageLib 2.0, and did a full clobber build (102 minutes). Apprunner can't find any libjpeg and libpng functions even though both libraries are statically linked in; as a result, it won't display any JPEG or PNG images (bug 5459). This seems to be a problem in the dynamic registry/XPCOM stuff.

Update 1999-04-30:   Boy, bug 5459 caused many headaches this week. The original problem was fixed by some minor modifications to Makefile.in in the modules/libimg/pngcom and modules/libimg/jpgcom directories, and for Red Hat / glibc-based folks, that appears to have fixed everything. But Greg's libc5-based system still won't touch his system libpng.so.2, even though that's what it links libnspng.so against. PNG images simply don't get rendered. Using the Mozilla-built copy of libpng.so (linked as libpng.so.2) works just fine, however. Maybe this is related to the fact that the system library is an optimized build while the Mozilla-built one is a debug version? To top it off, ldd 1.9.5 doesn't work on shared objects even though ld.so 1.9.5 will load them for any app but Mozilla.

And regardless of which libpng is installed, OBJECT images of any flavor fail to display (bug 5829).

Update 1999-05-01:   OK, headaches finally eliminated. Mozilla always compiles with its own png.h and pngconf.h even when it links against a system libpng.so (bug 5842). On top of that, Mozilla's pngconf.h is from libpng 0.95 even though the rest of its libpng files are from 1.0.2 (bug 5841), and this is specifically what causes the incompatibility with the system libpng.so.2 on Greg's system.

Update 1999-05-06:   Oops. The problem of Mozilla compiling with its own PNG headers (bug 5842) turns out to have been pilot error. You have to do "make clobber_all" after switching from Mozilla libpng to system libpng; "make clobber" doesn't remove the symlinks in dist/include (or anything under dist, for that matter). Greg did a clobber_all build with no other changes, and indeed the system PNG header files were used (along with the system PNG library). Nothing broke, either. Bug 5842 is therefore closed out as ``resolved/invalid.''

Relevant Source Files

Note: Much of this is out of date at the moment; it applied to MozillaClassic, not SeaMonkey. Front-end stuff that used to live under cmd now lives under gfx/src. 

modules/libimg/png/				libpng 1.0.2 (originally 0.95b)
modules/libimg/src/				main ImageLib routines (generic)

modules/libimg/public/ni_pixmp.h		NI_TrueColor type, etc.
modules/libimg/public/MIMGCB.h			IMGCB callback macros
modules/libimg/public/MIMGCBIF.h		IMGCBIF callback macros (?)



modules/libimg/src/png_png.cpp			similar to libpng's example.c

    process_data ()	[ not used; declared static, defined non-static ]

    info_callback ()
      [ various libpng set-transformation calls ]
      png_read_update_info ()
      png_set_dims (png_ptr->io_ptr, png_ptr)

    il_create_alpha_mask (ic, xoffset, w, h)	[ called by png_set_dims() ]
      [ ic->mask = XP_NEW_ZAP(IL_Pixmap) ]
      [ ic->mask->header->color_space = IL_CreateGreyScaleColorSpace(1, 1); ]
      [ sets width and height of mask ]
      [ widthBytes hardcoded to assume *bit* mask, not alpha mask ]

    row_callback ()
      [ sets ic but doesn't use it ]
      il_emit_row (png_ptr->io_ptr, 0, new_row, 0, png_ptr->width, row_num,
                   1, ilErase, png_ptr->pass)		[ scale.cpp ]




modules/libimg/src/ipng.cpp			ImageLib interface to libpng

    il_png_init (ic)
      [ ic->src_header->color_space->type = NI_TrueColor ]
      [ ic->src_header->color_space->pixmap_depth = 24 ]

    il_png_write (ic, buf, len)	[ called by IL_StreamWrite() in if.cpp ]
      png_set_progressive_read_fn (png_ptr, buf, info_callback, row_callback,
        end_callback)		[ first time only ]
      [ png_ptr->io_ptr = ic ]
      png_process_data (png_ptr, info_ptr, buf, len)

    png_set_dims (ic, png_ptr)	[ called by info_callback() in png_png.cpp ]
      [ should be named il_png_set_dims() ]
      [ possible bug:  sets width, height based on png_ptr values ]
      [ possible bug with gray+alpha images (channels == 2) ]
      [ uses both png_ptr->io_ptr and ic:  identical ]
      il_create_alpha_mask (io_ptr, 0, width, height)	[ png_png.cpp ]
      il_init_image_transparent_pixel (ic)		[ if.cpp ]
      il_size (ic)					[ if.cpp ]
        [ this is where the max-image-size limitation lives (8000 pixels) ]
      il_setup_color_space_converter (ic)		[ color.cpp ]




modules/libimg/src/scale.cpp			most of the good stuff!

    il_timeout_callback (closure)	[ closure == ic, apparently ]
      il_flush_image_data (ic)
      FE_SetTimeout(il_timeout_callback, ic, delay)

    il_partial (ic, row, row_count, pass)
      [ ic->multi:  multi-image GIF, for example? ]
      [ if no delay ] il_flush_image_data (ic)
      FE_SetTimeout (il_timeout_callback, ic, ROW_OUTPUT_INITIAL_DELAY)

    il_flush_image_data (ic)
      IMGCBIF_UpdatePixmap(ic->img_cx->img_cb, ic->img_cx->dpy_cx, image, 0,
                           row, img_header->width, end_row - row + 1);
      IMGCBIF_UpdatePixmap(ic->img_cx->img_cb, ic->img_cx->dpy_cx, mask, 0,
                           row, mask_header->width, end_row - row + 1)
      [ IMGCBIF_UpdatePixmap() is defined in modules/libimg/public/MIMGCBIF.h as
        (effectively) ic->img_cx->img_cb->vtable->UpdatePixmap, which points at
        _IMGCB_UpdatePixmap() in:
		cmd/xfe/images.c		(Unix/X)
		cmd/macfe/central/mimages.cp	(Macintosh)
		cmd/winfe/feimage.cpp		(Windows)
      ]
      il_pixmap_update_notify (ic)			[ if.cpp ]
      il_progress_notify (ic)				[ if.cpp ]

    il_scale_RGB_row (src, src_len, dest, dest_len)
      [ scales row of RGB pixels ]

    il_scale_CI_row (5 args..., transparent_pixel_color)
      [ scales row of indexed-color pixels ]

    il_alpha_mask (many args)
      [ this is where the alpha-dither-to-bitmask trick goes ]

    il_generate_scaled_transparency_mask (many args)
      [ used for GIF (binary) transparency; also PNG binary transparency? ]

    il_reset_background_pixels (5 args)
      [ apparently for dithered pseudocolor displays only ]

    il_generate_byte_mask (5 args)
      [ for pseudocolor displays only? ]

    il_overlay (5 args)

    il_emit_row (many args)	[ called by row_callback(); 400 lines ]
      il_generate_scaled_transparency_mask (many args)	[for GIFs]
      il_alpha_mask (many args)				[for alpha PNGs]
      [ this is where the interlace bug lives ]
      [ tons of other stuff ]
      il_partial (ic, drow_start, row_count, pass)




modules/libimg/src/if.cpp

    IL_StreamWrite (ic, str, len)	[ called by something in netlib? ]
      il_image_stopped (ic)
      (*ic->write) (ic, str, len)	[ il_png_write(), etc. ]
      il_progress_notify (ic)

    IL_StreamFirstWrite (ic, str, len)
      il_type (ic->type, str, len)	[possibly override MIME type ]
      [ ic->write = il_png_write() or il_jpeg_write() or ... ]



gfx/src/nsImageRenderer.cpp
gfx/src/nsImageRequest.cpp
gfx/src/nsBlender.cpp



gfx/src/gtk/nsImageGTK.cpp



gfx/src/windows/nsImageWin.cpp



gfx/src/mac/nsImageMac.cpp

Obsolete Source Files 

cmd/xfe/images.c

    _IMGCBIF_NewPixmap ()
      [ also hardwired to assume mask_depth == 1 ]

    _IMGCBIF_UpdatePixmap ()
      XPutImage ()

    _IMGCBIF_DestroyPixmap ()

    _IMGCBIF_DisplayPixmap ()	[ 150 lines ]
      fe_DrawMaskedImageWithClipRegion (dpy, drawable, img_x_pixmap,
        mask_x_pixmap, width, height, img_x_offset, img_y_offset,
        x_offset, y_offset, fe_drawable->clip_region);

    fe_DrawMaskedImageWithClipRegion ()
      FE_ForEachRectInRegion (clip_region, fe_FillPixmapRectFunc, &fill_closure)

    fe_FillTiledPixmapRectFunc ()
      XP_OffsetRect ()
      XP_IntersectRect ()
      XFillRectangle ()




cmd/xfe/region.c

    FE_ForEachRectInRegion (region, func, closure)
      func (closure, &rect)	[ fe_FillPixmapRectFunc() in images.c ]




cmd/winfe/feimage.cpp

    _IMGCBIF_NewPixmap ()

    _IMGCBIF_UpdatePixmap ()

    _IMGCBIF_DestroyPixmap ()

    _IMGCBIF_DisplayPixmap ()
      displayContext->fe.cx->DisplayPixmap ()		[ cxdc1.cpp ? ]




cmd/winfe/cxdc1.cpp

    CDCCX::DisplayPixmap()
      [ appears to be actual compositing code ]
      MaskBlt ()					[ Windows call? ]




cmd/macfe/central/mimages.cp

    _IMGCBIF_NewPixmap ()

    _IMGCBIF_UpdatePixmap ()

    _IMGCBIF_DestroyPixmap ()

    _IMGCBIF_DisplayPixmap ()
      [ no transparency support at all?? ]

Here are the other PNG-related pages at this site:

Last modified 6 May 1999 by newt@pobox.com , you betcha.

Copyright © 1998-1999 Greg Roelofs.