As part of the 31st Picture Coding Symposium, to be held in Cairns Australia, from 31 May to 3 June, an evaluation of existing and future image compression technologies will be conducted. This activity will include formal evaluation of contributions in an independent and objective manner and the results of the evaluation will be reported both as part of the PCS symposium record and as input documents to standardization bodies.
Contributions to this evaluation event are expected to include existing standardized technologies as baselines, but PCS is particularly keen to receive image compression technologies that are new and not currently standardized. The mix of new and standardized technologies will allow the outcomes of this exercise to have an impact on future standardization activities and technology adoption by industry. Contributors are not required to be the original developers of a technology, but are expected to represent the technology fairly, consulting with the original developers as fully as possible and clearly indicate authorship, sources and references.
While this evaluation exercise is independent of any standardization body, it should be noted that the ISO working group JTC 1/SC 29/WG 1, known as JPEG, has an open Call for Information (CfI) on new image coding technologies and is looking forward to input at its 69th meeting in Warsaw, from June 22-26, 2015. The JPEG group has asked that it be allowed to use the output materials from this PCS exercise as input to its CfI. Contributors of new image coding technologies may find the PCS evaluation to be a convenient and robust mechanism for promoting new approaches. Any response to the CfI or future standardization will be authored by the Chair of this evaluation event and contributors to the exercise with coordination provided by the Chair.
Contributions – Input Required
Contributions should provide an explanation of the basis of the compression algorithm. However, compression performance alone is not the sole feature of interest for practical deployment of an image compression technology. For this reason, contributors are asked to identify how the following features are, or might be, addressed by the contributed technology:
- Spatial accessibility (regions of interest)
- Resolution accessibility (access at different scales)
- Amenability to subjective optimization (pre- or post-compression)
- Ability to address content with varied precision, larger numbers of colour components, Wide gamut colour space capability or larger resolutions than those explicitly tested, including the ability to compress high dynamic range content.
Additionally, contributors are asked to comment on the complexity of the scheme, in the following terms:
- Memory complexity
- Computational complexity
- Software complexity, including potential execution speed indicating the used platform and conditions
- Hardware complexity, to the extent that specific information may be available
- Asymmetry between encoding and decoding
In addition to coding of the mandatory set of images, each contributor is invited to submit a maximum of four (4) of their own test images that could serve to enhance the exercise. To facilitate this, the following requirements are imposed:
- Contributed test images should be formatted as 8- or 12-bit per sample uncompressed TIFF files, as sRGB colour images. Greyscale images can be accommodated, as sRGB images with all three components equal.
- Capture conditions for the contributed test images (device and lens settings) should be documented.
- Contributed test images should not previously have been compressed using any lossy coding scheme.
All contributed test images should be free from any legal encumbrances that would prevent their public use in academic or commercial research and evaluation exercises and a statement to this effect must be made available. The “Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)” creative commons license found at http://creativecommons.org.au would be a suitable license under which to contribute images to this evaluation process.
Submission of Contributions: Detailed Information
- The evaluation event will take the form of a PCS Special Session. Contributions to this Special Session will be reviewed and potentially published as part of the proceedings of symposium. Moreover, the usual no-show guidelines apply to contributions, meaning that contributors are expected to represent their submissions in person at the symposium.
- All contributors are expected to produce a paper, formatted in accordance with the paper template for PCS’2015, to be uploaded to the site specified on registration of intent with their final submission. The document should provide a description of the technology, specific features of the implementation tested, and address the complexity and features of the contributed technology, citing existing public-domain literature wherever appropriate. This paper will be reviewed alongside the evaluation process and changes may be required. PCS reserves the right to reject papers and contributions based on reviewer reports.
- All contributors are expected to provide executable compression (encoding) and decompression (decoding) tools that are capable of accepting 8- or 12-bit uncompressed TIFF files as input (compression) or generating them as output (decompression). These tools should be command-line executables capable of running on 64-bit Linux platforms, 64-bit Windows platforms, or both. An execution script should be provided that demonstrates the way in which the tools should be invoked to compress and decompress images.
- Contributors are at liberty to upload their own compressed files, for the standard image test corpus, but they may also leave the upload folder unpopulated, in which case the uploaded compression tool will be used to populate the folder during testing. Contributors should, however, note that test images submitted by other contributors will also be compressed and decompressed using their tools during testing.
- Since subjective and objective testing will be performed, it is acceptable for contributors to provide two different encoding tools, or two different sets of encoding parameters: one to compress for objective evaluation; and one to compress for subjective evaluation. The upload site will contain separate folders for each case, but if only one folder is populated with a compression tool and sample compression script, the results will be used for both objective and subjective testing.
- Full instructions on access to tools, datasets and the submission of final contributions will be provided to those who indicate intent to contribute by the stated deadline – see below.
The formal assessment process to be undertaken in this evaluation exercise is concerned only with compression performance. In particular, a corpus of images should be compressed at a variety of different compressed bit-rates (bits per pixel) and the reconstructed images will be evaluated with respect to a set of distortion measures. Distortion will be measured as follows:
- Objective measures:
- Mean Squared Error (expressed as PSNR)
- Multi-scale SSIM
- Subjective assessment:
- Mean Opinion Scores
Objective measures will be defined by scripts made available to those contributors who register interest. It should be noted that PSNR calculation will not be weighted or converted between color spaces.
For the purposes of the assessment, contributors should note that:
- The encoding algorithm should be prepared to receive one (only!) free parameter that balances between quality and rate in an implementation-defined way. No other free parameters are allowed. The evaluation process will use this parameter to compress input images to a set of pre-defined bit-rates from 0.1 bpp to 2bpp (bits per pixel).
- The evaluation process may need to crop and/or clip the images to make them suitable for subjective evaluation. This requires contributors to provide a suitable derivative license e.g. (CC BY-NC-SA 4.0) for any content contributed.
Test imagery for the evaluation will be drawn from a sizeable repository that is maintained by JPEG and will be made available to contributors via the PCS website. The test set will be limited to 24bpp RGB photographic content.
Venue and Dates
The evaluation special session will take place on the afternoon of June 3, 2015 and be run as part of the:
2015 Picture Coding Symposium (31 May – 3 June)
Pullman Reef Hotel Casino,
Contributors are expected to represent their contributions in person, by registering for PCS’2015, and making themselves available to both present their contribution and participate in the discussion that will form an integral part of the evaluation exercise.
Meeting Format and Intended Outcomes
The evaluation exercise is expected to involve the following elements:
- A panel formed from key representatives from industry and academia, chaired by Prof. Ian S. Burnett, Dean of the Faculty of Engineering and Information Technology, University of Technology, Sydney.
- A report from image quality evaluation experts, to be drawn from eminent laboratories around the world, based on contributed submissions.
- An input document (max 5 pages, following the PCS paper template) from each contributor.
- Presentations by panelists, contributors and a representative of the quality evaluation activity.
- Discussion of evaluation results and implications of other features associated with each of the evaluated technologies.
- Post-meeting write-up of an overview paper by the chair and panelists, to be supplemented by inclusion of some or all of the input papers provided by contributors (at the discretion of the panel), for inclusion as an addendum to the proceedings of the 31st Picture Coding Symposium.
- Response to the CfI issued by JPEG, to be delivered at its 69th Warsaw meeting from June 22-26, 2015.
- Links to download and upload sites, as well as paper formatting instructions, to be available at http://www.pcs2015.org
- Indication of intent to contribute: 1 April, 2015
- Final submission of all materials: 1 May, 2015
- Chair: Prof. Ian Burnett (email@example.com)