No Pixel Left Behind: Filling Gaps in Anime Colorization

Masahiro Kono1, Akinobu Maejima2,3, Yuki Koyama1, Yotam Sechayk1, Takeo Igarashi1
1 The University of Tokyo; 2 OLM Digital, Inc.; 3 IMAGICA GROUP Inc.
CHI '26: The ACM CHI Conference on Human Factors in Computing Systems (Full Conference Paper)
& WISS '25: Workshop on Interactive Systems and Software (Demo Presentation)
ACM Library Paper (Full/EN) Paper (Short/JP) Posters Demo Video Slides Code (Coming Soon)
Teaser (EN) Teaser (JP)

GapFill is a specialized tool designed to assist professional anime colorists in addressing small, unpainted areas. (a) Colorization with the paint bucket tool often (b) leaves small enclosed regions (“gaps”) unpainted. When GapFill is activated, (c) automatic gap detection with circular highlights is triggered, and (d) these gaps are temporarily filled with suggested colors using our domain-specific deep learning method. (e) When the user hovers over a highlight, a magnified view pops up, allowing inspection without manual zooming. (f) Dragging within a highlight activates a color-picker mode for correcting the suggestion. (g) If suggestions appear correct, users can sweep across highlights to apply them at once.

Abstract

Animation production workflows often involve digital colorization of line art, where small unpainted regions (“gaps”) frequently occur and remain an underexplored challenge. We conducted a formative study in Japanese animation (anime) pipelines and found that while the paint bucket tool is widely used for base coloring, tiny enclosed areas are frequently overlooked, resulting in time-consuming manual detection and filling. We introduce GapFill, a tool grounded in professional practices that reduces the effort of gap detection, zooming, and color selection. Our deep-learning method suggests appropriate fill colors by referencing surrounding regions, leveraging the flat-color nature of anime-style images. In a user study with 13 professional colorists, our system improved performance and usability in gap-filling tasks over conventional methods. The study also suggested that prediction accuracy alone is not the primary factor for usability, that appropriate colors can be contextually ambiguous, and that GapFill can complement existing tools depending on users' trust in new AI-powered assistance.

Web Demo (Used in the user study)

The interactive web demo will be available here. (Web page coming soon.)

Temporary URL: https://gapfill-webtest-855.pages.dev/

Demo Video

Background and Conventional Methods

Manual colorization of line art with flat colors using the Paint Bucket (Flood Fill) tool. Small enclosed areas often remain unpainted (“gaps”) due to unintended line intersections especially at sharp corners. These gaps are often difficult to detect and time-consuming to fix manually.

Colorization using the paint bucket

Manual colorization of line art using the Paint Bucket tool.

Gap Example

Small enclosed areas often remain unpainted (“gaps”) due to unintended line intersections especially at sharp corners.

Below are the conventional methods for addressing unpainted small gaps that were identified through our formative study.
(reference: https://tips.clip-studio.com/en-us/articles/591)

Black Light Method

(a) Black Light Method: Temporarily replace all colors with black in a single click, making gaps appear as white dots for easy detection.

Leftover Pen

(b) Leftover Pen: Fill all enclosed areas along the stroke with the specified color.

Enclose And Fill

(c) Enclose and Fill: Fill all gaps within the enclosed area using a lasso-like tool.

While these tools are convenient, the process still requires repetitive actions such as detecting gaps, zooming, and selecting colors.

Proposed Tool: “GapFill”

Design Principles

Features of GapFill

Unpainted Gap Detector with Circular Highlights; Automatic Color Suggestion for Filling Unpainted Gaps

Gap Detector with Circular Highlights and Automatic Color Suggestion: GapFill automatically detects and highlights small unpainted enclosed regions to reduce the manual effort required for gap detection. It also temporarily overlays each detected gap with a predicted fill color to reduce the manual effort of color selection.

Hover-Activated Pop-up Magnification for Quick Inspection

Hover-Activated Pop-up Magnification: GapFill provides a hover-triggered localized magnification around each detected gap within the highlight, reducing the need for manual zooming during quick inspection.

In-Circle Color-pick for Correcting AI Suggested Colors

In-Circle Color-pick for Correcting Colors: GapFill provides manual correction of AI-suggested fill colors through a familiar color-picker interaction within the highlight, allowing users to efficiently resolve few mispredicted gaps without manual zooming.

Out-Circle Sweep-to-Apply and Apply-All Button

Out-Circle Sweep-to-Apply and Apply-All Button: GapFill supports batch application of AI-suggested colors via a familiar brush-like interaction when the cursor is outside the highlight, enabling efficient filling of multiple gaps.

Method for Automatic Color Prediction

Color Prediction via Region Correspondence (EN)

We use an indirect color prediction approach that learns correspondences between regions rather than predicting colors directly. By leveraging the flat-color characteristics of anime-style images and the color-independent nature of unpainted gaps, our U-Net-based model predicts which surrounding large region shares the same color as the target gap and robustly suggests an appropriate fill color.

Creating Synthetic Training Dataset

We create a synthetic training dataset by extracting line art, segmenting it into enclosed regions, and treating small regions below a pixel threshold as potential unpainted gaps. Using local image patches centered on these regions, the model is trained to associate each gap with the nearest large same-color region.

Special Thanks

This project is based on results obtained from GENIAC (Generative AI Accelerator Challenge, a project to strengthen Japan’s generative AI development capabilities), a project implemented by the Ministry of Economy, Trade and Industry (METI) and the New Energy and Industrial Technology Development Organization (NEDO), Japan Grant Number JPNP20017. This work was also supported by JST, CRONOS, Japan Grant Number JPMJCS25K1.
Finally, we thank the professional colorists at OLM Asia SDN BHD for their participation and valuable feedback in our formative and user studies, and we also express our sincere gratitude to the members of ©IIS-P / Ponnomichi Production Committee for kindly providing the image data.

Citation