Commit 6b643c8a authored by njean42's avatar njean42
Browse files

Initial commit

parents
Kumiko, the Comics Cutter is a set of tool for generating information about
panels locations within a comic books pages, and more.
Copyright (C) 2018 njean42
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
# Introduction
![Kumiko mascot by Cthulhulumaid, [CC-BY-SA 4.0](http://creativecommons.org/licenses/by-sa/4.0/ "Creative Commons License")](artwork/kumiko-big.png "Kumiko mascot by Cthulhulumaid")
*Kumiko, the Comics Cutter* is a set of tools to compute useful information about comic book pages, panels, and more.
Its main strength is to find out the **locations of panels** within a comic's page (image file).
*Kumiko* can also compile information about panels for all pages in a comic book, and present it as one piece of data (JSON-formatted object).
*Kumiko* makes use of the great (freely licensed) [opencv](https://opencv.org/) library, which provides image processing algorithms of all sorts.
Mainly, the contour detection algorithm is used to detect panels within an image.
# Demo
*TL;WR* Too Long; Won't Read the whole doc?
Try reading [an *xkcd* episode](xkcd.html) panel by panel. :)
# Philosophy
*Kumiko* aims at being a functional library to extract information from comic pages / books.
The goal is to provide a set of tools that is usable beforehand, to extract all needed information.
External programs can later use the generated information for different purposes: panel-by-panel viewing, actual splitting of an image down into panels, etc.
## Panel-by-panel comic reading
Bing able to jump from one panel to the next was the original idea behind *Kumiko*.
![[xkcd](https://www.xkcd.com) by Randall Munroe, [#208](https://www.xkcd.com/208/), [CC BY-NC 2.5](https://creativecommons.org/licenses/by-nc/2.5/)](doc/img/xkcd.png "xkcd")
Comic viewers usually imply a very common *page-by-page reading paradigm*.
You read a page, possibly zooming on it to be able to read speech bubbles, then click, tap, press a key or swipe to go to the next page.
With knowledge about panels locations, we can imagine a comic reader that also offers *panel-by-panel reading*.
This is especially interesting for **small screens**, on which you probably can't read the texts if a whole page is displayed.
We provide an example implementation of such an reader application.
Read more in the [example Reader documentation](doc/Reader.md).
# Requirements
`apt-get install python-opencv` will install the only necessary library needed: *opencv* (2.x).
This should do the trick for Debian distros and derivatives (Ubuntu, Linux Mint...).
If you successfully use *Kumiko* on any other platform, please let us know!
From memory (TODO: check), this implies using python version 2.x, since *opencv* 3.x runs on python 3.x.
# Usage
See the [usage doc](doc/Usage.md) for details on how to use the *Kumiko* tools.
# Numbering
The numbering is left to right, top to bottom for now (see roadmap).
Here is an example of how *Kumiko* is going to number panels by default (numbers and red lines not in the original picture).
![[Pepper & Carott](https://www.peppercarrot.com/) by [David Revoy](https://www.davidrevoy.com), [episode 2](https://www.peppercarrot.com/en/article237/episode-2-rainbow-potions), [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)](doc/img/numbering.png "Pepper&Carrot")
# Limitations
For now, *Kumiko* only deals with **white-ish backgrounds**.
Panels within a comic page will be any "object" that has non-white boundaries (not necessarily vertical or horizontal lines).
If you have ideas on how to programmatically guess the background color of a page, please let us know!
# Short- and longer-term awesome features (roadmap)
## Kumiko library
* be able to detect panel contours on pages with **non-white background**
* implies being able to determine the background color: histogram, probing of some kind? (worst case: manually?)
* provide an option for **right-to-left numbering**, critical e.g. for mangas
## Back-office (validation / edition tool)
Let's face it: we probably can't ensure that *Kumiko* can perfectly find out the panels in *any* image.
There is a huge diversity of panel boundaries, layouts and whatnot.
This is why we should have some kind of back-office / editing tool that lets a human editor:
* validate pages
* add, delete, move or resize incorrect panels
* report bugs
* ...
Such a tool should edit the JSON file representing a comic book information, for later use by other programs that'll rely on it.
# Example of a panel-by-panel reader application
## Concept
We provide an example implementation of a *panel-by-panel reader* as an example of what *Kumiko* can be used for.
Once the command-line tool has generated a complete JSON representation (see the [usage doc](Usage.md)) of a page panels, our example reader can provide an interface that shows only the first (zoomed) panel of a page, and allows to jump forward to the next panel(s).
Try reading [an *xkcd* episode](../xkcd.html) panel by panel. :)
Use the left and right arrow keys or click-touch the screen to navigate through panels.
## Reader implementation details
See the `reader.js` file for use in a browser or application webview.
Some event bindings are defined in the `xkcd.html` file.
# Command-line usage
## Help
More detailed help can be found by running `kumiko --help`.
## Get panel information for one page (image file)
kumiko -i /path/to/comicbook/page001.jpg
The result would look like the following:
[
{
"filename": "comicbook/page001.jpg",
"size": [1920,2549],
"panels": [
[148,108,1628,874],
[147,999,358,538],
...
]
}
]
(Note that the resulting JSON-formatted data is already a list, with just one object representing the extracted information about your image.
This is for compatibility with generating information for several pages in one directory, see next section.)
### Panels
The *size* gives us the *[width,height]* of our image, in pixels.
The panel details are also expressed in pixels.
The 4 values representing a panel are given in the following order: **[x,y,width,height]**, with *x* and *y* representing the **top-left corner** of the panel.
Note that, although Kumiko can detect panels with arbitrary shapes, it will return a rectangular bounding box for each one.
This is because in the end, many things are rectangular: image files, our screens to display them...
We may want an option in the future to get more accurate polygons instead.
### File names
The `filename`s in our JSON object are relative to the working directory from which you invoked the `./kumiko` command.
## Get panel information for all pages in one comic book
kumiko -i /path/to/comicbook/
The result of such a command would be a JSON-formatted array of all images in the folder:
[
{
"filename": "comicbook/page001.jpg",
"size": [1920,2549],
"panels": [
[148,108,1628,874],
[147,999,358,538],
...
]
},
{
"filename": "comicbook/page002.jpg",
"size": [1920,2550],
"panels": [
[149,109,1631,737],
...
]
}
...
]
The image order is alphabetical, which is based on a commonly encountered *page001, page002, ...* naming for pages.
## Debug
You can pass `kumiko` a `--debug` parameter that tells you are craving debugging information.
This will create an image for each page that Kumiko analysed, renamed `$filename-040-contours-numbers.jpg`.
These files will be found in the `debug` subfolder (should exist and be writable).
Panel contours and image numbering is drawn onto every debugging image.
This diff is collapsed.
#!/usr/bin/env python
import sys, os, json
import argparse
from kumikolib import Kumiko
parser = argparse.ArgumentParser(description='Kumiko CLI')
# Utilities
parser.add_argument('-d', '--debug', action='store_true', help='Debug mode, creates files with extra panel information and more')
# Input/Output
parser.add_argument('-i', '--input', nargs=1, required=True, help='A file or folder name')
parser.add_argument('-o', '--output', nargs=1, help='A file name to save out to')
args = parser.parse_args()
k = Kumiko({
'debug': args.debug
})
file_or_folder = args.input[0]
if os.path.isdir(file_or_folder):
info = k.parse_dir(file_or_folder)
elif os.path.isfile(file_or_folder):
info = k.parse_images([file_or_folder])
else:
print "--input (-i) is not a file or directory: '"+file_or_folder+"'"
sys.exit()
info = json.dumps(info)
if args.output:
f = open(args.output[0], 'w')
f.write(info)
f.close()
print info
#!/usr/bin/env python
import os
import cv2 as cv
class Kumiko:
options = {}
img = False
def __init__(self,options={}):
if 'debug' in options:
self.options['debug'] = options['debug']
else:
self.options['debug'] = False
if 'reldir' in options:
self.options['reldir'] = options['reldir']
else:
self.options['reldir'] = os.getcwd()
def read_image(self,filename):
return cv.imread(filename)
def parse_dir(self,directory):
filenames = []
for root, dirs, files in os.walk(directory):
for filename in files:
filenames.append(os.path.join(root,filename))
filenames.sort()
#filenames = filenames[0:10]
return self.parse_images(filenames)
def parse_images(self,filenames=[]):
infos = []
for filename in filenames:
infos.append(self.parse_image(filename))
return infos
def parse_image(self,filename):
img = self.read_image(filename)
# TODO: handle error
size = list(img.shape[:2])
size.reverse() # get a [width,height] list
infos = {
'filename': os.path.relpath(filename,self.options['reldir']),
'size': size,
'panels': []
}
gray = cv.cvtColor(img,cv.COLOR_BGR2GRAY)
tmin = 220
tmax = 255
ret,thresh = cv.threshold(gray,tmin,tmax,cv.THRESH_BINARY_INV)
# OpenCV 3.2+
# See https://docs.opencv.org/master/d4/d73/tutorial_py_contours_begin.html
#im2, contours, hierarchy = cv.findContours(thresh, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)
#cv.imwrite('tmp/contour.jpg',im2)
#cv.imwrite('tmp/hierarchy.jpg',hierarchy)
#print contours
# OpenCV 2.4
# https://docs.opencv.org/2.4/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html
contours, hierarchy = cv.findContours(thresh, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE)
# Get panels out of contours
for contour in contours:
arclength = cv.arcLength(contour,True)
epsilon = 0.01 * arclength
approx = cv.approxPolyDP(contour,epsilon,True)
x,y,w,h = cv.boundingRect(approx)
# exclude very small panels
if w < infos['size'][0]/15 or h < infos['size'][1]/15:
continue
contourSize = int(sum(infos['size']) / 2 * 0.004)
cv.drawContours(img, [approx], 0, (0,0,255), contourSize)
panel = [x,y,w,h]
infos['panels'].append(panel)
if len(infos['panels']) == 0:
infos['panels'].append([0,0,infos['size'][0],infos['size'][1]]);
for panel in infos['panels']:
x,y,w,h = panel
panel = {
'x': x,
'y': y,
'w': w,
'h': h
}
# Number infos['panels'] comics-wise (left to right for now)
self.gutterThreshold = sum(infos['size']) / 2 / 20
infos['panels'].sort(cmp=self.sort_panels)
# write panel numbers on debug img
fontRatio = sum(infos['size']) / 2 / 400
font = cv.FONT_HERSHEY_SIMPLEX
fontScale = 1 * fontRatio
fontColor = (0,0,255)
lineType = 2
n = 0
for panel in infos['panels']:
n += 1
position = ( int(panel[0]+panel[2]/2), int(panel[1]+panel[3]/2))
cv.putText(img,str(n),position,font,fontScale,fontColor,lineType)
if (self.options['debug']):
cv.imwrite(os.path.join('debug',os.path.basename(filename)+'-040-contours-numbers.jpg'),img)
return infos
def sort_panels (self,p1,p2):
[p1x,p1y,p1w,p1h] = p1
[p2x,p2y,p2w,p2h] = p2
p1b = p1y+p1h # p1's bottom
p2b = p2y+p2h # p2's bottom
p1r = p1x+p1w # p1's right side
p2r = p2x+p2w # p2's right side
# p1 is above p2
if p2y >= p1b - self.gutterThreshold and p2y >= p1y - self.gutterThreshold:
return -1
# p1 is below p2
if p1y >= p2b - self.gutterThreshold and p1y >= p2y - self.gutterThreshold:
return 1
# p1 is left from p2
if p2x >= p1r - self.gutterThreshold and p2x >= p1x - self.gutterThreshold:
return -1
# p1 is right from p2
if p1x >= p2r - self.gutterThreshold and p1x >= p2x - self.gutterThreshold:
return 1
return 0 # should we really fall into this case?
class Reader {
constructor(options) {
if (!options.comicsPath) {
console.error('no comicsPath given in options');
return;
}
if (!options.container) {
console.error('no container given in options');
return;
}
this.comicsPath = options.comicsPath;
this.bigimgsize = 1080;
this.container = $('<div class="container"/>');
options.container.append(this.container);
this.container.css({ position: 'relative', width: '90%', height: '90%' });
try {
this.currpage = JSON.parse(localStorage.currpage);
} catch (e) {
this.currpage = {};
}
if (!this.currpage[this.comicsPath])
this.currpage[this.comicsPath] = 0;
this.nbDigits = options.nbDigits ? options.nbDigits : 1;
this.currpanel = 0;
window.addEventListener('orientationchange', function () {
setTimeout( function () { this.gotoPanel(this.currpanel); }.bind(this), 500); // slight delay to make it work better, not sure why :)
}.bind(this));
}
loadNextPage() { this.loadPage(this.currpage[this.comicsPath]+1); }
loadPrevPage() { this.loadPage(this.currpage[this.comicsPath]-1); }
getCurrentPage() { return this.currpage[this.comicsPath]; }
getPages () {
return this.comic;
}
loadPage(page=false) {
// don't go to a page below 0, or above the number of pages in this comic
if (page == -1)
return;
if (this.comic && page == this.comic.length) {
this.currpage[this.comicsPath] = this.comic.length-1;
return;
}
if (page === false)
page = (this.comicsPath in this.currpage) ? this.currpage[this.comicsPath] : 0;
this.currpage[this.comicsPath] = page;
localStorage.currpage = JSON.stringify(this.currpage);
if (this.comic) {
this.kumikoReady();
return;
}
// else
$.get(this.comicsPath, function (comic) {
this.comic = comic;
this.kumikoReady();
}.bind(this));
}
kumikoReady () {
var page = this.currpage[this.comicsPath];
var imginfo = this.comic[page];
var img = $('<img class="pageimg" src="'+imginfo['filename']+'"/>');
img.css({
position: 'absolute',
'width': '100%',
'height': '100%'
});
this.container.children('img').remove();
this.container.prepend(img);
this.drawPanels(imginfo);
this.dezoom();
if (this.currpanel == 'last')
this.currpanel = $('.panel').length-1;
else
this.currpanel = 0;
if ($('input[name=panelview]').is(':checked'))
this.gotoPanel(this.currpanel);
this.container.parent().trigger('page-changed');
}
zoomOn (panel,callback) {
var growth = this.container.parent().width() / panel.width();
var max = 'x';
var ygrowth = this.container.parent().height() / panel.height();
if (ygrowth < growth) {
growth = ygrowth;
max = 'y';
}
// var margin = growth * 0.1;
var newcss = {
top: - panel.position().top * growth,
left: - panel.position().left * growth,
height: this.container.height() * growth,
width: this.container.width() * growth
};
// center panel horizontally or vertically within container's parent?
if (max == 'x')
newcss.top += (this.container.parent().height() - panel.height() * growth) / 2;
else
newcss.left += (this.container.parent().width() - panel.width() * growth) / 2;
$('.panel.zoomTarget').removeClass('zoomTarget');
panel.addClass('zoomTarget');
this.container.addClass('zoomed');
this.container.animate(newcss,300,callback);
}
dezoom () {
var size = this.getImgSize();
var newcss = {
width: size.w,
height: size.h,
left: 0,
top: 0
};
this.container.removeClass('zoomed');
this.container.css(newcss);
}
getImgSize () {
var size = {
w: this.container.parent().width(),
h: this.container.parent().height()
};
var imgsize = this.comic[this.currpage[this.comicsPath]]['size'];
var ratio = imgsize[0] / imgsize[1];
if (size.w > size.h * ratio)
size.w = size.h * ratio;
else if (size.h > size.w / ratio)
size.h = size.w / ratio;
return size;
}
gotoPanel (i) {
$(window).scrollTop( $('#result').position().top )
if (i < 0) {
this.dezoom();
this.currpanel = 'last';
this.loadPrevPage();
return;
}
var newPanel = $('.panel').eq(i);
if (newPanel.length > 0)
this.zoomOn(newPanel);
else {
this.dezoom();
this.loadNextPage();
}
}
nextPanel () { this.currpanel++; this.gotoPanel(this.currpanel); }
prevPanel () { this.currpanel--; this.gotoPanel(this.currpanel); }