Илья Поляков
216 lines
11 KiB
Markdown
216 lines
11 KiB
Markdown
---
|
|
name: pdf-to-kindle
|
|
description: Convert a text-layer PDF (usually one generated by calibre from an ebook) into EPUB/AZW3 for Kindle while preserving italics, bold, chapter headings, sidebars/journal blocks, images and cross-page paragraphs. Use when the user asks to read a PDF book on Kindle or another e-reader, to convert a PDF to EPUB/AZW3/MOBI, or complains that a converted book lost its italics, merged its chapters, or reflows badly on the device. Not for scanned PDFs (no text layer) — those need OCR first. Also covers translating such a book into another language before packing it, delegated to the external book_translator project, and generating a TTS audiobook from it with an integrity check for dropped fragments.
|
|
---
|
|
|
|
# PDF to Kindle
|
|
|
|
`ebook-convert book.pdf book.epub` silently destroys formatting: poppler infers
|
|
style from font names and misses abbreviated ones like `MinionPro-It`, and
|
|
calibre labels body text as `<h2>` when the most common font is not the body
|
|
font. Measured on a 399-page novel: 435 italic runs in the PDF became 1 `<i>`
|
|
tag, and 2817 body paragraphs became `<h2>`.
|
|
|
|
`scripts/pdf2html.py` extracts spans with PyMuPDF, where style comes from span
|
|
properties, and emits semantic XHTML. calibre is then used only as the
|
|
XHTML→EPUB packer.
|
|
|
|
## Workflow
|
|
|
|
1. **Verify a text layer exists.** `pdfinfo` and `pdffonts` on the file. No
|
|
embedded fonts / no extractable text means a scan — stop and say OCR is
|
|
needed; this skill does not apply.
|
|
2. **Check tooling.** `ebook-convert` must be on PATH. For PyMuPDF, prefer an
|
|
existing interpreter that has it; otherwise build a throwaway venv in the
|
|
scratchpad — do not install into the system Python:
|
|
|
|
```bash
|
|
python3 -m venv "$SCRATCH/venv" && "$SCRATCH/venv/bin/pip" -q install pymupdf
|
|
```
|
|
|
|
3. **Profile the fonts** before converting anything:
|
|
|
|
`python scripts/pdf2html.py --fonts book.pdf`
|
|
|
|
Read off: the body font (largest character count), the italic variant, the
|
|
heading sizes, and any secondary family used for sidebars, journal entries,
|
|
chat logs or slides.
|
|
4. **Tune the thresholds** in `block_kind()` and `style()` to that output. The
|
|
defaults target a 15pt/letter calibre layout: `>= 28` or a display font is a
|
|
title, `>= 24` a chapter/part `h1`, `>= 17` an `h2` subtitle, a secondary
|
|
family is `p.note`. `INDENT_X` (default 88) is the x-coordinate that
|
|
separates an indented first line from a continuation line — verify it
|
|
against the real `x0` values, not by assumption.
|
|
5. **Convert to XHTML:**
|
|
|
|
`python scripts/pdf2html.py book.pdf out/book.html`
|
|
|
|
6. **Verify before packing** (see Verification). Fix thresholds and re-run until
|
|
the counts are sane. Cheap to iterate; do not skip to packing.
|
|
7. **Pack with calibre**, always passing explicit TOC XPaths — without them
|
|
calibre applies its own heuristics and re-breaks the chapters:
|
|
|
|
```bash
|
|
ebook-convert out/book.html "Title.epub" \
|
|
--title="Title" --authors="Author" --language=en \
|
|
--cover=out/images/cover.jpg \
|
|
--level1-toc='//h:h1' --level2-toc='//h:h2' \
|
|
--page-breaks-before='//h:h1' \
|
|
--no-default-epub-cover
|
|
ebook-convert "Title.epub" "Title.azw3"
|
|
```
|
|
|
|
Take title/author from the user or the book's own title page — PDF metadata
|
|
is often an ASIN or a filename.
|
|
8. **Deliver both files** and say what each is for: `.epub` for Send to Kindle
|
|
(Amazon converts server-side), `.azw3` for USB copy into `documents/`.
|
|
Delete intermediate artifacts left in the user's directories.
|
|
|
|
## Verification
|
|
|
|
Never report success on the converter's own summary line alone. Check:
|
|
|
|
```bash
|
|
python3 - <<'EOF'
|
|
import re
|
|
t = open('out/book.html').read()
|
|
print('italic:', t.count('<i>'), 'bold:', t.count('<b>'))
|
|
print('h1:', len(re.findall(r'<h1', t)))
|
|
print('double spaces:', re.sub('<[^>]+>', '', t).count(' '))
|
|
EOF
|
|
```
|
|
|
|
- italic count near zero on a novel means `style()` missed the font-name pattern;
|
|
- an `h1` count in the hundreds means a size threshold is too low and body text
|
|
is being promoted;
|
|
- many double spaces means line joining is off;
|
|
- list the extracted headings (`grep -o '<h1[^>]*>.\{0,60\}'`) and read them —
|
|
they become the TOC, and a wrong one is obvious at a glance;
|
|
- read one full page of body text and confirm paragraphs merge across page
|
|
breaks and hyphenated words are rejoined.
|
|
|
|
After packing, confirm the EPUB: `ebook-meta` for metadata, and the `.ncx`
|
|
navPoint count for the TOC size.
|
|
|
|
## Optional stage: translation
|
|
|
|
Only when the user asks for a translated book. It slots between step 6 and
|
|
step 7 — translate the XHTML, then pack the translated file with calibre.
|
|
|
|
Translation is delegated to an external project,
|
|
[`vetermanve/book_translator`](https://github.com/vetermanve/book_translator)
|
|
(DeepSeek or a local Ollama model). **Never modify that repository** — it is
|
|
driven through its documented CLI and file formats only. `scripts/translate.py`
|
|
is the bridge: it writes the repo's input format, shells out to
|
|
`03_translate_parallel.py --all`, and reads the repo's output format back.
|
|
|
|
1. **Check the book actually needs translating.** The script does this first
|
|
and refuses to burn API credits on a book already in the target language:
|
|
|
|
`python scripts/translate.py book.html out.html --repo <clone> --workdir <dir> --check-only`
|
|
|
|
It reports block/chapter/character counts and the detected source language.
|
|
`--force` overrides the refusal.
|
|
2. **Set up the external repo once** (a plain clone; never edit it) and its
|
|
credentials — a `.env` in the working directory with `USE_LOCAL_MODEL=false`
|
|
and `DEEPSEEK_API_KEY=…`, or `USE_LOCAL_MODEL=true` plus `OLLAMA_MODEL` for a
|
|
local model. **Its `requirements.txt` is incomplete** — `pip install openai
|
|
pyyaml python-dotenv` as well. Both `openai` and `pyyaml` are imported at
|
|
runtime and missing from that file; without `pyyaml` every single request
|
|
dies inside `_create_system_prompt` *before* reaching the API, and the tool
|
|
reports "API запросов: 0, ошибок: N" while writing `[UNTRANSLATED]` stubs.
|
|
Write the `.env` under `umask 077`, and never echo the key into logs or
|
|
command output.
|
|
3. **Translate:**
|
|
|
|
`python scripts/translate.py book.html book.ru.html --repo <clone> --workdir <dir> --workers 12`
|
|
|
|
Resumable: the external repo tracks completed chapters and skips them on a
|
|
re-run, so an interrupted run costs nothing to restart.
|
|
4. **Read the reported counts.** "без перевода" above zero means a chapter came
|
|
back with a different paragraph count and kept its original text; "разметка
|
|
потеряна" counts paragraphs where the model mangled the inline-tag markers
|
|
and the italics were dropped rather than corrupted. Both are expected to be
|
|
near zero — a large number means the translator misbehaved, not that the
|
|
bridge is broken. The script then checks the output's actual language and
|
|
exits non-zero if the text is still the source language or contains
|
|
`[UNTRANSLATED]` stubs — **matching paragraph counts do not prove anything
|
|
was translated**, since the external tool substitutes the original on API
|
|
failure. Do not pack a file that failed this check.
|
|
**Re-running after a failure needs the state cleared:** the external repo's
|
|
`progress/` directory marks those chapters complete and will skip them.
|
|
Delete `<workdir>/progress`, `<workdir>/context`, and
|
|
`<workdir>/translations` before the retry.
|
|
5. Pack `book.ru.html` with `--language=ru` and translated `--title`/`--authors`.
|
|
|
|
How formatting survives a translator that only speaks plain text: inline `<i>`
|
|
and `<b>` become `⟦i⟧…⟦/i⟧` markers before the text leaves, and are restored
|
|
after. Every paragraph's markers are balance-checked on the way back. Images,
|
|
`<h1>`/`<h2>` structure, and block order never leave this side — only the text
|
|
of each block round-trips, and blocks are reassembled by index.
|
|
|
|
`scripts/test_translate.py` covers the marker round-trip, the broken-marker
|
|
fallback, language detection, and a full split/rebuild against a faked
|
|
translator response — no network, no API key. Run it after touching the bridge.
|
|
|
|
## Optional stage: audiobook
|
|
|
|
Also delegated to `book_translator` (`05_create_audiobook.py`, Microsoft
|
|
edge-tts — free, needs internet). Two wrappers live in `scripts/audiobook.py`;
|
|
the external repo is still never edited.
|
|
|
|
1. **Strip markup markers first.** The translated JSON still holds the
|
|
`⟦i⟧` markers from the translation stage — TTS would read them aloud:
|
|
|
|
`python scripts/audiobook.py prep <workdir>/translations <workdir>/translations_tts`
|
|
|
|
Point the external script at the *stripped* copy; the original keeps its
|
|
italics for the EPUB.
|
|
2. **Synthesize:** `05_create_audiobook.py --translations-dir <…>/translations_tts
|
|
--voice dmitry --rate '+0%'`. Fragments are **one per paragraph** (the
|
|
`--paragraphs-per-group` flag is not used by the loop), named
|
|
`chapter_NNN_intro.mp3` / `chapter_NNN_para_NNNN.mp3` under
|
|
`audiobook/temp_audio/`.
|
|
3. **Verify before the temp files are deleted** — `cleanup_temp_files()` wipes
|
|
`temp_audio/`, and after that only the merged file can be checked:
|
|
|
|
`python scripts/audiobook.py verify <workdir>/translations_tts <workdir>/audiobook`
|
|
|
|
It flags chapters missing fragments, zero-byte/undecodable mp3s, and
|
|
chapters whose duration falls short of what their character count predicts.
|
|
The seconds-per-character baseline is the median across chapters, so it
|
|
self-calibrates to whatever voice and `--rate` were used. Exit code is
|
|
non-zero when anything is wrong.
|
|
4. **Re-running fills gaps cheaply** — the external script skips any fragment
|
|
file that already exists *and is non-empty*, so delete the bad ones `verify`
|
|
named and run it again. It never re-checks that an existing file is sane,
|
|
which is exactly why step 3 exists.
|
|
|
|
Known limits of the external stage: it merges everything into a single
|
|
`audiobook_complete.mp3` with no per-chapter files and no chapter marks —
|
|
poor for players; splitting is not implemented here. The phonetics stage
|
|
(`07_extract_terms.py` + `08_generate_phonetics.py`) is worth running first for
|
|
a technical book, or the Russian voice will mangle every English term.
|
|
|
|
`scripts/test_audiobook.py` builds real silent mp3s with ffmpeg and checks that
|
|
`verify` catches a missing fragment, an empty file, and passes a clean book.
|
|
Needs ffmpeg; no network.
|
|
|
|
## What the script does
|
|
|
|
- style from span font names (`-It`, `Italic`, `Bold`, `Semibold`) → `<i>`/`<b>`;
|
|
- headings by font size, sidebars by font family;
|
|
- one PDF block = one paragraph; an unindented block that opens a page is
|
|
appended to the previous page's paragraph;
|
|
- trailing hyphens dropped, adjacent `</i><i>` runs merged, doubled spaces collapsed;
|
|
- images written to `images/`, cover rendered from page 1 at 150 dpi;
|
|
- absolute positioning is deliberately discarded so text reflows at any font size.
|
|
|
|
## Limits
|
|
|
|
- Headers/footers are not stripped — calibre-made PDFs have none. A typeset PDF
|
|
with running heads and page numbers needs blocks filtered by `y` coordinate
|
|
near the margins; add that before trusting the output.
|
|
- Multi-column layouts are not handled; block order would need column sorting.
|
|
- Thresholds are per-layout constants. Always re-run `--fonts` for a new book.
|