Files
media-batch-tools/README.md
T

292 lines
22 KiB
Markdown

# Media Batch Tools
Small Python scripts for organizing photo collections and preparing or encoding edited videos.
## Scripts
When started without paths, the path-based tools ask for one file or directory per line until an empty line is entered. Tab completes paths on Linux, including the photo and audio output-directory fields. The repository tracks the entry-point scripts as executable, so on Linux they can also be run as `./photo_metadata.py`, `./audio_tool.py`, and so on. If a graphical session starts one without a terminal, it opens in Konsole or another supported terminal.
### `photo_metadata.py`
Interactive metadata cleanup for photo collections:
```bash
python3 photo_metadata.py path/to/file/or/directory [...]
```
With no paths, the script asks for one file or directory per line until an empty line is entered. This is equivalent to supplying several command-line or dragged paths. Directory inputs are scanned recursively. The script reads metadata, asks for the choices it needs, prints a preview, and only then applies changes in place after confirmation.
Current features:
- supports `.jpg`, `.jpeg`, `.heic`, `.arw`, `.mp4`, `.mov`, and `.mts`
- reads and writes metadata through `exiftool` from `PATH`
- optional direct time shifts, such as `+1:30`, `-02:00:00`, or `+1h 2m`
- optional reference-clock correction using a source photo timestamp or filename plus the correct time
- keyboard-driven settings table with timezone gap filling and optional fixed timezone offsets
- artist/author set, clear, or leave unchanged
- optional timestamp renaming: adjust timestamps already in filenames, add one, replace the stem, or leave names unchanged
- video filenames use the beginning timestamp by subtracting rounded duration from the video metadata timestamp
- Sony-style video sidecars like `C0011.MP4` plus `C0011M01.XML` are moved and renamed with the video
- automatic photo burst/HDR grouping per camera
- optional panorama project (`.pto`) grouping and reference updates
- inferred subsecond metadata for grouped same-second photo bursts when the group has no existing subseconds
- optional capture-time inference from a safe timestamp filename, then `ModifyDate` as a fallback
The default output directory is determined from the selected items. A single selected folder is the default. For multiple selected files/folders, their last common ancestor is used. The first settings row can replace it with an absolute path or a path relative to that default. Automatic burst/HDR and `.pto` group folders are created inside the selected output directory.
The `Organize/group` setting has three modes. **No moving** leaves files in their existing directories. **Flat** moves every selected media file into the output directory without creating groups. **Grouping** moves ungrouped files there and creates burst/HDR and optional panorama-project group directories. Space cycles the mode; burst size and `.pto` controls are available only in Grouping mode.
Panorama project files use the `.pto` extension. When enabled, every media file referenced by a `.pto` file belongs to the same group. If those files already belong to burst/HDR groups, the groups are merged. `.pto` files are moved into the group folder with their referenced files, renamed from the first and last referenced media stems, and their file references are rewritten to the new relative paths.
Reference-clock correction is useful when one photo shows a reliable clock. For example, if `DSC01234.JPG` has camera metadata `2026:06:02 22:10:00`, but the clock in the photo shows `22:13:25`, choose reference mode, enter `DSC01234.JPG` as the source, and enter `22:13:25` as the correct time. The script computes a `+00:03:25` shift and applies that correction to the selected files. If you omit the date in the correct time, the script chooses the date closest to the source timestamp, within 12 hours.
The settings table requires an ANSI-capable terminal because it uses arrow keys and inline editing. Konsole is supported. Use Up/Down to select a row, Left/Right to select an option, Space to toggle an option, and Enter to continue. The preview separates common changes, inferred timestamps, warnings, and file operations before asking for confirmation.
Dolphin may follow its configured file-type action instead of executing Python files; in that case, use its **Run in Konsole** action or run `./photo_metadata.py` from Konsole.
In the reference-photo field, Tab completes names from the selected photos (not videos). Ambiguous matches complete their common prefix and show possible names; continue typing to narrow the selection.
### `grouping.py`
Fast manual grouping for selected files:
```bash
python3 grouping.py path/to/file/or/directory [...]
```
With no arguments, it lists the supported input forms and asks for selections until an empty line is entered. For example:
```text
Selection: /photos/DSC0010.JPG DSC0020.JPG
Selection: "/photos/Extra photo.JPG"
Selection: "/photos/Panorama A" "Panorama C"
Selection: /photos/DSC01*.JPG
Selection: /photos/DSC0021.JPG 5
Selection:
```
Each line accepts one path, a wildcard pattern, two range endpoints, or a path followed by a positive integer count. Quote any path/name containing spaces. A range's second endpoint can be just a name or an absolute path, but must be in the first endpoint's directory. Both endpoints must be files or both must be directories. Selection uses case-insensitive alphabetical filename order, includes both endpoints, and skips items of the other type. Reversed endpoints work too. Invalid lines are reported and can be re-entered.
`*` matches zero or more characters within a filename or directory name. Other wildcard syntax is treated literally. As with shell wildcards, hidden names require an explicit leading dot. Patterns are only allowed in single-path entries, never with a range endpoint or count. Matched folders are collected recursively, as with explicitly selected folders. If a pattern spans several directories, their common parent becomes its working directory.
A count includes the starting item and selects following items of the same type in its directory: `/photos/A.jpg 5` selects five files; `/photos/FolderA 3` selects three folders. Too few remaining items is an error, not a shortened selection. A purely numeric second token is a count; use an absolute path to specify a numeric filename as a range endpoint.
Tab completes the first endpoint from filesystem paths. For the second endpoint, it suggests only names in the first endpoint's directory, restricted to the same type (files or folders). Starting the second endpoint with `/` or `~` completes that directory first, then its matching names. Completion quotes names containing spaces.
Interactive selections must resolve to the same working directory. For paths, ranges, and counts, this is the selected items' parent directory, even when selecting just one folder. All selections are combined into one group; overlapping selections are collected only once. With command-line or dragged paths, the original working-directory rules still apply:
- one dragged folder is the working directory
- multiple dragged files/folders use their last common ancestor as the working directory
The script recursively collects all file types from selected folders and moves the files into one flat folder named `[first stem]-[last stem]` inside the working directory. Existing group folders are reused, and filename collisions receive `-1`, `-2`, etc. Input directory structure is not preserved.
Selected folders and their descendants are removed afterward if empty, including pre-existing empty folder trees and selections containing no files at all. Folders with remaining files are kept. The working directory is never removed.
### `view_metadata.py`
Show all readable metadata for the first file provided:
```bash
python3 view_metadata.py path/to/file/or/directory [...]
```
This script only displays one file. If several files are given, it displays the first one. If a directory is given, it searches recursively and displays the first file it finds.
It runs ExifTool as:
```bash
exiftool -a -u -ee -G0:1 -s file
```
The important choices are:
- `-a` shows duplicate tag names instead of hiding later duplicates.
- `-u` includes unknown tags ExifTool can identify well enough to print.
- `-ee` extracts embedded metadata where ExifTool supports it, which is useful for some video/container formats.
- `-G0:1` shows both the broad metadata container and the more exact location, such as `EXIF:IFD0` or `QuickTime:Track1`.
- `-s` uses compact tag names, which are easier to copy into scripts than the descriptive labels.
The old `-G` style shows broad groups like `EXIF`, `File`, or `QuickTime`. That is readable, but it can hide where inside a container a value came from. Family 1 groups, from `-G1`, show more exact locations like `IFD0`, `ExifIFD`, `MakerNotes`, or `Track1`, but sometimes lose the broader context. `-G0:1` combines both, so it is the most useful default for debugging metadata without changing which normal tags are extracted. The organization option changes labels, not the underlying metadata extraction; `-a` and `-u` are the parts that affect whether duplicate or unknown printed tags are included.
### `copy_metadata.py`
Copy meaningful metadata from one file to another:
```bash
python3 copy_metadata.py
python3 copy_metadata.py source.jpg destination.jpg
```
If run without files, it asks for source and destination paths. If exactly two files are dragged onto it, it asks which one is the source. Any other number of dragged files is an error.
The script copies normal metadata with ExifTool `-TagsFromFile`, while excluding file/system tags, composite tags, image dimensions, pixel dimensions, and resolution fields that should remain specific to the destination file.
### `audio_tool.py`
Extract, remux, or encode audio from audio files and videos:
```bash
python3 audio_tool.py path/to/file/or/directory [...]
```
The tool probes every selected file, warns about files without audio, and leaves those files untouched. When multi-stream inputs have the same stream layout, it offers one stream selection for those inputs; single-stream inputs always use their only stream. Different multi-stream layouts fall back to their first stream with a warning.
The keyboard-driven settings table defaults to encoding. Use Up/Down to select a setting and Left/Right to change choices. Bitrate, quality, FLAC compression, and output directory are edited by typing; bare bitrates such as `192` are treated as `192k`, and invalid values remain open for correction.
Choose `remux` to preserve source audio codecs, or `encode` to produce AAC, Opus, FLAC, or MP3. `Native` keeps each codec's typical audio extension. `MKA` is the broadly compatible universal option. `M4A` is offered only when every relevant codec can be stored in MP4; non-AAC/ALAC M4A combinations are allowed but flagged as less compatible with players.
A blank output directory places each result beside its source. An absolute path collects every result there; a relative path is resolved from the displayed working directory. The working directory is the dragged directory when exactly one directory is supplied, otherwise it is the inputs' last common ancestor. Missing output directories are created when processing starts.
The tool skips, remuxes, or re-encodes sources that already match the requested output according to the selected matching-output policy. Audio-only `.mkv` and `.mka` are treated as the same container family, as are `.mp4` and `.m4a`. Video inputs are always extracted into a new audio-only output. Existing and same-batch name collisions receive `-1`, `-2`, and so on.
AAC presents both the built-in FFmpeg encoder and FDK AAC. FDK AAC uses the normal FFmpeg on `PATH` when it provides `libfdk_aac`. Otherwise the tool asks for an absolute path to an FDK-enabled FFmpeg, verifies it, and remembers that path in the user's Media Batch Tools settings for later runs. It never silently substitutes the built-in encoder when FDK AAC was selected.
### `metadata_editor.py`
Interactively inspect, edit, or remove metadata fields in one or more files:
```bash
python3 metadata_editor.py path/to/file/or/directory [...]
```
The ANSI terminal table includes fields found in the selected files plus common writable fields that may be missing. Differing bulk values are shown as `(multiple values)`; selecting and changing one replaces it in every selected file. Use Up/Down to navigate, type to replace a value, Enter to confirm text, Delete to mark a field for removal, and Esc to cancel editing.
The `GPS coordinates` row accepts decimal degrees or ASCII degrees/minutes/seconds without requiring a degree-sign key:
```text
58.660600, 16.448300
58 39 38.16 N, 16 26 53.88 E
58:39:38.16N, 16:26:53.88E
```
Negative decimal values mean south/west. N/S/E/W are accepted with either form, but contradictory values such as `-58 N` are rejected. The editor writes normalized EXIF GPS coordinate and hemisphere-reference tags.
### `video_encode.py`
Prepare, validate, encode, and remux Avisynth-edited videos:
```bash
python3 video_encode.py path/to/file/or/directory [...]
```
Source videos are untouched until you explicitly choose a final action after encoding. The script recursively collects `.mp4`, `.mov`, `.mkv`, `.mts`, `.m2ts`, and `.avi` files, builds a collapsed drag tree, and creates a workspace in `video_workspace/`.
The workflow is:
1. Probe the source files with `ffprobe` and ExifTool, including frame timing, embedded timezone, location, and HDR metadata.
2. Create editable `.avs` scripts in `video_workspace/` and generated source/validation files in `video_workspace/.source/`.
3. Edit the visible `.avs` files. Existing editable files are kept unless you confirm an overwrite.
4. Press Enter for fast blank-frame validation, `r` for real-frame validation, or `q` to stop after workspace creation.
5. Review encoding settings, encode accepted scripts, then leave outputs in the workspace or move them to the source directories.
Generated source scripts use FFMS2 `FFVideoSource`, mark every frame with source identity properties, and leave the source clip in `last`. Custom Avisynth filters must preserve those frame properties. Validation renders every output frame and rejects scripts that lose or corrupt the identity mapping.
The visible scripts document the available helpers, including `MBT_Drop`, `MBT_DropEvery`, `MBT_Info`, `MBT_AbsoluteTimeStretch`, `Resize`, `RotateCrop`, `MBT_CorrectMatrix`, `MBT_ToSDR`, `Cropf`, `DeShake`, and `FixContrast`. `MBT_Drop` omits frames while preserving their timeline duration; normal operations such as `Trim` and `SelectEven` remove source time. `MBT_AbsoluteTimeStretch()` adjusts the absolute timestamps used for metadata and `MBT_Info()` without changing playback speed.
On Windows, ending a visible `.avs` file with a commented `#32bit` or `# 32-bit` marker selects a bundled/configured 32-bit runner for that script. Linux uses the 64-bit runner.
The bundled runner is used automatically when found. Build it on Linux with:
```bash
tools/avisynth_runner/build_linux.sh
```
On Windows, use the bundled `tools/avisynth_runner/mbt_avs_runner.exe` with installed AviSynth+. Set `MBT_AVS_RUNNER`, `MBT_AVS_RUNNER64`, or `MBT_AVS_RUNNER32` to override runner locations. Generated scripts normally rely on FFMS2 autoload; set `MBT_FFMS2_PLUGIN` to generate `LoadPlugin("...")`, or `MBT_ASK_FFMS2_PLUGIN=1` to ask interactively.
#### Encoding
- x265 is the default encoder. x264, x265, and unchanged-video stream copy are available when compatible with the validated output.
- The tool supports CFR, VFR, trims, playback-speed changes, `MBT_Drop`, normal frame deletion, and mixed deletion/drop edits.
- VFR and normal-deletion output preserves frame timing through timestamp-v2 files and MKVToolNix. x264 receives those timestamps through `--tcfile-in`; x265 receives them during the muxing step.
- Video filenames can use the edited beginning timestamp in a chosen timezone. Video container metadata is written with the edited ending timestamp. Meaningful source metadata, including location, is copied while physical stream properties remain specific to the output.
- Existing output names can be overwritten or given `-1`, `-2`, and so on. Same-batch collisions always receive a suffix.
- Profile and level can be unset, calculated as the per-clip minimum, or selected as a floor. x264 is unavailable for output above 10-bit, and output above 12-bit is rejected.
- MP4 and MKV support AAC, Opus, FLAC, copied audio, or no audio. Stream-copy compatibility is checked before encoding.
Audio is read from the original source by default. An optional `[visible script stem]_audio.avs` is rendered to temporary WAV and used instead when present. AAC audio is trimmed and tempo-adjusted for the validated output. Audio stream copy is limited to whole-video and simple beginning/end trims, so normal frame deletion and speed changes require audio re-encoding or no audio.
After successful encoding, choose to leave files in `video_workspace/`, replace the originals, or move originals to `video_workspace/originals/` before moving the new files to the source directories.
#### HDR
HDR clips remain HDR unless the script calls `MBT_ToSDR()`. HDR-preserving x265 encodes retain validated color properties and static HDR metadata. HDR10+ additionally requires `hdr10plus_tool`; other dynamic HDR formats currently require video stream copy. `MBT_ToSDR()` uses HDRTools and offers `MPC` (default), `Hable`, `Mobius`, `Reinhard`, `ACES`, and `BT2446C`. Tone-mapped output should be visually checked before relying on it for archival work.
#### Non-interactive Encoding
Set `MBT_ENCODE_VIDEO=1` to encode non-interactively after validation. The following optional variables configure that run:
- `MBT_VIDEO_TIMESTAMP_NAMES=1` names outputs as `YYYYMMDD_HHMMSS.mp4`; `0` keeps original-style workspace names.
- `MBT_VIDEO_TIMEZONE=+03:00` selects the timezone used for timestamp filenames. If omitted, one consistent embedded source timezone is preferred; conflicting or missing source offsets fall back to the system timezone. No timezone is needed when timestamp filenames are disabled.
- `MBT_VIDEO_CONTAINER=mp4` or `mkv` selects the output container. MP4 is the default.
- `MBT_VIDEO_CODEC=x264`, `x265`, or `copy` selects FFmpeg `libx264`, `libx265`, or direct source-video remuxing; x265 is the default. `copy` is only available for unchanged validated video timelines.
- `MBT_VIDEO_CRF=16` sets CRF. Interactive defaults are 16 for x264 and 21 for x265.
- `MBT_VIDEO_PRESET=slow` sets the encoder preset.
- `MBT_VIDEO_PROFILE=minimum` and `MBT_VIDEO_LEVEL=minimum` calculate per-clip constraints. A named profile or level acts as a requested floor. Omit either variable to apply no constraint for it.
- `MBT_VIDEO_THREADS=auto` (the default) leaves encoder thread selection automatic. Set it to a positive integer to limit x264 threads or x265 worker pools.
- `MBT_VIDEO_AUDIO=aac`, `opus`, `flac`, `copy`, or `none` selects audio handling. AAC, Opus, and FLAC are valid in either supported container, subject to player compatibility.
- `MBT_VIDEO_AUDIO_BITRATE=192k` sets AAC bitrate.
- `MBT_VIDEO_AUDIO_PITCH=preserve` or `shift` selects whether speed-changed AAC audio keeps pitch or shifts pitch with playback speed. The default is `preserve`.
- `MBT_VIDEO_FINAL_ACTION=leave`, `replace`, or `backup` selects the final disposition.
- `MBT_VIDEO_DELETE_WORKSPACE=1` deletes `video_workspace` after moving outputs.
Missing executables are requested by absolute path for the current run when possible. `mkvmerge` also checks the standard Windows installation path.
## Testing
Run the Python tests with:
```bash
python3 -m unittest discover -s unit_tests
```
The pure Python tests always run. The video end-to-end scenario tests skip unless the local test fixtures and toolchain are available: `tests/`, ExifTool, FFmpeg/FFprobe, FFMS2 for Avisynth, and the bundled `mbt_avs_runner`.
Current video scenarios copy fixture videos to temporary directories before processing and validate:
- CFR MP4 encode with AAC audio and metadata copy
- VFR MP4 trim with copied GPS metadata and adjusted end timestamp
- normal frame deletion such as `SelectEven()` with segmented AAC audio
- mixed normal frame deletion plus `MBT_Drop`
- `MBT_Drop(start,end)` and `MBT_DropEvery(...)` helper behavior
- MKV output with Opus audio
- MP4 video/audio stream-copy remuxing
## Requirements
- Python 3.10 or newer
- ExifTool available as `exiftool` on `PATH`
- FFmpeg available as `ffmpeg` and `ffprobe` on `PATH` for `video_encode.py` and `audio_tool.py`
- MKVToolNix available as `mkvmerge` on `PATH` for VFR or frame-deletion output (Windows also checks its standard installation path)
- the x264 command-line encoder available as `x264` on `PATH` for timestamp-aware x264 output
- `hdr10plus_tool` available on `PATH` only when re-encoding HDR10+ video while retaining HDR
- FFMS2 available to Avisynth as `FFVideoSource` before generated `.avs` scripts can load real videos, either through plugin autoload or a `LoadPlugin` path
- AviSynth+ development files and a C compiler are needed only if building the bundled runner from source
On Debian/Ubuntu-like Linux systems, ExifTool is commonly installed with:
```bash
sudo apt install libimage-exiftool-perl
```
FFmpeg/FFprobe is commonly installed with:
```bash
sudo apt install ffmpeg x264 mkvtoolnix
```
On Windows, install ExifTool and make sure the `exiftool` command is available in a normal terminal. For timestamp-driven video output, also install MKVToolNix; the usual `C:\Program Files\MKVToolNix\mkvmerge.exe` location is detected automatically, otherwise add `mkvmerge` to `PATH`. Timestamp-aware x264 output additionally needs the x264 command-line encoder as `x264` on `PATH`.
No virtual environment is required. The scripts use only the Python standard library plus external command-line tools.
## Safety
`photo_metadata.py` edits files in place, but it prints a preview and asks for confirmation before writing metadata or moving files. Test media should be copied to a temporary directory before running destructive checks.
`video_encode.py` creates files under `video_workspace/`, including encoded outputs when requested. It only modifies source directories or source videos if you explicitly choose a final action that moves outputs back to the source folders.
`tests/`, `old/`, and `video_workspace/` are intentionally ignored by git. The `tests/` directory is for reusable source fixtures that should not be modified directly.
## AI Note
This code was written with AI assistance and should be reviewed and tested before relying on it for important media archives.