Per #4419, we need a way of handling context for translations.
This adds a script to convert translation files at compile time. This means that we can use Format.JS files which support storing descriptions in the <key>.description JSON key, and it will correspond with the <key>.defaultMessage translation value.
Example usage:
"unknown_camera": {
"defaultMessage": "Speed camera ahead",
"description": "TEST"
}
No generative AI used.
I have linted the shell scripts with shellcheck and it's all false positives or preexisting warnings:
⬢ [toolbx comaps]$ shellcheck -o all migrate.sh
⬢ [toolbx comaps]$ shellcheck -o all tools/unix/generate_json_strings.sh
In tools/unix/generate_json_strings.sh line 19:
find translations/ -type f -name 'localize.json' -print0 | xargs -P"${JOBS}" -0 -I'{}' bash -c '
^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.
In tools/unix/generate_json_strings.sh line 21:
jq -c '"'"'map_values(.defaultMessage)'"'"' "1ドル" > "$(echo "1ドル" | sed "s|^translations/|data/|")"
^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.
For more information:
https://www.shellcheck.net/wiki/SC2016 -- Expressions don't expand in singl...
⬢ [toolbx comaps]$ shellcheck -o all configure.sh
... lots of style warnings that were already there
⬢ [toolbx comaps]$
Performance:
⬢ [toolbx comaps]$ time tools/unix/generate_json_strings.sh
Starting...
Finished
real 0m0.485s
user 0m1.704s
sys 0m2.131s
Tested no regressions (~/comapsdata is post-migration and post generate_json_strings.sh, ~/comaps/data has a checkout of main):
⬢ [toolbx data]$ pwd
~/comaps/data
⬢ [toolbx data]$ head -c50 ~/comapsdata/categories-strings/ar.json/localize.json; echo
{"@category_eat":"طعام|مكان لتناول �
⬢ [toolbx data]$ head -c50 categories-strings/ar.json/localize.json; echo
{
"@category_eat": "طعام|مكان لتناول
⬢ [toolbx data]$ head -c50 ~/comapsdata/categories-strings/zh-Hans.json/localize.json; echo
{"@category_eat":"在哪儿吃|食物","@category_
⬢ [toolbx data]$ head -c50 categories-strings/zh-Hans.json/localize.json; echo
{
"@category_eat": "在哪儿吃|食物",
"@catego
⬢ [toolbx data]$ head -c50 ~/comapsdata/categories-strings/en.json/localize.json; echo
{"@category_eat":"Where to eat|eat|Food","@categor
⬢ [toolbx data]$ head -c50 categories-strings/en.json/localize.json; echo
{
"@category_eat": "Where to eat|eat|Food",
"@cate
⬢ [toolbx data]$ find *-strings -type f -exec bash -c 'diff <(jq . {}) <(jq . ~/comapsdata/{})' \;
⬢ [toolbx data]$ find *-strings -type l -exec bash -c 'diff <(jq . {}) <(jq . ~/comapsdata/{})' \;
⬢ [toolbx data]$ cd ~/comapsdata
⬢ [toolbx comapsdata]$ find *-strings -type f -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
⬢ [toolbx comapsdata]$ find *-strings -type l -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
⬢ [toolbx comapsdata]$ sed -i -e 's/Abkhazia/Foo/' countries-strings/en.json/localize.json
⬢ [toolbx comapsdata]$ find *-strings -type f -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
2c2
< "Foo": "Abkhazia",
---
> "Abkhazia": "Abkhazia",
⬢ [toolbx comapsdata]$ find *-strings -type l -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
2c2
< "Foo": "Abkhazia",
---
> "Abkhazia": "Abkhazia",
⬢ [toolbx comapsdata]$
Deployment steps:
-
Make sure there are no unmerged translations and block new translations
-
Rebase and run migrations
Check out this branch, then run in the repository root directory:
#!/usr/bin/env bash # === 1. Move translations to new path to ensure history is retained # porcelain, i know... sorry, linus. git fetch origin main git rebase origin/main # get rid of untracked and ignored files (these actually do exist) rm -rf data git checkout -- data # move data mkdir data/translations mv data/*-strings data/translations/ # commit changes git add data git commit -m "[1/2] Migrate translations" -s # === 2. Delete everything from 1 and actually run the migration rm -rf data git restore --source=HEAD~ data tree data # copy symlinks verbatim (they are all relative) find data/ -type l -name 'localize.json' -exec bash -c ' mkdir -p "$(dirname "$(echo "1ドル" | sed "s|^data/|data/translations/|")")" cp -d "1ドル" "$(echo "1ドル" | sed "s|^data/|data/translations/|")" ' inline-shell '{}' \; # convert and copy data find data/ -maxdepth 3 -type f -name 'localize.json' -exec bash -c ' mkdir -p "$(dirname "$(echo "1ドル" | sed "s|^data/|data/translations/|")")" jq --indent 4 '"'"'map_values({defaultMessage:.})'"'"' "1ドル" > "$(echo "1ドル" | sed "s|^data/|data/translations/|")" ' inline-shell '{}' \; # commit changes git add data/translations git commit -m "[2/2] Migrate translations" -s -
Merge this PR
-
In Codeberg Translate, go to the Operations -> Settings -> Files tab for each affected Component
-
For each one:
- Set File format to Format.JS JSON file
- In Monolingual base language file, change
data/prefix totranslations/ - Restore the old file-format-specific settings if needed: Sort JSON keys=off; JSON indentation=0; Avoid spaces after separators=off; DOS line endings=off
-
Check that translations are synchronizing correctly.
Closes #4419
Per #4419, we need a way of handling context for translations.
This adds a script to convert translation files at compile time. This means that we can use [Format.JS files](https://docs.weblate.org/en/weblate-2026.5/formats/formatjs.html) which support storing descriptions in the `<key>.description` JSON key, and it will correspond with the `<key>.defaultMessage` translation value.
Example usage:
```
"unknown_camera": {
"defaultMessage": "Speed camera ahead",
"description": "TEST"
}
```
No generative AI used.
<hr>
I have linted the shell scripts with shellcheck and it's all false positives or preexisting warnings:
```
⬢ [toolbx comaps]$ shellcheck -o all migrate.sh
⬢ [toolbx comaps]$ shellcheck -o all tools/unix/generate_json_strings.sh
In tools/unix/generate_json_strings.sh line 19:
find translations/ -type f -name 'localize.json' -print0 | xargs -P"${JOBS}" -0 -I'{}' bash -c '
^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.
In tools/unix/generate_json_strings.sh line 21:
jq -c '"'"'map_values(.defaultMessage)'"'"' "1ドル" > "$(echo "1ドル" | sed "s|^translations/|data/|")"
^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.
For more information:
https://www.shellcheck.net/wiki/SC2016 -- Expressions don't expand in singl...
⬢ [toolbx comaps]$ shellcheck -o all configure.sh
... lots of style warnings that were already there
⬢ [toolbx comaps]$
```
Performance:
```
⬢ [toolbx comaps]$ time tools/unix/generate_json_strings.sh
Starting...
Finished
real 0m0.485s
user 0m1.704s
sys 0m2.131s
```
Tested no regressions (`~/comapsdata` is post-migration and post `generate_json_strings.sh`, `~/comaps/data` has a checkout of main):
```
⬢ [toolbx data]$ pwd
~/comaps/data
⬢ [toolbx data]$ head -c50 ~/comapsdata/categories-strings/ar.json/localize.json; echo
{"@category_eat":"طعام|مكان لتناول �
⬢ [toolbx data]$ head -c50 categories-strings/ar.json/localize.json; echo
{
"@category_eat": "طعام|مكان لتناول
⬢ [toolbx data]$ head -c50 ~/comapsdata/categories-strings/zh-Hans.json/localize.json; echo
{"@category_eat":"在哪儿吃|食物","@category_
⬢ [toolbx data]$ head -c50 categories-strings/zh-Hans.json/localize.json; echo
{
"@category_eat": "在哪儿吃|食物",
"@catego
⬢ [toolbx data]$ head -c50 ~/comapsdata/categories-strings/en.json/localize.json; echo
{"@category_eat":"Where to eat|eat|Food","@categor
⬢ [toolbx data]$ head -c50 categories-strings/en.json/localize.json; echo
{
"@category_eat": "Where to eat|eat|Food",
"@cate
⬢ [toolbx data]$ find *-strings -type f -exec bash -c 'diff <(jq . {}) <(jq . ~/comapsdata/{})' \;
⬢ [toolbx data]$ find *-strings -type l -exec bash -c 'diff <(jq . {}) <(jq . ~/comapsdata/{})' \;
⬢ [toolbx data]$ cd ~/comapsdata
⬢ [toolbx comapsdata]$ find *-strings -type f -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
⬢ [toolbx comapsdata]$ find *-strings -type l -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
⬢ [toolbx comapsdata]$ sed -i -e 's/Abkhazia/Foo/' countries-strings/en.json/localize.json
⬢ [toolbx comapsdata]$ find *-strings -type f -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
2c2
< "Foo": "Abkhazia",
---
> "Abkhazia": "Abkhazia",
⬢ [toolbx comapsdata]$ find *-strings -type l -exec bash -c 'diff <(jq . {}) <(jq . ~/comaps/data/{})' \;
2c2
< "Foo": "Abkhazia",
---
> "Abkhazia": "Abkhazia",
⬢ [toolbx comapsdata]$
```
<hr>
Deployment steps:
1. [ ] [Make sure there are no unmerged translations](https://codeberg.org/comaps/comaps/src/branch/main/docs/TRANSLATIONS.md#resolving-conflicts) and block new translations
2. [ ] Rebase and run migrations
Check out this branch, then run in the repository root directory:
```
#!/usr/bin/env bash
# === 1. Move translations to new path to ensure history is retained
# porcelain, i know... sorry, linus.
git fetch origin main
git rebase origin/main
# get rid of untracked and ignored files (these actually do exist)
rm -rf data
git checkout -- data
# move data
mkdir data/translations
mv data/*-strings data/translations/
# commit changes
git add data
git commit -m "[1/2] Migrate translations" -s
# === 2. Delete everything from 1 and actually run the migration
rm -rf data
git restore --source=HEAD~ data
tree data
# copy symlinks verbatim (they are all relative)
find data/ -type l -name 'localize.json' -exec bash -c '
mkdir -p "$(dirname "$(echo "1ドル" | sed "s|^data/|data/translations/|")")"
cp -d "1ドル" "$(echo "1ドル" | sed "s|^data/|data/translations/|")"
' inline-shell '{}' \;
# convert and copy data
find data/ -maxdepth 3 -type f -name 'localize.json' -exec bash -c '
mkdir -p "$(dirname "$(echo "1ドル" | sed "s|^data/|data/translations/|")")"
jq --indent 4 '"'"'map_values({defaultMessage:.})'"'"' "1ドル" > "$(echo "1ドル" | sed "s|^data/|data/translations/|")"
' inline-shell '{}' \;
# commit changes
git add data/translations
git commit -m "[2/2] Migrate translations" -s
```
3. [ ] Merge this PR
4. [ ] In Codeberg Translate, go to the Operations -> Settings -> Files tab for each affected Component
- https://translate.codeberg.org/projects/comaps/countries/
- https://translate.codeberg.org/projects/comaps/search-synonyms-aliases/
- https://translate.codeberg.org/projects/comaps/tts/
5. For each one:
1. Set **File format** to **Format.JS JSON file**
2. In **Monolingual base language file**, change `data/` prefix to `translations/`
3. Restore the old file-format-specific settings if needed: Sort JSON keys=off; JSON indentation=0; Avoid spaces after separators=off; DOS line endings=off
6. Check that translations are synchronizing correctly.
Closes #4419
134 KiB