Translation tools and thoughts

[ian]

I’d like to get the translations under control before there are too many of them. it is a real pain to update translations while developing:

1. Enter a tag for the translation in the base.h file
2. Add the tag and the text to en-us.h (base translation)
3. Repeat for every other translation

It means that things don’t get added as consistently as they should.

I whipped up a little python script that’s just an idea at this point:

h2json.py

Usage: python script.py <base-translation.h> <other-translation.h>
>python h2json.py en-us.h zh-cn.h

This script processes our current .h translation files into JSON. It reads the master translation (en-us.h), populates keys with all available text values from other translation (zh-cn.h), and saves to a simple JSON file (zh-cn.json). Some of our translations (chinese) are old and have missing values, this fills them with default English text and saves everything as JSON for the next step.

json2h.py

>python json2h.py

This script:

• Reads the base translation (en-us.h)
• Creates a new base.h with all the keys from en-us.h (base.ht is the template file)
• Processes all the other .json files in the folder into .h files, filling in with default values from en-us.h where things are missing. (translation.ht is the template file).

Proposed workflow:

1. Developers add a single new tag and text to en-us.h
2. h2json.py runs at compile time (or with a folder change watch), reading en-us.h and creating a matching base.h file. Then translation .json is parsed to .h files with defaults filled for missing values.

I think this would be a lot more usable than the current situation.

The other part of this will be a web browser app that (hopefully) reads translation .json files from our github repo for easy online translation additions and updates.

I pushed the scripts to a new branch, but it’s really small so I attached it here too. If you’re able to try out a python script please share your thoughts.

translation.zip (39.6 KB)

[ian]

One last thing for the night. Here’s a modern javascript webpage (probably needs to be run from a server like laragon) that fetches the .json from github and shows it in rows for editing.

Click download JSON to get a .json file straight from the browser. (sample data only at the moment).

TODO

• Create new translation option (what to name download?)
• Reload JSON option (resume work? save in browser indexDB?)
• show base values alongside translation
• enter URL of file to work on (load)

test.zip (1.6 KB)

[ian]

Translations can now be created and edited in your browser.

• Open the translation web page
• To edit an existing translation choose it from the Load Translation File drop down menu
• To create a new translation, choose en-us.json (the base translation)
• Translate the text fields
• Click Download JSON to download your translation text as a .json file
• Post the .json file in the forum, or submit a pull request on GitHub

Of course, once you have a JSON file you can edit it in a text editor.

The web app pulls straight from the firmware repo. Changes to the repo show up immediately.

I added a bit of documentation.

[ian]

Does anyone know how to handle pointers to complex variables?

void translation_set(uint32_t language)
{
    switch(language)
    {
        case 2:
            t = (char **) &bs_ba;
            break;         
        case 1:
            t = (char **) &pl_pl;
            break;        
        /*case 2:
            t = (char **) &zn_ch;
            break;*/
        case 0:
        default:
            t = (char **) &en_us;
            break;
    }
}

Right now there is an ugly switch statement that assigns a pointer to the right array of translations.

static char const * const en_us={.....}

This is the type of the array of translations.

(some type) translations{
&en_us,
&pl_pl,
&bs_ba
}
t = (char **) &translations[language];

I want to put the translation pointers into a struct and then use the index to assign the t pointer.

Pointers are first thing in the morning work though, so I’ll leave it here in case anyone knows.

Bug report

Forgot: language setting is saved, but it is not reloaded on restart. I’ll fix that.

[LouD]

@ian for the wtfos configurator we used crowdin.com to have the community help create translations.
example: Translating WTFOS Configurator to Dutch language - Crowdin

[ian]

Wow, I’m really glad to see .json is an acceptable format for localization. This is such great stuff.

Is that a site/app that I can use to host our own json files for localization? Are there any developer tools to integrate it?

It used to be such a pain to add new text to translation - make a key, then update every translation with that key plus default value. What happens in practice is that everything but english is commented out and falls behind.

I did some updates with our new python script. Only add an entry in english.h, the tags file and the other translations are automatically generated in a fraction of a second. And, it generates .h files so there’s no need to setup an extensive toolchain just to get our source compiling.

[LouD]

@ian I made a quick example project for the en-us.json file here:
https://crowdin.com/editor/test-project-for-bp5/4/en-nl?view=comfortable&filter=basic&value=0

Ps. there is a 14 day trial to this site before you have to pay

[LouD]

As example I translated a couple of texts and downloaded the ‘Dutch’ json:

Easy peasy lemon squeezy

[henrygab]

There is an issue with the prior design of the JSON localization.

Steps to repro:

1. Find a string in one of the translations that isn’t actually translated yet (identical to EN-US string)
2. Edit the EN-US string for that existing string ID

Expected results:

The translations which have not actually translated the string get the updated EN-US string displayed.

Actual results:

Once an EN-US string is copied to the translation JSON, it is never updated, even if the EN-US string is updated.

Fix

Any non-translated string should be removed from the translation’s JSON.

Any editing webpage should load the EN-US as well as the translation JSON. If the string exists only in EN-US, it’s not been translated. The buspirate firmware was already updated to automatically load the EN-US string in such a case.

[ian]

This is a good point. So one update to the python script to skip instead of fill, and an update to the little web app to use en-us as the base over the translation file.

[henrygab]

Yes … and two more manual steps are needed:

1. Automatic removal of matching English strings from translation files
2. [optional] Manual review of remaining strings in translation files, to remove English strings that were not automatically removed.

[henrygab]

Not sure how to best tackle this, but there’s a third issue that will eventually arise related to format arguments… See GitHub issue 185 for details on issue and recommended way to improve.