TM Manager is a tool devised to clean tmx files by removing damaged, deprecated or noisy TUs (Translation Units).

Cleaning a TM

1. Go to TM Manage on the Console. A default starting page will open.

A basic configuration of preset filters will be available to start with.
It is possible to cut two types of subsets from the original tmx:
1. Sample size: type the number of segments that will be copied from the original TMX file into the cleaned file.
2. Test set size: type the number of segments that will be chosen randomly and extracted from the TMX file. The final, cleaned TMX will no longer include these TUs. It can be used later for automatic scores calculation.

Both settings are optional.

Click on Filters to display all available rules for cleaning the tmx.

Choose any relevant filter to be applied to clean the TM by selecting the checks (see TM Manager for a detailed description of available filters)

If you select a filter configuration that you want to reuse for future cleanings, you can save the selection by clicking on Save new preset

Give your congifuration a name, and a new button with your saved settings will appear in the Preset section

Upload the file to be cleaned by clicking the Upload files button.

In the Upload window:
1. select source and target language
2. upload the tmx file to be cleaned
3. (Optional) upload a reference glossary

Click the Start button to begin the cleaning.
After process is finished, a notification will be shown to download the files and you will also receive an email with a link to download the file.

Cleaning Results

The cleaning results will be provided as a zip file containing:

clean.tmx: the cleaned tmx, i.e. the new tmx which will not contain the removed TUs
duplicates.txt: a file containing a list of duplicates that were deleted from the final cleaned tmx
filtered.xlsx: a detailed list of all TUs that were deleted from the final tmx and the reason why (i.e. the name of the filter that trigged the removal, segment by segment)
report.yaml: a numeric analysis of the status of the TM, detailing how many TUs were deleted and why (i.e. the name of the filter that trigged the removal, counting how many segments were removed for that reason)
sample.tmx: a file with copied segments (see point 3)
test_set.tmx: a file with the subset of TUs for automatic scoring (see step 3)

Filters

Filters remove TUs from the original TMX file. The following section provides a detailed description of the elements that each filter will remove.

Filter	Description	Recommendations

Filter	Description	Recommendations
no_empty	Segments with empty source or target.	Default for clean TM. Essential for removing incomplete segments.
not_too_long	Segments containing more than 512 characters.	Identifies long segments that might be difficult for translators or cause issues with CAT tools.
not_too_short	Segments containing less than 3 words.	Identifies very short, potentially unhelpful segments (e.g., single words, numbers, or punctuation marks).
length_ratio	Segments in which one side is 70% longer than the other.	Identifies segments where the length disparity between source and target is significant, indicating possible omissions or expansions.
no_identical	Target same as source.	Default for clean TM. Crucial for removing segments where the source and target are identical, which can inflate TM size with irrelevant data.
no_literals	Segments containing any of the following: "Re:", "{{", "%s", "}}", "+++", "***", ’=\".	Identifies segments that might contain code snippets, placeholders, or email subject line indicators.
no_breadcrumbs	Segments containing more than 2 breadcrumbs characters.	Identifies segments that might contain navigation paths or structural elements that are not true translatable content.
no_repeated_words	Segments containing consecutive repeated words.	Default for clean TM. Crucial for removing segments with unintentional word repetitions, often indicative of errors or poor quality.
no_unicode_noise	Segments containing too many characters from weird Unicode sets.	Identifies segments with corrupted or irrelevant Unicode characters, which can indicate encoding issues or data corruption.
no_space_noise	Segments containing too many consecutive spaces.	Identifies segments with excessive or inconsistent spacing, which can affect TM matching and consistency.
no_paren	Segments containing too many parentheses or brackets.	Identifies segments that are overly complex due to excessive use of parentheses or brackets, which might indicate structural or formatting issues.
no-escaped_unicode	Segments containing unescaped Unicode.	Identifies segments with unescaped Unicode, which can cause improper character rendering and display issues in various tools and platforms.
no_bad_encoding	Segments containing mojibake.	Default for clean TM. Crucial for removing segments with unreadable characters due to incorrect encoding.
no_wrong_language	Segments whose source or target are not in expected language.	Default for clean TM. Crucial for maintaining language integrity within your TM and ensuring that segments are correctly categorized by language.
no_only_numbers	Segments containing more than a 50% of numbers.	Identifies segments that are primarily numerical, as these often don't require translation and can inflate TM size.
no_urls	Segments containing URLs.	Identifies segments that are primarily URLs, which may not be translatable content and can affect TM matching.
no_titles	Segments in which all words are uppercased or title cased.	Identifies segments that might be headings or proper nouns.
no_only_symbols	Segments that consist of more than a 90% of non-alphabetic characters.	Identifies segments that are primarily made up of symbols, which are often not meaningful translatable content.
no_script_inconsistencies_sl	Segments containing characters from different scripts / writing system in source.	Identifies segments with script inconsistency within source language, which can impact font rendering and text processing.
no_script_inconsistencies_tl	Segments containing characters from different scripts / writing system in target.	Identifies segments with script inconsistency within target language, which can impact font rendering and text processing.
no-explicit text	Segments containing porn-like language. Trained on porn video content.	Use with caution and discretion. Identifies offensive or inappropriate content. Its effectiveness depends on the specific data, and it may require further manual review to ensure no false positives.
sensitive_data	Segments containing data similar to personal data.	Use with caution and discretion. Identifies sensitive personal information. Its effectiveness depends on the specific data, and it may require further manual review to ensure no false positives.
inc_transl_sl	Segments containing translation inconsistencies in source.	Identifies segments with potential inconsistencies within source segments, which might indicate issues with original content creation or previous processing.
inc_transl_tl	Segments containing translation inconsistencies in target.	Identifies segments with potential inconsistencies within target segments.
no_number_inconsistencies	Segments with mismatching numbers.	Default for clean TM. Crucial for ensuring numerical consistency between source and target, preventing common translation errors.
dedup	Segments containing duplicates or near duplicates of other TU.	Default for clean TM. Highly recommended for reducing TM size and improving lookup efficiency by removing redundant entries.
no_glued_words	Segments containing too many uppercased characters between lowercase characters in source or target.	Identifies instances of "glued words" or improper capitalization that can affect readability and machine processing.
lm_filter	Segments with a low fluency score from language model. Considers the entire segments and its fluency, grammar, etc. Use with care on short segments.	LLM-based. High computational cost. Use for longer segments where fluency and grammar are critical. Avoid on short segments as it may yield less reliable results.
bifixer	Fixes mojibake, turns HTML entities into characters they represent, replaces characters from wrong alphabets with the correct ones, normalizes punctuations and spaces, fixes common orthographic error for supported languages.	LLM-based. High computational cost. A powerful tool for comprehensive text normalization and error correction. Use when significant text cleaning is required, but be mindful of its processing time.
bicleaner	Alignment check by calculating mutual translation probability score based on	Identifies segments with a low likelihood of a pair of sentences being mutual translations, i.e. wrong alignments.