IT TN Fixes for #166 by zoobereq · Pull Request #183 · NVIDIA/NeMo-text-processing

zoobereq · 2024-06-14T18:27:01Z

What does this PR do ?

A fix for issue #166 reported for Italian TN. Fixes the bug where the sentence-final period in sentences ending with domain is incorrectly normalized as part of the domain. The PR also includes support for social media tags, updated tests and a fix to test_sparrowhawk_normalization.sh which blocked Sparrowhawk testing.

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

for more information, see https://pre-commit.ci

tbartley94 · 2024-07-15T00:35:03Z

@@ -1,5 +1,3 @@
-Avv.~avvocato


Whhy deletion?

tbartley94 · 2024-07-15T00:35:23Z

-        domain = pynutil.delete("domain: \"") + domain + pynutil.delete("\"")
+        domain = pynutil.delete('domain: "') + domain + pynutil.delete('"')
        protocol = (
-            pynutil.delete("protocol: \"")


just alias protocol as its own variable

tbartley94 · 2024-07-15T00:35:31Z

        domain = convert_defaults + pynini.closure(insert_space + convert_defaults)
        domain @= verbalize_characters

-        domain = pynutil.delete("domain: \"") + domain + pynutil.delete("\"")


alias domain as its own variable

tbartley94 · 2024-07-15T00:35:40Z


        verbalize_characters = pynini.cdrewrite(graph_symbols | graph_digit, "", "", NEMO_SIGMA)

-        user_name = pynutil.delete("username: \"") + add_space_after_char() + pynutil.delete("\"")


alias username as its own variable

tbartley94 · 2024-07-15T00:36:31Z

-            + pynutil.insert("\"")
-        )
+        # email
+        username = pynutil.insert('username: "') + all_characters + pynutil.insert('"') + pynini.cross("@", " ")


alias username as its own variable

tbartley94 · 2024-07-15T00:36:47Z

        protocol = protocol_start | protocol_end | (protocol_start + protocol_end)
-        protocol = pynutil.insert("protocol: \"") + protocol + pynutil.insert("\"")
-        graph |= protocol + insert_space + (domain_graph | domain_common_graph)
+        protocol = pynutil.insert('protocol: "') + protocol + pynutil.insert('"')


alias protocol as its own variab.e

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

for more information, see https://pre-commit.ci

+from nemo_text_processing.text_normalization.en.graph_utils import (
+    NEMO_ALPHA,
+    NEMO_DIGIT,
+    NEMO_SPACE,
+    GraphFst,
+    at,
+    colon,
+    domain_string,
+    double_quotes,
+    double_slash,
+    file,
+    http,
+    https,
+    period,
+    protocol_string,
+    slash,
+    triple_slash,
+    username_string,
+    www,
+)


Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

for more information, see https://pre-commit.ci

ekmb · 2024-07-19T05:46:04Z

@tbartley94 , @zoobereq this PR changes en/graph - CI folder for Eng should've been updated
Eng SH tests are failing afeter this commit https://github.com/NVIDIA/NeMo-text-processing/commits/main/

This reverts commit d83577c.

tbartley94 · 2024-07-19T18:20:01Z

@ekmb How do I open a revert with a signature sign off? Trying to revert the branch but it's triggering Dco failures.

This reverts commit d83577c. Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Fixes for #166 Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Implements aliases for common string literals Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleans up unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <alexcui1994@gmail.com>

* Fixes for NVIDIA#166 Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Implements aliases for common string literals Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleans up unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

* Fixes for NVIDIA#166 Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Implements aliases for common string literals Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleans up unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fixes for #166 Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Implements aliases for common string literals Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleans up unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

zoobereq and others added 2 commits June 14, 2024 14:22

Fixes for #166

f35361b

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

e2968ff

for more information, see https://pre-commit.ci

zoobereq changed the title ~~Fixes for #166~~ IT TN Fixes for #166 Jun 14, 2024

zoobereq requested a review from tbartley94 June 26, 2024 15:29

zoobereq marked this pull request as ready for review June 26, 2024 15:29

zoobereq requested a review from mgrafu July 10, 2024 15:10

tbartley94 requested changes Jul 15, 2024

View reviewed changes

zoobereq and others added 2 commits July 16, 2024 12:00

Implements aliases for common string literals

9b272e7

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

20cc8cf

for more information, see https://pre-commit.ci

github-advanced-security AI found potential problems Jul 16, 2024

View reviewed changes

zoobereq and others added 2 commits July 16, 2024 12:07

Cleans up unused imports

ea2b7b8

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

03317b7

for more information, see https://pre-commit.ci

tbartley94 approved these changes Jul 16, 2024

View reviewed changes

zoobereq removed the request for review from mgrafu July 17, 2024 13:04

zoobereq merged commit d83577c into main Jul 17, 2024

ekmb deleted the IT-TN-Fixes branch July 19, 2024 00:31

tbartley94 added a commit that referenced this pull request Jul 19, 2024

Revert "IT TN Fixes for #166 (#183)"

b76a582

This reverts commit d83577c.

tbartley94 mentioned this pull request Jul 19, 2024

Revert "IT TN Fixes for #166" #190

Closed

zoobereq added a commit that referenced this pull request Jul 23, 2024

Revert "IT TN Fixes for #166 (#183)"

d5f65fc

This reverts commit d83577c. Signed-off-by: Simon Zuberek <szuberek@nvidia.com>


		verbalize_characters = pynini.cdrewrite(graph_symbols \| graph_digit, "", "", NEMO_SIGMA)

		user_name = pynutil.delete("username: \"") + add_space_after_char() + pynutil.delete("\"")

Conversation

zoobereq commented Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Before your PR is "Ready for review"

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zoobereq Jul 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Check notice

ekmb commented Jul 19, 2024

Uh oh!

tbartley94 commented Jul 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zoobereq commented Jun 14, 2024 •

edited

Loading

zoobereq Jul 16, 2024 •

edited

Loading