Roadmap for the post and term locales proposal

Here’s what I feel we need to do to investigate the proposal to add locales to posts and terms, and to test whether this is feasible and desirable.

Write a proof of concept plugin which:

  • Implements a new column in wp_posts and wp_terms
  • Implements an enhancement to WP_Query

Then test this against some existing multilingual and non-multilingual plugins, looking for:

  • Performance
  • Any bugs or unexpected behaviours introduced

I also think it would be good to look at how locale information for posts and terms might be exposed via the proposed WordPress REST API endpoints (I knowledge or preconceptions here).

I’m happy to have help on this journey.

The next multilingual WordPress weekly…

The next multilingual WordPress weekly chat is on Monday 30 November at 17:00 GMT

We’ll be talking about:

We’ve got leaders for the above topics/proposals, and we need help in the following areas:

  • Developers to build, review, and discuss implementations of these ideas
  • Others with an active interest to help understand the shape and size of the community in terms of statistics and usage

We’re not discussing or critiquing existing multilanguage and translation plugins… instead we’re talking about how WordPress core can make life easier for the developers (plugin developers, theme developers, site developers) and users of WordPress sites using multiple languages.

See you there! 🙂

New Proposal for handling strings languages: Introduction

This proposal is intended to make easier to develop multilingual-ready themes and plugins. As 98% of WordPress sites are single-language, our goal is to enable multilingual functionality, while avoiding complexity for single-language sites.

The intention is to allow themes and plugins to indicate what elements, which they create, will require translation on multilingual sites. Different multilingual plugins can use that information and implement the multilingual operation in whatever way they choose.

Custom post types and taxonomy

When running a multilingual site, custom posts and taxonomy can be ‘translatable’ or not. If items of a custom type are translatable, the user expects to see different content per language. In some cases, even if the site is multilingual, certain CPTs should be language-indifferent. For example, if a site offers ‘downloads’, they will probably be the same, no matter in what language visitors browse the site.

Custom fields and ‘option’ entries

Custom fields and ‘options’ are a bit more complex than CPTs. Custom fields may require translation, may need to be the same for all languages or should be language-indifferent.

How to set language information

Developers will have two options to indicate if elements require translation:

  • Using JSON files
  • Using filters

JSON files allow to pre-compile the language information for elements. These files offer zero overhead during runtime for single-language sites. When running in a multilingual site, the multilingual plugin will read these JSON files and ‘know’ what elements require translation. JSON files are best for ‘static’ elements, which the theme or plugin always create. For example, a real estate theme may create CPTs for ‘properties’ and some fields for properties. A JSON file in the theme folder will indicate that these elements require translation.
Filters are best for dynamic elements, which cannot be determined when developing the code. They add minimal overhead during runtime (to go through filters that nothing uses). A plugin that creates field according to user inputs will need to use these filters to indicate language information, as these fields are not known when coding the plugin.

Example – announcing translatable texts in WooCommerce

The following example, shows a configuration file for translating post meta and ‘option’ entries in WooCommerce.

This would be the JSON file for WooCommerce:

{
  "post-meta": [
      {
        "action": "copy",
        "name": "_backorders"
      },
      {
        "action": "copy",
        "name": "_pricing_rules"
      }
      {
        "action": "translate",
        "name": "_crosssell_ids"
      },
      ...
    ]
  ,
  "options": [
      { "name": "woocommerce_shop_page_title" },
      {
        "name": "woocommerce_new_order_settings",
        "key": [
          { "name": "subject" },
          { "name": "heading" }
        ]
      },
      ...
      { "name": "woocommerce_demo_store_notice" },
    ]
}

You can see that the custom fields ‘_backorders’ and ‘_pricing_rules’ need to be copied between languages. The field ‘_crosssell_ids’ is user-translatable.

We can implement the same using API calls, like these:

// Get the theme option
$cross_sell_ids = get_option( 'my_theme_footer_text' );
$cross_sell_ids = get_translated_string( 'option', 'my_theme_footer_text', $cross_sell_ids );
// Or...
$cross_sell_ids = apply_filters( 'translated_string', get_option( 'my_theme_footer_text' ), 'option', 'my_theme_footer_text' );

You can read the complete technical proposal in the following Google Doc: https://goo.gl/Ynp1ek.

Current status: need help :)

Read our initial intentions post

As I said in the WordPress Core slack channel: We’ve lost @markoheijnen‘s involvement I think permanently, and @sciamannikoo is taken up with work issues so not able to give time (currently) to the proposal on translating data stored in options, similarly I’ve not had the time to devote to the post/term locale project ​although I still intend to​ 😦

So:

  • We’re looking for more people to get involved
  • We’re taking a break this week and next, back Monday 26 October, although we’ll pop in on the intervening Mondays not for a full chat but to keep the cadence and see if anyone is around who wants to join in
    …ping me or @jennybcappaert if you’re interested in helping out, please 🙂

#core-channel, #core-multilingual

Core Multilingual Support chat notes,…

Core Multilingual Support chat notes, Monday 14 September 2015.
Transcript starts at 16h UTC.

@sc0ttkclark joined us in the discussion after giving @markoheijnen and I admin access to the WP Compare doc. As he’s working on the Fields API project, he was inquiring as to whether there is anything with regard to locale and the core propositions that we’re working on that he should be aware of / that he might take into special consideration.

i assume you have the same concerns with Customizer API

We have not, as of yet, discussed our current proposals/research in terms of other APIs.

Summary:

  1. What can be localized and translated through po/mo should be.
  2. Where fields are dependent posts they will inherit locale.
  3. Are there cases where fields aren’t dependent on posts?
  4. Are there cases where fields shouldn’t inherit locale?

 

Core Multilingual Support chat notes,…

Core Multilingual Support chat notes, Monday 31 August 2015.
Transcript starts at 16h12 UTC.

This was a bank holiday in England, but brief discussion was had about:

  1. @chouby mentioning during previous meeting working on a plugin that uses taxonomies  to define locale for posts (and possibly terms). Would be great to get feedback from him about his progress and what challenges he’s run into with it.
  2. My idea to tack on a fourth mission to this project: data. Following Matt’s comments at WCEU about the lack of data, I’m going to start gathering some to use toward a base argument for our general proposal.

@alvarogois popped in at the end of the meeting to say he’s pinged Alessandro Senese about this project and our weekly meetings – author of another plugin we haven’t yet discussed, Ceceppa Multilingua.

Defining a post or term language

I’m going to try to set this out in stages, so people can take issue with each or any particular stage. 🙂

WordPress provides no way to define a locale for posts and terms. This means that plugin authors must define one, which adds complexity to all content queries during the actions and filters run for each request, and may also add complexity to the data structures. The lack of a built-in API for locale definition additionally complicates saving content, though that is significantly less impactful that the content retrieval issues.

Examples:

  • WPML stores locale in an additional table, and this table is joined to allow retrieval of the content in the correct language
  • Polylang uses a taxonomy to store locale, and this taxonomy is referenced in queries for the content
  • Babble uses a parallel post type or taxonomy for each language, which complicates the data structure

Each of the above solutions is further unique to that particular plugin, which means that any other plugin author, theme author, or developer, who wants to act on content of a given type is presented with a complications specific to each; what language is the post in, what post type represents the language for this post, etc.

In broad outline, I propose:

  • WordPress core provides an API to allow a locale to be defined for a post or term, the locale should default to “unspecified”, and data structure(s) to store this
  • WordPress provides an API to allow content queries, for posts and terms, to request content with a specific locale

Currently I envisage the implementation as:

  • wp_insert_post and related functions accept a parameter to allow a locale to be defined for a post or term, the locale should default to “no locale”
  • The locale designation is stored in an additional column in wp_posts and wp_terms respectively
  • wp_query allows an additional parameter to specify content from a particular locale

One thing which has occurred to me: If some content gets assigned a locale, but other content does not, (e.g. if a multi-locale aware plugin has been active and is not inactive) what content should be retrieved? It’s possible the desired behaviour for a content query on a site which has no interest in multiple locales is different to that of a site which is aware of multiple locales.

Translating taxonomy

There are two parts for this. The strings/labels and the taxonomy name itself.

Labels and slugs

This is relative easy since you can use the i18n function __(). Which is also stated in the comments in “News from Polylang“. However this can become more tricky when having multiple languages on the same installation like Babble, WPML, Polylang etc. This since it’s part of the rewrite rules which are stored in the option table and because of that have no idea of the language context. When using Multilingual Press then this isn’t an issue and the __() function is all you need to do.

”Translating” the taxonomy name

With the name I mean the first argument of register_taxonomy() and not the key ‘name’ of $labels. In general you don’t need to do anything with this. The question is all how is a plugin going to translate the terms and in case of Babble they do it with ghost taxonomies. And because of that they do need to “translate” the taxonomy name as in ‘categories’ will become ‘categories_de_de’. Now it does become tricky since WordPress doesn’t support filtering that name to something else in functions like get_the_category_list() or it equivalents get_the_tag_list() and get_the_term_list().

The functions itself also don’t have the same function signatures. Here the code of WordPress need to be analysed to see how this filtering can be added and how in other cases we can make it easier for plugins to translate terms. So for babble that would mean adding filters in places where we want to change the taxonomy name.

Next steps

I haven’t had the time to dive into every single plugin and will start doing that. With the main goal is to check if plugins support the translation and how the support it. I will try to write something up in the next 2 days and also see how it can be added to Compare WP spreadsheet for multilingual plugins.

Core Multilingual Support chat notes,…

Core Multilingual Support chat notes, Monday 3 August 2015.
Transcript starts at 16h UTC.
Discussion centered around Andrea’s solution (below) for Proposal 2—translating for non gettext strings—in which he proposes a filter.
Marko has added the need to wrap this filter in a function, to account for and avoid human error when applying the filter (comment also below).
Marko also stated that he would have a draft for Proposal 3 by the next meeting (today!)

Jenny notes that she is better at facilitating meetings than at writing up meeting notes.

Proposal: filter for translating any non gettext string

Options and user meta may need to be translated in many cases.

Let’s consider 3 scenarios:

  1. Site settings such as Site Title, Tagline
  2. Theme settings such as a footer disclaimer
  3. User bio

1 is stored in wp_options and likely is the same for 2 (if author is using $wp_customizer or wp_options).

3 is stored in wp_usermeta.

Very similar to how gettext function work, we could consider to define a convention for a new filter.

WordPress core won’t need any change on its code for implement this, because this would have to be implemented by themes and plugins authors (or multilingual plugins).

This filter will need 2 arguments:

  1. The string to filter (of course)
  2. An associative array
    The idea of using an associative array rather than a given list of arguments, lies on the fact that this could allow to extend the amount and type of data this filter can receive.
    Also this helps keeping the signature small.
    • domain: is the equivalent of a text-domain
    • context: as for gettext, we may need to provide a context
    • name: this is a way to identify the string we want to translate: not necessarily needed, but if we store these translation somewhere, we most likely need that)

I still have no ideas of what this filter handle could be named (suggestions are welcome!), but let’s suppose it will be ‘translate_string’.

Possible implementations of the 3 scenarios could be:

  1. WP domain:
    • $translated_blogname = apply_filters( 'translate_string', get_bloginfo('name'), array( 'domain' => 'WP', 'name' => 'blogname' ) );
    • $translated_tagline = apply_filters( 'translate_string', get_bloginfo('description'), array( 'domain' => 'WP', 'name' => 'blogdescription' ) );
  2. Theme domain (where domain would be the theme slug):
    • $translated_footer_disclaimer = apply_filters( 'translate_string', $wp_customize->get_setting( 'footer_disclaimer' ), array( 'domain' => 'theme-slug', 'name' => 'footer_disclaimer' ) );
  3. user-meta domain (here I’m making up a custom domain):
    • $translated_user_bio = apply_filters( 'translate_string', get_usermeta($user_id, 'description'), array( 'domain' => 'user-meta', 'context' => $user_id, 'name' => 'description' ) );