Multibyte string on string::string_print

[hyeshik]

BPO	479898
Nosy	@gvanrossum, @loewis, @hyeshik
Files	python_mbstring_diff.txt: patch to Objects/stringobject.c configure.in.diff.txt: 2nd) autoconf detect for mbtowc(), iswprint() pyconfig.h.in.diff.txt: 2nd) autoconf detect for mbtowc(), iswprint() stringobject.c.diff.txt: 2nd) new clean(on my view) patch for Objects/stringobject.c mb3.diff: 3rd) revised (includes patch for stringobject.c, configure.in and pyconfig.h.in)

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2002-10-11.05:38:32.000>
created_at = <Date 2001-11-09.07:10:11.000>
labels = ['interpreter-core']
title = 'Multibyte string on string::string_print'
updated_at = <Date 2002-10-11.05:38:32.000>
user = 'https://github.com/hyeshik'

bugs.python.org fields:

activity = <Date 2002-10-11.05:38:32.000>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2001-11-09.07:10:11.000>
creator = 'hyeshik.chang'
dependencies = []
files = ['3743', '3744', '3745', '3746', '3747']
hgrepos = []
issue_num = 479898
keywords = ['patch']
message_count = 10.0
messages = ['38131', '38132', '38133', '38134', '38135', '38136', '38137', '38138', '38139', '38140']
nosy_count = 3.0
nosy_names = ['gvanrossum', 'loewis', 'hyeshik.chang']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue479898'
versions = []

[hyeshik]

Many multibyte language users are difficult to see
native characters on list or dictionary and etc.
This patch allows printing multibyte on UNIX98-
compatible machines; mbtowc() is ISO/IEC 9899:1990
standard C-API function.

[loewis]

Logged In: YES
user_id=21627

Even though I think this patch is correct in principle, I
see a few problems with it:

1. Since it doesn't fix a bug, it probably cannot go into 2.2.
2. There is no autoconf test for mbtowc. You should test
   this in configure, and then conditionalize your code on
   HAVE_MBTOWC.
3. There is too much code duplication. Try to find a
   solution which special-cases the escape codes (\something)
   only once. For example, you may implement a trivial mbtowc
   redefinition if mbtowc is not available, and then use mbtowc
   always.

[gvanrossum]

Logged In: YES
user_id=6380

I don't understand the point of using mbtowc() here.

The code extracts a wide character, but then it uses
isprint() on it, and as far as I know, isprint() is not
defined on wide characters, only on 'unsigned char' (and on
-1).

Isn't what the author wants simply to is isprint(c) instead
of (c < ' ' || c >= 0x7f)???

[loewis]

Logged In: YES
user_id=21627

You are right, the code should use iswprint instead.

The point is that multiple subsequent bytes can make up a
single printable character. Not every character above 127 is
necessarily printable (e.g. in Latin-1, only characters
above 160 are printable). Likewise, a single byte may not be
printable, but a combination will print fine. So this code
is supposed to catch only those cases where printing will
actually work.

[hyeshik]

Logged In: YES
user_id=55188

Yes, it should be changed to iswprint on Linux systems.
(but, isprint of BSD systems was designed for wide
characters)
As loewis told, EUC codes of Korea, Japan, Taiwan doesn't
use 0x7F-0x9F for printable character. So, I think that
using mbtowc is unavoidable.

[gvanrossum]

Logged In: YES
user_id=6380

Still, the patch as it exists is unacceptable -- it needs
configure support to decide whether to use mbtowc() and
whether to use iswprint() or isprint() (I would hope on BSD
there is also an iswprint(), to be standard-conforming).

[hyeshik]

Logged In: YES
user_id=55188

I uploaded 2nd patches which contains configure support.
Unfortunately, Citrus(new generation locale support for
*BSDs) didn't implemented iswprint() yet. but *BSDs
supports wide character via Rune Locale isprint() func.

[hyeshik]

Logged In: YES
user_id=55188

Oops, one mistake. sorry.

stringobject.c:646

else if (_ISPRINT(c)) {
-> 
else if (cr > 0 && _ISPRINT(c)) {

(to detect whether mbtowc failed to convert)

[loewis]

Logged In: YES
user_id=21627

Thanks for the patch, committed as

configure 1.343;
configure.in 1.354;
pyconfig.h.in 1.51;
stringobject.c 2.190;

I'm not quite sure that your correction is correct: If we
invoke iswprint, cr is already guaranteed to be >0, since we
otherwise goto nonprintable.

[loewis]

Logged In: YES
user_id=21627

The patch was causing too many problems, so I had to back it
out.