mapit_global_import

hello,
when i try to import with ./manage.py mapit_global_import --verbosity 2 --commit data/cache-with-political/
i've got this error behind, is there a solution to correct it easily ?

WARNINGS:
?: (1_8.W001) The standalone TEMPLATE_* settings were deprecated in Django 1.8 and the TEMPLATE S dictionary takes precedence. You must put the values of the following settings into your defa ult TEMPLATES dict: TEMPLATE_CONTEXT_PROCESSORS, TEMPLATE_LOADERS.
Loading any admin boundaries from data/cache-with-political/
Finding language codes...
Loading type O02
Loading all KML in O02
[0% complete] Loading /var/www/xxxxxxxxxx
Ecuador.kml
Ecuador
No area existed in the current generation with that OSM element type and ID
Area ID: 1
[0% complete] Loading /var/www/mapit.fixmystreet.fr/mapit/data/cache-with-political/O02/relatio n-1152720-ÐбÑ
Ð°Ð•Ð¸Ñ - ÐÒ§ÑнÑ.kml

UnicodeEncodeError: 'ascii' codec can't encode characters in position 21-27: ordinal not in range(128)

[dracos]

Sorry to hear you've had problems. Could you provide the entire traceback? It should include the line number that's going wrong. It looks like it's the print line of the country name, but I can't be sure without the full details.

One possibility, could you try downgrading Django to 1.7 and seeing if it runs okay then? I'm not sure what difference that would make, but it's possible that import script hasn't been tested under 1.8 and there is some subtle difference that needs looking at.

It should work okay, here's that area imported in our installation: http://global.mapit.mysociety.org/area/687137.html
We need to update our own installation, so I will set that running now and see if I come across any similar issues.

./manage.py mapit_global_import --verbosity 2 --commit data/cache-with-political/

System check identified some issues:

WARNINGS:
?: (1_8.W001) The standalone TEMPLATE_* settings were deprecated in Django 1.8 and the TEMPLATE S dictionary takes precedence. You must put the values of the following settings into your defa ult TEMPLATES dict: TEMPLATE_CONTEXT_PROCESSORS, TEMPLATE_LOADERS.
Loading any admin boundaries from data/cache-with-political/
Finding language codes...
Loading type O02
Loading all KML in O02
[0% complete] Loading /var/www/mapit.testserveur.fr/mapit/data/cache-with-political/O02/relatio n-108089-Ecuador.kml
Ecuador
No area existed in the current generation with that OSM element type and ID
Area ID: 1
.
[0% complete] Loading /var/www/mapit.testserveur.fr/mapit/data/cache-with-political/O02/relatio n-1111111-Deutschland.kml
Deutschland
Invalid KML:O02/relation-1111111-Deutschland.kml
No area existed in the current generation with that OSM element type and ID
Area ID: 2
.
[0% complete] Loading /var/www/mapit.testserveur.fr/mapit/data/cache-with-political/O02/relatio n-1124039-Monaco.kml
Monaco
No area existed in the current generation with that OSM element type and ID
Area ID: 3
.
[0% complete] Loading /var/www/mapit.testserveur.fr/mapit/data/cache-with-political/O02/relatio n-114686-Estados Unidos Mexicanos.kml
Estados Unidos Mexicanos
No area existed in the current generation with that OSM element type and ID
Area ID: 4
.
[0% complete] Loading /var/www/mapit.testserveur.fr/mapit/data/cache-with-political/O02/relatio n-1152720-ÐбÑ
Ð°Ð•Ð¸Ñ - ÐÒ§ÑнÑ.kml
Traceback (most recent call last):
File "./manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/var/www/mapit.testserveur.fr/virtualenv-mapit/local/lib/python2.7/site-packages/django /core/management/init.py", line 351, in execute_from_command_line
utility.execute()
File "/var/www/mapit.testserveur.fr/virtualenv-mapit/local/lib/python2.7/site-packages/django /core/management/init.py", line 343, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/www/mapit.testserveur.fr/virtualenv-mapit/local/lib/python2.7/site-packages/django /core/management/base.py", line 394, in run_from_argv
self.execute(_args, *_cmd_options)
File "/var/www/mapit.testserveur.fr/virtualenv-mapit/local/lib/python2.7/site-packages/django /core/management/base.py", line 445, in execute
output = self.handle(_args, *_options)
File "/var/www/mapit.testserveur.fr/virtualenv-mapit/local/lib/python2.7/site-packages/django /core/management/base.py", line 623, in handle
label_output = self.handle_label(label, **options)
File "/var/www/mapit.testserveur.fr/mapit/mapit_global/management/commands/mapit_global_impor t.py", line 166, in handle_label
xml.sax.parse(kml_filename, kml_data)
File "/usr/lib/python2.7/xml/sax/init.py", line 33, in parse
parser.parse(source)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 119, in parse
self.prepareParser(source)
File "/usr/lib/python2.7/xml/sax/expatreader.py", line 111, in prepareParser
self._parser.SetBase(source.getSystemId())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 21-27: ordinal not in range(128)

[dracos]

Okay, so that error is being thrown by the XML reader (so I don't think you need to bother downgrading Django, I can't see that changing anything). The KML file that it is erroring on, relation 1152720, is that file valid XML? Is there anything about it you can see that looks wrong?
On my server, the file is listed as: ...mapit/data/cache-with-political/O02/relation-1152720-Абхазия - Аҧсны.kml so my guess is the issue lies with your server in some way. Is it running a Unicode locale?

[dracos]

I have replicated the issue locally. It looks to be a bug in python 2.7.3, fixed in 2.7.4: https://bugs.python.org/issue11159
If you can't upgrade to a newer 2.7 python, we can try and find a workaround, I imagine. I'm not sure why we haven't come across this in the past.

[dracos]

I don't really follow the second half of your comment, I'm afraid – if your terminal isn't capable of displaying UTF-8 characters correctly (could you try setting it to UTF-8) then you will be seeing all sorts of weird characters that will interfere with your terminal display.

The issue is definitely with the import script, as I've said; I have pushed a fix to that script that should hopefully allow this script to work in python 2.7 prior to 2.7.4, thanks for letting us know about it :)

yes was my terminal from others computers not in UTF-8, but now i've got this behind, any clues to correct it ?
(virtualenv-mapit)mapit@mapit:/mapit$ ./manage.py migrate mapit
Operations to perform:
Apply all migrations: mapit
Running migrations:
Applying mapit.0001_initial... OK
Applying mapit.0002_auto_20141218_1615... OK
(virtualenv-mapit)mapit@mapit:/mapit$ ./manage.py loaddata global
Installed 23 object(s) from 1 fixture(s)
(virtualenv-mapit)mapit@mapit:~/mapit$ ./manage.py mapit_generation_create \

--commit --desc "Initial import of MapIt Global data"

Creating generation...
...saved: Generation 1 (inactive)
(virtualenv-mapit)mapit@mapit:~/mapit$ ./manage.py mapit_global_import --verbosity 2 --commit data/cache-with-political/
Loading any admin boundaries from data/cache-with-political/
Finding language codes...
Loading type O02
Loading all KML in O02
[0% complete] Loading data/cache-with-political/O02/relation-108089-Ecuador.kml
Ecuador
No area existed in the current generation with that OSM element type and ID
Area ID: 1
Traceback (most recent call last):
.................
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 65, in execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "area_id" violates not-null constraint

I solved this error with a simple upgrade of Django > 1.8" in the virtualenv.
For information I had updating python to 2.7.4 in the virtualenv like you told me.
Django 1.7 with python2.7.4 failed on the mapit_global_import script with an area_id error.
From now import is working so when it'll finish i'll tell you more about the results.