Salt v3000 crashes with unless clause if pip is missing

[thusoy]

Description of Issue

archive.extracted (and possibly others?) fails if setting an unless clause and pip is not installed. Works on 2019.2.3.

Setup

# archive-test
vault:
    archive.extracted:
        - name: /usr/local/bin/
        - source: https://releases.hashicorp.com/vault/1.3.2/vault_1.3.2_linux_amd64.zip
        - source_hash: sha256=b49de8fd508eb9c2c222fa0f38e23546fb28991af2e8bfdb9bbe381a380f9baa
        - archive_format: zip
        - enforce_toplevel: False
        - unless:
            - echo hello

Steps to Reproduce Issue

salt-call state.sls archive-test without pip installed. Removing the unless clause or installing python-pip resolves the issue.

Versions Report

Masterless.

Salt Version:
           Salt: 3000

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.4
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-6-amd64
         system: Linux
        version: debian 9.4

There's a very long stacktrace when it crashes, this is the beginning and end:

[ERROR   ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 110, in <module>
    del pip
NameError: name 'pip' is not defined
[ERROR   ] Failed to import states pkg, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkg.py", line 77, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states pkgbuild, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py", line 48, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
...

[ERROR   ] Failed to import states zpool, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1607, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/zpool.py", line 75, in <module>
    import os
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'salt.loaded.int.module.state'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3000', 'console_scripts', 'salt-call')()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 445, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 119, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 232, in call
    func.__module__].__context__.get('retcode', 0)
KeyError: 'salt.loaded.int.module.state'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3000', 'console_scripts', 'salt-call')()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 445, in salt_call
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 119, in run
    ret = self.call()
  File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 232, in call
    func.__module__].__context__.get('retcode', 0)
KeyError: 'salt.loaded.int.module.state'

[thusoy]

FWIW I'm seeing the same issue on Debian Buster (10).

[dynek]

I feel lucky to not use Saltstack (or the combo Salt and Debian) in a production environment. Pretty much each time there's a release since I started using it there are such problems.
So there is my +1 for Debian 10.3 and Salt 3000+ds-1 (also happens on an old Debian 8.11 that I'm migrating)

[max-arnold]

It looks like the old problem manifested itself in a different way: #53570

Below is a quick fix:

patches/pip.sls:

{% if grains.saltversion == "3000" %}
patch:
  pkg.installed

pip_state_patch:
  file.patch:
    - name: '{{ grains.saltpath }}'
    - source: salt://patches/files/3000-pip.diff
    - strip: 2
    - require:
        - pkg: patch

restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - file: pip_state_patch
{%- endif %}

patches/files/3000-pip.diff:

diff --git a/salt/states/pip_state.py b/salt/states/pip_state.py
index 0f762752d0..97a9feeb48 100644
--- a/salt/states/pip_state.py
+++ b/salt/states/pip_state.py
@@ -100,14 +100,13 @@ except ImportError:
     import sys
     pip_related_entries = [
         (k, v) for (k, v) in sys.modules.items()
-        or getattr(v, '__module__', '').startswith('pip.')
+        if getattr(v, '__module__', '').startswith('pip.')
         or (isinstance(v, types.ModuleType) and v.__name__.startswith('pip.'))
     ]
     for name, entry in pip_related_entries:
         sys.modules.pop(name)
         del entry
 
-    del pip
     sys_modules_pip = sys.modules.pop('pip', None)
     if sys_modules_pip is not None:
         del sys_modules_pip

Run sudo MINION state.apply patches.pip to apply the fix.

Another option is to override the pip_state.py module. However, the process is more complex.

[ymasson]

Thanks @max-arnold

[ferricoxide]

Thanks for opening this. Was wondering why all my Travis jobs started turning red for content I han't changed.

[stratusjerry]

Sounds like #54755 (comment)