close
Skip to content

pmdaCacheResize vs PMDA_CACHE_SAVE from previous PCP versions #2571

@natoscott

Description

@natoscott

Awhile ago I encountered an issue where the bpf metrics were misbehaving. Symptoms were:

  • I knew the PMDA was installed, and even re-Installed it to be sure, to be sure.
  • Running 'pminfo bpf' returned "Unknown metric name"
  • Noone else was seeing this, QA showed no issues.

I didn't have time to look into it previously, but Murphy's Law kicked in the day before @sfeifer and I gave a 2hr PCP tutorial this past week, and my laptop exhibited the same issue again.

What I believe happened is as follows:

  • I was mucking around with the PMID cluster cache in this PMDA awhile back, including removing the use of the CacheResize call, and instances were persisted in this state (with very large cluster ID hash values).
  • Later, a subsequent fresh install of PCP returned the machine to its normal cache state ... however the on-disk cache still contained these now-invalid entries.
  • As a result, the PMIDs were all messed up (incorrect cluster IDs loaded from disk into the cache.c data structures ... not sure exactly what happens then within libpcp_pmda, but visible result is that the resized cache IDs have been subverted and are invalid (too large) for existing bpf modules' cluster IDs.
  • This creates a mismatch within the PMDA and all PMID lookups fail subsequently.
  • Removing the saved cache files resolves the problem and everything works correctly once more.

It's not an issue that has hit anyone other than me AFAIK. However, its a subtle corner case so I figure I'll open this issue to record it for posterity. Perhaps some hardening could be added to the cache LOAD code to check the incoming "instance" IDs are less than the limit, discarding them if not? Not a high priority anyway, bit of a wierd one - it would only really become an issue if we introduce a call to pmdaCacheResize in some PMDA where it has previously persisted cache state.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions