[lora] fix: lora unloading behvaiour

[sayakpaul]

What does this PR do?

For the text encoder, we set:

diffusers/src/diffusers/loaders/lora_base.py

Line 194 in cdaf84a

text_encoder._hf_peft_config_loaded = None

We should also do the same for the denoiser but doesn't:

diffusers/src/diffusers/loaders/peft.py

Lines 680 to 688 in cdaf84a

	def unload_lora(self):
	if not USE_PEFT_BACKEND:
	raise ValueError("PEFT backend is required for `unload_lora()`.")
	from ..utils import recurse_remove_peft_layers
	recurse_remove_peft_layers(self)
	if hasattr(self, "peft_config"):
	del self.peft_config

This PR fixes that.

The problem without this would surface when running the test this PR adds (i.e., add an adapter, unload it, and then try to add it again).

I have also taken the liberty to rename a function to add_adapters_to_pipeline() which I think is more appropriate.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for this bugfix, LGTM.

Regarding the name, I believe both are a bit imprecise, as the method both adds and runs asserts. Not a big deal, just my 2c.