close
Skip to content

aarch64: expose SMT topology in cpu-map and allow smt: true - #6154

Open
NathanChenNVIDIA wants to merge 5 commits into
firecracker-microvm:mainfrom
NathanChenNVIDIA:aarch64-smt-support-aug
Open

aarch64: expose SMT topology in cpu-map and allow smt: true#6154
NathanChenNVIDIA wants to merge 5 commits into
firecracker-microvm:mainfrom
NathanChenNVIDIA:aarch64-smt-support-aug

Conversation

@NathanChenNVIDIA

Copy link
Copy Markdown

Changes

  • Emit a /cpus/cpu-map node on aarch64 (one cluster, one core per vCPU by default) and give each cpu@N an explicit phandle so the map can reference it.
  • When smt is enabled and there is more than one vCPU, pair consecutive vCPUs (2i, 2i+1) as thread0/thread1 under core i instead of as independent cores.
  • Accept smt: true in machine-config on aarch64, using the same rule as x86_64: vcpu_count must be 1 or even. Update swagger, CHANGELOG, and tests.
  • Add FDT unit tests for the map with and without SMT, and an aarch64 integration test that checks guest lscpu reports two threads per core.
  • This describes topology to the guest only. It does not pin vCPUs to host SMT siblings, change MPIDR, or alter scheduling.

Reason

aarch64 guests currently have no cpu-map, so Linux treats each vCPU as its own core (thread_id = -1). SMT pairing on arm64 comes from the device tree (parse_core() in arch_topology.c), not from MPIDR, so describing thread siblings in cpu-map is enough for the guest to show a real SMT topology. The API already had an smt flag; rejecting it only on aarch64 blocked that topology from being configured.

Testing

On aarch64, a 2-vCPU guest with smt: true and guest kernel 6.1 reports:

  • lscpu: CPU(s)=2, Thread(s) per core=2, Core(s) per cluster=1, Cluster(s)=1
  • /sys/devices/system/cpu/cpu0/topology/thread_siblings_list: 0-1
  • /proc/device-tree/cpus/cpu-map/cluster0/core0/: thread0, thread1
    test_cpu_topology skips on this host (kernel ≥ 6.14). The new ARM hwloc
    expectations in that helper have not run here.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

  • I have read and understand CONTRIBUTING.md.
  • I have run tools/devtool checkbuild --all to verify that the PR passes
    build checks on all supported architectures.
  • I have run tools/devtool checkstyle to verify that the PR passes the
    automated style checks.
  • I have described what is done in these changes, why they are needed, and
    how they are solving the problem in a clear and encompassing way.
  • I have updated any relevant documentation (both in code and in the docs)
    in the PR.
  • I have mentioned all user-facing changes in CHANGELOG.md.
  • If a specific issue led to this PR, this PR closes the issue.
  • When making API changes, I have followed the
    Runbook for Firecracker API changes.
  • I have tested all new and changed functionalities in unit tests and/or
    integration tests.
  • I have linked an issue to every new TODO.

  • This functionality cannot be added in rust-vmm.

@Manciukic Manciukic added the Status: Awaiting assignee Indicates that an issue or pull request is awaiting action from its assignee. label Sep 2, 2026
@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.06%. Comparing base (fa6bfae) to head (776c33e).

⚠️ Current head 776c33e differs from pull request most recent head f36c092

Please upload reports for the commit f36c092 to get more accurate results.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6154      +/-   ##
==========================================
+ Coverage   82.99%   83.06%   +0.07%     
==========================================
  Files         277      277              
  Lines       31311    31198     -113     
==========================================
- Hits        25985    25915      -70     
+ Misses       5326     5283      -43     
Flag Coverage Δ
5.10-m5n.metal 83.29% <ø> (+0.07%) ⬆️
5.10-m6a.metal 82.65% <ø> (+0.06%) ⬆️
5.10-m6g.metal 80.12% <100.00%> (+0.07%) ⬆️
5.10-m6i.metal 83.29% <ø> (+0.07%) ⬆️
5.10-m7a.metal-48xl 82.65% <ø> (+0.07%) ⬆️
5.10-m7g.metal 80.12% <100.00%> (+0.07%) ⬆️
5.10-m7i.metal-24xl 83.26% <ø> (+0.07%) ⬆️
5.10-m7i.metal-48xl 83.26% <ø> (+0.06%) ⬆️
5.10-m8g.metal-24xl 80.12% <100.00%> (+0.07%) ⬆️
5.10-m8g.metal-48xl 80.12% <100.00%> (+0.07%) ⬆️
5.10-m8i.metal-48xl 83.26% <ø> (+0.06%) ⬆️
5.10-m8i.metal-96xl 83.26% <ø> (+0.07%) ⬆️
5.10-m9g.metal-48xl 80.12% <100.00%> (+0.07%) ⬆️
6.1-m5n.metal 83.31% <ø> (+0.07%) ⬆️
6.1-m6a.metal 82.68% <ø> (+0.07%) ⬆️
6.1-m6g.metal 80.12% <100.00%> (+0.07%) ⬆️
6.1-m6i.metal 83.31% <ø> (+0.07%) ⬆️
6.1-m7a.metal-48xl 82.67% <ø> (+0.07%) ⬆️
6.1-m7g.metal 80.12% <100.00%> (+0.07%) ⬆️
6.1-m7i.metal-24xl 83.32% <ø> (+0.06%) ⬆️
6.1-m7i.metal-48xl 83.33% <ø> (+0.07%) ⬆️
6.1-m8g.metal-24xl 80.12% <100.00%> (+0.07%) ⬆️
6.1-m8g.metal-48xl 80.12% <100.00%> (+0.08%) ⬆️
6.1-m8i.metal-48xl 83.32% <ø> (+0.07%) ⬆️
6.1-m8i.metal-96xl 83.32% <ø> (+0.07%) ⬆️
6.1-m9g.metal-48xl 80.12% <100.00%> (+0.07%) ⬆️
6.18-m5n.metal 83.31% <ø> (+0.07%) ⬆️
6.18-m6a.metal 82.68% <ø> (+0.07%) ⬆️
6.18-m6g.metal 80.23% <100.00%> (+0.07%) ⬆️
6.18-m6i.metal 83.32% <ø> (+0.08%) ⬆️
6.18-m7a.metal-48xl 82.67% <ø> (+0.06%) ⬆️
6.18-m7g.metal 80.22% <100.00%> (+0.07%) ⬆️
6.18-m7i.metal-24xl 83.32% <ø> (+0.06%) ⬆️
6.18-m7i.metal-48xl 83.32% <ø> (+0.08%) ⬆️
6.18-m8g.metal-24xl 80.22% <100.00%> (+0.07%) ⬆️
6.18-m8g.metal-48xl 80.22% <100.00%> (+0.07%) ⬆️
6.18-m8i.metal-48xl 83.33% <ø> (+0.07%) ⬆️
6.18-m8i.metal-96xl 83.32% <ø> (+0.07%) ⬆️
6.18-m9g.metal-48xl 80.22% <100.00%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JackThomson2

Copy link
Copy Markdown
Contributor

Hey @NathanChenNVIDIA thank you for the PR it seems the CI is currently failing with this error can we take a look at it?


=================================================================== short test summary info ====================================================================
--
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_ON-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_ON-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_OFF-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_OFF-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_ON-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_ON-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_OFF-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_OFF-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_ON-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_ON-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_OFF-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_OFF-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | ============================================ 12 failed, 1485 passed, 128 skipped, 16 warnings in 325.04s (0:05:25) =============================================

Walter Wang and others added 3 commits September 8, 2026 19:31
Emit a minimal cpu-map -- one cluster containing N cores -- and give
each cpu node an explicit phandle so the cluster/core entries can
reference it.

Signed-off-by: Walter Wang <walterw@nvidia.com>
Extend the cpu-map node so that, when SMT is enabled, consecutive vCPUs
(2*i, 2*i+1) are emitted as the two `thread` nodes of core `i` instead
of as two independent cores.

arm64 Linux derives thread siblings and "Thread(s) per core" purely
from the device tree: parse_core() in drivers/base/arch_topology.c reads
the `thread%d` child nodes, and store_cpu_topology() -- the fallback
used when no cpu-map is present -- hardcodes thread_id to -1. MPIDR_EL1
is not consulted, so the cpu-map node alone is sufficient to expose SMT
and no vCPU register state needs to change.

With SMT disabled, or with a single vCPU, the emitted tree is unchanged.

Signed-off-by: Nathan Chen <nathanc@nvidia.com>
Remove the aarch64-specific rejection of smt: true in machine-config
updates. The same validation rules as x86_64 now apply: vcpu_count must
be 1 or even when SMT is enabled. Update the affected tests, API
documentation, and changelog accordingly.

Signed-off-by: Nathan Chen <nathanc@nvidia.com>
Update integration tests to verify SMT can be configured on aarch64 and
that guest CPU topology reflects the configured thread count. Add
focused 2-vCPU and 4-vCPU cases that assert two threads per core.

Signed-off-by: Nathan Chen <nathanc@nvidia.com>
@NathanChenNVIDIA

Copy link
Copy Markdown
Author

Hey @NathanChenNVIDIA thank you for the PR it seems the CI is currently failing with this error can we take a look at it?


=================================================================== short test summary info ====================================================================
--
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_ON-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_ON-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_OFF-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.18.44-PCI_OFF-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_ON-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_ON-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_OFF-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-5.10.260-PCI_OFF-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_ON-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_ON-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_OFF-HTT_ON-2] - AssertionError: depth 3 does not have the expected value
  | FAILED integration_tests/functional/test_topology.py::test_cpu_topology[vmlinux-6.1.182-PCI_OFF-HTT_ON-16] - AssertionError: depth 3 does not have the expected value
  | ============================================ 12 failed, 1485 passed, 128 skipped, 16 warnings in 325.04s (0:05:25) =============================================

Hi @JackThomson2 , thanks for taking a look. I made a fix to test_topology.py so that only the depth 6 hwloc output is updated in accordance with the associated changes to core count. I reverted the changes to depths 3, 4, and 5 because we do not yet have the associated updates to the device tree to advertise shared L1/L2 caches between sibling threads on ARM64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status: Awaiting assignee Indicates that an issue or pull request is awaiting action from its assignee.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants