Add Hybrid Solver by maxpaik16 · Pull Request #112 · polyfem/polysolve

maxpaik16 · 2026-07-09T17:00:27Z

Added CPU and GPU implementations of hybrid linear solver. The CPU solver is only built if MPI is found, and the GPU solver is only built if CUDA is found.

Added cuDSS wrapper

Added test case to verify hybrid linear solvers are converging well for a system with ill-conditioning due to contact

codecov · 2026-07-09T17:13:43Z

Codecov Report

❌ Patch coverage is 75.12315% with 202 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.45%. Comparing base (e38cfac) to head (74a4d45).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
src/polysolve/linear/CPUHybridSolver.cpp	77.34%	174 Missing ⚠️
src/polysolve/linear/hybrid_utils/DisjointSet.cpp	22.22%	14 Missing ⚠️
src/polysolve/linear/HypreSolver.cpp	41.66%	7 Missing ⚠️
src/polysolve/linear/CPUHybridSolver.hpp	40.00%	6 Missing ⚠️
src/polysolve/linear/HypreSolver.hpp	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #112      +/-   ##
==========================================
- Coverage   80.73%   79.45%   -1.29%     
==========================================
  Files          51       55       +4     
  Lines        2118     2940     +822     
  Branches      280      377      +97     
==========================================
+ Hits         1710     2336     +626     
- Misses        408      604     +196

Flag	Coverage Δ
polysolve	`79.45% <75.12%> (-1.29%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

teseoch · 2026-07-10T17:50:32Z


+if(POLYSOLVE_WITH_MPI)
+    # If the option is ON (either via auto-detect or forced by the user), ensure it is actually found
+    find_package(MPI REQUIRED)


i have mixed feeling about this, i would rather have a warning and disalbe if no mpi

teseoch · 2026-07-10T17:51:12Z


+# Silently detect MPI to set the default option
+find_package(MPI QUIET)
+if(MPI_CXX_FOUND)


here too, the philosophy of polysolve is user select, if system dosent have it, warning then disable

I think MPI should be disabled by default just like CUDA. Both are troubling dependencies.

iiiian · 2026-07-10T20:54:11Z

+            m_nrows = A.rows();
+            m_ncols = A.cols();
+            m_nnz = A.nonZeros();
+


It's fine to support int32 index only, but please throw on large index somewhere. Do hybrid support int64 index type?

I now only enable cuDSS/hybrid solvers if POLYSOLVE_LARGE_INDEX is false. Of course, this isn't the ideal solution but I believe multiple of the solvers assume int32 and properly handling int64 indices deserves a separate PR.

iiiian · 2026-07-10T21:04:21Z

        "doc": "Use preconditioned residual norm for termination check."
+    },
+    {
+        "pointer": "/CPUHybrid",


This is just my personal opinion but maybe setting other than tolerance should be in advanced? I don't think user would want to tweak stuff like gmm iter.

* Run the hybrid solver on hypre's threads-as-ranks backend The hybrid solver needed MPI, which meant the whole application had to be launched under mpirun. That is fine for a driver but not for a library: anything embedding polysolve inherited the requirement. hypre can now build its MPI surface on threads of one process (HYPRE_ENABLE_THREAD_MPI), so the ranks become threads and mpirun goes away. The resulting binary links no libmpi at all. What this needed: - tmpi_mpi_compat.hpp: the few MPI facilities hypre's backend does not already provide. hypre maps MPI_Bcast/Allreduce/Scatterv/... onto its own surface already; missing were the MPI-3 shared-memory windows (with threads, "shared memory" is just a pointer broadcast), MPI_IN_PLACE, and MPI_Init/Initialized/Finalized. - The ranks are started by the solver rather than by mpirun. The first hybrid solver constructed calls hypre_tmpi_team_start(), the calling thread becomes rank 0 and keeps driving, and the workers sit in the existing command loop. They now return from their thread function instead of std::exit(0), and the last solver destroyed shuts the team down so the rest of the process sees a one-rank world again. - is_running_worker_loop and worker_registry are thread_local. They were per-rank only because each rank used to be a process; as threads they were shared and the workers raced on them. MPI remains available: POLYSOLVE_WITH_MPI still selects a rank-parallel hybrid solver, it just no longer implies OpenMPI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add an SPMD benchmark driver and reporting scripts for the hybrid solver The Catch2 tests cannot time the multi-rank path: they drive the solver from rank 0 while the other ranks sit in the worker loop, and under mpirun at more than one rank the suite is torn down when a worker leaves the loop. Timing that measures the teardown, not the solve. tests/bench_spmd.cpp is the same source built into both an OpenMPI and a thread-MPI build, so the two backends are compared on one driver rather than on two that happen to look similar. It builds a 7-point Laplacian of a given grid size, solves it, and prints setup/solve/total plus the relative residual so a run that converged differently cannot be mistaken for a run that was merely faster. scripts/bench_hybrid.sh sweeps rank counts on both backends. It takes an flock so two sweeps cannot halve each other's scores, refuses to start against a loaded machine rather than quietly reporting contended numbers, and interleaves the backends inside each repetition so drift hits both equally. scripts/bench_report.py renders the CSV as Markdown: best-of-N rather than mean, the worst run-to-run spread so the reader can judge whether a gap is real, and a check that both backends reached the same residual at each rank count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Take nano-mpi as the MPI, instead of a backend inside hypre The threads-as-ranks backend this branch was written against lived inside hypre, behind a HYPRE_ENABLE_THREAD_MPI build mode, and needed a 183-line compatibility shim here to fill the gaps. It is now a library of its own: https://github.com/danielepanozzo/nano-mpi which changes what polysolve has to do. nano-mpi installs a header named mpi.h, so hypre is built as an ordinary MPI build -- HYPRE_ENABLE_MPI=ON, find_package(MPI) as it always did -- and the shim deletes entirely: * MPI_IN_PLACE, MPI_Init/Initialized/Finalized, and const-correct collectives are all native now. * MPI-3 shared-memory windows are native too. They were the last thing keeping the shim alive, and they are the one part of MPI's one-sided chapter that is trivially true when ranks are threads: the window is a real allocation plus everyone's offset into it. nano-mpi implements the layout MPI actually promises rather than the shim's simplification, so MPI_Win_shared_query answers correctly for every rank, not just rank 0. hypre_tmpi_team_start/join -> nanompi_team_start/join HYPRE_TMPI_NUM_THREADS -> NANOMPI_NUM_RANKS The one piece of machinery this adds is cmake/nanompi-as-mpi/FindMPI.cmake. hypre calls find_package(MPI REQUIRED) internally and would otherwise find a system Open MPI -- whose ranks are processes, needing a launcher polysolve has no way to invoke. The shim is on CMAKE_MODULE_PATH only when POLYSOLVE_WITH_MPI is on, and it points MPI at nano-mpi. It gives MPI::MPI_C the header path and nothing else, deliberately: hypre puts that target in CMAKE_REQUIRED_LIBRARIES and runs check_c_source_compiles, and try_compile() exports it into a scratch project where a link interface naming nanompi::nanompi is a hard error. Whoever links MPI links nanompi::nanompi explicitly instead. The probe in question is for MPI_Comm_f2c, which nano-mpi does not declare -- it has no Fortran bindings and cannot, since Fortran SAVE storage is per-process by language rule -- so it correctly comes out false. Verified on macOS/arm64, Release: unit_tests: All tests passed (1521 assertions in 23 test cases) otool -L bench_spmd | grep -i mpi -> nothing bench_spmd 40 CPUHybrid, 64k unknowns: 1 rank setup 0.040 solve 0.098 relres 3.02e-11 2 ranks setup 0.028 solve 0.057 relres 7.57e-11 4 ranks setup 0.014 solve 0.041 relres 1.97e-11 nano-mpi is pinned to v0.1.0. hypre still points at the fork branch, because the re-entrancy fix it needs (nine globals made thread-local) is still an open PR upstream; that pointer moves when it lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add a CI job that actually builds the hybrid solver Every existing job configures with POLYSOLVE_WITH_MPI=OFF, which leaves the hybrid solver out of the build entirely. A green CI on this branch therefore said nothing about the thing the branch is for. This job turns MPI on. Ranks are threads of the test process, so there is nothing to install and nothing to launch -- the Linux dependency list is the same as the others minus the mpi package. Beyond building and running ctest it checks two things the port could plausibly get wrong and still pass tests: * that no MPI runtime is linked into the binary, since the whole point is that ranks are threads and not processes; * that the solver converges at 1, 2 and 4 ranks, since a rank count that silently produces a wrong decomposition would otherwise look like a pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Move to nano-mpi v0.1.1 The Linux CI job added in the previous commit found what the macOS build did not: AMGCL's MPI headers need MPI_CXX_DOUBLE_COMPLEX and MPI_CXX_FLOAT_COMPLEX (amgcl/mpi/util.hpp maps std::complex<T> onto them unconditionally), MPI_Exscan (mpi/partition/util.hpp) and MPI_Ialltoall (mpi/coarsening/pmis.hpp). v0.1.1 has all four, plus the rest of the nonblocking collectives and a Windows port. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

maxpaik16 added 2 commits July 9, 2026 12:54

add hybrid solvers and cuDSS + tests

d9c9c65

formatting fixes

36e8a7b

maxpaik16 and others added 6 commits July 9, 2026 13:24

pass nullptr to MPI_init

3773540

init all hypre vector fields

08d4721

add includes for hypre internals

654928a

add workspace management

f670bb1

all ranks update memory/execution policies

6d852d0

Merge branch 'main' into hybrid-solver

77e2142

maxpaik16 mentioned this pull request Jul 10, 2026

Add (optionally MPI-parallelized) hybrid solver polyfem/polyfem#518

Open

teseoch reviewed Jul 10, 2026

View reviewed changes

iiiian reviewed Jul 10, 2026

View reviewed changes

Comment thread tests/test_linear_solver.cpp

iiiian reviewed Jul 10, 2026

View reviewed changes

disable mpi by default

bc147c1

This was referenced Aug 6, 2026

Add a persistent rank team so libraries can embed hypre danielepanozzo/hypre#4

Merged

Run the hybrid solver on nano-mpi: ranks as threads, no mpirun #114

Merged

danielepanozzo and others added 5 commits September 2, 2026 23:00

use MPI by default to enable hybrid solver

3700940

remove openmpi comparisons

d8333ab

disable mpi for windows

6e1d0c2

set mpi default behavior

74a4d45

Navigation Menu

Uh oh!

Add Hybrid Solver - #112

Add Hybrid Solver#112
maxpaik16 wants to merge 14 commits into
mainfrom
hybrid-solver

maxpaik16 commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

teseoch Jul 10, 2026

Uh oh!

maxpaik16 Jul 25, 2026

Uh oh!

teseoch Jul 10, 2026

Uh oh!

iiiian Jul 10, 2026

Uh oh!

maxpaik16 Jul 25, 2026

Uh oh!

iiiian Jul 10, 2026 •

edited

Loading

Uh oh!

maxpaik16 Jul 25, 2026

Uh oh!

Uh oh!

iiiian Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

maxpaik16 commented Jul 9, 2026

Uh oh!

codecov Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

teseoch Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

maxpaik16 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

teseoch Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

iiiian Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

maxpaik16 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

iiiian Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxpaik16 Jul 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

iiiian Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Jul 9, 2026 •

edited

Loading

iiiian Jul 10, 2026 •

edited

Loading