Conversation
…me->PythonInteropRuntime, namespace ->NumSharp.Interop.PythonNet
Clarify the pythonnet interop's public surface by naming types for what they
convert and where they live, ahead of folding the fluent extensions in.
Types:
- PythonConvert -> NDArrayInterop (the four-verb NDArray<->Python engine;
files PythonConvert{,.Export,.Import}.cs -> NDArrayInterop{,.Export,.Import}.cs)
- InteropRuntime -> PythonInteropRuntime (process-wide session state / lifetime
registries; file InteropRuntime.cs -> PythonInteropRuntime.cs)
Namespace:
- NumSharp.Interop -> NumSharp.Interop.PythonNet across all sources, plus the
csproj RootNamespace, so the pythonnet backend sits under its own sub-namespace
(leaving room for sibling interop backends).
Callers updated:
- NumpyCodec, Pythonic (np/ctypes/weakref/builtins veneer), NumSharpPythonExtensions,
and every doc/<see cref> reference now point at the new type names.
- All NumSharp.Interop.UnitTests files add 'using NumSharp.Interop.PythonNet;' and
reference NDArrayInterop.* / PythonInteropRuntime.*.
Pure rename: no behavioral change. Source and test projects build clean (net8.0;net10.0).
…op NumSharpPythonExtensions Make NDArrayInterop the single provider of both the conversion engine and its fluent extension methods, removing the separate wrapper classes. Removed: - NumSharpPythonExtensions and PyObjectNumSharpExtensions (whole file). They were thin forwarders whose only job was to expose the engine verbs as extension methods on NDArray / PyObject. NDArrayInterop now provides the extensions directly: - The engine verbs themselves became extension methods (this on the first param), so ONE method serves both call styles — NDArrayInterop.ToNumpy(nd) (static, used by the codec and ~15 test sites) AND nd.ToNumpy() (fluent) resolve to it. Applies to ToNumpy(NDArray), ToNumpy(NDArray, bool), ToNumpyCopy, ToMemoryView, and ToNDArray(PyObject). A same-named forwarding extension would have collided with the engine signature, so making the engine method itself the extension is the only way to keep both forms. - The two distinctly-named fluent aliases are re-added on NDArrayInterop next to what they delegate to: ToPython (NDArray -> ToNumpy; the pythonnet-shaped name that still out-resolves object.ToPython()) and AsNDArray (PyObject -> ToNDArrayView; the numpy array/asarray 'As... shares' name). ToNDArrayView stays a plain static — AsNDArray is its fluent face, keeping the To=copy/As=share mnemonic intact. Extension surface is preserved EXACTLY (same names, same namespace NumSharp.Interop.PythonNet), so no test or user 'using' changes — nd.ToNumpy(), nd.ToPython(), nd.ToNumpyCopy(), nd.ToNumpy(copy: true), nd.ToMemoryView(), py.ToNDArray(), py.AsNDArray() all keep working. Docs synced (README + website pythonnet.md/numpy-net.md): PythonConvert -> NDArrayInterop, using NumSharp.Interop -> using NumSharp.Interop.PythonNet, and the 'extension methods' sections now state NDArrayInterop provides them. Verified: source + test build clean (net8.0;net10.0); full interop suite 103/103 passing against Python 3.12 + numpy 2.4.2 (exercises every fluent form and the lifetime/shutdown contracts).
… + process-wide NDArrayInterop.RequireGIL Every conversion verb now takes `bool? requireGIL = null` as its last parameter (ToNumpy view + copy-routing overload, ToPython, ToNumpyCopy, ToMemoryView, ToNDArray, ToNDArrayView, AsNDArray). Effective policy: the parameter when non-null, else the new process-wide NDArrayInterop.RequireGIL (volatile-backed, default true). - true -> Py.GIL() exactly as before (re-entrant under an outer scope). - false -> a shared no-op IDisposable replaces Py.GIL() so every conversion body keeps its `using (...)` shape verbatim; the CALLER must already hold the GIL (an enclosing Py.GIL() block). Motivation: skip the per-call PyGILState_Ensure/Release + Py.GILState allocation in hot loops converting under one outer acquisition, and an escape hatch where PyGILState is problematic. - AcquireGil reads the policy exactly once per call, so a concurrent RequireGIL flip cannot split one conversion across two policies. - Scope: conversion verbs only. The background machinery (deferred lease drain, engine-shutdown drain) is deliberately UNTOUCHED — it runs on ThreadPool/shutdown threads that cannot inherit a caller's GIL and must keep acquiring regardless of the policy. - Overload safety: nd.ToNumpy(true/false) still binds to the (source, bool copy) overload (bool -> bool is exact, bool -> bool? needs lifting) — pinned by a semantics test (a copy shares nothing). - New AssemblyInfo grants InternalsVisibleTo to NumSharp.Interop.UnitTests and NumSharp.DotNetRunScript (same friend pattern as NumSharp.Core), so the AcquireGil factory is asserted precisely (singleton no-op guard vs real Py.GILState). Probed the hard way (dotnet_run + python_run break harnesses, embedded AND python-hosted, pythonnet 3.0.5 and 3.1.0): - Under a caller-held GIL every verb works GIL-free: functional matrix, codec round-trip under the global opt-out, 40k-conversion hot loop, 4 threads x 2000 conversions under per-thread GIL, RequireGIL toggle storm — all clean, LiveExports/LiveImports settle to zero. - Engine shutdown with the opt-out active and LIVE conversions outstanding: import leases force-drained, orphaned exports swept after engine death — the machinery is provably immune to the policy. - Misuse (requireGIL:false without holding the GIL) is an immediate 0xC0000005 at the first C-API call — both with a released thread state (after BeginAllowThreads) and with no thread state at all (fresh thread). Documented as undefined behavior, same as raw C-API misuse. - THE TRAP (found by the python-hosted probe, then pinned in-suite via PyGILState_Check): a .NET method/delegate body invoked FROM Python does NOT hold the GIL — pythonnet's MethodBinder releases it around managed bodies, so the one call site that looks like it inherits the GIL does not; requireGIL:false there AVs. Docs warn explicitly: only pythonnet's argument/return marshaling (where the codec runs) executes under the GIL, never the body itself. Timing note: under an already-held GIL the re-entrant Py.GIL() costs ~nothing measurable at 20k conversions — the opt-out is about control, not speed. Tests: GilPolicyTests (14) — factory identity (shared no-op singleton, real GILState, null fallback, per-call override), cross-thread contention proof that requireGIL:true actually blocks while another thread owns the GIL, the PyGILState_Check == 0 callback-body pin (resolved from Runtime.PythonDLL via NativeLibrary — no hardcoded paths), all verbs GIL-free under a held GIL, global opt-out + codec, deferred-drain immunity while the opt-out is active, dtype-gate-before-GIL, overload pin, 200-iteration hot loop. Full interop suite: 117/117 on both net8.0 and net10.0.
…nInteropRuntime->PythonRuntimeInterop Continues the interop naming pass started in 11a02db (PythonConvert->NDArrayInterop, InteropRuntime->PythonInteropRuntime). Both public/internal type names now lead with the domain noun they belong to, so the '*PythonInterop' / '*RuntimeInterop' suffixes read consistently and group together in the file tree and IntelliSense. Renames: - NDArrayInterop -> NDArrayPythonInterop (public static partial API surface) NDArrayInterop.cs -> NDArrayPythonInterop.cs NDArrayInterop.Export.cs -> NDArrayPythonInterop.Export.cs NDArrayInterop.Import.cs -> NDArrayPythonInterop.Import.cs - PythonInteropRuntime -> PythonRuntimeInterop (internal session/runtime plumbing) PythonInteropRuntime.cs -> PythonRuntimeInterop.cs Pure identifier rename — no behavioral change. All in-tree references updated across the interop sources (NumpyCodec, Pythonic, the Export/Import partials) and the full NumSharp.Interop.UnitTests suite, including doc-comment <see cref> targets. Diff is symmetric (248 insertions / 248 deletions). Note: the public API type NDArrayInterop is renamed to NDArrayPythonInterop — a breaking change for any downstream referencing it by name (branch is pre-release interop work).
… default (+ fix a pythonnet WRITABLE-on-readonly-memoryview segfault it exposed)
Replaces the codec's two bool policy flags (EncodeAsView/DecodeAsView) with
ONE NumpyCodecMode enum used in both directions (EncodeMode/DecodeMode):
Auto (default) - zero-copy VIEW when the dtype/layout permits, an
independent COPY only when a view is impossible. Never a
blanket copy. This is the 'clever resolving' requested.
View - always share; DECLINE the conversion (return no value) if a
view is impossible, rather than silently copying.
Copy - always an independent copy (no shared memory, no Py_buffer
lock, total coverage).
DecodeAnyBuffer (which python types decode) is orthogonal and unchanged.
Decode Auto is a real fallback (NumpyCodec.TryDecodeView ?? TryDecodeCopy):
contiguous / strided-numpy sources stay zero-copy views (read-only sources
become NON-WRITEABLE views), while complex64, big-endian, and non-contiguous
non-numpy exporters -- which have NO zero-copy NumSharp representation --
transparently become copies instead of failing. The common case (contiguous
numpy) succeeds on the first try, so the exception-driven fallback costs
nothing there. Encode Auto == View (a view and a copy have identical dtype
coverage on export, both needing a numpy-expressible dtype; only Decimal is
unrepresentable and falls through to pythonnet's CLR wrapping either way); the
copy fallback is wired anyway for a uniform contract.
BREAKING: the codec's decode default flips from copy to view-first. Under the
registered codec, pyObj.As<NDArray>() on a contiguous source is now a shared,
mutating, source-locking view. Set DecodeMode=Copy for the old detached-
snapshot behavior. Documented in README with the shared-memory/lock caveat.
--- The crash this default exposed, and its fix ---
Flipping decode to view-first routed memoryview/bytes/matrix through the VIEW
import path for the first time, which HARD-CRASHED the test host (native
0xC0000005, no managed exception). Root-caused with dotnet_run probes: it is a
pythonnet 3.0.5 bug -- GetBuffer(PyBUF.WRITABLE) on a read-only *memoryview*
segfaults (plain bytes throws BufferError cleanly; a memoryview does not).
AcquireBuffer probed WRITABLE-by-exception, so every read-only memoryview
import hit it.
Fix (NDArrayPythonInterop.Import.cs): read the exporter's own
memoryview.readonly (already have the metadata view open) and only REQUEST
PyBUF.WRITABLE when the source is actually writable -- never probe it on a
read-only source. This is also strictly better: one fewer guaranteed-to-fail
C-API call + exception on every read-only import. AcquireBuffer now takes a
sourceReadonly hint; a defensive catch remains for the writable-but-refused
edge. ToNDArray (copy) never requested WRITABLE, so it was always safe -- which
is why the old copy-default decode never hit this.
Verified: 150 abandoned memoryview/bytes/matrix views now decode + GC-finalize
+ drain cleanly; the minimal repro (memoryview(b'ab') view) survives; the full
suite runs to completion (was aborting mid-run). All view/copy/lock/writeability
semantics unchanged (transparency probe: complex64 view->throw/copy->widen,
memoryview[::2] view->throw/copy->linearize, read-only bytes -> non-writeable
view whose write throws, live bytearray view -> Python resize LOCKED; lifetime
probe: ARC survives parent-dispose+GC, GC reclaims abandoned views, no early
free under pressure).
Tests:
- CodecModeTests (new, 12) - the Auto/View/Copy x encode/decode matrix via
direct codec instances (one process, all three modes; the sticky global
RegisterCodec can only pin one): Auto decode views a contiguous source and
falls back to copy for complex64 / sliced-memoryview / (non-writeable) bytes;
View declines the unviewable ones with no silent copy; Copy always snapshots;
encode Auto shares, Copy detaches, Decimal CLR-wraps; Default is Auto both.
- CodecTests.Decode_NdarrayToNDArray_IsACopy -> _DefaultsToView (the default is
now a shared view); Decode_CoversSubclassesAndBufferBuiltins now disposes its
decoded views (they are leases under Auto, not owned copies).
- NumpyNetInteropTests.Codec_DecodesNumpyNetArrays_AndScopesInterleave updated
to the view contract + disposes the view.
Full interop suite: 129/129 on net8.0 and net10.0 (three consecutive clean
runs; the crash was previously deterministic).
…ffset/reversed memoryview) — a third zero-copy import route Probed for cases we DECLINED as unviewable but actually can view, and found a big one: non-contiguous exporters that are NOT numpy arrays. The old view path had two routes — C-contiguous PEP 3118 (PyBuffer SIMPLE/WRITABLE) and non-contiguous NUMPY arrays (__array_interface__) — and threw 'not C-contiguous and not a numpy array' for everything else, so a sliced/offset/reversed memoryview only ever COPIED. But a memoryview exposes its own .shape/.strides, and the buffer protocol hands back the base pointer via a PyBUF.STRIDED request even when the layout is non-contiguous. So they ARE viewable. Added a third route, ViewViaBufferStrides: read shape/strides/itemsize/readonly from the exporter's memoryview, take the base pointer from GetBuffer(STRIDED for writable / STRIDED_RO for read-only), normalize the window for negative strides (same math as the __array_interface__ path), and build the strided view. Only genuinely irreducible layouts still decline — complex64 (no 16-byte reinterpret), big-endian, and non-element strides — which in codec Auto mode become the copy fallback. AcquireBuffer was generalized to take the (writable, readonly) PyBUF flag pair so both the contiguous (WRITABLE/SIMPLE) and strided (STRIDED/ STRIDED_RO) routes share one readonly-gated acquisition (the WRITABLE-on-readonly memoryview segfault guard from the prior commit still applies to STRIDED). Impact: under the codec's Auto default, As<NDArray>() on a sliced/reversed/offset memoryview (or a strided array.array memoryview) is now a zero-copy shared view instead of a copy — more cases viewed, fewer copies. Probed the whole way (dotnet_run, pythonnet 3.0.5): GetBuffer(STRIDED_RO/STRIDED) returns a correct base pointer for [::2], [1::2] (offset), [::-1] (negative stride), and read-only sources; metadata strides reconstruct byte-exact; leases release cleanly (verified drain to 0 with the same Finalizer.Instance.Collect() pump the test-base leak gate uses). CPython forbids multi-dim memoryview slicing, so non-numpy >1-D strided inputs can't even be constructed — 1-D is the realistic scope; numpy >1-D strided still flows through __array_interface__. Tests: - StridedBufferViewTests (new, 13) — the round trip the request asked for: read -> mutate via NumSharp -> Python reads -> mutate via Python -> NumSharp reads -> compare, for strided/offset/reversed bytearray memoryviews, strided array.array (i4 and f8), read-only strided (non-writeable, guarded write throws; refused without opt-in), derived-slice-still-aliases, export round trip proving np.shares_memory (real view not copy), resize-lock, values==copy path, the complex64-strided unviewable boundary (declines/copies), and codec-Auto decoding a strided memoryview as a shared view. - CodecModeTests: the two stale 'sliced memoryview is non-viewable' cases now assert the opposite (Auto and View both share it), and the decline/fallback cases moved to strided complex64 (genuinely unviewable). Full interop suite: 144/144 on net8.0 and net10.0 (net8.0 run twice; the previously-fixed host crash stays gone).
… any buffer exporter — ctypes goes crash -> view Censused 48 exporter varieties through the view path to find every case we COPY but could VIEW. Result: 45 view / 2 copy, and the two copies (complex64, sub-item strides) are genuinely unrepresentable — so the codec was already at its view ceiling for the types it accepted. The real remaining gap was elsewhere: an entire exporter family that did not merely copy, it CRASHED. 1) Lease acquisition now goes through the exporter's memoryview, not the raw object. pythonnet 3.0.x's obj.GetBuffer is per-exporter buggy: on a raw ctypes array it hard-crashes (0xC0000005) for EVERY flag — SIMPLE, WRITABLE and STRIDED_RO alike — which took down BOTH the view and the copy path (copy also calls GetBuffer(SIMPLE)), so ctypes was simply unusable. The memoryview over the very same memory leases cleanly, so AcquireBuffer is now handed the memoryview wrapper. Py_buffer.obj retains it, which keeps the source pinned, so the wrapper is disposed right after and every safety property survives — verified: bytearray resize-lock still LOCKED, numpy resize(refcheck) still REFUSED, read-only sources still non-writeable, shared mutation still both ways. This also hardens every OTHER exporter against the same class of bug. 2) The codec accepts any buffer exporter, not a name allowlist. A ctypes array's tp_name is generated per element type AND length (c_long_Array_4, c_double_Array_3, ...), so it can never be enumerated. CanDecode now also accepts a type carrying PEP 688's __buffer__ (Python 3.12+), which probes True for every exporter (ctypes/numpy/bytes/bytearray/memoryview/array.array/ BytesIO buffers) and False for every non-buffer builtin (dict/int/str/list/ float) — so it never widens past real exporters. On Python <= 3.11 the dunder does not exist, the check is inert, and the four built-in names still apply. Net effect on Auto: ctypes arrays (and any C-extension buffer object) now decode as zero-copy shared views instead of crashing/being declined. Census (dotnet_run, pythonnet 3.0.5, numpy 2.4.2) — VIEW: bytes, bytearray, memoryview, BytesIO.getbuffer, all 12 array.array typecodes, all 4 ctypes cases, every numpy dtype except c8 (incl. f2/c16 and big-endian i1 — single-byte endianness is irrelevant), every numpy layout (strided/reversed/transposed/ fortran/broadcast/read-only/0-d), and every memoryview cast/strided form. COPY: c8, sub-item stride. Unsupported by both paths: big-endian multi-byte. Tests (+4, suite 144 -> 148 on net8.0 and net10.0; net8.0 run twice): - ImportTests.View_CtypesArray_SharesMemoryBothWays / _CoversOtherElementTypes — ctypes c_int/c_double/c_ubyte round trip: read -> NumSharp write -> Python reads -> Python write -> NumSharp reads. - CodecModeTests.CanDecode_AcceptsAnyBufferExporter_AndRejectsNonBuffers — the capability check accepts the un-allowlistable ctypes type and still rejects dict/int/str/list/float. - CodecModeTests.Auto_Decode_CtypesArray_IsAView — the payoff: through Auto, a ctypes array comes back a shared view and the write reaches Python.
…gnose Python/pythonnet version mismatches
The package declared `pythonnet 3.0.1`, which shipped a DEFAULT that cannot run
Python 3.12+. NuGet resolves LOWEST-applicable, never latest, so the declared
floor IS the out-of-box experience: a bare `dotnet add package` pulled 3.0.1
(Python 3.7-3.11) even with 3.1.0 sitting on nuget.org, and the first
`PythonEngine.Initialize()` on a modern Python died with a bare
`MissingMethodException: Failed to load symbol PyUnicode_AsUnicode`.
WHAT WAS MEASURED (all 7 stable v3 releases: 3.0.0 3.0.1 3.0.2 3.0.3 3.0.4 3.0.5 3.1.0)
1. The SOURCE was never the constraint. It compiles clean against every stable
v3, on both net8.0 and net10.0, touching only 29 pythonnet members — all of
which exist in 3.0.0. The public API 3.0.0 -> 3.1.0 is purely additive; the
only removals in the whole line are `Runtime.Py_Main` and a
`PythonDerivedType` codegen overload, neither used here.
2. The ASSEMBLY REFERENCE is the real floor. pythonnet is strong-named and bumps
AssemblyVersion on every patch (3.0.0.0 ... 3.1.0.0). .NET rolls FORWARD
silently; a downgrade is a hard FileNotFoundException. Verified by swapping
Python.Runtime.dll under a fixed app:
compiled@3.0.0 -> runs on 3.0.0 / 3.0.5 / 3.1.0 all OK
compiled@3.0.5 -> runs on 3.0.0 FileNotFoundException
Full zero-copy roundtrips (export -> numpy.sum == 66, reimport, and a strided
[::2, ::-1] view import == 44) pass compiled@3.0.0 on both 3.0.5 and 3.1.0.
3. WHY 3.0.5 AND NOT 3.0.0, despite 3.0.0 binding widest. Each release hard-caps
the Python it can drive, but `MinSupportedVersion` has been 3.7 for the ENTIRE
v3 line — only the ceiling moves. Read from each package's own
`PythonEngine.MaxSupportedVersion`, not the release notes:
3.0.0 Python 3.7 - 3.10
3.0.1 / 3.0.2 Python 3.7 - 3.11
3.0.3 / 3.0.4 Python 3.7 - 3.12 (NoopFormatter added in 3.0.4)
3.0.5 Python 3.7 - 3.13 <- the floor
3.1.0 Python 3.7 - 3.14 (notes drop 3.7-3.9; guard still says 3.7)
So 3.0.5 is a strict SUPERSET of 3.0.0-3.0.4 in Python coverage. Lowering the
floor to 3.0.0 would buy ZERO extra reach — no Python needs an old pythonnet —
while making the default install unable to run anything past 3.10. It would
only tolerate a foreign hard-pin, which NU1605 flags anyway.
Independent corroboration: README already documents the .NET 8+ shutdown
workaround `RuntimeData.FormatterType = typeof(NoopFormatter)`, and
NoopFormatter does not exist before 3.0.4 — the old 3.0.1 floor did not
guarantee the package's own documented advice.
4. The upper bound `4.0.0` keeps a future breaking pythonnet 4.x from being
resolved into consumers automatically. 3.1.0 remains opt-in (verified: an
explicit `<PackageReference Include="pythonnet" Version="3.1.0" />` resolves
to 3.1.0 against this range) — that is the path for Python 3.14.
RUNTIME DIAGNOSTIC
`PythonRuntimeInterop.VerifyPythonnetSupportsRunningPython` runs once per engine
session from EnsureEngine and turns an out-of-range pairing into an actionable
message naming the package version to install, e.g.
Python 3.14 is not supported by the loaded pythonnet 3.0.5 (it supports
Python 3.7 - 3.13). Upgrade pythonnet to 3.1.0 or later:
<PackageReference Include="pythonnet" Version="3.1.0" />
Two traps this had to handle:
- `MaxSupportedVersion` carries int.MaxValue in its build AND revision fields
as an open upper bound (literally 3.13.2147483647.2147483647), so a full
Version comparison is meaningless — compare on major.minor only.
- The check must never be the reason interop fails: `PythonEngine.Version` is
sys.version free-text, so parsing is best-effort and bails silently.
It cannot catch every case — overshooting the ceiling usually kills the caller's
own `PythonEngine.Initialize()` before any NumSharp code runs — but it does catch
the dangerous combination that initializes while out of range. Boundary logic
verified across in-range / above-ceiling / below-floor / unknown-future-Python /
unparseable inputs.
Gates: 148 interop tests x net8.0 + net10.0 = 296 pass. Packed nupkg verified
internally consistent — assembly references Python.Runtime 3.0.5.0 and the nuspec
declares [3.0.5, 4.0.0) for both TFMs. (A stale obj/ will keep the OLD compiled
reference after editing this range; the packed assembly ref must be re-checked
after a clean restore, not assumed.)
Test project comment corrected: its 3.0.5 pin is now exactly the library floor,
so CI exercises the version a default install actually resolves.
…ay(ctypes) segfaulted The memoryview-wrapper acquisition landed on the VIEW path only; ToNDArray's contiguous branch still did obj.GetBuffer(PyBUF.SIMPLE) on the raw object, which is the exact call that hard-crashes pythonnet 3.0.x on a ctypes array (every flag, not just WRITABLE). So while AsNDArray(ctypes) worked, ToNDArray(ctypes) — and DecodeMode=Copy on one — still took the process down with 0xC0000005. The metadata memoryview is already open two lines above; read the bytes through it. Only a read-only SIMPLE lock is needed and it is released as soon as the blit completes, so nothing about the copy's ownership or lifetime changes. Found while walking the import flow end to end, which is exactly the sort of asymmetry a partial fix leaves behind: the view path was hardened, the copy path that Auto falls back to was not. Test: ImportTests.Copy_CtypesArray_Works — copies a ctypes c_int[4] and asserts the write does NOT reach the source (proving it is a copy, not a view). Suite 148 -> 149 on net8.0 and net10.0.
…et docs to match the current design New page docs/website-src/docs/interop/zero-copy-model.md explaining the approach behind view-vs-copy, with runnable examples throughout: - The two semantics side by side (memory, propagation, cost, effect on the Python object, coverage, survival past PythonEngine.Shutdown). - Auto as view-first/copy-fallback, shown as the actual two-line switch (TryDecodeView(pyObj) ?? TryDecodeCopy(pyObj)) so the '??' IS the fallback, plus when to reach for View (a silent copy would be a bug) vs Copy. - The three routes that produce a view — C-contiguous exporters, non-contiguous numpy via __array_interface__, and non-contiguous NON-numpy via a PyBUF.STRIDED pointer + the memoryview's own strides — including the negative-stride window normalization code. - The three things that genuinely cannot be viewed, each with the why: complex64 (8 vs 16 bytes; viewing it as float32[...,2] would silently change the caller's dtype), big-endian multi-byte (single-byte big-endian DOES view), and sub-item strides — worked through byte by byte with the as_strided repro showing why elem[1] reads 65536, and why ordinary slicing can never produce one. - Read-only sources still view (non-writeable), and why IsWriteable == false is a reliable 'this took the view path' signal. - The trade-off: a live view holds a Py_buffer lock, so Dispose() is what makes the release deterministic. - Measured coverage table: 45 view / 2 copy across 48 exporter varieties, and the note on why leases are acquired through the memoryview rather than the raw object (pythonnet's per-exporter GetBuffer crashes). - The GIL policy (requireGIL / RequireGIL) with the Python->.NET callback trap. - A 'choosing, in one table' summary. Refreshed the existing pages, which had drifted and now contradicted the code: - NDArrayInterop -> NDArrayPythonInterop throughout (class was renamed). - Codec policy table rewritten for NumpyCodecMode (Auto/View/Copy) — the old EncodeAsView/DecodeAsView bools no longer exist, and the documented 'decode defaults to copy' is now the opposite. - 'two zero-copy routes' -> three, plus the memoryview-acquisition note and the measured coverage. - DecodeAnyBuffer now documents the PEP 688 __buffer__ capability check. - GIL section documents the requireGIL opt-out and the callback trap. - Version table corrected: STRIDED/STRIDED_RO are used too. - numpy-net.md's codec paragraph said decode is 'a safe copy by default', which Auto inverted — corrected to the shared-view default (matching the test that already asserts it). Wired into docs/toc.yml and the interop index. docfx build: 0 errors (the one warning is a pre-existing bad bookmark in NDIter.md).
Second pass over the PRE-EXISTING pages — the first one only repaired outright
contradictions, this one fills the gaps they still had.
pythonnet.md
- Intro: names the full exporter set the bridge actually handles (ctypes arrays,
BytesIO buffers, custom C extensions — not just numpy/memoryview/bytes), states
the measured 45-view/2-copy result, and links the model page. Added an
on-this-page nav strip.
- Installation: replaced the stale one-liner ('3.12/3.13 need >= 3.0.4') with the
real Python-to-pythonnet matrix (3.7-3.10 -> 3.0.0, 3.11 -> 3.0.1,
3.12 -> 3.0.3, 3.13 -> 3.0.5, 3.14 -> 3.1.0) and — newly documented — the
version guard the library already implements but nobody had written down: it
validates the pairing once per session and replaces pythonnet's opaque
'Failed to load symbol <PyFoo>' with a message naming the package version to
install. Includes the verbatim error text.
- Quick Start: added the codec form, so the zero-explicit-calls path is visible
next to the manual one.
- Four Verbs: real signatures including the trailing requireGIL, a note that they
are static members of NDArrayPythonInterop, and the ToNumpy(nd, copy: true)
routing overload.
- Version Compatibility: rewritten — Python 3.7-3.14 bounded by pythonnet,
net8.0/net10.0, and an explicit 'two pythonnet defects the bridge routes
around' (PyBuffer metadata flags; per-exporter GetBuffer crashes) so the
memoryview-acquisition design reads as a deliberate workaround rather than a
quirk.
- NEW Troubleshooting table: 13 symptoms mapped to cause and fix — the version
errors, read-only refusals, both resize-lock errors (BufferError and numpy's
refcheck ValueError) pointed at the lock section, owndata, big-endian, decimal,
complex64-widening, 'As<NDArray>() returned a view' (the Auto default), the
access-violation-means-you-lied-about-the-GIL case, and post-shutdown crashes.
- Testing: current counts (149 x2 frameworks), what the suite now covers (three
routes, codec modes, GIL policy) and the commands to run it.
index.md
- Explained what the release hook actually buys (fires on the LAST reference,
original or derived view, Dispose or GC — so neither runtime needs to know the
other's lifetime rules).
- NEW 'Three questions every bridge must answer': share-or-duplicate, who
releases and when, and what sharing costs the other side — turning the page
from a description of NumSharp's primitives into guidance for writing a bridge.
docfx build: 0 errors; all new in-page and cross-page anchors resolve (the single
warning is the pre-existing bad bookmark in NDIter.md).
… dependency floor to [3.0.5, 4.0.0) Version table gains a third column — the full Python range each pythonnet drives, so the mapping is readable in both directions instead of only 'what is my minimum': | Your Python | Minimum pythonnet | That pythonnet supports | | 3.7 - 3.10 | 3.0.0 | 3.7 - 3.10 | | 3.11 | 3.0.1 | 3.7 - 3.11 | | 3.12 | 3.0.3 | 3.7 - 3.12 | | 3.13 | 3.0.5 | 3.7 - 3.13 | | 3.14 | 3.1.0 | 3.7 - 3.14 | Every bound was read out of the packages themselves rather than the release notes — restored each version and printed PythonEngine.Min/MaxSupportedVersion: 3.0.0 -> 3.7-3.10, 3.0.1 -> 3.7-3.11, 3.0.3 -> 3.7-3.12, 3.0.5 -> 3.7-3.13, 3.1.0 -> 3.7-3.14. MinSupportedVersion is 3.7 for the whole v3 line, so each release is a strict superset of its predecessors (and MaxSupportedVersion really does carry int.MaxValue in its build/revision fields, as the guard's comment says). While verifying, found the surrounding prose still documenting the OLD floor: the package now requires [3.0.5, 4.0.0) but three places still said '>= 3.0.1'. That is not a cosmetic drift — per the csproj rationale, NuGet resolves LOWEST-applicable, so the floor IS the out-of-box experience, and the old 3.0.1 floor 'silently shipped a default that could not run Python 3.12+'. Corrected: - Installation now leads with what actually happens by default: the 3.0.5 floor drives Python 3.7-3.13 with zero configuration, and ONLY Python 3.14 needs an explicit <PackageReference Include="pythonnet" Version="3.1.0" />, shown inline. - Table caption explains that rows below 3.0.5 are the mapping the error message quotes back at you, not versions you can resolve through this package. - Version Compatibility row documents the real range including WHY the 4.x cap exists (a future breaking major must not be resolved into consumers) and why raising the floor cost nothing (superset). - numpy-net.md: Numpy.NET pins pythonnet 3.0.1, so spelled out that NuGet unifies it upward to the bridge's 3.0.5+ — which is what lets the pair run Python 3.12+ at all. - zero-copy-model.md: the PyBuffer flag defect is not 3.0.1-specific; it spans 3.0.x. docfx build: 0 errors (the single warning is the pre-existing bad bookmark in NDIter.md).
The interop project and its 149-test suite were invisible to CI: build-and-release.yml
built/tested only test/NumSharp.UnitTest and packed only NumSharp.Core + NumSharp.Bitmap,
and no workflow builds the solution as a whole. So nothing compiled
src/NumSharp.Interop.pythonnet — a break in code that reaches into NumSharp.Core internals
would have landed unnoticed — and the package it is fully configured to ship had no path
to nuget.org.
New 'interop-test' job (gate + build check)
- Own job rather than extra steps on 'test': the embedded-CPython setup must not slow
down or destabilise the main gate, and a red here is attributable to interop alone.
- Same OS matrix as 'test' (windows/ubuntu/macos-latest) and the same
continue-on-error on ubuntu.
- Python 3.12 + numpy>=2.0. 3.12 is inside pythonnet's supported range — PythonSession
rejects anything outside 3.7-3.13 — and has numpy 2.x wheels on all three OSes.
- libpython discovery is deliberately NOT reimplemented in YAML: it is left to
PythonSession's own probe, so CI exercises the shipped code path instead of a
parallel copy that could drift. When the probe finds nothing every interop test
reports Inconclusive, so a runner without a usable libpython goes SKIPPED, never red.
- 'Report Python host' prints exactly the values that probe consumes (version,
base_prefix, INSTSONAME, LIBDIR), so a silent skip is diagnosable from the log
rather than by reproducing locally.
- Both TFMs (net8.0, net10.0), Release, no category filter — the suite carries no
OpenBugs/HighMemory tests and every test already self-asserts a
LiveExports/LiveImports leak baseline.
Release pipeline
- validate-release now needs [test, interop-test]: the interop ships as a package, so
a broken interop must block the release.
- build-nuget builds and packs the interop alongside Core/Bitmap. It is packed with the
same -p:Version, which matters because its NumSharp.Core ProjectReference packs as a
NuGet dependency on "NumSharp" $(Version) — the csproj's hardcoded 0.60.0 is only a
local-dev default and must not leak into a release.
- publish-nuget needed no change: it already loops over every artifacts/*.nupkg, so the
package now reaches nuget.org with the rest.
- Release notes gain the install line and the NuGet badge row.
Verified locally against the current branch, not assumed:
- YAML parses; job graph is test/interop-test -> validate-release -> build-nuget ->
create-release + publish-nuget.
- Release build of the interop test project: 0 errors.
- CI-identical build+pack with -p:Version=0.99.0-ci emits
NumSharp.Interop.pythonnet.0.99.0-ci.nupkg (+ .snupkg) whose nuspec carries
NumSharp -> 0.99.0-ci (co-versioned, NOT 0.60.0), pythonnet -> [3.0.5,4.0.0),
lib/net8.0 + lib/net10.0, and the packed README.
- dotnet test Release: 149/149 pass on net8.0 and 149/149 on net10.0.
…est jobs
The 'test' job carried 'continue-on-error: ${{ matrix.os == 'ubuntu-latest' }}'
with the note "Ubuntu has intermittent OOM issues - allow failure while
investigating", and the new 'interop-test' job mirrored it for consistency.
That is not flake tolerance, it is a blindfold: a continue-on-error leg is
reported green to dependent jobs, so ANY Ubuntu-only regression — a real
Linux-specific bug, not just an OOM — passes the gate silently and still
lets validate-release proceed. Linux is a first-class target; a red Ubuntu
leg must fail the run and get fixed (or the underlying OOM addressed
directly), not be waved through.
Both jobs now fail the run on any matrix leg. No other change.
…of 149 tests were silently skipping
The first green CI run of the new interop-test job was a FALSE green on macOS:
ubuntu-latest 149/149 ran
windows-latest 149/149 ran
macos-latest 9 ran, 140 SKIPPED
PythonSession.Probe resolved libpython as Path.Combine(LIBDIR, INSTSONAME).
That is right for an ordinary unix shared build (libpython3.12.so.1.0 under
LIBDIR — which is why ubuntu passed), but a macOS FRAMEWORK build (what
python.org ships and what actions/setup-python installs on macOS runners)
reports INSTSONAME as a RELATIVE framework path:
INSTSONAME Python.framework/Versions/3.12/Python
LIBDIR /Library/Frameworks/Python.framework/Versions/3.12/lib
Combining those yields
/Library/Frameworks/Python.framework/Versions/3.12/lib/Python.framework/Versions/3.12/Python
which does not exist. Discovery failed, so EnsureOrInconclusive marked every
engine-backed test Inconclusive — reported as skipped, and 'Passed!' overall.
The job went green while proving nothing about macOS.
FindUnixPythonLibrary now walks candidates and takes the first that exists:
1. Path.Combine(LIBDIR, INSTSONAME) unix shared build
2. Path.Combine(prefix, filename(INSTSONAME)) macOS framework — the binary
sits directly in the version
prefix, so the FILE NAME
('Python') is what resolves
3. LIBDIR/libpython<maj>.<min>.{so,dylib} non-framework / missing INSTSONAME
4. prefix/lib/libpython<maj>.<min>.{so,dylib}
Windows is untouched (it takes the base_prefix/pythonXY.dll branch and already
resolved correctly — INSTSONAME/LIBDIR are None there).
This is a test-harness fix; no shipped library code changes. It is exactly the
class of masking the previous commit removed continue-on-error for: a green
check that silently covered an entire platform.
…x the four claims that were wrong
The interop docs were prose about code: every fenced sample, table row and quoted
error message was hand-written and hand-verified once, then left to rot. This turns
all of it into a gate. 53 new tests (149 -> 202 on each of net8.0/net10.0), one
`DocExamples_*` class per page, each test reproducing a doc snippet as literally as
the harness allows and asserting exactly what the surrounding prose promises.
DocExamples.PythonnetPage.cs 24 tests — pythonnet.md
DocExamples.ZeroCopyModelPage.cs 17 tests — zero-copy-model.md
DocExamples.InteropIndexPage.cs 7 tests — index.md (Python-free: the page
documents NumSharp primitives)
DocExamples.DocIntegrity.cs 5 tests — the prose itself
(numpy-net.md already had its 1:1 file, NumpyNetInteropTests; all 9 of its
"Proven by" citations verified to resolve.)
Writing them found four things that were not true:
1. `src/.../README.md` — the NuGet package README — still told users to call
`NDArrayInterop.RegisterCodec()`, six occurrences, for every commit since the
type was renamed to `NDArrayPythonInterop`. Shipped samples that cannot compile.
Fixed, and gated: `EveryInteropTypeNameMentionedByTheDocs_ExistsInTheShippedAssembly`
scans the pages plus the packaged README for `*Interop` names and fails on any
the assembly does not declare. A rename is invisible to a markdown file, so the
check has to be explicit.
2. `index.md`'s bridge primitive claimed `resize(refcheck: true)` protects a wrapped
foreign buffer. It does not. Written exactly as the page showed it, the array
believes it OWNS the block, so a growing `resize` succeeds: fresh NumSharp memory,
release hook fired, bridge left pointing at nothing (measured — the base address
really does move). The pythonnet import path avoids this by `Alias`ing the
storage, which is what gives an imported view numpy's `owndata == False` and the
`cannot resize this array: it does not own its data` refusal. Added that caveat
to the page with the one-line fix, and pinned both halves.
3. The coverage census (`45 view / 2 copy across 48 varieties`, quoted in four
places) was a remembered figure. It is now a test:
`Coverage_Census_MeasuresTheDocumentedTotals` builds all 50 varieties, pushes
each through the view path, falls back to copy when the view declines, prints the
table and asserts the totals — 47 view, 2 copy, 1 rejected. The numbers moved
because the census is freshly constructed (6 ctypes element types where the
original sampled 4), not because coverage changed; the 2 copies (complex64,
sub-item stride) and the 1 rejection (big-endian multi-byte) are the same floor.
Doc table now carries per-row counts so the total is auditable.
4. The Troubleshooting row for numpy's `cannot resize an array that references or is
referenced by ...` was reachable only from the import direction (a NumSharp view
leasing a numpy array), not the export direction I first assumed — an exported
array hits the owndata error first. Test now exercises the row's real cause.
Also verified-as-correct and now pinned rather than trusted: the Quick Start and
codec samples run line for line; all four verbs' documented semantics; the extension
block's `To… copies / As… shares` convention; all 7 Layout Fidelity rows; all three
import routes; the whole dtype table in both directions; every Troubleshooting
symptom the bridge itself throws; the version table; the `as_strided` sub-item-stride
example down to its `[0, 65536]` and overlapping bytes.
DELIBERATELY NOT DONE: bending production code to make docs testable. The total
change to shipped code here is ONE WORD — `MinimumPythonnetFor` private -> internal,
using the `InternalsVisibleTo` the assembly already declares — so the docs' Python
-> pythonnet table is pinned to the mapping the runtime guard actually uses (two
copies of the same knowledge that must never disagree). An earlier draft also
extracted the guard's message into `DescribeUnsupportedPython(...)` purely so the
verbatim error block could be asserted for a version pairing this process cannot be
in; that is test-induced design damage and was reverted. The drift-prone half of
that block — the version it tells you to install — is pinned instead from the
outside, by parsing the page and comparing against `MinimumPythonnetFor`
(`TheVersionAdviceQuotedByPythonnetMd_IsWhatTheGuardWouldSay`). The line: widening
visibility for a test is fine; reshaping production code for one is not.
Supporting changes:
- csproj copies the interop pages + packaged README to the test output. They fail
LOUDLY when absent — a doc gate that silently asserts nothing when its inputs
vanish is worse than no gate.
- pythonnet.md's suite size is now asserted against the assembly by reflection, so
the sentence cannot drift (it caught two of my own additions while writing this).
… could not fail
Ubuntu CI failed on ConvolveSame_TightLoop_DoesNotLeakWorkingSet
("Expected value to be less than 20L, but found 25L") while Windows and
macOS passed. The assertion was invalid, and the memory growth it picked
up came from a real defect the test never named.
WHY THE TEST COULD NOT WORK
---------------------------
It ran 200 'same'-mode convolves and asserted Process.WorkingSet64 grew
by < 20 MiB, claiming to guard the `using` on the `full` intermediate
added in 528a0f0.
- The guarded buffer is 1063 doubles (~8.5 KB). 200 of them are 1.7 MiB,
already under the 20 MiB threshold, so reintroducing the leak could
never trip it.
- Measured directly, running the loop with and without releasing `full`:
trial 0: with using = 4 MiB without using = 0 MiB
trial 1: with using = 0 MiB without using = 0 MiB
trial 2: with using = 1 MiB without using = 0 MiB
The reading is uncorrelated with the invariant, and on trial 0 the
FIXED code scored worse than the BROKEN code.
- WorkingSet64 is process-wide OS RSS. Linux does not trim it on
GC.Collect, so allocator churn reads as permanent growth on Ubuntu and
not on Windows. That is the whole platform split.
THE REAL DEFECT
---------------
ConvolveFullTyped<T> read both operands as
`Converts.ToDouble((object)aPtr[j])`. The callee takes `object`, so the
JIT cannot elide the boxes: two boxed doubles (2 x 24 B) on EVERY
inner-loop iteration.
arange(1000) convolved with arange(64), mode 'same':
3,078,968 B of managed garbage per call
587.3 MiB over the test's 200-iteration loop
= 345x the 1.7 MiB signal the test believed it was measuring
ToDoubleUnboxed<T> replaces it: `typeof(T) == typeof(X)` folds to a
compile-time constant per generic instantiation, so exactly one branch
survives the JIT and the read becomes a plain load. Same shape as the
write-back chain already in the method. Complex still discards the
imaginary part, exactly as Converts.ToDouble(object) does, so semantics
are unchanged.
per call: 3,078,968 B -> 5,944 B (518x less)
over 200: 587.3 MiB -> 1.16 MiB
Allocation is now FLAT in the work done: measured identical (5,944 B for
'same', 1,448 B for 'full') at nv = 64, 128, 256 and 512, an 8x range of
element-pairs.
Verified value-preserving: 468 rows of {13 dtypes} x {4 value sets, incl.
negatives, fractions and narrow-type overflow} x {3 kernels} x {full,
same, valid} are bit-for-bit identical before and after the change.
THE REPLACEMENT TEST
--------------------
NdArray.Convolve.UsingTests.cs -> NdArray.Convolve.AllocationTests.cs.
Both NumPy-parity tests are unchanged. The working-set test gives way to
ConvolveSame_DoesNotAllocatePerElementPair, which convolves the same
1000-element array against a 64-tap and a 512-tap kernel and asserts the
larger does not allocate 2x the smaller.
The ratio IS the assertion, so there is no magic threshold to tune: the
bug is defined by allocation tracking the element count. Both figures come
from the repo's AllocationTests.MinAllocated (best-of-N floor, since
GC.GetTotalAllocatedBytes is process-wide and noise can only ever add).
It has teeth. Against the pre-fix code it fails on both TFMs:
found 24,582,112 B, expected < 6,155,952 B (net8.0)
found 24,582,080 B, expected < 6,155,888 B (net10.0)
and passes after. The class doc records the measurements above so a
working-set assertion is not reintroduced here.
The `using` on `full` is correct and stays. It is simply not observable
from a runtime probe at this scale, so it rests on review rather than on
a test that only looked like one.
Suite (CI filter, both TFMs): 11,652 passed / 0 failed / 11 skipped.
… prose Follow-up to 2ef3906, which shipped a `DocExamples_DocIntegrity` class that read the interop markdown out of the test output and asserted things about its TEXT: that every test name a page cited still existed, that the suite size it quoted was real, that the pythonnet version in its error block matched the guard, that no page named a renamed type. The csproj copied the pages (and the packaged README) next to the test binaries to feed it. That was the wrong instrument. Documentation prose is fluid: rewording a sentence, renaming a heading, dropping a citation or adding a page are all normal edits, and none of them should be able to turn a build red. A gate like that does not protect the reader, it taxes the writer — and it quietly makes the tests depend on file layout and on a build rule nobody remembers. What survives is the part that was always the point: the 48 `DocExamples_*` tests reproduce the code examples and assert the BEHAVIOUR the pages claim. They read no files. Every documented assumption they check is encoded as test data in C#: - the Python -> pythonnet table is a dictionary in the test, compared against `PythonRuntimeInterop.MinimumPythonnetFor` (the mapping the runtime guard uses), so the two copies of that knowledge cannot disagree; - the coverage census builds all 50 exporter varieties and asserts 47 view / 2 copy / 1 rejected as constants; - every verb, route, dtype row, layout row and thrown message is exercised directly. A page can now be rewritten however its author likes. It can only break the build by claiming something that is false. Removed with it: `pythonnet.md`'s exact test count ("202 tests"), which nothing can keep honest without policing the file — it is stated qualitatively now. The stale `NDArrayInterop` -> `NDArrayPythonInterop` names in the packaged README stay fixed; that class of rot is caught structurally instead, since every DocExamples test calls the real type and a rename breaks compilation. Suite: 197 tests, green on net8.0 and net10.0. No change to shipped code.
…p-lease deadlock
Closes gaps found auditing the four dtype maps (ToNumpyDtypeStr /
FromNumpyDtypeStr / ToBufferFormat / FromBufferFormat) for whether they
cover everything NumSharp can represent, plus the logically-adjacent
interop paths that feed them. Also fixes a real deadlock the new scenario
coverage surfaced.
Import dtype coverage (FromBufferFormat / ToNDArray)
----------------------------------------------------
Char previously had NO import route (export-only, round-tripping as
UInt16). Text buffers now map by width:
- PEP 3118 'u' itemsize 2 (Windows wchar_t: array.array('u'),
ctypes.c_wchar) -> zero-copy Char VIEW. A UTF-16 code unit IS
System.Char, so this is bit-exact. NumPy itself lists 'u' as
unsupported ("UCS-2 strings"), so this legitimately exceeds NumPy:
NumSharp natively has the dtype NumPy lacks.
- 'u' itemsize 1 -> Byte (degenerate single-byte text unit).
- UCS-4 text -- numpy '<U1' (exports the count-prefixed '1w'), 4-byte
'u' (linux/macOS wchar_t), Python 3.13's array.array('w') -> ToNDArray
copy-NARROWS to Char, symmetric with the existing complex64 widen.
BMP-only: an astral code point needs a surrogate pair, so it throws
with a precise message rather than silently emitting half a pair.
The view path declines (FromBufferFormat throws copy guidance); codec
Auto then falls back to the copy transparently.
FromNumpyDtypeStr gained the guided '<c8' (complex64) and '<U1' (UCS-4)
messages so the non-contiguous __array_interface__ path reports the same
"widens/narrows via ToNDArray" guidance as the buffer-format twin,
instead of the generic "has no NumSharp dtype".
Also: Windows np.longdouble was wrongly rejected -- numpy exports buffer
format 'g' at every width, and MSVC long double IS IEEE double, so 'g'
itemsize 8 now views as Double; extended-precision widths keep a guided
rejection.
__array_interface__ robustness (ViewViaArrayInterface)
------------------------------------------------------
- The 'data' field is type-gated before parsing. PySequence_Tuple would
turn a bytes 'data' (which PIL.Image emits) into a tuple of byte
VALUES, promoting the first pixel byte to a pointer -- a poisoned
view one strict As<bool> away from an access violation. Non-tuple /
missing / short 'data' now raise guided NotSupportedException; a
buffer-object 'data' (PIL) is refused cleanly in both view and copy.
- The readonly flag is read by truthiness, not As<bool>, so the 0/1 int
flags real-world producers emit are accepted (previously an opaque
InvalidCastException) and carry writability correctly.
ToMemoryView: contiguous offset windows
---------------------------------------
np.unstack / np.split children are C-contiguous views at offset>0 over
the whole parent block. ToMemoryView now exports exactly their
[offset, offset+size) window (address + offset*itemsize) instead of
rejecting offset != 0; non-contiguous views still throw with guidance.
Deadlock fix: lease drain vs GIL-bouncing dealloc (PythonRuntimeInterop)
------------------------------------------------------------------------
An mmap's dealloc wraps its unmap syscalls in Py_BEGIN/END_ALLOW_THREADS,
so PyBuffer_Release DROPS the GIL mid-call and re-takes it. DrainPending
(and OnEngineShutdown) held _drainGate ACROSS the disposal, so a second
drain that won the dropped GIL parked on the gate while the disposing
thread waited to re-take the GIL -- a GIL/gate lock-order inversion
(confirmed via dotnet-stack: one thread in PyBuffer_Release holding the
gate, the ThreadPool worker holding the GIL parked on Monitor.Enter).
The gate now guards only dequeue + teardown-state reads; disposal runs
outside it. ImportLease disposal is idempotent, so concurrent drains
alternating over the queue stay correct. Repro went from hanging 5/5 to
surviving 3/3.
Refactor: DtypeCompatibilityKind
--------------------------------
Replaced the two booleans threaded through ToNDArray -> CopyBuffer
(widenComplex64, narrowUcs4) with a single private enum
{ Blit, WidenComplex64, NarrowUcs4 }. ResolveDtypeCompatibility now
classifies the element type once and returns both the kind and the
destination dtype it implies; CopyBuffer switches on it. Illegal states
(both bools true) are unrepresentable. Pure refactor, no behavior change.
Tests + docs
------------
- New ViewabilityMatrixTests pins ~88 import scenarios to
view/copy/rejected: builtins, mmap, SharedMemory.buf, np.memmap, numpy
scalars, numpy layouts (offset slices, columns, read-only diagonal()),
ctypes (multi-dim, 0-d scalar, c_bool/c_char/c_wchar), memoryview casts
(typed, 2-D/3-D, toreadonly), every array.array typecode, text buffers,
and the __array_interface__ fakes -- with platform-computed
expectations for wchar width and longdouble.
- DtypeMapTests: 'u'/2->Char, 'u'/1->Byte, UCS-4 guided throws, 'g'
width rule, FromNumpyDtypeStr complex64/UCS-4 guidance.
- ExportTests: ToMemoryView on unstack/split children (window
correctness, write-through, untouched neighbours).
- DocExamples (pythonnet / zero-copy-model pages) extended in lockstep
with the prose; the 50/47/2 census is unchanged.
- Docs: dtype table Char row, "four things that cannot be viewed" + a
UCS-4 section, README, and codec/route XML docs.
Full interop suite green: 216/216 on net8.0 and net10.0.
…ocExamples gate
The interop documentation set is rebuilt from scratch against the agreed writing
specification (docs/plans/interop-docs-spec.md, committed here) and fully gated:
every behavioural claim on every page names a test that exists and passes.
Suite: 234 tests, green on net8.0 and net10.0, zero skips on a full third-party
stack (torch, PIL, pyarrow, pandas, polars, cv2).
Pages (docs/website-src/docs/interop/):
- index.md the landing page: the three-capability contract every bridge
builds on (raw layout access, external-memory wrapping with a
release hook, last-reference ARC release) + the bridge table,
incl. the byte-exact .npy/.npz row. Its gates run without
Python - the contract is NumSharp's alone.
- pythonnet-numpy.md the package reference: the four verbs, measured costs (view
verbs flat in n, copy verbs linear, crossover ~10K elements),
layout fidelity incl. the ndarray->c_char_Array_N base chain,
the three import routes + the 50-variety census (47 view /
2 copy / 1 rejected), lifetime both directions, the codec,
the GIL, dtypes, versions.
- np-frombuffer.md (authored in the previous session, now fully gated) reaching
any Python library through ToMemoryView + the buffer protocol.
- numpy-net.md coexistence with SciSharp's Numpy.Bare on one shared engine:
wrap (new NDarray(nd.ToNumpy())) / unwrap (their.self
.AsNDArray()) idioms, the Numpy.NET GIL rule, a matmul
cross-check against NumSharp's own, lifetime across three
facades, the CS0433 flavor trap.
The four previous pages (index, pythonnet, zero-copy-model, numpy-net) are
deleted UNREAD per the clean-slate decision; no prose was carried over. The
toc.yml Interoperability node is rewired to the new filenames.
New discovery documented and gated on the way: pythonnet caches DECODER LOOKUP
MISSES per (Python type, target type) pair, so an As<NDArray>() attempted before
RegisterCodec() permanently poisons that exact pair for the engine session - even
after registration - while untouched pairs work fine. Page 2 documents 'register
at startup' as the rule with the observed transcript;
Codec_RegisterBeforeFirstConversion_OrThePairIsPoisoned proves both branches
using ctypes array types (c_short_Array_7 vs c_short_Array_8), whose per-length
tp_names give the test a probe pair no other test can ever touch.
Second discovery: NumSharp's refcheck refusal reproduces NumPy's message
INCLUDING its internal line wrap ('...references or is referenced\nby another
array in this way.\nUse the np.resize function or refcheck=False') - gates
assert both fragments rather than a sentence-spanning wildcard.
Gate suite (test/NumSharp.Interop.UnitTests/):
- InteropTestBase.SkipUnless(module): imports under the GIL, Assert.Inconclusive
when absent - third-party gates are real proof where the package exists and
silent where it doesn't (spec par-8.3).
- DocExamples.NpFrombufferPage.cs (27 tests) - every claim of page 3, incl. the
flat-vs-linear timing ORDERING gate (generous margins, never absolute numbers),
the 11-probe import census, seven third-party gates, and a socketpair
round-trip that makes the thesis's 'a socket' claim true by test.
- DocExamples.PythonnetNumpyPage.cs (23 tests) - every claim of page 2; the
50-variety census relocated here from the deleted zero-copy class; the version
table compared against PythonRuntimeInterop.MinimumPythonnetFor so the two
copies of that knowledge cannot drift; verbatim troubleshooting symptoms.
- DocExamples.NumpyNetPage.cs (10 tests) - every claim of page 4 (wrap,
unwrap, slices, dtypes, matmul cross-check, lifetime, codec-through-wrapper,
pythonnet version unification with Numpy.Bare).
- DocExamples.InteropIndexPage.cs (7 tests, rewritten for the new page) - the
Python-free contract plus the bridge-assembly pin.
- Deleted: DocExamples.PythonnetPage.cs, DocExamples.ZeroCopyModelPage.cs - their
unique assertions (the census, the version table) moved into the new classes;
everything else stays covered by the standing suite (GilPolicyTests,
CodecModeTests, ImportTests, ExportTests, LifetimeTests, ...).
Spec updates ratified with the user this session: par-2.1 content-driven section
count confirmed (the spec's one open item, closed); par-3.1 gate links are
reference-style - one [gate] definition per target test file at the page bottom.
Verification: all 73 gate names cited across the four pages resolve to real
code; dotnet test green on net8.0 and net10.0 (234/234 both); docfx build clean
for the interop pages and the rewired toc (the one remaining warning is
pre-existing in NDIter.md); every transcript block measured live on CPython
3.12.12 / numpy 2.4.2 / pythonnet 3.0.5 / Numpy.Bare 3.11.1.33 / Windows 11.
…otes New docs/bugs/ knowledge base holding the two behavioural discoveries surfaced while writing the interop documentation gates (commit d970424): - pythonnet-decoder-cache-poisoning.md pythonnet's PyObjectConversions layer memoizes decoder resolution per (Python type, CLR target type) pair INCLUDING misses, so an As<NDArray>() attempted before RegisterCodec() permanently poisons that exact pair for the engine session - registration returns true, auto-encode works, fresh pairs decode, but the poisoned pair keeps throwing InvalidCastException ('numpy.ndarray' value cannot be converted to NumSharp.NDArray). Covers the minimal repro, the observed five-line transcript, the pair-granularity proof (ctypes per-length tp_names: c_short_Array_7 vs _8), root cause (cache cleared only by PyObjectConversions.Reset at engine shutdown), workarounds (register at startup; explicit verbs bypass the codec entirely), the dual-branch gate that keeps it pinned suite-safely, and the upstream-fix note. - numpy-resize-message-line-wrap.md NOT a defect - deliberate byte-level NumPy parity: NDArray.resize's refcheck refusal reproduces numpy's shape.c string including its two embedded newlines ('...references or is referenced\nby another array in this way.\nUse the np.resize function or refcheck=False'), so any single-line matcher (test wildcard, grep, docs quote) silently fails - exactly how the gate suite's only two first-run failures happened. Records both source origins (NDArray.resize.cs ~96-99 mirroring numpy _core/src/multiarray/shape.c ~101-105), numpy's own docstring dodging the wrap with an ellipsis (_add_newdocs.py:4248), the house matching pattern (assert per-line fragments, never span the wrap), the trap-free sibling message ('does not own its data'), and the rule for future gate authors: check vendored numpy C strings for embedded \n before pinning any parity error text.
Reworked the interop docs spec and index page to use noun/verdict-style subsection headings, stronger first-sentence claims, and deeper explanatory prose around the contract mechanics (layout, wrapping, release, and ownership). Updated bridge/compliance linking, added the new docs image asset, and synced Interop index doc-example test comments with the revised page language.
…igning Rebasing NumSharp.Interop.pythonnet onto journey3 moves it into journey3's consolidated strong-naming world (a single repo-root Open.snk driven by Directory.Build.props, with keyed InternalsVisibleTo everywhere). The package and its test project were authored before that overhaul, so two build breaks surfaced — both fixed exactly as NumSharp.Core / NumSharp.Interop.OpenBLAS already are: - src/NumSharp.Interop.pythonnet/AssemblyInfo.cs: its InternalsVisibleTo grants to NumSharp.Interop.UnitTests and NumSharp.DotNetRunScript were keyless, which is CS1726 from a now-strong-named assembly. Add PublicKey= via a PythonNetFriendKey const, mirroring OpenBLAS's BlasFriendKey. - test/NumSharp.Interop.UnitTests/NumSharp.Interop.UnitTests.csproj: it pinned AssemblyOriginatorKeyFile to ..\NumSharp.UnitTest\Open.snk — one of the five key copies journey3 deleted (CS7027 "key file not found") — and re-set SignAssembly plus a now-dead SIGNING define in a Publish-only PropertyGroup. Drop all three so it inherits Directory.Build.props like NumSharp.UnitTest. The Core-side keyed grants for NumSharp.Interop.pythonnet and NumSharp.Interop.UnitTests were folded into the rebased commits that introduce those assemblies. StrongNameTests stays green — it iterates the friend declarations and asserts each names the key, pinning no count. Verified: pythonnet, NumSharp.Interop.UnitTests and NumSharp.UnitTest build 0 warnings / 0 errors on net8.0 and net10.0.
…BLAS_PATH + OS-install scan + 32-bit names) and document it Runtime discovery in CBlasNative.AutoCandidates gains two tiers and both scipy distributions' bare names; README/GEMM_PARITY/CLAUDE.md are updated to match. CBlasNative.cs: - NUMSHARP_OPENBLAS_PATH: additive, NON-binding discovery location(s) (path-separator delimited files/dirs), probed after the bundled parity asset and before the ambient scan. The sibling of the BINDING NUMSHARP_PARITY_BLAS: "also look here", silently skipped when it holds no BLAS. - SystemBlasDirectories(): scans the conventional install prefixes of the package managers OpenBLAS's own install docs list (apt multiarch, Homebrew keg, MacPorts, conda tree, vcpkg triplet, /opt/OpenBLAS, /usr/lib64 ...) plus OPENBLAS_HOME / OPENBLAS_ROOT / VCPKG_ROOT / CONDA_PREFIX roots. Ranked BELOW the bundled asset and numpy.libs because these builds are a different compiler/config -> NOT byte-parity with NumPy (a correct, fast BLAS, not a bit-identical one). - Bare-name fallback gains the 32-bit spellings (libscipy_openblas, scipy_openblas) alongside the 64-bit ones. Both scipy distributions already bind at the symbol layer (scipy_cblas_*64_ ILP64 and plain scipy_cblas_* LP64); scipy-openblas32 is the ONLY build PyPI publishes for 32-bit x86 (win32/i686). Order is now: [binding] path/NUMSHARP_PARITY_BLAS -> bundled -> NUMSHARP_OPENBLAS_PATH -> numpy.libs -> system installs -> bare names. The new tiers cannot disturb parity: they rank below the parity sources and the binding row bypasses AutoCandidates. All 30 MatmulParityBackendTests remain green; interop + test project build clean. Docs: - README.md: discovery list 1..6, NUMSHARP_OPENBLAS_PATH + install-root rows in the Environment table, and a note that scipy-openblas64 (ILP64) and scipy-openblas32 (LP64) are both first-class (32 mandatory for 32-bit x86). - docs/GEMM_PARITY.md: §6.1 discovery table extended to 6 rows, a dated "Discovery widened (2026-08-13)" subsection, and a pointer from the stale §3 order to §6.1. - .claude/CLAUDE.md: discovery-order sentence + env-knob line updated. Verified against PyPI: scipy-openblas32 and scipy-openblas64 ship parallel versions (both the 0.3.31.22.0 pin and 0.3.34.106.0 latest); 32 additionally covers win32/i686. Not included (design only, not yet built): the consumer build-time version override (props/targets + NUMSHARP_OPENBLAS_VERSION/_URL/... knobs).
… bundled) + PATH sweep as last resort Two discovery-order changes in CBlasNative.AutoCandidates, per request: - NUMSHARP_OPENBLAS_PATH moves to the FRONT of the discovery chain — tried before the bundled asset and everything else, so a caller who sets it wins. Still NON-binding: it falls through to the rest when it holds no loadable BLAS (the BINDING override remains NUMSHARP_PARITY_BLAS, handled in Load() before AutoCandidates runs). Parity-by-default is unaffected — the variable is unset in the common case, leaving the bundled asset first. - PathEnvDirectories(): a new LAST-resort tier that sweeps every directory on PATH for a BLAS under a non-standard file name (a renamed OpenBLAS, a vendor CBLAS) that the bare loader names would miss. Broadest/most expensive scan; lazy iteration means it never runs when an earlier tier binds. New order: [binding] path/NUMSHARP_PARITY_BLAS -> NUMSHARP_OPENBLAS_PATH -> bundled -> numpy.libs -> system installs -> bare names -> PATH sweep. No parity regression: the binding row still bypasses AutoCandidates, the bundled asset stays ahead of numpy.libs, and the two ambient scans rank last. 30/30 MatmulParityBackendTests green; interop + test project build clean. Docs (README.md, docs/GEMM_PARITY.md §6.1, .claude/CLAUDE.md) updated to the 7-tier order and the corrected NUMSHARP_OPENBLAS_PATH semantics (priority-first, not "additive after bundled").
…overy model Adds docs/OPENBLAS_DELIVERY_DESIGN.md — a proposed design covering: - Bundle = PyPI-based default shipped in the nupkg, but moved to LAST resort (only when no tooling is found), and autoinstall renamed BundleAutoinstall. - numpy.libs discovery removed (never grab a numpy install's OpenBLAS). - Override on the PackageReference: OpenBlasPath (read-in-place, no version enforcement) or OpenBlasVersion (download from PyPI at build -> global temp cache -> output; hard-required). Env (NUMSHARP_OPENBLAS_*) wins over metadata. Override takes priority over bundle. - Same-artifact invariant: the bundle-extracted lib and the build-downloaded lib are byte-identical and share runtimes/<rid>/native; the wheel is discarded after extraction (no stray artifacts). - Transitive delivery via buildTransitive: OpenBlas* metadata honoured on ANY PackageReference (incl. MyPackage -> NumSharp.Interop.OpenBLAS), enabled by default; OpenBlasDelivery=package lets a dependent bake a pinned version into its OWN nupkg. - Build->runtime handoff via a source marker that flips the priority of runtimes/<rid>/native between "required override" (tier 2b) and "bundle" (tier 4). Status: DESIGN (proposed). §10 lists 6 open decisions needing sign-off — chiefly 10.1, that bundle-last drops parity-by-default (recovered by pinning a version). Supersedes the discovery model in GEMM_PARITY.md §6.1 once implemented.
…(marker tiers, rename, numpy.libs removal)
Implements the RUNTIME half of docs/OPENBLAS_DELIVERY_DESIGN.md (the BUILD half —
buildTransitive targets, download/stage task — comes next). §10.1 was resolved with
explicit sign-off: the bundle stays ABOVE machine tooling (the doc's alternative),
preserving parity-by-default; the originally-stated bundle-last model was rejected
because a distro OpenBLAS is not byte-identical to NumPy 2.4.2 and binding it by
default would have made every "bit-identical" claim conditional on the machine's
package-manager history.
Discovery order now (CBlasNative.Load orchestrates; each tier is its own stream):
1. BINDING explicit path / NUMSHARP_PARITY_BLAS (unchanged, exclusive, fatal miss)
2a. OVERRIDE NUMSHARP_OPENBLAS_PATH (env wins over metadata), then a path-mode
source marker recorded by the build; non-binding, falls through
2b. OVERRIDE a version override staged by the build — HARD-REQUIRED: a miss throws
BlasRequiredOverrideException, never falls through to the bundle or
machine tooling (the pin is a contract; substituting a same-layout,
plausibly same-bytes library is the failure a pin exists to prevent);
sha256-verified per candidate when the marker pins a hash
3. BUNDLE runtimes/<rid>/native — the zero-config parity default (§10.1)
4. MACHINE system install dirs -> bare loader names -> PATH sweep
TOOLING (numpy.libs REMOVED)
New: OpenBlasSourceMarker (openblas.source.json) — the build-phase handoff that flips
a staged folder between "required version override" (tier 2b) and "bundle" (tier 3).
Needed precisely because the design's "same binary" invariant makes the two
content-identical: only the marker can tell them apart. Schema:
{ mode: version|path|none, distribution, version, sha256, required, path }
mode=version defaults required=true (an explicit required:false downgrades a miss to
fall-through); path resolves relative to the marker's own directory; probing covers
<base>/openblas.source.json and <base>/runtimes/<rid>/native/ for both ProbeBases
(AppContext.BaseDirectory + assembly dir). TryFind never throws (module-init path);
an existing-but-unparsable marker gets one stderr line and is ignored.
New: BlasRequiredOverrideException : DllNotFoundException — its own type so the
module-load path can tell a broken CONTRACT from the ordinary no-BLAS-anywhere case.
Blas.BundleAutoinstall catches it, reports to stderr, and leaves the backend
UNINSTALLED (never substituted): throwing from a [ModuleInitializer] would surface as
TypeInitializationException at an unrelated first touch, breaking the "merely
referencing the package cannot break the app" promise; an explicit Blas.Enable()
throws the full exception. Everything else stays silent as before.
Renames (design §4/§9): Blas.AutoInstall -> Blas.BundleAutoinstall;
NUMSHARP_BLAS_AUTOINSTALL=0 -> NUMSHARP_BLAS_BUNDLE_AUTOINSTALL=0 with the old
spelling honoured as a deprecated alias for one release. The test-run guard
(BlasEngineAutoInstallGuard) now sets both spellings.
Removed: PythonLibDirectories() — the numpy.libs scan over VIRTUAL_ENV/CONDA_PREFIX/
PYTHONHOME/every python on PATH (design Goal 5: never grab OpenBLAS out of a numpy
installation; the bundle already IS that binary at the pinned version). CONDA_PREFIX
survives only as a machine-tooling root in SystemBlasDirectories (a conda-installed
openblas is tooling; a numpy is not). The no-BLAS error text no longer points at
numpy.libs — it now names the bundle and the env knobs.
Refactor kept behavior-preserving: the old AutoCandidates loop body became
TryLoadCandidates (a loadable non-CBLAS candidate still THROWS from Bind with the
previous binding untouched, exactly as before); the ambient stream (bundle -> system
-> bare names -> PATH sweep) is AmbientCandidates; ProbeBases was extracted from
BundledDirectories so the marker probe and the asset probe agree on where "next to
the app" is. Blas.IsBundledLibrary is now marker-aware: a folder the marker declares
an override read-location reports false even though layout (and possibly bytes)
match the bundle.
Gates: new OpenBlasDeliveryTests (10) — the rename + [ModuleInitializer] pin, both
env spellings suppress, autoinstall installs with no opt-out, version-marker
hard-require (throws with the bundle loadable — proving no fall-through), sha
mismatch refuses the file, loud-but-not-fatal module-load behavior, path-marker
falls through / binds in place, numpy.libs tier stays deleted. MatmulParityBackendTests
30/30, StrongNameTests 6/6, matmul_parity corpus tier green. One test needed
CBlasNative.Unload() to reproduce process-start state: Blas.Enable() reuses an
already-loaded library and skips discovery (its documented short-circuit), which in
production cannot precede the marker check because markers are written at build time.
Docs synced: OPENBLAS_DELIVERY_DESIGN.md (§10.1 RESOLVED, tier diagram, §4/§8
implemented notes, checklist flipped); GEMM_PARITY.md §6.1 (new 7-row order,
numpy.libs removal note); package README (discovery order, env table incl. the
deprecated alias); .claude/CLAUDE.md OpenBLAS section.
…ve targets, PyPI fetch task, caching, transitive + package modes
Implements the BUILD half of docs/OPENBLAS_DELIVERY_DESIGN.md (§5–§7) — the design is
now fully implemented. Verified end-to-end by the new scripted integration gate
tools/verify_build_override.sh: 9 steps against the REAL packed nupkg restored from a
local feed, all green (default no-op, metadata version override + runtime binding,
cache hit, wrong-version hard fail, version+path combined, override removal, path-only
fall-through, both publish layouts, no-wheels invariant).
buildTransitive/NumSharp.Interop.OpenBLAS.{props,targets} ship in the nupkg (csproj
packs them plus a pack-time copy of tools/openblas-manifest.json as
buildTransitive/openblas-manifest.json — the shared extraction contract, §5.4; the
tools/ copy stays the single checked-in pin). NuGet flows buildTransitive to DIRECT
AND TRANSITIVE consumers, which is what gives a dependent package's consumers the
identical override experience (§6.1): the resolve target scans EVERY PackageReference
for OpenBlas{Version,Path,Distribution,Feed,Sha256,Delivery} metadata; environment
variables (NUMSHARP_OPENBLAS_*) beat metadata; per-§10.2 the highest version among
references wins (logged) and a version-vs-bare-path conflict across references is a
hard error.
The inline RoslynCodeTaskFactory task (NumSharpOpenBlasFetchTask) performs §5.3's
pipeline: resolve → PyPI JSON (<feed>/pypi/<dist>/<version>/json) → pick the wheel by
the RID's platform-tag anchor (derived from the packed manifest's wheel filename, so
the RID→distribution/tag map is literally fetch_openblas.py's RID_MAP — the "same
binary" invariant §5.4) → download the wheel to a temp file → verify the wheel sha256
from the index → extract the ONE native member (prefer /lib/, largest — exactly
pick_native_member) → verify the extracted lib sha256 (explicit OpenBlasSha256, or the
manifest pin when the version+distribution equal the default; a non-pypi.org feed
REQUIRES an explicit sha, §10.6) → store in the global per-user cache keyed
<distribution>/<version>/<rid>/<sha256>/ (%LOCALAPPDATA%/NumSharp/openblas or
$XDG_CACHE_HOME/$HOME/.cache; EXTRACTED LIBS ONLY — the wheel is deleted, Goal 7;
cache hits serve offline) → copy over the bundle in the output (idempotent: skipped
when the destination already hashes to the key) → write the openblas.source.json
marker. Any version-mode failure is Log.LogError → THE BUILD FAILS (Goal 3, §10.5) —
never a silent fallback to a different binary.
Marker placement (the §8 handoff, as implemented): next to the staged binary with no
"path" field (its own directory is the read location), plus a root-level marker
carrying "path" only when staging went to a custom OpenBlasPath the runtime's
app-relative probe could not otherwise find. A flattened RID-specific publish stages
into the publish root, so its marker IS the root one. Stale-marker hygiene across
every transition: override removed → MSBuild <Delete> clears both locations; mode
switched (version↔path, default↔custom dir) → the task deletes the superseded
location. Two bugs found and fixed while proving this: the flattened case initially
deleted the marker it had just written (destDir == output root — now guarded by
SamePath), and a default→custom-dir switch left the old rid-dir marker behind.
Staged RID defaults to the host ($(NETCoreSdkPortableRuntimeIdentifier)) or
$(RuntimeIdentifier) when set. OpenBlasDelivery=package (§6.2) stages ALL manifest
RIDs and adds binary+marker per RID as TfmSpecificPackageFile pack content — verified:
MyPackage.Test.nupkg carried runtimes/<rid>/native/{lib,openblas.source.json} for all
8 RIDs, every sha equal to the manifest pin, win-arm64 correctly scipy-openblas32.
NuGet nearest-wins makes the dependent's copy beat this package's transitive bundle.
delivery=package + OpenBlasPath is rejected (they contradict). Publish staging hooks
CopyFilesToPublishDirectory: portable → runtimes tree + rid marker; RID-specific →
flattened next to the app + root marker; both verified to bind at runtime
(bundled=False, product correct).
Two MSBuild traps are encoded in comments because each silently broke the feature
during bring-up: (1) a target's Condition is evaluated BEFORE its DependsOnTargets
run, so gating the stage target on the resolve-set property made it never fire — the
has-override gate sits on the TASK element (evaluated at execution, after the
dependency), leaving default builds with nothing but a cheap item scan and the task
never compiled; (2) RoslynCodeTaskFactory compiles against netstandard2.0's DEFAULT
references and simple-name <Reference> items fail with MSB3755 on the dotnet-sdk
MSBuild — the task therefore uses only netstandard2.0 surface (HttpClient, ZipArchive
over FileStream instead of ZipFile.OpenRead, SHA256) and a small self-contained
MiniJson reader/writer instead of System.Text.Json (bonus: works under VS's .NET
Framework MSBuild; the strict parser fails loudly — it parses PINS). Documented as
§5.3 implementation deviations.
The verify script hardens against two host quirks: MSYS mktemp paths are invisible to
the Windows dotnet/NuGet processes (cygpath -m normalization; the embedded
nuget.config feed path was silently breaking restore), and dotnet pack defaults to
Release on .NET 8+ while this repo's incremental build can declare stale Release
outputs up-to-date — the script force-rebuilds (-t:Rebuild) before packing, which is
also how a stale pre-change DLL got packed and masqueraded as a runtime bug during
bring-up.
Gates: verify_build_override.sh 9/9; OpenBlasDeliveryTests 10/10;
MatmulParityBackendTests 30/30; matmul_parity corpus tier green. Docs synced:
OPENBLAS_DELIVERY_DESIGN.md (status IMPLEMENTED, §5.3 deviations, §11 checklist all
DONE, §12 references), package README (override section + env table), .claude/CLAUDE.md.
…cy alias, self-healing cache, adversarial gates
Finalizes docs/OPENBLAS_DELIVERY_DESIGN.md: no backwards compatibility is carried
anywhere in the OpenBLAS area, and the delivery mechanism was attacked deliberately —
the three real weaknesses that fell out are fixed and pinned by new gates.
LEGACY REMOVED (no alias, no compat):
- The pre-rename NUMSHARP_BLAS_AUTOINSTALL spelling is RETIRED OUTRIGHT and ignored —
the "deprecated alias for one release" plan was dropped because the rename never
shipped in a release, so there is no installed base to migrate. BundleAutoinstall
now reads ONLY NUMSHARP_BLAS_BUNDLE_AUTOINSTALL; the test-run guard sets only the
new name; README/design/GEMM_PARITY/CLAUDE.md scrubbed. The REMOVAL itself is
pinned: RetiredAutoinstallAlias_NoLongerSuppresses proves the old spelling is a
no-op (autoinstall proceeds despite it), so a quietly reintroduced alias turns a
test red. (The verify script no longer unsets the dead variable either. A stale
NumSharp.Interop.BLAS.0.60.0.nupkg under gitignored packages/ is a build artifact
of a long-gone package id, not a repo asset — nothing to remove from the tree.)
BREAK-IT FINDINGS, FIXED:
1. Poisoned cache trusted on hit (build task): FindInCache trusted the directory
NAME as the content hash without verifying the FILE — a truncated write (killed
build) or a tampered entry would be staged silently and then hard-fail at RUNTIME
forever (the marker's sha check refuses it) when the BUILD could have healed.
Every cache hit now re-hashes the entry; a mismatch is logged ("DISCARDING
poisoned cache entry ..."), the entry deleted, and the download re-runs — the
cache self-heals (§10.6: never silently substitute; fetch_openblas.py treats its
poisoned wheel cache the same way). Cost: one ~20 MB hash per warm override build.
2. Non-atomic cache writes: File.Copy straight to the content-hash name meant a
killed build could strand a half-written file that (1) would then have had to
catch. Writes now go through a unique temp INSIDE the entry directory + rename.
3. Concurrent-build race: two msbuild processes downloading the same entry could
clobber each other mid-copy and fail spuriously; with temp+rename the loser of
the rename keeps the winner's file — identical verified content by construction —
and only a genuine IO failure still throws.
4. Runtime IsBundledLibrary vs file-valued marker paths: a path-mode marker may
legally name the library FILE (Expand accepts files), but DeclaresOverrideFor
compared directories only, so a library bound through such a marker misreported
as the bundle. A file-valued marker entry now also matches its parent directory
(guarded File.Exists check, no false positives from unrelated parents).
COVERAGE GROWN — OpenBlasDeliveryTests 11 -> 20 (all green):
- Priority pins: an explicit binding (NUMSHARP_PARITY_BLAS) bypasses a BROKEN
version marker (tier 1 over 2b — a dead pin cannot take down a caller who named
their binary); NUMSHARP_OPENBLAS_PATH outranks a broken version marker (2a over
2b — env wins over build metadata).
- Hostile markers: corrupt JSON is ignored and discovery proceeds (one stderr line,
no module-load break); an unknown mode is neither an override nor an error
(forward compatibility — the folder reads as the bundle); required:false
downgrades a version-marker miss to fall-through (the schema's escape hatch).
- Layout/format edges: a relative marker path resolves against the MARKER's own
directory (xcopy-deployable pair); a rid-dir marker with no root marker is found
(the §6.2 packed-dependent layout); the sha comparison is case-insensitive
(Convert.ToHexString is uppercase); a file-valued path marker binds that file AND
reads as an override (pins fix 4).
verify_build_override.sh grown 9 -> 15 steps (all green end-to-end):
5. tampered artifact (wrong expected sha) -> CHECKSUM MISMATCH build failure —
the tamper-detection proof; deliberately re-downloads one wheel per run
6. non-pypi.org feed without an explicit sha -> refused BEFORE any network
7. invalid OpenBlasDelivery value -> refused, names the valid set
8. OpenBlasDelivery=none -> the reference's metadata is ignored entirely
9. version on one reference vs bare path on another -> the §10.2 hard conflict
10. poisoned cache -> discarded, re-downloaded, staging completes (pins fixes 1-3)
The script now rewrites the consumer csproj fresh per scenario (write_csproj) instead
of sed-mutating it, so no step depends on a previous step's edits.
Gates: OpenBlasDeliveryTests 20/20, verify_build_override.sh 15/15,
MatmulParityBackendTests 30/30, matmul_parity corpus tier green, FULL CI-style suite
12,733 passed / 0 failed. Docs synced: design doc (status IMPLEMENTED & FINALIZED,
§4/§9 rename tables record the outright retirement, §7 cache-integrity notes, §11
checklist), package README (cache verification note, env table), GEMM_PARITY (guard
env name), .claude/CLAUDE.md (self-healing cache + gate inventory).
…full doc sweep Read every tracked doc that grep-hits the OpenBLAS delivery terms and reconciled each to the finalized model. Two genuine staleness bugs, both from the incremental edits not fully catching the §10.1 "bundle above tooling" resolution: - GEMM_PARITY.md §1: the "initial cut" discovery narrative still listed the numpy.libs scan as a live tier and deferred to §6.1 for the "real" order. Rewritten as an accurate summary of the current order (binding → override paths → version override → bundle → machine tooling), noting numpy.libs is no longer an auto-scanned tier, and pointing at both §6.1 and OPENBLAS_DELIVERY_DESIGN.md. - OPENBLAS_DELIVERY_DESIGN.md §8: two numbering/framing errors against the doc's OWN resolved §3 ordering. "Tiers 2 and 4 both look at runtimes/<rid>/native" → the override tier (2b) and the BUNDLE tier (3) — the bundle is tier 3, not 4 (tier 4 is machine tooling). And "last-resort when it is the bundle" still carried the rejected bundle-last framing → "the parity default (tier 3, above machine tooling but below any override)". Verified clean, no action needed: - AGENTS.md is a symlink to .claude/CLAUDE.md (the "Replace Claude imports with symlinks" commit) — already current from the CLAUDE.md edits. - README.md: read in full, already reflects the whole new system (override tiers, build-time override section, transitive/package modes, self-healing cache note, retired autoinstall with no alias). - GEMM_PARITY §6.1 (7-row flat table) and the design doc §3 (4-tier grouped scheme) number the same order at different granularity — both internally consistent and cross-referenced, not a contradiction. - The two oracle skill references hit only "delivery" in the test-coverage sense. - SIGNING_HANDOVER.md references the package only for strong-naming / release-pipeline — orthogonal to delivery; the buildTransitive additions are unsigned text assets that pack automatically, so its signing/release story is unchanged. A cross-doc sweep confirms every surviving "last resort" / "numpy.libs" / bare "NUMSHARP_BLAS_AUTOINSTALL" mention is legitimate (the §10.1-rejection rationale, the PATH-sweep-is-last-resort note, the removal changelog, or the rename-mapping table).
… close the coverage gap Audit follow-up to the NDScope/weaver hardening: the scoped set (53) covered the hot reduction/statistics/set-op/selection surface but NOT every composition that owns transient NDArray intermediates. The stats class turned out covered transitively (std/var -> scoped ReduceStd/ReduceVar; median/percentile/quantile + nan* twins -> scoped QuantileEngine), but the linear-algebra and core-math compositions delegated to nothing scoped and reclaimed their temps only via the finalizer. Scope the eight transient owners with [NDScoped] (the weaver injects the scope; sites keep their original bodies): - np.cross multiply/subtract/negative/concatenate/astype fold - np.kron multiply/tile/reshape/broadcast_to - np.outer multiply/expand_dims (the @out param is a caller input -> Returns no-op) - np.tensordot (core) transpose/reshape views + dot product + final reshape-over-product view - np.linalg.matrix_power binary exponentiation (matmul folds; inv for n<0); work=a for n>=0 is an untracked input, results are fresh copies/products - np.diff / np.ediff1d subtract/concatenate/ravel/astype - EinsumContract the einsum contraction core. Its single-operand VIEW path returns a writeable view re-aliased from the OPERAND's storage; under the scope the view is yielded (survives) and keeps the operand buffer alive via ARC, so einsum('ii->i', a)[:] = 1 still writes through to a (pinned by EinsumContractionTests.ViewPath_DiagonalIsAWriteableView_ThatWritesThrough). All eight are safe by the weaver's structural invariant (inputs constructed before the scope opens are never tracked; the return value is yielded; a returned view over a tracked transient stays valid by refcount), so the migration is byte-neutral by construction. Verification (DISPOSAL-GUIDELINES section 12): weaver reports woven 61 (was 53), 0 NDW errors, both TFMs; affected-op tests 410/410 (einsum/tensordot/cross/kron/outer/matrix_power/diff/ediff1d + the einsum writeable-view contract); NDScopeWeaveTests gate 61/61 on net8.0 AND net10.0; FuzzMatrix byte-exactness 88/88; full NumSharp.Tests net8.0 13992/0 (unchanged from baseline). DISPOSAL-GUIDELINES section 11 gains a coverage note: what is covered directly vs transitively, the deliberate non-targets (single-kernel ufuncs, views, kernel-driven nan-mean/std/var, the thin product wrappers inner/vdot/vecdot/matvec/vecmat), and the remaining opportunistic tail (grid/creation, N-D FFT, polynomial) to migrate when a profile shows them hot.
…r ([NDScoped])
Exhaustive-sweep follow-up: after the LinAlg/Math tier (e2cb6600), walk the remaining np.*
composition surface and scope the transient owners that carry clear reclaim value. Woven count
61 -> 78 (+17), each site keeps its original body (weaver-injected).
Newly scoped (all weaver-compatible NDArray returns, no static NDArray caching, safe by the
weaver's structural invariant):
- linalg.norm abs/power/sum/sqrt/max reduction composition (hot: optimization loops)
- linalg.multi_dot Cormen-ordered chain of pairwise dot products
- vander per-column power composition
- polynomial family poly, polyval (Horner), polyder, polyint, polyadd/polysub (PolyAddSub),
polymul (convolution) — NDArray-returning, own concatenate/multiply temps
- N-D FFT (8) fft2/ifft2/fftn/ifftn/rfft2/rfftn/irfft2/irfftn — each RawFftNd is a pure
per-axis 1-D composition owning one intermediate NDArray per axis; the @out
param is a caller input (Returns no-op). Byte-exactness is delicate here, so
re-confirmed against the fft.jsonl fuzz tier.
Deferred (documented in DISPOSAL-GUIDELINES section 11), NOT scoped:
- carrier-return compositions the weaver rejects (NDW003) -> hand-scope if profiled hot:
meshgrid/mgrid/ogrid (grid result structs), polydiv/polyfit (tuple / PolyfitResult),
and the factorization tuples svd/qr/eig/eigh/lstsq/slogdet (also backend-only).
- one-shot structural ops pad/insert/delete/block and the r_/c_ construction indexers -
section 11 rates these low-priority (not hot-loop / small-N); migrate opportunistically.
Verification (section 12): weaver woven 78, 0 NDW errors, both TFMs; FuzzMatrix byte-exactness
88/88 (fft/poly/linalg tiers exercised); affected-op tests 404/404 (Fourier / Polynomial /
linalg norm+multi_dot / vander + the NDScopeWeaveTests gate at 78/78 on net8.0 AND net10.0); full
NumSharp.Tests net8.0 13992/0 (unchanged from baseline). A lone RandomParity_MultivariateNormal
'failure' seen while filtering was a pre-existing [OpenBugs] test the ~Norm filter matched by
coincidence (MultivariateNormal), not a regression.
…ted functions The existing nesting tests use HAND-WRITTEN scopes (NDScopeTests) or assert structure (NDScopeWeaveTests proves every [NDScoped] method carries a scope local). Nothing drove the WOVEN Core methods themselves through nesting behaviorally. New NDScopeWeaveNestingTests does, in three shapes, each pinning values + zero net buffer strands + untouched inputs: - Real woven-calls-woven IL: np.roll(a) with axis=null recurses into the woven roll(ravel, shift, 0), so two woven scopes nest with NO hand-written scope anywhere. The zero-strand variant proves BOTH nested scopes reclaim (a leaked inner scope would read ~N strands). - Enclosing scope over many woven calls (cross/diff/roll/kron/outer/vander/polyval/linalg.norm) whose results are all DROPPED: each woven method's child scope re-parents its result up to the enclosing scope, which reclaims them on close; inputs stay untouched. Plus a woven result fed into another woven call under the same scope (result-flow between nested functions). - A deep chain of nested functions (NestLevel reproduces the weaver's exact Open/temp/Returns pattern) with a real woven roll at the leaf: value + input-safety + a zero-strand variant, and the exception path — a throw after nested woven calls under an enclosing scope reclaims at every level and leaves inputs intact. 7 tests, green on net8.0 AND net10.0; the full Lifetime.NDScope suite is 34/34.
… model) Evaluation-driven tests validating that NDScope is thread-CONFINED: the scope stack is [ThreadStatic] (t_current + single-slot t_pool), every instance field is touched only by the owning thread, and there is no shared mutable state and no lock. New NDScopeThreadSafetyTests fills the angles NDScopeStressTests leaves: - SharedReadOnlyInput_ConcurrentScopedOps_UntouchedAndCorrect: one shared array read by woven scoped ops on 8 threads x 500 iters; asserts it stays untracked (TrackingScope == null), alive, and value-correct (roll flat[0] == 64-k) throughout. - ThreadLocalIsolation_WorkerDoesNotSeeMainScope_NorIsReclaimed: with a scope open on the main thread, a worker thread sees NDScope.Current == null and its array is not tracked by (nor reclaimed by) the main scope — direct proof of thread-local isolation. - ConcurrentScopes_EachThreadSeesOnlyItsOwnStack: 8 threads open/nest/close scopes concurrently, each always sees its own scope as current and its own temps tracked by its own scope. - CrossThreadHandoff_ReceiverAttachesIntoScope_Reclaims: full detach -> hand off -> NDScope.Attach into the RECEIVER's scope -> reclaim-on-close cycle across threads. - ManyThreads_NestedWoven_Concurrent_ZeroErrors_BoundedStrands: the woven nested op (roll axis=null -> roll) on 8 threads x 800 iters, value-correct with a bounded strand slope. The Debug thread-affinity asserts on Returns/Attach/Detach are live during these runs and never trip. 5 repeated net8.0 runs + net10.0 all green; full Lifetime suite 46/46.
…orm overload gap
Cross-referencing the [NDScoped] set against the AUTHORITATIVE api inventory
(coverage/NumSharp.Tools.ApiInventory: 562 public functions across np/ndarray/np.linalg/
np.fft/np.random) for the first time surfaced composition gaps the earlier directory-based
sweep missed. Woven 78 -> 86 (+8), each an NDArray-returning composition:
- linalg.pinv conjugate + svd + reciprocal + matmul (backend-only)
- linalg.cond svdvals + divide
- linalg.matrix_rank svd + amax + multiply + sum (>=2d path)
- linalg.tensorinv reshape + inv + reshape
- linalg.tensorsolve reshape + solve + reshape
- linalg.norm the int[]-axis overload (ravel + abs^2/dot + sqrt + reshape) -- an
OVERLOAD-COVERAGE gap: the int? overload was scoped, this workhorse (also
the target of vector_norm/matrix_norm) was not, so a direct int[] call leaked.
- fft.hfft / ihfft conjugate + irfft/rfft + norm swap (Hermitian composition)
All byte-neutral by the weaver's structural invariant. Verified: woven 86, 0 NDW errors;
FuzzMatrix 88/88; 602 linalg/fft tests + NDScopeWeaveTests gate at 86/86.
Honest status: this does NOT yet make coverage exhaustive against the inventory -- a real
composition tail remains (linalg eigvals/roots, ndarray.choose, np extract/place/angle/
searchsorted/mask_indices, structural append/stack/tile/bmat, and the entire unaudited
np.random). Carrier-returns polydiv/polyfit/svd/qr/eig tuples stay hand-scope-deferred.
…tion of 10 np.* functions Per-function verification on two axes at once: (1) correctness vs NumPy across representative cases, and (2) allocation discipline -- every buffer the function allocates (internal transients AND result) is reclaimed, measured as pool acquisitions - releases over 200 cycles with the result disposed each cycle (a single undisposed internal per call reads as ~200; inputs are also asserted untouched -- never disposed, never adopted into the function's scope). Five heavy COMPOSITIONS that drive many np.* internally -- np.cross (astype/moveaxis/multiply/ subtract/negative/concatenate), np.corrcoef (cov/dot/sqrt/divide/outer/clip), np.union1d (ravel/ concatenate/unique), np.kron (multiply/tile/reshape/broadcast_to), np.vander (power/concatenate) -- and five kernel/manipulation ops -- np.clip, np.sort, np.take_along_axis, np.roll (2d axis=null, woven roll nesting woven roll), np.where. Measured deficits are EXACTLY 0 for all ten, and the acq/rel counts confirm the compositions really do allocate internal transients that all get reclaimed: corrcoef 1600/1600 (8 buffers/ cycle), vander 1000/1000 (5/cycle), cross 600/600 (3/cycle), union1d 400/400 (2/cycle) -- so the scope is doing real work, not passing because there was nothing to reclaim. 5 repeated net8.0 runs + net10.0 green; full Lifetime suite passes together.
…o double-free) Covers the interaction of a hand-written NDArray.Dispose() on a TRACKED array with the scope's later reclamation sweep, which disposes the same array again. Load-bearing property: freed EXACTLY once -- Dispose is CAS-idempotent, so the sweep's second dispose is a no-op and the buffer is never double-returned to the pool (a double-free would hand the same buffer to two later allocations = silent aliasing corruption). Seven cases, asserting via pool-counter deltas + IsReleased + IsDisposed: - ManualDisposeOfTrackedTemp_ScopeSweepIsNoOp_FreedExactlyOnce: manual Dispose frees the buffer and leaves the temp STILL tracked; the sweep visits it again and must add ZERO further releases. - DoubleManualDispose_ThenScopeSweep_FreedOnce: two hand disposes + the sweep = one free. - ManualDisposeBase_YieldedView_StaysValid_FreedOnceAtViewDispose: dispose a base whose reshape view is yielded; ARC keeps the buffer alive, the view reads correctly, freed once at the view's dispose. - ManualDisposeThenReturnsSame_Graceful_NoCorruption: disposing what you return (a user bug) yields a disposed array gracefully, never a crash/double-free; the input stays untouched. - ExceptionAfterManualDispose_ScopeFinallySweep_FreedOnce_InputUntouched: throw after a manual dispose; the finally sweep must not free the temp again; input untouched. - MixedManualAndScopeReclaim_Balanced_NoDoubleFree_NoLeak: the §6 mid-method direct-Dispose pattern mixed with scope reclaim over 200 cycles; deficit >= 0 (no double-free = Releases > Acquisitions) AND <= 8 (no leak). - ManualDisposeThenScopeSweep_NoAliasing_HostileProbe: churn the pattern 100x, then allocate two arrays, fill with distinct values, and prove no cross-corruption (the strong aliasing detector). Result: flawless. All freed exactly once; a fixed exact-count assertion had to account for the scalar transient that 'input * 2' allocates (the sweep correctly reclaims it) -- switched that case to arange (one buffer, no hidden transient). 3 net8.0 runs + net10.0 green; Lifetime suite 63/63.
…ey3 branch state The Deploy Docs `api-coverage` gate regenerates test/inventory/generated and diffs it against the checked-in snapshot. The branch had drifted +56 test methods (13,703 -> 13,759) as the NDScope lifetime suites (NumSharp.Tests.Lifetime.* — BufferReleaseSweepTests / NDScopeManualDisposeTests / …) landed without regenerating the committed dashboard, so the gate went red (the actual docs DEPLOY is master-push-only and was skipped; only the PR verify job ran). Refreshed from the exact inventory the CI run generated (ubuntu, LF-normalised) so the diff is byte-clean and will also pass on the master-merge deploy. Data-only — this commit adds/removes no code and no test methods.
This change updates the changelog authoring playbook and release-note formatting to use ASCII hyphen separators, move breaking changes to the end of the changelog, and tighten the package/dashboard sweep rules and verification checklist. It also relocates the CHANGELOG_STYLE reference under the changelog skill folder and refreshes the 0.70.0 release notes to match the new house style.
…parity (fix flaky x64/arm64 CI) Convolve_SmallRealKernel_StaysManaged_ByteExact compares a MANAGED Vector<T> sliding-dot against LIVE numpy's own SIMD dot. Both pick their reduction kernel at runtime by CPU (x64 AVX2 vs AVX-512 lane width, arm64 NEON FMA-contraction), so the last bit agrees on some GitHub runners and differs by 1 ULP on others — observed byte[8] 0x55 vs 0x56 on BOTH ubuntu-x64 and windows-x64, though the test had passed on the same x64 runners in 3 prior runs (02e6929 / f0187d2). Strict byte-exactness is not a valid contract for this one managed-vs-live-numpy cell. New AssertSlidingWithinUlp asserts <= 1 ULP per element (still fails on any real divergence). It subsumes the former arm64-only SkipByteExactOnArm64 on this test — the divergence was never arm64-specific (memory confirms arm64 is also 1 ULP here). The backend-routed sliding tests stay STRICT byte-exact: scipy-openblas at threads=1 is deterministic; only the managed path is SIMD-dispatch-fragile. SkipByteExactOnArm64 remains for Lstsq/Polyfit (x64-deterministic LAPACK, arm64-only residual rounding). Verified on Windows: full SlidingDotLiveParityTests class 11/11 green (Convolve within 1 ULP, the 10 strict backend-routed tests unchanged); net8.0 + net10.0 compile.
…d convolve parity Follow-up to 6c96beb. The <=1-ULP tolerance fixed the x64 flake (interop ubuntu+windows green) but CI proved the arm64 assumption wrong: removing SkipByteExactOnArm64 exposed that the managed convolve diverges from live numpy by ~12942 ULP at a near-cancellation output (element 236) on the macOS runner — a genuine cross-arch numeric difference, NOT the 1-ULP rounding the memory claimed, so no small ULP bound can (or should) absorb it. Re-added SkipByteExactOnArm64 for this test (its original 02e6929 purpose was correct). x64 keeps the tolerance, widened to <=2 ULP (1 observed + 1 margin for AVX2-vs-AVX-512 runner variation) to prevent recurrence; any real divergence still fails by orders of magnitude. Verified on Windows: SlidingDotLiveParityTests 11/11 green; net8.0 + net10.0 compile.
…nvolve ULP-tolerance edit The tests-oracle inventory records each test method's source LINE, so the AssertSlidingWithinUlp / UlpDistance helpers + arm64-skip comments added to SlidingDotLiveParityTests.cs (6c96beb / 7e6c127) shifted the line numbers of the tests below them — drifting tests-oracle-report.{csv,json} even though NO test method was added or removed (summary.md / tests-oracle-manifest.json counts are unchanged). Refreshed from 7e6c127's CI-generated inventory so the Deploy Docs api-coverage diff is byte-clean. Data-only, and being inventory-only it touches no test lines, so it does not itself re-drift.
…t structs The [NDScoped] IL weaver previously rejected (NDW003) every return shape beyond a bare NDArray/NDArray[]/scalar, forcing tuple- and result-struct-returning boundary methods to be hand-scoped. Extend the weaver + NDScope so it weaves the two carrier shapes NumPy's surface uses, so those methods keep their 100% original body like every other [NDScoped] method. Weaver (tools/NumSharp.Weaver/ScopeWeaver.cs), two new RetKinds in Classify: - NDArrayTuple: a System.ValueTuple of 2..4 NDArray-likes (modf/polydiv/qr/eig/svd/lstsq/ average_returned/...). Emits `scope.Returns((a,b[,...]))` through new NDScope.Returns<T1,...> overloads — the overload decomposes the tuple in managed C#, so no field access is needed. ResolveRefs now keys the Returns family by GenericParameters.Count (was: param-is-ArrayType, which would have mis-bound the tuple overloads onto ReturnsOne). - Carrier: a result struct implementing the new internal interface NumSharp.INDArrayCarrier (explicit `void YieldTo(NDScope)` yielding each member). Emits a boxing-free `ldloca retVar; ldloc scope; constrained. <struct>; callvirt YieldTo`. Why the interface, not weaver-emitted field decomposition: the first design had the weaver ldfld each struct field directly. It throws FieldAccessException at RUNTIME — the CLR grants a nested type access to its enclosing type's privates but NOT the reverse, so a woven method in `np` cannot read np.UniqueCountsResult's private auto-property backing fields (or _grids/_outputs). The struct's own YieldTo can, so members are yielded from inside the struct. NDW003 now fires only for genuinely unsupported carriers (object/collection field, >4-arity or mixed tuple, struct without the interface), and a hand-scoped supported carrier with [NDScoped] is skipped rather than erroring. NDScope (src/NumSharp.Core/Backends/NDScope.cs): add Returns<T1,T2>/<T1,T2,T3>/<T1,T2,T3,T4> tuple overloads (also usable for hand-scoping) and the INDArrayCarrier interface. All 8 carrier result structs implement it (weave-ready): UniqueResult, UniqueAll/Counts/InverseResult, MeshgridResult, OGridResult, MGridResult, PolyfitResult. Annotated to prove both paths end-to-end with genuine transients and no backend dependency: - np.polydiv (tuple) — [NDScoped] (was unscoped). - np.unique_counts / unique_inverse / unique_all (result structs) — CONVERTED from hand-scoped to [NDScoped] (equivalent: the reshaped-inverse view survives its base's reclamation via ARC). Gates (all green, net8.0 + net10.0): new NDScopeWeaveCarrierTests (7 — members survive, dropped results re-parent into an enclosing scope, internal temps leave zero strands); the woven-assembly coverage gate; Sort/MultiOutput/Poly/Manip/Creation/GroupA/Conversion FuzzMatrix tiers bit-exact vs NumPy 2.4.2 (unique family + polydiv byte-neutral); Statistics/average unchanged (no overload- resolution regression at existing Returns call sites); ILVerify reports ZERO findings in the woven carrier methods (constrained.callvirt + tuple Returns are fully verifiable). Docs: DISPOSAL-GUIDELINES.md decision table + deferred section + API updated; NDScopedAttribute doc updated.
…8, mixed components) Extend the [NDScoped] weaver's tuple support beyond the arity-2..4 all-NDArray shape to ANY ValueTuple/Tuple — arity up to 8, and tuples whose components are not all NDArrays. NDScope: add `ITuple Returns(ITuple)` — iterates the tuple and yields every NDArray component through Returns<T>(T), skipping non-NDArray components (a scalar, a count, …); a null tuple is a no-op. Covers both value-type ValueTuple and reference-type System.Tuple (both implement ITuple). Weaver (ScopeWeaver.cs): new RetKind.Tuple alongside the existing NDArrayTuple. - NDArrayTuple (ValueTuple of 2..4 NDArrays) keeps the strongly-typed Returns<T1,…> overload — no box, the optimal path for every tuple np actually returns today (polydiv/qr/eig/svd/lstsq/modf/…). - Tuple (any other ValueTuple/Tuple: arity 5..8, a non-NDArray component, or a reference Tuple) emits `box <ValueTuple>` (skipped for a reference Tuple) + `callvirt Returns(ITuple)`. Detected by name (System.ValueTuple`N / System.Tuple`N), so no external assembly resolver is needed. ResolveRefs finds Returns(ITuple) by its zero-generic ITuple parameter. No in-tree [NDScoped] method returns a 5..8 or mixed tuple, so — exactly like the weaver's existing out-NDArray[] egress branch — the ITuple path's runtime semantics are pinned on the underlying NDScope method rather than a woven consumer: NDScopeWeaveCarrierTests calls `scope.Returns((ITuple)t)` in the IDENTICAL box+callvirt shape the weaver injects (a C# `(ITuple)valueTuple` cast IS the box), across a 5-element mixed tuple, a reference Tuple, a null tuple, and re-parenting into an enclosing scope. Verified: polydiv still takes the typed path and unique_counts still uses YieldTo (IL-inspected — the ITuple path is purely additive, existing woven methods unchanged); the whole weave suite (18) + the unique/polynomial regression are green on net8.0. Docs: DISPOSAL-GUIDELINES.md decision table + NDScopedAttribute doc updated.
…close the aliased-base use-after-free (SlicingWithNegativeIndex1, macOS CI) Symptom: intermittent red on test (macos-latest) — SlicingWithNegativeIndex1 read 0L where 8L was stored. The test builds `new UnmanagedStorage(np.arange(10).GetData(), shape)`: the arange NDArray temp dies immediately while its buffer is aliased by a bare UnmanagedStorage, which holds NO counted ARC reference (only NDArray ctors AddRef). When GC ran between construction and the reads, ~NDArray -> InternalArray.Release() -> refcount 1 -> 0 -> claim-free CAS -> buffer returned to SizeBucketedBufferPool UNDER the live alias; the next same-size allocation overwrote it. Deterministic repro on Windows (forced GC + same-size np.full(7777) thief): 2000/2000 iterations freed, 1999/2000 read 7777 instead of 8. Green-at-7e6c1279 / red-at-edc946c4 (a data-only commit) proved the GC-timing race. Root cause: the finalizer path claimed the eager free. An NDArray being UNREACHABLE proves nothing about OTHER reachable aliases of the same block — bare UnmanagedStorage / IArraySlice handles from GetData() are invisible to the refcount. Freeing at refcount-0 is only sound where a caller asserts lifetime, i.e. the deterministic Dispose()/NDScope path. Fix: split the drop in two. - Disposer.Abandon() (new, plumbed as IArraySlice.Abandon -> ArraySlice<T> -> UnmanagedMemoryBlock<T>): decrement WITHOUT the 0 -> -1 claim-free transition; count rests at live-zero, so a later TryAddRef (a new NDArray wrapping a still-live alias) legitimately revives 0 -> 1, and a stray call on a released block self-heals exactly like Release. - ~NDArray now calls Abandon() instead of Release(). Dispose() (and with it NDScope eager reclamation and ndarray.resize) is untouched and keeps the eager free-and-pool at refcount 0. Why this is safe and leak-free: the block's inner Disposer has its own finalizer that frees regardless of refcount, gated only on reachability — exactly the right authority for the non-deterministic path. An abandoned buffer whose aliases all died is freed (and pool-returned) in the following GC cycle; an abandoned buffer with a live alias stays valid for as long as the alias is reachable. Verified: fixed build 0/2000 UAF hits; pool-reuse probe 49/49 buffers still recycle through SizeBucketedBufferPool after finalization; buffer readable under the alias, refcount drained to 0. Test updates (ArcLifecycleTests): three tests pinned the OLD contract — each held the slice (a reachable alias!) and asserted IsReleased==true after finalization, i.e. asserted freed-memory-reachable-through-a-live- alias, the exact hazard fixed here. - Finalizer_ReleasesUnmanaged_WhenDisposeMissed -> Finalizer_AbandonsRef_ButNeverFreesUnderALiveAlias: asserts the buffer survives finalization while the alias is held (and stays readable), the counted ref drains to 0, and — via a WeakReference to the inner Disposer — that the block IS reclaimed once the last alias dies. Every touch of the slice box is confined to NoInlining helper frames: a single-shot test method JITs at tier-0, whose untracked eval-stack temps root the box until method end (measured: the drop-phase probe never dies inside the test's own frame, dies immediately when confined to a helper frame). - UndisposedNDArray_BecomesCollectable_ViaFinalizer: same contract flip (buffer NOT freed under the held alias, refcount 0). - ReshapeNonContiguous_AllocatesNewOwningBuffer: the orphan intermediate now abandons at finalization — assert refcount drains to 0 and the buffer stays alive under the held c_slice. - NEW AliasedBaseBuffer_SurvivesFinalizerOfItsLastNDArray: the CI failure made deterministic — 200 iterations of the exact SlicingWithNegativeIndex1 idiom with forced GC + a same-size allocation that would steal a wrongly pooled buffer. Verification: full NumSharp.Tests suite (CI filter) 13858/13858 green on net10.0 AND net8.0 Release; ArcLifecycleTests 50/50; FuzzMatrix oracle gate 85 passed / 3 host-pin skips on both TFMs; SlicingWithNegativeIndex1 + SlicingWithNegativeIndex green. The 16 sibling `new UnmanagedStorage(nd.GetData(), ...)` sites across the test suite are all covered by the same class-level fix.
…sts abandon-contract update Regenerated by CI (Deploy Docs run 32754479619, ubuntu) and committed from the downloaded numsharp-tests-oracle-inventory artifact — never regenerated locally (Windows-vs-ubuntu output drift). Drift source: commit 09b47a4 renamed Finalizer_ReleasesUnmanaged_WhenDisposeMissed to Finalizer_AbandonsRef_ButNeverFreesUnderALiveAlias, added AliasedBaseBuffer_SurvivesFinalizerOfItsLastNDArray (+1 test method), and shifted ArcLifecycleTests.cs line numbers.
…counted-ref protection Support [NDScoped] on a boundary method that RETURNS a lower-layer buffer (IArraySlice / UnmanagedStorage) not wrapped in an NDArray — the third "other return kind" beyond tuples/carriers. The hazard this addresses: the scope's reclamation unit is the NDArray, which owns the single counted ARC reference on its buffer; NDScope reclaims tracked NDArrays via the DETERMINISTIC Release path (eager free at refcount 0, "asserting no alias outlives" — see the ~NDArray ABANDON contract, commit 09b47a4). A bare IArraySlice/UnmanagedStorage is an UNCOUNTED alias, so a scoped method returning one would have its buffer freed out from under the caller the instant the scope Releases the intermediate NDArray that shares it (the SlicingWithNegativeIndex1 UAF class). NDScope: add Returns(IArraySlice) and Returns(UnmanagedStorage) — each takes a counted reference (TryAddRef, a stable ARC API) on the returned buffer so the scope's Release can no longer reach 0; the buffer survives, and the reference is deliberately abandoned (never Released), so the block's finalizer reclaims it on unreachability — the same non-deterministic backstop a bare buffer already relies on. Weaver: new RetKind.Storage — a return typed exactly NumSharp.Backends.Unmanaged.IArraySlice or NumSharp.Backends.UnmanagedStorage emits `ldloc scope; ldloc retVar; callvirt Returns(slice/storage); stloc retVar`. ResolveRefs finds the two overloads by their zero-generic parameter type. Deliberately DEFERRED (documented): auto-TRACKING intermediate bare buffers for eager reclamation — hooking bare-buffer construction would double-count against the NDArray's reference, lands in the parallel ARC rework, and has no consumer (compositions own NDArrays, whose transients are already reclaimed transitively). Only the RETURN is protected. No in-tree method returns a bare buffer, so (like the ITuple/out-param branches) the semantics are pinned on the NDScope methods the weaver targets: NDScopeWeaveCarrierTests proves a yielded slice/ storage survives the scope's Release of its NDArray (readable afterward), with a control showing an un-yielded bare alias IS freed (the hazard the counted ref fixes). Full Lifetime suite (75) green. Docs: DISPOSAL-GUIDELINES.md decision table + a "scope of support" note; NDScopedAttribute doc.
Add docs/website-src/docs/ndscoped.md — a website page documenting the [NDScoped] build-time scope
weaving, framed (like the OpenBLAS interop page) as an optional installable package, NumSharp.Weaver,
that you reference to enable the transform on your own composition methods (NumSharp.Core already uses
it internally on ~78 np.* methods).
Follows the house optional-package layout: one-line intro + "why it exists" (the finalizer-lag problem
vs NumPy's deterministic refcounting), a dotnet-add-package quick start, the before/after the weaver
injects (open scope / try-finally / Returns at each ret), the full return-shape decision table
(NDArray, NDArray[], typed ValueTuple, ITuple for any-arity/mixed tuples, INDArrayCarrier result
structs, bare IArraySlice/UnmanagedStorage, out params, the NDW002-004 rejections), the NDScope
runtime API + hand-scoping, the MSBuild integration (per-TFM, re-sign, PDB, idempotent/incremental),
when-to-use guidance, the SkipNDScopeWeave/ILVerify escape hatches, and an NDW error-code
troubleshooting table. Cross-links buffering.md (ownership) and il-generation.md (IL); gate links to
the real NDScope{,Weave,WeaveCarrier,WeaveNesting}Tests + StrongNameTests. Added to docs/toc.yml after
"Buffering & Memory".
… double-dispose
Adds NDScopeDoubleDisposeTests, pinning that the woven [NDScoped] np.* methods which touch
UnmanagedStorage internally — those that explicitly Dispose() a transient (np.diff, np.ediff1d, and
nonzero's documented `materialized?.Dispose()` ARC-release) or alias a buffer into a returned view
(np.nonzero, np.einsum diagonal) — never release a buffer twice.
The concern: the weaver injects a scope that reclaims every tracked NDArray at method exit, so a
method that ALSO disposes one of those transients itself would, if unguarded, release the same buffer
reference twice — a double-free / premature-free under a live view.
It cannot happen, for two audited reasons this gate exercises at runtime:
1. NDArray.Dispose is idempotent (an Interlocked _disposed guard), so an explicit dispose followed by
the scope's dispose releases the buffer's one reference exactly once.
2. No [NDScoped] method releases a buffer DIRECTLY — none call InternalArray.Release()/DangerousFree()
(verified by source audit); every release goes through the idempotent NDArray.Dispose. The .Storage
accesses in [NDScoped] methods are read-pointer / reshape-metadata / alias-view / fill — never a
release.
Tests:
- ExplicitTransientDispose_PlusScopeDispose_ReleasesBufferExactlyOnce — the mechanism made observable:
a scope tracks a base, the base is explicitly disposed (the np.diff/nonzero pattern), and a view
sharing its buffer is yielded; the buffer must survive scope exit (proving no double-release under
the live view) and free exactly once when the view — the last owner — drops.
- YieldedViewOntoDisposedBase_SurvivesBothDisposes — a yielded column view outlives both the explicit
dispose and the scope (the nonzero shape).
- WovenStorageMethods_UnderGCPressure_NoDoubleFree — 12k iterations of np.diff/ediff1d/nonzero(non-
contiguous)/where/einsum under forced GC, the exact conditions Default.NonZero documents as
reproducing a double-free AccessViolation in Release; a double-release would crash or read garbage.
Green on net8.0 + net10.0.
…e explicit-layout union (-896 B per UnmanagedStorage) Every UnmanagedStorage carried 15 strongly-typed 64 B ArraySlice<T> fields of which exactly ONE was ever populated (the lane matching the array's dtype) -- 960 B embedded per instance, ~896 B of it permanently default. They are now a single [StructLayout(LayoutKind.Explicit)] nested struct TypedSlices whose 15 lanes all overlap at [FieldOffset(0)], accessed as _slices.Int32 / _slices.Double etc. Pure mechanical rename on every read/write site; the getter read path is byte-for-byte identical (_slices.X[_shape.GetOffset(indices)]). Why the overlap is CLR-legal: ArraySlice<T>'s layout is identical for every T (T occurs only behind pointers: T*/void*/long/bool), and its single managed reference -- the non-generic Disposer inside UnmanagedMemoryBlock<T> -- sits at the same offset with the same type in all 15 lanes, so the GC ref map is well-formed. A malformed map would be a TypeLoadException at first touch (the loader's own check), never silent corruption. Aliasing is never crossed: only the _typecode-matched lane is written or read over a storage's lifetime. The ARC refcount is untouched -- TryAddRef/Release flow entirely through the boxed InternalArray, which is kept verbatim. Changes: - UnmanagedStorage.cs: 15 field decls -> TypedSlices union + _slices; renames in the 12 scalar ctors, 12 array ctors, and both SetInternalArray 15-case switches; codegen template comments updated (_array#1 -> _slices.#1). - UnmanagedStorage.Getters.cs: the 30 direct typed getters renamed. - UnmanagedStorage.Cloning.cs: Alias(Shape)'s typed-lane mirror is now ONE 64 B struct copy (r._slices = _slices) replacing the per-dtype IL-emitted field copier (ConcurrentDictionary lookup + delegate invoke + ldfld/stfld). - DirectILKernelGenerator.StorageAlias.cs: DELETED -- its GetField("_array"+typecode) reflection has no target anymore and its single caller is gone; ~140 lines + a DynamicMethod emit path retired. - GeneratedDelegates.cs: the copier-cache census entries removed (count / sum term / per-cache clear / ClearAll call) -- a 5th touched file the design doc's edit inventory missed. - TypedSlicesUnionTests.cs (NEW, 6 tests): union type-loads + is exactly one slice wide (64 B); lane==Address agreement across all 15 dtypes on owned and sliced storage; Alias struct-copy carries the live lane; view write-through; clone independence incl. the 16-byte lanes; dispose-base-view-survives. - docs/UNMANAGED_STORAGE_UNION_DESIGN.md: the design + safety proof, now stamped IMPLEMENTED with measured outcomes. - .claude/CLAUDE.md: dropped the deleted .StorageAlias.cs from the Direct partials table. Measured (net10.0 x64 Release, this host): - UnmanagedStorage MethodTable.BaseSize: 1088 -> 192 B - per-array managed footprint (int32x10): 1424 -> 528 B (-896 B, exactly the 14 dead lanes) - scalar GetInt32 x100M: 130 -> 113 ms (no regression; smaller object, better locality); Alias(Shape) 32.8 ns/alias - type loads on net8.0 AND net10.0 with no TypeLoadException Verification (design doc section 5, all green): - full suite net10.0 CI filter: 14,043 passed / 0 failed - FuzzMatrix differential gate (bit-exact vs NumPy 2.4.2, all 15 dtypes x layouts): 88/88, zero diffs - net8.0 union/ARC/lifetime/weave leg: 89/0 - GC stress (the explicit-layout-specific risk): 1.2M arrays+ARC views churned under forced compacting gen2 waves on Workstation+Concurrent AND Server+Concurrent, plus a GCStress=0xC pass -- 2.43M typed-lane verifications total, 0 corruption, 0 crashes - interop ShutdownLeakTests green; weaver weaves the new struct cleanly (NDScope gates in the full run)
The Deploy Docs workflow's 'Verify checked-in Tests & Oracle dashboard data' gate diffs the checked-in test/inventory/generated/ against a fresh generate_test_inventory.py run and fails on any staleness. The typed-slice union change added 6 tests (Backends/Unmanaged/TypedSlicesUnionTests) without refreshing the dashboard; regenerated: 13,782 test methods, 116,971 corpus rows, 366 op keys.
…026-08-24_020e6543 [skip ci] Publish the official benchmark run produced by benchmark/run_benchmark.py: the op/dtype/N matrix (BenchmarkDotNet vs a warm NumPy 2.4.2) plus the five appended subsystems (NDIter, Layout, Operand, Cast, Fusion) and the Managed / OpenBLAS backend profiles merged into the canonical backend-aware matrix. Provenance (from snapshot MANIFEST): - Run timestamp 20260824-132525; harness wall time ~23301s (~6.5h). - Git HEAD at run: 020e654 (feat(lifetime): weave carrier returns). - Host: 13th Gen Intel Core i9-13900K, Windows 11 (10.0.26200), .NET SDK 10.0.101 (net10.0 Release), Python 3.12.12, NumPy 2.4.2. - Sizes 1K / 100K / 10M, same seeds both sides; join on (op, dtype, N, scenario); effective timing = fastest valid backend profile. Convention NPY/NS (>1 = NumSharp faster). Refreshed: - benchmark/benchmark-report.md (backend-aware matrix + subsystem sections). - Subsystem sheets: cast/fusion/layout/nditer/operand *_results.{md,tsv} + nditer cards. - OpenBLAS profiles: openblas/openblas_results.{managed,openblas}.json + .md/.tsv. - Website data: docs/website-src/docs/data/benchmark-report{,.managed,.openblas}.json. New tracked history snapshot benchmark/history/2026-08-24_020e6543/ (report .md/.json/.csv, managed+openblas profile json, numpy-results.json, all subsystem sheets, cards, MANIFEST) and benchmark/history/latest repointed to it. Headline (MANIFEST): NDIter 1.30x geomean (77% of NumPy's time, 72 win / 53 lose over 125 cells); Cast 1054/1568 comparable cells >=1.0x. Note: SciSharp.NumSharp.sln (unrelated NumSharp.Weaver project addition) deliberately left uncommitted — not part of this benchmark run.
…n doc Closes the design's acceptance checklist: Build and Release green on the union merge commit 793949f (run 32761521466) and on tip 8a039e8 (run 32762118299); ILVerify woven-vs-unwoven delta 0 and pre-vs-post-union delta 0 (modulo compiler closure renumbering from the StorageAlias.cs deletion); post-merge compatibility with the ~NDArray abandon fix verified (gates 110/0 + 600K-array GC-stress leg on the merged tree).
…r NuGet package NumSharp.Weaver now packs as a tools-only NuGet package that WEAVES the project it is installed on instead of becoming a dependency of it: install it, mark methods [NDScoped], and the packaged NDScopeWeave target rewrites the intermediate assembly after each per-TFM compile exactly like NumSharp.Core's own self-weave. Not-a-dependency, by construction: - IncludeBuildOutput=false + SuppressDependenciesWhenPacking: the nupkg carries NO lib/ and NO dependency entries (Mono.Cecil rides inside tools/net8.0/any/, never expressed to the consumer's graph). - DevelopmentDependency=true: `dotnet add package NumSharp.Weaver` writes PrivateAssets="all" on the consumer's PackageReference by itself, so the weaver never flows into the consumer's own package as a dependency. - build/ only (deliberately NOT buildTransitive/): weaving applies exactly where the package is installed, never to projects that merely reference a woven library. - The tool payload is the PUBLISH output (weaver + Cecil + runtimeconfig with RollForward=Major so an SDK-10-only box runs the net8.0 tool; UseAppHost=false). TRAP: PrivateAssets="all" on the Mono.Cecil PackageReference EXCLUDED it from the publish output (the SDK treats such packages as non-published dev dependencies) -- keeping Cecil out of the consumer's graph is SuppressDependenciesWhenPacking's job, so the reference deliberately carries no PrivateAssets. Cross-assembly weaving (a consumer's NDScope lives in the REFERENCED NumSharp): - WeaverAssemblyResolver: exact-path map built from the compiler's own reference list (@(ReferencePathWithRefAssemblies) written as a response file, passed via --refs) plus directory-probing fallback; reference assemblies are fine because the weaver imports member SIGNATURES, never bodies. - ResolveRefs falls back from module.GetType to the assembly references (NumSharp-named refs first, type forwarders honoured) and IMPORTS every member into the woven module (an identity pass-through for the in-module self-weave). - Classification guards relaxed from name-prefix to resolver-backed resolution: consumer-defined INDArrayCarrier result structs, NDArray subclasses and consumer/BCL enums classify exactly like in-module ones (System./Microsoft. fast-skip avoids opening BCL ref packs for answers the classifier already knows). - New error NDW001: [NDScoped] methods present but NDScope unresolvable -- a loud configuration error instead of silently shipping unwoven methods. - An assembly with no [NDScoped] methods (or whose targets are all hand-scoped already) is left byte-for-byte UNTOUCHED: no rewrite, so its compile-time signature and determinism id survive. Program: --refs <rsp>; an empty --snk is tolerated (unsigned consumers); the summary reports 'assembly unchanged' when nothing wove. Consumer-facing surface: NDScopedAttribute and INDArrayCarrier go PUBLIC in NumSharp.Core -- the ndscoped docs already promise both to consumers, and the woven cross-assembly YieldTo/Returns calls must pass the CLR's runtime accessibility check. Without the package the attribute stays inert metadata: adding/removing the weaver never changes results, only WHEN transient buffers are reclaimed. Packaged MSBuild target (build/NumSharp.Weaver.targets): AfterTargets=CoreCompile with a per-TFM incremental marker, the refs rsp via WriteLinesToFile (immune to command-line length limits), re-sign only a genuinely full-signed project (SignAssembly and not DelaySign/PublicSign and KeyOriginatorFile exists), SkipNDScopeWeave and NDScopeWeaveILVerify escape hatches identical to the self-weave, and a NumSharpWeaverToolDll override for pointing at a locally built tool. TRAP: an XML comment cannot contain '--' (MSB4024 at import time) -- option names in targets comments must be spelled in prose. CI: the release workflow builds + packs the weaver alongside the four library packages (verify-signing throwaway pack included) and lists it in the release notes; verify_strong_name.cs learns the tools-only shape -- it asserts the build/ + tools/ payload (tool, Cecil, targets, no lib/) instead of failing on the missing lib/, with the key check n/a since the tool never loads into a user app and Mono.Cecil carries Cecil's key, not NumSharp's. Gate: tools/verify_weaver_package.sh -- a 9-step scripted nupkg-flow run against a real multi-TFM (net8.0;net10.0) consumer restored from a local feed: CI-shape pack (build -t:Rebuild then pack --no-build, proving Publish-under-NoBuild stages tools/), package-shape assertions, the dotnet-add-package PrivateAssets UX, per-TFM weave of all three consumer return shapes (bare NDArray, ValueTuple-of-NDArrays, consumer-defined INDArrayCarrier struct) verified BOTH by reflection over the shipped IL and by values surviving their scopes, marker incrementality (with one settling build -- the P2P chain may legitimately recompile once when the surrounding global-property context changes), SkipNDScopeWeave (whose red consumer run also proves the step-6 gate non-vacuous), and the consumer's own nuspec depending on NumSharp but NOT on NumSharp.Weaver. Dev-loop trap encoded: the constant package version means NuGet's GLOBAL cache shadows freshly packed copies, so the script evicts the numsharp.weaver cache entry before restore. All 9 script steps green; NDScope*/StrongName gates 68/68 green on net8.0 (self-weave regression-free; Core currently weaves 90 methods).
…s; codebase parity
Prove-and-break pass over the NumSharp.Weaver package. 13 adversarial consumer shapes
were run against the packed nupkg; two real defects surfaced and are fixed here, the
battery's five highest-value probes are folded into the committed gate (9 -> 14 steps),
and the package is now enumerated everywhere the other four packages are.
Defects found and fixed:
- CROSS-PLATFORM PATHS: build/NumSharp.Weaver.targets spelled the tool path with
backslashes; the path reaches Exec's SHELL (cmd on Windows, sh on Linux/macOS), and a
raw backslash path breaks the Unix side. Now forward slashes +
$([MSBuild]::NormalizePath()), the NumSharp.Interop.OpenBLAS buildTransitive
convention; the csproj's pack globs (build/**, $(PublishDir)**/*) likewise.
- CI NoWarn OVERRIDE: the workflow passes a GLOBAL -p:NoWarn, and a global property
overrides the csproj's own NoWarn accumulation -- the weaver's by-design NU5100
(assemblies under tools/) and NU5128 (no lib per TFM) suppressions were lost in CI
packs. Both codes now ride env.DOTNET_NOWARN, like NU5048 already did.
Adversarial battery (all green after the fixes; scratch probes, not committed):
PackageReference-NumSharp consumer (lib/ resolution), strong-named consumer (re-signed
with its own key, correct token cc7b13ffcd2ddd51, StrongNameSigned CorFlag set, runs),
public-signed consumer (woven, NO re-sign attempt), attribute-free consumer (untouched
no-op), weaver-without-NumSharp consumer (clean no-op), NDW002+NDW003+NDW004 all
surfacing as MSBuild errors and failing the build, transitive isolation (AppB
referencing woven LibA is NOT woven; its [NDScoped] stays inert), embedded-PDB
consumer, path-with-spaces consumer, -p:Version/PackageVersion lockstep
(9.9.9-test nupkg), Debug-configuration weave, dotnet-publish output woven, and the
packed consumer's lib/ dll byte-identical to the woven bin dll.
Gate grows 9 -> 14 steps (tools/verify_weaver_package.sh), making the battery's core
permanent: step 1 additionally packs NumSharp.Core into the local feed (versions parsed
from the nupkg names, digit-glob so a future 1.x still matches); 10 = the REAL product
path, a consumer taking NumSharp itself as a PackageReference so the weaver resolves
NDScope out of lib/net8.0/NumSharp.dll instead of the P2P ref assembly; 11 = the three
rejection shapes must FAIL the build with their codes; 12 = attribute-free no-op
('nothing to do', assembly unwritten); 13 = transitive isolation (weave stays on the
installed project -- PrivateAssets/build-not-buildTransitive proven, not assumed);
14 = strong-named consumer re-signed with its own key, verified down to the
CorFlags.StrongNameSigned bit read straight out of the PE by python (a delay-signed
image carries the key but leaves the bit clear), and still green at runtime.
Codebase parity with the other packages: ARCHITECTURE.md gains the NumSharp.Weaver row
in the project/package boundary table and a build-time node in the flow diagram;
DISPOSAL-GUIDELINES' weaver section drops the stale '(internal)' on the attribute and
documents the consumer-facing package + its gate. Root README and the website index
list no optional packages (only 'dotnet add package NumSharp'), so no edit there; the
website toc already carries docs/ndscoped.md; release workflow, release-notes body,
badge table and the strong-name verifier were wired in the previous commit.
OracleSurfaceCoverageTests stays green with the two newly public types.
… scars
Mined github.com/Nucs/JsonSettings src/JsonSettings.Autosave/build (the battle-tested
out-of-process AspectInjector weave + re-sign targets) for the production failure modes
it encodes, mapped each against NumSharp.Weaver, and adopted the three that apply.
Adopted (all three verified end-to-end):
- HOST SELECTION, no bare-PATH assumption: the Exec ladder now prefers
$(DOTNET_HOST_PATH) (set by the dotnet CLI / .NET-Core MSBuild for exactly this
hand-off; the SDK-documented discovery for a build step's own host) and falls back to
PATH 'dotnet' only where it is unset (VS devenv / Full-framework MSBuild, whose SDK
installer registers dotnet machine-wide). Overridable via $(NumSharpWeaverDotnetHost).
Applied to BOTH the packaged build/NumSharp.Weaver.targets and NumSharp.Core's own
inline NDScopeWeave target. Proven live in both regimes: a dotnet-CLI build resolves
to the full 'C:\...\dotnet.exe' path, and a REAL VS 2022 MSBuild.exe consumer build
with DOTNET_HOST_PATH scrubbed falls back to 'dotnet', weaves, and the app verifies
its own NDScope local at runtime.
- ANCHOR SET: BeforeTargets grows _TimeStampAfterCompile;AfterCompile (AspectInjector's
own anchors) so the SDK's post-compile timestamping stays consistent with the
rewritten assembly and AfterCompile-hooked IL rewriters (MAUI XamlC and friends) see
the WOVEN assembly instead of racing it. MSBuild ignores anchors naming targets a
project never defines, so this is inert elsewhere. Mirrored into Core's inline target.
- CONTENT PRE-SCAN (their '__a$_instance' marker-scan technique): using [NDScoped]
compiles a TypeRef whose name lands as an ASCII run in the metadata #Strings heap,
and ASCII runs survive lossy UTF-8 decoding of arbitrary binary intact — so
File.ReadAllText(...).Contains('NDScopedAttribute') is a reliable was-it-used probe
on fresh CoreCompile output. Attribute-free consumers (and rebuilds where a foreign
rewriter bumped the assembly past the marker) now skip the tool spawn entirely; a
false positive (the name in a user string literal) merely spawns the tool, which
no-ops. No false negative exists on compiler output. The gate's step 12 accepts the
new skip message alongside the tool's own nothing-to-do line.
Scars audited and NOT applicable — the design is already immune, now deliberately so:
- their PART 1 core bug (in-process task leaks Cecil ReadWrite handles into reused
MSBuild nodes; CreateAppHost then dies on the locked dll): we are out-of-process by
construction with InMemory reads, and the OS closes everything at tool exit — their
fix converges on our architecture (their -nodeReuse:false child-MSBuild machinery is
unnecessary for a plain console child);
- the CULTURE/CALENDAR scar (%(ModifiedTime) formats with the current culture's
CALENDAR while Touch parses Time invariant → Persian/Umm-al-Qura locales die with
'Not a valid Win32 FileTime'; their ToString('o') round-trip fix): our Touch stamps
bare AlwaysCreate (no Time string round-trip at all), the targets format no dates or
numbers, and the tool runs InvariantGlobalization=true with ordinal comparisons;
- ADVICE STACKING on re-weave (their measured one-save-becomes-N bug behind both the
content check and the XamlC restamp target): our transform is per-method idempotent
(an already-scoped body is skipped) and a nothing-woven pass does not rewrite the
file — live-proven this session: deleting the marker and rebuilding reports
'woven 0, already-scoped 90, assembly unchanged' where AspectInjector would have
double-woven;
- RE-SIGN-AFTER-STAMP re-weave loop (sn -R bumps the assembly past the marker): our
re-sign happens inside the tool's single write, BEFORE the marker touch — the
ordering hazard cannot exist;
- sn.exe discovery / Windows-only re-sign / warn-when-missing: Cecil signs the blob
in-process, cross-platform, no external tool;
- command-line overflow of reference lists: already a response file (--refs);
- semicolon-in-path property tearing (their NJS1007): we pass quoted ARGUMENTS to a
console tool, never -property: values through a child MSBuild command line.
Gate: all 14 verify_weaver_package.sh steps green against the hardened targets;
NDScope*/StrongName suites 68/68; Core self-weave verified under BOTH dotnet-CLI
MSBuild and VS 2022 Full-framework MSBuild.exe.
…allel weaves
tools/stress_weaver.sh complements the 14-step correctness gate with scale and
concurrency: a python generator emits 1,015 [NDScoped] members (20 classes x 50 + 15
specials) cycling every weave shape — bare NDArray, ValueTuple, NDArray[],
INDArrayCarrier struct, [NDScoped] PROPERTY (the CollectTargets property branch at
scale), bool+out NDArray, void+out NDArray, return-inside-try/finally, scalar
(scope-only), hand-scoped+attributed (idempotence, 91 instances), multi-return switch,
40-statement bodies and 4-deep nested try/finally — consumed three ways:
A SOURCE MODE ('referencing as a project'): ProjectReference to NumSharp.Core plus a
ReferenceOutputAssembly=false ProjectReference to the weaver csproj (builds the
tool) plus a direct <Import> of build/NumSharp.Weaver.targets with
$(NumSharpWeaverToolDll) pointed at the tool's bin output — the in-repo recipe,
multi-TFM net8.0;net10.0.
B NUPKG MODE: NumSharp AND NumSharp.Weaver restored from a local feed as
PackageReferences, multi-TFM, plus a weave-cost probe (consumer-only Rebuild with
vs without SkipNDScopeWeave).
P PARALLEL MODE: 8 independent libs x 120 members built -m:8 so up to 8 weaver
processes run CONCURRENTLY against the same cached tool, plus a weaver-free app
referencing all 8 (transitive isolation at scale) that validates every lib.
Each mode's runner reflects EVERY [NDScoped] method and property in the shipped IL
(the NDScopeWeaveTests NDScope-local invariant, at scale) and then executes every
generated member 25 times with forced full GCs between sweeps — a mis-scoped result
whose buffer was wrongly reclaimed surfaces as pool-reuse value corruption, which the
per-shape expected values catch.
First-run results, all green:
- A: woven 924 / already-scoped 91 on BOTH TFMs; runners 1015/1015 woven, 25 sweeps,
0 failures each; incremental build does not re-weave; deleting the markers and
rebuilding reports 'woven 0, already-scoped 1015, assembly unchanged' on both TFMs —
idempotence at scale with no file rewrite.
- B: identical coverage/value results from the packed nupkg; weave cost ~1s TOTAL for
2 TFMs x 1015 members on a consumer-only rebuild (spawn + Cecil + weave + re-write).
- P: 8 concurrent weaves of 'woven 109, already-scoped 11' with no contention, the app
itself untouched, aggregate 960/960 validated under the same GC hammering.
Net: ~125K woven-member executions under forced GC across the three modes, 0 value
failures, 0 coverage misses — both consumption modes hold at scale and under -m:8
process concurrency.
…1ms window)
The op-matrix Python timer collapsed sub-microsecond O(1)/scalar/view ops (broadcast,
reshape, transpose, dtype queries, format_float, splits, index generators) to a noisy
~0.0000 ms mean with an undefined NPY/NS ratio. Rework the measurement so every op gets a
credible average, and fix the two mis-scaled/absent cases the O(1) sweep surfaced.
numpy_benchmark.py benchmark():
- Rewritten to a fixed per-test rule: ALWAYS >=50 timed samples (never fewer, never 1),
each batching 'inner' calls (a count computed per test, not fixed) so the TOTAL
measurement window spans >=1 ms. A sub-us op therefore runs tens of thousands of times
batched; a >1 ms/call op does inner=1, i.e. exactly 50 real single calls (never collapses
toward 1). Per-call cost is first estimated with a growing PILOT batch (one calibration
call is too noisy for a sub-us op); a 1.5x safety margin keeps the actual window >=1 ms.
Reported iterations is the total calls executed. Real-work O(N) ops are unchanged
(inner=1, 50 samples).
SliceBenchmarks.cs + numpy twin:
- The three fixed-work slice ops (a[..].copy(), np.copy(a[..]), a[..]*2) sliced a constant
~900 elements at every N, so their cross-N comparison was meaningless. They now slice the
middle 80% [N/10:9N/10] and scale with N; renamed accordingly, join-key verified identical.
The pure view-creation slices stay O(1) views.
N=1 scalar-dispatch tier:
- Wire the N=1 ('scalar x scalar') pure dispatch-overhead tier, scoped to the api suite via
a new --with-scalar flag (run_benchmark passes it only for that suite). ApiSurface DType/
Text classes gain [Params(Scalar, Small)]; nested_iters' reshape is guarded for N=1; the
real-work poly/IO cases stay at the standard size.
Report precision:
- merge-results rounds numsharp_ms/numpy_ms to 6 decimals (was 4) so a sub-100 ns mean and
its ratio survive; merge-backend-profiles displays sub-us cells with adaptive precision.
Validated end-to-end: np.result_type went from '0.0000 ms / --' to '0.000005 ms / 39.2x'.
Register the NumSharp.Weaver project in SciSharp.NumSharp.sln, including its build configuration mappings and solution folder placement. This makes the weaver buildable and discoverable as part of the main solution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NumSharp 0.70.0
It took a while but we are at 85% NumPy API coverage.
This is a huge milestone for the .NET ecosystem as NumSharp grows to matureness.
This branch also delivers integration with NumPy's OpenBLAS backend and full integration with Python giving a new angle of use cases for NumSharp to a point NumSharp is an unmanaged memory and math interop with the python ecosystem. I believe in integration rather than competition thus the large scale of support from PyTorch to Pillow.
OpenBLAS is a rapidly developed ecosystem NumSharp will eventually replace with a simpler version but that requires porting of 100k-300k lines of code to achieve complete mathematical parity. OpenBLAS roughly powers 30% of NumPy's.
📦 New NuGet Packages
Two optional companion packages ship for the first time, co-versioned with NumSharp 0.70.0 (both depend on NumSharp.Core).
All nugets are now published as signed nuget packages.
dot,matmul,inner,vdot,vecdot,matvec,vecmat,tensordot,multi_dot,matrix_power.solve,inv,det,slogdet,tensorsolve,tensorinv.cholesky,qr,svd,svdvals.eig,eigvals,eigh,eigvalsh.lstsq,pinv,matrix_rank,cond,norm.correlate,convolve.arr.ToNumpy()/arr.ToPython()out;pyObj.AsNDArray()/pyObj.FromArrayLike()in.RegisterCodec()once, then pythonnet's ownobj.ToPython()/pyObj.As<NDArray>()round-trip transparently.Auto(view when possible, else copy),View(share or decline),Copy(always independent).FromArrayLikeimports any PEP 3118 exporter, strided/offset/reversed included; read-only stays non-writeable:📊 Dashboards & Docs
Three living dashboards ship on the documentation site, each generated from the same CI artifacts the release gates run on.
✨ New APIs & Modules
np.random.default_rng- the full modern PCG64Generator, byte-identical streams to NumPy 2.4.2 -e868d8ae(+754b7476,febfbbdd,f491c499).default_rng- entry point (seed /SeedSequence/BitGenerator/PCG64overloads).random,integers,standard_normal,normal,exponential,uniform,standard_gamma,gamma,choice,shuffle,permutation,permuted- the Generator draw surface.random_integers,bytes- the legacy RandomState helpers.np.fft.*- the whole 18-function Fourier module, a pure-managed pocketfft port, bit-exact incl. float32/float16 values -3b9d5cfb,a525e355,4cb91898.fft,ifft,fft2,ifft2,fftn,ifftn- complex forward/inverse (1-D/2-D/N-D).rfft,irfft,rfft2,irfft2,rfftn,irfftn- real-input transforms.hfft,ihfft- Hermitian-symmetric transforms.fftfreq,rfftfreq,fftshift,ifftshift- sample-frequency & shift helpers.np.einsum- Einstein summation, now computing and planning -7d2d7a2f(+d78e07db,b61b0998),bb63ba48.einsum- contracts via the matrix products (rides OpenBLAS when the package is referenced).einsum_path- greedy/optimal contraction planner, byte-exact info string.np.r_/np.c_/np.ix_/np.s_/np.index_exp- the grid & slice-expression DSL, 131/131 bit-exact vs NumPy 2.4.2 -00dfe402(+3c63734d,7eea4f7f,c4e27523).np.ogrid/np.mgrid/np.meshgrid- open-mesh / dense-mesh / coordinate-matrix grid constructors, differential bit-exact vs NumPy 2.4.2 -19feaed2,7f558d05,4e8c3925.NDIterRefengine (37 cases probed side-by-side, all identical) -8bd882b3,7112cbe4.np.nditer,np.ndindex,np.ndenumerate- the boxed iterators, full flag/error parity.np.nested_iters,ndarray.flatiter- nested-loop iterators + a write-through flat iterator.b3505398(+8cad3025).partition,argpartition- kth-element partial sort (value + index).lexsort- indirect stable multi-key sort;sort_complex- real-then-imag complex sort.nanargmax,nanargmin- NaN-aware argmax/argmin.np.take_along_axis- the per-slice gather (theargsort/argmaxinverse), NumPy 2.4.2 parity, 24,000+ fuzz cases bit-exact, ≥1.5× faster on every measured variation -f351600a(+88550d13,7091a3c9).np.select- pick each element from the first choice whose condition is true, NumPy 2.4.2 parity (fused single-pass kernel on the contiguous path) -fc10404d(+42d96a14).np.isin+intersect1d/union1d/setxor1d/setdiff1d- element-wise membership + sorted set algebra, NumPy 2.4.2 parity (1.9-13.5× faster) -bfe952d5(+27632ed5).unique_values,unique_counts,unique_inverse,unique_all, 102/102 bit-exact vs NumPy 2.4.2 across 13 dtypes -bec1c497.np.diagfamily + triangular ops - 13 functions, 165/165 side-by-side parity with NumPy 2.4.2 -27b9b012(+a7782984).diag,diagflat,fill_diagonal- diagonal build & in-place fill.tri,tril,triu- triangular masks & extraction.diag_indices,diag_indices_from,tril_indices,tril_indices_from,triu_indices,triu_indices_from,mask_indices- index generators.np.*products; byte-parity via the OpenBLAS backend when referenced -53d7764f(+81509766,297f883f,74aa5d5a).inner,vdot,vecdot,matvec,vecmat,tensordot,multi_dot,matrix_power.956f3392(+2628c921,d6a50593).poly,roots,polyfit,polyval- construction / fitting / evaluation.polyadd,polysub,polymul,polydiv,polyder,polyint- arithmetic & calculus.poly1d- the polynomial object;vander- Vandermonde matrix.savetxt→loadtxtround-trips -a1920a4a,17a1ff8a(+80a0ed50,d39ff824).np.savetxt,np.loadtxt,np.fromstring.9fa48041(+615f1ee5).arcsinh,arccosh,arctanh- primary ufuncs;asinh,acosh,atanh- Array-API aliases.deviceconformance (CPU shim) -ebba2cbf.ndarray.device,ndarray.to_device, anddevice=onarray/zeros/ones/empty/arange/ ….np.kron,np.cross- Kronecker & cross products -7bcad845,73019dce.np.cov,np.corrcoef- covariance & Pearson correlation -92dc537b,aaf731b2.np.choose- index-into-choices gather -aaa41ef2.np.nancumsum,np.nancumprod- NaN-aware cumulative scans -0370c0aa.np.digitize,np.bincount- bin-index + integer histogram, bit-exact vs NumPy 2.4.2 -f2cefba2,12f484c3.np.correlate- sliding cross-correlation (managed SIMD; OpenBLAS byte-parity below) -12f484c3.np.bmat- block-matrix assembly -6ba24752(+d5621d57).np.real,np.imag,np.angle,np.conjugate/np.conj- complex component / phase accessors (post-FFT spectrum extractors) -8b0ac701,d0081b6d.np.iterable- NumPy's pure iterability predicate -ce560796(+8cf54d35).np.isfortran- F-contiguity predicate (a.flags.fnc) -30453696.np.linalgfactorisation surface and complex128dot/matmulare listed under New NuGet Packages above (they compute via the OpenBLAS backend) -dc448acc,f5ec6276,d09e4376,6ee562da.🧩 ndarray surface
ndarraymember parity with NumPy 2.4.2 -data- the memoryview buffer object (np.MemoryView); accepted zero-copy byarray/asarray/frombuffer/ … -25ae7053(+4072577d,bc544403).byteswap- width-dispatched endian byte-swap -67994cbc.getfield,setfield- byte-field views -4b07b71d.real,imag,conj,conjugate- complex accessors -7765ce50.itemsize,nbytes,fill,flags- metadata members -792a9f14(+aee7cbab,27a19ae4);setflags- write/align control -275f089c.all,any,clip,take,repeat,squeeze,trace, …) -06869352.⚡ Performance
Ratios are NumPy ÷ NumSharp - higher is better (
x2= twice NumPy's speed);xLOW->xHIGHspans the worst→best measured cell across sizes and dtypes.1088 bytes -> 192 bytes- NDArray allocation size has been reduced by utilizing StructLayout.Explicit.x0.98->x74-np.uniquefamily routed through the radix sort core -5df10897(+35d12699).x1.6->x5.2-percentile/median/quantilepivot-stack block-partition quickselect -8a1376ff.x0.4->x2.75-np.argpartitionon the same block/pivot-stack path -75a1d873.x1.35->x11-np.isinhash-set membership replaces sort+searchsorted -bd96d541.x15.6- blocked GEBP double GEMM for transposed-Bdot(2.9→43 GFLOP/s) -97e9e82a(+7d680eb1).x40->x249- typednp.nditer<T>/nditer_chunks<T>, allocation-free iteration (chunks +Vector<T>hits 249×) -d58f3728.x1.0->x4.5-take/put/placeelement-copy specialization + gather prefetch (takewent from x0.68 losing to winning everywhere) -88550d13.x1.04->x11.7- float32exp/log/sin/cos/tanh+rad2deg/deg2radreimplemented as bit-exact NumPy kernel ports (tanhalso replaces the float64 loop) -ecdb4581,6bab5754,f5f21ff3.🎯 Parity & Fixes
ndarray.flags/setflags- full NumPy 2.4.2 parity across the whole layout/producer space, hardened by a 1104-case differential oracle (owndata/writeable/contiguity, squeeze-as-view, split-child contiguity, read-only reduction scalars) -275f089c,53b5d82e,ca1b0fac.searchsorted- complex lexicographic order +result_typekey promotion (no more silent key down-cast) + NaN-as-largest total order -93abe13d,cc676ea8,f2cefba2.np.take/np.putindex validation matches NumPy - a negative index undermode='raise'normalizes once (np.take(a, [-1])addresses the last element instead of throwing), and a non-castable float/complex index raises the verbatimTypeErrorinstead of silently truncating -fc10404d,88550d13.np.correlate/np.convolve- OpenBLAS byte-parity via the new sliding-dot seam -d0be3132.broadcast_tois read-only,broadcast_arraysis writeable, and writing a non-writeable view now raises NumPy's verbatim message instead of silently corrupting the shared source -1eadb83b,6fb518c0,1cc67d47(+baf41c89).size×itemsizeoverflow,reshape(-1, …), andexpand_dimsaxis now raise NumPy's verbatim texts instead of silent wrong-size allocations or raw .NET exceptions -c2552d6a.ones(3000)@ones(3000)=2048 saturation), stacked/fancy indexing into zero-sized arrays, and the 0-d boolean setter -03d0f0c8,7636100a,f6e258c0.SetIndicesNDNonLinear), bit-exact vs NumPy 2.4.2 across all 15 dtypes -ff68bf14.astype(copy: false)never mutates the caller's array on a dtype conversion, matching NumPy -e5274cdc.ndarray.view(dtype)of a different-itemsize dtype now follows NumPy 2.x's last-axis-contiguous rule, soarr[::2].view(int32)works instead of throwing -970ee7f1.np.matmulgains the full ufunc keyword surface (out=/axes=/axis=/keepdims=/dtype=/casting=/order=), andnp.dot/np.outergainout=-73019dce(+87ff5797).argmax/argminbugs the sort audit exposed - the Decimal and Char flat paths and a NaN-tie ordering - now match NumPy 2.4.2 -8cad3025.np.uniquefull-parameter parity with NumPy 2.4.2 - the axis path's slab equality is corrected so each NaN sub-array is distinct and signed-zero sub-arrays collapse (a real unique-row-count bug for floats/complex),sorted=/equal_nan=are accepted, an out-of-range axis raises the verbatimAxisError, the bare-return overloads (np.unique(ar, axis: 0)) andintersect1d(return_indices:)now port verbatim, andUniqueResultfields are case-identical to NumPy -9f573dd5,262eefd7,0151a832.np.linalgfactorisations without a backend now raise a typedOpenBlasMissingBackendException- derives fromNotSupportedExceptionso existing catches still work, and names theNumSharp.Interop.OpenBLASpackage to install (was a bareNotSupportedException) -d1347c36.b2a8374b:np.ascontiguousarray/np.asfortranarray- a 0-D input returns a length-1 view (shares storage), matching NumPy's ndim≥1 contract.np.eye/np.ones-Charfills numeric one U+0001, not the character'1'.np.full_like- preserves the source array's dtype;fill_value's CLR type no longer selects the result dtype.np.linspace- floors inexact values before an integer-dtype cast and pins the endpoint tostopexactly.np.einsum- a scalar (ndim==0) contraction keeps its()shape instead of promoting to(1,).np.angle(deg: true)- a 0-DHalf/Singleresult keeps its float tier instead of promoting toDouble.var/std/cumsum/cumprod/all/any) andndarray.fillnow run at unlimited ndim like the rest of NumSharp -8f34e8ff,7fa96750.🧰 Testing & Tooling
out=/where=(3,727 cases over out × mask layouts), result-kinds + verbatim-error + iterator-trace, IEEE special-values (nan/±inf/±0/subnormal), and a truthful-vs-precise precision channel -bc91dd25,6cd1de9b,0882edbb,359e9d3c,76f0c918.np.randomsampler byte-parity divergences (f/pareto/standard_cauchy/binomial/negative_binomial/multinomial/multivariate_normal/gamma(shape<1)), now pinned as known[OpenBugs]issues (not yet fixed) -31a178f2.cf559a1a,03415ec9,5ff54a72.💥 Breaking Changes
np.random.bytes/Generator.bytesnow returnNDArray<byte>instead ofbyte[], so draws >2 GiB succeed (NumPynpy_intpparity) -44d2e7d9.ndarray.stridesnow reports bytes per axis (was elements), matching NumPy'sPyArray_STRIDES-6ef30215.np.unique(ar)now returns aUniqueResultstruct instead of a bareNDArray, sonp.unique(ar)[k]selects the k-th output (use.values[k]for the k-th value); it converts implicitly toNDArray/NDArray[]so most call-sites are unchanged -17f571ef.np.mgrid/np.meshgriddrop their legacy non-NumPy signatures:mgrid[...]is now an indexer (was a 2-arg method) andmeshgridis variadic returningMeshgridResult(was a fixed 2-tuple +Kwargs) -7f558d05,4e8c3925.PublicKeyTokenchanges fromnulltocc7b13ffcd2ddd51(published NumSharp had shipped unsigned since 2019); every consumer (TensorFlow.NET, Pandas.NET, Gym.NET) must recompile -478d550d.NUMSHARP_<AREA>_<SETTING>scheme with no back-compat aliases -NUMSHARP_GUARD_PAGES(shipped in 0.60.0) becomesNUMSHARP_DEBUG_GUARD_PAGES, and the OpenBLAS/pythonnet knobs take_LIBRARY/_SEARCH_PATH/_USE_BUNDLED/_PYPI_FEED_URL/_REQUIRE_ENGINEnames -079d1859.NDArray.Normalize()(a non-NumPy extension) is marked[Obsolete]in favour ofnp.clip()-b701843e.