Conversation
|
Testing this downstream, report below. Works great. The #423 change is known (we are accessing internals in fill, trivial update to match upstream). The 8x slower comment is known too, it's due to np.take's vectorized gather. It is something we could fix here, but it's not reduce specific; we could address this in a followup for all reductions (would need per-axis lut + bulk copies). I'm not sure how difficult that is, can investigate afterwards. The overflow bins now work, too, so the old boost-histogram "experimental" warning can go away :). 🤖 AI text below 🤖 Verdict: yes, the feature works for us. The whole home-grown What I set up (all uncommitted, for the trial):
Feedback on the feature itself:
So the PR can come out of draft as far as boost-histogram verification goes; the perf note is the one thing I'd consider addressing upstream first. |
|
@HDembinski This is ready for review, by the way. There were several decisions along the way ( |
| /// Tag type for the axis constructor that picks bins (see algorithm::reduce) | ||
| struct pick_tag {}; | ||
|
|
There was a problem hiding this comment.
Using a tag type is clean, but I don't like it because it becomes part of the public API and we managed so far without adding these things. This should be an implementation detail and not part of the public API.
A pickable axis is detected via the special constructor, and it looks like the constructor is already distinct from any other constructors, so I think we can do this without a public tag type. Please give it a try.
Otherwise looks good and it's a nice feature.
There was a problem hiding this comment.
It wasn't originally unambiguous with std::vector, but the current form actually is, I believe, due to the two pointers. (The arbitrary metadata entry was the issue before). I'll push a version that doesn't have a pick tag if it works in a minute.
|
@henryiii There are some conflicts here, can you give it another iteration? I also added a comment. I think this feature is in a good shape, but I want to see whether we can avoid the tag type. |
Add a pick command to algorithm::reduce, which selects an arbitrary subset of bins by index from an axis which is not ordered, like the category axis. Unlike slice, the picked bins do not have to be adjacent, and they may be given in any order, which reorders the bins in the new axis. Counts in bins which are not picked are added to the overflow bin, if it is present, consistent with how slice treats unordered axes; otherwise they are discarded. Axes opt into picking with a new special constructor which accepts the original axis and a vector of bin indices to keep. The new trait axis::traits::is_pickable detects this constructor, mirroring is_reducible. axis::category implements it; ordered axes throw invalid_argument, since picking a non-adjacent subset from them would require changing the axis type, which reduce does not support yet. Fixes boostorg#275 Assisted-by: ClaudeCode:claude-fable-5
Windows CI failed two ways: - C1128 (too many sections) when compiling algorithm_reduce_test.cpp; add /bigobj for the test under CMake, matching the b2 build which already sets it globally and the existing fill/operators tests. - C4702 (unreachable code, treated as error) for the non-pickable static_if branch in reduce(). Unlike is_reducible (true for all standard axes), is_pickable is false for most axes, so that branch is codegen'd and MSVC flags the value after the noreturn throw. Guard the function with a 4702 pragma, mirroring detail/fill.hpp. Assisted-by: ClaudeCode:claude-opus-4.8
The is_pickable trait matched any axis whose generic (iterable, metadata) constructor accepts a vector of indices, e.g. variable<double, std::vector<int>>, which broke compilation of reduce() with any command on such axes. The pick constructor now takes an axis::pick_tag so only axes that opt in match the trait. Also mirror the MSVC /bigobj flag in test/Jamfile, simplify the pick branch of the fill loop, hoist index validation out of the static_if, and copy metadata via metadata_base(src) like the shrink constructors. Assisted-by: ClaudeCode:claude-fable-5
Replace the per-cell linear search over the picked indices with a lookup table built once per axis. Picking 1000 of 2000 categories in a 2000x500 histogram drops from ~200 ms to ~15 ms, on par with slice. The table also routes the underflow bin of a hypothetical pickable axis to the overflow bin instead of reading out of bounds; the Axis concept now documents this. Also move the MSVC pragma block above the doc comment and clarify in the guide that pick cannot be combined with other commands. Assisted-by: ClaudeCode:claude-fable-5
pick accepts an optional slice_mode like slice. In crop mode the counts of unpicked bins and of the original overflow bin are discarded instead of being moved to the overflow bin. Assisted-by: ClaudeCode:claude-fable-5
…tructor The pick constructor was the only user-facing signature in the library that requires std::vector. A pointer range matches the iterator-pair idiom of the other axis constructors and the primitive arguments of the reduce constructor, and does not freeze a container type into the Axis concept. Assisted-by: ClaudeCode:claude-fable-5
- Store the pick lookup table in a dedicated reduce_command member instead of overwriting the pick list mid-algorithm. - Cross-reference the primary pick() overload docs instead of duplicating them. - Drop the per-test /bigobj flag from the Jamfile; the project requirements already apply it to all MSVC tests. Assisted-by: ClaudeCode:claude-fable-5
The pick constructor takes a pointer range, which no generic axis constructor accepts, so is_pickable can detect it without a tag type. This keeps the pick mechanism out of the public API surface. Add trait checks that an axis whose metadata is constructible from a pointer is not mistaken for pickable. Assisted-by: ClaudeCode:claude-fable-5-1
Use reduced_end for the pick path instead of repurposing end.index, so the per-bin loop has one overflow-redirect tail. Index the lookup table by old index + 1 to cover flow bins without a ternary. Give the non-pickable lambda an explicit return type so the noreturn throw ends the body, which removes the MSVC C4702 pragma. Assisted-by: ClaudeCode:claude-fable-5-1 ClaudeCode:claude-fable-5-1
This implements a pick command for
algorithm::reduce, closes #275. Adds a new constructor:To be pickable, an axis must have this constructor, using
axis::pick_tagto opt in. Pointers are used in the signature to avoid forcing a specific collection likestd::vector(and C++17'sstd::spanis not available); this is similar to(It begin, It end)iterator-pair constructors ofaxis::variableandaxis::category. Concrete pointers are used instead of a template so thataxis::traits::is_pickablecan detect the constructor withstd::is_constructible. Categorical axes gain this new constructor.Picking uses
pick({2, 0})(astd::vector). The items here do need to be a specific collection, similar to thestd::initializer_listconstructors ofaxis::variableandaxis::category, because the type-erased reduce_command must store the indices in a concrete container.🤖 AI text below 🤖
This adds a
pickcommand toalgorithm::reduce, which selects an arbitrary subset of bins by index from an axis which is not ordered, like the category axis. Unlikeslice, the bins do not have to be adjacent, and the new axis keeps them in the order given.Summary:
A(const A&, axis::pick_tag, const std::vector<index_type>&), detected by the newaxis::traits::is_pickabletrait. Thepick_tagdisambiguates the constructor from the generic(iterable, metadata)axis constructors, which could otherwise match by accident and break compilation ofreducefor unrelated axes. Onlycategoryopts in.pickcannot be combined with another reduce command on the same axis; indices must be unique and in range.