Share Rcpp object/pointer between R and Python?

[gregorgorjanc]

This would be very cool and would simplify a lot, but I don't know how to do this. Here is a question on Rcpp repo RcppCore/Rcpp#1439 (reply in thread)

In meanwhile we have serialisation (disk-write-and-read) in place between R and reticulate Python (both ways) via ts_py_to_r() (see https://github.com/HighlanderLab/tskitr/blob/020cc396fbf5a6a710d3ded6345bcb2ce308f5b2/tskitr/R/tskitr-package.R#L282) and ts_py_to_r() (see https://github.com/HighlanderLab/tskitr/blob/020cc396fbf5a6a710d3ded6345bcb2ce308f5b2/tskitr/R/tskitr-package.R#L247)

[gregorgorjanc]

Got good feedback at RcppCore/Rcpp#1439 (comment). Effectively we have two next actions from there:

• simplify the "pointer code" (see https://github.com/HighlanderLab/tskitr/blob/878d539c5508369875b9003a27701c679392ce52/tskitr/src/tskitr.cpp#L105-L106 and other parts) and study how the units R package does it https://github.com/r-quantities/units/blob/29a13c0d62304ab97a0961a194b6d5ceaf616b36/src/units_types.h#L12-L13 --> DONE at 5524ccc
• study how arrow passes pointers between R and Python by studying arrow:::r_to_py.Array() and arrow:::py_to_r.pyarrow.lib.Array() (note that the r_to_py() and py_to_r() are S3 generics, so we could develop tskitr:::r_to_py.TreeSequence() and tskitr:::py_to_r.TreeSequence(), I guess) --> DONE at Is it possible to share Rcpp object/pointer between R and Python? RcppCore/Rcpp#1439 (comment)

So, in principle we know what should be done here. The more important question however is if we should really be doing this and what would be the benefit of passing pointers between R and reticulate Python instead of serialising (writing/reading) to disk using ts_r_to_py() and ts_py_to_r(). I assume time will tell and we will revisit this issue/question once we start working with large tree sequences where disk serialising could become a hassle. Related to #12 and #3

[gregorgorjanc]

A really well worked example of how this could be done is in RcppCore/Rcpp#1439 (comment)

[gregorgorjanc]

Another link to the same discussion at RcppCore showing my test of the proposed solution - seems very doable RcppCore/Rcpp#1439 (reply in thread)