Simultaneous support for Python 2 and Python 3#46
Simultaneous support for Python 2 and Python 3#46teeks99 wants to merge 21 commits intoboostorg:developfrom
Conversation
Were you able to make any more progress with this? Are you running your regressions under python 3? I see in collect_and_upload_logs.py row 182 they're dealing with it as a text file. Opening that file as binary, then Similarly in process_xml_file() row 103. Neither will get as far as the |
|
I haven't looked at this in a very long time, but I believe that issue is still there. I'm still using python 2 for the regression runs. |
|
Asking AI about this, and so far here are issues it noticed:
should be: This was changed in one location already correctly, but in another place it should still be fixed.
There is an instance of this code: It recommends: Again the same footnote applies: this was fixed in one place correctly, it should be added in another location. It would be great to complete this pull request. There are multiple options.
Let me know what you think. If I should work on it, then it would be helpful to know the steps to replicate the bugs you saw before. A step by step explanation so I could reproduce the errors locally. |
|
@sdarwin Thanks for looking into this. Both your changes seem legit, but as you see from my comment above, there are also unicode/byte issues I've encountered when trying to actually run the regression tests myself, that static code analyses (like yours) won't catch. Take a look at the boost regression matrices. You'll see that @teeks99 is still faithfully running them (thank you!), but that he's the only one. There's a problem on the server side: when I ran the regressions (under Python 3), I got as far as the results uploading, which should put them on that matrix, but the matrix never updated. So whichever script that resides in that server, that transforms uploaded regression results to the matrix, fails for my uploads (and anyone else who's tried it, though maybe it's only me), but clearly continues to works for his. We need to troubleshoot that, and only an admin like yourself has permission to watch it (or see its logs). Once we're seeing results reliably, then I can fork from this pull request, work through remaining runtime issues, and ultimately do a pull request with higher confidence that it will work end to end. You'll notice too that the master branch matrix is completely neglected. So running master tests to both verify its operation and put something there would also be good. See also: Test Matrix Docs. |
|
Hi Jim,
In terms of uploads, the previous configuration was that uploads go to "results.boost.org" and then were displayed on the earlier "www.boost.org". Now... I have rebuilt the ftp server, and using the same URL "results.boost.org" and the same method, results actually go to a directory on "regression.boost.org", which hosts the matrix. We can attempt to debug this. If you are able to upload results, tell me the filenames (and dates and sizes, although not really required) and I will run |
This is a bit of a tangent for this thread, but the master tests are running, they are all just failing due to an issue in the codebase. This has been ongoing for many months. |
|
This thread claims that the issue would be resolved if graph_parallel develop is merged to master. But it has been merged. |
|
I just tried the graph_parallel tests on the current master, and they (mostly) work for me. (I get two compile errors caused by missing |
Great to hear. I have a regression result that uploaded early afternoon, file name I did have to work through the master branch jamfile failure that @teeks99 mentions above. I'll do pull req on that after this is resolved. Very small change. |
|
Thanks. afk, will be able to check Mon.
…On Sat, Apr 18, 2026, 7:55 PM Jim B ***@***.***> wrote:
*jimmbelll* left a comment (boostorg/regression#46)
<#46?email_source=notifications&email_token=AAJTCS77RAQREFCKGPRQ46D4WQW2RA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRXGQ4TMNBWGI32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4274964627>
We can attempt to debug this. If you are able to upload results, tell me
the filenames (and dates and sizes, although not really required) and I
will run locate on the server to see if they are present.
Great to hear. I have a regression result that uploaded early afternoon,
file name jcbell-msys2-mingw-w64.zip ; I see that the master matrix
<https://regression.boost.org/master/developer/summary.html> has updated
since but has not picked up my result. Hopefully that gives you enough to
go on.
I did have to work through the master branch jamfile failure that
@teeks99 mentions above <#m_-1474967229374152954_issuecomment-4273616831>.
I'll do pull req on that after this is resolved. Very small change.
—
Reply to this email directly, view it on GitHub
<#46?email_source=notifications&email_token=AAJTCS77RAQREFCKGPRQ46D4WQW2RA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRXGQ4TMNBWGI32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4274964627>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJTCS3HSX5JHG42FFZBS234WQW2RAVCNFSM6AAAAACX5FM6V6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DENZUHE3DINRSG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@teeks99 How difficult is it for you to switch all your regressions to python 3? Which 3.x version would you run? As you may know, python has lots of evolution (and deprecation) throughout 3.x. Not sure how much it affects this code, but also not sure how much effort should go into staying compatible with 2.x, especially pre-2.7, and even pre 3.9. (Does boost have a policy here?) |
|
I'd love to switch to python 3, hence why I started this incomplete MR. It would take a small amount of time to switch over and would save me a ton of time not having to build python2 from source for inclusion in my runner images. I would like support for a fairly old version of python 3. I have ubuntu trusty images that still run tests on gcc-6, and that |
|
There's no need to use Trusty for anything anymore, much less GCC 6. |
|
I'm sure I could find a different way such as building gcc6 from scratch, but I have it working with trusty now, and would kind of like to leave it alone. |
|
GCC 6 is available on Ubuntu 18.04 via apt-get from the built-in repos. |
|
I've confirmed that I can update my images to 18.04 and keep the existing gcc runs. That gives us python 3.8. |
|
Before, boost_wide_report.py matched: now adding this: The effect is that the jcbell zip file is uploaded, ingested, unzipped, and available on the website. Many files such as: https://regression.boost.org/master/jcbell-msys2-mingw-w64.html That is progress, step by step. The summary page still doesn't update. However the page is generated during CI and the contents are the same as what's shown: https://regression.boost.org/master/developer/summary.html CI: https://app.circleci.com/pipelines/github/boostorg/regression It would be helpful to upload all the master branch files, and debug the master branch, why have those stopped? In terms of upgrading python, most of the progress so far has been to "Simultaneous support for Python 2 and Python 3", and that seems like a natural strategy, because then you don't have to be overly concerned about scheduling or cut-over methods or migrating containers. The PR can be merged whenever it's working. Convenient. Just complete the steps of "Simultaneous support for Python 2 and Python 3". It's almost done. After a transition there could be further work to improve and optimize scripts. At that point abandon Python 2 or break compatibility with Python2, but that happens later. |
Ok, wow, I had no idea there was such a specific/narrow list of matches. Seems fragile (my case), and like it discourages new testers pretty much by design. I can imagine malicious uploads (and you may want to scrub the URL from your comment above mentioning it). Is that the rationale? Is there a history of such? If so, can we filter by content (markers in the file) vs. a name match like this? Also, a technique to obscure the upload URL in the source code might be helpful, since github makes the source more available than in the olden days. With python you can do it on import with a technique limited only by your imagination. A fun puzzle. |
Seems like you alone have the permission to do this. I'm glad to look at a log if that helps.
My PR forked from this will have all fixes in it, and I'll submit it after the master branch debug. I'm a git novice: I'd love to somehow merge mine with this PR to link it with this discussion. What's the git magic to achieve that? Also, @teeks99 I'd love for you to drop my set of changes into your environment(s) for your runs before the merge. As you know with python, there's no substitute for running it. And maybe then you update your fork and this is the PR after all. |
The builds depend on test results being uploaded to results.boost.org. Here is how the upload directory looks for the 'develop' branch: It's full of recent uploads. And then on master: Nothing since Dec, and those might be from the server migration, and actually be even older. The existing python scripts filter and only use the last 1 month worth of data, so the information from December would be excluded and not shown. This explains the current situation, right? teeks99 is not uploading any data. That is presumably due to a build error. It should be reported and analyzed and tested, maybe here, or in the boostorg/boost github repository. Then, log files are visible at https://app.circleci.com/pipelines/github/boostorg/regression . It's another important component to analyze. But for now, secondary to the known issue above.
"Maintainers are allowed to edit this pull request." If you are a maintainer of a repo, you can directly modify a PR, and add more commits directly. However, it's probably safer to open a separate parallel PR, if you are an admitted "git novice". :-) No problem. And it's not particularly worse. We can just merge the newer pull request. |
Yes. The master branch jamfile issue is a show-stopper, halting the regression, and I have a fix. |
|
I don't understand why you're encountering this issue and I am not. What is your fix? |
|
Ah, I see it. It's not the same issue. This is |
Yes: two instances of '../../system/build' near each other, only references in the file. That directory doesn't exist, but '../../filesystem/build' right next to it does, and doesn't error. This halts b2 (bjam) immediately. With this change all libraries seem to process. |
|
Good news! The master matrix updated with my two tests successfully! And results look sane. (Note that the Release View and User View links at the very top give 404 errors.)
I'm not sure what I'm looking at here, but some observations:
What's the relationship between this circleci "pipeline" and the matrix-building code running on the server? |
|
The design of boostorg/regression is extremely unusual. The relevant scripts are strewn across all those branches: develop, ci/test-reports, ci/test-reports-circle2, . No master branch exists. Nor should it, necessarily, based on the current configuration. There is no correlation between processing boostorg libraries and the master branch of boostorg/regression. If you examine files in develop and ci/test-reports, I believe most of the logic can be found in the python scripts. The actions which occur are distributed between the CircleCI jobs, and the server itself. Most processing happens in CircleCI. But not everything. The server will extract zip files, and copy them to an html directory for web hosting. Actually... even that step is done through CircleCI. |
So, specifically, when regression/reports/src/boost_wide_report.py line 115 executes Where exactly does that message wind up? A bit off topic but very useful information. |
It's in the CircleCI output. 
|
4 participants
I've made the changes to follow the program through in both version of python. This Fixes #43.
Please don't merge yet, there are two known defects.
Different results
The windows version shows slightly different results on windows. Looking at the develop results the
teeks99-test-py23-p2-wandteeks99-test-py23-p3-wthey are the same except for thebuildresults which are missing for the python 3 version.Different output
On linux (running python 3.5), I get an exception running through the log for regex. Running with Python 3 the file
results/boost/bin.v2/libs/regex/example/icu_example.test/gcc-8~c++2a/debug/threading-multi/visibility-hidden/test_log.xmlcontains:whereas the python 2 version contains:
The difference on the
£198.12vs.<C2><A3>198.12causes aUnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 1803: ordinal not in range(128). Any ideas what could be causing the differences in the xml files? Something about the environment that runs the tests?