PdfJS inconsistencies in sentence highlight when page rotated

[minump]

Check the issue with incosistent labels and highlights when previewing in web page. This happens for landscape pages within a pdf - when pdf page is rotated.
Test with the input pdf and consort statement

1. Why there is no highlights in page 18 of the original pdf when showing in web browser - we expect to see label 26 there.
2. Why these is no vertical highlights in page 20 of the original pdf when previewing in web browser - we expect to see label 13 there.
3. Why there is a mismatch in position of highlighted sentence in page 20 of the original pdf when previewing in web browser -
   a. Sentences "The effect sizes and the corresponding confidence intervals were first computed in each of the 50 imputed data sets before they were averaged to a single value." and "Further objectives and secondary clinical outcomes not reported in this table can be seen in the Supplementary Appendix" are to be highlighted. But the highlights seem to be misaligned.

2024.01.09.24300978v3.full_highlighted.pdf

2024.01.09.24300978v3.full.pdf

2024.01.09.24300978v3.full_highlights.json

2024.01.09.24300978v3.full_statement_labels.csv

This issue might be related to the separate height and width issue for each page. #137

[minump]

Some findings that are shared in the email related to this issue (copy-pasting the email here):

2024.01.09.24300978v3.full.pdf - original pdf input file
2024.01.09.24300978v3.full.json - grobid output
2024.01.09.24300978v3.full.csv - input to the extractor (clowder process that is responsible for all processing). Contains grobid coordinates for each sentences
2024.01.09.24300978v3.full_predicted.csv - model output. Contains the model predictions labels and scores.
2024.01.09.24300978v3.full_statement_labels.csv - the model prediction labels are converted into the labels/item numbers for the chosen guideline.
2024.01.09.24300978v3.full_highlights.json - the labels+highlights+scores+coodinates are read from this file to be processed by PyMuPDF and PdfJS.
2024.01.09.24300978v3.full_report.pdf - downloaded report produced by PyMuPDF
2024.01.09.24300978v3.full_highlighted.pdf - original pdf with highlights produced by PyMuPDF

Of these files, 2024.01.09.24300978v3.full_highlights.json would have all the required information to visualizing the highlighted pdf.

Findings:

1 Mismatch in sentence highlights and position.

During the discussion we found 2 vertical sentences that were highlighted in oval shape. See report pdf page 26 ( page 18 in the original pdf) and page 31 (page 20 in the original pdf).

Sentence 1 - page 18 in original pdf:

1. Checking the output of the model and grobid:

A. Text: "." - a period is labelled as 26 with score 0.95. The coordinates are "18,765.34,426.04,7.40,2.22" from grobid.
B. Text: "; https://doi.org/10.1101/2024.01.09.24300978 doi: medRxiv preprint" is labelled as 26 with score 0.94.

2. Checking the model output csv file (2024.01.09.24300978v3.full.csv):
   A. See row 130. Text: "." is labelled as as '38a_Outcome_results' with score 0.95.
   B. See row 133, the sentence "; https://doi.org/10.1101/2024.01.09.24300978 doi: medRxiv preprint" is labelled as '38a_Outcome_results' with score 0.94.

Sentence 2 - page 20 in original pdf:

1. Checking the output of the model:
   Sentence : "is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity." is labelled as 13-Intervention and Comparator-Details
   This sentence is given the below coords output from grobid:

20,773.34,219.50,7.40,5.78;20,773.34,227.50,7.40,11.12;20,773.34,240.85,7.40,49.81;20,773.34,292.88,7.40,14.67;20,773.34,309…7,7.40,24.90;20,773.34,457.39,7.40,11.12;20,773.34,470.74,7.40,27.12;20,773.34,500.08,7.40,6.22;20,773.34,508.53,7.40,37.35

2. Checking the output of the model:

A. See row 140 in the csv file. Text : "is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity" is labelled as 11a_Intervention_Description with score 0.7.

2 The sentences are correctly highlighted in the file 2024.01.09.24300978v3.full_highlighted.pdf. See page 18,20. No oval shape highlights. This means that PyMuPDF code works well with the page rotation.

Conclusion:

1. The labels + sentences come from the model output.
2. The 2024.01.09.24300978v3.full_highlighted.pdf (original pdf with highlights produced by PyMuPDF) has all the highlights correctly. No weird oval shapes, no missing labels/highlights. See page 18,20.
3. This means that there is something happening when the report is generated - when the report table checklist is joined with the highlighted pdf.

Further investigation needed :

1. Not sure why the highlight rendering changes when a checklist table is appended to the highlighted pdf using PyMuPDF.
2. Why there is no highlights in page 18 of the original pdf when showing in web browser - we expect to see label 26 there.
3. Why these is no vertical highlights in page 20 of the original pdf when previewing in web browser - we expect to see label 13 there.
4. Why there is a mismatch in position of highlighted sentence in page 20 of the original pdf when previewing in web browser -
   a. Sentences "The effect sizes and the corresponding confidence intervals were first computed in each of the 50 imputed data sets before they were averaged to a single value." and "Further objectives and secondary clinical outcomes not reported in this table can be seen in the Supplementary Appendix" are to be highlighted. But the highlights seem to be misaligned.

In the above issues,
Issue 1 - could be PyMuPDF appending an html and a pdf using fitz module.
Issue 2 and issue 3 - could be PDFJs and margins issue in the web browser. PdfJS might not render vertical highlights correctly.