Rewrite emphasis handling by facelessuser · Pull Request #1632 · Python-Markdown/markdown

facelessuser · 2026-09-07T19:36:05Z

This is a complete rewrite of how emphasis handling is done.
Drop use of multiple regex patterns run in multiple passes and instead evaluate delimiters, nested or otherwise, and build up HTML elements.
Both * and _ are handled at the same time.
Smarter handling of delimiters: left should not be followed by space and right should not be preceded by space.
Try to consume tokens as much as possible until a full element is constructed (with children if any).
If an outer set of tokens cannot be resolved, but one or more sub tokens can, render the first sub token span and cache the remaining ones for subsequent reentry and render those until the cache is exhausted.
Two tests results were updated to match new behavior.

Description

AI Assistance Disclosure

No AI tools were used in preparing this PR.
If AI tools were used, I have disclosed which ones, and fully reviewed and verified their output.

Checklist

This PR follows the contribution guidelines.
The code follows the Code Style Guide.
The commit message follows the Commit Message Style Guide.
I have added or updated relevant docs, including release notes if applicable which follow the [Documentation Style Guide](Documentation Style Guide).
I have added or updated relevant tests.
I have not requested, and will not request, an automated AI review for this PR.

- This is a complete rewrite of how emphasis handling is done. - Drop use of multiple regex patterns in run in multiple passes and instead evaluate delimiters, nested or otherwise, and build up HTML elements. - Try to consume tokens as much as possible until a full element is constructed (with children if any). - If an outer set of tokens cannot be resolved, but one or more sub tokens can, render the first sub token span and cache the remaining ones for subsequent reentry and render those until the cache is exhausted. - Two tests results were updated to match new behavior.

facelessuser · 2026-09-07T19:38:38Z

This is purposely a draft and is being made available for testing. More tests should be written, etc. For review.

The proposal is to first release this in Pymdown Extensions, and once vetted in the real world, make it the default approach, but I will let Python Markdown make the decision to release it before then if desired.

waylan · 2026-09-07T23:33:25Z

Awesome! I haven't had an opportunity to fully review the code, but the idea seems sound. The one thing that gives me pause is that we are reducing the number of inline processors to run. In itself, that is not a problem, but it may be that some third-party extensions are expecting to inject their processors between a few of the now-combined processors. I'm not certain if that will cause issues for those extensions or not. Therefore, my inclination is to consider this a backward incompatible change. Presumably the release notes should include a warning for third-party extension devs. I don't think there is any way we can gracefully make these changes through deprecations.

Interestingly, the change this relies on already was pushing the next release into at least a minor release. Adding this to the same release would be ideal from that perspective. But I also see the desire to test more thoroughly. I'm undecided about it for now.

facelessuser · 2026-09-07T23:58:25Z

Yep, I'm under no illusion that this is not a big change. While I think that if people notice anything, it will be minor differences, or actually have it handle certain nesting more like other parsers, it is a complete and total rewrite from the ground up, and that does carry some risk until fully vetted.

The one thing that gives me pause is that we are reducing the number of inline processors to run.

Totally understood. I wouldn't design it differently, but I get why this is a flag that it's a bigger change and may have other surprising impacts outside of what becomes emphasis and what does not.

I've fiddled with the regex for a while now to try to make case X better, or make case Y more performant, but I think I finally just reached the limit of what could be done with multiple regex patterns applied in sequence, and finally did what I always knew was the answer. But to be honest, without the recent fix in Python Markdown, I couldn't implement this.

Personally, I'm releasing it as a "breaking" change in Pymdown Extensions, despite expecting/hoping most people won't notice differences.

I'm okay with whatever Python Markdown wants to do, and I'm happy to clean it up for merge if we want to push it out early rather than later, but I'll leave it in the draft state unless I hear a push to get it in. At the very least, it is available to be freely tested and put through its paces, and I think that is the most important thing; we now have something that can be evaluated for as long as we feel is necessary.

This is a prevented measure to ensure the processor is always in a good state. This situation has never been observed, but if it did occur, this would allow the processor to reset its state and continue properly. Only extensions have a way to reset, processors don't. Nor do they have a way to detect when a reset would be needed. - Add a current time for each Markdown run. - Have DelimiterProcessor compare the each run time, and if it has changed, perform a reset of the stack.

facelessuser · 2026-09-08T14:45:26Z

I've cleaned things up a bit and fixed failing CI stuff. Over the next couple of days, I'll bring over some more tests to make sure we have good coverage and checks that assert expected logic for more advanced nesting cases.

The only new addition is that I added a last_run to the Markdown object, which is just a timestamp. We can use this to tell if we are in a new run or not.

Inline processors don't have a reset hook, that requires an actual, registered extension, but since we do preserve a state between calls so that we can efficiently serve up already found emphasis, it is possible if the Markdown object were to crash for any reason, the processor could be left in a bad state on a rerun. So adding this timestamp allows us to see that we are in a new run, so we can reset the state if it hasn't already been done.

This is a preventive measure, and I haven't actually encountered this case, but it is certainly a plausible scenario.

- Fix an issue in element building where we should have exited - When an ambiguous `***` is partially consumed by a smaller start, the remainder should be a non-ambiguous start. - Don't consume an ambiguous opening with an end that is smaller than the opening. - `***` are always consumed, even when ambiguous.

facelessuser · 2026-09-10T20:09:44Z

Unless I find any issues or I get feedback for more changes, I'm going to consider this done. I think I've caught the main concerns.

facelessuser · 2026-09-10T20:19:42Z

The only thing I may consider is possibly scanning the full buffer the first time. Currently, we stop after we get a complete element or exhaust the buffer, searching to complete all the potential spans we find. With little effort, we could probably just always scan until the end, and then return elements from the cache. I don't think you gain much additional speed per se, but there is no reason we can't just complete the scan.

Anyway, I seem to have figured out how to get this working without the latest performance fix in Python Markdown, so I could release this approach in Pymdown Extensions. I think you lose a little performance on a few very specific cases, but it is generally faster than it was before the rewrite, so it's a win either way. The performance fix just makes some specific cases even faster. At the very least, this just means Python Markdown can possibly take more time considering things if required.

facelessuser · 2026-09-11T13:21:13Z

I'm running these changes through CommonMark test suite. I'm going to see what kind of tests fail and see if it is reasonable for us to address them.

I know they handle punctuation differently, but the Python regex engine does not expose Unicode Properties, so I probably won't bother with anything there.

I know they parser underscore and asterisk at the same time, something we don't, so I'm not worried about that. It's something we could consider in the future though.

I'm possibly expecting some issues with the fact we consider emphasis tokens of 1, 2, or 3. Their logic may handle larger spans a bit different. Well see.

I'm expecting some difference in how Python Markdown parses things related to order, but that is expected.

Anything else may be worth at least taking a look at.

I know we aren't a CommonMark parser, but just curious how much closer we are, and what do we currently miss.

- When consuming a chain of tokens greater than 2, consume in increments of 2 until all are consumed or 1 is left, and then consume 1. This means that the previous assumption that triple tokens yield `` is now ``. While the syntax is different the logic is the same. This is a necessary change in order to properly consume tokens. - Small clean up of pattern building - Add common mark tests

facelessuser · 2026-09-12T01:29:42Z

Running this through CommonMark, basically everything I thought was confirmed. With changes, I can get all to pass but 1 to pass, and that's because we currently run underscore and asterisk parsing in separate passes. I'm okay with that. If we change our mind in the future, we can certainly change it, but for now, I'm okay with that.

The remaining test failures fall into two buckets. I have fixes for both, but I'll hold off on the last one to get feedback.

Token chains longer than 3

We currently don't handle chains longer than 3, but I have a fix for this. I think this is a reasonable thing to address now that we aren't reliant on multiple regex patterns.

It doesn't affect any pre-existing tests logically, but it does swap  as  for triple tokens. This is an algorithmic requirement as it is how tokens are logically consumed. There really isn't a way around it, but it doesn't change how rendering behaves.

Because this actually improves parsing, I'll push a commit with this change. I guess if there is strong opposition, I can revert it.

EDIT: it may be possible to special case things to avoid the flip, but its not my preferred approach.

Punctuation

We don't handle punctuation as they do, but I could patch this fairly easily. We don't have to handle punctuation like CommonMark, but we could.

Python doesn't currently expose Unicode properties in the regex engine, but we could create a string of the characters for the Unicode property and simply insert them into the pattern. It's pretty easy and works pretty well. We could simplify it to ASCII punctuation as well.

If this isn't wanted, we can simply omit it as a choice.

I'll hold off on this to see if there is desire for this.

facelessuser · 2026-09-12T01:43:17Z

Some redundant PL2004 and PL2007 tests are failing. I was running unittest directly, and these were not run. There is nothing new here, and it just indicates what I described earlier: the swap of . I'll disable these as they are covered elsewhere.

When building elements, we no longer need to account for regions of 3 tokens as they are always broken up into regions less than 3.

facelessuser · 2026-09-12T12:18:25Z

I've confirmed that I can isolate and avoid the  flip without affecting any other behavior. While I feel nothing is gained by preserving this for the sake of not changing it, if my argument is that there is no logical change, there is no reason it can't be preserved as well.

If we absolutely do not want these tags to flip, I have a simple adjustment that fixes this. The algorithm will still prefer chunking large spans as , but when the remainder is exactly three, we'll chunk first as  then  (inward out).

That would leave us with the question of whether we want to apply CommonMark punctuation awareness or not.

We should grab all tokens, regardless of whether of what size we are looking for. Chunk large spans accordingly a delimiter that only requires single delimiters should chunk by one, all others by two. Limit adding new starting spans if a delimiter only requires double tokens. Also fix our double flag logic.

A delimiter processor that requires only double length tokens could fail on single width start

- This ensures Python Markdown behavior of nested strong/em tags, remain as they were prior to this change, ensuring the the absolute minimal difference between tested cases.

facelessuser · 2026-09-12T20:53:57Z

I went searching for it, but I recalled a conversation previously with someone's PR where the the flipping of strong/em nesting came up, but I can't seem to find it.

I recalled that at the time change of behavior was not desired, so I've updated the implementation to ensure legacy behavior. We can always change it back, if we desire, but I'm going to approach this by changing as little as possible in regard to prior tests. That's the best chance of getting this in.

facelessuser · 2026-09-12T21:15:35Z

Okay, so, in the end, only the two previous tests I mentioned in the beginning changed, and we were able to add faster and more robust behavior related to nesting.

I've been running this through its paces, and I think I've fixed everything I can, unless someone else stumbles on something new.

The CommonMark punctuation handling for emphasis is still on the table, and I ran the change against our current tests; it only impacts the CommonMark tests, allowing the rest to pass. No other Python Markdown tests are impacted. I realize Python Markdown may not want these, but I just wanted to at least lay out what the impact would be in case there was interest.

I can happily apply the punctuation change if you are curious; reverting it is no big deal as well.

Altogether, being able to minimize the impact on legacy tests to two tests surprises even me.

I spent longer on this than I planned, so I'm taking a break from looking at this stuff for now, but I'm happy to answer any questions that get posted.

Ensure the cache point is after the last region start, but before or equal to the next region start so we can properly adjust the offset when returning the next region from the cache.

If the final delimiter we are comparing against is greater than 2, consume it anyway.

facelessuser · 2026-09-14T12:04:39Z

I was thinking about things more. Python Markdown, I know, doesn't generally care about CommonMark compliance, and I know Punctuation rules will be different; I imagine there may be pushback for that. So, unless I am asked to add that in, I'm going to assume it is not wanted.

Over the weekend, I was able to combine underscore and asterisk parsing into one, which I hadn't planned on doing. This allows for more sane interactions of * and _ when nested together, so I think I'll make that my last change. The Delimiter processor will allow for additional tokens to be registered/deregistered, so we can ensure the legacy underscore can easily be swapped out.

BetterEm, a 3rd party plugin, will allow for full CommonMark compliance with punctuation logic if people want that, and for Python Markdown core, we can settle on the better nesting improvements and call it a day.

DelimiterProcessor can now handle multiple delimiters at the same time allowing for more natural nesting. If the `em_strong` inline processor already exists, tokens are added to the existing processor; otherwise, a new delimiter processor is added.

facelessuser · 2026-09-14T21:20:33Z

Okay, I pushed the handling of * and _ at the same time. I'm done, unless I find more issues, for real this time. This provides the least changes possible while providing the best nesting possible, without introducing new behavior.

The only thing we've omitted from the final solution is punctuation handling, which would be new behavior, and swapping of  nesting for ***case***. Either could easily be added if we ever change our mind, but if not, we have sane, nesting with less surprises.

facelessuser · 2026-09-15T14:51:49Z

I've done an initial release of the full CommonMark implementation in BetterEm. It's exactly like this implementation except for it adds the punctuation stuff and the normalized usage of <super> nesting which we special handle and avoid here for triple cases. So I should be getting real world feedback now.

We'll backport any fixes here if any issues that impact the Python Markdown implementation are brought to light. I'm hoping we won't see anything related to basic parsing of nested non-punctuation aware emphasis and HTML element building as those would be the only things that impact us here, and I feel like I've tested this area a lot, but you never know.

waylan · 2026-09-15T15:49:03Z

the punctuation stuff

What is the punctuation stuff that you are referring to? As I'm not interested in implementing a Commonmark parser I am not intimately familiar with all aspects of the spec.

facelessuser · 2026-09-15T16:04:03Z

Yep, I've left out all punctuation rules from the Python Markdown official implementation.

As for what it is, it's just additional constraints that allows for more reasonable handling of emphasis around Unicode punctuation and symbols (example).

Again, I'm most interested in sane nesting in Python Markdown core, not specifically CommonMark compliance. So I've focused on providing that work with as little changes to default tests as possible. We aren't far off from being able to provide compliance, but not my ultimate goal here.

In the end, we only impact the two tests I pointed out in the beginning. If only that is desired, I feel we have provided a massive improvement, and it fixes a few performance issues as well.

facelessuser · 2026-09-15T17:22:25Z

I guess I'll at least post the additional rules here for anyone curious:

A left-flanking delimiter run is a delimiter run that is (1) not followed by Unicode whitespace, and either (2a) not followed by a Unicode punctuation character, or (2b) followed by a Unicode punctuation character and preceded by Unicode whitespace or a Unicode punctuation character. For purposes of this definition, the beginning and the end of the line count as Unicode whitespace.

A right-flanking delimiter run is a delimiter run that is (1) not preceded by Unicode whitespace, and either (2a) not preceded by a Unicode punctuation character, or (2b) preceded by a Unicode punctuation character and followed by Unicode whitespace or a Unicode punctuation character. For purposes of this definition, the beginning and the end of the line count as Unicode whitespace.

facelessuser · 2026-09-15T17:25:24Z

Normally, we just care about the whitespace rules.

waylan · 2026-09-15T18:42:00Z

Hmm, I thought we followed those rules (or perhaps a slight variation of them). Didn't we used to match word boundaries in our old regex which would have mostly matched that behavior? I realize that not all punctuation characters are treated the same for word boundaries, so there may be some edge cases that don't line up exactly. I wonder what tests of ours fail if that was added.

facelessuser · 2026-09-15T19:06:10Z

Our previous logic is very, very basic. We have word handling for _, but that's it. We avoid an isolated delimiters (e.g. test * test), but previously allowed odd things like:

test* test*

Which yields the following, which I personally think is not great.

<p>test<em> test</em></p>

The rewrite constrains patterns such that a start/left can't be followed by space, and an end/right can't be preceded by space. This allows us to handle complex emphasis as we can now identify a left, right, or ambiguous delimiters and make sensible decisions accordingly to avoid what was shown above.

Punctuation just extends things further, and it changes no existing tests except for the new tests added under the CommonMark test class, which would now yield proper results. I think the only tweak I added to the rules was to also give it some awareness of placeholder boundaries (STX and ETX) as those are specific to our implementation.

Tell you what, I'll push the change, and if you hate it, I'll back it out. Keep in mind that the punctuation property will resolve sometime in the future to just \p{P}\p{S} as there have been commits to start adding proper Unicode regex to the Re on CPython main. So it's just a matter of time before we are able to use proper Unicode properties.

facelessuser · 2026-09-15T19:14:25Z

I've pushed the punctuation rules. Only tests changes under CommonMark tests are observed.

That would leave the only difference between us and CommonMark being the order that ***casse*** nests strong/em, which I've currently forced to maintain the status quo.

waylan · 2026-09-15T20:27:17Z

Those are all weird edge cases that I would have never thought to create tests for. It appears that the new behavior here does not match the reference implementation. That said, I don't think we exactly matched some of the edge cases in this area anyway. Personally, I don't have an opinion on what the "correct" behavior should be. I see no point in spending effort to work on this (the punctuation stuff), but now that the work is done, that's fine, I guess.

facelessuser · 2026-09-15T20:36:53Z

Considering what we have here is essentially what I've released 3rd party (except for forcing strong/em nest order), I'm basically done here unless I get feedback on some unexpected behavior that needs to be corrected.

I'll let you decide if/when you want to move forward with this. If you change your mind about some aspects of what is present in this PR, let me know.

waylan · 2026-09-15T21:38:27Z

I'm going to sit on this for a bit to see what transpires with your release. After a reasonable period of time with no bugs, feel free to comment here to remind me to review and merge this.

waylan added the work-in-progress A partial solution. More changes will be coming. label Sep 7, 2026

facelessuser commented Sep 8, 2026

View reviewed changes

Comment thread markdown/inlinepatterns.py Outdated

facelessuser added 7 commits September 8, 2026 06:44

Remove outdated comment

59fdb1d

Consolidate reset behavior and ensure we update our tracked run

99594f2

Have other reset locations call the reset function as well

19d247f

Add return type

b82bd46

Fix link errors

dd24bcd

Add a changelog item

ee81828

Consolidate duplicate code

7f76bfb

waylan reviewed Sep 9, 2026

View reviewed changes

Comment thread markdown/core.py Outdated

facelessuser added 7 commits September 10, 2026 09:28

Separate cache info defines from reset

abe63fb

Fix lint

08d6519

Fix a possible scenario where the next region is contiguous

c2a0b0e

Rework how resetting is performed

ee8114e

Remove unused module

cf01269

Spelling

116de54

waylan reviewed Sep 10, 2026

View reviewed changes

Comment thread tests/test_syntax/inline/test_emphasis.py Outdated

Use test harness dedent

8f616ab

facelessuser added 2 commits September 11, 2026 19:45

Ignore redundant tests

1eb6be6

Remove dead code

cfab483

When building elements, we no longer need to account for regions of 3 tokens as they are always broken up into regions less than 3.

facelessuser added 3 commits September 12, 2026 10:04

A double width only delimiter processor could fail on single width token

8a8e668

A delimiter processor that requires only double length tokens could fail on single width start

Avoid flipping strong/em tag nesting for triple emphasis tokens

4f1b093

- This ensures Python Markdown behavior of nested strong/em tags, remain as they were prior to this change, ensuring the the absolute minimal difference between tested cases.

facelessuser added 2 commits September 13, 2026 06:58

Forgot to commit last fix, better cache point

c3a73b4

Ensure the cache point is after the last region start, but before or equal to the next region start so we can properly adjust the offset when returning the next region from the cache.

Another fix that missed getting committed

e7efd85

If the final delimiter we are comparing against is greater than 2, consume it anyway.

Add punctuation logic

a2ef4e1

Fix typo

54284be

Uh oh!

Conversation

facelessuser commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

AI Assistance Disclosure

Checklist

Uh oh!

facelessuser commented Sep 7, 2026

Uh oh!

waylan commented Sep 7, 2026

Uh oh!

facelessuser commented Sep 7, 2026

Uh oh!

Uh oh!

facelessuser commented Sep 8, 2026

Uh oh!

Uh oh!

facelessuser commented Sep 10, 2026

Uh oh!

facelessuser commented Sep 10, 2026

Uh oh!

Uh oh!

facelessuser commented Sep 11, 2026

Uh oh!

facelessuser commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Token chains longer than 3

Punctuation

Uh oh!

facelessuser commented Sep 12, 2026

Uh oh!

facelessuser commented Sep 12, 2026

Uh oh!

facelessuser commented Sep 12, 2026

Uh oh!

facelessuser commented Sep 12, 2026

Uh oh!

facelessuser commented Sep 14, 2026

Uh oh!

facelessuser commented Sep 14, 2026

Uh oh!

facelessuser commented Sep 15, 2026

Uh oh!

waylan commented Sep 15, 2026

Uh oh!

facelessuser commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facelessuser commented Sep 15, 2026

Uh oh!

facelessuser commented Sep 15, 2026

Uh oh!

waylan commented Sep 15, 2026

Uh oh!

facelessuser commented Sep 15, 2026

Uh oh!

facelessuser commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

waylan commented Sep 15, 2026

Uh oh!

facelessuser commented Sep 15, 2026

Uh oh!

waylan commented Sep 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

facelessuser commented Sep 7, 2026 •

edited

Loading

facelessuser commented Sep 12, 2026 •

edited

Loading

facelessuser commented Sep 15, 2026 •

edited

Loading

facelessuser commented Sep 15, 2026 •

edited

Loading