Tracking Issue for explicit-endian String::from_utf16

[CAD97]

View all comments

Feature gate: #![feature(str_from_utf16_endian)]

This is a tracking issue for versions of String::from_utf16 which take &[u8] and use a specific endianness.

Public API

impl String {
    fn from_utf16le(v: &[u8]) -> Result<String, FromUtf16Error>;
    fn from_utf16le_lossy(v: &[u8]) -> String;
    fn from_utf16be(v: &[u8]) -> Result<String, FromUtf16Error>;
    fn from_utf16be_lossy(v: &[u8]) -> String;
}

Steps / History

• Implementation: Add explicit-endian String::from_utf16 variants #95967
• Final comment period (FCP)¹
• Stabilization PR

Unresolved Questions

• Ideal naming; options include from_utf16le, from_utf16_le, from_le_utf16, from_le_utf16_bytes, and other such combinations.
• Should these methods get the with_capacity+push implementation used for from_utf16 while collect doesn't reserve capacity? (Collecting into a Result<Vec<_>> doesn't reserve the capacity in advance #48994)
• Tweaks to the error type: FromUtf16Error currently displays as "invalid utf-16: lone surrogate found" which isn't correct for an error due to odd byte length.
  • The from_utf16 method takes &[u16] so can't have the odd-length problem that &[u8] can.
  • Fixed in Clarify UTF-16 decoding errors #149696

Footnotes

1. https://std-dev-guide.rust-lang.org/feature-lifecycle/stabilization.html ↩

[zachs18]

Perhaps as an unresolved question: with these added, FromUtf16Error's Display impl is no longer always accurate; it says "invalid utf-16: lone surrogate found", but these functions introduce a new failure case: the &[u8] was of odd length. Making FromUtf16Error hold information about which kind of error occurred would require making it not a ZST anymore, which could degrade performance since currently Result<String, FromUtf16Error> is (non-guaranteed-ly) null-pointer-optimized to be the same size as String. (see below)

Alternately, they could return some new FromUtf16BytesError type which can represent both errors, so that String::from_utf16 can still return the null-pointer-optimized Result<String, FromUtf16Error>.

(Alternately, FromUtf16Error's Display impl could be updated to say something like "invalid utf-16: lone surrogate found, or odd length byte string passed".)

[CAD97]

To note, Result<String, enum { L, R }> is still niched. The data pointer is null and the other 2×usize are available to carry the Err payload. The only performance hit would be constructing or inspecting the error payload.

But that said, I also think just rendering the error as invalid utf-16 would be sufficient. Adding a new variant to the existing enum is also fine, but I don't think making a new error type is particularly helpful.

An alternative would be to panic if given an odd-length slice, since that's trivial to precheck. But not a particularly good alternative.

[eirnym]

Any news on this tracking issue? I'd love to use it

[Havunen]

Yeah, I have been using these for months without issues. This API is my last one that requires nightly version of Rust. I would love to move back to stable :)

Is there anything we can help to get this stabilized?

[Oakchris1955]

I agree with the people above, I think that we should try to get this stabilized ASAP

[Havunen]

Hi rust lang libs team, I'm asking for help how to proceed stabilizing str_from_utf16_endian feature. I believe it is ready for "Final comment period" FCP.

@rustbot ping libs-team

@rust-lang/libs-api

[rustbot]

Error: Parsing ping command in comment failed: ...' libs-team' | error: expected end of command at >| ' I'm ask'...

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

[rustbot]

Error: Only Rust team members can ping teams.

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.