How is this advantageous over str.format? [LWN.net]

How is this advantageous over str.format?

Posted Jan 22, 2025 20:14 UTC (Wed) by siddh (subscriber, #169663)
Parent article: A revamped Python string-formatting proposal

From a cursory reading, it seems to me the only thing it has over str.format is that it detects the variable from the name, so code becomes shorter.

We anyways need to create a function to process the template, which is just equivalent to processing a format string but without the explicit argument passing (which can be just a dict).

What am I missing?

to post comments

How is this advantageous over str.format?

Posted Jan 23, 2025 13:29 UTC (Thu) by rrolls (subscriber, #151126) [Link] (12 responses)

What you're missing is the paragraph starting "The programmer can then provide any sort of template processing" and the code example following it.

f-strings (and str.format, and so on) are for producing strings directly; whoever writes the f-string or the call to str.format (say, Alice) takes responsibility for making sure every substitution is formatted (e.g. escaped) correctly. And Alice has to remember to escape every single substitution individually. There is a wealth of experience now on SQL injections, JS injections, and the like, which all ultimately result from Alice forgetting to call an escape function, or forgetting to sanitise, or misunderstanding which mechanism should be used where, or not knowing about other solutions such as parameterised SQL queries.

t-strings on the other hand are a tool that is designed to work in conjunction with library code. Someone - Bob - writes a library - LibFoo - for producing perfectly escaped HTML, or perfectly escaped SQL, or whatnot. Alice then imports LibFoo, and writes a call to it, passing a t-string with some substitutions. If an f-string, or str.format was used, then LibFoo only gets the fully formatted output string, and when it sees syntax, it can't tell whether Alice intended that to be treated as syntax, or whether it was in some user-supplied input that Alice fed to it via a substitution. But with a t-string, LibFoo knows exactly which parts were written by Alice and intended to be treated as syntax, and which parts were passed in via substitutions: it can then escape the substituted values (in the case of SQL, it can either escape them, or pull them out entirely and turn the whole thing into a parameterised query).

There's a couple of other advantages too. First, it's possible to write a t-string once and then pass it to several different library functions. Each function could do something different with it; with an f-string, that's simply not possible (you'd have to write the f-string repeatedly). One use case for this is in logging SQL queries: say you make a tool that needs to a) execute parameterised queries in a real database, and b) log the same queries for debugging, with parameters filled in in-place for human reading. Without t-strings, there are certainly ways to achieve this, but they're not simple nor do they produce particularly readable code.

Second, it becomes possible for code assist tools to detect the language that's actually being used inside the t-string. This means that if you write some HTML inside a t-string, and pass it to a function that declares (via typing) that it will process the t-string as HTML, syntax highlighting for HTML can be applied to the code inside the t-string, rather than it just being all one color; the same would work for any other language. If in future regular expressions move to using t-strings, this'll be immensely helpful: currently, some editors just basically hard-code raw strings (`r"..."`) to be rendered with regex highlighting, which is useful when you're writing a regex, but is annoying when you're using a raw string for any other reason.

I think t-strings are best seen as a way to embed any other language within Python code - without Python needing to know anything about that language. IMO, it's an incredibly powerful tool that every language should have, and yet I've never previously come across one that does, which is why I was so excited to see this PEP appear. You can put CSS and JS inside HTML, yes, but HTML, and editors and tools that work with it, are all specifically designed to support this - and then it's a similar argument for the case of putting HTML/CSS/JS inside PHP. None of these are generic - but t-strings are.

How is this advantageous over str.format?

Posted Jan 23, 2025 15:16 UTC (Thu) by siddh (subscriber, #169663) [Link] (8 responses)

What I meant was

    def html(t_str):
        final = ""
        
        for item in template:
            match item:
                case str() as s:
                    final += s
                case Interpolation() as i:
                    final += sanitise(i)

        return final

    evil = "[script]alert('evil')[script]"
    template = t"[p]{evil}[/p]"
    sanitised = html(template)

is equivalent to

    def html(given, args):
        for k, v in args.items():
            args[k] = sanitise(v)

        return given_str.format(**args)

    args = {"evil": "[script]alert('evil')[/script]"}
    template = "[p]{evil}[/p]"
    sanitised = html(template, args)

Whatever you said is equally applicable to the second case IIUC.

(Used [] instead of <> since LWN HTML comment parses it...)

How is this advantageous over str.format?

Posted Jan 23, 2025 15:56 UTC (Thu) by daroc (editor, #160859) [Link] (4 responses)

You can use HTML entities (< and >) if you want to type < and > in comments. It is a little unwieldy to do the escaping yourself, but you get used to it.

How is this advantageous over str.format?

Posted Jan 23, 2025 19:06 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

Can we get an MD mode for comments, please?

How is this advantageous over str.format?

Posted Jan 27, 2025 6:49 UTC (Mon) by rsidd (guest, #2582) [Link]

For the case of code, just working support for the PRE tag would be an improvement. > and < are generally permitted within PRE tags and verbatim code with unescaped symbols is allowed there. LWN supports PRE tags but doesn't allow these unescaped symbols.

MD mode

Posted Jan 29, 2025 10:05 UTC (Wed) by smurf (subscriber, #17840) [Link] (1 responses)

*Strongly* seconded.

But somewhat off topic here. ;-)

MD mode

Posted Jan 29, 2025 13:52 UTC (Wed) by daroc (editor, #160859) [Link]

Sorry, I probably should have replied to Cyberax — it's on my list, but our site-development time is currently being spent on a number of anti-bot measures, because we've seen a big surge in the new year.

We'll get to it at some point, though!

How is this advantageous over str.format?

Posted Jan 23, 2025 18:58 UTC (Thu) by rrolls (subscriber, #151126) [Link] (2 responses)

It doesn't look particularly exciting in these textbook cases, but the benefits will quickly show themselves once you have more complicated real-world cases.

With t-strings, you could easily have something like this:

def render_item_row(item: Item) -> HTML:
  return HTML(t"<tr><td>{item.name}</td><td>{item.date}</td><td>{item.author}</td></tr>")

Your "equivalent" would look like this:

def render_item_row(item: Item) -> HTML:
  return HTML(
    "<tr><td>{name}</td><td>{date}</td><td>{author}</td></tr>",
    {"name": item.name, "date": item.date, "author": item.author}
  )

which is unwieldy, unintuitive, and has a lot of boilerplate... or perhaps you might do

def render_item_row(item: Item) -> HTML:
  return HTML(
    "<tr><td>{name}</td><td>{date}</td><td>{author}</td></tr>",
    dataclasses.asdict(item)
  )

if Item happened to be a dataclass, but then as well as that requirement, the code doesn't make it obvious that those fields actually exist in the class.

Additionally, if you are using code assist, the 1st code block above will immediately get a red squiggly under any nonexistent field. But in both the 2nd and 3rd cases, code assist would not be able to help you if the text between { } was incorrect, or (in the 2nd case) if the dict keys were incorrect.

How is this advantageous over str.format?

Posted Jan 23, 2025 20:33 UTC (Thu) by siddh (subscriber, #169663) [Link] (1 responses)

> which is unwieldy, unintuitive, and has a lot of boilerplate...

It's not IMO. In fact for complex code you'd anyways have to create intermediary variables to make the code cleaner (what if name has to be item.original_name sometimes?).

I do agree t-strings looks cleaner, but from the passing to function POV, it just feels like new syntactic sugar to me for a function call.

Also, it's named "template" but IIUC it is just a locally bound object invalid outside of its scope, like you can't pass it around generically and have the vars substituted without wrapping in a function call which then leads to same thing.

So all-in-all the advantage just boils down to introducing the new type so str can be explicitly disallowed? For eg. sqlite3 could throw exception on using str. That will indeed force less mistakes in processing unsantised input where it's done, but may also make the already-assumed dumb programmer more careless...

But I do get your point now and appreciate it more. Thanks!

How is this advantageous over str.format?

Posted Jan 28, 2025 17:31 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

> IIUC it is just a locally bound object invalid outside of its scope, like you can't pass it around generically and have the vars substituted without wrapping in a function call which then leads to same thing.

You can indeed pass t-strings around arbitrarily, it's just that the vars are eagerly evaluated at construction time. Which is fine IMHO, because a lazy t-string would just be way too much magic for one bit of syntax (and you can always use a lambda if that's really what you wanted).

How is this advantageous over str.format?

Posted Jan 27, 2025 9:59 UTC (Mon) by epa (subscriber, #39769) [Link] (2 responses)

Agreed. This is something I have often found missing from Perl, too: how can I make a string with variable interpolation, but do my own custom interpolation?

It could bridge the gap between the safe but clunky way of creating an SQL statement with bind parameters, and the much more convenient and readable, yet dangerous, approach of pasting together an SQL string with variable interpolation. Most programming languages create too much "temptation" here. If you can say t"select * from mytable where id = {x}" and have it executed safely (which could be as simple as checking that x contains a number, not a string) then you won't have to choose between safety and comfort.

I would also use it for filenames. Writing f"/foo/bar/{leafname}" is handy but can be dangerous if leafname contains "..". You can use various libraries to build up a path from components, but again they're awkward compared to just writing out the path with interpolated parts. With t-strings, make_path(t"/foo/bar/{leafname}") could check that the value you're interpolating is a single path component, unless you explicitly permit otherwise.

How is this advantageous over str.format?

Posted Jan 28, 2025 11:47 UTC (Tue) by taladar (subscriber, #68407) [Link] (1 responses)

SQL in particular might not be the best use case, not only are there statements you can create with string interpolation that can't be handled by parameter binding (e.g. anything that changes the structure of the statement), preparing statements also has other performance benefits over ad-hoc queries.

How is this advantageous over str.format?

Posted Jan 28, 2025 17:26 UTC (Tue) by epa (subscriber, #39769) [Link]

It does often perform better to make a prepared statement and then call it several times with different values of the bind parameters. But just as often, the difference is negligible. The query may be a one-off in the lifetime of the script. But anyway, if you have a t-string based interface

t'select name from mytable where id = {myid}'

then the SQL library is free to transform that into a prepared statement which it can execute hygienically and even cache for future calls of the same t-string.