Pandoc 3.4 released [LWN.net]

Version 3.4 of the Pandoc document-conversion tool has been released. Notable changes in this release include a new ANSI output format (for console output), a switch to WeasyPrint as the PDF engine for HTML to PDF conversion, the ability to position captions above or below tables and figures, and much more.

to post comments

Printing web pages to PDF

Posted Sep 11, 2024 6:20 UTC (Wed) by epa (subscriber, #39769) [Link] (19 responses)

Converting an HTML document to PDF sounds like a simple request but is hard to do programmatically. Only a full web browser has complete and well-tested support for all CSS stylings, but the browser is not designed for use as a library or a standalone command-line tool. You have to take the desktop application (with all of its dependencies for sound, GUI toolkit, SSL and a dozen others) and automate it.

That sounds like an opportunity for Mozilla. Take Firefox’s renderer (Gecko, I believe) and package it as a self-contained program without most of the dependencies needed by a browser. JavaScript support could also be left out. It would take a HTML file and write PDF, or a PNG image at the chosen size. Release that tool alongside Firefox each time Firefox is released. This would promote greater use of their renderer, which has to be a good thing for the project, as Firefox is always in danger of becoming obsolete as Chrome and its relatives become the only rendering engine people test with.

Printing web pages to PDF

Posted Sep 11, 2024 6:55 UTC (Wed) by Wol (subscriber, #4433) [Link] (4 responses)

Even a full-fledged browser is incapable of printing a web page ... I usually end up doing a "save web page", and editing it in Writer or similar to get something approximating the original ...

Cheers,
Wol

Printing web pages to PDF

Posted Sep 11, 2024 9:19 UTC (Wed) by jengelh (subscriber, #33263) [Link] (3 responses)

>Even a full-fledged browser is incapable of printing a web page

"Good parsing // Good layouting // Good printing. Pick two, you can't have all three"? ;-)

Then again, making hardcopies is falling out of favor, so I guess browsers made a sensible choice.

Printing web pages to PDF

Posted Sep 11, 2024 9:52 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

But wasn't the whole point of the web that you could read it on the output medium of your choice? Okay, I know that ship has sailed, but when the output on paper bears no resemblance whatsoever to the output on screen (and those are the main pages I want to print!!! ARGH!!!) then something's gone *badly* wrong.

Cheers,
Wol

Printing web pages to PDF

Posted Sep 12, 2024 0:18 UTC (Thu) by khim (subscriber, #9252) [Link] (1 responses)

> then something's gone *badly* wrong.

Nothing went wrong. HTML and Web was designed to deliver information, but then it got commercialized and thus something had to become a product.

You are not paying anything on most web sites thus you are the product and shouldn't complain.

If you do pay for the web site access (like with LWN) they are, usually, pretty easy to print.

Printing web pages to PDF

Posted Sep 12, 2024 6:31 UTC (Thu) by oldtomas (guest, #72579) [Link]

I know this meme.

After having watched the fireworks for nearly 35 years, I think it needs an update.

You are not the product, you are the means of production (think cows, milk).

And, BTW., you often ave no choice *even when you pay for it* (think electric cars, smart TVs, you name it).

Printing web pages to PDF

Posted Sep 11, 2024 19:29 UTC (Wed) by jond (subscriber, #37669) [Link]

It is hard: WeasyPrint is very nice though. At least for authoring documents intended to be converted to PDF; pandoc’s requirements are more broad.

Printing web pages to PDF

Posted Sep 11, 2024 19:53 UTC (Wed) by karkhaz (subscriber, #99844) [Link] (10 responses)

I believe you should be able to implement this fairly easily using headless mode in Firefox. It's used for example by brow.sh to render a web-page on a server to be browsed client-side, but you could just as well use it to open and then print a documen to PDF.

Printing web pages to PDF

Posted Sep 12, 2024 5:40 UTC (Thu) by epa (subscriber, #39769) [Link] (9 responses)

Yes, you can automate Firefox, but you need the whole of Firefox. Which might be too big a dependency, on servers where a graphical interface is not used, or on “enterprise” distros where core libraries are a few years old (for building from source). It’d be nice to build a standalone renderer that doesn’t have nearly as many prerequisites as a web browser. That would run a bit faster too.

Printing web pages to PDF

Posted Sep 12, 2024 9:51 UTC (Thu) by roc (subscriber, #30627) [Link] (8 responses)

Most likely a custom renderer is not going to run faster than a modern browser, which is optimised up the wazoo.

Just use Firefox in headless mode. It will support everything, it will be secure, and it will be fast.

Printing web pages to PDF

Posted Sep 13, 2024 6:12 UTC (Fri) by tzafrir (subscriber, #11501) [Link] (1 responses)

Optimized for the same task? For instance, a browser is optimized to show the page ASAP in a reasonable shape. Converting to PDF is a different task that can use various shortcuts you cannot use when rendering to a screen.

Printing web pages to PDF

Posted Sep 25, 2024 17:07 UTC (Wed) by sammythesnake (guest, #17693) [Link]

I haven't looked into it recently (I *used to be* a web developer) but I'm pretty sure Firefox ha[s/d] a number of tunable parameters* that affect the accommodations for "time to first render" it whatever it's called. When used in a headless context, I suspect a fair amount could be done with these to shift the balance away from low latency and toward high throughput.

* Stuff like how resources are fetched, and how much content is fetched before beginning the layout calculations. Layout often needs redoing as content continues to load (a wider cell appears in a table, the size of an image is now known etc.) but there tuneables affect how hard the rendering engine jumps the gun (hoping to guess right) vs. waiting to actually know the answer - somewhat analogous to speculative execution in a CPU...

Printing web pages to PDF

Posted Sep 13, 2024 7:09 UTC (Fri) by epa (subscriber, #39769) [Link] (5 responses)

That’s my point. Don’t use a custom renderer. Use Firefox’s renderer. But without pulling in support for SSL and 3D acceleration and JIT compilation and all the other things that make a web browser such a huge target to build from source and quite heavyweight to start up.

Printing web pages to PDF

Posted Sep 13, 2024 9:44 UTC (Fri) by james (guest, #1325) [Link] (4 responses)

HTML on disk, maybe.

But the subject of the thread is "print web pages to PDF". There you do want TLS / SSL in general: very often the HTTP site will redirect you to HTTPS and not serve the HTML you want over HTTP¹, and if not now, then it may do in the future². And some users may have governance standards that require encryption.

You might even want JIT compilation to speed up the rendering of a Javascript-heavy page if you're rendering enough of them.

Because the experience of the last 30 years, at least, is that usually you do want one tool that does everything in the problem domain (e.g. Unicode), not multiple simple tools that each do a subset (e.g. ISO 8859-*). It will come with a whole load of stuff you don't want and might despise, but it will come with everything each user wants — and you get to share debugging of the common parts.

¹ For example,
curl --proto-redir -all http://lwn.net/
shows you can't print the LWN homepage without TLS.

² Anyone else think that once all major browsers do HTTPS first, security-conscious sites are either going to shut down port 80, or just serve a very simple "Your browser is too old" page for all HTTP URIs? And this is likely to happen in the next few years?

Shutting down port 80

Posted Sep 13, 2024 10:45 UTC (Fri) by farnz (subscriber, #17727) [Link] (1 responses)

I think it's going to be a long time before security-conscious sites stop redirecting plaintext HTTP traffic to HTTPS; there's going to be a long tail of things like old links and bookmarks that are better redirected via 301/308 codes than sent to a "please upgrade" page, especially since there's no rewriting of old links without the redirect.

Shutting down port 80

Posted Sep 13, 2024 14:42 UTC (Fri) by james (guest, #1325) [Link]

Web browsers are increasingly pushing HTTPS First mode, where the browser itself will rewrite http:// to https://.

For example,

Chrome will automatically upgrade all http:// navigations to https://, even when you click on a link that explicitly declares http://. This works very similarly to HSTS upgrading, but Chrome will detect when these upgrades fail (e.g. due to a site providing an invalid certificate or returning a HTTP 404), and will automatically fallback to http://. This change ensures that Chrome only ever uses insecure HTTP when HTTPS truly isn't available, and not because you clicked on an out-of-date insecure link.

Firefox is doing something similar.

Once that's universal on current browsers, banks and the like have a financial incentive not to make things work for the few holdouts running insecure browsers.

Printing web pages to PDF

Posted Sep 13, 2024 13:34 UTC (Fri) by epa (subscriber, #39769) [Link]

You are right, it was a poor choice of title. I was thinking of only plain HTML (with CSS and images), which is the domain of weasyprint and similar programs.

Printing web pages to PDF

Posted Sep 13, 2024 13:49 UTC (Fri) by epa (subscriber, #39769) [Link]

And I entirely agree that you want one tool in the problem domain (in this case, a web browser) rather than several smaller ones. That is essentially what my idea is trying to achieve. There is one project, Firefox, and it provides a complete web browser. But that same code could also be used when you just want to convert HTML to PDF — the only difference being that it’s built to run standalone and have many fewer dependencies.

Currently we have the proliferation of different tools you want to avoid, because to generate PDF many choose a separate program like weasyprint. Not because the rendering is better than Firefox (it isn’t). But because you can’t easily get the Firefox renderer without dragging in the whole dependency tree for a full web browser (and then somewhat awkwardly automating it). So we have this fragmentation between the browser and various special-purpose tools. If Mozilla included a “convert” tool built from the same codebase and released at the same time — yet with much tighter dependencies and a smaller set of source files used — many might switch to it.

Printing web pages to PDF

Posted Sep 12, 2024 14:03 UTC (Thu) by eduperez (guest, #11232) [Link] (1 responses)

Don't forget that some parts of the page (both content and styling) are loaded or modified dynamically, using JavaScript; so you also need a JavaScript runtime.

Printing web pages to PDF

Posted Sep 13, 2024 7:10 UTC (Fri) by epa (subscriber, #39769) [Link]

The use case I had in mind was converting “HTML documents” to PDF. Not web pages in general, which as you note, may require JavaScript.

Markdown viewer

Posted Sep 11, 2024 7:12 UTC (Wed) by oliwer (subscriber, #40989) [Link] (2 responses)

The new ANSI output is perfect for reading Markdown documents from the console. The man output works pretty well too, but without color.

Markdown viewer

Posted Sep 11, 2024 9:21 UTC (Wed) by jengelh (subscriber, #33263) [Link] (1 responses)

How hard could it possibly be to convert a 24-bit color #rrggbb triplet to the 16-color palette, then emit "\m[blue]" as appropriate into the manpage?

Markdown viewer

Posted Sep 11, 2024 9:50 UTC (Wed) by Wol (subscriber, #4433) [Link]

You're forgetting something. Will the whatever-it-is reading the manpage understand \m[blue]? THAT is the problem, when you don't know what is consuming your output, you need to be very careful what you emit. As little bobby delete tables will tell you ...

Cheers,
Wol