LWN: Comments on "Some useful tools for binary formats"

Thanks

buck — Thu, 02 Mar 2023 06:47:07 +0000

Have to agree with a prior comment: Nice article!

A whole ecosystem of tools i had no knowledge of

Have enjoyed many of your past contributions as well, and appreciate their variety, clarity, deeply informative nature, and ubiquity of links. Please keep up the good work.

(Which is to take nothing away from our humble editor and the rest of the crew, who continue to faithfully pound the kernel and software development beats, with like aplomb and trademark wit)

Some useful tools for binary formats

bartoc — Wed, 01 Mar 2023 21:52:35 +0000

Yeah I was surprised this was left out, it's my goto when I need to mess around with binary formats because it's data specification mini-language is so terse but covers so much ground. It's also just really nice to be able to interactively get all kinds of different sorts of printouts. I kinda wish there was something like this "Katai Struct" program that used that instead of yaml, katai's yaml is so very verbose I almost feel like I'm not saving much time.

Actually, I think ASN.1 is basically the ultimate language for binary format description, but it's really the kitchen sink and I don't know of any parser generator implementation that can really deal with the whole specification.

Some useful tools for binary formats

iabervon — Wed, 01 Mar 2023 17:06:16 +0000

There's a similar point with libgcc: sometimes the compiler chooses to generate a call to a non-obvious implementation of a common pattern instead of a direct transformation of your source, and then the license of that implementation becomes relevant. Such things tend to be permissively licensed, so that the compiler is as widely useable as possible. On the other hand, things like regular expression libraries tend to have more runtime support and also tend to expect that a system will include the compiler and not just its output, so they don't tend to produce something that is only a derived work of the input to the compiler. So it can be a question of whether the project thinks of itself like GCC or libpcre as far as how some of the code is licensed.

Some useful tools for binary formats

sur5r — Wed, 01 Mar 2023 14:26:10 +0000

I didn't mean to undermine your work, sorry if it sounded like that. It was just that my mind connected "Cutter" and "Reverse Engineering" with radare2 and I was trying to understand if it was related or just a name collision.

Kaitai not (really) included in Fedora

pebolle — Wed, 01 Mar 2023 14:17:55 +0000

kaitai looks interesting. I wanted to play around with it on Fedora. Turns out there is only a python3-kaitaistruct package. Fedora doesn't seem to ship the ksc, the compiler.

I guess Fedora doesn't ship the compiler because Fedora last contained SBT, the Scala build tool, in Fedora 29. That's about five years ago. And SBT is mentioned as a requirement for building the compiler. So I fear playing with this in Fedora requires a lot of yak shaving, involving a language (Scala) that I know only by name.

Has anyone tried to do this nevertheless?

(It's a bit odd that Fedora ships that Python package, because it apparently only allows you to run ksc generated Python code, but not to generate kaitai code by itself. As you seem to need that compiler to do that.)

So my playing around with kaitai stopped after discovering that Fedora doesn't include the tools to do it easily. Which took me an embarrassing amount of time.

Some useful tools for binary formats

anton.kochkov — Wed, 01 Mar 2023 13:07:39 +0000

We forked multiple years ago and there is a significant difference in the code and features. We aren't "just a fork" for quite a while. I recommend checking the blog articles to get a glimpse: https://rizin.re/posts/

Some useful tools for binary formats

donaldh — Wed, 01 Mar 2023 11:24:15 +0000

The article says:

> the run-time libraries for the generated code use the MIT or Apache-2.0 license

So, yes, it's just the libraries that the generated code needs, not the generated code itself. You can presumably license the generated code any way you like.

Some useful tools for binary formats

sur5r — Wed, 01 Mar 2023 08:16:40 +0000

The "cutter" name left me confused at first, but it's indeed a fork of the radare2 stuff. Might be worth noting.

Some useful tools for binary formats

LtWorf — Wed, 01 Mar 2023 06:41:43 +0000

Nice article!

> the generated code use the MIT or Apache-2.0 license. Thus generated code can even be used in proprietary applications.

This had me confused. I was under the impression that generated code just carries the copyright of the source. Which is why gcc's output can have any license the author desires.

But I guess the situation here is that the generated code links/imports a certain library that is using that license, rather than being under that license in itself.

Some useful tools for binary formats

anton.kochkov — Wed, 01 Mar 2023 04:49:38 +0000

There is also a realm of working with the executable binary formats in UNIX-y way, not only the binary data formats. One good example is Rizin framework (and a Cutter Qt GUI to it). Apart from standard features for RE tool like disassembly, analysis, signatures, decompilation, binary diffing, it offers some of the features that are suitable for reversing data formats as well - inspecting raw data, calculating hashes, entropy inspection, magic search, diffing, and so on.

See https://rizin.re for more information. Disclaimer: I am one of the developers of the tool.