| From: |
| Alan Schmitt <alan.schmitt-AT-polytechnique.org> |
| To: |
| "lwn" <lwn-AT-lwn.net>, "cwn" <cwn-AT-lists.idyll.org> |
| Subject: |
| Attn: Development Editor, Latest OCaml Weekly News |
| Date: |
| Tue, 19 Jun 2018 15:24:21 +0200 |
| Message-ID: |
| <m24lhy9aey.fsf@charm-ecran.irisa.fr> |
Hello
Here is the latest OCaml Weekly News, for the week of June 12 to 19,
2018.
Table of Contents
─────────────────
New tool to generate the CWN
You like parmap? You like parany?
opam-check-all: Are your packages broken for the upcoming OCaml releases?
OCaml job opportunity at Citrix in Cambridge UK
Omod 0.0.1
Tools for logic programming in OCaml
opam-ci: first release of a tool to check the health of your packages
Does OCaml ecosystem have CSS preprocessors?
Old CWN
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
New tool to generate the CWN
════════════════════════════
Archive: [https://framagit.org/brab/cwn]
/The editor announced:/
Thanks to the hard work of [Enzo Crance], we now have an OCaml-based
tool to generate the OCaml weekly news. We now use [Org mode] for the
conversion to html and text. Please don't hesitate to [contact me] for
any bug report or suggestion.
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[Enzo Crance] https://github.com/cloudyhug
[Org mode] http://orgmode.org/
[contact me] mailto:alan.schmitt@polytechnique.org
You like parmap? You like parany?
═════════════════════════════════
Archive:
[https://discuss.ocaml.org/t/you-like-parmap-you-like-para...
/UnixJunkie announced:/
So, here comes parpair: [https://github.com/UnixJunkie/parpair]
The ultimate step in minimalist parallel programming.
Of course, your function needs to be pretty heavy in order for the
computation to be accelerated.
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
opam-check-all: Are your packages broken for the upcoming OCaml releases?
═════════════════════════════════════════════════════════════════════════
Archive:
[https://discuss.ocaml.org/t/ann-opam-check-all-are-your-p...
/Continuing this discussion, Kate announced:/
The service now has a proper domain name: [http://check.ocamllabs.io/]
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
OCaml job opportunity at Citrix in Cambridge UK
═══════════════════════════════════════════════
Archive:
[https://discuss.ocaml.org/t/ocaml-job-opportunity-at-citr...
/Christian Lindig announced:/
The Citrix office in Cambridge develops the [XenServer] hypervisor
which is open source and builds on [Xen]. The toolstack part of the
product, which orchestrates the VMs, is developed in OCaml (see
[GitHub]) and the team is looking for an engineer with strong affinity
to functional programming in the Unix environment. Feel free to send
me an email at [christian.lindig@citrix.com] with any questions.
Below is the official job description.
[https://jobs.citrix.com/job/CITRA005811517/Software-Engin...
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[XenServer] https://www.citrix.co.uk/products/xenserver/
[Xen] https://www.xenproject.org
[GitHub] http://www.github.com/xapi-project
[christian.lindig@citrix.com] mailto:christian.lindig@citrix.com
Omod 0.0.1
══════════
Archive:
[https://sympa.inria.fr/sympa/arc/caml-list/2018-06/msg000...
/Daniel Bünzli announced:/
I'd like to announce `omod' which provides data-driven module toplevel
loaders.
Like `ocamlfind' it allows to load compilation objects and their
dependencies in the toplevel. However `omod' works without additional
metadata files and with top level (compilation unit) module names
rather than ocamlfind package or archive names.
This means you only need to know the name of the module you want to
load rather than having to remember in which `ocamlfind' package or
archive it lives.
It should work well in `ocaml', `ocamlnat'[0] and `utop'. To get
started type:
┌────
│ opam install omod
└────
and follow the [tutorial][1]. More information on how `omod' came to
be is given below [2].
omod is ISC licensed and can be found at
[http://erratique.ch/software/omod]
Best,
Daniel
P.S. Currently `omod' doesn't work with opam system compilers.
[0] One issue with `ocamlnat' is that some libraries lack their `cmxs'
files. E.g. the lack of `threads.cmxs' prevents the `Core' module
from loading. But other ones like `Irmin', `Containers' or `Base' do
however load in `ocamlnat' via `omod'.
[1]: [http://erratique.ch/software/omod/doc/Omod.html#tutorial]
[2] This is basically a dependency-less rewrite of `odig''s toplevel
loaders which will be removed from `odig'. It solves the following
problems they had:
1. Too many dependencies. They brought too many dependencies in the
toplevel including `compiler-libs' which are problematic in
`ocamlnat'. `omod' has no dependency, so using it limits toplevel
scope pollution to the `Omod' module only which loads fine in
`ocamlnat'.
2. Resolution for multiple load sequences. `odig' had no support for
choosing between multiple load solutions (e.g. for library
variants). `omod' has both a variant specification syntax and an
interactive mode to resolve them.
3. Slow. They were too slow on large libraries as resolution was done
in the (bytecode) toplevel itself, `omod''s performance (and API)
should be improved for scripting however it is reasonably fast for
your toplevel interactive usage.
4. Outdated information in long running toplevel sessions. They
provided no mecanism to update the package information whenever the
package install base changed.
To solve 1., 3. and 4. the `Omod' API invokes the `omod' binary with
the `load' sub-command (whose textual output you can check when you
suspect there's a problem) to find out the load sequences.
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Tools for logic programming in OCaml
════════════════════════════════════
Archive:
[https://discuss.ocaml.org/t/tools-for-logic-programming-i...
/Continuing this thread from last week, th3rac said:/
Also [https://github.com/c-cube/datalog], a Datalog implementation
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
opam-ci: first release of a tool to check the health of your packages
═════════════════════════════════════════════════════════════════════
Archive:
[https://discuss.ocaml.org/t/ann-opam-ci-first-release-of-...
/Anil Madhavapeddy announced:/
/Cross posted from
[https://github.com/ocaml/infrastructure/wiki/Using-the-op...
Have you submitted a new opam package after testing it on your
desktop, and then wondered if it builds on OpenSUSE or CentOS, or on
an ARM or PowerPC architecture, or against a different version of the
OCaml compiler, or if older version continues to work after a few
years of being published? But you don't have the resources to check
all these interactions manually, especially as the opam package
database now contains thousands of revisions of OCaml source code. The
new `opam ci' plugin comes to the rescue! The remainder of this post
describes its uses, how to triage issues and then fix them.
`opam-ci' provides an interface to the opam2 continuous integration
cluster, which regularly rebuilds the full package repository across a
variety of OCaml compiler versions, operating system distributions and
CPU architectures. These builds are done regularly in remote
infrastructure and the results are pushed to a [metadata repository]
where they are fetched by a CLI client to let you query the status of
your packages.
• `opam ci status' shows a dashboard of the build results across this
matrix. Packages can be filtered by maintainer substrings or tag
names in the opam package description, so you see only those
relevant to you.
• `opam ci logs' will show you the build errors so you can fix
them. It also generates a Dockerfile of the precise build to
reproduce the environment locally for you.
To get started, try these commands with the maintainer argument
replaced with your own information or tags:
┌────
│ # show all the failing MirageOS packages
│ opam ci status -m org:mirage | less -R
│ # show all the packages maintained by anil@recoil.org
│ opam ci status -m anil@recoil.org --filter=all | less -R
│ # show all the packages failing on the latest RC of the OCaml compiler
│ opam ci status --filter=variants:rc | less -R
│ # display all failure logs for the mirage-xen package
│ opam ci logs mirage-xen
└────
The status view shows a panel of icons that represent different
combinations of ways to build opam packages. From left to right, these
are:
• /Compiler:/ The circled numbers represent OCaml compiler versions (a
circled 6 is OCaml 4.06, a circled 7 is 4.07, and so on).
• /Distro/: The square letters indicate different OS distributions. D
is Debian, F is Fedora, A is Alpine, U is Ubuntu and S is OpenSUSE.
• /CPU Architecture/: The small circled letters represent different
CPU architectures. x represents x86_64, a is arm64 and p is
PowerPC64LE.
Some compiler variants are also tested to track down specific
problems, shown by the icons to the far right of the display. (see
[below](#fix-specific-issues-around-ocaml-features) for more
information on these).
• /safe-string/: The ss icon is for 'safe-string' failures, which
would happen in OCaml 4.06 due to the switch to immutable strings.
• /flambda/: The fl icon is for packages that fail to compile with the
flambda variant of the compiler.
• /release-candidate/: The flag icon is for packages that fail to
compile with the latest release candidate of OCaml; this is useful
to figure out how much of the ecosystem works with a
soon-to-be-released compiler.
The colours indicate the result of the build: white indicates the
package was not built due to constraints, green is a successful build,
yellow indicates the build was skipped due to a dependency failure,
red is a direct build failure of that package, and magenta and blue
indicate package metadata errors such as a failure of the solver to
find a solution or the package sources being unavailable. /(note:
colors are currently mandatory but this [needs to be improved])/
[metadata repository] https://github.com/ocaml/obi-logs
[needs to be improved] https://github.com/ocaml/obi/issues/6
Uses
────
Check your own package builds
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
See all failing builds by specifying the maintainer:
┌────
│ opam ci status -m anil@recoil.org | less -R
└────
See all builds, including successes:
┌────
│ opam ci status -m anil@recoil.org -f all | less -R
└────
Check on a project's libraries
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
You can specify additional `-m' fields, which match based on the
`maintainer:' or `tags:' field in the opam metadata. For example, the
[MirageOS] uses `org:mirage' in its tags to group libraries:
┌────
│ opam ci status -m org:mirage | less -R
└────
[MirageOS] https://mirage.io
Package you have forgotten about
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Sometimes you constrain a package due to an incompatibility with a
newer version of OCaml, but then forget to release a new version.
You can query for "lagging" packages whose latest version is
incompatible with the latest release of OCaml due to constraints:
┌────
│ opam ci status -f lagging | less -R
└────
Fix specific issues around OCaml features
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
Newer releases of OCaml come with some backwards incompatible changes.
You can find packages with some of those particular problems.
◊ Migration to safe-string
OCaml shifted from mutable to immutable strings by default in OCaml
4.06, but there is a variant of the compiler with the old behaviour.
You can list packages which break with the new immutable default, but
that work with the older setting.
┌────
│ opam ci status -f variants:ss | less -R
└────
◊ Flambda inliner compilation
There is an experimental inliner available with the `flambda'
configure time variant of the compiler. A few packages fail to compile
with the new option, so you can list those explicitly to determine
what's wrong:
┌────
│ opam ci status -f variants:fl | less -R
└────
◊ Testing on trunk / release candidate OCaml
When there are release candidates for OCaml, it is helpful to test
packages on those pre-release versions. You can find packages that
compile successfully on a stable release but fail on the bleeding edge
compiler:
┌────
│ opam ci status -f variants:rc | less -R
└────
Note that this shows actual build failures. You can use the "lagging"
filter to find packages that have been constrained to prevent them
from being compiled on the latest version entirely, which is useful to
figure out what needs porting.
┌────
│ opam ci status -f lagging | less -R
└────
Community service
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
You can find all unmaintained packages that might seem some assistance
since their maintainer fields are blank:
┌────
│ opam ci status -f orphaned | less -R
└────
Identifying issues
──────────────────
Once you have identified a package that is failing, you can inspect
the build logs to figure out how to fix the issue.
Firstly, use `opam ci logs' to find out what's wrong. For example, I
might run this on the `xen-gnt-unix' package.
┌────
│ $ opam ci logs xen-gnt-unix
│ xen-gnt-unix: multiple build failures found with different configuration parameters.
│ Please refine the command to select exactly one of the following:
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=ubuntu-18.04
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=fedora-28
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06+flambda --arch=amd64 --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06+default-unsafe-string --arch=amd64
--distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.05 --arch=amd64 --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.04 --arch=amd64 --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=alpine-3.7
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=ppc64le --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=arm64 --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=opensuse-42.3
│ opam-ci logs xen-gnt-unix.3.0.1 --compiler=4.06 --arch=ppc64le --distro=debian-9
│ opam-ci logs xen-gnt-unix.3.0.1 --compiler=4.06 --arch=amd64 --distro=opensuse-42.3
└────
If just one failure is found, then the build logs are shown for that
failure. If there is more than one failure, the output will give you a
more precise command line to enter to select just one of the failures,
as shown above.
Just pick the first one, and the output shows us the abbreviated
failure log, and some metadata such as which git revision of the
[opam-repository ([https://github.com/ocaml/opam-repository]) the
package was built against.
┌────
│ opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=ubuntu-18.04
│ ====> xen-gnt-unix.3.0.0 4.06 Ubuntu 18.04 amd64 (exit code 31) (opam-repository 8425e617):
│ <snip>
│ ### output ###
│ # File "/home/opam/.opam/4.06/lib/io-page/META", line 1, characters 0-0:
│ # Error: Library "io-page-unix" not found.
│ # -> required by library "io-page.unix" in /home/opam/.opam/4.06/lib/io-page
│ # Hint: try: jbuilder external-lib-deps --missing -p xen-gnt-unix @install
└────
This then lets you hopefully see a path to fixing the issue.
Fixing issues
╌╌╌╌╌╌╌╌╌╌╌╌╌
The problems you find can range from a few root causes. If you think
of more, please add them here by updating the [wiki page] or posting
to the discussion forum.
[wiki page]
https://github.com/ocaml/infrastructure/wiki/Using-the-op...
◊ Incorrect opam package constraints
If a package used to build but subsequently starts failing, it is
probably because some dependencies have changed their interfaces. You
can use opam package constraints to fix this by identifying which the
offending dependency is from the build failure, and modifying your
package to only select
/Feature: [#5] tracks including dependency information into the
opam-ci metadata directly./
A package occasionally also becomes uninstallable due to dependency
constraints resulting in an impossible situation. This is normally
caught by the opam repository maintainers, but mistakes slip through.
In this case, you'll need to figure out the constraints that let your
package install.
/Feature: [#7] tracks including `opam lint' output in opam-ci status
to make finding these problems easier./
[#5] https://github.com/ocaml/obi/issues/5
[#7] https://github.com/ocaml/obi/issues/7
◊ OCaml compiler version
OCaml now releases on a 6-8 month release schedule, and the march of
progress occasionally breaks existing code. In this case, maintainers
can release a new version of their package that works, but older
releases are still tracked in opam and should be constrained to
prevent them being selected by the new compiler.
One example is the migration to safe-string, which broke a large
number of packages out of the box in OCaml 4.06.0. In this case, you
might see an issue like this:
┌────
│ $ opam-ci logs syslog.1.4 --compiler=4.06 --arch=amd64 --distro=debian-9
│ # File "syslog.ml", line 196, characters 50-53:
│ # Error: This expression has type bytes but an expression was expected of type
│ # string
└────
In this case, you need to prevent this package from being installed on
OCaml 4.06.0 or higher, and release a new version of the package with
the functionality fixed. The opam1 constraint looks like:
┌────
│ available: [ ocaml-version < "4.06.0" ]
└────
◊ Operating System portability
A very common situation is that you test your package on your local
desktop, but cannot try it on the huge number of Linux distributions
out there. Thanks to the magic of Docker containers, the opam-ci
shows build information on many distributions: Ubuntu, Debian, CentOS,
Alpine, OpenSUSE and OracleLinux across several versions (the full
list is [here]).
To pick on an example of `zstd', the `opam ci status zstd' shows that
it builds on Debian and Ubuntu, but fails on CentOS and Alpine.
Inspecting the opam file reveals why:
┌────
│ $ opam show zstd --raw
│ <snip>
│ depexts: [
│ ["libzstd-dev"] {os-distribution = "debian"}
│ ["libzstd-dev"] {os-distribution = "ubuntu"}
│ ]
└────
The `depexts' field is driven by [opam-depext] which understands a
large number of operating systems. You can fix this by including the
packages for Alpine and CentOS and submitting the fixes. The
opam-repository CI will test your submitted fixes against the matrix
of operating systems and verify if the fix worked or not, or you can
use [Docker locally] via the [OCaml containers](Containers) before
submitting the fix.
[here]
https://github.com/avsm/ocaml-dockerfile/blob/master/src-...
[opam-depext] https://github.com/ocaml/opam-depext/tree/2.0
[Docker locally] https://www.docker.com/docker-mac
◊ CPU portability
OCaml runs on a variety of CPU architectures, but most of us develop
on x86. The bulk builds also run regularly on arm64 and ppc64le, which
exposes portability bugs in C bindings quite often.
The AFL testing system fails on ARM for example:
┌────
│ opam-ci logs afl.2.52b --compiler=4.06 --arch=arm64 --distro=debian-9
│ ### output ###
│ # [*] Checking for the ability to compile x86 code...
│ # /tmp/cctyEiSt.s: Assembler messages:
│ # /tmp/cctyEiSt.s:10: Error: unknown mnemonic `xorb' -- `xorb %al,%al'
│ #
│ # Oops, looks like your compiler can't generate x86 code.
│ #
│ # Don't panic! You can use the LLVM or QEMU mode, but see docs/INSTALL first.
│ # (To ignore this error, set AFL_NO_X86=1 and try again.)
└────
In this case, you can fix the portability issue in your package and
release a new version. To prevent the older (already released)
packages from being selected by the opam solver on that architecture,
you can add a `cpu' constraint in the `available:' field. An example
is the `yaml.0.2.0' package that was broken on PowerPC and fixed in
`yaml.0.2.1':
┌────
│ $ opam show yaml.0.2.0 --raw
│ <snip>
│ available: arch != "ppc64"
└────
Contributions
─────────────
We would love to see more contributions to opam-ci and the associated
infrastructure. The most obvious way you can get involved is by
looking at the build failures and submitting fixes to the
[opam-repository] to help us maintain the rapidly growing package
database.
If you would like to work on the CLI tool itself, then feel free to
get started and look at the [issue list] and the [contributions
guidelines] to get started, and/or post on the [OCaml discussion
forum]. You can also add more triaging tips to this wiki to help other
users.
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[opam-repository] https://github.com/ocaml/opam-repository
[issue list] https://github.com/ocaml/obi/issues
[contributions guidelines]
https://github.com/ocaml/obi/blob/master/CONTRIBUTING.md
[OCaml discussion forum] https://discuss.ocaml.org
Does OCaml ecosystem have CSS preprocessors?
════════════════════════════════════════════
Archive:
[https://discuss.ocaml.org/t/does-ocaml-ecosystem-have-css...
/Volodymyr Melnyk asked and Nadeem Bitar replied:/
Does OCaml's ecosystem have CSS preprocessors except CCSS
(cause I want to write CSS in OCaml)?
I've used [bs-css] to write CSS in OCaml.
It allows you to write something like:
┌────
│ open CSS
│ let card = style
│ [
│ display flexBox;
│ flexDirection column;
│ boxShadow ~y:(px 3) ~blur:(px 5) (rgba 0 0 0 0 0.3);
│ ]
└────
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[bs-css] https://github.com/SentiaAnalytics/bs-css
Old CWN
═══════
If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].
If you also wish to receive it every week by mail, you may subscribe
[online].
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[Alan Schmitt]
[send me a message] mailto:alan.schmitt@polytechnique.org
[the archive] http://alan.petitepomme.net/cwn/
[RSS feed of the archives] http://alan.petitepomme.net/cwn/cwn.rss
[online] http://lists.idyll.org/listinfo/caml-news-weekly/
[Alan Schmitt] http://alan.petitepomme.net/
--
OpenPGP Key ID : 040D0A3B4ED2E5C7
Monthly Athmospheric CO₂, Mauna Loa Obs. 2018-05: 411.25, 2017-05: 409.65
