Reworking Fedora release schedules

The Fedora distribution has had a habit of missing its release targets over the years, but has also tried to target releases at certain times of the year (early May and late October). That led to a rather short development cycle for Fedora 25 as its predecessor was substantially delayed. Fedora project leader Matthew Miller recently floated an idea on the fedora-devel mailing list that might plausibly help the chronic delayed-release problem and perhaps have other beneficial effects: move Fedora to an annual release cycle. There was more to it than just that, of course, and support for the idea was mixed at best, but the conversation makes it clear that Fedora is willing to look at fairly radical changes as it moves forward.

Miller's message started by noting a graph of distinct IP addresses connecting to the Fedora mirror network for updates, which shows Fedora 25 performing quite well compared to earlier releases. The graph is not a direct measure of the number of Fedora installations, by any means, but does provide some rough measure, especially when compared to previous Fedora releases. "One week after the release, we're at about the 40k mark, and previous releases going back to F21 were at around 30k at that time."

But he also noticed something of a trend:

Annual releases?

He noted there might be public relations advantages to having one big release per year. Moreover, the Modularity and Generational Core initiatives "give us an opportunity to rethink how we are doing releases entirely". Those efforts are aimed at reworking how the distribution is put together, such that it can better support "modules" that encapsulate kind of a "unit of functionality" (e.g. a web server) with all of its dependencies. Modules may have different lifecycles than other components of the operating system, which would be built up using several different components (Base Runtime, System Runtime, and Shared Components). All of those would be collected up into a "Generational Core":

That would certainly be a sweeping change for the distribution. But Miller's thoughts go even further:

He solicited input and listed a number of features that he would like to see in any kind of plan going forward. The list included things like predictable calendar dates for releases, increasing QA time and maintaining or increasing the quality of the QA efforts, allowing the QA and release engineering teams some time to "breathe" and to work on infrastructure, maximizing user growth, and so on. In some ways, it all comes down to "not being on a hamster wheel which routinely bursts into flame".

Reaction

But Gerald B. Cox was not convinced by Miller's concern about newer releases stepping on the toes of older ones.

The recent cycle for Fedora 25 was planned for five months so it would, hopefully, land in late October, which was something that Miller pushed for. He now thinks that was a mistake and is questioning whether the frequency of Fedora releases is worth all of the effort that goes into them. But Michael Catanzaro is not so sure. He is concerned that Fedora would lose its place as the premier GNOME distribution if it moved to annual releases:

Miller is thinking that the ".1" update he proposed could incorporate even larger changes like a newer GNOME release, however. But the underlying Modularity initiative is also being questioned. Cox pointed to a comment by Kevin Kofler that was strongly critical of the idea, likening it to the "RPM hell" days. But Chris Murphy said that Modularity is not required in order to effect Miller's plan. There was some discussion of what the end-of-life strategy for the .0 and .1 pieces might be, but Miller seemed to agree that it was a workable scenario.

Another barrier to Miller's original idea is that Fedora release engineering is not prepared to handle all the parts and pieces. Dennis Gilmore pointed out that some significant work would need to be done to support it:

More concrete

All of that led Miller to propose two "more concrete" options for ways to do an annual Fedora release with one large update six months or so later. His proposal did that without "taking modularity into account in any way; it's 'how we could do this with our current distro-building process'". In both cases, Fedora 26 would be targeted for a June a release, and 26.1 (or whatever it is called) would created from that base.

The first option would batch up all of the non-security updates starting in October and test them together "in Some Handwavy Way for serious problems and regressions" before releasing the whole batch at the end of the month. The other option he suggested was to branch the release a month or two after its release; updates to Fedora 26 would be added to the 26.1 branch, which would get frozen and validated in much the same way as a regular release (though without the Alpha stage). Fedora 26 would continue on as usual until January when it would be switched over to use the 26.1 repository.

While there was some discussion of the technical parts of the plan—the second option had far more support than the first—much of the thread concerned the justification for even looking at changing the roughly six-month release cycle at all. Several participants seemed to think that the reasons for even considering the change had not been established. For example, "drago01" asked: "Which problem are you trying to solve with those proposals?"

In answer, Miller pointed back to the list of properties in his original email. But drago01 and others are skeptical that a sweeping change as he has suggested will actually preserve those properties. Peter Robinson was concerned that Miller's proposal might create more hamster wheels (flaming or otherwise) than it would actually eliminate. Adam Williamson was also unenthusiastic:

Overall, Miller's thoughts were a bit scattered, at least partly because he was posting in and around a conference he was attending. When he mentioned perhaps considering four-month cycles, Williamson was not seeing how that meshed at all:

He also wondered why Modularity and Generational Core were not part of the picture given that is the direction Fedora seems to be heading. At that point, Langdon White spoke up to try to link those two initiatives into the picture:

The problem, of course, is that the Modularity and Generational Core initiatives are still rather new and a lot is left to be done to get to the vision that White outlined. That may well mean that the 2017 releases of Fedora 26 and Fedora 27 proceed along the usual lines, with possibilities of larger changes sometime beyond that time frame. That is why Miller is trying to find a middle ground, though his justifications for that seem weak to several in the thread.

Some of these proposed changes are being driven by the new container-oriented world that is starting to change how packages and software stacks are delivered to users. Things are clearly changing in the world of distributions and it is not entirely clear what the "right" path forward is. One suspects we may see more of these kinds of discussions over time.

Comments (6 posted)

Adopting DNSSEC

The Domain Name System (DNS) is an amazing technological achievement, but it suffers from a historical excess of trust, which makes it possible for people who rely on it to be lied to. The DNS Security Extensions (formally DNSSEC-bis, more usually just DNSSEC) are a mechanism for including robust trust information within the DNS. Here we discuss briefly what DNSSEC does, how it does it, and how (and whether) you can use it to secure your domains.

The problem

DNS is the distributed system that maps domain names to IP addresses; it is one the largest distributed databases in the world. However, its design dates back to an earlier era when there was more trust around; specifically, information propagated by the DNS is often cached, while only being validated by checking the IP address of the supplier.

There are at least two problems with this. First, just because a DNS packet comes from the configured resolver is not necessarily a good reason to assume the content is trustworthy. And second, because much DNS information is propagated via UDP, which makes spoofing sender IP addresses trivial, a DNS packet claiming to be from the configured resolver's IP address may not actually come from that resolver.

The fascinating field of security research has unearthed many ingenious attacks against cryptographic protocols, such as power analysis, which are elegant and effective but unlikely to be seen in quantity in the wild anytime soon. DNS spoofing, though, is widespread; it's easy to do, and lots of people are doing it, for all sorts of reasons.

In March 2014, Turkey blocked access to Twitter after recordings damaging to the Turkish government were leaked there. The country did this by instructing Turkish ISPs to return wrong IP address information when twitter.com domains were resolved, so that DNS would instead return IP addresses that took users to a government web page announcing the block. Turks quickly discovered that setting their DNS servers to foreign recursing resolvers such as Google's would bypass the block, and some went a step further by spray-painting the IP addresses of Google's DNS servers on public buildings. Other governments engage in similar practices on a routine basis.

The solution

It's been slow in coming, but DNSSEC is now pretty much here. It enables cryptographic signatures to be distributed in the DNS alongside existing information, so that resolvers that care about such things can ask for it; they can also rely upon the answers they get. This is done by defining a few new DNS Resource Record (RR) types:

• RRSIG: This one's the workhorse; it's a signature for an RR, made with the appropriate key pair. To verify the authenticity of some RR that has just been queried, whether it was an A record, an MX, a CNAME, or some other record, the accompanying RRSIG is the starting point.
• DNSKEY: This asserts one of the keys (yes, plural, we'll come to that) for a particular zone or subzone. This is something given out to the world for your own zone(s), much like an SSH server announcing its own public key on connection.
• DS: This is how trust is distributed. It's a fingerprint for the primary DNSKEY record, generated by the zone's parent (usually, the registrar) and distributed as part of the glue. Like any other RR, it also comes with an RRSIG record, also generated by the zone's parent. This is how people can trust your DNSKEY records when you return them in response to a query. Its existence also functions as a flag that DNSSEC extensions should be checked by a capable resolver for queries in the zone in question.
• NSEC3: It is as important to sign your negative responses as your positive ones, otherwise any attacker could simply sit there returning faked NXDOMAIN responses for queries about all your hostnames. NSEC3 provably says that no such RR exists, in such a way as to cover a range of possible name queries without enabling a brute-force search of the namespace.

The chain of trust ends up with the public key that is used to sign the DNS root. This is the DNSSEC equivalent of your browser's CA bundle, and it should already be known to any resolver that intends to do DNSSEC.

Can I use it? Should I? How do I?

Not all top-level domains (TLDs) currently implement DNSSEC. Wikipedia's list of TLDs includes information on which do and do not support DNSSEC, but it's pretty widely supported, and if you're not doing registration in some obscure country-code TLD just to get a cute domain name, you probably can deploy DNSSEC.

The following shows how I enabled DNSSEC on teaparty.me.uk by following a procedure which was developed from NLNetLabs' excellent HOWTO, to whose authors I am grateful. You will need to be using a BIND implementation that is DNSSEC-capable, but I'm using BIND 9.8.2 on CentOS 6, which is pretty old, and it works fine.

First, you have to generate your zone keys. Second, you have to include your keys in your zone file and sign your zone. Finally, you have to generate your DS record, and propagate it through your registrar.

So, first off, generate the keys. They come in two types: a key-signing key (KSK), which is used only to sign other keys, and a zone-signing key (ZSK), which is used to sign all the other RRs. These do not have to be different keys, but best practice is to use a different KSK and ZSK. Key lengths and algorithm choices are a matter for you to decide. Also, the choice between /dev/random and /dev/urandom is yours to make; my server has a hardware random-number generator (RNG) attached, without which this process might have taken some time. Note also that I do this in the parent of the directory that holds the zone files themselves, as I find it convenient to keep the keys there.

    [root@lory dns]# dnssec-keygen -r/dev/random -f KSK -a RSASHA1 -b 2048 teaparty.me.uk
    Generating key pair...............................+++ ..........................+++ 
    Kteaparty.me.uk.+005+02104
    [root@lory dns]# dnssec-keygen -r/dev/random        -a RSASHA1 -b 2048 teaparty.me.uk
    Generating key pair.....+++ .................+++ 
    Kteaparty.me.uk.+005+60996

Here, the -b chooses the key length and the -a is used to choose the cryptographic algorithm to be used. The last line of the output in each case is the base name of the key files, one ending .key and the other ending in .private, which have just been created in the working directory. The filename includes the zone name, the algorithm type (005=RSA/SHA-1), and a five-digit random key identifier which helps detect key rollovers (about which more may be found here). Now, let's extract the meat, that being the new keys, from those .key files we just created:

    [root@lory dns]# grep DNSKEY *teaparty.me.uk*.key
    Kteaparty.me.uk.+005+02104.key:teaparty.me.uk. IN DNSKEY 257 3 5 AwEAAdCM/LeSga8...
    Kteaparty.me.uk.+005+60996.key:teaparty.me.uk. IN DNSKEY 256 3 5 AwEAAcQxmpGFwWw...

The keys in the above output have been truncated to avoid filling your screen with gibberish. Note the leading 256 and 257 in the DNSKEY RRs: an even number indicates a ZSK and an odd number, a KSK. Your KSK is your zone's primary key, from which we will later derive your DS record.

Next up is to put the two DNSKEY RRs into your zone file, then sign it with the keys you have just created. Signing is accomplished with the dnssec-signzone command:

    [root@lory ~]# cd /var/named/chroot/var/dns/primary/
    [root@lory primary]# /usr/sbin/dnssec-signzone -o teaparty.me.uk \
        -f teaparty.me.uk.signed -K .. -k Kteaparty.me.uk.+005+02104.key \
	-e +3024000 -N unixtime teaparty.me.uk Kteaparty.me.uk.+005+60996.key
    dnssec-signzone: warning: teaparty.me.uk:8: no TTL specified; using SOA MINTTL instead
    Verifying the zone using the following algorithms: RSASHA1.
    Zone signing complete:
    Algorithm: RSASHA1: KSKs: 1 active, 0 stand-by, 0 revoked
			ZSKs: 1 active, 0 stand-by, 0 revoked
    teaparty.me.uk.signed

First, I changed into the directory where the zone file is (in whose parent, you will recall, the keys are kept). Then I signed the zone; important flags include:

• -o specifies the zone origin (domain name); this can be inferred from the filename, but I prefer to keep it explicit.
• -f specifies the file that will contain the signed zone; I will need to alter my named.conf file so that the zone is taken from this signed file in future.
• -K specifies where to find key files.
• -k specifies the file containing the KSK.
• -e specifies the date until which I wish the signatures to remain valid. +3024000s is now + 35 days, which is designed to let me automate this signing with cron on the first of every month without the signatures ever expiring, even if the leap second people at the IERS decide to put quite a lot of extra seconds onto the end of any given month.
• -N specifies that the zone serial number (SN) in the new, signed zone file will be Unix time. I could use -N keep to preserve the SN from the input unsigned, zone file, but I prefer this.

Finally, the two arguments are the name of the zone file (which for me is simply the name of the zone), and the ZSK filename.

The last step is to generate your DS record from your KSK. This has already been done by signing the zone; you will notice in the zone file directory a file called, in my case, dsset-teaparty.me.uk.. This contains a couple of potential DS records that differ only in the digest type being used ("1" versus "2") and thus also in the digest of the key itself:

    [root@lory primary]# cat dsset-teaparty.me.uk.
    teaparty.me.uk.         IN DS 2104 5 1 1E8AB98D...
    teaparty.me.uk.         IN DS 2104 5 2 440A7FA1... 6F8640BE

This can also be done manually with the dnssec-dsfromkey command, which can be fed either the zone file or the KSK file. Although RFC 4034 says that having a DS record with digest type 1 (SHA-1) is mandatory, RFC 4509 defines the digest type 2 (SHA256) digest, and specifies that digest type 1 should be ignored if type 2 is present, in keeping with the move away from SHA-1. Most people seem to publish only the SHA-256 (digest type 2) record, and this works well for me.

So I will give the digest type 2 record to my registrar through whatever mechanism it has decided is appropriate. For registrations in .uk I do it through Nominet's website, being a registrar myself; alternatively, one would go through one's registrar. For .com/.net/.org domains, which I register through an old friend, I email the DS record to him under cover of PGP, and he passes it up the chain of registrars until it gets to someone who can put it into the proper TLD zone file.

And is it working?

You can check for DNSSEC capability in your normal resolution chain by using dig +trace. Try:

    $ dig +trace www.google.com

If you see a bunch of RRSIG and DS records being returned in the chain along with the more usual A and NS records, you're DNSSEC-capable. If your normal resolution chain isn't capable, try:

    $ dig +trace www.google.com @8.8.8.8

That will show whether your client is capable and so configured. You'll need a DNSSEC-capable resolver to go any further.

If I take a DNSSEC-capable resolver, such as Fedora 24's, using a capable resolution chain such as Google's public servers, and I ask for explicit signature checks, I should get confirmation that all is well:

    $ dig +sigchase test.teaparty.me.uk @8.8.8.8
    ...
    ;; Ok this DNSKEY is a Trusted Key, DNSSEC validation is ok: SUCCESS

If I alter the RRSIG on my A record by one byte, increment the zone's SN, reload the zone, and repeat the test, I should get indication of a problem, ending with:

    ;; Impossible to verify the Non-existence, the NSEC RRset can't be validated: FAILED 

The usual sources of error are cut-and-paste failings, publishing a DS record without a signed zone (recall that the presence of the record is the indicator that a zone should be signed), or major clock skew in your DNS server. The latter can cause problems because TTLs in DNSSEC RRs are absolute timestamps, unlike the relative TTLs of pre-DNSSEC RRs; if clocks are badly wrong, RRs that are already out-of-date can be propagated, and these will fail to validate.

So in summary, DNSSEC isn't complex; honestly, it's not. It's fiddly, to be sure; many small parts have to be assembled in the right order. But once this is done, it just works, to the net benefit of all. If you are in charge of any DNS infrastructure, and you haven't come to grips with this already, now might be a good time for you to tackle DNSSEC.

Comments (64 posted)

AMD's Display Core difficulties

Back in 2007, the announcement that AMD intended to reverse its longstanding position and create an upstream driver for its graphics processors was joyfully received by Linux users worldwide. As 2017 approaches, an attempt by AMD to merge a driver for an upcoming graphics chip has been rejected by the kernel's graphics subsystem maintainer — a decision that engendered rather less joy. A look at this discussion reveals a pattern seen many times before; the positions and decisions taken can seem arbitrary to the wider world but they are not without their reasons and will, hopefully, lead to a better kernel in the long run.

A quick timeline

Back in February, Harry Wentland posted a patch set adding the new "DAL display driver" to the AMD GPU subsystem; this driver, he said, would "allow us going forward to bring display features on the open amdgpu driver (mostly) on par with the Catalyst driver." It was not a small patch, adding 279 new source files containing nearly 94,000 lines of code. That code saw little public discussion and was never merged, though it has become clear that some private discussion took place.

In March, Alex Deucher proposed that DAL should be merged, saying that it was to be the core of the new display stack; his goal was to get it into the 4.7 release. Graphics maintainer Dave Airlie made it clear that this was not going to happen, saying that: "I think people are focusing on the minor comments and concerns and possibly deliberately ignoring the bigger concern that this code base is pretty much unmergeable as-is." His biggest complaint had to do with the overall design, which involved a lot of abstraction code that tried to hide the details of working with the kernel from the rest of the code. Others echoed his concerns and, indeed, the code was not merged for 4.7 or any other kernel released since then.

The current chapter began on December 7, when Wentland posted an RFC note saying that this code (now going by the name "display core") was needed to support the next generation GPU coming out of AMD. The company, he said, has based all of its efforts on the display core code, using it as the foundation for all of its quality-assurance work, its OEM preloads, and more. And, he noted: "DC behavior mirrors what we do for other OSes". That last point is important; the display core code helps the company maintain the driver across multiple operating systems by hiding a lot of system-specific interfaces.

This time, Daniel Vetter complained about the abstraction layers in the code and described why they were not acceptable in the mainline kernel. Airlie responded more strongly, saying that this code would not be merged in its current form:

As one might expect, a certain amount of back-and-forth resulted; the AMD developers were not pleased by this response. It can only have felt like a slap to a group of developers who were trying to do the right thing by getting support for their hardware into the mainline kernel. Even so, they stayed with the discussion, which remained almost entirely civil, and which, in the end, seems to be leading to a viable path forward.

The trouble with midlayers

There are a number of complaints with the AMD driver code; it is not often that tens of thousands of lines of new code are free of problems. But the biggest issue has to do with the midlayer architecture. A midlayer, as its name suggests, sits between two other layers of code, typically with the purpose of hiding those outer layers from each other. In this case, for example, the display core code tries to hide the details of low-level hardware access, allowing the upper-layer driver to run on any operating system.

The kernel community has a long experience with midlayers, and that experience is generally seen as being bad. For an extensive discussion of the midlayer pattern in the kernel, this 2009 article from Neil Brown is recommended reading. A midlayer architecture can bring a whole raft of long-term maintainability issues in general; the graphics developers are also concerned about some more specific issues.

The idea behind AMD's midlayer is to abstract out the Linux-specific details in the driver. That may be desirable for somebody trying to maintain a cross-platform driver; it also helps the AMD developers get the Linux driver working before the hardware engineers move on to the next product and stop answering questions. But code structured in this way is painful for people trying to maintain the Linux kernel. Understanding higher-level code becomes harder when that code does not follow the normal patterns used by graphics drivers; that can be hard for maintenance in general, but it can become a severe impediment to any sort of refactoring work. As Airlie put it:

A hardware abstraction layer must be maintained to meet the needs of code for other operating systems — code that the Linux kernel developers cannot see (and probably don't want to). In effect, that means that nobody outside of the originating company can touch the midlayer code, making community maintenance impossible. If members of the community do try to patch the midlayer — often to remove code that, from the kernel's perspective, is redundant — they will run afoul of the driver maintainers, who may well try to block the work. If they are successful in doing so, the result is code in the community kernel that is essentially off-limits for community developers.

Functionality placed in a midlayer, rather than in common code, has a high likelihood of developing its own behavioral quirks. As a result, drivers using the midlayer will behave differently from other drivers for similar hardware, often in subtle ways. That creates pain for application developers, who no longer have a single driver interface to work with.

A midlayer will also tend to isolate its developers from the common core code. The midlayer will be fixed and improved, often to work around shortcomings in the common layer, rather than improving the common layer itself. Kernel developers would rather see that effort going into the common code, where it benefits all users rather than one driver in particular. Developers who work on this support code have a say in the direction it takes, while developers who work on a midlayer generally do not. So things tend to diverge further over time, with the driver developers feeling that the core is not developed with their needs in mind.

Finally, midlayer-based code has a certain tendency to get stuck on older kernel versions; indeed, the current display core code is still based on 4.7. That makes it hard to maintain as the kernel goes forward. In this case, Vetter summarized this aspect of the problem by saying: "I think you don't just need to demidlayer DAL/DC, you also need to demidlayer your development process." Code intended for the mainline needs to be developed and tested against the current mainline, or it will inevitably fall behind.

The way forward

Regardless of how one views the odds of seeing the Year of the Linux Desktop in the near future, it seems certain that those odds can only be worse in the lack of AMD GPU drivers. The blocking of such a driver — otherwise on track to be in mainline before the hardware ships — thus looks like a step backward for a subsystem that has already struggled to gain support for current hardware.

Chances are, though, that this standoff will be resolved more quickly than people might expect. The AMD developers were not unaware of the problems and, it seems, not unwilling to fix them. Deucher said:

Some work has indeed been done since the early posting of this patch set; and, it is said, about one-third of the midlayer code is gone. Vetter made it clear that this work had been seen and appreciated:

The code that had been removed so far is, naturally enough, the easiest third to take care of; getting rid of the rest of the midlayer will require some significant refactoring of the code. Vetter provided a roadmap for how that work could be done; Wentland and AMD developer Tony Cheng agreed that the path seemed workable. Wentland acknowledged that things could have been done better at AMD, saying: "We really should've started DC development in public and probably would do that if we had to start anew." Actually getting all that work done may take a while; it must compete with other small tasks like making the driver actually work for existing and upcoming hardware.

One might conclude that what we are really seeing here is a negotiation over just how much of this work must be done before the code is deemed good enough that the rest of the fixes can be done in the mainline. Maintainers tend to worry about giving way in such situations because, once they have merged the code, they have given up their strongest leverage and can find that the developers become surprisingly unmotivated to finish the job. Arguably, that is a relatively unlikely outcome here; AMD has been trying to improve its upstream development for nearly a decade and its developers know what the rules are.

The most likely outcome, thus, is that this driver is delayed for perhaps a few more development cycles while the worst problems are taken care of and some sort of convincing story exists for the rest. Then it will be welcomed into the kernel as a welcome addition to mainline support for current hardware, and users worldwide will have something to celebrate. The Year of the Linux Desktop, unfortunately, may be a little slower in coming.

Comments (64 posted)

ModSecurity for web-application firewalls

Many web applications depend on a web-application firewall (WAF) as an important part of their security strategy. There are a few free-software options in the WAF space, but they are all limited in scope. The only multi-purpose project is ModSecurity, which has been around for almost 15 years now. In fact, ModSecurity is often the base that is used by commercial WAF offerings because of its flexibility and the granular control that it offers. But readers may not be familiar with ModSecurity, so an introduction to it is in order.

In 2006, the payment-card industry (PCI) mandated the use of a WAF in order to have a secure online offering, but did not specify a particular WAF implementation. That allowed many companies to offer WAFs of varying quality as the key component for PCI compliance. That variance gave WAFs a bad name; ModSecurity is one of the few brands that stands out from the rest.

It all started in 2002 when the original author, Ivan Ristić, wrote a tool to monitor application traffic in the Apache web server. Web servers do not give the administrator easy access to the request bodies and the responses. The same is true for reverse proxies that are put in front of these servers as gateways. So he wrote an Apache module that would provide this functionality and then developed a rule language to inspect said traffic and perform access control.

In 2006, Ristić sold ModSecurity but continued as a developer of it until 2009, when he left the project to pursue other ideas such as Hardenize. In 2010 ModSecurity ended up in the hands of Trustwave where it was re-licensed under the Apache license (instead of the original GPLv2). The new license made it more attractive to integrate ModSecurity into other products. That led to its adoption by various commercial vendors. Apache is at the core of many WAFs. In addition, there is now a ModSecurity module in those WAF boxes.

The license change also allowed companies to feel more comfortable porting the software to other web servers. Hence, Microsoft contributed an IIS port. The ModSecurity developers ported the code to NGINX, where it found a growing user community. Since ModSecurity was originally an Apache module, the ports were difficult to do and maintenance is painful. That is why a complete rewrite is underway. The idea is to develop ModSecurity into a standalone daemon that will communicate with the web server via an API. This daemon, which is in a functional alpha stage now, will eventually become libModSecurity 3.0. Meanwhile, the stable release branch stands at 2.9.1.

The rule language has always been part of ModSecurity; these days there is also a group of complementary features like schema validation of XML payloads, GeoIP lookup, support for remote blacklists, and even the injection of cross-site request forgery (CSRF or XSRF) tokens into the HTTP traffic.

Writing rules

But let's focus on the rule language, which is a domain-specific language for ModSecurity. The SecRule directive gives access to the full requests and responses via over one hundred different variables. Naturally, this covers all of the headers but it also allows access to individual arguments by parsing the payload of requests and responses as well. This feature also handles XML and JSON request content types, giving fine-grained access to individual parameters in the requests.

A variety of operators allow for inspection of those variables. The workhorse is the regular expression operator. But numerical comparison operators like @lt and @gt exist as well and a fast parallel-matching operator complements the set.

Depending on the return code of the operator, you can carry out a variety of actions. To deny access would be the most obvious one. But internal variables combined with persistent storage allow for a great many use cases that go beyond simple one-shot access control. HTTP is a stateless protocol up to version 1.1. ModSecurity, on the other hand, allows keeping track of a user session, caching expensive lookup operations, or communicating between different server threads in a straightforward way.

Let's look at a simple example of a rule:

    SecRule ARGS:surname "!@rx ^[a-z0-9._-]*$" "id:1000,deny"

This rule inspects the surname request parameter. It helps protect the service using a whitelist of allowed characters that it enforces. The @rx defines the regular expression operator. It is negated and works on the pattern defined immediately afterward. The operator will return a match if the surname parameter contains any character outside of the set listed in the square brackets. If we have a match, the deny action comes into play: it blocks the request and returns an HTTP status code of "403 Forbidden" to the client by default. The id part of the rule is an arbitrary integer identifier that is used in other rules and for logging; it is mandatory for every rule.

The example rule could be inspecting the payload of an HTTP POST request on a registration form. There are many reasons to inspect the payload arguments before passing them to the application. Registration forms usually lead to inserts into SQL databases. Maybe we are facing a situation where an SQL-injection vulnerability was discovered in that part of the application. Our little rule would thus be meant to protect the database storage. But even if no vulnerability was discovered yet, such protection can be important. Unless you are sure that the invocation of the SQL insert statement is 100% safe, you may want to add a second layer of protection. The WAF is providing this security with rules like the one in our example.

A construct such as this also allows you to operate an insecure application with a so-called virtual patch. Virtual patching is a stupid idea — it would better fix an application than trying to lock down something known to be insecure. But as we have seen, fixing applications can take a lot of time. Besides, an in-depth security approach will assume applications are always insecure so they need to be protected with an additional layer of security.

The rule language has a lot more power than what the example above would suggest. Take the following construct as a demonstration of that: it counts the number of occurrences of individual parameters in a request and denies the request if a parameter has been submitted more than once (an attack dubbed HTTP parameter pollution):

    SecRule ARGS_NAMES "@unconditionalMatch" \
	    "id:1001,\
	    pass,\
	    setvar:'TX.paramcounter_%{MATCHED_VAR_NAME}=+1'"

    SecRule TX:/paramcounter_.*/ "@gt 1" \
	    "id:1002,\
	    deny,\
	    msg:'HTTP Parameter Pollution (%{TX.1})',\
	    chain"
	    SecRule MATCHED_VARS_NAMES "TX:paramcounter_(.*)" \
		    "capture"

The first rule builds up an internal hash and counts the number of occurrences for every parameter. The second rule cycles over the hash and checks if any of the parameter has a counter that is higher than 1. If that is the case, it manipulates the parameter name to get a well formatted alert message with the help of another chained rule and the use of regular expression back references.

Whitelisting

In 2005, firewall expert Marcus Ranum insisted that security systems need to be configured with a default deny policy on all levels. If you put that idea into practice on a WAF, you write a presumably long list of SecRule constructs like the one in our first example. You define value domains for every parameter of an application. But you will also cover the HTTP request method, the permissible HTTP headers, the cookies, and the list of acceptable location paths within the request line. You describe the application traffic down to the byte-level. This is called positive security.

ModSecurity allows for such a rule set. However, the big security benefit is overshadowed by the cost of maintenance. Because of the tight rule set, every change to the application will result in requests being blocked. That is not really practical. But in real life, there is a benefit to using partial whitelists. Such a rule set only covers parts of the application where you have identified specific threats. Perhaps in the login and registration forms, for example. There is public access to these resources. This makes them easy targets and qualifies them as a good place for a partial whitelist. Once the user has authenticated, the risk is smaller and you can decide to go without the whitelisting for the rest of the application.

Still, rule writing is hard and takes some experience. Unfortunately, the online documentation of ModSecurity is not very comprehensive. It is also scattered over multiple sites and many blog posts and tutorials. This is why the ModSecurity Handbook by Ristić has achieved a quasi-official status. The book came out in 2010 and it is now showing its age. Ristić asked me to write the second edition of the handbook. I spent the whole summer on this venture and the manuscript is now complete. While the book is being finished, you can purchase an early-access online version from the publisher, Feisty Duck.

Predefined rule sets

Given the difficulty of writing rules and the disadvantages of whitelisting rules, most people do not write their own rules. They use predefined sets of blacklist rules instead. ModSecurity blacklist rules are a series of rules that try to catch exploit payloads before they hit a server. In their basic form, these lists of patterns look like a list of Snort signatures, or they resemble a long list of exploit patterns you might find in anti-virus databases. Using the blacklist rules on a web server leads to performance problems. And like Snort or an anti-virus, you are only catching known exploits. New attacks will evade the rule set; as will re-implementations of known exploits with a different fingerprint.

So this approach has not been successful. But what is more successful is generic blacklisting. That is, inspecting the traffic and looking for signs of an attack. If you parse an argument within a request and it smells like an SQL injection, then you can deny access. SQL injections come in a variety of forms, so you will need a variety of rules to catch them. The OWASP ModSecurity Core Rule Set is a collection of rules that provides this functionality. A second article will describe this rule set and the new features provided by the CRS 3.0 release.

Comments (4 posted)

Security quotes of the week

— Matthew Garrett reviews a lock on Amazon

— Tom Spring in Threatpost

Comments (3 posted)

apt: code execution

Package(s):	apt	CVE #(s):	CVE-2016-1252
Created:	December 14, 2016	Updated:	December 19, 2016
Description:	From the Debian advisory: Jann Horn of Google Project Zero discovered that APT, the high level package manager, does not properly handle errors when validating signatures on InRelease files. An attacker able to man-in-the-middle HTTP requests to an apt repository that uses InRelease files (clearsigned Release files), can take advantage of this flaw to circumvent the signature of the InRelease file, leading to arbitrary code execution.
Alerts:	Ubuntu USN-3156-2 apt 2016-12-16 Ubuntu USN-3156-1 apt 2016-12-13 Debian DSA-3733-1 apt 2016-12-13

Comments (none posted)

atomic-openshift: information disclosure

Package(s):	atomic-openshift	CVE #(s):	CVE-2016-8651
Created:	December 8, 2016	Updated:	December 14, 2016
Description:	From the Red Hat advisory: An input validation flaw was found in the way OpenShift handles requests for images. A user, with a copy of the manifest associated with an image, can pull an image even if they do not have access to the image normally, resulting in the disclosure of any information contained within the image. (CVE-2016-8651)
Alerts:	Red Hat RHSA-2016:2915-01 atomic-openshift 2016-12-07

Comments (none posted)

golang: denial of service

Package(s):	golang	CVE #(s):
Created:	December 13, 2016	Updated:	January 16, 2017
Description:	From the Red Hat bugzilla: The net/http package's Request.ParseMultipartForm method starts writing to temporary files once the request body size surpasses the given "maxMemory" limit. It was possible for an attacker to generate a multipart request crafted such that the server ran out of file descriptors.
Alerts:	Mageia MGASA-2017-0019 golang 2017-01-14 Fedora FEDORA-2016-7eea952041 golang 2016-12-17 Fedora FEDORA-2016-fad6fc78dd golang 2016-12-13 Fedora FEDORA-2016-a1563af431 golang 2016-12-13

Comments (none posted)

gst-plugins-bad: code execution

Package(s):	gst-plugins-bad0.10	CVE #(s):	CVE-2016-9809
Created:	December 8, 2016	Updated:	February 6, 2017
Description:	From the Debian-LTS advisory: An out of bounds heap bug was found in the H264 parser in gst-plugins-bad0.10.
Alerts:	openSUSE openSUSE-SU-2017:0388-1 gstreamer-0_10-plugins-bad 2017-02-04 openSUSE openSUSE-SU-2017:0291-1 gstreamer-0_10-plugins-bad 2017-01-26 openSUSE openSUSE-SU-2017:0149-1 gstreamer-plugins-bad 2017-01-16 openSUSE openSUSE-SU-2017:0152-1 gstreamer-plugins-bad 2017-01-16 openSUSE openSUSE-SU-2017:0157-1 gstreamer-0_10-plugins-bad 2017-01-16 CentOS CESA-2017:0021 gstreamer1-plugins-bad-free 2017-01-09 CentOS CESA-2017:0018 gstreamer-plugins-bad-free 2017-01-09 openSUSE openSUSE-SU-2017:0072-1 gstreamer-plugins-bad 2017-01-08 Scientific Linux SLSA-2017:0021-1 gstreamer1-plugins-bad-free 2017-01-05 Scientific Linux SLSA-2017:0018-1 gstreamer-plugins-bad-free 2017-01-05 Oracle ELSA-2017-0021 gstreamer1-plugins-bad-free 2017-01-05 Oracle ELSA-2017-0018 gstreamer-plugins-bad-free 2017-01-05 Red Hat RHSA-2017:0021-01 gstreamer1-plugins-bad-free 2017-01-05 Red Hat RHSA-2017:0018-01 gstreamer-plugins-bad-free 2017-01-05 SUSE SUSE-SU-2016:3297-1 gstreamer-plugins-bad 2016-12-30 Fedora FEDORA-2016-ca6cc3ce3e gstreamer1-plugins-bad-free 2016-12-10 Fedora FEDORA-2016-9c63b652a8 gstreamer-plugins-bad-free 2016-12-09 Debian-LTS DLA-736-1 gst-plugins-bad0.10 2016-12-08

Comments (none posted)

gstreamer-plugins-bad-free: two vulnerabilities

Package(s):	gstreamer-plugins-bad-free	CVE #(s):	CVE-2016-9812 CVE-2016-9813
Created:	December 12, 2016	Updated:	December 22, 2016
Description:	From the Red Hat bugzilla: CVE-2016-9812: An out-of-bounds heap read vulnerability was found in gst_mpegts_section_new caused by improper size checks. CVE-2016-9813: An out-of-bounds read leading to null pointer dereference vulnerability in mpegts parser in _parse_pat was found.
Alerts:	openSUSE openSUSE-SU-2017:0149-1 gstreamer-plugins-bad 2017-01-16 openSUSE openSUSE-SU-2017:0152-1 gstreamer-plugins-bad 2017-01-16 CentOS CESA-2017:0021 gstreamer1-plugins-bad-free 2017-01-09 openSUSE openSUSE-SU-2017:0072-1 gstreamer-plugins-bad 2017-01-08 Scientific Linux SLSA-2017:0021-1 gstreamer1-plugins-bad-free 2017-01-05 Oracle ELSA-2017-0021 gstreamer1-plugins-bad-free 2017-01-05 Red Hat RHSA-2017:0021-01 gstreamer1-plugins-bad-free 2017-01-05 SUSE SUSE-SU-2016:3297-1 gstreamer-plugins-bad 2016-12-30 Oracle ELSA-2016-2974 gstreamer-plugins-bad-free 2016-12-21 Fedora FEDORA-2016-ca6cc3ce3e gstreamer1-plugins-bad-free 2016-12-10 Fedora FEDORA-2016-9c63b652a8 gstreamer-plugins-bad-free 2016-12-09

Comments (none posted)

gst-plugins-base: code execution

Package(s):	gst-plugins-base0.10	CVE #(s):	CVE-2016-9811
Created:	December 8, 2016	Updated:	January 30, 2017
Description:	From the Debian-LTS advisory: An out of bounds heap read issue was found in gst-plugins-base0.10.
Alerts:	openSUSE openSUSE-SU-2017:0311-1 gstreamer-plugins-base 2017-01-28 openSUSE openSUSE-SU-2017:0325-1 gstreamer-0_10-plugins-base 2017-01-28 Mageia MGASA-2017-0021 gstreamer 2017-01-27 openSUSE openSUSE-SU-2017:0161-1 gstreamer-plugins-base 2017-01-16 openSUSE openSUSE-SU-2017:0148-1 gstreamer-plugins-base 2017-01-16 openSUSE openSUSE-SU-2017:0145-1 gstreamer-0_10-plugins-base 2017-01-16 Fedora FEDORA-2016-4fff0cbc66 gstreamer1-plugins-base 2016-12-10 Fedora FEDORA-2016-a17657197c gstreamer-plugins-base 2016-12-11 Fedora FEDORA-2016-4c8140241f gstreamer-plugins-base 2016-12-09 Debian-LTS DLA-735-1 gst-plugins-base0.10 2016-12-08

Comments (none posted)

httpd: denial of service

Package(s):	httpd	CVE #(s):	CVE-2016-8740
Created:	December 9, 2016	Updated:	January 27, 2017
Description:	From the Red Hat bugzilla entry: The HTTP/2 protocol implementation (mod_http2) had an incomplete handling of the LimitRequestFields directive. This allowed an attacker to inject unlimited request headers into the server, leading to eventual memory exhaustion.
Alerts:	openSUSE openSUSE-SU-2017:0297-1 apache2 2017-01-27 Gentoo 201701-36 apache 2017-01-15 Slackware SSA:2016-358-01 httpd 2016-12-23 Fedora FEDORA-2016-b39fedec11 httpd 2016-12-11 Fedora FEDORA-2016-260d22944d httpd 2016-12-08

Comments (none posted)

jasper: multiple vulnerabilities

Package(s):	jasper	CVE #(s):	CVE-2015-8751 CVE-2016-9262 CVE-2016-9387 CVE-2016-9388 CVE-2016-9557 CVE-2016-9560
Created:	December 9, 2016	Updated:	December 14, 2016
Description:	From the Arch Linux advisory: - CVE-2015-8751 (denial of service): An integer overflow flaw was found in the way the JasPer's library jas_matrix_create() function parsed certain JPEG 2000 image files. A specially crafted file could cause an application using JasPer to crash.- CVE-2016-2089 (denial of service) - CVE-2016-9262 (arbitrary code execution): A number of overflows were found in jasper causing use after free vulnerability triggered by a crafted image. - CVE-2016-9387 (denial of service): An integer overflow in jpc_dec_process_siz was found that can be triggered by crafted image file when given as input to imginfo. - CVE-2016-9388 (denial of service): An improper error handling was found in the RAS encoder/decoder triggering assertion tests that result in denial of service. - CVE-2016-9557 (denial of service): A signed integer overflow vulnerability has been discovered in jas_image.c triggered by a crafted image. An option max_samples has been added to the BMP and JPEG decoders to restrict the maximum size of image that they can decode. This change was made as a (possibly temporary) fix to address security concerns. - CVE-2016-9560 (arbitrary code execution): A stack buffer overflow vulnerability has been discovered in jpc/jpc_dec.c duo to an out of bounds array write triggered by a crafted image.
Alerts:	Debian DSA-3785-1 jasper 2017-02-09 openSUSE openSUSE-SU-2017:0101-1 jasper 2017-01-10 SUSE SUSE-SU-2017:0084-1 jasper 2017-01-08 Debian-LTS DLA-739-1 jasper 2016-12-10 Arch Linux ASA-201612-9 jasper 2016-12-09

Comments (none posted)

jasper: buffer overflow

Package(s):	jasper	CVE #(s):	CVE-2016-8654
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the Debian LTS advisory: Heap-based buffer overflow in QMFB code in JPC codec.
Alerts:	Debian DSA-3785-1 jasper 2017-02-09 openSUSE openSUSE-SU-2017:0101-1 jasper 2017-01-10 SUSE SUSE-SU-2017:0084-1 jasper 2017-01-08 Debian-LTS DLA-739-1 jasper 2016-12-10

Comments (none posted)

kernel: two vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2015-1350 CVE-2016-7917
Created:	December 8, 2016	Updated:	December 14, 2016
Description:	From the openSUSE advisory: CVE-2015-1350: The VFS subsystem in the Linux kernel 3.x provides an incomplete set of requirements for setattr operations that underspecifies removing extended privilege attributes, which allowed local users to cause a denial of service (capability stripping) via a failed invocation of a system call, as demonstrated by using chown to remove a capability from the ping or Wireshark dumpcap program CVE-2016-7917: The nfnetlink_rcv_batch function in net/netfilter/nfnetlink.c in the Linux kernel did not check whether a batch message's length field is large enough, which allowed local users to obtain sensitive information from kernel memory or cause a denial of service (infinite loop or out-of-bounds read) by leveraging the CAP_NET_ADMIN capability
Alerts:	SUSE SUSE-SU-2017:0494-1 the Linux Kernel 2017-02-17 openSUSE openSUSE-SU-2017:0458-1 kernel 2017-02-13 SUSE SUSE-SU-2017:0437-1 the Linux Kernel 2017-02-09 SUSE SUSE-SU-2017:0333-1 kernel 2017-01-30 SUSE SUSE-SU-2017:0181-1 kernel 2017-01-17 Debian-LTS DLA-772-1 kernel 2017-01-01 openSUSE openSUSE-SU-2016:3050-1 kernel 2016-12-08

Comments (none posted)

kernel: multiple vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2015-8962 CVE-2015-8963 CVE-2015-8964 CVE-2016-7913 CVE-2016-8646 CVE-2016-9178 CVE-2016-9794
Created:	December 8, 2016	Updated:	December 14, 2016
Description:	From the openSUSE advisory: CVE-2015-8962: Double free vulnerability in the sg_common_write function in drivers/scsi/sg.c in the Linux kernel allowed local users to gain privileges or cause a denial of service (memory corruption and system crash) by detaching a device during an SG_IO ioctl call CVE-2015-8963: Race condition in kernel/events/core.c in the Linux kernel allowed local users to gain privileges or cause a denial of service (use-after-free) by leveraging incorrect handling of an swevent data structure during a CPU unplug operation CVE-2015-8964: The tty_set_termios_ldisc function in drivers/tty/tty_ldisc.c in the Linux kernel allowed local users to obtain sensitive information from kernel memory by reading a tty data structure CVE-2016-7913: The xc2028_set_config function in drivers/media/tuners/tuner-xc2028.c in the Linux kernel allowed local users to gain privileges or cause a denial of service (use-after-free) via vectors involving omission of the firmware name from a certain data structure CVE-2016-8646: The hash_accept function in crypto/algif_hash.c in the Linux kernel allowed local users to cause a denial of service (OOPS) by attempting to trigger use of in-kernel hash algorithms for a socket that has received zero bytes of data CVE-2016-9178: The __get_user_asm_ex macro in arch/x86/include/asm/uaccess.h in the Linux kernel did not initialize a certain integer variable, which allowed local users to obtain sensitive information from kernel stack memory by triggering failure of a get_user_ex call CVE-2016-9794: A use-after-free in ALSA pcm could lead to crashes or allowed local users to potentially gain privileges
Alerts:	SUSE SUSE-SU-2017:0494-1 the Linux Kernel 2017-02-17 SUSE SUSE-SU-2017:0471-1 kernel 2017-02-15 SUSE SUSE-SU-2017:0464-1 kernel 2017-02-15 Oracle ELSA-2017-3515 kernel 3.8.13 2017-02-10 Oracle ELSA-2017-3515 kernel 3.8.13 2017-02-10 SUSE SUSE-SU-2017:0437-1 the Linux Kernel 2017-02-09 SUSE SUSE-SU-2017:0407-1 kernel 2017-02-06 Oracle ELSA-2017-3514 kernel 4.1.12 2017-02-07 Oracle ELSA-2017-3514 kernel 4.1.12 2017-02-07 SUSE SUSE-SU-2017:0333-1 kernel 2017-01-30 SUSE SUSE-SU-2017:0181-1 kernel 2017-01-17 Oracle ELSA-2017-3508 kernel 4.1.12 2017-01-12 Oracle ELSA-2017-3508 kernel 4.1.12 2017-01-12 Oracle ELSA-2017-3509 kernel 3.8.13 2017-01-12 Oracle ELSA-2017-3509 kernel 3.8.13 2017-01-12 Oracle ELSA-2017-3510 kernel 2.6.39 2017-01-12 Oracle ELSA-2017-3510 kernel 2.6.39 2017-01-12 Ubuntu USN-3167-2 linux-ti-omap4 2017-01-11 Ubuntu USN-3169-4 linux-snapdragon 2017-01-11 Ubuntu USN-3169-3 linux-raspi2 2017-01-11 Ubuntu USN-3169-2 linux-lts-xenial 2017-01-11 Ubuntu USN-3168-2 linux-lts-trusty 2017-01-11 Ubuntu USN-3167-1 kernel 2017-01-11 Ubuntu USN-3168-1 kernel 2017-01-11 Ubuntu USN-3169-1 kernel 2017-01-11 Mageia MGASA-2017-0004 kernel-tmb 2017-01-03 Mageia MGASA-2017-0003 kernel-linus 2017-01-03 Debian-LTS DLA-772-1 kernel 2017-01-01 Mageia MGASA-2016-0429 kernel, kmod 2016-12-29 SUSE SUSE-SU-2016:3248-1 the Linux Kernel 2016-12-22 SUSE SUSE-SU-2016:3252-1 the Linux Kernel 2016-12-22 SUSE SUSE-SU-2016:3217-1 the Linux Kernel 2016-12-21 Ubuntu USN-3161-4 linux-snapdragon 2016-12-20 Ubuntu USN-3161-3 linux-raspi2 2016-12-20 Ubuntu USN-3161-2 linux-lts-xenial 2016-12-20 Ubuntu USN-3161-1 kernel 2016-12-20 SUSE SUSE-SU-2016:3203-1 kernel 2016-12-20 SUSE SUSE-SU-2016:3188-1 kernel 2016-12-16 SUSE SUSE-SU-2016:3146-1 kernel 2016-12-14 openSUSE openSUSE-SU-2016:3118-1 kernel 2016-12-13 openSUSE openSUSE-SU-2016:3061-1 kernel 2016-12-08 openSUSE openSUSE-SU-2016:3050-1 kernel 2016-12-08 openSUSE openSUSE-SU-2016:3058-1 kernel 2016-12-08

Comments (none posted)

kernel: two vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2016-9756 CVE-2016-9793
Created:	December 8, 2016	Updated:	January 11, 2017
Description:	From the Mageia advisory: Linux kernel built with the Kernel-based Virtual Machine(CONFIG_KVM) support is vulnerable to an information leakage issue. It could occur on x86 platform, while emulating instructions in 32bit mode. A user/process could use this flaw to leak host kernel memory bytes (CVE-2016-9756). A bug in SO_{SND|RCV}BUFFORCE setsockopt() implementation allows CAP_NET_ADMIN users to set negative sk_sndbuf or sk_rcvbuf values. A user could use this flaw to cause various memory corruptions, crashes and OOM (CVE-2016-9793).
Alerts:	SUSE SUSE-SU-2017:0494-1 the Linux Kernel 2017-02-17 SUSE SUSE-SU-2017:0471-1 kernel 2017-02-15 SUSE SUSE-SU-2017:0464-1 kernel 2017-02-15 openSUSE openSUSE-SU-2017:0458-1 kernel 2017-02-13 openSUSE openSUSE-SU-2017:0456-1 kernel 2017-02-13 SUSE SUSE-SU-2017:0437-1 the Linux Kernel 2017-02-09 SUSE SUSE-SU-2017:0407-1 kernel 2017-02-06 SUSE SUSE-SU-2017:0333-1 kernel 2017-01-30 SUSE SUSE-SU-2017:0181-1 kernel 2017-01-17 Oracle ELSA-2017-3508 kernel 4.1.12 2017-01-12 Oracle ELSA-2017-3508 kernel 4.1.12 2017-01-12 Oracle ELSA-2017-3509 kernel 3.8.13 2017-01-12 Oracle ELSA-2017-3509 kernel 3.8.13 2017-01-12 Oracle ELSA-2017-3510 kernel 2.6.39 2017-01-12 Oracle ELSA-2017-3510 kernel 2.6.39 2017-01-12 Ubuntu USN-3167-2 linux-ti-omap4 2017-01-11 Ubuntu USN-3169-4 linux-snapdragon 2017-01-11 Ubuntu USN-3169-3 linux-raspi2 2017-01-11 Ubuntu USN-3170-2 linux-raspi2 2017-01-11 Ubuntu USN-3169-2 linux-lts-xenial 2017-01-11 Ubuntu USN-3168-2 linux-lts-trusty 2017-01-11 Ubuntu USN-3167-1 kernel 2017-01-11 Ubuntu USN-3168-1 kernel 2017-01-11 Ubuntu USN-3169-1 kernel 2017-01-11 Ubuntu USN-3170-1 kernel 2017-01-11 Mageia MGASA-2017-0004 kernel-tmb 2017-01-03 Mageia MGASA-2017-0003 kernel-linus 2017-01-03 openSUSE openSUSE-SU-2017:0002-1 kernel 2017-01-02 Debian-LTS DLA-772-1 kernel 2017-01-01 Fedora FEDORA-2016-5aff4a6bbc kernel 2016-12-13 Fedora FEDORA-2016-5cb5b4082d kernel 2016-12-13 Fedora FEDORA-2016-107f03cc00 kernel 2016-12-13 Fedora FEDORA-2016-9c17cb9648 kernel 2016-12-10 Fedora FEDORA-2016-5ec2475e3f kernel 2016-12-10 Fedora FEDORA-2016-bbe98c341c kernel 2016-12-09 Mageia MGASA-2016-0415 kernel 2016-12-08

Comments (none posted)

kernel: multiple vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2016-7910 CVE-2016-7911 CVE-2016-7914 CVE-2016-7916
Created:	December 9, 2016	Updated:	February 22, 2017
Description:	From the openSUSE advisory: - CVE-2016-7910: Use-after-free vulnerability in the disk_seqf_stop function in block/genhd.c in the Linux kernel allowed local users to gain privileges by leveraging the execution of a certain stop operation even if the corresponding start operation had failed - CVE-2016-7911: Race condition in the get_task_ioprio function in block/ioprio.c in the Linux kernel allowed local users to gain privileges or cause a denial of service (use-after-free) via a crafted ioprio_get system call - CVE-2016-7914: The assoc_array_insert_into_terminal_node function in lib/assoc_array.c in the Linux kernel did not check whether a slot is a leaf, which allowed local users to obtain sensitive information from kernel memory or cause a denial of service (invalid pointer dereference and out-of-bounds read) via an application that uses associative-array data structures, as demonstrated by the keyutils test suite - CVE-2016-7916: Race condition in the environ_read function in fs/proc/base.c in the Linux kernel allowed local users to obtain sensitive information from kernel memory by reading a /proc/*/environ file during a process-setup time interval in which environment-variable copying is incomplete
Alerts:	Ubuntu USN-3207-2 linux-lts-trusty 2017-02-22 Ubuntu USN-3206-1 linux, linux-ti-omap4 2017-02-22 Ubuntu USN-3207-1 kernel 2017-02-22 SUSE SUSE-SU-2017:0494-1 the Linux Kernel 2017-02-17 SUSE SUSE-SU-2017:0471-1 kernel 2017-02-15 SUSE SUSE-SU-2017:0464-1 kernel 2017-02-15 SUSE SUSE-SU-2017:0437-1 the Linux Kernel 2017-02-09 SUSE SUSE-SU-2017:0407-1 kernel 2017-02-06 SUSE SUSE-SU-2017:0333-1 kernel 2017-01-30 Debian-LTS DLA-772-1 kernel 2017-01-01 Ubuntu USN-3159-2 linux-ti-omap4 2016-12-20 Ubuntu USN-3160-2 linux-lts-trusty 2016-12-20 Ubuntu USN-3159-1 kernel 2016-12-20 Ubuntu USN-3160-1 kernel 2016-12-20 openSUSE openSUSE-SU-2016:3061-1 kernel 2016-12-08

Comments (none posted)

kernel: two vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2016-9777 CVE-2016-9755
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the Red Hat bugzilla: CVE-2016-9777: Linux kernel built with the Kernel-based Virtual Machine(CONFIG_KVM) support is vulnerable to an out-of-bounds memory access issue. It could occur on x86 platform, while servicing I/O APIC requests with larger vcpu_id. A guest user/process could use this flaw to crash the host kernel resulting in DoS or it could potentially be used to escalate privileges on a host. CVE-2016-9755: A GPF in network stack was reported and traced down to negative nh offset in nf_ct_frag6_queue(). Problem is that all network headers before fragment header are pulled. Normal ipv6 reassembly will drop the skb when errors occur further down the line. netfilter doesn't do this, and instead passed the original fragment along. That was also fine back when netfilter ipv6 defrag worked with cloned fragments, as the original, pristine fragment was passed on.
Alerts:	Ubuntu USN-3190-2 linux-raspi2 2017-02-09 Ubuntu USN-3190-1 kernel 2017-02-03 Fedora FEDORA-2016-9c17cb9648 kernel 2016-12-10 Fedora FEDORA-2016-5ec2475e3f kernel 2016-12-10 Fedora FEDORA-2016-bbe98c341c kernel 2016-12-09

Comments (none posted)

kernel: privilege escalation

Package(s):	kernel	CVE #(s):	CVE-2016-9576
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the openSUSE advisory: A use-after-free vulnerability in the SCSI generic driver allows users with write access to /dev/sg* or /dev/bsg* to elevate their privileges.
Alerts:	SUSE SUSE-SU-2017:0494-1 the Linux Kernel 2017-02-17 SUSE SUSE-SU-2017:0471-1 kernel 2017-02-15 SUSE SUSE-SU-2017:0464-1 kernel 2017-02-15 openSUSE openSUSE-SU-2017:0458-1 kernel 2017-02-13 SUSE SUSE-SU-2017:0437-1 the Linux Kernel 2017-02-09 SUSE SUSE-SU-2017:0407-1 kernel 2017-02-06 Oracle ELSA-2017-3514 kernel 4.1.12 2017-02-07 Oracle ELSA-2017-3514 kernel 4.1.12 2017-02-07 SUSE SUSE-SU-2017:0333-1 kernel 2017-01-30 Mageia MGASA-2017-0004 kernel-tmb 2017-01-03 Mageia MGASA-2017-0003 kernel-linus 2017-01-03 Debian-LTS DLA-772-1 kernel 2017-01-01 Mageia MGASA-2016-0429 kernel, kmod 2016-12-29 SUSE SUSE-SU-2016:3248-1 the Linux Kernel 2016-12-22 SUSE SUSE-SU-2016:3252-1 the Linux Kernel 2016-12-22 SUSE SUSE-SU-2016:3217-1 the Linux Kernel 2016-12-21 SUSE SUSE-SU-2016:3203-1 kernel 2016-12-20 SUSE SUSE-SU-2016:3188-1 kernel 2016-12-16 SUSE SUSE-SU-2016:3146-1 kernel 2016-12-14 openSUSE openSUSE-SU-2016:3118-1 kernel 2016-12-13 Fedora FEDORA-2016-5aff4a6bbc kernel 2016-12-13 Fedora FEDORA-2016-5cb5b4082d kernel 2016-12-13 Fedora FEDORA-2016-107f03cc00 kernel 2016-12-13 openSUSE openSUSE-SU-2016:3086-1 kernel 2016-12-12 openSUSE openSUSE-SU-2016:3085-1 kernel 2016-12-12

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2016-9919
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the CVE entry: The icmp6_send function in net/ipv6/icmp.c in the Linux kernel through 4.8.12 omits a certain check of the dst data structure, which allows remote attackers to cause a denial of service (panic) via a fragmented IPv6 packet.
Alerts:	openSUSE openSUSE-SU-2017:0456-1 kernel 2017-02-13 SUSE SUSE-SU-2017:0181-1 kernel 2017-01-17 Ubuntu USN-3170-2 linux-raspi2 2017-01-11 Arch Linux ASA-201612-14 linux-zen 2016-12-13 Arch Linux ASA-201612-11 linux-grsec 2016-12-10 Arch Linux ASA-201612-10 kernel 2016-12-10

Comments (none posted)

libgsf: denial of service

Package(s):	libgsf	CVE #(s):	CVE-2016-9888
Created:	December 12, 2016	Updated:	January 18, 2017
Description:	From the CVE entry: An error within the "tar_directory_for_file()" function (gsf-infile-tar.c) in GNOME Structured File Library before 1.14.41 can be exploited to trigger a Null pointer dereference and subsequently cause a crash via a crafted TAR file.
Alerts:	Fedora FEDORA-2017-21edf966b0 mingw-libgsf 2017-01-17 Mageia MGASA-2016-0427 libgsf 2016-12-29 Fedora FEDORA-2016-4871c26b3c libgsf 2016-12-16 Fedora FEDORA-2016-6ba1694d89 libgsf 2016-12-11 Debian-LTS DLA-740-1 libgsf 2016-12-11

Comments (none posted)

mozilla: multiple vulnerabilities

Package(s):	firefox	CVE #(s):	CVE-2016-9080 CVE-2016-9894 CVE-2016-9896 CVE-2016-9903
Created:	December 14, 2016	Updated:	December 14, 2016
Description:	From the Arch Linux advisory: - CVE-2016-9080 (arbitrary code execution): Mozilla developers and community members Kan-Ru Chen, Christian Holler, and Tyson Smith reported memory safety bugs present in Firefox 50.0.2. Some of these bugs showed evidence of memory corruption and we presume that with enough effort that some of these could be exploited to run arbitrary code. - CVE-2016-9894 (arbitrary code execution): A buffer overflow in SkiaGl caused when a GrGLBuffer is truncated during allocation. Later writers will overflow the buffer, resulting in a potentially exploitable crash. - CVE-2016-9896 (arbitrary code execution): Use-after-free while manipulating the navigator object within WebVR. Note: WebVR is not currently enabled by default. - CVE-2016-9903 (cross-site scripting): Mozilla's add-ons SDK had a world-accessible resource with an HTML injection vulnerability. If an additional vulnerability allowed this resource to be loaded as a document it could allow injecting content and script into an add-on's context.
Alerts:	openSUSE openSUSE-SU-2016:3310-1 firefox 2016-12-31 openSUSE openSUSE-SU-2016:3184-1 firefox 2016-12-16 Ubuntu USN-3155-1 firefox 2016-12-13 Slackware SSA:2016-348-01 firefox 2016-12-13 Arch Linux ASA-201612-15 firefox 2016-12-14

Comments (none posted)

mozilla: multiple vulnerabilities

Package(s):	firefox thunderbird	CVE #(s):	CVE-2016-9893 CVE-2016-9895 CVE-2016-9897 CVE-2016-9898 CVE-2016-9899 CVE-2016-9900 CVE-2016-9901 CVE-2016-9902 CVE-2016-9904 CVE-2016-9905
Created:	December 14, 2016	Updated:	February 3, 2017
Description:	From the Arch Linux advisory: - CVE-2016-9893 (arbitrary code execution): Mozilla developers and community members Jan de Mooij, Iris Hsiao, Christian Holler, Carsten Book, Timothy Nikkel, Christoph Diehl, Olli Pettay, Raymond Forbes, and Boris Zbarsky reported memory safety bugs present in Firefox 50.0.2 and Firefox ESR 45.5.1. Some of these bugs showed evidence of memory corruption and we presume that with enough effort that some of these could be exploited to run arbitrary code. - CVE-2016-9895 (access restriction bypass): Event handlers on marquee elements were executed despite a strict Content Security Policy (CSP) that disallowed inline JavaScript. - CVE-2016-9897 (arbitrary code execution): Memory corruption resulting in a potentially exploitable crash during WebGL functions using a vector constructor with a varying array within libGLES. - CVE-2016-9898 (arbitrary code execution): Use-after-free resulting in potentially exploitable crash when manipulating DOM subtrees in the Editor. - CVE-2016-9899 (arbitrary code execution): Use-after-free while manipulating DOM events and removing audio elements due to errors in the handling of node adoption. - CVE-2016-9900 (information disclosure): External resources that should be blocked when loaded by SVG images can bypass security restrictions through the use of data: URLs. This could allow for cross-domain data leakage. - CVE-2016-9901 (insufficient validation): HTML tags received from the Pocket server will be processed without sanitization and any JavaScript code executed will be run in the about:pocket-saved (unprivileged) page, giving it access to Pocket's messaging API through HTML injection. - CVE-2016-9902 (content spoofing): The Pocket toolbar button, once activated, listens for events fired from it's own pages but does not verify the origin of incoming events. This allows content from other origins to fire events and inject content and commands into the Pocket context. Note: this issue does not affect users with e10s enabled. - CVE-2016-9904 (information disclosure): An attacker could use a JavaScript Map/Set timing attack to determine whether an atom is used by another compartment/zone in specific contexts. This could be used to leak information, such as usernames embedded in JavaScript code, across websites.
Alerts:	Oracle ELSA-2017-0238 thunderbird 2017-02-02 Oracle ELSA-2017-0238 thunderbird 2017-02-02 Ubuntu USN-3165-1 thunderbird 2017-01-28 Slackware SSA:2017-023-01 firefox 2017-01-23 Debian-LTS DLA-782-1 icedove 2017-01-13 Debian DSA-3757-1 icedove 2017-01-11 Fedora FEDORA-2017-7af4c910c2 thunderbird 2017-01-09 Mageia MGASA-2017-0006 thunderbird 2017-01-06 Fedora FEDORA-2017-7c870ccc88 thunderbird 2017-01-06 openSUSE openSUSE-SU-2017:0026-1 thunderbird 2017-01-04 Gentoo 201701-15 firefox thunderbird 2017-01-04 Gentoo 201701-15 firefox 2017-01-03 Slackware SSA:2016-365-02 thunderbird 2016-12-30 openSUSE openSUSE-SU-2016:3308-1 thunderbird 2016-12-30 openSUSE openSUSE-SU-2016:3307-1 thunderbird 2016-12-30 openSUSE openSUSE-SU-2016:3310-1 firefox 2016-12-31 SUSE SUSE-SU-2016:3210-1 firefox 2016-12-21 SUSE SUSE-SU-2016:3222-1 firefox 2016-12-22 SUSE SUSE-SU-2016:3223-1 firefox 2016-12-22 Oracle ELSA-2016-2973 thunderbird 2016-12-21 Oracle ELSA-2016-2973 thunderbird 2016-12-21 CentOS CESA-2016:2973 thunderbird 2016-12-21 CentOS CESA-2016:2973 thunderbird 2016-12-21 Scientific Linux SLSA-2016:2973-1 thunderbird 2016-12-21 CentOS CESA-2016:2973 thunderbird 2016-12-21 Red Hat RHSA-2016:2973-01 thunderbird 2016-12-21 openSUSE openSUSE-SU-2016:3184-1 firefox 2016-12-16 CentOS CESA-2016:2946 firefox 2016-12-16 CentOS CESA-2016:2946 firefox 2016-12-16 CentOS CESA-2016:2946 firefox 2016-12-16 Mageia MGASA-2016-0420 firefox 2016-12-15 Debian-LTS DLA-743-1 firefox-esr 2016-12-15 Scientific Linux SLSA-2016:2946-1 firefox 2016-12-14 Oracle ELSA-2016-2946 firefox 2016-12-14 Oracle ELSA-2016-2946 firefox 2016-12-14 Oracle ELSA-2016-2946 firefox 2016-12-14 Ubuntu USN-3155-1 firefox 2016-12-13 Slackware SSA:2016-348-01 firefox 2016-12-13 Debian DSA-3734-1 firefox-esr 2016-12-14 Arch Linux ASA-201612-15 firefox 2016-12-14 Red Hat RHSA-2016:2946-01 firefox 2016-12-14

Comments (none posted)

openjpeg: code execution

Package(s):	openjpeg	CVE #(s):	CVE-2016-1923
Created:	December 8, 2016	Updated:	December 14, 2016
Description:	From the CVE entry: Heap-based buffer overflow in the opj_j2k_update_image_data function in OpenJpeg 2016.1.18 allows remote attackers to cause a denial of service (out-of-bounds read and application crash) via a crafted JPEG 2000 image.
Alerts:	Gentoo 201612-26 openjpeg 2016-12-08

Comments (none posted)

openjpeg2: denial of service

Package(s):	openjpeg2	CVE #(s):	CVE-2016-9573 CVE-2016-9572
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the Red Hat bugzilla: CVE-2016-9572: A NULL pointer dereference flaw was found in the way openjpeg decoded certain input images. Due to a logic error in the code responsible for decoding the input image, an application using openjpeg to process image data could crash when processing a crafted image. CVE-2016-9573: A heap buffer overflow flaw was found in the way openjpeg decompressed certain input images. Due to an insufficient check in the imagetopnm() function, an application using openjpeg to process image data could crash when processing a crafted image.
Alerts:	Debian DSA-3768-1 openjpeg2 2017-01-20 openSUSE openSUSE-SU-2017:0207-1 openjpeg2 2017-01-19 openSUSE openSUSE-SU-2017:0185-1 openjpeg2 2017-01-17 openSUSE openSUSE-SU-2017:0155-1 openjpeg2 2017-01-16 Fedora FEDORA-2016-52a1b18397 mingw-openjpeg2 2017-01-01 Fedora FEDORA-2016-89ee54c661 mingw-openjpeg2 2017-01-01 Mageia MGASA-2016-0426 openjpeg2 2016-12-29 SUSE SUSE-SU-2016:3270-1 openjpeg2 2016-12-27 Fedora FEDORA-2016-0b80dcfe5a openjpeg2 2016-12-11 Fedora FEDORA-2016-fc8577bf00 openjpeg2 2016-12-09

Comments (none posted)

openstack-cinder, openstack-glance: denial of service

Package(s):	openstack-cinder and openstack-glance	CVE #(s):	CVE-2015-5162
Created:	December 8, 2016	Updated:	February 16, 2017
Description:	From the Red Hat advisory: A resource vulnerability in the Block Storage (cinder) and Image (glance) services was found in their use of qemu-img. An unprivileged user could consume as much as 4 GB of RAM on the compute host by uploading a malicious image. This flaw could lead possibly to host out-of-memory errors and negatively affect other running tenant instances. (CVE-2015-5162)
Alerts:	Red Hat RHSA-2017:0282-01 openstack-cinder, openstack-glance, and openstack-nova 2017-02-15 Red Hat RHSA-2017:0153-01 openstack-cinder 2017-01-19 Red Hat RHSA-2017:0156-01 openstack-cinder 2017-01-19 Red Hat RHSA-2017:0165-01 openstack-cinder 2017-01-19 Red Hat RHSA-2016:2991-01 openstack-cinder/glance/nova 2016-12-22 Red Hat RHSA-2016:2923-01 openstack-cinder and openstack-glance 2016-12-07

Comments (none posted)

perl-DBD-MySQL: use after free

Package(s):	perl-DBD-MySQL	CVE #(s):	CVE-2016-1251
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the CVE entry There is a vulnerability of type use-after-free affecting DBD::mysql (aka DBD-mysql or the Database Interface (DBI) MySQL driver for Perl) 3.x and 4.x before 4.041 when used with mysql_server_prepare=1.
Alerts:	openSUSE openSUSE-SU-2017:0252-1 perl-DBD-mysql 2017-01-23 Gentoo 201701-51 DBD-mysql 2017-01-23 Fedora FEDORA-2016-bf6c3ea62c perl-DBD-MySQL 2016-12-17 openSUSE openSUSE-SU-2016:3090-1 perl-DBD-mysql 2016-12-12 Fedora FEDORA-2016-673cbb6bb4 perl-DBD-MySQL 2016-12-13 Fedora FEDORA-2016-302f840ecf perl-DBD-MySQL 2016-12-10

Comments (none posted)

php5: multiple vulnerabilities

Package(s):	php5	CVE #(s):	CVE-2016-9933 CVE-2016-9934
Created:	December 13, 2016	Updated:	January 2, 2017
Description:	Several vulnerabilities were found in PHP. See the PHP 5.6.28 changelog for more information.
Alerts:	Ubuntu USN-3196-1 php5 2017-02-14 Debian-LTS DLA-818-1 php5 2017-02-07 openSUSE openSUSE-SU-2017:0061-1 php7 2017-01-08 openSUSE openSUSE-SU-2017:0081-1 php5 2017-01-08 openSUSE openSUSE-SU-2017:0006-1 gd 2017-01-02 Debian DSA-3751-1 libgd2 2017-01-01 Mageia MGASA-2016-0422 php 2016-12-22 openSUSE openSUSE-SU-2016:3239-1 php5 2016-12-22 openSUSE openSUSE-SU-2016:3228-1 gd 2016-12-22 Debian-LTS DLA-758-1 libgd2 2016-12-22 Debian DSA-3732-2 php-ssh2 2016-12-21 Slackware SSA:2016-347-03 php 2016-12-12 Debian DSA-3732-1 php5 2016-12-13

Comments (none posted)

php: unknown

Package(s):	php	CVE #(s):	CVE-2016-9935
Created:	December 13, 2016	Updated:	December 19, 2016
Description:	One of the vulnerabilities described in the PHP 5.6.29 changelog.
Alerts:	Gentoo 201702-29 php 2017-02-21 Ubuntu USN-3196-1 php5 2017-02-14 Debian-LTS DLA-818-1 php5 2017-02-07 Arch Linux ASA-201701-28 php 2017-01-19 openSUSE openSUSE-SU-2017:0061-1 php7 2017-01-08 openSUSE openSUSE-SU-2017:0081-1 php5 2017-01-08 Mageia MGASA-2016-0422 php 2016-12-22 openSUSE openSUSE-SU-2016:3239-1 php5 2016-12-22 Debian DSA-3737-1 php5 2016-12-16 Slackware SSA:2016-347-03 php 2016-12-12

Comments (none posted)

php-php-gettext: code execution

Package(s):	php-php-gettext	CVE #(s):
Created:	December 8, 2016	Updated:	December 14, 2016
Description:	From the Red Hat bugzilla entry: A code injection vulnerability was found in php-gettext. Evaluating the plural form formula in ngettext family of calls can execute arbitrary code if number is passed unsanitized from the untrusted user.
Alerts:	Fedora FEDORA-2016-2460f713a1 php-php-gettext 2016-12-10 Fedora FEDORA-2016-077d0ed22e php-php-gettext 2016-12-11 Fedora FEDORA-2016-a571b97ebb php-php-gettext 2016-12-08

Comments (none posted)

php-simplesamlphp-saml2: incorrect signature verification

Package(s):	php-simplesamlphp-saml2	CVE #(s):
Created:	December 13, 2016	Updated:	December 14, 2016
Description:	From the SimpleSAMLphp advisory: The SAML2\Utils class offers a set of methods that allow to verify an XML digital signature against a given key. In particular, the validateSignature() method receives a signature and a key to verify it, and throws an exception in case there is any error, either caused by incorrect input or an invalid signature. This method uses the verify() method from the RobRichards\XMLSecDSig class to verify the signature with the given key, which in turn will end up calling openssl_verify() depending on the signature algorithm used.
Alerts:	Fedora FEDORA-2016-5c32bae671 php-simplesamlphp-saml2_1 2016-12-13 Fedora FEDORA-2016-b000091725 php-simplesamlphp-saml2_1 2016-12-13 Fedora FEDORA-2016-8b1f72df21 php-simplesamlphp-saml2_1 2016-12-13 Fedora FEDORA-2016-5c32bae671 php-simplesamlphp-saml2 2016-12-13 Fedora FEDORA-2016-b000091725 php-simplesamlphp-saml2 2016-12-13 Fedora FEDORA-2016-8b1f72df21 php-simplesamlphp-saml2 2016-12-13

Comments (none posted)

python-html5lib: cross-site scripting

Package(s):	python-html5lib	CVE #(s):	CVE-2016-9909 CVE-2016-9910
Created:	December 14, 2016	Updated:	January 4, 2017
Description:	From the Arch Linux advisory: - CVE-2016-9909 (cross-site scripting): A potential cross site scripting vulnerability was found in python- html5lib due to unquoted attributes that need escaping in legacy browsers. - CVE-2016-9910 (cross-site scripting): A potential cross site scripting vulnerability was found in python- html5lib due to unquoted attributes that need escaping in legacy browsers.
Alerts:	Mageia MGASA-2017-0001 python-html5lib 2017-01-03 Arch Linux ASA-201612-12 python2-html5lib 2016-12-13 Arch Linux ASA-201612-13 python-html5lib 2016-12-13

Comments (none posted)

python-XStatic-jquery-ui: cross-site scripting

Package(s):	python-XStatic-jquery-ui	CVE #(s):	CVE-2016-7103
Created:	December 9, 2016	Updated:	January 20, 2017
Description:	From the Red Hat advisory: It was found that a parameter of the dialog box feature of jQuery UI was vulnerable to cross site scripting. An attacker could use this flaw to execute a malicious script via the dialog box when it was displayed to a user. (CVE-2016-7103)
Alerts:	Red Hat RHSA-2017:0161-01 python-XStatic-jquery-ui 2017-01-20 Red Hat RHSA-2016:2932-01 python-XStatic-jquery-ui 2016-12-09 Red Hat RHSA-2016:2933-01 python-XStatic-jquery-ui 2016-12-09

Comments (none posted)

roundcube: code execution

Package(s):	roundcube	CVE #(s):	CVE-2016-9920
Created:	December 9, 2016	Updated:	December 30, 2016
Description:	From the Debian-LTS advisory: It was discovered that there was a vulnerability where a remote user could execute arbitrary commands in Roundcube, a webmail solution for IMAP servers, by sending a specially crafted email. This was due to lack of sanitisation of the arguments to PHP's "mail" function.
Alerts:	Mageia MGASA-2016-0430 roundcubemail 2016-12-30 Gentoo 201612-44 roundcube 2016-12-24 Fedora FEDORA-2016-b4896f20b3 roundcubemail 2016-12-14 Fedora FEDORA-2016-60753c3dcd roundcubemail 2016-12-13 Fedora FEDORA-2016-d361d188d9 roundcubemail 2016-12-11 Debian-LTS DLA-737-1 roundcube 2016-12-08

Comments (none posted)

spip: cross-site scripting

Package(s):	spip	CVE #(s):	CVE-2016-9152
Created:	December 9, 2016	Updated:	December 14, 2016
Description:	From the Debian-LTS advisory: It was discovered that there was a cross-site scripting (XSS) vulnerability in spip, a website publishing engine, which allowed remote attackers to inject arbitrary web script or HTML via the "rac" parameter.
Alerts:	Debian-LTS DLA-738-1 spip 2016-12-08

Comments (none posted)

sqlite: code execution

Package(s):	sqlite	CVE #(s):	CVE-2015-7036
Created:	December 8, 2016	Updated:	December 14, 2016
Description:	From the Fuzzing Project report: Passing the command ".\" will cause a one byte heap overflow in the function resolve_backslashes(). Parsing a malformed database file will cause a heap overflow of several bytes in the function sqlite3VdbeExec(). This only matters if your attack scenario involves parsing untrusted database files.
Alerts:	Gentoo 201612-21 sqlite 2016-12-08

Comments (none posted)

subversion: denial of service

Package(s):	subversion	CVE #(s):	CVE-2016-8734
Created:	December 12, 2016	Updated:	January 9, 2017
Description:	From the openSUSE bug report: Subversion's mod_dontdothat module and clients using http(s):// are vulnerable to a denial-of-service attack caused by exponential XML entity expansion. The attack, otherwise known as the "billion laughs attack", targets XML parsers and can cause the targeted process to consume an excessive amount of CPU resources or memory.
Alerts:	Mageia MGASA-2017-0009 subversion 2017-01-07 Fedora FEDORA-2017-c629f16f6c subversion 2017-01-06 openSUSE openSUSE-SU-2016:3073-1 subversion 2016-12-09

Comments (none posted)

systemd: denial of service

Package(s):	systemd	CVE #(s):	CVE-2013-4393
Created:	December 13, 2016	Updated:	December 14, 2016
Description:	From the CVE entry: journald in systemd, when the origin of native messages is set to file, allows local users to cause a denial of service (logging service blocking) via a crafted file descriptor.
Alerts:	Gentoo 201612-34 systemd 2016-12-13

Comments (none posted)

unzip: buffer overflows

Package(s):	unzip	CVE #(s):	CVE-2014-9913 CVE-2016-9844
Created:	December 13, 2016	Updated:	January 13, 2017
Description:	From the Debian LTS advisory: "unzip -l" (CVE-2014-9913) and "zipinfo" (CVE-2016-9844) were vulnerable to buffer overflows when provided malformed or maliciously-crafted ZIP files.
Alerts:	Mageia MGASA-2017-0015 unzip 2017-01-13 Fedora FEDORA-2016-80a2fba8aa unzip 2016-12-16 Fedora FEDORA-2016-3b4de2babd unzip 2016-12-16 Debian-LTS DLA-741-1 unzip 2016-12-13

Comments (none posted)

w3m: multiple vulnerabilities

Package(s):	w3m	CVE #(s):	CVE-2016-9443 CVE-2016-9621 CVE-2016-9622 CVE-2016-9623 CVE-2016-9624 CVE-2016-9625 CVE-2016-9626 CVE-2016-9627 CVE-2016-9628 CVE-2016-9629 CVE-2016-9630 CVE-2016-9631 CVE-2016-9632 CVE-2016-9633
Created:	December 14, 2016	Updated:	December 14, 2016
Description:	From the CVE entries: An issue was discovered in the Tatsuya Kinoshita w3m fork before 0.5.3-31. w3m allows remote attackers to cause a denial of service (segmentation fault and crash) via a crafted HTML page. (CVE-2016-9443, CVE-2016-9622, CVE-2016-9623, CVE-2016-9624, CVE-2016-9625, CVE-2016-9627, CVE-2016-9628, CVE-2016-9629, CVE-2016-9631, CVE-2016-9632) Duplicate of CVE-2016-9429 (CVE-2016-9621) An issue was discovered in the Tatsuya Kinoshita w3m fork before 0.5.3-33. Infinite recursion vulnerability in w3m allows remote attackers to cause a denial of service via a crafted HTML page. (CVE-2016-9626) An issue was discovered in the Tatsuya Kinoshita w3m fork before 0.5.3-33. w3m allows remote attackers to cause a denial of service (global buffer overflow and crash) via a crafted HTML page. (CVE-2016-9630) An issue was discovered in the Tatsuya Kinoshita w3m fork before 0.5.3-33. w3m allows remote attackers to cause a denial of service (infinite loop and resource consumption) via a crafted HTML page. (CVE-2016-9633)
Alerts:	Gentoo 201701-08 w3m 2017-01-01 openSUSE openSUSE-SU-2016:3121-1 w3m 2016-12-14

Comments (none posted)

webkit-gtk: multiple vulnerabilities

Package(s):	webkit-gtk	CVE #(s):	CVE-2014-4409 CVE-2014-4410 CVE-2014-4411 CVE-2014-4412 CVE-2014-4413 CVE-2014-4414 CVE-2014-4452 CVE-2014-4459 CVE-2014-4465 CVE-2014-4466 CVE-2014-4468 CVE-2014-4469 CVE-2014-4470 CVE-2014-4471 CVE-2014-4472 CVE-2014-4473 CVE-2014-4474 CVE-2014-4475 CVE-2014-4476 CVE-2014-4477 CVE-2014-4479 CVE-2015-1074 CVE-2015-1080 CVE-2015-1084 CVE-2015-5789 CVE-2015-5790 CVE-2015-5791 CVE-2015-5792 CVE-2015-5798 CVE-2015-5802 CVE-2015-5826
Created:	December 13, 2016	Updated:	December 14, 2016
Description:	From the Gentoo advisory: Multiple vulnerabilities have been discovered in WebKitGTK+. A remote attacker can use multiple vectors to execute arbitrary code or cause a Denial of Service condition.
Alerts:	Gentoo 201612-41 webkit-gtk 2016-12-13

Comments (none posted)

xen: privilege escalation

Package(s):	xen	CVE #(s):	CVE-2016-9637
Created:	December 8, 2016	Updated:	December 21, 2016
Description:	From the SUSE advisory: CVE-2016-9637: ioport array overflow allowing a malicious guest administrator can escalate their privilege to that of the host
Alerts:	Mageia MGASA-2017-0012 xen 2017-01-09 openSUSE openSUSE-SU-2017:0008-1 xen 2017-01-02 openSUSE openSUSE-SU-2017:0007-1 xen 2017-01-02 Gentoo 201612-56 xen 2016-12-31 SUSE SUSE-SU-2016:3273-1 xen 2016-12-27 Fedora FEDORA-2016-bcbae0781f xen 2016-12-23 Oracle ELSA-2016-2963 xen 2016-12-20 CentOS CESA-2016:2963 xen 2016-12-20 Scientific Linux SLSA-2016:2963-1 xen 2016-12-20 Red Hat RHSA-2016:2963-01 xen 2016-12-20 SUSE SUSE-SU-2016:3174-1 xen 2016-12-16 Fedora FEDORA-2016-cc2916dcf4 xen 2016-12-19 SUSE SUSE-SU-2016:3156-1 xen 2016-12-14 openSUSE openSUSE-SU-2016:3134-1 xen 2016-12-14 SUSE SUSE-SU-2016:3083-1 xen 2016-12-12 SUSE SUSE-SU-2016:3067-1 xen 2016-12-09 SUSE SUSE-SU-2016:3044-1 xen 2016-12-07

Comments (none posted)

xen: information disclosure

Package(s):	xen	CVE #(s):	CVE-2016-9384
Created:	December 12, 2016	Updated:	December 14, 2016
Description:	From the SUSE bug report: Along with their main kernel binary, unprivileged guests may arrange to have their Xen environment load (kernel) symbol tables for their use. The ELF image metadata created for this purpose has a few unused bytes when the symbol table binary is in 32-bit ELF format. These unused bytes were not properly cleared during symbol table loading. A malicious unprivileged guest may be able to obtain sensitive information from the host. The information leak is small and not under the control of the guest, so effectively exploiting this vulnerability is probably difficult.
Alerts:	Mageia MGASA-2017-0012 xen 2017-01-09 Gentoo 201612-56 xen 2016-12-31 openSUSE openSUSE-SU-2016:3134-1 xen 2016-12-14 SUSE SUSE-SU-2016:3067-1 xen 2016-12-09

Comments (none posted)

zabbix: two vulnerabilities

Package(s):	zabbix	CVE #(s):	CVE-2016-4338 CVE-2016-9140
Created:	December 13, 2016	Updated:	December 14, 2016
Description:	From the Gentoo advisory: Multiple vulnerabilities have been discovered in Zabbix. A remote attacker could possibly execute arbitrary code with the privileges of the process, or cause a Denial of Service condition.
Alerts:	Gentoo 201612-42 zabbix 2016-12-13

Comments (none posted)

Kernel release status

The 4.9 kernel is out, released on December 11. Some of the headline features in 4.9 include improved security with virtually mapped kernel stacks, the memory-protection keys system calls, the BBR congestion-control algorithm, support for the Greybus bus architecture, shared extents in the XFS filesystem (which will be used to support lightweight copy operations among other things), and much more. The code name has also been changed to "Roaring Lionus". In the end, 16,216 non-merge changesets were pulled for the 4.9 release, making this development cycle the busiest ever by far.

The 4.10 merge window is open; see the separate article below for a summary of what has been pulled into the the mainline thus far.

Stable updates: 4.8.13 and 4.4.37 were released on December 8, followed by 4.8.14 and 4.4.38 on December 10. The 4.8.15 and 4.4.39 updates are in the review process as of this writing; they can be expected on or after December 15.

Comments (none posted)

Quotes of the week

— Jani Nikula

— Paul McKenney

Comments (none posted)

4.10 Merge window part 1

When Linus Torvalds released the 4.9 kernel on December 11, he repeated his warning that the 4.10 merge window would be shorter than usual to accommodate the holidays. Maintainers were advised to get their pull requests in early, and many have gotten the word. As of this writing, three days into the 4.10 merge window, nearly 6,900 non-merge changesets have been pulled into the mainline repository, a pace that seems likely to continue for a while.

Some of the more interesting user-visible changes merged so far are:

• The writeback throttling patches have been merged; this should help to keep systems more responsive in the face of heavy background writeback activity.

• The arm64 architecture has gained support for user-space probes (uprobes). Also, arm64 can now emulate the "privileged access never" feature, keeping the kernel from accessing user-space memory in unintended ways.

• The networking layer now supports per-UID network packet routing, a feature brought in from the Android world, where it is used to configure per-app routing policies.

• Support for IPv6 segment routing, a source-routing mechanism, has been added to the networking code.

• There is a new BPF map type that implements a least-recently-used (LRU) list. Said new map type does not, unfortunately, come with any new documentation describing how it works. The curious can glean some information from this changelog and the test code.

• It is now possible to attach a BPF program to a control group which then performs network filtering for all processes within that group.

• BPF programs can be attached to the networking lightweight tunnel infrastructure; these programs can make observations about passing traffic or modify that traffic on the fly. The sample code shows how this facility can be used.

• The netfilter subsystem has gained support for "stateful objects", counters and quotas in particular. See this post for some information on this feature.

• On EFI-firmware systems, the kernel can now store some random data as a firmware variable, then use it at boot time to initialize the random-number generator. This allows entropy to be carried over from one boot to the next and should make for stronger randomness immediately after the system boots.

• The perf events subsystem and user-space tooling have seen a lot of changes this time around; see this commit message for a list.

• The "mediated device" mechanism allows software-defined devices to be made available via the VFIO interface. It is intended to facilitate making devices available to virtualized guests. See Documentation/vfio-mediated-device.txt for details.

• The user-space LED driver subsystem has been merged; see Documentation/leds/uleds.txt and this article for more information.

• New hardware support includes:
  • Processors and systems: Intel CPUs with "Turbo Boost Max Technology".

  • Clocks: Mediatek MT2701 clocks, Qualcomm MSM8994 global clock controllers, Qualcomm MSM8994 pin controllers, Qualcomm RPM-based clock controllers, HiSilicon Hi3798CV200 and HI3516CV300 clock controllers, Allwinner A64 clock controllers, and Renesas r8a7743 and r8a7745 clock pulse generators.

  • Graphics: Silicon Image SII8620 HDMI/MHL bridges, TI TFP410 DVI/HDMI bridges, ZTE VOU display controllers, Hisilicon Hibmc GPUs, NXP i.MX23/i.MX28/i.MX6SX MXSFB LCD controllers, and Amlogic Meson display controllers.

  • Industrial I/O: Analog Devices AD7766/AD7767 analog-to-digital controllers (ADCs), ST Microelectronics STM32 ADCs, Texas Instruments LMP91000 potentiostats, Texas Instruments TMP108 temperature sensors, Honeywell ABP pressure sensors, ACCES 104-QUAD-8 quadrature encoders, Domintech ARD10 3-axis accelerometers, MiraMEMS DA311 and DA280 digital accelerometers, STMicroelectronics HTS221 temperature-humidity sensors, and Invensense MPU-3050 gyroscopes.

  • Miscellaneous: ST FDMA DMA engines, Altera Arria10 system resource GPIO controllers, NI PXI NIC78bx LED controllers, Amlogic S905/GX* SD/MMC host controllers, Cadence SD/SDIO/eMMC host controllers, Qualcomm ADSP peripheral image loaders, Synopsis Designware I2S audio interfaces, Analog Devices ADV7511 HDMI Audio interfaces, MediaTek USB3 dual role USB controllers, NXP LPC18XX one-time programmable memory, Broadcom one-time programmable memory controllers, Holtek Ht16K33 LED controllers, Altera SoCFPGA Arria10 FPGA programmers, Fintek F81532/534 USB-to-serial adapters, Marvell Armada 3700 SPI controllers, Freescale i.MX LPSPI controllers, and Microchip TC654/TC655 fan controllers.

  • Networking: Mellanox Technologies SwitchIB and SwitchIB-2 Infiniband switches, Oxford Semiconductor OXNAS dwmac Ethernet controllers, Amlogic Meson GXL Internal PHYs, QorIQ DPAA Ethernet controllers, and Alacritech Slicoss gigabit Ethernet cards.

Changes visible to kernel developers include:

• The new CONFIG_BUG_ON_DATA_CORRUPTION config variable will cause the kernel to oops when data corruption is detected; it is currently applied to the linked-list implementation.

• The low-level mutex implementation has been rewritten to get rid of most of the architecture-specific code.

• The CPU-idle notifier has been deleted after the final user was removed from the kernel.

• All existing CPU hotplug notifiers have been converted to the new state-machine architecture and the notifier mechanism itself is set to be removed by the end of the merge window.

• The configuration mechanism recognizes a new imply keyword. It functions like select except that the named option can still be turned off manually if desired.

• The new vfree_atomic() function frees memory like vfree(); that freeing is deferred to a workqueue, though, so this function can be called in atomic context.

• There is a new mechanism for tracking functional dependencies between otherwise unrelated devices; see the documentation for details.

• The s390 architecture can now make use of the contiguous memory allocator subsystem.

The 4.10 merge window would normally be expected to close on December 25, but it is clear that the merging of patches will stop by the 23rd at the latest. There are a number of important repositories yet to be pulled, so we can expect there to be a fair number of changes merged before that date. As usual, LWN will continue to follow the merge window through the end.

Comments (2 posted)

A way forward for BFQ

The BFQ ("budget fair queuing") I/O scheduler has seen several discussions on the kernel mailing lists since its debut in 2014. By changing the scheduling algorithm and adding a number of heuristics, BFQ is able to provide much more responsive block I/O behavior, especially when dealing with rotational devices. Despite its merits, though, BFQ has never found its way into the mainline. Now, however, it looks like the primary roadblock is about to be removed.

The objections to BFQ have varied over the years. Initially it was proposed as a new scheduler to be added to the four that the kernel already supports; at that time, BFQ developer Paolo Valente was told that the BFQ functionality should be added to the existing CFQ scheduler instead. In February 2016, he came back with a patch set that attempted to do that, but those patches were (necessarily) intrusive and did not get far. Among other things, reviewers complained about replacing the well-tested CFQ scheduler. The bigger complaint, though, which was echoed after the most recent posting in October, is that BFQ uses the legacy kernel block API rather than the newer multiqueue API.

The multiqueue interface was added in response to scalability issues in the block layer; it allows I/O request queues to be split into multiple subqueues that can be handled on a per-CPU basis. It is seen as the way forward, and it is natural for the block-layer developers to want to see a large piece of new infrastructure using it. There is only one little difficulty, though: BFQ is an I/O scheduler, and the multiqueue code has no provision for I/O scheduling. Leaving out scheduling might make sense if the multiqueue API were to be used only with high-end, solid-state drives, but it also makes that API mostly useless for rotational devices. And, unsurprisingly, it makes the integration of a new I/O scheduler hard.

If rotational storage were on its last legs, neglecting to properly support it in the future block API might make sense. But, despite the ongoing advances in solid-state storage devices, rotational storage seems likely to stay around for some time yet. That suggests that adding I/O scheduling to the multiqueue API would be a good thing to do.

Making such an addition was suggested to Valente, but he did not feel qualified to do the job. A developer who is qualified is Jens Axboe, the maintainer of the block layer and the author of the multiqueue API. So his recent posting of multiqueue I/O scheduling support is a good sign for anybody who is interested in seeing the BFQ scheduler find its way into the mainline. This work is still in an early stage, but Valente is already working on porting BFQ over to this new API; Axboe is offering ongoing help in getting that work done.

So the final chapter in this long story appears to be coming into focus. BFQ will almost certainly find its way in as the first nontrivial I/O scheduler for the multiqueue block API. It is not clear whether there will ever be a need for any others. CFQ, which works well with the legacy APIs, will stay in place without big changes, but it will slowly lose relevance over time. It's a reasonably happy ending that might have been happier had it happened a year or two ago, but sometimes these things take time.

Comments (1 posted)

The LPC Android microconference, part 1

The Linux Plumbers Android microconference was held in Santa Fe on November 3rd; it had a large number of topics to cover, many in the form of lightning talks, trying to expose developers to a number of recent developments connected to Android. There was strong attendance from community members as well as from Google, including developers working on Android, Chrome OS, and Brillo. This article is the first in a two-part series describing the conversations that took place during this microconference.

Note that the topics covered elsewhere in LWN (recent and future scheduler efforts and Brillo kernel maintenance) will not be covered here.

Android code in staging

The first speaker was John Stultz, who quickly covered the current state of the Android code in the staging directory (slides [PDF]). With around 150 patches applied in the last year, it hasn’t seen a huge amount of change recently — but that is not a trivial patch rate either. Highlights were recent progress with upstreaming the Android Sync framework and the removal of the timed output and timed GPIO drivers. One interesting item was that no new features have been added to staging in the last year.

The remaining Android code in staging has come down to three drivers: ashmem, the low-memory killer, and ION. Efforts to upstream ashmem’s memory unpinning feature have stalled, so input was requested on whether we should upstream it as-is (as was done with binder) or try instead to add shrinker-based unpinning to the memfd feature.

Android has already integrated a user-space based low-memory killer utilizing memory-pressure notifiers. However, due to the high cost of calculating who to kill in user space once the notification has happened, this implementation has not shipped on any devices as of yet, leaving most devices still making use of the in-kernel low-memory killer. Discussion on the ION memory allocator was deferred to Laura Abbott’s session later in the day.

John left the session with the question of whether staging has been particularly useful to upstreaming Android functionality, and if folks thought more features (providing some specific examples) should be pushed via staging or not.

Background updates

Rom Lemarchand from Google gave a lightning talk on Android background updates (slides [PDF]). Before starting his talk, he pulled out a phone which he had avoided applying a recent update to. He hit the "update" button and began his talk while holding the phone up. His purpose was to show how much faster and less painful the update process can be.

While the update progressed, he provided an overview of the simplified (from the user’s perspective) background-update process, then contrasted it with the complex state machine needed between the bootloader and kernel in order to allow this process to work. The android background update was adapted from Chrome OS; it uses a dual partition layout, so there must be duplicate partitions for anything updated in an over-the-air update (usually the boot and system partitions). This additional storage cost was mitigated by using squashfs and minimizing the number of applications available over the Play Store that are stored on the system partition. The Android developers also introduced a new layer to handle the communications between the kernel and the bootloader.

There are a few additional kernel patches needed for ramfs booting and dm-verity validation for the rootfs. The system will fail back to the old partition if there’s a validation failure, so the end result is an update mechanism that lowers the chance of bricking a device. Additionally, Android now does the application rebuilding step in the background, so one doesn’t have to wait for that after installing an update. Of course, before he had finished his talk, his phone had already rebooted into the updated image and was ready to use.

vDSO issues

Next up was Kevin Brodsky from ARM on the vDSO on arm64 (slides [PDF]). Kevin outlined how a vDSO is a shared library for user space that is provided by the kernel; it allows certain kernel system calls like clock_gettime(), to be processed entirely in user space, avoiding context switches and greatly improving performance. He provided a detailed look at how the vDSO is implemented before diving into his work implementing 32-bit vDSO support for arm64.

While the performance of 32-bit apps in a 64-bit environment might seem unimportant, there are many Android apps that have not been built for 64-bit systems and may not be updated in the near future, as moving to 64 bits comes with its own costs (pointer sizes double, etc) that can negatively affect performance. Additionally, Chrome OS on arm64 platforms currently runs a 32-bit-compiled user space. So performance for 32-bit apps is important for many common use cases.

He outlined some of the details of his proposed patches, including complications like the fact that one has to have a 32-bit toolchain installed to build the 32-bit vDSO when building an arm64 kernel, as well the need to have support added to the C library implementations in order to enable the vDSO which, unfortunately, didn’t make it for the Android Nougat release. He then covered some performance results, and again pointed to his proposed patch set submitted upstream.

Kernel behavioral analysis

Next up was a talk on kernel behavioral analysis and visualization by KP Singh from Google (slides [PDF]). KP covered a number of tools he has been using and working on. The first was Linux Integrated Systems Analysis (LISA) which has been heavily used by ARM to automate running workloads and collecting traces. He covered Trace Analysis And Plotting in Python (TRAPpy), which parses trace files and allows for visualization. Then finally there is the Behavioral Analysis and Regression Toolkit (BART), which allows expected behaviors to be defined and then tested against the trace logs, allowing for behavior regressions to be caught.

Defining behaviors in BART is done by by first establishing a test case, such as one that generates a number of threads that run for 10% of the time. Then, if one expects such lightweight tasks to be bound to the "little" CPUs in a system, one can specify that the residency of those tasks should be on the "little" cluster for 75% of the time. He presented example charts generated from TRAPpy that showed CPU usage across the system; it illustrated the small tasks being properly packed onto the little CPUs, keeping the big CPUs idle. He then showed how BART would view that data and validate the earlier statement made as true. BART has a number of ways of testing scheduler behavior, allowing assertions about temperature and load averages to be made.

He plugged his phone into his laptop and did a live demo, running tests using LISA, which can be used via a web page. LISA generated interactive TRAPpy graphs, also shown in that web page, as well as various BART assertions that evaluated to true or false. This generated a fair amount of discussion around the Jupyter web-interface used, which seemed new to many in the crowd. While the tooling can be used directly in the shell, it was noted that Jupyter Python notebooks are common in academic research, and these rich tools are useful for the type of performance analysis used in the examples.

There were some questions about the limits of doing browser-based interactive trace charts, but those limits are mostly a function of the amount of memory on the host rendering the web page. There was also interest in how other test frameworks might be able to use these tools, one idea being that kselftests might also be able to adapt some of these detailed behavioral tests as a way to watch for regressions.

Large pages in ION

The first graphics-focused discussion was about large pages in ION, by John Reitan from ARM (slides [PDF]). He covered some of his efforts to prototype a compound-page heap in ION. On some display and IOMMU hardware, 2MB pages are needed for rotation. As the kernel's native page size is 4K, this could previously only be done by using a special carve-out heap to guarantee allocations using 2MB of contiguous physical pages. His solution, which is designed to replace the carve-out heap usage, maintains a pool of pre-allocated and zeroed 2MB pages, which are asynchronously filled. One optimization he was hoping to achieve was to avoid internal fragmentation of the allocations, as the 2560x1440 buffers he was using only need 14.065 MB, causing only 64KB of the the last 2MB page to be used. Packing tails wasn’t an option as the memory does have to be virtually contiguous, but he could use that extra space to store other small allocations.

He prototyped this on a Nexus 10 tablet to better represent how his allocator would work when there are numerous other device drivers active and doing things like pinning memory. The Nexus 10, while old now, had a world-leading high-resolution display at the time of its release, so it's still relevant to mid-range devices today. When initially developed, John explained, the Nexus 10 had a number of issues with the GPU pinning 4K pages, which was worked around by reserving a large 348MB carve-out for the GPU. Also, the IOMMU would cause display corruption resulting from TLB misses when enabled, which using 2MB pages avoids. So another 256MB carve-out was set aside to make that work.

His compound page heap is able to replace these two carve-outs and can be used for all ION allocations including camera and video-decoding buffers, freeing up more than half a gigabyte of reserved memory. He ran this code through a number of scripted tests to stress the system. The system was overall more responsive, but unfortunately, he found that, after a week or two, the tests would fail and the device would not wake up. This ended up being due to failed allocations, as the GPU was pinning pages and fragmenting the new heap. So, for the next steps, he wants to improve the GPU driver to support page migration and avoid causing fragmentation.

He discussed his plans to try to enable ION to support order-9 allocations and trying to upstream the heap code. He also wants to follow the efforts on the new Unix device memory allocator that was discussed during XDC 2016.

ION

Next was Laura Abbott from Red Hat covering the status of ION (slides [PDF]). Laura was mostly wanting to have a discussion around the goals and needs folks have for this allocator. ION has lingered in staging for a long time, and while it's been getting patches for things like a new query ioctl() and cleaning up dead code, there has been little progress toward addressing its fundamental issues. There’s unfortunately no one who is working on ION full time and, most problematically, the users of ION are, for the most part, proprietary out-of-tree graphics drivers, so there is no real reference platform that the community has any interest in.

Laura thinks that the current iterative approach to fixing ION in the hope of upstreaming it isn’t working and should be given up. The ION driver as it exists today is not self-contained like binder, so its not really something that could be upstreamed as-is. Her proposal is to freeze ION development to try to force development focus elsewhere. She said she would still take bug fixes but, since upstreaming ION isn’t really a goal in itself, we really need to find a good solution.

She also pointed to the new Unix device memory allocator, which is trying to address some of the same issues. This brought up some discussion in the room around the somewhat old debate of ION/Gralloc style user-space constraint awareness vs in-kernel constraint solving, as was proposed in the attach-and-allocate-at-mapping efforts made previously.

Another discussion point was whether there is a need for a centralized dma_buf allocator. Both in-kernel constraint solving and centralized allocation were previously proposed with cenalloc. Currently, the upstream approach requires user space to allocate buffers using driver-specific ioctl() calls; if there’s a constraint on those buffers, user space has to do the allocation from the most constrained driver. This seems to work for upstream developers, so these new approaches don’t appear to have much immediate benefit, but it was interesting to hear how folks looking at the new Unix device memory allocator for some more advanced use cases might be "rediscovering" some of the issues that ION was created to solve.

Hardware composer and drm_hwcomposer

Next on the schedule was Greg Hackmann, Sean Paul, and Zach Reizner from Google talking about Hardware Composer 2.0 (HWC2) and the drm_hwcomposer implementation (slides [PDF]). Greg started by providing an overview of what the hardware composer does: compositing a number of different buffers in the most efficient way possible. It is mostly used to save power by doing the composition using the display hardware's support for multiple planes and overlays, rather then compositing the buffers in the GPU (which is less power-efficient). He then reviewed some of the changes from the original HWC framework to HWC2, which entails a richer API with some function name clarifications and the use of non-speculative fences instead of the speculative ones used before. He then showed some diagrams illustrating the differences between how the APIs are used. He pointed to the implementation and documentation that is in the AOSP source, and promised a reference implementation for the Nexus 9 device soon.

Sean and Zach discussed their efforts on drm_hwcomposer, which is a kernel modesetting-based HWC implementation that was initially done for the Pixel C tablet. They showed how the HWC2 framework simplifies some of the implementation details and got into some complexities of using overlays on some hardware. It turns out that, while doing composition in the display hardware is more power-efficient than using the GPU for screens where there isn’t much change, the in-display composition is still re-done over and over, which consumes memory bandwidth. In that case, doing composition in the GPU once can save that bandwidth. To balance this tradeoff, the screen is broken up into rectangular regions separating parts that are changing from parts that are mostly static. Then only the changing portions are sent in via the overlay.

They outlined their GL compositor, which uses a shader per layer, allowing each rectangular region to be generated with a single drawing call. For the most part, this implementation is generic, but they have some device-specific logic in the planner code, which is used every time composition changes; it helps map SurfaceFlinger layers to hardware planes. There was some discussion at that point that having hardware-specific planners isn’t great, but it's better to have a mostly generic drm_hwcomposer with small, device-specific planners rather than having a bunch of forked drm_hwcomposers for each device.

Mainlining Sync

Gustavo Padovan from Collabora was up next to talk about his efforts to move the Android Sync Framework out of the staging directory (slides [PDF]). He started by providing an overview of how the Android Sync framework works; it provides the concept of a timeline, which is a monotonic counter to which sync points can be added. There can be multiple timelines in the system at one time. One or more of those sync points can be merged together into what Android calls a "sync fence". These sync fences can be exported and shared in user space via file descriptors. User space can wait on those sync fences, which will be signaled when all the sync points contained have been signaled by the monotonic counter reaching their value.

Gustavo covered his upstreaming efforts to bring explicit fencing to mainline. He removed the sync points and timelines. replacing them with the DRM fence implementation, and reworked the sync fence implementation, renaming it to "sync file". He has also implemented patches to Android’s libsync library to be able to support both legacy Android sync fences as well as sync files. He clarified the types of fences DRM uses, which can be "in fences", which are fences that can be waited on, and "out fences", which are per-CRT and are signaled on every scanout, informing the user that the previous buffer can be reused. Support for explicit fencing must be added to each driver and, at this point, freedreno is done and work is in progress on i915 and virgl. User-space support is also needed in mesa, which is in progress by Rob Clark. drm_hwcomposer already supports explicit fencing; it has been a good test case for the upstream efforts, but work there continues.

There were some questions as to what benefits explicit fencing brings over implicit fencing, and it was clarified that the performance should be better. There were also questions of how to test this infrastructure, which, Gustavo clarified, can be done using the sw_sync driver. Finally, folks asked about the status of the patches to libsync in AOSP, which were clarified to be in their 3rd revision and are getting quite close. Developers from Google commented that the next revision should be able to be merged.

(See also: coverage of Gustavo's XDC talk on mainlining the sync framework.)

More to come

As mentioned at the beginning, this microconference covered a large range of topics. The second part in this series, to appear shortly, will finish out our coverage of this gathering; stay tuned.

Comments (none posted)

Debian considering automated upgrades

The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users.

The proposal was brought up during the Debian Cloud sprint in November by longtime Debian Developer Steve McIntyre. The rationale was to make sure that users installing Debian in the cloud have a "secure" experience by default, by installing and configuring the unattended-upgrades package within the images. The unattended-upgrades package contains a Python program that automatically performs any pending upgrade and is designed to run unattended. It is roughly the equivalent of doing apt-get update; apt-get upgrade in a cron job, but has special code to handle error conditions, warn about reboots, and selectively upgrade packages. The package was originally written for Ubuntu by Michael Vogt, a longtime Debian developer and Canonical employee.

Since there was a concern that Debian cloud images would be different from normal Debian installs, McIntyre suggested installing unattended-upgrades by default on all Debian installs, so that people have a consistent experience inside and outside of the cloud. The discussion that followed was interesting as it brought up key issues one would have when deploying automated upgrade tools, outlining both the benefits and downsides to such systems.

Problems with automated upgrades

An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so.

For example, Noah Meyerhans, a Debian developer, argued that "any reasonably well managed production host is going to be driven by some kind of configuration management system" where competent administrators can override the defaults. Debian, for example, provides the policy-rc.d mechanism to disable service restarts on certain packages out of the box. unattended-upgrades also features a way to disable upgrades on specific packages that administrators would consider too sensitive to restart automatically and will want to schedule during maintenance windows.

Reboots were another issue discussed: how and when to deploy kernel upgrades? Automating kernel upgrades may mean data loss if the reboot happens during a critical operation. On Debian systems, the kernel upgrade mechanisms already provide a /var/run/reboot-required flag file that tools can monitor to notify users of the required reboot. For example, some desktop environments will popup a warning prompting users to reboot when the file exists, although Debian doesn't currently feature an equivalent warning for command-line operation. Vogt suggested that the warning could be shown along with the usual /etc/motd announcement.

The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues.

Furthermore, system reboots are only one part of the problem. Currently, upgrading packages only replaces the code and restarts the primary service shipped with a given package. On library upgrades, however, dependent services may not necessarily notice and will keep running with older, possibly vulnerable, libraries. While libc6, in Debian, has special code to restart dependent services, other libraries like libssl do not notify dependent services that they need to restart to benefit from potentially critical security fixes.

One solution to this is the needrestart package which inspects all running processes and restarts services as necessary. It also covers interpreted code, specifically Ruby, Python, and Perl. In my experience, however, it can take up to a minute to inspect all processes, which degrades the interactivity of the usually satisfying apt-get install process. Nevertheless, it seems like needrestart is a key component of a properly deployed automated upgrade system.

Benefits of automated upgrades

One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:

One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.)

Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.

Possible non-issues

That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users.

Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.

Automating major upgrades?

While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical.

While major upgrades are not currently automated in Debian, they are usually pretty simple: edit sources.list then:

    # apt-get update && apt-get dist-upgrade

But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line.

The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.

Future work

Ubuntu currently defaults to "no automation" but, on install, invites users to enable unattended-upgrades or Landscape, a proprietary system-management service from Canonical. According to Vogt, the company supports both projects equally as they differ in scope: unattended-upgrades just upgrades packages while Landscape aims at maintaining thousands of machines and handles user management, release upgrades, statistics, and aggregation.

It appears that Debian will enable unattended-upgrades on the images built for the cloud by default. For regular installs, the consensus that has emerged points at the Debian installer prompting users to ask if they want to disable the feature as well. One reason why this was not enabled before is that unattended-upgrades had serious bugs in the past that made it less attractive. For example, it would simply fail to follow security updates, a major bug that was fortunately promptly fixed by the maintainer.

In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community.

More information about the configuration of unattended-upgrades can be found in the Ubuntu documentation or the Debian wiki.

Comments (20 posted)

Distribution quote of the week

— Lars Wirzenius

Comments (2 posted)

Release for CentOS Linux 7 (1611) on x86_64

CentOS Linux has released version 7.3-1611 of its Enterprise Linux clone. "This release supersedes all previously released content for CentOS Linux 7, and therefore we highly encourage all users to upgrade their machines. Information on different upgrade strategies and how to handle stale content is included in the Release Notes."

Full Story (comments: 6)

Release for CentOS userland 7(1611) on armhfp

CentOS 7.3-1611 is available for armhfp compatible machines.

Full Story (comments: none)

KDE neon User LTS Edition Out Now

KDE Neon User LTS Edition has been released. "This comes with Plasma 5.8 LTS, updated for new bug fix releases (e.g. 5.8.5 is out at the end of this month) and will not change to Plasma 5.9 when they becomes available. A common critisism of LTS editions is that it just means users get old versions with known bugs. KDE neon User LTS Edition comes with the latest KDE Applications and it comes with the latest KDE Frameworks release and Qt 5.7, so all the KDE software we ship is the latest stable version. Along with other KDE neon editions we’ll also ship the HWE updates for Linux and Mesa when they become available."

Comments (none posted)

Announcing OLPC OS 13.2.8 with Sugar 0.110

OLPC OS 13.2.8 for XO-1, XO-1.5, XO-1.75, and XO-4 has been released. It includes the Sugar 0.110 desktop. The release notes have details.

Full Story (comments: none)

openSUSE Board election for 2016

Nominations are open for openSUSE board elections. There are 2 seats open. You may also apply for openSUSE membership in order to vote or stand for a seat. Nominations and the membership application period end January 1, 2017.

Full Story (comments: none)

Distribution newsletters

• DistroWatch Weekly (December 12)
• Lunar Linux weekly news (December 9)
• openSUSE News (December 8)
• openSUSE Tumbleweed – Review of the Week (December 9)
• Tails report (November)
• Ubuntu Weekly Newsletter (December 11)

Comments (none posted)

Remembering a friend: Matthew Williams (Fedora Community Blog)

Over at the Fedora Community Blog, Brian Proffitt writes about Fedora member Matthew Williams who passed away recently from cancer. "Matthew’s passion to constantly improve the software and hardware with which he worked created a tireless advocate for the Fedora Project, and his presence was felt at conferences across the nation: SCaLE, Ohio LinuxFest, and the former Indiana LinuxFest, an Indianapolis-based event that he helped found. Matthew also devoted time to interviewing and archiving notable figures in the free and open source software communities to learn what drove people to work on their projects. He was also very driven to share what he knew, launching the Open FOSS training site in 2015 to help new Linux users with getting involved with any Linux distribution. While he was active in the Fedora community, Matthew was also very involved with Ubuntu as well."

Comments (none posted)

An update on the Linux Test Project

In the roughly two years since I wrote a couple of introductory articles on the Linux Test Project (LTP), lots has happened. Much of the effort spent falls into the "work as usual" category, but there are some interesting bits to present. The goal of LTP is to write and maintain test cases for the Linux kernel and related code. There are about three thousand test cases that are made up mostly of various system call tests, a POSIX conformance test suite, I/O stress tests, namespace and control-group (cgroup) controller tests, network tests, and so on. For an introduction to the project, have look at the first article that describes LTP goals, a bit of history, and shows how to run the tests. The second article describes the test library and explains how to write a test case.

For those who have never heard of it, LTP is large automated kernel test suite; I have been one of its maintainers for about five years now. I started to work on LTP for SUSE in order to eliminate unreliable test cases and, in retrospect, that proved to be more work than I had ever imagined. I got my first LTP patch accepted in the middle of 2009 and became co-maintainer of the project a bit more than year later.

Development statistics

Let's start with a few, mostly boring, statistics. The project has produced timely releases roughly every four months. The number of patches and contributors has showed a similar pattern over the last two years or so:

Git tag	# patches	# authors	Lines added	Lines removed
20140828	311	24	106326	138946
20150119	139	21	9627	10201
20150420	183	23	31911	42167
20150903	272	27	16324	18149
20160126	191	29	14681	26907
20160510	169	24	12255	9378
20160920	236	28	11462	27741

If you are wondering why the line stats for the tag 20140828 are more than ten times bigger than the other entries, then the answer is simple: there were two patches that moved around the TI-RPC test suite, thus contributing 79452 to lines removed and then the same number to lines added. If you subtract this number from the diff stats, the result is more consistent with the rest as you get 26874 lines added and 59494 lines removed.

I would like to point out that the tag 20160510 was the first-ever development cycle with more lines added than removed. Which hints that, after six years of effort, the project is finally getting closer to a state where it is done with cleaning up the code base and is finally putting much more effort into developing new test cases.

Much harder to produce are statistics on the increase in test coverage. Relatively easy, but to some extent wrong, is to count additions and deletions in runtest files. A runtest file is a list of binaries to be executed during the test run, hence counting these should at least show some trend. However there are two flaws that undermine this approach's credibility. First, a single binary can do many tests and sometimes it makes perfect sense to put new checks into existing binaries. Second, we have been getting rid of big test cases containing hundreds of tests, which took half an hour to execute, by splitting them into reasonable-sized tests.

Git tag	Tests added	Tests removed
20150119	36	36
20150420	41	17
20150903	93	35
20160126	450	97
20160510	60	23
20160920	700	1232

This table was created by tracking changes in the runtest/ directory between two releases and counting lines that start with a plus or minus sign followed by a letter in order to ignore changes to white space and comments. As you can see, LTP was splitting big test cases into smaller chunks in 20160126 and 20160920 and also did some reorganization in the latter.

Bugs found

A test suite would be useless if it failed at its basic task of finding bugs. Unfortunately I have no clear data on who uses the LTP or what bugs it has found unless that information can be pieced together from bits scattered around somewhere in public. I would love to get more direct feedback on what test cases have found bugs, which in turn would help LTP developers prioritize their work better.

For instance, grepping for "ltp" and "LTP" strings in the kernel Git repository shows thirteen patches that were produced in order to fix a bug found by LTP in 2016. Which does not seem like all that much, but hopefully that is just a tip of the iceberg. For example, the memcg migration kernel lockup bug was found by LTP, but I only know that since I was personally involved.

As I wrote in the previous article, the LTP is being run on the 0-Day testing cluster and, for some time now, test failures are CCed to the LTP mailing list as well. That effort has found a few bugs already. However, since it has been a few months since the last failure report, I asked the developers behind the 0-Day project how things are going. I got an answer from Fengguang Wu that they run a subset of the LTP regularly for bleeding-edge kernel branches and that this initiative has found seven bugs in the last year. But I've also been told that they have some problems with a few test cases. Clearly there is some room for improvement as getting more test cases running on the 0-Day infrastructure would pay off greatly in the long term.

Most notable changes

The following are descriptions of what I think were the most interesting changes during last two years, sorted from oldest to newest.

All except "test passed" messages now include source file name and line number, which helps developers to quickly connect a particular error or failure with the corresponding source code. While this does not seem that important at first glance, it helps greatly in practice, since you don't have to reproduce the failure in order to figure out what exactly went wrong. There is a huge difference between "BROK: fork() failed: EAGAIN" and "test01.c:33: BROK: fork() failed: EAGAIN".

Most of the sleep() calls were replaced with a proper IPC synchronization, which not only fixes rare and hard to reproduce failures but also saves a couple of minutes of test suite runtime.

One of the bigger changes in the past two years was the arrival of the new test library, which is easier to use, cleaner, and race-free. I sometimes call it a "driver-model for test cases" since the test no longer implements the main() function and instead exports a structure holding function pointers and some bit flags telling the test library what should be initialized before the test functions are called. To give you a better idea of the changes look at the umount02 test case using the old library and compare it with umount02 using the new library.

The simple fact that main() was moved to the test library not only allowed moving most of the boiler-plate code there as well, but also enabled running the test in a child process, while the main process watches for timeouts and cleans up after the child in the event it was killed or crashed. The other two notable changes that the new library brought are that there is now one global cleanup function for the whole test and that the test results are now stored in a chunk of shared memory and updated atomically. The change for the cleanup function was needed because passing, possibly different, cleanup functions to each test library function proved to be too difficult to maintain correctly.

The test results are now stored in a page of shared memory, which turned out to be the best solution for all of the project's needs. No longer is there a requirement to make sure that test results are propagated correctly from a child process. When the child calls a corresponding result-reporting function, the results are guaranteed to be stored as the function returns. And since the results could be changed from different processes at the same time, they have to be updated using atomic operations. A side effect of this is that the library is now also thread-safe, thus killing two birds with one stone. Shared memory is also used to implement interprocess synchronization based on the futex() system call, thus eliminating many sleep() calls in the test cases as well. The only downside is that, at the time of writing, only ~80 out of ~1000 test cases use it, so converting all of the test cases will probably take another year or two.

Last but not least, a new shell library, in the spirit of the new C library, is being developed and many shell test cases are being cleaned up at the same time.

Future work and challenges

One constant challenge for the LTP project is that kernel development never ceases and keeping pace with it is difficult given the obvious disparity between the manpower put into the LTP and the Linux kernel. In particular, the development of cgroups, which have been a moving target for quite some time, has been challenging. The subtle changes in their behavior tends to break tests more often than I would like.

Part of the reason for this is that Linux developers ran out of proven Unix interfaces to copy and started to explore new designs. Mistakes are bound to happen and sometimes it's inevitable that they have to be corrected in a non-backward-compatible manner. You may counter that by saying that introducing incompatible changes in the user-space API is forbidden, but that only applies to changes that actively break something useful and it seems that there are cases where the test suite is the only one depending on a particular behavior. In that case, preserving the old and problematic behavior is questionable.

Take, for instance, LTP issue #93. The problem here is that as a side effect of kernel commit b2052564e66d ("mm: memcontrol: continue cache reclaim from offlined groups") some charge may stay accounted for even after the subgroup has been deleted, which prevents use_hierarchy from being toggled once we make use of the memory cgroups. That applies even after we clean up and remove all the subgroups. Most likely this is a non-issue, since those who want to change the default would do so once the system boots up, but test cases do toggle the switch in order to test both scenarios.

Another reoccurring problem is that, while kernel interfaces are being created, the developers involved do not even consider that the added APIs should be testable. As a result, writing automated test cases is often more complex and includes some amount of guesswork. Consider, for instance, the readahead() system call. The call may truncate the count argument silently if it's deemed too big. The maximal value depends on the kernel version, sysfs knobs, the filesystem used, and so on. If the system call returned the truncated count argument instead of zero on success it would be much easier to assert the expected increase in cached pages. Another example is that the resolution of the getrusage() CPU timers depends on the value of CONFIG_HZ, which is hard to guess at runtime.

On the brighter side, one idea I have in my mind is to be more flexible with releases. At this point, it takes two months on average for a fix to a test case to reach a release; since LTP does not have the manpower to backport fixes to the latest release I would like to get more lean. Ideally, the project would adopt the release-on-green approach, which could be fully automatic. What I would like to do is an automated LTP run on a few distributions once a week and generate a release in the case that no regressions have been introduced.

Conclusion

A fair amount of progress has been made by the project in the last few years. LTP has gotten rid of the "too broken to be useful" sticker, which itself is something to be proud of. In the years to come, I would like to invest more into increasing the coverage, writing more complex test cases, and being smarter about these two points. But there is a limit to what LTP developers can achieve by themselves. We need your help. Even just contributing ideas for automated test cases is something that would help LTP developers greatly since that would save a large amount of time spent on research. So let me end here with a hope for better cooperation between LTP and Linux kernel developers.

Comments (3 posted)

Development quote of the week

— Alexander Deucher

Comments (none posted)

Krita 3.1 released

Version 3.1 of the Krita image editor is available. "Krita 3.1 is the result of half a year of intense work and contains many new features, performance improvements and bug fixes. It’s now possible to use render animations (using ffmpeg) to gif or various video formats. You can use a curve editor to animate properties. Soft-proofing was added for seeing how your artwork will look in print. A new color picker that allows selecting wide-gamut colors. There is also a new brush engine that paints fast on large canvases, a stop-based gradient editor." See the release notes for more information.

Comments (none posted)

LLVM's new versioning scheme

The LLVM compiler project is the latest to do away with long-lived major version numbers. "With our next release though, we're changing this, to increase the version number by '1.0' with every major release, which means that the first major release after LLVM 4.0 will be LLVM 5.0 (expected September 2017)."

Comments (none posted)

Nextcloud 11 released

Nextcloud 11 has been released with many security and scalability improvements. "Nextcloud 11 introduces Apache Solr powered Full Text Search, enabling users to find words or phrases in text, pdf and common office documents on internal, external, shared and encrypted storage. The next generation Federation technology introduces a central lookup server, enabling Nextcloud users to find each other irrespective of the server their account resides on. The experimental Spreed app integrates secure, peer to peer audio and video chat in Nextcloud."

Comments (34 posted)

Development newsletters

• Emacs News (December 5)
• Emacs News (December 12)
• What's cooking in git.git (December 12)
• What's cooking in git.git (December 13)
• Git Rev News (December 14)
• This week in GTK+ (December 12)
• OCaml Weekly News (December 13)
• Perl Weekly (December 12)
• PostgreSQL Weekly News (December 11)
• Python Weekly (December 8)
• Ruby Weekly (December 8)
• This Week in Rust (December 13)
• TUGboat (December 8)
• Wikimedia Tech News (December 12)

Comments (none posted)

FSFE Newsletter - December 2016

This edition of the Free Software Foundation Europe's newsletter covers FSFE's 15th birthday, Russian bill makes free software a public priority, FSFE General Assembly annual meeting, and several other topics.

Full Story (comments: none)

KDE e.V. Community Report - 2nd Half of 2015

KDE e.V. has released its community report for the second half of 2015. "Over nineteen years producing high quality free software, spreading open culture, and creating a thriving community, KDE has become a huge umbrella organization supporting all sorts of FOSS-related projects. As a consequence, an even more inclusive, diverse, and open community has grown, with opportunities we couldn't have envisioned some years ago."

Comments (3 posted)

CFP Deadlines: December 15, 2016 to February 13, 2017

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
December 31	March 2 March 3	PGConf India 2017	Bengaluru, India
December 31	April 3 April 7	DjangoCon Europe	Florence, Italy
January 1	April 17 April 20	Dockercon	Austin, TX, USA
January 3	May 17 May 21	PyCon US	Portland, OR, USA
January 6	July 16 July 23	CoderCruise	New Orleans et. al., USA/Caribbean
January 8	March 11 March 12	Chemnitzer Linux-Tage	Chemnitz, Germany
January 11	February 15 February 16	Prague PostgreSQL Developer Day 2017	Prague, Czech Republic
January 13	May 22 May 24	Container Camp AU	Sydney, Australia
January 14	March 22 March 23	Vault	Cambridge, MA, USA
January 20	March 17 March 19	FOSS Asia	Singapore, Singapore
January 31	May 16 May 18	Open Source Data Center Conference 2017	Berlin, Germany
February 6	May 8 May 11	OpenStack Summit	Boston, MA, USA
February 12	June 9 June 10	Hong Kong Open Source Conference 2017	Hong Kong, Hong Kong

If the CFP deadline for your event does not appear here, please tell us about it.

Events: December 15, 2016 to February 13, 2017

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
December 27 December 30	Chaos Communication Congress	Hamburg, Germany
January 16	Linux.Conf.Au 2017 Sysadmin Miniconf	Hobart, Tas, Australia
January 16 January 17	LCA Kernel Miniconf	Hobart, Australia
January 16 January 20	linux.conf.au 2017	Hobart, Australia
January 18 January 19	WikiToLearnConf India	Jaipur, Rajasthan, India
January 27 January 29	DevConf.cz 2017	Brno, Czech Republic
February 2 February 3	Git Merge 2017	Brussels, Belgium
February 4 February 5	FOSDEM 2017	Brussels, Belgium
February 7 February 9	AnacondaCON	Austin, TX, USA

If your event does not appear here, please tell us about it.