Lessons from open source in the Mexican government

The adoption of open-source software in governments has had its ups and downs. While open source seems like a "no-brainer", it turns out that governments can be surprisingly resistant to using FOSS for a variety of reasons. Federico González Waite spoke in the Open Government track at SCALE 22x in Pasadena, California to recount his experiences working with and for the Mexican government. He led multiple projects to switch away from proprietary, often predatory, software companies with some success—and failure.

González Waite began by noting that he is a Mexican/Kiwi ("there's not many of us", he said with a grin), who spent nine or ten years in high-level roles in the Mexican government "promoting open-source implementations". Among other things, he was the CTO for the Ministry of Foreign Affairs; "I am actually responsible for Mexicans having an electronic passport today." That was one of the projects that he led and part of it was done with open-source software, which is something that people find to be amazing, González Waite said. He served in the office for national strategy under President Andrés Manuel López Obrador, eventually moving into the CEO role at the National Research and Innovation Center for Mexico.

In all of his roles, he advocated using open source within the government; it is never easy to be a "change-maker" of that sort, he said. He left the government at the recent change of presidents (to Claudia Sheinbaum) and is now working on "helping people do their own transformations" to open source, while keeping an eye out for the next big thing. He noted that he would take questions at the end and he might not be able to answer them all due to confidentiality responsibilities, but he is "no longer a public official, so that gives me a lot more leeway to talk freely".

Why?

"So why open source for the government?" One reason is to cut costs; Mexico is a financially small, developing country that is always looking to reduce costs, he said. Paying for licensing was costing a lot of money that could be used to do other things. The López Obrador administration passed an austerity plan into law, so that various actions (e.g. government officials traveling out of the country for work) needed presidential-level approval. That also affected purchasing licenses and the plan pushed the use of open source.

Another major reason is to work toward Mexico having "IT sovereignty". One major problem that the country has had is that its IT leadership is not technical, which means that people were signing off on projects and licenses without understanding what they were buying. "They really didn't know if they are were getting good value for [the] money".

There is a need to build up the talent within the government to support open-source software; "there's no point in building a whole infrastructure on open source if you don't have the talent to keep it up". There has been a lot of effort to bring in new people and to train existing workers to that end. One goal is to go from being an IT-consumer nation to being an IT-producing nation. It is surprising to many who think of Mexico as a manufacturing and assembly nation that the country does actually produce a lot of technology, he said. Much of what was being produced within the government, though, was being siphoned off by the private sector; more recently, that is changing, so that the government can package and even sell its IT development.

Another goal is for Mexico to become more self-sufficient, so that it is not locked-in by vendors of various sorts. In his most recent role, he was able to see what was happening all across the government. One common thread is that when agencies were asked why they were spending so much on a particular service, they claimed they had no choice, even though there are lots of other companies offering the same services. It turned out that various contracted companies had corruptly put the software licenses they bought for the government into their own names, leading to a lock-in for their services. Moving to open source can break those and other kinds of locks.

There are multiple agencies within the government to handle various pieces of the technology puzzle, González Waite said. He headed INFOTEC, which is the IT service provider to the rest of the government. It managed the largest telecommunications project in the world at that time to place 6,000 BTS antennas, while deploying over 30,000km of fiber-optic cable, in various places throughout the country. That was part of the CFE Internet Para Todos project, which is aimed at the democratization of access to the internet; prior to that, a large region in Mexico had no internet because it was not commercially viable.

All of the technology projects within the federal government require approval from the president's office; that has been part of the country's laws for quite a while, González Waite said, but had not been enforced. Under the López Obrador administration, reviews of those projects often found that they could use open source, rather than the expensive proprietary solution being proposed, so the projects needed to switch.

Using open-source software has been enshrined in Mexican law since 2021, when it was specifically authorized for government agencies. González Waite went briefly through a few pieces of legislation to show that the government is serious about cutting costs and gaining control of its infrastructure through the use of open-source software.

Foreign affairs

He led a project for the foreign affairs agency; it started with separating the data that was being stored so that some of it was put into the cloud, while the most sensitive data was still stored in the government's data center in Mexico. The agency was resistant to moving any of its data into the cloud, because it was supposedly all "national security data", but there was no data classification that specified which data was actually sensitive. When pressed, the agency claimed that anything ever produced in or by a consulate qualified; he pointed out that meant that consulates sending party invitations and holiday greetings were violations of the national security act. That led to an effort to classify the data and, thus, to foreign affairs being the second government agency to store data in the cloud.

Another part of the project was to move away from Oracle and to PostgreSQL. That change led to various threats and intimidation from the company when it learned of the change, González Waite said. "They told me that the entire passport system of the country was going to fall down" and that it would be his fault that Mexico could not let anyone into or out of the country. "Guess what? That didn't happen."

It turned out that the employees working on the database had heard of PostgreSQL, but never used it. So his team reached out to the Mexican PostgreSQL community for assistance and advice. It took around three months to migrate the information out of Oracle into PostgreSQL. The team took advantage of the shift to restructure the database "because we found that our storage provider was being a little bit naughty", storing the data three or four times in order to charge more money. "Mexico is still a very corrupt country, like a lot of the countries in the world", he said. When you make decisions that take money out of people's pockets, they "start getting really nasty".

The "biggest game-changer" from the switch was that the agency now had direct access to its data and was not reliant on partners and contractors to provide applications and reports. That led to the ability to develop an electronic passport for Mexico. The platform for that was developed in-house, "so we are not paying any licensing"; various open-source software libraries were used in the platform, including for handling biometric information, he said.

Education

In addition to the need for retraining technical staff on open-source topics, his team discovered that students at universities and technical schools were not getting exposed to open-source software either. INFOTEC started a large internship program to place students into government agencies, but found that the students were unable to perform their duties due to a lack of training. That led to creating three-month boot camps for software development and cybersecurity for the interns. The interns were getting paid for that time, which made the training expensive for the agencies, but it brought the interns up to speed so that they could be productive going forward.

Along the way, he visited the communication lab at a technical university and saw that all of the equipment there was from Cisco. The students there did not know how to, say, configure a firewall for any other type of equipment or using open-source software tools—and they were fearful about learning anything else. This is, González Waite said, part of the strategy of the companies, "where they were pretty much giving away the equipment to the universities, but creating a handicap for the students".

To combat that, INFOTEC developed four online educational programs, adding cloud computing and data center operations to the two boot camp subjects, all based on open-source software. Those were two-year courses that included practical work within the government at various levels (local, state, or federal). There was also a need for a "fast track", so the two boot camps were translated into 22-week training courses for those subjects. These open-source education projects reached across borders, he said, which was "a very cool thing"; El Salvador has adopted them and more than 5,000 people have enrolled. Beyond that, the cybersecurity course has been modified to target non-technical government employees in Mexico and more than 3,000 have taken advantage of it.

A different form of education came about in a project to use Mifos for banks in Mexico. Mifos is an open-source platform for handling financial transactions of various kinds. One of the big barriers to getting the project off the ground in Mexico was in educating the regulatory body (National Commission on Banking) about open-source software. His team spent around six months in meetings with the commission to do that education. That paid off and, as the project gained momentum, the regulators had the context needed, which smoothed the path. The project partnered with the Mifos community; some members of the community traveled to Mexico to meet with the regulatory bodies as well.

The other big challenge from this project was in being able return code that was developed for Mexican banking to the Mifos project. Banking institutions have security concerns about releasing code, but the Mifos community wanted to add the code to its repository and tout the deployment, which was one of the largest ever. That was a conversation that played out over three years and eventually resulted in a complicated scheme where older versions of the code could be added to Mifos after two subsequent releases were rolled out to production.

The banking project had gotten its start in part because of a study showing the enormous amount of money being spent to run the existing infrastructure for handling certain types of loans. But, of the two different deployments that came out of the project, the one directly targeting saving a big chunk of that money ended up being canceled—after it had already been proven to work. Another deployment, which basically took the code from the first and reworked it to handle different kinds of loans, was smaller in size, but was switched over to successfully.

The main difference is that the successful project was done for a recently restructured agency that lacked the bias and entrenched interests that the other agency had. The leadership in that organization was better, he said, and also a bit desperate to fairly rapidly show the president's office that it was doing its job effectively.

Lessons

The regulations around open source helped a lot, "because it gave a legal framework to explain to people why we were doing stuff", but it was not enough. Whenever the idea of using open-source software was raised, there was fear because of a lack of knowledge about it, but, because the decision-makers were government officials, there was legal fear on top of that. "That liability could get you into jail, so don't change the technology, leave whatever there is, pay those millions, and nothing will happen, you will be fine."

Technology is often seen as the problem, he said, but he generally found that the problems were due to using obsolete technology and a lack of knowledge about the data being handled. There is often no documentation of the data and its structure, coupled with no understanding of that by the people in charge of it. Poor leadership in the agencies is another barrier; there needs to be a champion for a change of this sort, who understands what needs to be done and properly assigns people to work on it. Many Mexican government officials are political appointees who do not want to get involved in managing a project, so they hand it off to someone else who is already doing too many other things.

Another reason that projects would fail was because the leadership decided that switching to open source meant that no money would be needed in the future. But servers, developers, system administrators, and so on still require a budget if the project is going to succeed. Beyond that, some of the government employees did not have any interest in acquiring the skills needed to switch to open-source software. Many were happy to hand the whole job off to a vendor due to lack of knowledge or "other personal motives".

The biggest lesson that he learned was that projects for switching to open-source software solutions was that either "you win big or you lose big". He and his team never encountered a technological or regulatory problem that could not be fixed; both technology and regulations can be rewritten, "it just takes time and it takes effort". They were able to anticipate most of the loss scenarios in advance because they saw that logical arguments were not prevailing; when that happens, "there's only one explanation and that's an alternative motive". His "number one recommendation" is to ensure, even before the project gets started, that it has the right champion and backing inside the agency or organization; that is the real determiner for whether a project will succeed or fail.

In answer to a question about license audits (to enforce compliance with proprietary-software terms) that was asked by a Mexican citizen who used to work at INFOTEC as a contractor, González Waite said that all of the large proprietary software companies "are big bullies". He has been called into the US embassy and been threatened because Mexico was using technology that was not from the US; those threats were dialed back when he explained that the government also used software and services from Amazon, Google, and Microsoft. Various companies use the US government to bully other countries, but they also use license audits as a reaction to projects that move to open-source software. Every time a successful switch happened, "six months later there was an audit"; having the right legal team helps defend against those tactics, he said.

Earlier, González Waite had noted that the new presidential administration had not specified the use of open-source software in any of the legislation that has been enacted so far. Another Mexican asked him whether he was concerned that the new administration was turning away from open source. "I think it's too early to know that", he said. For now, the relevant legislation from the previous administration has not been undone, but that will come sometime soon, and he is hopeful that the open-source efforts will continue.

The final question that he took was about handling obsolescence in the libraries and other dependencies within government systems; many of those projects will run for decades, but the dependencies may not be maintained for that long. González Waite agreed that it is a major problem; many governments are run by politicians with a short attention span. The politicians are looking for recognition, which leads to votes, and then they move onto the next thing. In Mexico, it is particularly problematic because of the presidential change every six years; most contracts and plans do not stretch past that transition point. It is important to address that problem, because technology is changing so quickly, though it would take major legislation to do so in Mexico—for now, that problem remains unsolved.

A video of just the talk should appear soon on SCALE's YouTube page, but the full set of talks from the room is available with this talk as the first of the day.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for funding to travel to Pasadena for SCALE.]