| LWN.net needs you! Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing |
Back in 2012, we started a quest to find a free replacement for the QuickBooks Pro package that is used to handle accounting at LWN. As is the way of such things, that project got bogged down in the day-to-day struggle of keeping up with the LWN content treadmill, travel, and other obstacles that the world tends to throw into the path of those following grand (or not so grand) ambitions. The time has come, however, to restart this quest and, this time, the odds of a successful outcome seem reasonably good.
More to come.
Proprietary systems like QuickBooks do not provide that access; instead, accounting data is stored in a mysterious, proprietary file format that is difficult to access — especially if one is uninterested in developing on Windows using a proprietary development kit. Locking up data in this way makes moving to a competing system hard, naturally, though a number of (proprietary) alternatives have found a way. It also makes it hard to get company data into the system in any sort of automated way. LWN operates with a set of scripts that convert data into the IIF format for importing, for example.
Recently, as has happened in many other settings, accounting has moved into the cloud; web-based accounting systems abound. These systems can be convenient if you're not trying to do anything that its developers haven't anticipated, and they often interface well with various other sources of financial data, such as banks, payroll services, credit-card processors, and more. But online accounting services take the proprietary-file problem and turn it up a notch, in that now even the file is inaccessible. Somehow, these services tend not to prioritize making it easy for customers to extract their own data, so moving away is even harder than it was before. Making backups can require the (paid) intervention of third-party providers.
There are a number of other interesting questions raised by online accounting services. If the company goes out of business or is forced off the net by a DDOS attack, the accounting data is inaccessible at best, and perhaps gone altogether. The temptation to find other uses for all that business data will be tempting, and not all of these companies are particularly good at resisting temptation. It also makes an attractive target for attackers, as does the widespread sharing of credentials needed to make the integration with other services work.
In other words, even when software-freedom issues are not considered, use of an online accounting system seems like an unacceptable risk for a business to take.
So what is an acceptable solution for a business that wants to maintain control over its own accounting processes and data? As was the case in 2012, there are a number of free-software accounting systems out there, none of which seem like a perfect drop-in replacement for a tool like QuickBooks. But, while no conclusion has been reached as of this writing, it seems like there should be some options that are good enough. So we are going to spend some time playing with these systems using years' worth of real accounting data; the results will appear back here.
In particular, we will be looking at the alternatives from these points of view:
Doubtless there will be other concerns that will arise as actual systems are put into operation.
The first step, though, will be focused on the challenge of extracting accounting data from QuickBooks — a challenge which, it turns out, can be overcome. The next article will describe that process while, it is hoped, also filling in some documentation that is desperately needed by a certain financial software project and including the release of the scripts that were written to get the job done.
Stay tuned for that exciting installment, and those that follow. Your
editor shall endeavor to finish the job without getting distracted this
time around. That should be doable: even writing about accounting is more
enjoyable than, say, going anywhere near anything to do with grsecurity.
At the end, with any luck at all, LWN will have managed to wean itself from
the one piece of proprietary software that is part of its operations.
Restarting the free accounting search
Posted Jul 28, 2017 21:24 UTC (Fri) by pj (subscriber, #4506) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 21:17 UTC (Sat) by pturmel (✭ supporter ✭, #95781) [Link]
Restarting the free accounting search
Posted Jul 30, 2017 20:21 UTC (Sun) by tonyblackwell (subscriber, #43641) [Link]
Restarting the free accounting search
Posted Jul 30, 2017 20:27 UTC (Sun) by tonyblackwell (subscriber, #43641) [Link]
Restarting the free accounting search
Posted Jul 28, 2017 21:59 UTC (Fri) by karkhaz (✭ supporter ✭, #99844) [Link]
I hope you realise that the very first transaction you'll need to use your new accounting software for will be reimbursing your readers for the coffee-damage to their keyboards and monitors.
Restarting the free accounting search
Posted Jul 28, 2017 22:10 UTC (Fri) by lkurusa (guest, #97704) [Link]
Restarting the free accounting search
Posted Jul 28, 2017 22:24 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 1:24 UTC (Sat) by drag (guest, #31333) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 5:22 UTC (Sat) by OrbatuThyanD (subscriber, #114326) [Link]
i'm so sorry I commented on that other thread :(
Restarting the free accounting search
Posted Jul 29, 2017 9:15 UTC (Sat) by dottedmag (subscriber, #18590) [Link]
These systems will not help you with taxes per se, but the rest is a breeze -- all your transactions are out in the plain text file, with parsers and producers available. There might be some plugin for generating these forms though. I haven't actively looked for them.
I am using Beancount and have added several importers for PDF invoices (via pdftotext) and CSV files produced by banks, each one takes less than one evening to make.
Restarting the free accounting search
Posted Jul 29, 2017 12:50 UTC (Sat) by flussence (subscriber, #85566) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 13:08 UTC (Sat) by pabs (subscriber, #43278) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 17:29 UTC (Sat) by josh (subscriber, #17465) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 20:05 UTC (Sat) by flussence (subscriber, #85566) [Link]
Restarting the free accounting search
Posted Aug 3, 2017 16:45 UTC (Thu) by emorrp1 (subscriber, #99512) [Link]
libwebkitgtk-1.0, which is in the process of being removed from all the major distributions (#790204 in debian), so GnuCash have (very very recently) migrated to gtk+3.
Restarting the free accounting search
Posted Jul 29, 2017 18:50 UTC (Sat) by jebba (✭ supporter ✭, #4439) [Link]
Restarting the free accounting search
Posted Aug 1, 2017 7:39 UTC (Tue) by timrichardson (subscriber, #72836) [Link]
Restarting the free accounting search
Posted Jul 29, 2017 23:29 UTC (Sat) by fujimotos (guest, #111905) [Link]
For my experience, every company I've worked for relies on these cloud-based services.
They seem to just take these services for granted, and do not much care about the data
lock-in/availability things.
But inside, I always wonder what happens if a data breach strikes SalesForce/Slack/Github
someday. My co-workers shares internal sales documents on Slack. Our entire source
codes are on Github's private repositories. So...?
The wired part is nobody seems to be thinking about them, as if it's just out of sight.
I'd really like to hear how others manage to handle this problem.
Restarting the free accounting search
Posted Jul 30, 2017 12:48 UTC (Sun) by Funcan (subscriber, #44209) [Link]
Restarting the free accounting search
Posted Jul 30, 2017 17:01 UTC (Sun) by anselm (subscriber, #2796) [Link]
But inside, I always wonder what happens if a data breach strikes SalesForce/Slack/Github someday.
It may be not all that unlikely that the dedicated security guys at SalesForce/Slack/Github are better trained and funded than the ones at your company (if you have any, that is).
What I would be more worried about is SalesForce/Slack/Github's being acquired by your biggest competitor.
Restarting the free accounting search
Posted Jul 31, 2017 9:45 UTC (Mon) by mfuzzey (subscriber, #57966) [Link]
Given the difficulty of getting your (often mission critical for many types of service) data out, what happens if the service provider goes out of business or decides to multiply their prices by 10?
Plain git repos are easy to move but bug trackers, documentation services, accounting services etc much less so.
There are also a lot of online diagramming tools being used which may become inaccessible (or at least become uneditible) if the providers disappear.
For short lived use cases the tradeoffs may be worthwhile as the risk is limited in the short term. But for stuff that needs to be available and modifiable in 10+ years I'm not so sure...
Restarting the free accounting search
Posted Jul 31, 2017 13:07 UTC (Mon) by felixfix (subscriber, #242) [Link]
There is nothing special about cloud vendors.
Restarting the free accounting search
Posted Jul 31, 2017 13:35 UTC (Mon) by pizza (subscriber, #46) [Link]
If your landlord screws you over, you have volumes of laws and regulations to keep you able to at least negotiate while looking for other alternatives. The same for eminent domain seizure, which doesn't just happen overnight. If your roof A/C dies, you bring in portable units to keep your essential equipment cool for as long as it takes.
As for hotels, there are many out there, and conventions are known out a year ahead of time, making it trivial to plan around such things. Ditto for a service center; that's a true commodity.
But your accounting platform is _not_ something trivial to migrate, especially at the last second. That going away will literally put you out of business.
Restarting the free accounting search
Posted Jul 31, 2017 13:50 UTC (Mon) by felixfix (subscriber, #242) [Link]
Saying there is legal recourse implies legal recourse is speedy. That's a laugh!
Saying conventions are known a year in advance ignores all sorts of complications which can happen and have happened. Fires, Legionnaire's Disease, bureaucratic snafus, anything can disrupt plans. Conventions may prove to be more popular than expected. The President may schedule a campaign speech or international political event and disrupt business as usual.
How many big name companies have forgotten to renew their domain names?
You rent five floors in a skyscraper. Where are you going to put those portable A/C units, which require sealed rooms and outside air? Where are you even going to get those portable A/C units when everyone else in the building also wants them? Maybe it will only take a few days -- how long can you afford to be shut down?
Who cares if your finances are being held hostage if your ISP gets knocked out by a hurricane? What if yoru internet connection is clobbered by a ship dragging anchor?
Finances are not the only, or even the most, critical part of a company. That's like saying tires are the most critical part of a vehicle.
Restarting the free accounting search
Posted Jul 31, 2017 14:24 UTC (Mon) by pizza (subscriber, #46) [Link]
Yes, absolutely anything can go wrong at any time. That's why you make contingency plans, prioritized based on their likelihood. That's why, living in a place prone to hurricanes, I have a week's worth of supplies stashed in two different counties, generators (and fuel), and even portable A/C units. Along with plywood, tarps, and a decent chainsaw. And nobody is ever surprised by a hurricane!
The only way I can mitigate against a failure of an external service is to bring it in-house. For some things (eg accounting, short-term power loss) that's practical. For others, it's not.
Restarting the free accounting search
Posted Jul 31, 2017 16:02 UTC (Mon) by tialaramex (subscriber, #21167) [Link]
Legionnaire's looks at first glance much less devastating than, say, a big fire, but in both cases and many others the only way your business will actually have any real continuity is if it's inherently multi-site. That means in _practice_ not in theory there are several distinct physical locations from which all core elements of the business operate.
Most businesses are far too small for that, so in _practice_ their continuity plan for a lot of eventualities is "Go bankrupt". My favourite burrito place in London was flooded out, went bankrupt shortly after re-opening. It doesn't matter that they didn't cause the flood, or that they worked their backsides off to re-open as quickly as possible, the damage was fatal to the business.
Restarting the free accounting search
Posted Jul 31, 2017 17:49 UTC (Mon) by farnz (subscriber, #17727) [Link]
And, coming back round to the topic of cloud services, "go bankrupt" is a perfectly good continuity plan for a sufficiently unlikely eventuality (it was my previous employer's continuity plan for "what happens if our site is hit by a nuclear explosion", for example). The question then becomes whether or not a failure of any given cloud service is sufficiently unlikely that "go bankrupt' is a reasonable continuity plan.
You also have to have a reasonable assessment of the likelihood of your outsourced service failing as compared to (e.g.) your in-house service being hit by a catastrophic failure. Accounting in the cloud makes business sense if the likelihood of your cloud service failing is much smaller than the likelihood of any in-house service suffering a terminal failure.
This does, BTW, currently tend towards preferring big centralised services; if (to choose a random example) Salesforce.com goes under, your creditors are much more likely to accept that this is an exceptional event worthy of offering special terms than if your office server explodes in a shower of sparks and cannot be replaced in a reasonable time period. Not least because there's a decent chance that your creditor can't tell what you owe them until they recover from the shock, too.
As a non-business example, my continuity plan for "home hit by a hurricane" is "panic!".
Restarting the free accounting search
Posted Jul 31, 2017 17:55 UTC (Mon) by pizza (subscriber, #46) [Link]
We came close -- Last summer, a hurricane turned a few degrees to the north at the last moment and missed us entirely. Since then, the worst has been a power outage and an internet service outage, each completely shutting us down for about half a day. (Ironically, the same things that make us more resilient to major disasters makes us more vulnerable to the more run-of-the-mill stuff..)
But your point about being multi-site is well taken; our "continuity" plans only really work because we're so geographically dispersed. Fortunately, this particular site has never had to go into full disaster _recovery_ mode.
(BTW, I am familiar with how bad Legionniare's can get, having had a rather close encounter with it in my youth)
Restarting the free accounting search
Posted Jul 31, 2017 19:56 UTC (Mon) by anselm (subscriber, #2796) [Link]
(BTW, I am familiar with how bad Legionniare's can get, having had a rather close encounter with it in my youth)
When I was a student at Edinburgh University in the late 1980s, Legionella bacteria were found in a water tank of the air-conditioning system that served the university's computing centre. This was during an early summer that by Scottish standards was quite warm, and running the computers without air conditioning was not an option. Hence the computing centre was duly shut down for at least a week – I don't quite remember the specifics – so the A/C units (all of them) could be thoroughly sanitised. While the operations staff tried to keep a rudimentary service running, this was a huge hassle as lots of people had exams and deadlines for papers or projects coming up and couldn't do the requisite work. For a company, it would probably have been even more of a problem.
So, Legionnaires' disease is not to be trifled with. I don't think anybody actually contracted it at the time, which was just as well.
Restarting the free accounting search
Posted Jul 31, 2017 14:28 UTC (Mon) by mfuzzey (subscriber, #57966) [Link]
The difference between cloud services and most of the physical services (buildings / hotels / internet connectivity etc) is that the physical services are generally commodity and it is at least *possible* (which doesn't necessarily mean easy nor cheap, especially in a rush) to find a substitute.
When the data in the cloud is *your* data (be it accounts, code or whatever), it is, by definition, unique to you and hence irreplaceable.
Now, of course, you can also loose all your data on your own servers if you don't have proper backups etc in place but that is at least your own fault.
Many people assume that the cloud means the cloud provider handles all that.
While they will (one would hope at least!) transparently handle disk crashes etc while you have a contract with them, they may not provide a way for you to get your data out (conveniently or at all) if you decide to leave them or they cease to exist.
Maybe you can, in some circumstances, sue a cloud provider who looses your data - but that won't get it back if it's really lost (provider goes broke, auctions off all the servers and the disks are erased...)
Restarting the free accounting search
Posted Jul 31, 2017 15:48 UTC (Mon) by smurf (subscriber, #17840) [Link]
In contrast, if you store your database in the cloud you can trivially do a SQL dump and be back online with your own SQL server. Or move to some other AWS-compatible hoster.
In contrast², if you run the accounting system on your own hardware you can at least limp along, and you have time to look for another solution or a migration path, if/when your software provider goes belly-up.
Restarting the free accounting search
Posted Jul 31, 2017 16:09 UTC (Mon) by felixfix (subscriber, #242) [Link]
Worrying about cloud failures is pointless in comparison. The real world doesn't seem to have many cloud problems.
Restarting the free accounting search
Posted Jul 31, 2017 17:33 UTC (Mon) by pizza (subscriber, #46) [Link]
The real world, however, has many _connectivity_ problems.
Restarting the free accounting search
Posted Jul 31, 2017 17:34 UTC (Mon) by felixfix (subscriber, #242) [Link]
Restarting the free accounting search
Posted Jul 31, 2017 17:39 UTC (Mon) by felixfix (subscriber, #242) [Link]
Restarting the free accounting search
Posted Jul 31, 2017 19:01 UTC (Mon) by pizza (subscriber, #46) [Link]
It doesn't matter if the service outage is caused by incompetence by the provider, bankruptcy, hackers, puntive price increases, popular uprising, road work, solar flares, a rogue ISP in Pakistan pushing bad routes, or a gardener with a shovel. Regardless of the reason, the symptom is the same -- the service is not available, and if you need it to function as a business, you're sunk until it's available again.
One has to plan for outages.
Restarting the free accounting search
Posted Jul 31, 2017 19:12 UTC (Mon) by farnz (subscriber, #17727) [Link]
But note that "an outage of this magnitude kills the business" is a reasonable plan, if an outage of that magnitude is expected to be a rare enough event. You could, after all, plan for "aliens obliterate the continent I'm currently on", but it's such an unlikely event that there's no point planning for it.
Similarly, a small business may reasonably plan on the basis that loss of a cloud service for more than (say) 5 days kills the business, and that they'll ensure that they're close enough to up-to-date to cope with a 5 day outage.
Restarting the free accounting search
Posted Jul 31, 2017 19:46 UTC (Mon) by drag (guest, #31333) [Link]
Losing access to data or having data breach is a existential threat for almost all types of businesses. Much more serious then, say, being kicked out of a building.
If you depend on cloud providers too much and don't have a alternative way to retrieve and use your data then you are tying the fate of your business into the fate of your cloud providers. Their accidents can mean you are left dead in the water.
Even the best cloud provider has 'lost accounts'.. virtual machines go dead, customers are kicked out of their own accounts, etc. etc. Stuff happens. Their loss is your business. Your loss could be your business, as well. Your risk is much higher then theirs.
I don't have a problem with cloud services, but it really doesn't change much when it comes to dealing with backups and restores.
Restarting the free accounting search
Posted Aug 10, 2017 9:10 UTC (Thu) by oldtomas (guest, #72579) [Link]
https://www.theregister.co.uk/2017/08/06/cisco_meraki_dat...
(Cisco, of all things).
The other recent one I remember was also a big name (perhaps the one with the A? Not sure). Interesting -- in both cases it was (at least officially) "human error", the equivalent of the classical "rm -Rf" at the wrong time/place.
I'm waiting for the first "automatic error" -- when all those Puppets, Dockers, Lambdas and Unikernels and things are managed by some deep-learned network. That will be... weird.
Restarting the free accounting search
Posted Aug 10, 2017 18:02 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]
It didn't result in any data loss, as far as I'm aware there have been no data-loss events with S3.
Restarting the free accounting search
Posted Aug 11, 2017 15:31 UTC (Fri) by Wol (subscriber, #4433) [Link]
So why is it that pretty much ALL companies that suffer a serious failure in their accounting systems cease trading within the year ...
While a failure anywhere else usually only causes serious disruption for a couple of months.
Mind you, thinking about your tyre analogy, maybe tyres ARE the most critical part of the vehicle, after all, a motorway blowout can easily kill you instantly. Most other failures can be repaired.
Cheers,
Wol
Govt requirements
Posted Jul 30, 2017 0:07 UTC (Sun) by smurf (subscriber, #17840) [Link]
* be able to select how to do sales tax; file them on billing or on payment?
* lock down individual transaction logs immutably. These days the best way to do that would probably be to hash the file, and store the hash on some public blockchain.
* actually *have* (multiple) transaction logs, not just ledgers. The difference is important: locking a ledger means you can't change the VAT after filing it / your bank account balance after reconciling with the bank's end-of-year statement, while locking the transaction logs involved in the VAT / bank account means you can't change the other legs of the entries, i.e. which invoice triggered the VAT payment / which accounts the balance was credited to. The latter type of locking is mandated by law in some jurisdictions.
Govt requirements
Posted Aug 12, 2017 22:15 UTC (Sat) by Wol (subscriber, #4433) [Link]
A guy said he'd been asked by the accountants to provide a utility that would permit modifying any field in the accounts. He was suspicious, but the accountant people pressed him so he obliged.
A while later, the fraud people came in, and when they discovered this little utility, the accountant bunch said "that programmer wrote it, blah blah blah" so the fraud guys knocked on his door.
Unknown to the accountants ... the programmer said "Oh, that utility? Here's a log of *every* time it's been used, the change made, and the user who made it". The fraud guys had a field day as they went through it :-) Talk about covering your backside :-)
(The accountants had been moving sales figures around so that things "looked good" - dunno why but something to do with moving things between quarters.)
Cheers,
Wol
Govt requirements
Posted Aug 14, 2017 9:36 UTC (Mon) by sdalley (subscriber, #18550) [Link]
Restarting the free accounting search
Posted Jul 31, 2017 3:25 UTC (Mon) by gfa (subscriber, #53331) [Link]
I wonder if 2017 will be the year of the grsecurity flame, it is time for systemd to move out of the spot and let others to be flamed.
LOL
Restarting the free accounting search
Posted Aug 6, 2017 18:32 UTC (Sun) by GoodMirek (subscriber, #101902) [Link]
Restarting the free accounting search
Posted Aug 7, 2017 13:58 UTC (Mon) by ortalo (guest, #4654) [Link]
Another question more on topic, why accept that payroll processing be done as a service? Is it so different from accounting (software)?
NB: Personnally, I have been using KMyMoney since 2009 and I am pretty happy about it - though of course, personal finance managers may be far from having all the features you expect for a business.
Restarting the free accounting search
Posted Aug 7, 2017 14:24 UTC (Mon) by pizza (subscriber, #46) [Link]
Restarting the free accounting search
Posted Aug 7, 2017 14:26 UTC (Mon) by corbet (editor, #1) [Link]
"That Colorado thing"With regard to payroll processing: it's complex. Taxes to various governments, paperwork for various governments, withholdings for employee benefits, direct deposits, electronic filings, etc. Multiply that if you have employees in more than one US state. Payroll services are set up to deal with all of that; the services is cheap these days, to the point that it's just not worth even thinking about.
Restarting the free accounting search
Posted Aug 11, 2017 15:42 UTC (Fri) by Wol (subscriber, #4433) [Link]
Cheers,
Wol
Restarting the free accounting search
Posted Aug 7, 2017 14:34 UTC (Mon) by karkhaz (✭ supporter ✭, #99844) [Link]
After reading that sentence, I glanced up to check whether LWN-Tux likes to enjoy a spliff with his morning paper. But he seems disappointingly sober.
Restarting the free accounting search
Posted Aug 15, 2017 1:54 UTC (Tue) by fest3er (guest, #60379) [Link]
Restarting the free accounting search
Posted Dec 9, 2017 3:37 UTC (Sat) by benreu (guest, #120149) [Link]
Restarting the free accounting search
Posted Dec 9, 2017 8:49 UTC (Sat) by smurf (subscriber, #17840) [Link]
Restarting the free accounting search
Posted Dec 9, 2017 12:17 UTC (Sat) by benreu (guest, #120149) [Link]
Restarting the free accounting search
Posted Dec 10, 2017 14:12 UTC (Sun) by smurf (subscriber, #17840) [Link]
This is one of the things Gnucash gets right. Check out their transaction/split data structure.
NB: you might want to "git push" the code …
Restarting the free accounting search
Posted Dec 11, 2017 23:34 UTC (Mon) by benreu (guest, #120149) [Link]
Uhm... just to clarify, I am not looking for more work. I have a lot of features I would like to implement that I would use, not counting all the features people have already told me I should add ;-)
I posted here because it seemed like this is a fairly recent conversation. I thought maybe some of these people would be interested in helping and/or collaborating. If you really want to help or influence the direction of PyGtk Posting, please contact me through email or start a new discussion here https://sourceforge.net/p/pygtk-posting/discussion/
I will not clutter LWN with random noise, therefore consider this my last post here.
Copyright © 2017, Eklektix, Inc.
This article may be redistributed under the terms of the
Creative
Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds