Xero Transaction Limits: Nov 2018 test results
There are questions about how well Xero performs under "higher loads". Our first concern was for medium-sized wholesalers. Xero talks about limits as low as 1000 'transactions' a month. Our testing shows that Xero performs well at much higher transaction volumes. The first test results are based on 3 years of 2500 transaction rows per month (and then matching payments uploaded via a bank CSV). The second test run was much larger.
Note that after some discussions with Xero and based on our own experience, we (GrowthPath) believe that businesses can interpret the 'limit' of 1000 invoices a month as advisory. This limit does not form part of the terms of service. In Jan 2017 Xero published some small changes to the API limit, and these limits, which are enforced, allow a much, much higher load than Xero's advisory limits. Hence, stress testing is relevant because you will reach the practical limits of Xero before you exceed API limits. Exactly what are those practical limits? That's the point of this article.
For a medium sized wholesaler, we assume that Xero is the backend to a cloud-based inventory package, such as Unleashed or Trade Gecko, so we did not use Xero's inventory module.
Compared with a B2C (retail) business, a wholesaler has fewer customers, fewer invoices but invoices which have many more lines. Most sales are on credit, so reconciliation of bank feeds is important.
Testing was initially carried out in June 2015 with a standard Xero trial account and then updated in November 2018 with much higher loads, in preparation for a talk at Melbourne AccountTech 2018. The browser used was Firefox on a fast Linux notebook. The testing environment ranged from an office 100 Mb/s connection (at the computer) to 5 Mb/s Indonesian hotel wifi.
Data was created with python scripts and automatically imported with CSV files, and browser automation was used for bank recs, document approvals and simultaneous invoice entry, using Python & Splinter. We also did some uploads via the API mainly to load Xero for testing. GrowthPath can perform custom load testing as a consulting assignment if these results do not provide the required insight.
How does the number of invoices affect Xero's performance?
There are two patterns of use a system can see. There is "offline", where a user is doing reports and inquiries while there is little or no live transaction entry , and there is transactional use, when sales are being entered.
The back ends of modern computer systems can handle massive amounts of data very easily, so it is actually surprising to hear that cloud systems hit performance problems at fairly low levels of transactions, although it is precisely these complaints that lead to this testing.
The first phase of testing looks at how the size of data in a Xero file affects reporting. Because the data was loaded in flatfile before testing began, the testing activities described below were the only use of the file at that time. This is like a business doing reporting after hours when there are no orders being entered, for example. Therefore, this testing looks for the impact of historical data on reporting and searching performance.
We learnt these things about Xero and transaction load (Nov 2018 testing):
a) Total transactions load is not a factor. We loaded large amounts of data, and saw no degradation of performance loading data or entering invoices. That is, a CVS load of a thousand invoices was as fast the 80th time as it was the first time. This is good news if you are using Xero as a backend for an ERP system.
b) Reporting performance gets slow if the report downloads many rows, that is, if it is detailed report, such as an account transactions report. And "download" is the key word: Xero's reports first compile all data on the server, and then download all the data. This second phase can be slow or very, very slow, depending not on Xero, but on your bandwidth. Your browser can lose patience waiting for the download, making it look like Xero has a performance problem, when it doesn't (although you can argue that Xero could download pages of data at a time, rather than the whole lot). We consider that our testing broke Xero when the failure happened on the server.
c) the legacy BAS report ("Full BAS") is as terrible in 2018 as it was in 2015, but now there is an alternative: the Simple BAS, which is much faster but the Simple BAS audit report is still an early failure: it it the weakest link.
Considering that the BAS Audit report is required functionality, we conclude that our Xero broke at a monthly load between 300K and 400K invoice lines. (which is 60,000 to 80,000 five-line invoices), if we can use the SImple BAS.
This is obviously much bigger than Xero's suggested limits.
Xero Simultaneous users: failure with three or more people entering sales orders.
Xero makes no licensing limit on the number of users. Our 2018 testing reproduced a problem we saw in 2015.
However, we discovered in our testing that simultaneous entry of invoices causes problems. Xero assigns a next invoice number when a user starts entering an invoice. This number is not reserved, so if another user starts a new invoice, it will get the same number. In fact, five users keying in invoices would all get the same number; a number is only reserved when an invoice is saved (including saving as a draft, as well as sending it straight to approval). Xero attempts to fix duplicate numbers when the invoice is saved, but this approach doesn't work very well. Our testing shows that with more than two users entering invoices at the same time, invoices are overwritten.
We did not find a workaround for this, part from the obvious one: use a front-end sysem such as a cloud ERP for order entry.
Manually entering invoice numbers is possible, but it would require each user to have a pool of numbers to use, and each user would need to manually keep track of the next number. A number is reserved when you save and continue editing, but this only works if you can guarantee that only one user does that at any given time. Xero is not ready for >= 3 users doing order entry via the browser. Possibly it would be ok with two, but more than that is asking for trouble (lost orders).
2018 Xero High Transaction Volume Testing Results: Part 1
The testing database has data spread over 3.5 years.
Apart from the AR data here, it also has >30K bank lines (mostly reconciled as AR payments), and >30K AP documents. The sales data was spread over 50 customers.
Expenses were spread over about ten accounts. GST is accrual method.
A Xero trial account was used, set to Australian locale. Australian GST is a value added tax. Every AR & AP invoice line in the test database was subject to GST.
The highest montly load tested was 400K lines a month. That's about non-stop entry of 40 invoices lines a minute, 8 hours a day Monday to Friday.
We rate the upper usable transaction limit to be around 300K lines a month. There seems to be no limit on the historical volume of data.
| Apr 2020 | Jan 2020 | Jan 2017 | March 2018 | Feb 2018 | Jan 2018 | |
| Low | Low | Low | Medium | High | Extreme | |
| AR Invoices (@25 lines | 250 | 500 | 1,000 | 6,000 | 11,000 | 16,000 |
| AR Lines | 6,250 | 12,500 | 25,000 | 150,000 | 275,000 | 400,000 |
| Task | Results | |||||
| Enter Sale | FAST | FAST | FAST | FAST | FAST | FAST |
| P&L, Month/Qtr* | FAST | FAST | FAST | FAST | FAST | FAST |
| P&L, Sales drill down | FAST | BANDWIDTH | BANDWIDTH | BANDWIDTH | BANDWIDTH | BANDWIDTH |
| Simple BAS, Month | FAST | FAST | FAST | FAST | FAST | FAST |
| Simple BAS, Month: Audit Report | SLOW | SLOW | SLOW | TIMEOUT | ||
| Balance Sheet | FAST | FAST | FAST | FAST | FAST | FAST |
2018 Xero High Transaction Volume Testing Results: Part 2
| Apr 2020-June 2020 | Jan 2019-March 2019 | Jan 18 to March 18 | |
| Low | Medium | High | |
| AR Invoices (@25 lines | 750 | 3000 | 33,000 |
| AR Lines | 18,750 | 75,000 | 825,000 |
| Task | Results | ||
| Enter Sale | FAST | FAST | FAST |
| P&L, Month/Qtr* | FAST | FAST | FAST |
| P&L, Sales drill down | BANDWIDTH | BANDWIDTH | TIMEOUT |
| Simple BAS, Month | FAST | FAST | FAST |
| Balance Sheet | FAST | FAST | FAST |
| Report results key | ||
| FAST | < 10 seconds | |
| SLOW | < 60 seconds | |
| VERY SLOW | > 60 seconds but succeeds | |
| TIMEOUT | timeout | |
| BANDWIDTH | Performance is limited by your network speed, not Xero. Once testing confirmed the download began before a timeout, a score of BANDWIDTH was entered | Note: the Account Transactions report, the report used for P&L drilldowns, Xero compiles and downloads all data to your browser. It is then very fast to move betwen pages, since you already have all the data. An alternate design more common in large data applications is to send only one page at a time to your browser: this makes the initial response much faster, but it is slower to move between pages |
Xero Performance: Conclusions
This testing was quite rigorous, although it is not real world testing. Apart from the bug which makes multi-user entry of invoices at risk of data loss, the performance was much better than we expected. We stuck to wholesaler-type scenarios, where transaction volume appears as invoices were quite a few lines, but a relatively small number of customers (well within the 5000 limit suggested by Xero). The total transaction volume was much, much higher than the limits Xero mentions, and performance was good.
There are real world reports of Xero performing poorly under higher load. We would like to investigate those. We predict that these problems are likely to be in businesses with many customers and a constant stream of invoice load, perhaps via the API which is sending invoices real time in batches of one.
Work arounds: Consolidation
The most common workaround for poor transactional performance is to consolidate entry via a separate system. This means that the invoice line details of sales are merged before exporting to Xero. A retailer will usually have a Point of Sale system which records the high volume of transactions, with only summary records going to the finance system. A wholesaler may use a cloud ERP system such as Unleashed, Dear or Cin7 (reviewed here) to capture the details of sales and inventory transactions, sending only consolidated sales to Xero.
Inbal Steinberg of ConvertWorx helped one of her clients by developing a retrospectice consolidation script, which consolidated invoices already loaded to Xero ; this approach was very successful.
Custom Stress Testing
If you are thinking of changing to Xero and are concerned about whether it will handle your business, please get in touch with us. We will design a test scenario matching your business, and we can explore how Xero performs.