Xero Transaction Limits and larger businesses
There are questions about how well Xero performs under "higher loads". Our first concern was for medium-sized wholesalers. Xero talks about limits as low as 1000 'transactions' a month. Our testing shows that Xero performs well at much higher transaction volumes. The first test results are based on 3 years of 2500 transaction rows per month (and then matching payments uploaded via a bank CSV). The second test run was much larger.
Note that after some discussions with Xero and based on our own experience, we (GrowthPath) believe that businesses can interpret the 'limit' of 1000 invoices a month as advisory. This limit does not form part of the terms of service. In Jan 2017 Xero published some small changes to the API limit, and these limits, which are enforced, allow a much, much higher load than Xero's advisory limits. Hence, stress testing is relevant because you will reach the practical limits of Xero before you exceed API limits. Exactly what are those practical limits? That's the point of this article.
For a medium sized wholesaler, we assume that Xero is the backend to a cloud-based inventory package, such as Unleashed or Trade Gecko, so we did not use Xero's inventory module.
Compared with a B2C (retail) business, a wholesaler has fewer customers, fewer invoices but invoices which have many more lines. Most sales are on credit, so reconciliation of bank feeds is important.
Testing was carried out in June 2015 with a standard Xero trial account. Browsers used were Firefox and Chrome on a Macbook. Data was imported with CSV files, and some browser automation was done for bank recs and document approvals, using Python & Splinter. We also did some uploads via the API mainly to load Xero for testing.
How does load affect Xero?
There are two patterns of use a system can see. There is "offline", where a user is doing reports and inquiries while there is little or no live transaction entry , and there is transactional use, when sales are being entered.
The back ends of modern computer systems can handle massive amounts of data very easily, so it is actually surprising to hear that cloud systems hit performance problems at fairly low levels of transactions, although it is precisely these complaints that lead to this testing.
The first phase of testing looks at how the size of data in a Xero file affects reporting. Because the data was loaded in flatfile before testing began, the testing activities described below were the only use of the file at that time. This is like a business doing reporting after hours when there are no orders being entered, for example. Therefore, this testing looks for the impact of historical data on reporting and searching performance.
The testing we have done indicates that performance problems will come more from simultaneous load on the system rather than the volume of historical transactions. Even with transaction volumes 10 times what Xero indicates as the upper limit, Xero shows no ill-effects due to transaction history. However, the performance impact of transaction entry is greater.
Note that this is different to MYOB, which suffers serious performance problems when the volume of historical data gets "high" (and that limit is very low due to the rudimentary database backend in use), as well as not performing well under transaction entry. The transaction volume entered here would severely stress MYOB.
Xero Simultaneous users
Xero makes no licensing limit on the number of users.
However, we discovered in our testing that simultaneous entry of invoices causes problems. Xero assigns a next invoice number when a user starts entering an invoice. This number is not reserved, so if another user starts a new invoice, it will get the same number. In fact, five users keying in invoices would all get the same number; a number is only reserved when an invoice is saved (including saving as a draft, as well as sending it straight to approval). Xero attempts to fix duplicate numbers when the invoice is saved, but this approach doesn't work very well. Our testing shows that with more than two users entering invoices at the same time, invoices are overwritten. This is actually the first real problem our testing showed while investigating Xero at higher loads. It's a bad bug.
We did not find a workaround for this. Manually entering invoice numbers is possible, but it would require each user to have a pool of numbers to use, and each user would need to manually keep track of the next number. A number is reserved when you save and continue editing, but this only works if you can guarantee that only one user does that at any given time. Xero is not ready for multiple users doing order entry via the browser. Possibly it would be ok with two, but more than that is asking for trouble (lost orders).
Load Scenario 1: Medium sized wholesaler ($3m to $5m)
Three years (36 months), each of 70 invoices per month with 20 lines, GST on. Five customers. Bank statement: one receipt per invoice, auto-matched. This may not be every "$5m wholesaler" but it is about right.
Therefore each month contains about 2500 transactions (invoices lines and bank feed). In our test environment we did not include AP, payroll: therefore it misses transaction volume in these areas. However, a business which is scaling up is likely to more likely to see transaction volume growth in the area of sales. There are no inventory transactions in this scenario because they are managed in the supply chain front end. If you are using Xero for detailed inventory transactions, our test scenario may not be accurate.
For this test, transactions were loaded via CSV (so we did not stress test the API). The files were loaded one per year (so one file of invoices, and one bank statement).. Each file took only a few minutes to upload and approve the resulting draft invoices. The bank statements also loaded very quickly. The upload was designed so that Xero would immediately recognise matches (each invoice totalled to a unique amount, matching receipts on the bank statement). However, there is no way to automatically approve the suggested matches: it is not done via the API. This was done with a script which automated the browser interface, simulating a human clicking the "OK" buttons.
Performance tests and results (after loading three years of transactions)
|Manual entry of a sale||Quite fast. Saving the invoice took three seconds. It is about the same as an empty database.|
|Manual entry of a journal||Less than three seconds. This is a normal performance|
|View contact record and then sales per contact. Drill down to an invoice||Fast as normal|
|!2 month P&L||A few seconds|
|GST Reconciliation Report||Less than five seconds for an entire year|
After this transaction load, Xero performs normally. There is no evidence of any slow-down in reporting or transaction entry.
Load scenario 2: Larger wholesaler ($10m to $15m)
Three years (36 months), each of 200 invoices per month with 25 lines, GST on. Fifty customers.
Plus 200 supplier invoices per month, of 5 lines each. Fifty vendors. This is about 6500 lines per month (including the bank feeds showing payments).
Bank statement: one receipt per invoice (for both AR & AP), auto-matched.
Results: Performance due to transaction history
These tests were carried out as the only user on the system, when no processing was taking place (the 'offline' mode). Therefore these tests show the impact of the historical data loaded, but not the impact of those transactions actually being entered live (the "simultaneous user" impact).
Summary: in this test database, which ended up with even more data than indicated here, offline performance was fine. Reporting, inquiries and order entry were fine, virtually indistinguishable from the smaller test set. Any performance issues with Xero seems unlikely to be due to the sheer historical volume of data.
|Test||Results (when the only user)|
|Manual entry of a sale||It is about the same as an empty database.|
|Manual entry of a journal||Less than three seconds. This is a normal performance|
|View contact record and then sales per contact. Drill down to an invoice||As Fast as normal|
|!2 month P&L||Less than three seconds (even with one other busy session). Normal.|
|GST Reconciliation Report||Less than five seconds for an entire year. For three years, still only a few seconds.|
Evidence of a CPU bottleneck to due to simultaneous activity
While one session was importing invoices, an interactive session creating a new sales order faced a 45 second wait to save an invoice so performance was really suffering. This is a very intensive action.
However, while one browser session was interactively approving draft invoices, another session entering invoices experienced typical performance. New invoices were consistently saved in less than five seconds.
Using the Xero API to load invoices while entering invoices and running reports: Performance impact
We decided to upload invoices via the API using Python 3.5 (using the PyXero library). The Xero API is limited to 5000 calls a day (according to documentation (Jan 2017 update)), but each API call make more than one invoice. Initially we loaded 100 invoices per call, as Drafts, but this was too quick for our purposes: we wanted to do some manual interactions with Xero in the browser while the load was occurring. So we reduced it to 5 invoices per time: the time taken for API calls is more related to making a call, more than what you do with it, so it's more efficient to load 100 invoice per time than 20 calls of 5 each. The invoices were the same 25 line invoices as above. So incidentally we observed that the API also performs well despite a very large volume of history in the Xero file, and that with decent design, the API limit of 5000 calls can actually support a very high number of transactions. Before the Jan 2017 update, the limit was 1000 a day, and we considered that sufficient if used wisely, so the new limit is more than enough.
While this import was running, we tried reporting and invoice entry. We found the performance to be acceptable and close to normal. There were occasions were steps became slow, but we couldn't reliably repeat this.
Xero Performance: Conclusions
This testing was quite rigorous, although it is not real world testing. Apart from the bug which makes multi-user entry of invoices at risk of data loss, the performance was much better than we expected. We stuck to wholesaler-type scenarios, where transaction volume appears as invoices were quite a few lines, but a relatively small number of customers (well within the 5000 limit suggested by Xero). The total transaction volume was much, much higher than the limits Xero mentions, and performance was good.
There are real world reports of Xero performing poorly under higher load. We would like to investigate those. We predict that these problems are likely to be in businesses with many customers and a constant stream of invoice load, perhaps via the API which is sending invoices real time in batches of one.
The most common workaround for poor transactional performance is to buffer entry via a separate system. A retailer will usually have a Point of Sale system which records the high volume of transactions, with only summary records going to the finance system. A wholesaler may use a cloud system such as Unleashed to capture the details of sales and inventory transactions. The Australian cloud online store Neto, for example, supports (out of the box) two solutions. Firstly, it supports businesses which are using Unleased to manage stock by sending orders there, and if Neto is directly connected to Xero, it offers a batching option where it consolidates invoices prior to transferring them to Xero.
Custom Stress Testing
If you are thinking of changing to Xero and are concerned about whether it will handle your business, please get in touch with us. We will design a test scenario matching your business, and we can explore how Xero performs.