Designing distributed invoice number counting system

Question

I am looking for a way to design a system that can provide a linear and incremental counting for invoice number accros a scalable system.

At this time, I have four pools of two servers (two pool for europe and two pool for america -> total eight servers ). This system handle sports subscriptions and can generate invoice at two moment :

• When the customer subscribes for the first time, all along the day, especially before a match or a sport event

• When the subscription comes at his until date and renew, we run a batch that seeks all subscriptions to renew at the rate/parallelization we want.

All theses subscriptions are distribued through an amqp and federated exchanges , e.g : pool1 handle all subscriptions coming from the load balancer (lb1) and put them into a queue (queue_pool1), then later, the queue is depiled by servers of pool1 and subscriptions are handled into pool1, except when the queue_pool1 is full, so the excess messages go into the federated exchange that feeds queue to the exchange of the pool2 (queue_pool2).

At this time, we use a linear counting based on a timestamp +micro seconds base10 to 36, when we generate it, we put it on a shared memcached and a check is provided into this memcached before using the number invoice generated, but for legal purpose we have to change it to an incremental counter. However, we can use a dedicated counter with a prefixed indicator like this I1-NNNNNNN, I2-NNNNNNNN, but not holes between numbers invoices. etc. (Best would be an unique counter for all servers.)

When we run renews, the rate is around 20/40 subscriptions in parallel. Depending on our partner api payment, we are at 10-20 renews per seconds. We can control the rate.

When we have a big event, we are around 5 to 10 subscriptions per seconds. We can control the rate but if the rate is too slow, customers will expect delay into their services activation.

Each pool handle almost around 1000 requests per second for all incoming traffic.

• Steve: You need 8 servers to handle a load of 10 invoices per second? And you can't logically have a counter that is both distributed and incremented sequentially in time order, but at just 20 increments per second, why on earth would you even wish to distribute it?
• mik3fly-4steri5k: well, servers are on different datacenter, we expose only 4 at time, by doing active-active. The goal is to have 4 servers at any time. You have to take in count, that for one subscription, you can have almost 1000 others request per second of any type (cart, user account creation, eligilibity, resolving voucher code ..) @Steve
• Stack Exchange Broke The Law: Have you considered running one copy of one service that counts invoice numbers? It should be able to handle 100000 invoice numbers per second, or so.
• mik3fly-4steri5k: Behind each invoice, we call our payment api that takes between 20-80ms to execute. At his best, we can do 500 invoices per second for each server @user253751,
• Steve: Have you considered decoupling the payment-taking process from the invoice generation process? It is quite normal in business for invoices to be raised some time before from payment, and in the event the expected payment fails you can make compensating transactions (i.e. cancel the invoice). I suspect you're trying to make it all part of one atomic transaction rather than following a normal accounting process.

Answer 1

Indeed, many countries around the world, enforce a legal requirement to use sequential numbering for invoices. The aim is to reduce tax fraud: non-reported invoices (potential tax evasion) are easy to spot. In some countries, the sequential numbering must even be chronologically consistent with the time stamps.

Remark: In some countries, this requirement is not absolute: you can have a small gap here and there, provided you can justify the reason of the gap in a convincing manner. Be aware that what could convince a software engineer might look likean attempt of fraud by a tax auditor, the kind of persons who are relatively impermeable for tech-speak and tech reasons. So better keep the logs of those crashes that made you loose numbers...

Strong sequential numbering is a hard constraint and bottleneck:

• you can have only one counter.
• Moreover, the counter must be persisted all times to cope with system failire (i.e. no caching possible).
• Finally, the counter should be managed in the same transaction than the invoice, so that either the invoice is created and the number used or the invoice fails and the counter is not updated.
• As a consequence, invoice creation would be a sequential bottleneck.

Potential implementations that meet these requirements are:

• Traditional architecture: use one single database for invoicing. The database server will be the bottleneck. The only scaling is vertical.

• Synchronous service architecture: use a sequential numbering service. However, if you want to avoid gaps in the numbering, you'd need to use a 2PC protocol, which might align the overall performance on the slowest node. Or increase parallelism, if you do not have a chronological conistency constraint in your country and could hence reuse a number that was assigned to a failed invoice creation.

• Asynchronous service architecture (typically microservices): Use a single invoicing service, that generates invoices asynchronously in response to queued invoicing requests according to the saga pattern. This allows to absorb peaks accumulated in the queue in less busy moments of the day.

In many countries, it is however allowed to have several independent series of sequential numbers. This requires each serie to have a different prefix or range, and each without gap. You need to check this with a chartered accountant in your tax jurisdiction. In this case, you may use a different numbering for new subscriptions and for renewals (this is the kind of arguments that is convincing for tax auditors, because it's easy to understand that there's a different invoicing process for both cases). Eventually, you may consider N independent ranges for N independent server nodes. The simpler the distribution scheme (by shop, by geographical area, ...) the simpler it can be explained to/understood by a tax auditor.

• Steve: "what could convince a software engineer might look likean attempt of fraud by a tax auditor, the kind of persons who are relatively impermeable for tech-speak and tech reasons" - those silly fools! How dare they reject "tech reasons", and expect "software engineers" to abide by laws and accounting practices! There is a serious lesson here though, that almost anything concerned with the recording of money transactions in a business is the domain of a regulated profession called accountancy, and there are laws and practices that must be followed under threat of criminal penalties.
• Christophe: @Steve ;-) I think they just try to do their job to the best possible extent, but in a domain where they cannot afford to trust what they cannot understand. It's certainly not without reasons. Invoices and cash registers are more and more regulated (see this document page 12-13). In some countries you even have to get such products certified. Fortunately, tax authorities have now special departments that deal with tech controls & approvals
• Christophe: @Steve by the way, software engineers are not only expected to abide by laws but also to know these in their domain. When they don't, they risk jail, as numerous examples show (e.g. Diesel gate, filehsaring software and many more)
• Steve: Agreed. Because systems exist which are understandable, robust, flexible, and auditable, auditors and tax inspectors have every reason to reject that which cannot be understood, because at the very least the un-understandable is written by amateurs who will probably calculate things wrong and lose large numbers of records by accident, and at worst it will be written to conceal systematic fraud. The starting point for a developer of an accounting system, should be to look at (or ask) how the law says accountants must do it, not to invent a new system that is inflexible and unlawful.
• Steve: Yes it's quite bizarre how many developers seem to think that no laws apply to their work, that there is no accountability whatsoever.

Answer 2

You can't distribute an incremental counter practically by definition. (as Steve points out) And as you point out there is a legal requirement for invoices to have an incrementing counter.

However, I think you are missing other ways around this problem.

Things to note:

• Invoicing and taking payment are two separate things.
• Creating an invoice and sending it to a customer are two different things
• Invoices can be canceled by credit notes

I would suggest the following solution

1. Have a single SaveInvoice() API which takes an invoice without the number and saves it, adding the incremental invoice number and adding it to your accounting software
2. You can now distribute the invoice calculation. (if that is CPU intensive)
3. Have a single GetNextPendingInvoice() API, which returns (locking as appropriate) a pending invoice
4. Have a Single AddPayment() API which adds payment to your accounting system and marks invoices as paid.
5. You can now distribute your payment collection, which will usually be the slow bit. Looping over unpaid invoices and attempting to take and record payments.