By chance, that customer I'm referring was also in
the gambling sector and also had operations in South America, indeed
they operated worldwide, but I was consulting for them because they
wanted to address several kinds of fraud they had discovered in their
slot machines in certain countries in America. So this article
sparked my memories, and, as I think it was an interesting project
and there some lessons to learn, here you have a small article
describing it.
General View
As I said, the customer I was prospecting at that
time -I will not disclose its name-, operated casinos all around the
world with a lot of success. However, from time to time they
discovered certain kinds of frauds or even bugs in their slots
machines or casino systems they wanted to address. The main problem
with this kind of issues it is not to find them out, but when do you
find them out. If you have a system that alerts you as soon as a
fraud -or a bug- is happening, you can stop the loss immediately and
remediate it quickly. If, on the contrary, you allow it to run for
days, weeks or even months, the loss increases as a snowball running
downhill.
Problems to Address
One example of a bug was in relation with one of
their slots machines. It was not their more popular machine, it
offered a new game that was only used by a marginal group of users.
However, at the end of a given month, when the usage data of the
machines arrived the headquarters, the team of business analysts
found that this specific type of machine had increased in popularity
in an amazing 4000%!!
For some weird reason suddenly everybody wanted to
use that machine. So they decided to understand why.
The next discovery was that, in spite of the spike in
the number of users, the machine was losing huge amounts of money,
but the ratio of winning/losing games did not favor the gamblers, so
something very funny was going on.
After retiring the machines from the casinos and
sending a team of engineers to find it out, they discovered the real
reason of the sudden popularity of the slot machine and why it was
losing money: the software had a bug, a very subtle one.
The machine allowed gamblers to pay with cash,
including bank notes, returning change of the amounts not used. The
bug consisted in the wrong recognition of a very particular kind of
bank note when running one single bet in the machine: when a $20 note
was inserted in the machine, and just one bet was used -valued in
$1-, the machine returned $99 in change instead of $19, as if the
note introduced was a $100 one.
It was a very weird bug, as it didn't happen if the
gambler used any other number of bets than one, and only with $20
notes. But word spread quickly among gamblers and they quickly
exploited this bug on their benefit. Unfortunately for the company,
they needed a couple of months to find it out.
[Note: I used $ as the currency to explain it in a
simple way, but the real currency was not in dollars]
To try to stop this things to happen again the
company opened a tender to invite vendors to provide a solution. At
that time I was working for IBM and we had a wonderful product for
real-time/streaming data analysis: Streams. Indeed, it is still being
sold by IBM, and allegedly is the best product in this category of
software -I'll write a comparison about this products in the future,
hopefully-.
The Solution
My proposal was relatively simple: Streams would
gather all the information from the different systems installed on
the casinos and would analyze it in real-time(1). Then, using stored
historical information, Streams would decide it the information was
normal or if there were any anomalies that an operator should check
in detail. Then, the information would also be stored in the
historical databae to be part of the knowledge base for future
reference in the search of anomalies, with an additional piece of
information: once an operator investigated the detected anomalies, it
would annotate them to inform the system if they were worthy to
investigate or if they were false possitives ths should be ignored,
improving the accuracy of the system.
If you think about this, it resembles an Artificial
Intelligence system...
Obviously, this is more easily said than done. The
project involved not only the use of the streaming technology, but
the processes to gather the information from the slot machines and
other casino systems, the design of the data-warehouse to store the
historical information, the algorithms to detect the possible frauds
in real-time, etc.
To add value to the project, my proposal also
included monitoring the video cameras in real time in search of
suspicious behaviours, not only from the players, but from the
maintenance personnel: we were told of cases where the maintenance
people acted in connivence with malicious gamblers installing the
most exotic devices inside the machines to win illegally.
But not only fraud could be addressed. Another
interesting feature that could be addressed with this project was
preventive maintenance and detection of failure in the systems. As it
happens with cars or other kinds of machinery, there are scheduled
maintenance intervals, but sometimes failures happen before the
maintenance periods are met, so by monitoring the systems in real
time you could learn when a system is about to fail, by using
historical information and anomaly detection. Then, by approaching a
100% availability of the machine, it means that the system is almost
100% of the time available to produce a benefit.
In sum, we realized that by just decreasing the fraud
in the systems by a 2% -a very easy and non-ambitious target-, the
project would not just pay for itself but produce additional benefits
in a 10 month period. The other added benefits -reducing maintenance
periods and other kinds of fraud- were not included in the proposal
as we wanted to stay on the safe side, but we estimated that they
could add another 12-15% increase in the final results of the systems
being monitored. Regarding fraud detection, the 2% estimation was
also a very conservative one. Our previous experience showed we could
easily achieve a 18-23% of fraud reduction, which is a significant
amount of money as they estimated the undetected fraud in more than
$100M
Aftermath
What happened after all? The customer decided not to
buy our solution. We may have done if for free and claim only a
percentage of the fraud detected, and it would have been the project
of the year but our internal rules did not allow that, and anyway the
customer didn't want us to do it anyway.
As I said, they asked several vendors and consultancy
firms to propose solutions, and they trusted the most their own
internal auditor, one of the big 4, so there was no room left for a
pure technical provider.
Their auditor-consultant recommended to just audit
the results as they were doing, and not spend money on anybody else:
just in them... And the customer believed them. It happens a lot,
once you earn the trust of your client, it is more likely that they
will follow you even if it is not the best possible solution.
In any case, that company was later involved in
internal fights and the CIO left the company, so the project quite
likely would not have been deployed.
Conclusion
Well, it is you who have to reach your own
conclussions, right?
Just kidding. Regarding only the streaming project,
real-time (streaming) analysis of information is a very powerful tool
that can provide immediate results and mitigate, if not totally
avoid, future problems. And if used in combination with AI/ML
techniques and the proper use of historical information it is just
fantastic.
There has been a lot of advance in several
open-source platforms to address those kind of issues and with
relatively very small investment a lot can be achieved with these
solutions, but its use is not as widespread as I'd like it to be.
We'll have to work on this.
---------------------------------------------------
