Machine Learning in Finance

How AI is used for trading, fraud detection, insurance & personalised banking

There is a lot to be gained for finance businesses in applying AI. And - in fact - plenty are already doing so successfully. To give you an idea - here’s a summary of the top applications of AI in Finance.

AI for trading

We can’t talk about Machine Learning in finance without talking about Machine Learning for trading. In fact, there is so much to talk about that we dedicated a whole article to Machine Learning for Trading

I summarised most important points here:

AI doesn’t replace traders – It helps them

Traders need to process a very large amount of information – to stay up to date with what’s happening and develop informed hunches about where things are going.

They also conduct rigorous statistical analyses to decipher the details of small-market inefficiencies.

All of this takes time – and since trading is a competitive game, the faster you are, the more of the pie you get.

AI helps make traders faster and more accurate in their analyses by summarising news (e.g., with sentiment analysis), improving predictions of external factors (such as rainfall, commodity supply volumes, or election outcomes), and automating the fine-tuning of high-frequency trading machines.

Automated & profitable AI trading is mostly a fantasy

AI is still much weaker than human brains when it comes to making decisions in complex systems where you compete against other human beings - like you do in stock markets – so be wary of anyone that promises a profit based on pure-play AI systems.

There are lots of common mistakes that make it seem like this is possible – when in fact it isn’t.

For more details about why this is the case – and learn more about the ways you can use AI instead – check out our in-depth article on Machine Learning for Trading

Making portfolio management better & cheaper

Portfolio managers tell you where to invest your money – depending on your financial goals and the market fluctuations you can deal with (your risk tolerance).

They pick different types of investments and arrange a portfolio (a list of assets plus the percentage of your money that should go into each one). If the market changes – which it constantly does – this portfolio needs to be recalibrated by moving your money around between assets.

For a human being, this is a time-intensive process – so banks charge a lot for this service and only offer it to their wealthy clients. But good investment advice helps everyone, no matter how large your savings are.

That’s why there’s been a rapid rise in “robo-advisors.” Not only are they very cheap – they actually do their job better than more expensive, human portfolio managers.

Why? Because many human advisors can’t stop themselves from promising their clients superior returns via “secret tips” – which, in an efficient market, is always an illusion, and hence a bad idea. These “tips” end up adding a lot of risk to your portfolio – and might wipe out a big chunk of your savings. (I’ve seen this happen a couple of times – and it’s not pretty.)

The secret of why AI is so useful in portfolio management is that its almost purely a statistical process. Every factor that goes into determining your ideal asset mix (portfolio) can easily be expressed in numbers: your age, income, family status, when you’ll need money (for a house, kids, education, etc.), and your risk tolerance – all easy to enter as numbers in a database.

Based on this data, AI can predict your preferences (like when you’re likely to need how much money). Once you have this information, then the correct allocation of your money is simply a matter of solving a mathematical formula – which is a simple task for any computer.

Want to learn more about the theory behind statistical asset allocation? – Check out Modern Portfolio Theory.

Strengthening fraud detection

With increasing digital commerce (through credit cards and online payments), the incentive to commit fraud is also constantly increasing.

Traditional fraud models rely on simple rule systems – checklists designed to catch fraudsters. The biggest drawback of these simple means of identifying fraud is that they produce a lot of false alarms. Which means legitimate customers are denied purchase transactions – and they’re rightfully upset, and will go and buy somewhere else.

Machine Learning can produce more accurate fraud-detection checklists while also reducing false alarms. Why are AI models better? These are the Top 3 Reasons:

  • Data driven: Machine Learning models go through all the examples of fraudulent purchases and – free of human bias – find the exact patterns that differentiate fraudsters from customers.
  • More complex: Machine Learning models can easily capture thousands of small patterns and use them to check a transaction – something that isn’t realistic if you’re writing a checklist by hand. This means the models are more accurate and produce fewer false alarms.
  • Continually updating: Fraudsters are inventive and constantly try new things to trick the system. Once you’ve found fraud that your system hasn’t caught yet, it only takes a few minutes to update the model so it learns how to spot the new trick from now on. The great thing is: it doesn’t matter whether you’re fighting one inventive fraudster or thousands – an AI system can learn all their patterns just as easily, with no need to cut corners or generalise.

Fraud systems are mostly side-kicks, like algorithms in trading: a human being still needs to do the final check on all the transactions tagged as fraud. Why? Because 1-1, humans are still better at reading other humans’ behaviour – Machine Learning just makes our work easier.

Want to learn more? We wrote an article about using AI to better detect Mobile Click Fraud.

Making insurance underwriting more accurate & personal

Similar to portfolio management (above), insurance underwriting is a job with a clear set of quantified factors (inputs), quantified goals, and fixed outputs: the price of the insurance and its premiums over time.

This is the ideal environment for automated statistical decision-making* – meaning: Machine Learning.

For years, underwriters have already been using simpler statistical models to estimate the underwriting risk and decide on the right premiums.

Machine Learning models can go further and make almost the entire underwriting job cheap and scalable. The key is the insurance company’s data: the thousands of contracts they’ve already calculated – with human input – are the ideal basis on which to train a Machine Learning model to learn to write contracts itself.

Machin Learning models like this open the door to cheap, individualised underwriting – which is especially helpful in the B2C space (such as life insurance).

But that’s just the beginning – the insurance business is stastical through-and-through, so there are many more AI applications. Here are two examples:

Good drivers pay less for car insurance

Some insurance providers are already predicting your risk of having an accident based on the way you drive (your telematic signature). They use algorithms to learn which driving behaviours are signs of a safe driver – and then they offer those drivers cheaper rates.

This sets insurance companies apart from the competition and attracts safe drivers who don’t want to pay for risky drivers’ mistakes.

Predicting which insurance you’ll choose

Another way Machine Learning makes insurance companies’ lives easier is by predicting the exact insurance option you’ll prefer in the end. This makes your advisor’s job much easier, because they can propose the options that make the most sense to you right away.

And this is just the tip of the iceberg

There are many more applications, such as: predicting whether you’ll be a loyal customer, whether you’ll default on the insurance or file a claim in the future.

Targeted sales of banking services

Banks have a lot of very valuable data on you – besides knowing your age, income, and where you live, they also know your exact spending behaviour. To put it mildly, this reveals a lot about you. Maybe more than your online browsing behaviour.

If a bank used all the information hidden in this data, it could learn a lot about you – including which services you’re interested in right now.

Banks hire advisors to give some personal advice – but they only have time for the big clients, and an advisor needs a lot of experience to read the signs correctly.

Machine Learning can find the relevant information in your data and predict which bank products fit your current interests:

  • A new credit card or a higher credit limit?
  • A temporary loan?
  • Or maybe a mortgage, because you’re behaving like someone who plans to buy a house soon?

Not only that, banks can also use Machine Learning to predict the likelihood that you’ll repay that credit, loan, or mortgage – and hence which interest rate they should charge you.

This will increasingly lead to proactive lending. Instead of waiting for you to apply for a loan, thanks to algorithmic profiling, banks will offer a tailored loan to you – and also to millions of other customers.

Churn prediction - Predicting whether you’ll switch banks

Once you decide to switch banks and let your advisor know, it’s usually too late to convince you to stay. It would be much better if they saw the warning signs early and had a chance to make you happier before you decide to leave.

Banks can use the same profiles as above – built from your data and transaction histories - to predict whether you’re likely to switch to another bank. This gives advisors early warning of who to focus on and helps the bank retain many customers they would otherwise lose.

So why is AI so useful in finance?

Simply put, because finance is a game played with data and statistics. Financial products are mostly mathematical bets: statistical equations that – on average – should return a profit.

In the simpler and more repetetive parts of this game, we can substitute Machine Learning for human brainpower and automate a lot of decisions.

That is, unless we’re directly competing against other human beings – as in trading or fraud detection, where human competition makes the game much more complicated. In those cases, we’re better off using our strongest weapon: our own brains.

Energy Transmission

Anticipating and Preventing Power Grid Failure

Massive power outages cause chaos for the general public, and they cost utility providers roughly $49 billion a year.

This wouldn’t be much of a problem if massive power outages were rare, but outages affecting more than 50,000 people have increased dramatically in recent years. This means utility companies need to find new ways of anticipating and managing these outages.

These days, smart grids are producing massive amounts of data, which means predicting and managing outages is easier than ever. Unlike traditional power grids, which are one-directional (meaning they only transmit power in one direction), smart grids are two-directional. They can capture data from every possible source in the grid at the same time as they’re providing electricity. They collect and monitor data from sources like smart meters, IoT devices, and power generation stations, providing a clear, real-time look at power usage.

Machine learning can use this data to anticipate and prevent massive power outages in the grid. Machine learning helps identify non-obvious patterns in the data that can be a precursor to grid failure, which helps maintenance teams preempt failure.

Balancing the Grid

Balancing the grid — making sure energy supply matches energy demand — is one of the most important jobs a transmission operator has. But renewable energy sources depend heavily on the weather, making them harder to predict.

Transmission operators spend millions each year fixing planning mistakes that lead to producing too much or too little power. In hybrid systems — which rely on both renewable energy sources and fossil fuels to generate electricity — these mistakes have to be corrected at the last minute by buying more energy or compensating power plants for the excess.

Machine learning is the most accurate method available to forecast the output of renewable energy. Advanced methods, like Long Short-Term Neural Networks (LSTMs), can weigh the many factors involved — wind, temperature, sunlight, and humidity forecasts — and make the best predictions. This saves money for operators and preserves resources for power plants.

Preventing Blackouts and Brownouts With Real-time Monitoring and AI Prediction

Power grids have a lot of obstacles to overcome in providing continuous energy to customers. Weather patterns, usage, internal failure, even wildcard incidents like lightning strikes and interference from wild animals can all affect power delivery.

Machine learning is increasingly being used to help predict potential brownout and blackout conditions. By feeding historical data into the AI and running Monte Carlo simulations to predict potential outcomes, grid operators can use machine learning to identify conditions that could lead to grid failure. And they can act accordingly.

Sensors like phase measurement units (PMU) and smart meters can provide usage information in real-time. When combined with both historical and simulation data, AI can help mitigate potential grid failure, using techniques like grid balancing and demand response optimization. Incidents that would otherwise have affected millions of people can be contained to a smaller area and fixed faster for less money.

Differentiate Power System Disturbances from Cyber Attacks

Cyber attacks are increasingly used to target important infrastructure, like shutting down hospitals with Ransomware attacks (when attackers break into the system and lock legitimate users out until a ransom is paid). With utility grids, a cyber attack can have widespread consequences and affect millions of users.

Detecting these attacks is critical.

Developers are using machine learning to differentiate between a fault (a short-circuit, for example) or a disturbance (such as line maintenance) in the grid and an intelligent cyber attack (like a data injection).

Since deception is a huge component of these attacks, the model needs to be trained to look for suspicious activity – things like malicious code or bots – that get left behind after the deception has occurred.

One such method uses feature extraction with Symbolic Dynamic Filtering (an information theory-based pattern recognition tool) to discover causal interactions between the subsystems, without overburdening computer systems. In testing, it accurately detected 99% of cyber attacks, with a true-positive rate of 98% and a false-positive rate of less than 2%. This low false-positive rate is significant because false alarms are one of the biggest concerns in detecting cyber attacks.

Balance Supply and Demand

Utility providers are looking for ways to better predict power usage while maintaining maintaining energy supply at all times. This becomes critical when renewable power sources (like solar or wind) are introduced into the grid.

Because these renewable power sources rely on elements beyond human control (like the weather), utility providers know they can’t always rely on renewables for continuous production. Knowing precisely when demand levels will peak allows utility providers to connect to secondary power sources (like conventionally generated electricity) to bolster the available resources and ensure constant service provision.

More and more utility providers are turning to machine learning for help. We can feed historical data into machine learning algorithms -- like Support Vector Machines (SVM) -- to accurately forecast energy usage and ensure sufficient levels and constant supply.

Detect Power Grid Faults

Current methods for detecting faults in the grid consume a lot of unnecessary time and resources. This creates a situation where power transmission is interrupted and customers are without electricity while faults are first located, then fixed.  

Machine learning can find faults quickly and more accurately helping you minimize service interruption for your customers.. Support Vector Machines (SVM) are combined with Discrete Wavelet Transformation (DWT) to locate faults in the lines using a traveling wave-based location method.

When we apply  DWT (a form of numerical and functional analysis that captures both frequency and location information) to the transient voltage recorded on the transmission line, we can determine the location of the fault by calculating aerial and ground mode voltage wavelets. So far, this method has detected fault inception angles, fault locations, loading levels, and non-linear high-impedance faults for both aerial and underground transmission lines.

Detect Non-Technical Power Grid Losses

In the energy world, “non-technical losses” means energy theft or fraud from the system.

There are two common types of non-technical losses. The first is when a customer uses more energy than the meter reports. The second involves rogue connections stealing energy from paying customers. To pull off this theft or fraud, bad actors can bypass smart meters completely or insert chips into the system that change how meters track energy use. Meter readers can also be bribed to report lower numbers (though thanks to smart meters, this is increasingly hard to do).

Because these non-technical losses cost $96 billion annually, utility providers are turning to machine learning to combat the problem.

We can help utility providers mine historical customer data to discover irregularities that indicate theft or fraud. These can be things like unusual spikes in usage, differences between reported and actual usage, and even evidence of equipment tampering.

Energy Distribution

Better Predict Energy Demand

Accurately predicting customers’ energy needs is critical for any utility provider. To date, we haven’t found an adequate solution for bulk energy storage, which means energy needs to be transmitted and consumed almost as soon as it’s produced.

We're using machine learning to increase the accuracy of these predictions. Historical energy use data, weather forecasts, and the types of businesses or buildings operating on a given day all play a role in determining how much energy is used.

For example, a hot summer day mid-week means more energy usage because office buildings run air conditioning at a high capacity. Weather forecasts and historical data can help identify those patterns in time to prevent rolling blackouts caused by air conditioners in the summer.

Machine Learning finds complicated patterns in the various influencing factors (such as day, time, predicted wind and solar radiation, major sports events, past demand, mean demand, air temperature, moisture and pressure, wind direction, day of the week, etc.) to explain the development of demand. Because machine learning finds more intricate patterns, its predictions are more accurate. This means energy distributors can increase efficiency and decrease costs when they buy energy – without having to make expensive adjustments.

Energy Generation

Predict Turbine Malfunction

Wind is a great renewable energy source, but wind turbine maintenance is notoriously expensive. It accounts for up to 25% of the cost per kWh. And fixing problems after they occur can be even more expensive.

Machine learning can help you get ahead of this problem. The goal is to reduce maintenance costs by catching problems before the turbine malfunctions. This is particularly important when wind farms are located in hard-to-access places, such as the middle of the ocean, which makes repair costs even higher.

Real-time data gathered with Supervisory Control and Data Acquisition (SCADA) can help identify possible malfunctions in the system far enough in advance to prevent failure.

For example, data from sensors found within the turbines – such as oil, grease, and vibration sensors – have been used to train machine learning models to identify precursors to failure, such as low levels of lubricant.

This method can train machine learning models to predict failures up to 60 days in advance.

Consumption / Retail

Accurately Predict Energy Prices

As personal power generation (using solar or wind power) gets easier and cheaper, consumers and businesses are increasingly producing their own power.

Personal power generation allows people to make, consume, and store their own energy. Depending on where they live, they may even be able to sell surplus power back to the local power utility.

Machine learning can help find the best time to produce, store, or sell this energy. Ideally, energy should be consumed or stored when prices are low and sold back to the grid when prices are high.

By looking at historical data, usage trends, and weather forecasts, machine learning models have made accurate predictions on an hourly basis. People with personal and business energy generation systems can use these predictions to make strategic decisions about whether to use, store, or sell their energy.

For example, Adaptive Neural Fuzzy Inference System (ANFIS) has been used to predict short-term wind patterns for wind power generation. This allows producers to maximize energy production and sell it when energy prices are at their peak.

Reduce Customer Churn

In open energy markets, where customers have a choice of utility providers, understanding which customers are going to churn out can be critical. Churn rates, which is the percentage of customers who stop using your service in a year, can be as high as 25%. Being able to predict churn and stay ahead of it is essential to survival.

Machine learning is helping utility owners predict when a customer is getting ready to churn out. By using techniques such as Cross-industry Standard Process for Data Mining (CRISP-DM), AdaBoost, and Support Vector Machines, as well as historical usage data, utility providers can identify key indicators of whether or not a customer is going to churn. These indicators include things like customer satisfaction, employment status, energy consumption, home ownership or rental status. A change in any of these can indicate a customer is getting ready to terminate their service.

When these indicators are identified far enough in advance, it’s possible to avoid churn by working with customers to solve any problems they’re experiencing.

Energy Trading

Predict Energy Prices

Just like natural gas and oil, wholesale energy is a market commodity. So naturally it's important for traders to be aware of market fluctuations and pricing when it comes to buying and selling energy.

To help make sense of the massive amounts of data used to make trading decisions, traders are increasingly turning to machine learning.

A mix of statistical analysis and machine learning can help commodity traders make better predictions. Classical statistical analysis techniques like time series analysis, Seasonal Autoregressive Integrated Moving Average (SARIMA), and regression models are used to deal with the data. And machine learning makes connections between the various data points.

What’s more, machine learning trains itself to make increasingly accurate predictions using the constant flow of real-time data.

Keep reading

No items found.
No blog posts found.