Subscribe now

When it comes to blockchain technology there is not a lot more we can add to the myriad of articles on the internet. For completeness a simple intro can be viewed below.

We will not go into cryptographic monetary assets like bitcoin and surely we will not get involved in the bitcoin vs blockchain discussion. We believe there are specific use cases of digital transformation where trustless distributed databases like blockchain are an ideal solution and this is the focus of this post.

In the classic innovation work Crossing the Chasm, the author begins with the diffusion of innovations theory and argues there is a chasm between the early adopters of the product (the technology enthusiasts and visionaries) and the early majority (the pragmatists). In the case of blockchain the early adopters are the users that started getting involved in the crypto world since the 2008 genesis paper of bitcoin up to late 2017, the peak of the blockchain hype bubble. We are now in the chasm phase in anticipation of the slower but deeper adoption of the technology by an early majority of enterprise users.

The hype that blockchain has generated the in the last few years seems to have steamed up the industry with various PoC projects in numerous different industries. In 2018 and 2019 we have already witnessed some of the experimental MVPs see the light of production and blockchain to become a vital part of the digital transformation strategy of different businesses. Like AI, Blockchain is high on the innovation agenda of all major enterprises.

There are public, private, protected or hybrid blockchain networks. Whilst the technical details and differences of each are not a focus here, the main gist of each is exactly the same as we have with the Internet when you compare it to Intranet, or LAN networks. Each of them has a very critical and very justified role to play in the technology ecosystem. And while public blockchains are receiving most of the media attention these days we feel hybrid networks are the future for most enterprise users.

New technologies, are new pathways, that enable you to activate new revenue models, that otherwise you couldn’t even conceive.

The main areas where blockchain’s impact will be most strongly felt are :

• Fintech : Banking, Payments, Capital Markets, Credit Scoring, Remittances, New Currencies

• Identity : Personal Data, Medical Records, Authentication

• Charity : Donations, ESG Investing, Project Monitoring

• Asset Tokenisation : Democratising ownership and transferability of real tangible and intangible assets

• Logistics : Supply chain, physical goods trade

Among the plethora of use cases there is an urgent need for early adopters to experiment easily with blockchain solutions to develop a clear understanding of the cost-benefits involved within the context of a wider enterprise digital transformation. That experimentation brings its own challenges, namely in two main areas : infrastructure and analytics.

In terms of infrastructure, despite the rich open source ecosystem of the toolsets created by the vibrant crypto community there is a lack of easy integration within enterprise workflows. Blockchain-based integrations require a whole new architecture paradigm. This brings a new challenge to the IT operations the enterprise : gone is the simplicity of CI/CD, monitoring, logging, alerting, cloud native integrations, or serverless stacks. There is no support team to ask for an SLA, on-call hours, or DevOps cookbooks. To introduce anything that uses blockchain as part of the solution, the enterprise has to be prepared to throw enough investment to facilitate an experimentation and eventually production-ready infrastructure. Problem is … when you cannot fully understand the benefits how do you decide how much to invest ?

Another issue at hand is the analytics required to understand the operational efficiencies of the blockchain experiments. Blockchain is all about trust. But how can you trust something, if there is no easy way to explore it and understand what is happening in the network and why ? Blockchain networks hold raw data that answers questions such as who interacts with whom, how much value is being exchanged and when, how does every move in the network impact the other parts of the network and so on. The mining of the rich graph data embedded in the blockchain is paramount to the validation of any PoC experiment and eventual enterprise deployment and productisation.

“I think it’s a technical tour de force”

- Bill Gates

Mentat guest blog post in collaboration with Evangelos Pappas, CEO of Ocyan, the enterprise ready cloud native blockchain platform.

When I was working in the ship repairs industry we had a saying: if we could make the ship sail with half a propeller or even one fin, ship-owners would use it immediately. What was meant with this is that shipping is a business that is extremely cost sensitive and ship-owners create operational margins out of ruthless cost cutting. There are of course periods of high margins as the market goes up. Lots of money is made during these times. But often these good times are followed by troughs in the business cycle, when ships are sold, decommissioned, scrapped, slow-steamed and possibly mishandled to make the business sustainable.

In particular employing ships in the spot market can have incredible revenue volatility. Spot markets, where vessels are chartered on daily floating rates are cases where free market conditions prevail and thus strategic advantages are quite short lived. Ship owners and operators are price takers and there is little room for negotiation. Consequently, ship-owners need to cut costs to make margins.

Anything between brilliant strategic thinking to reckless business manoeuvring are used to save costs in seaborne trade operations.

Shipping and related supply chains in general, have been operating for a long time in a regime of opaqueness and heavily constrained information flow. This is changing recently, not so much around the cost part, but in terms of opacity. Operations, under the pressure of regulators and charterers alike, are becoming more transparent. This puts a lot of pressure onto the supply chain and this is an irreversible fact. Think of the oil majors: chartering a crude carrier vessel that is subpar and results in an accident, is bad for everyone, catastrophic for the environment and ultimately severely damages the brand long term. Vetting of ships is one aspect but the broader operations of the principals are also of interest. Authorities on the other hand are tightening regulations on how ships can operate in different locations. The most vivid example is IMO2020 whereby ships should ensure that their SOx emissions are capped especially in emission control areas. Thirdly, the entire supply chain wants to optimize its operations and avoid uncertainty and associated bullwhip effects. Contracts are tightening and penalties are becoming more granular depending on the operation of a ship.

Under this pressure for transparency, the industry is obviously in transformation. All the stakeholders are seeking to optimize their operations beyond what was achievable by the great business minds of the past. Optimization requires precision, thus automation and most crucially data. Data flows are increasing and will continue to do so, as the ship and the supply chain interlace with a growing grid of sensors, IoT enabled devices, services and machinery. As the ships are sailing somewhere in solitude, satellites take up the load of conveying the data. Modelling and simulation are tasked with predicting the near future and providing guidance for routing and operations. The ship is also a big factory, packed with machinery and instruments that suffer breakdowns multiple times within a year. It also hosts a crew, that needs supplies, gets sick and — let’s be frank here — is sometimes anything between unreliable to reckless due to poor training. All this is happening in an adverse environment : nature is relentless and this is especially true for the open sea. Add to that the operation along with other ships of all sizes in ports, terminals or through straights, or in areas of the world where there’s political turmoil, piracy or difficult connectivity.

It is not the ship so much as the skillful sailing that assures the prosperous voyage

George William Curtis

The increase in data flows also introduces the dangers of cybersecurity. Hacking a ship is becoming a possibility. Not only are the connections vulnerable but also the ship’s systems. Moreover the crew has low IT literacy and can easily introduce malware. That creates a whole new area of potential innovation in a critical sector where awareness of cybersecurity is from non-existent to at best low.

This vast, complex ecosystem lends itself to complex but substantial business opportunities given the addressable market. The shipborne commerce is a healthy 90% of global trade and will continue to be so. The seaborne trade is a 500 billion USD yearly market with over 50,000 merchant ships trading internationally, transporting every kind of cargo. The world fleet is registered in over 150 nations, and manned by over one million seafarers of virtually every nationality. Cargo ships are technically sophisticated, high value assets which can cost over 200 million USD to build in some cases.

There’s a massive gap between the need for technology and the actual state of deployed technology, and that’s where a substantial opportunity lies. The traditional model of the operators controlling almost every aspect of the ship is slowly but surely getting challenged. The ship is a floating bundle of disparate systems that streams data to a remote HQ location where operation specialists may or may not review it to take decisions. The amount of data is overwhelming, making constant data connectivity expensive not to mention difficult to achieve in remote areas. Furthermore, the model suffers from its capacity to process data on-site and provide guidance. Let’s draw a parallel to the automotive industry here. Consider the car that needs to emit all its information to a central authority in order to determine its health state and how to steer it. That would be unacceptable. Although the tolerable latency is higher for ships, this is still a major bottleneck. Furthermore, the decisions to be taken are much more complex and involve a balance between several variable classes: operational, maintenance, contractual just to name a few. The fleet could be seen as a distributed computing system where a lot of decisions are calculated on board, and different pieces of information are directed to the different stakeholders. In addition, operators and ship owners are concerned about the financial performance of fleets and subsets of fleets so further decisions for the planning can be taken on-shore using different KPIs than those on the ship and require the coordination among the ships. The ship is therefore a key component, but there are several management views, the operators, the ship-owners, the port/terminal authorities, the agents. There is a clear and urgent need for true edge compute and analytics capabilities that do not rely on a constant satellite uplink to the HQ servers. On the other hand, adaptive and unsupervised anomaly detection models can determine automatically if any data needs to be sent back to HQ outside infrequent predetermined datacomms windows.

A number of existing and future technology solutions, adapted to operate in a true edge environment can establish a smart ship IoT platform that replaces the guesswork of the “noon-report excel attachment” process. The current outdated model of once-daily manual data entry and macro driven upload of a small subset of metrics back to HQ will be transformed dramatically. Automating data collection, computing with appropriate real time machine learning models optimal courses of action, helping the crew optimize operations using AI, automatically parsing unstructured data in legacy systems using NLP, reducing substantially fuel usage and environmental footprint via adaptive optimization, paving the path to eventual maritime asset autonomy … the list is endless.

Shipping is a massive global industry that is moving to new models of operational efficiency. Data will indeed be the new oil in the maritime technology whitespace.

Mentat guest blog post in collaboration with:

Dimitris Servis , PhD Naval Engineering

This post is focused on one feature of our framework: integration with scikit-learn. Sklearn is the flagship ML toolbox for Python, and growing by the day. To ignore their models and design patterns would be to reinvent the wheel.

So we have added a small example about how you can bring the full strength of scikit-learn to bear upon your detection problem, while still using dsio. Consider the following file, which you can find in the examples folder:

datastream.io/examples/lof_anomaly_detector.py

“Lof” stands for “Local Outlier Factor”, an old and well-tested technique for detecting anomalies in Euclidean space (although it can be generalised to any space for which you feel comfortable defining a distance metric). The basic idea of LOF is to identify points whose nearest neighbours are not so near, in comparison to other points in the dataset. This allows us to detect anomalies whose values might not look so abnormal when you compare them to the maximum and minimum values found in the dataset, but in truth they occupy an empty space somewhere in the middle, where no other data lives. For one-dimensional data it might be counterintuitive to imagine such anomalous gaps, but as the dimension of the data increases it becomes increasingly likely that your anomalies will not contain extreme values in all dimensions.

Scikit-learn contains an implementation of LOF, wonderfully explained here. However, the Sklearn framework itself does not contain an interface for anomaly detection: it only supports classification/regression (supervised learning) and clustering (unsupervised learning where the main objective is to assign datapoints to clusters, rather than to produce anomaly scores). This is by no means a crippling disadvantage: any clustering algorithm can be easily modified to produce an anomaly detector via ideas similar to LOF.

However, our proposed interface is a step forwards in recognising anomaly detection as a core data science problem category. We have followed sklearn design patterns in introducing it as a Mixin rather than an object, which means that you can use pretty much any class you want, as long as you introduce (or override if they exist already) the following methods:

fit, update, score_anomaly, flag_anomaly

Our fit function will be revised soon to follow sklearn conventions fully (currenty it only supports unidimensional input, so it diverges), so that you don’t have to worry about it. The score_anomaly function is probably the most important, as it produces the final output of the detectors, and the flag_anomaly function serves to produce a binary output if one is needed.

The update function

The update method present in AnomalyMixin is another key innovation of dsio whereby we request that all supported models feature a way to update their states given new data, rather than having to refit from scratch. We will be writing another post soon to delve deep into the world of model updates!

We have a long roadmap ahead of us, but, release often and release early, as they say. So here it is — a minimal viable full-stack Python anomaly detector:

pip install -e git+https://github.com/MentatInnovations/datastream.io#egg=dsio

Features

The purpose of the project is to perform the following functions:

• Consume data from a variety of file and stream formats.

• Transform data streams on the fly to derive statistics of interest such as aggregations, counts, sessions, groupings, or extract features.

• Model the resulting stream via unsupervised machine learning to capture normal baseline behaviour either globally, or at the level of a device/user.

• Score every new event by comparing it to the baseline model.

• Visualise anomalous events on a lightweight customisable dashboard, with a lightweight back-end, involving minimal fuss by the user.

In the spirit of a minimal first release, we start by supporting consumption from CSV files, filtered by column, a couple of basic modelling and scoring options, followed by visualisation via an Elastic-Kibana solution involving a dashboard which is auto-generated in accordance to the column names.

Bring-your-own detector

Those of you that read our previous post know that we are about to unleash some pretty powerful anomaly detection models in this project. But like any open-source project, our main ambition is to create a platform. So for the first release, we have offered two basic example detectors (see below), as a template for you to build your own! All you need to do is support some basic interfaces, like a way to update your model, a way to train it from scratch (this addresses the cold start problem), and a way to detect anomalies, which often will often involve a threshold on a scoring function that numerically describes how likely each new event appears in comparison to the model.

You can try one of our own detectors from the command line like this:

dsio --detector gaussian1d examples/data/cardata_sample.csv

to run against a sample dataset comprising IoT measurements from a car. But if you’d like to write your own, just add your module and run instead:

dso --modules examples/detector.py --detector Percentile1D examples/data/cardata_sample.csv

Here is the result:

We are looking forward to your feedback and contributions! We will be adding exciting contributions from our friends and colleagues in UK academia and industrial partners.

One of the core competencies of the Mentat team has been anomaly detection, in particular unsupervised streaming data anomaly detection.

Anomaly detection (also known as outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset. These events may indicate network intrusions, industrial component failures, financial fraud or health problems.

Classifying anomalies correctly and efficiently determines the usability and effectiveness of many algorithms. It is a horizontal technology that is core to data driven methodologies.

Googling for anomaly detection we find the Twitter R package with almost 2300 stars at github at the top. We were tempted to test that approach with a subset of our anomaly detection models for benchmarking on the same dataset. We have been working on a set of tools for scalable streaming anomaly detection under the project name datastream.io or dsio for short.

Given the unsupervised nature of the problem in a streaming context we need to take into account the following complications:

• Without labels, we must rely on a model of normal behaviour.

• All models of normal behaviour make assumptions about the data.

• It is important to make these assumptions robust to common sources of variation.

Periods

Periods are a common problem in anomaly detection: the natural fluctuation of the data means that it might not be as easy to detect local anomalies (for example, peaks that fall within the periods of the data). Overcoming this problem was a flagship property of the Twitter AnomalyDetector package:

Here we illustrate the anomalies detected by the Twitter package and our own detector, part of which will be open-sourced soon. Happily, they largely agree, and are both able to detect local outliers even when these lie within the normal range of the data.

Period shifts

However, anyone who has worked with real data will know that sometimes periods shift … This might not be true in calendar-driven events where periods are forced by day/week/month/year patterns, but, for example, in industrial IoT devices it is extremely common: deactivating a component for a little while, or pausing a drill, will cause an otherwise periodic signal to shift its period. Frequency-based methods such as Fourier Transforms are very confused by such behaviour. It turns out, so is Twitter’s method, if we shift the period mid-way by 1000 steps, and results in reporting a much greater number of anomalies, most of which are false alarms resulting from the introduction of bias into the estimate of the period by “pausing” the periodicity for a very short period of time, just once:

In contrast, our detector is not confused at the least and continues to flag the same anomalies.

Another way in which data can fool you is by the introduction of a trend. Although there are several techniques for de-trending time series, the point we are trying to make here is that a real-world generic anomaly detector should be robust to such disturbances without the user having to take special precautions according to the use case. Twitter’s detector is confused by the introduction of a trend, and ends up reporting no anomalies at all, whereas datastream.io remains stable:

Yet another way to get confused is when the signal suddenly jumps to a new mean level — a very common occurrence in real-world data, like in the case of a web server which suddenly becomes more popular due to a successful marketing campaign. This shift is handled poorly by Twitter’s AnomalyDetector package, which results in removing all anomalies from its report, as an (exaggerated) reaction to the increased range of the time series. Yet again datastream.io remains stable:

Conclusion

Looking at this simple univariate problem its easy to see there is significant room for improvement in the anomaly detection space. We have argued elsewhere and continue to argue here that robustness is the secret sauce that makes machine learning methodology truly generic and hence usable in the real world. Put differently, it increases the ROI of any investment in machine learning projects, because it reduces drastically the biggest cost: the amount of data preprocessing, cleaning and custom modelling that the data scientist needs to do before they can deploy their favourite method. That’s what datastream.io is all about: robustness.

We will start open sourcing some components of the stack and we are looking to create a community around robust anomaly detection, bringing together startups, researchers and practitioners. Feel free to reach out to us here should you want to be involved.

The new valve cost you less than 100 dollars, but the downtime set you back a million. Wouldn’t it be nice to have had some advance warning?

This is the promise of Predictive Maintenance: early warnings for critical failures to avoid downtime. Despite what the media hype might like you to think, the problem is exceptionally hard. Our toy example of air in, air out, oversimplifies real-world industrial control systems, which typically feature a large number of variables that all inter-depend in complex, non-linear fashions. Engineers and physicists will list all these variables and their physical properties, and a few hundreds of pages of mathematics later, they come up with a set of equations that prescribe what actions need to be taken at any given second (indeed, millisecond) to ensure the target variables take the values that they need to take at all times, so as to maintain constant pressure in a container, move a robotic arm in a certain way, etc. But wear and tear is unpredictable: by definition, it pushes the system into a state other than the one originally assumed by the mathematicians and the physicists. Indeed, ‘worn’ states are orders of magnitude harder to describe via physical modelling. Ask any physicist and they will confirm: describing the laws of motion of a bicycle is child’s play in comparison to describing the laws of motion of a bicycle with a nail stuck on its front tyre. In brief, the “physicist-driven” approach does not scale. Can we do better?

Data

Well, in the words of Lord Kelvin, “if you can’t measure it, you can’t improve it”. And this is precisely why Industry 4.0 is so important: it marks the switch from legacy technologies that involved largely hard-wired industrial systems with minimal data acquisition and data sharing abilities, to next-generation technologies featuring millisecond granularity remote sensing, with gigabit ethernet connections all the way from the IoT device to the cloud. And this is the foundation of the Industry 4.0 promise.

This paradigm shift opens the door to altogether new approaches. It is known from other areas such as machine translation, that with sufficient data, machine learning is able to compete with or even outperform tedious substantive case-by-case theory.

In the same way that machine learning coupled with huge multilingual online corpora managed to outperform decades of linguistic analysis and translation theory, we can expect that machine learning coupled with terabytes of continuous data streams from IoT devices will rapidly compete with physical modelling of each device.

The Stack

Indeed, that was our expectation when we entered the TechFounders program with the proposal to deploy our proprietary algorithms for anomaly detection on data streams against data from one of the largest German supplier of pneumatic and electrical automation components, Festo. Festo is launching at Hannover Messe its Festo Motion Terminal, an intelligent pneumatic automation platform replacing the functions of around 50 different individual components, a technical tour de force in “software defined hardware”. In 2016 Mentat used the new intelligent valve to develop and test AI-methods for condition monitoring.

Our engine has four components:

• A data collector deployed on the device (in our case, a Raspberry Pi connected via an ethernet cable to the actual component) would take care of the ETL process of transforming raw data generated from the device’s sensors into messages that we can consume downstream. This component is our data ingestion engine.

• An anomaly detector sitting on-the-edge (in our case, on-the-Pi), would convert the raw data signals into what we refer to as a “feature vector”: this is simply a list of metrics that describe various properties of the raw signal. Just like the signal varies over time, so will its properties, so features must be computed anew every few seconds, using either a sliding window implementation or some form of exponential decay. This component is called the feature extraction engine.

• The features then are scored according to a probabilistic profile of what normal behaviour should look like. At the first instance of “abnormal’ behaviour we raise a flag. We refer to this edge component as the decision engine, and it also forms part of our on-the-edge stack.

• The data (appropriate compressed and summarised) are also uploaded in regular batches to a cloud instance of our learning engine. This engine is responsible for profiling each device, and maintaining these profiles as more data accumulate. With every new data batch, the engine refreshes the device profiles, and if any of them changes significantly, it is deployed in the form of a “profile patch” to the IoT device.

The end product was designed to closely match Industry 4.0 specs:

• Alerts in real-time, without dependency on always-on cloud connections (if that sounds too stringent, consider a gas pipeline in the desert)

• High-Bandwidth, Low-CPU edge components: the raspberry Pi was connected via ethernet cable, but is nowhere near as powerful as our Amazon Cloud servers. Thankfully, it doesn’t need to be: once the device has been profiled, comparing its latest measurements against the profile is very cheap computationally (a little like going through a decision tree).

• Low-Bandwidth, High-CPU cloud components: we strived to compress the raw data signals as much as possible before we upload. This is critical because as the IoT proliferates, data production will once again exceed our abilities to store data, or in any case process it from cold storage. Indicatively, for a single device with 8 sensors per component and 8 components, each producing one measurement per millisecond, we obtained 250KB of data each second. This amounts to 21GB per day — and that’s just for one device. We expect to support thousands. We used two main techniques to achieve meaningful compression: feature extraction itself compresses by orders of magnitude, and can be followed by active learning techniques, whereby only parts of the raw signal that fail to conform with our device profile are communicated upstream, on the grounds that they are more to likely convey new information.

Context, context, context

Anomaly detection solutions are often plagued by the “cry wolf” problem: they tend to raise too many alerts, so that eventually people stop paying attention to them, which renders them useless. The sophistication of the algorithms and the robustness of the implementation are critical in overcoming this problem, but a factor which is often disregarded is context: the human expert might be aware of information that the algorithm is not. In our use case, the “context” of the operation of a device comprises the environmental conditions in which it operates, as well as the operational mode it is in: what sort of task has it been assigned to perform? Needless to say that a machine learning algorithm tasked with profiling a robotic arm that has never before moved upwards will raise a big red flag the first time it sees the arm move upwards, much to the confusion of the human operator, who is of course aware that such a move is both possible and normal.

We overcome this problem by allowing our profiles to be context-specific: along with the signal features, the learning and decision engines additionally receive data about the state and configuration the device is in, and condition the behavioural profile on this information.

Labels

Our toolbox is designed to operate without the need for any labels: it is not a requirement that an expert should manually tag certain data segments as corresponding to “abnormal” behaviour, and tag everything else as “normal”. Ours is referred to as an unsupervised approach, in contrast to supervised approaches where the algorithm learns to discriminate between examples labelled as “normal” versus examples labelled as “abnormal”.

The benefit of an unsupervised approach is that it does not rely on expensive manual labelling, and is not overly focused on past examples of abnormal behaviour. This gives it better generalisation ability. By the same token, however, unsupervised techniques will exhibit some loss of accuracy in comparison to supervised techniques when it comes to examples of device faults that have been previously seen in the historical dataset.

In the case of predictive maintenance, it is possible to consider a number of common faults and try and reproduce them in a lab, so as to produce a labelled dataset. This is still an expensive process, but one that happens anyway by way of testing components for wear and tear. So in some sense, a certain amount of labelled data is available anyway.

This gave us the chance to switch to what is known as semi-supervised mode, where our profile is instantiated via a classifier that seeks to classify each signal as coming from a healthy device, a device experiencing a failure of a known type (i.e., one that has been seen before in the labelled dataset), and a device experiencing a failure of an unknown type (unsupervised). This covered all the requirements of our industrial partners, so the pilot was on.

Results

When we received the first data batch, I was actually under the weather. My team pinged me on Slack, and I crawled out of bed to my computer to check our first set of results. I actually had to get out of bed because “results” at Mentat is never a single number... Our company has inherited from its founders a masochistic discipline in using best practices when it comes to assessing performance. Our CEO, developed this habit while devising statistically-driven trading strategies, where overestimating predictive accuracy during backtesting could cost you dearly. I, on the other hand, spent much of my time in academia researching this very topic. In any case, our results summaries always need a big screen to review.

But in this case, it didn’t. At a glimpse, we knew this was a huge success. A total of 14 different metrics all agreed that we had reached 99% accuracy (no matter how you measured it) with just 5% of the data.

Across several different modes of operation, and facing several different types of faults, the machine learning algorithms were (almost) always able to detect when something went wrong. Perhaps a little excess noise in a certain part of the signal, or a slightly earlier onset; or perhaps it was something more complex, like the fact that a spike in one of the 64 different sensor feeds was not immediately followed by a spike in another one, as it is usually does. Our feature engine could detect all of these different signatures — and a lot more, indeed, tens of thousands of them.

In truth, we were relieved to see our metrics rise up to 99% without any fine-tuning. A research paper might impress with accuracy even in the low 90s , but we knew full well that probability is a tough lover. If you are monitoring a single device over a period of a few days (as was the case in the historical dataset we were given), then 99% accuracy is more than sufficient. If, however, you monitor thousands of devices over months, or years, you had better start off with 99% accuracy, otherwise alert fatigue will kick in soon.

Our results confirmed not so much that the problem was “solved”, but rather that it is “solvable” with our choice of technology: a combination of sophisticated machine learning algorithms, with built-in failsafes to handle temporal variation, and a lean edge-cloud architecture.

The Future of Industrial IoT

The technical success of this initial proof of concept can only hint at the powerful implications for the future of the manufacturing industry.

Change brings opportunity and danger. The opportunities are obvious for manufacturers : better customer support, differentiation from the competition, software revenues via SaaS applications attached to their devices, better understanding of usage patterns, more uptime. The dangers are also pretty obvious : brands that have built a reputation of solid quality over generations will be competing with more agile manufacturers that can possibly be more aggressive in their cost plus pricing via creating intelligent revenue from software services.

We feel there is a seismic shift which will happen in the manufacturing sector in the next few years. The cards will be dealt again and the game will have new rules.

At Mentat we view drones as an agile flying platform for advanced sensors, hence the “Flying IoT”.

Consider the example of a remote wind farm that requires regular visual inspection of the wind turbines to detect and monitor cracks on the blades. It used to be necessary for an engineer to regularly conduct personal inspection visits to the site. A safer alternative would involve camera drones, remotely controlled from the ground by a qualified engineer. In this setup however the human remains the bottleneck: physical presence of a suitably qualified engineer/pilot is still required, a costly proposition.

An altogether superior solution would involve an autonomous drone, scheduled to perform a visual inspection of the entire farm at regular intervals, paying special attention to existing cracks, and optimising its flight path to take into account wind conditions, which can be particularly challenging in a wind farm scenario due to local vortices produced by the turning blades.

Multiple other use cases will be revolutionised by the advent of autonomous drones: inspecting forest areas for early detection of wildfires; surveillance in security deployments; collecting traffic statistics and managing first responders in urban environments; deliveries of products in an industrial or retail scenario, or critical supplies in an emergency management scenario. The list is endless and every day seems to bring another great use case for drones.

However, autonomous does not and should not mean entirely unsupervised. Miscalculations or faults in a drone’s on-board logic might lead it to fail its objective, or even to become a liability. The question then becomes: as autonomous drone technology scales, what sort of monitoring technology is able to scale with it and ensure safety and quality control without introducing bottlenecks ? Can we learn automatically from the trajectories and provide an intelligence layer for drones ?

The aim must be to minimise the human-to-drone ratio in any use case. One can envisage an alerting system that monitors all drones, prioritises them in terms of degree of concern, and asks for human input only in the most urgent cases.

Analogous alerting systems are already in place in other areas where software agents enjoy some degree of autonomy, such as robotic installations in manufacturing plants, and IT/cybersecurity. However, drones are an idiomatic case as they generate masses of live and fast geo-temporal data in the form of their 3D GPS tracks. Despite the abundance of GIS systems for storing such data, powerful solutions for analysing it are not available. This is a common theme in Big Data: easy to store, much harder to analyse ! Furthermore analysing this data must happen at the drone level (at the edge of the network) rather than using a data downlink (i.e. sending data to an on-premise or cloud based server).

The current state-of-the-art involves a technique known as geo-fencing, where basically a specific hard coded area of space is manually specified in the monitoring platform, and if any drone escapes that area an alert is generated. This technique cannot scale. If one tries to optimise the geo-fenced area, the manual configuration step will in effect fix the drone trajectory, which stands in the way of drone autonomy (a good analogy to keep in mind is the difference between a self-driving car, and a tram which runs on predefined rails).

The missing ingredient is a system designed to monitor intelligent agents. That system must itself be intelligent! Our solution is able to learn the preferred trajectories of each drone from the GPS tracks they generate, without any need for manual configuration.

When that trajectory profile in geospatial or temporal terms is violated, an alert is generated. Such a system offers incredible flexibility. First, it decouples monitoring from drone configuration, so that if a drone is suddenly reassigned to a different task, the system will initially raise an alert but then quickly adapt to the new route without the need for reconfiguration by a human. Second, it is able to profile drones controlled by entirely separate systems or entities, as long as it’s able to catch a glimpse of their GPS tracks. The learning algorithms can detect abnormal behaviour both at the macro level (“this drone is heading into territory it has never accessed before”), and the micro level (“this drone is performing odd manoeuvres that might indicate loss of stability or malfunction”). Some of the common shortcomings of geo-fencing are naturally overcome by our platform: for example, we can detect abnormal direction and/or speed, not just location (i.e., when a drone is within the bounds of its normal trajectory, but is moving in the reverse direction than usual). We can also grasp manoeuvres that surveillance drones typically employ to cover an area (such as the lawnmower or spiral manoeuvres — see use case 2 related to agriculture).

We have paid all due respect to the great forerunner of our system: target tracking software, but extended that methodology massively. Broadly speaking, the mathematics that underlie classical target tracking solutions are exceptionally accurate in forecasting trajectories of ballistic objects (where an initial or constant force defines the trajectory of an object such as a missile), or for short-term forecasting of autonomous objects with known constraints on their manoeuvrability (consider the difference in a fighter jet’s evasive manoeuvre versus the agility of a bumble bee). To track drones one must instead use more flexible methodology, inspired by advances in machine learning.

Below we include two video demonstrations. We make use of a 3D Unity front-end coupled with our streaming machine learning engine. This allows us to perform live demos, but it also expresses our view that a VR front-end is the right choice here: human supervisors need to understand drone trajectories in the context of the physical terrain they are navigating, but a constant live video feed is unrealistic due to both battery limitations, bandwidth constraints and cybersecurity issues.

In the above video, we demonstrate the ability of the system to learn a 3D trajectory from scratch. The cones in the video indicate the track the user is expected to follow using the controller (as a way of helping the “pilot” visualize the track), but the system has no prior knowledge of that, which explains why every time the drone turns during its first time round the track an alert is generated. However, as the drone repeats the track, the frequency of alerts (show in the bottom left corner as a time percentage and visually in the bottom right corner) decreases dramatically. At the end of the video, an excursion of the drone outside its typical trajectory is immediately flagged as anomalous, even while the drone still remains at the interior of the tracks, where this departure would have been missed by a classical geo-fencing solution.

We also show here the ability of the algorithms to understand patterns, which is a common set of manoeuvres employed by drones when they are trying to cover an area. Above is a a real GPS track from a drone in an agricultural use case, where a so-called “lawnmower” pattern is employed to cover the area. Here we focus on a collision alert use cases, where one drone (depicted in red in the video) is attempting to cross a region which is currently being surveyed by another drone (depicted in green in the video). The trajectory of the drone is determined autonomously, depending on its objective, the weather conditions, its battery power, as well as potentially other more complex criteria. The objective of the monitoring agent is to forecast potential collisions between the two drones.

Clearly the challenge here is to understand the recurring lawnmower pattern, typical in precision agriculture cases. Although the pattern is quite clear to a human, it is a great challenge to automated forecasters — it switches from linear (during take-off) to a recurring pattern, it involves a 3D diagonal movement which renders it asymmetric, and is awash with small departures/delays caused by wind. Moreover, the use case requires us to pick up the pattern very quickly, after just one or two repetitions. These features and requirements virtually incapacitate any classical “periodicity detector” that can be found in off-the-shelf time series or target tracking packages.

The detector instead does a great job picking up up the repeating pattern very early. The smoothness and accuracy of the pattern improves over time as the drone settles into its pattern, and can extend over very long horizons without losing accuracy. This in sharp contrast to most commercial target tracking software that are only able to forecast long horizons when the target is exhibiting what is technically known as “second-order stationarity”, a practical translation of that would be that the “steering wheel and gas pedal” are held in a fixed period (constant angular and linear acceleration).

It’s important to note that we don’t raise an alert as soon as the two forecasts start intersecting. Our prediction is in fact 4D, since it takes into account time (i.e., the speed of the drones). Therefore, forecasts are allowed to overlap as long as the drones never occupy the same space at the same time. That is why there is an alert for one of several intersections between the light blue and light red forecasted trajectories — and indeed that is the only one which would have led to a collision, as is evident near the end of the video.

The action that would follow a collision alert depends on the use case. It could trigger a human override in a centralised control scenario, or interact with the software agents in the drones in a distributed control scenario to avoid potential collisions. In this video simulation the red drone avoid the collision by moving higher temporarily. This capability becomes particularly useful in scenarios of multiple drones in challenging environmental conditions.

We are particularly grateful to have had the support of Ordnance Survey (Geovation) on the geospatial modelling side and of InnovateUK on the augmented & virtual reality side to bring this higher risk feasibility study to fruition. We are working on a number of drone data (raw and sensor/imaging) related projects which we will share soon.

Everything Changes

​So one reason why one should switch to online Random Forests is speed and scalability. But the other is that decision rules that were valid last week might be obsolete today. An immediate “hack” is to retrain your classifier in regular intervals, using a fixed number of the most recent observations. This is a great quick fix, but it suffers from two drawbacks:

• Your classifier is out-of-date most of the time (except just after retrains!)

• The length of the window size is a critical parameter for performance.

To demonstrate this latter issue, we run an experiment on a dataset knownn as ELEC2 which has become a benchmark in the streaming data literature. Each record contains a timestamp, as well as four covariates capturing aspects of electricity demand and supply for the Australian New South Wales (NSW) Electricity Market from May 1996 to December 1998. Labels indicate the price change related to a recent moving average.

Below we report the error rate (lower is better) achieved by a sliding window implementation of random forests using the randomForest package in R, for 8 different window sizes (left group of bars in the Figure below). When the dataset is ordered by timestamp, the best performing window size is 100, on the lower end of the scale. This is classic case of “more data does not equal more information”: using 100 times more data (w=10,000 vs w=100) almost doubles (175%) the error rate!!
To drive the point home, we took the same data, but presented it to the classifier in random order so that it was no longer possible to take advantage of temporal effects. In this case, without any temporal effects, indeed the accuracy improved with larger window sizes.
​
The advantage that a well-calibrated streaming method can have over its offline counterpart in a streaming context is quite dramatic: in this case, the best-performing streaming classifier has an error rate of 12%, whereas a random forest trained on the entire dataset (minus 10% withheld for testing) achieves an error rate of 24%. A fraction of the data, double the accuracy !

So it seems that you can have your cake and eat it. Or can you? Well, you can only achieve this fantastic result if the window size is well-tuned. And that’s a tricky business… Set it too small, and you will have to learn everything from scratch every few observations. Set it too long, and obsolete patterns will start misleading you. And to make things work, a technique like random forest consists of a composition of thousands of decision rules, some of which may be getting obsolete faster than others.

Put simply, any model that runs on a live stream must be updated regularly, but there are fundamentally better ways to update a model than to rebuild it from scratch. A bit like a human, successful online learning algorithms recognise that they have a finite memory, and that some patterns “age” faster than others. The brute force of a sliding window cannot take you very far in real applications.

The Mentat SRF

Mentat’s Streaming Random Forest has three unique features:

1. It makes use of the cutting edge in online learning techniques to stay up-to-date by letting patterns “age” at different rates.

2. It has a fixed memory footprint: a model update operation takes the same amount of time whether the data has seen 10 datapoints or 10 million datapoints.

3. It utilises an “ensemble-of-ensembles” techniques, by combining a streaming self-tuning implementation of random forests with a number of other, simpler but highly agile classifiers. These classifiers have the benefit that they can pick up on short-term linear trends much faster than a complex technique like a random forest, while allowing the latter to extract deeper patterns from the residual signal.

Indeed, in the example above, the SRF tool achieves the same result as the best-performing window, without any manual tuning whatsoever. This performance is identical to the state-of-the-art in the literature (see a related publication by our Chief Data Scientist here).

Empirical Study: Detecting MalwareThe Kaggle competition of 1999 introduced a benchmark dataset for classification on large datasets. It considered five types of traffic:

• Normal

• Attack Type 1: Probe

• Attack Type 2: Denial-Of-Service (DOS)

• Attack Type 3: user-to-root

• Attack Type 4: remote-to-local

This is an example of a multi-label classification problem: we don’t simply need to detect malicious traffic, but want to know what type of attack it is in real-time, so that we can take automatic action.
It was revealed that Probe and DOS traffic can be distinguished from normal traffic by very simple methods, such as a nearest neighbours classifier — the winning method only managed to improve accuarcy in these cases by a few percentage points. User-to-root and remote-to-local attacks were much more challenging, with the winning method achieving 92% and 87% error rate respectively — more than 8 out of 10 examples were misclassified in both cases.
​
Below we compare SRF to the winning method, in terms of the error rate (lower is better):

When it comes to DOS and Probe there wasn’t much to improve upon — the baseline method already achieved error rates below 5%. However, for remote-to-local and user-to-root, the ’99 Winner improved upon the baseline significantly (red is lower than blue in the last two columns). SRF blows it out of the water, with less than half the errors for remote-to-local, and a 40% decrease for user-to-root. These improvements look even more impressive when one takes into account two things:
​

• SRF was applied out-of-the-box to this dataset, whereas the 99 winner was bespoke.

• SRF maintains a fixed memory footprint

Description of the API

The Mentat SRF has a very simple API, with three components:

• A config file which describes:

• the number of input features, and their type: categorical (e.g., a colour, or profession) or numeric (e.g., height).

• the number of labels (e.g., “yes/no” versus “malicious/DOS/probe/…”).

Don’t worry, if you don’t provide a config file, we will guess.

• A predict call, which submits a new unlabelled datapoint, and returns its predicted label. Being mathematicians at heart we never just report the label alone, but also provide you with estimated probabilities, so you can know how confident we are in each prediction.

• An update call, which submits to our API a new datapoint, its timestamp and label.

To make it easier for you we can also accept multiple datapoints in one request. Our webservice for both predict calls is extremely fast and scalable, and our premiums service can support even the most demanding real-time applications, via Amazon Kinesis.

To sum up:

• no need for classifier selection and tuning (window size or otherwise)

• no need for you to host the decision engine for your web app

• no need for you to monitor the performance of your model and decide when and how to retrain it — our self-tuning engine monitors performance and ensures it remains optimal at all times

• no need for you to scale out as you get more data

For the enthusiasts out there, we can expose more control parameters, as well optionally return the updated decision rule as an XML file after an update call. We can also expose monitoring parameters such as the recent error rate or AUC (although we prefer to measure classification performance by the H-measure as well as a metric of how fast your data is aging.) If you want to be part of the private beta, please get in touch.

Feature extraction and unstructured data

Often datasets require a feature extraction step before they can be turned into the right format for classification. This step can be another crucial factor in performance. Our experienced data science team can support this on a consulting basis.

If your dataset or tasks involve unstructured data, such as free text, images, or networks, do not despair!. We can support feature extraction steps on all such data types, either using our own bespoke software or via integration with Alchemy API, as we are an IBM Watson Ecosystem Partner.

What else is coming?

Classification is a supervised technique: a bit like a child, the classifier needs to be shown labelled examples of the ground truth in order to extrapolate for the future. But children also learn by themselves, without explicit guidance by gradually developing an understanding of “normal behaviour” of their environment, and responding appropriately to any “anomalous events”.

This is also true of machine learning algorithms. For example, in cybersecurity, the most dangerous attacks come in the form of previously unseen types of threats (zero-day). A supervised classifier is by construction hopeless against such novel threats. Similarly, new types of fraud appear every day; new types of bugs are are introduced in software; and new customer behaviours can develop on a weekly basis.

When everything changes, profiling and anomaly detection is a necessary complement to agile, adaptive classification technology. This is why our core product, datastream.io, is designed to perform unsupervised profiling and anomaly detection at scale, as a service. Please stay tuned for our next blog post on this topic.

Originally published at www.ment.at on 10-Dec-2015

Let’s start with the simplest possible example, that of a linear sum: it can be easily computed in an incremental (exactly-once) processing manner:

​The answer in this case is exact: it agrees precisely with a batch sum. Such examples are referred to as exact incremental or exact online processing.

However, not all queries can be computed exactly in an online fashion. A classical example is top-N, or a median. To understand why, it is best to define our terms: what does incremental or online really mean? Put simply, it means that the memory and time requirements of the process do not grow over time — despite the fact that the number of data entries processed does (since data streams are by definition semi-infinite — in Tyler’s terminology: unbounded).

Indeed, to maintain a sum, one only needs one to store one floating number in memory (the last value of the sum). To maintain an average, one needs to store two: the sum, and a count of how many data entries have been seen so far. However, to maintain a median, one needs to keep in memory an array whose size grows with the total number of data entries seen. Similarly for top-N. Consequently, the best we can do in a streaming context is offer approximate answers to queries such as a median, top-N, or a percentile calculation.

“Approximate answers” has been a longstanding criticism of streaming processing. MapReduce enthusiasts have always argued that, no matter how big your dataset is becoming over time, unless you desperately need real-time answers, you should always scale out and run batch to ensure correctness.

Needless to say that, as Tyler also mentions, approximate queries could be implemented in batch mode, too, in time-critical contexts. In fact, the initial motivation behind online processing was precisely that: to generate fast approximations by sequentially processing massive datasets.

However, as soon as we move beyond simple analytics such as sums and top-Ns, and into the realm of machine learning, the situation becomes a lot more interesting. Very few batch Machine Learning algorithms have exact online counterparts (one example being linear regression which can be exactly computed online via the Recursive Least Squares algorithm). Therefore, your favorite batch random forest will only approximately match its online version.

Does that automatically make the online algorithm inferior, or sub-optimal to the batch version? The answer is a definite (and perhaps surprising): No !

A major distinction between analytical queries and machine learning algorithms is that the former are fixed functions of the data, such as a sum or a rank, whereas the latter is a process, rather than a function, whose objective is to discover hidden patterns or rules that generalise well against future datapoints. There is therefore no theoretical reason why a batch random forest must necessarily outperform a well-designed online random forest.

In the seminal book Principles of Data Mining the chairman of our advisory board Professor David Hand and his coauthors make this point really nicely. They break down a data mining algorithm into the following constituents:

• The task to be solved (e.g., classification or clustering)

• The structure of the model/pattern being fit to the data (e.g., decision tree)

• The scoring function which measures how good the fit is (e.g., squared loss)

• The search/optimisation algorithm which seeks high-scoring patterns

Possibly the easiest way to understand this is to consider a neural network with a fixed architecture (i.e., number of hidden layers etc.), which has to be trained on a dataset of labelled examples. Because neural networks are infamous for producing optimisation surfaces with a huge number of local optima, no gradient descent method is guaranteed to identify the global optimum, and, indeed, conjugate GD might produce a different answer than simple GD. In this sense, an online algorithm using Stochastic Gradient Descent is just another candidate, which might or might not outperform the batch candidates. In any case, it makes no sense to consider the SGD answer as an approximation to the GD answer. In truth, they are both approximations of the unknown “true” decision boundary, and, if they are well-designed, they should both approach the “true” answer as the size of the dataset increases. Which one gets there faster is largely dependent on the problem and the class of methods. Put differently, in machine learning, both batch and online algorithms are approximations, that use data and computational resources differently.

A clarification is in order at this point: sliding windows are an extremely inefficient way to perform streaming learning, precisely because they are designed as approximations to batch learning, albeit starved of information due to the fact that they use only a fraction of the available data. Overarching frameworks for online learning such as stochastic gradient descent and stochastic approximation ensure instead that information is extracted and stored in the most efficient manner possible and updated sequentially.

But, surely, you might argue, the batch paradigm has more computational resources and can therefore support superior algorithms, right? As a general statement, this argument makes sense, but in practice, several statistical phenomena conspire to make online algorithms extremely competitive. We only mention here a few, and promise to analyse further in a future post:

• Overfitting. Online learning proceeds to improve its answers by continuously comparing its predictions with the next datapoint. This means that, by design, online algorithms are less prone to overfitting than offline methods, where a great amount of discipline is needed to avoid that pitfall.

• Simulated annealing. When faced with complex optimisation surfaces riddled by multiple optima, the noise introduced by sequential data processing is a very successful way to escape local optima. In fact, online learning was first introduced in the machine learning literature as a way to train neural networks in batch mode, as it was observed that feeding the data to the learning algorithm sequentially can result in better answers.

• Temporal drift. Most real datasets feature a certain amount of temporal variation which might be too subtle or too complex to model explicitly (as opposed, for example, with clear periodicities or jumps). The data decay which is a built-in feature of most online learning algorithms offers robustness against this ubiquitous phenomenon, whereas batch algorithms need substantial refactoring to accommodate drift. This is in fact a critical and under-addressed point, which we may address in a subsequent post.

Finally, are streaming algorithms really “more complicated”? We think this question is moot. Some models admit simple algorithms whereas others don’t, and the same holds of streaming versions. Moreover, online learning is no longer as fragmented as it was ten years ago, since overarching frameworks such as Stochastic Approximation and Stochastic Gradient Descent haved matured to the extent that it is now possible to reuse a lot of components in a streaming machine learning library.

Let’s recap our key observations:

• In discussing the comparative accuracy of streaming vs batch processing, one must crucially distinguish between analytics and machine learning.

• For analytical queries, some cases such as a sum or a standard deviation are possible to exactly compute online (i.e., using constant resources), whereas others can only be approximated (e.g., top-N), albeit very well.

• In machine learning, both batch and online algorithms offer approximations to an “unknown truth” (e.g., a decision boundary or a cluster structure), and should be assessed according to their generalisation ability rather than their in-sample performance, so as to avoid overfitting. Therefore it makes no sense to consider online algorithms as approximations to batch algorithms.

• Despite their computational efficiency, online learning algorithms have built-in features that can often make them outperform batch counterparts.

Originally published at www.ment.at on 28-Sep-2015

What is it that makes Streaming Edge analytics different from Traditional analytics, or Big Data analytics? There are three main differences :
 
1. Scale with Velocity (not just Volume). A Hadoop query that runs a little slow is a discomfort; a streaming query that runs too slow is a disaster, resulting either in data loss or a system crash (or both), as new observations pile in waiting to be processed. Moreover, most edge use cases involve the need for real-time actionable insights with short lifespan, where “delayed” is just as bad as “missed”. 
 
2. You can’t store everything. By definition of the use case, some data will be lost. But the information contained in this data need not be lost, if your streaming analytics are done properly. The simplest example is a sum: it is trivial to compute it in a streaming fashion, via a cumulative sum that yields a 100% accurate answer without any data storage. Surprisingly many statistical insights can be extracted in this way. 
 
3. You can’t scale out easily (neither on CPU nor on RAM). Edge computing generally relies on few, small computers. Think 2000. Forget about machine learning techniques that need GPU farms to get started. 
 
 These are the constraints we listed on our white board at the South West of rainy London when we set out to design our demo for Cisco Live at sunny San Diego a few months back. And this is what we came up with.

The core idea came easily: we needed to deploy on a Raspberry Pi. These tiny very low cost devices are fairly close to the capabilities of a typical fog node. The Pi would receive a stream of data in one port at high frequency, process it in real-time, and output the resulting analytical insights in another port at a lower frequency.
 
 The challenge for the data replay was to stream the data at the same pace as the original dataset was being saved. In order to do so, we created a small tool coded in Elixir that would read and replay a dataset (CSV format) and re-stream over the network it in real time, re-timing each observation with the current timestamp. The tool would detect the original time delta between two observations and would make sure that we don’t emit a new observation too early or too late. 
 
 Since Linux’s Process Scheduler doesn’t give you any guarantee about when your process is going to be put asleep and how long this will last, we made sure to detect and discard any record that would be considered as “too old”. 
 
 I/O is the general bottleneck and in our case, reading the CSV dataset from the disk was of course slow. Furthermore, as the Process Scheduler was putting us asleep from time to time, we had to make sure that we had a buffer for the CSV records. 
 
 We therefore had a bunch of different Elixir (Erlang) Processes in charge of the entire pipeline: reading the CSV file, transforming each line in a new structured record, retiming the data and coercing some of the values, queuing the records, consuming the queue and streaming the records.
 
 In order to have a clear, visual and realistic demo, we decided to use a second RaspberryPI to run the data-replay tool. This way we could very easily plug and unplug the Ethernet cable that was linking them to stop and resume the consumption and analysis of the data.
 
 Since only part of the team could go to San Diego, we had to make sure that we could still deploy fixes and updates remotely. 
 
 We deployed an OpenVPN and a private Docker registry server in AWS to make sure that the technical team could deploy new containers with the latest version of the code and we provisioned the Raspberry PIs with Chef to ease the setup. In just a couple of commands the latest versions were deployed remotely round the world.

So one Rpi streaming 90 wind turbine sensor readings at 100 Hz each (or 9000 datapoints per second in total) going into our datastream.io core engine running on a single core of the CPU of the second Rpi, which then displayed a custom dashboard built for the demo. Anomalies were detected on all sensor streams in real time and our alert correlation technology gave a holistic view of the wind turbine health.

Originally published at www.ment.at on 13-Aug-2015

The IoE will treat the home as a living system of devices; the manufacturing sector as a society of robots and sensors; the retail sector as a never-ending multi-channel interaction; and the city as a hive of homes, retail outlets, factories, utilities and infrastructure. Accordingly, its disruptive nature will not primarily lie in the time-saving effect of automating simple tasks but rather feeding from, and feeding back into, intricate economies of network and of scale. The IoE will manifest itself in enabling previously unthinkable efficiencies and capabilities.

We are nearly there. Ubiquitous sensing, connectivity and processing are enabling continual data capture and exchange in a rapidly increasing variety of physical settings. These mini-nervous systems are still islands — the home, the hospital, the school, the manufacturing plant — but soon they will be subnets of an emerging interconnected network of the physical and industrial world. This is how it starts. But the end goal, invariably, is not just to sense, but to make sense of, to understand, to predict, to control. This is a superhuman task when faced with a sea of machine chatter. Machine nervous systems require machine minds.

Luckily, we can now build those. The last few years have seen a cascade of Machine Intelligence victories, reaching milestone after milestone: vision, translation, medical diagnosis. The list of tasks where machines now routinely outperform humans is growing at an unprecedented rate. All things considered, we don’t think it’s an exaggeration to say that the first half of the 21st century will be remembered for the convergence of two of the most disruptive technological waves in human history: the IoE, and Machine Intelligence. As machines become able to comprehend sensory input of various types and impossibly high dimensions, plugging them into a vast nervous system will mark a turning point in technological history. Machine Intelligence is the value driver that can turn continual streams of data into actionable insights.

A new paradigm is needed to address the challenges and opportunities of this new era of increasing data velocity and variety. We need machine intelligence designed to deal with data in motion. Data in motion will dominate the sea of data generated by the IoE. And the ability to efficiently process, analyze and produce actionable insights from streaming data in ways that can be consumed efficiently by humans will be paramount to bringing the vision to fruition. Real-life processes change over time and designing systems with adaptability and scalability as core principles is paramount. Business intelligence can yield a lot of insights, but we truly need to be able to deploy at scale the full arsenal of machine learning capability in order to leverage the wealth of structured and unstructured data generated by the IoE.

Mentat Innovations feels privileged to be able to collaborate with Cisco to deliver the vision of a connected, intelligent IoE to the world. We can now finally imagine a shift from diagnostics to prognostics in manufacturing plants, smart electricity grids and intelligent retail solutions. We can imagine a connected healthcare system offering early warnings for life-threatening conditions without additional burden on overworked doctors. Use case by use case, across all verticals, a bigger picture is emerging, of a new economy where ambient machine intelligence empowers humanity to build a better future. This starts now.

Tags: Christoforos Anagnostopoulos, Cisco, Internet of Everything, Internet of Things (IoT), IoE, IoT, Mentat Innovations

Originally published at blogs.cisco.com on 30-Mar-2015

It is important for two reasons. First, because whenever a piece of technology beats humans in something vital, this usually causes a revolution. When fire beat human saliva in making food safe and digestible, the modern human evolved. When stone tools beat human hands in warfare and hunting, the Stone Age began. Then iron beat stone. Steam. Electricity. The Digital Age. And now Artificial Intelligence.

It is hard for us to envisage the ‘next day’ where thinking machines routinely outperform humans. Massive unemployment, the potential for asymmetric warfare and a deepening rich-poor divide have all featured as potential elements of a dystopian AI future. Equally, eradication of disease, poverty and social injustice have also been put forward as a utopic alternative.

Hard to tell. Even harder to tell is whether machines will eventually dominate the human race, which some thinkers see as theoretically inevitable…geeky end-of-the-world enthusiasm aside, this is not as extreme as it sounds.

But there is a more practical, immediate concern, with real implications about the choice of methodology that AI companies like us are building right now. Our worry is that our immediate future will be plagued by overconfident machines. Put differently, if true wisdom is to know what you don’t know, machines are still pretty stupid.

Machine translation will always translate the input text, even if it has to clutch at straws and output junk. A bit like an overeager translation intern, it lacks the wisdom to say “I am not quite sure what this means, sorry”.

This carries risks that are all-too-real. Our foremost experience of AI will not be in the form of a charming golden robot with a British accent and a lovable sidekick, but rather in the form of hidden automation. And just like a human, an unsupervised machine that thinks too highly of itself can prove catastrophic.

So is it within our ability to code up “modesty” and “self-criticism”? Yes, and no.

Yes, because, to the relief of statistically-minded machine learning researchers most ML is now probabilistic, which simply means that answers are given in the form of a probability, rather than a straight yes/no. A picture might be a picture of a cat with 90% probability, or with 50% probability. It is then a matter of deciding how best to use this estimated uncertainty on a case-by-case basis, and, when it matters, one can always choose not to report the 50/50s.

However, uncertainty is subtler than that. Most Machine Learning is model-based, which means that its output (e.g., the probabilities above) is computed itself on the basis of certain assumptions about the data-generating process. These assumptions will, almost surely, be violated at some point. A thinking machine should then recognise that fact and report higher uncertainty.

Unfortunately, honest reporting of uncertainty remains challenging. Lets briefly examine why that is the case in the two flavours of AI that are ‘hottest’ right now: Bayesian AI, and Neural Networks (of which Deep Learning is the latest branch).

Bayesians are proud in relying on a coherent, all encompassing mathematical framework for expressing uncertainty, and they follow best practices in always reporting probability distributions, rather than point estimates. A Bayesian who is worried about a certain assumption will invariably express that assumption in a “fuzzier” manner, by introducing uncertainty about it in the form of yet another probability distribution — assumptions about assumptions. Unfortunately, even the most carefully stacked Bayesian model might still be betrayed by the real world, especially since the computational burden of the Bayesian calculus often force over-simplistic assumptions for the sake of faster computations. Now, assumptions are violated al the time — this is not a problem in itself, and to over-engineer the assumption set is not always the right decision. What is problematic, though, is that, if the model assumptions are wrong, it is really hard to know whether the answers are still somewhat valid. Ironically, for more complex models, the problem gets worse, because complex models fail in complicated ways. This results in the following well-kept secret: 40-year old tools such as linear and logistic regression are still the workhorses of business intelligence and advanced analytics. Users will call them “reliable”. The correct technical term is “robust”, which, in statistics parlour, means, roughly speaking, a model that tends to remain pretty accurate even when its assumptions somewhat break down. Not all is lost of course — Bayesians are aware of this problem, and are working on it. But to this day off-the-shelf AI will not generally come with strong robustness guarantees, these have to be engineered into the system separately.

What about deep learning? Such methods are often referred to as ‘black-boxes’ , and typically involve a highly parameterised input-output relationship fitted to the data via optimisation using a training dataset. These algorithms are fascinatingly complex and correspondingly powerful (indeed at Mentat we are building our own deep learning library for IoT cybersecurity right now). However, their complexity, and the fact that parameters of deep learning models are not really meaningful in themselves (technically speaking, they are not generally viewed as random variables or population parameters), makes it incredibly difficult for them to report their own uncertainty. Deep learning is in many ways similar to deep thinkers, or our lizard brains: it knows the answer instinctively, but cannot explain why.

That this should be a challenge is not surprising. Self-criticism, or, more broadly, self-reflection is one of the most elusive, and important, aspects of intelligence. Indeed, a certain school of thought in the philosophy of mind lists the ability to self-reflect as the defining characteristic of consciousness. Needless to say, humans do not always exhibit such intelligence themselves: cognitive biases, racism, fanaticism are all in many ways consequences of the inability to question one’s assumptions. However, the human race as a whole has historically demonstrated exquisite abilities to self-reflect, and reason about assumptions about assumptions, a little like the nested Bayesian model above.

So how can we build “self-critical” thinking machines? Our answer here at Mentat is that robustness and diagnostics are the unsung heroes in real-world deployments of AI.

The former consists in introducing failsafes into models that allow them to decrease their confidence when they feel overstretched (shrinkage in classical machine learning is a wonderful example of a simple solution to a difficult problem, and analogous techniques are applicable in almost all AI/ML algorithms, albeit with much greater care and technical difficulty at times). Failsafes against the possibility that the data-generating process changes over time is in fact a core innovation in our Machine Learning In Motion toolbox, and an unrecognised performance bottleneck in many competing offerings. Model diagnostics complement robustness and stem from an approach that was all-the-craze in the golden era of classical statistics, consisting in tracking performance indicators of the model, and taking suitable action when they take unreasonable values. This may sound mundane (who doesn’t keep track of their model’s performance?) but the art of diagnostics is in constructing the right statistic, and understanding the range of values it should be allowed to take without raising an alarm — a challenging mathematical problem, when done correctly. Thankfully, recent methodological developments can allow us to do both tasks with generality.

There is, of course, a lot more to “self-reflection” than what was explored here, but self-diagnostic, robust model-based ML is an excellent starting point.

Some time ago, a member of our university study group, after a long period of intense, silent thinking aimed at solving an infamous maths past paper question, announced: “Am I talking cr@p? That is the question.” Simply put we should not fully trust Thinking Machines until we can witness them spontaneously make a similar claim.

Originally published at www.ment.at on 17-Mar-2015

Originally published at www.ment.at on 19-Feb-2015

Data collection and storage technology is transforming our world. Bicycles are connected to the internet, smartphones will become our personal physicians and cities have already become cashless. A connected world, full of opportunity — a Big Data dreamland. And yet the old paradigm of storing all this data is challenged. The sheer volume sometimes makes it impossible. But a more alarming downside is the potential for privacy infringement. Embarrassing pictures on Facebook are only the beginning. Consider arbitrary third parties knowing your physical location, your habits inside the privacy of your own home. A Big Data nightmare. So is the Big Data dream in danger?

Should we have to choose between Big Data and Privacy?

Let’s get more specific, to remind ourselves of exactly what is at stake. The three big telcos know exactly where every single Londoner is at any given time. A high-street retail chain might be interested to see the demographics of the people walking past its store on a typical Sunday, right now, or even better, tomorrow afternoon, so that they can promote items that appeal to this particular customer segment. Golden third-party sharing opportunities. 
 
Smart metering. The National Grid needs to anticipate demand surges so as to balance the grid as a whole. It only has information accurate down to sub-station. In an ideal world National grid would be able to have information down to the postcode level, ensuring non-storable energy (e.g., renewables) is used optimally and demand peaks are handled efficiently. In an ideal world the utilities would be able to detect daily and hourly patterns of electricity demand at the household level, and offer personalised tariffs that incentivise off-peak consumption. The end product: a balanced grid, cheaper energy for all, and reduced energy waste. 
 
Neither the Telcos nor the utilities are able to deliver this without a huge privacy risk: it only takes one incident to bring down the full force of the law, and irreparably damage the company’s reputation. 
 
The EU has been pushing for tighter legislation around privacy for a while, largely driven by the potential for large-scale privacy infringement by the data behemoths: Facebook, Google and similar organisations plus the governments of this world. The reality though is that even SMEs or an innocent app can cause irreparable damage to an individual if they misuse private information. What is being challenged here is a “tacit agreement” between consumer and digital service providers that has so far served the latter much better than the former: “click here to accept the terms and conditions about data usage”. One click can no longer offer a get-out-of-jail card for the data holder. Consumers that initially give consent to share data now retain the right to change their mind. The EU aptly calls it “the right to be forgotten”, and, let’s be honest, it seems fair. But is it enforceable?

Mentat thinks John Oliver probably has it right : No way !

Private data, once stored outside the confines of your home and personal devices, are at risk. A healthy new ecosystem will no doubt grow around ‘safer data control’ and ‘data provenance’, but the reality is, every lock can be picked, and as the amount of private information being stored and shared increases exponentially, so does the risk of it being abused. So it does seem that Big Data and privacy might be incompatible, after all.
 
But what if we could think outside that box? What if we could have it all?
 
 We feel the real culprit is a decades-old Business Intelligence practice which we refer to as the “store everything, analyse later” approach. Indeed, the standard workflow of a Business Intelligence unit involves storing everything about everyone in massive data warehouses and pulling reports at arbitrary points in time later to satisfy the appetite of the next internal Powerpoint presentation, or to help determine the next-best-action during a customer service call.

This “keep it, just in case” rationale is outdated, and is, increasingly, seen as offensive, as it rests on an assumption that private data is a free, up-for-grabs resource. This is precisely the viewpoint that is becoming untenable. 
 
 At Mentat, we have a different viewpoint. Our cornerstone belief is that “value lies with information, not with data”. Value is derived from actionable insights, which in turn rely on processed information, not on raw data. Of course in some settings, you do need raw data (e.g., for billing or auditing), but in the vast majority of Business Intelligence tasks, it is aggregate insights you want, rather than the private data itself. Well, you are in luck: with streaming analytics technology you can keep the insights, but throw away the data. 
 
 So how does it work? A suitable analogy is that of an experienced store manager: they have learnt on the basis of past experience how to judge a character and sell differently, but they don’t remember the national insurance numbers and postal addresses of every single customer they have interacted with. This analogy cuts deep: a learning agent glimpses at new data, and uses it to update their view of the world, without storing every single bit and byte.

Going back to the use case of the mobility map, we can learn from the data how different customer segments move around the city in real-time, anticipate flows as they propagate — has a tourist bus just deposited hundreds of potential customers on Marble Arch? How likely are they to come near your store? We call this an information asset, in contrast to a data asset. The information asset only ever holds aggregated information, that cannot be traced back to the individual. So it’s not all or nothing: most of the useful insights that you need, don’t actually need the private data to be stored. In an ideal world, you shouldn’t have to face dilemmas like this one. 
 
Smart metering. We define our information asset: a geographical predictive map of energy demand patterns in real-time, down to the lowest granularity that allows anonymity: e.g., the first half of the postcode. The techniques we use are compliant: they rely on the same principles used by, say, the Office for National Statistics. But our reports are real-time, so that National Grid has time to act and match demand and supply.
 
 The same principle applies to any personal device in the Internet-of-Things. Privacy risks associated with wearable health sensors connected to smartphones will dwarf the risk of smart metering data. It is critical to rely on information assets that are privacy-respecting. Imagine a real-time map of asthma attacks in the city of London, split by age-group, offering warnings to individuals at risk. 
 
 Make no mistake: the digital economy will look different in one or two years from now, and privacy-enhancing service providers will need to introduce an array of technologies to ensure their data-driven business models remain intact while respecting consumer privacy. Better authentication systems, or smarter data management solutions, like our fellow EY challengers Sedicii and Exonar will no doubt be part of the solution. But when it comes to business analytics, forward-thinking companies that embrace the empowering technology of on-the-fly aggregation using in-motion analytics will proudly gain the reputation of respecting not only the letter, but also the spirit of the law: private information will be glimpsed at, to ensure that consumer and service provider alike benefit from the Big Data value chain, but it will then be forgotten, literally milliseconds later. That’s what drives us at Mentat: true innovation, which can helps us break free of false dilemmas.

Originally published at www.ment.at on 8-Jan-2015