COVID-19: Clustering National Patterns

The COVID-19 pandemic has produced vast quantities of publicly available data that holds a prominent global interest. While the integrity and accuracy of the data has been scrutinized and questioned from nation to nation, the data still provides a great jumping point to explore some fundamental data science tools.

In this post, we will explore how nations compared over their first 40 days of COVID-19 exposure for both infections and mortality. The data set for national infections and deaths is large enough that a brute force curve by curve comparison would be a challenge. One strategy is to group nations that followed similar infection and mortality curves together and compare the groups rather than the specific nations. One of the common algorithms for executing this type of analysis is k-means clustering.

K-means clustering is a type of unsupervised machine learning that analyzes a data set and assigns k cluster centers such that 1) the cluster center is the arithmetic mean of all points in the cluster and 2) all points are closer to their own cluster center than any other cluster center. K-means then yields k sets of data that can be used to represent the entire data set under study. Further to this, new data can be analyzed and assigned to these previously determined clusters and anything learned from the original clusters can be assumed to apply to the new data.

National Infections by K-Means Cluster

This plot represents how countries are assigned to 4 k-means clusters based on their first 40-days of reporting more than 50 COVID-19 infections. Interestingly, of the 121 countries in the data set, 105 of them were assigned to cluster 0. The remaining 16 countries were spread amongst the top 3 clusters with the US claiming cluster 3 all to itself. In addition to ending with a much higher number of infections at the end of 40 days, the US followed a unique trajectory through most of the data set.

The Python Abridgment

In the code below, modelData is a dataframe containing infection (or mortality) data as y. x is the offset day from the date the country recorded more than 50 infections (or 10 deaths). The testData dataframe is simply a listing of countries from modelData that have more than 40 days of recorded data. This is important because the KMeans function does not like NaN values and to backfill them with data would influence the outcome of the clustering algorithm.

print(modelData.head(5).to_markdown())

|    |   x |   y | index       |
|---:|----:|----:|:------------|
|  0 |   0 |  11 | Afghanistan |
|  1 |   1 |  14 | Afghanistan |
|  2 |   2 |  14 | Afghanistan |
|  3 |   3 |  15 | Afghanistan |
|  4 |   4 |  15 | Afghanistan |

The following code is the basics for executing the k-means clustering algorithm and plotting the results.

import pandas as pd
import numpy as np
from sklearn.cluster import KMeans
import plotly as plotly
import plotly.express as px

# set our number of clusters
cluster_num = 4

# build a dataframe based on our model and test set and drop data > 40 days
clusterData = modelData.loc[modelData['index'].isin(testSet.index.values)]
clusterData = clusterData[(clusterData['x'] < 40)]

# set the index to the country name and the x value which will give us a column for y data
# each country and x value will have 40 y values in a single column
# unstack will put the y values across the columns where each column is the x value
# reset the index and drop the 'index' column which is the country name
clusterData = clusterData.set_index(['index','x']).unstack().reset_index()
growthCluster = clusterData.drop(columns="index")

# execute the k-means algorithm and fit the data
kmeans = KMeans(n_clusters=cluster_num)
kmeans.fit(growthCluster)

# predict the cluster for each row of the data set
y_kmeans = kmeans.predict(growthCluster)

# add the predicted cluster back into the dataframe that carries the country name as the index
clusterData['Cluster'] = y_kmeans

# set the index to identify Country and Cluster and then 'stack' the results
# the index is cleared and the result is 'tidy' data
plotData = clusterData.set_index(['index','Cluster']).stack().reset_index()

# Plot the data using plotly express
fig = px.line(plotData, x="x", y="y", color="Cluster", line_group="index", hover_name="index",
                  line_shape="spline", render_mode="svg")

Summary of K-Means Infection Clusters 1, 2 & 3

If we were interested in doing a deeper study on the hardest hit nations, we could then drop the countries that fell into cluster 0, representing the flattest overall curve, and focus our attention on the countries in the other three.

Infection Cluster Centers

Potentially more interesting than knowing which cluster each country falls into, the lines representing each clusters geometric center's can also be plotted by pulling the lists from the k-means data ( _y = kmeans.cluster_centers_[i]) where i is 0 -3. These lines represent the 4 models that k-means clustering could be used to predict against for countries outside of the data set. These could also be used for similar future infections (COVID-26?) to help nations determine which model they tracking closest to.

National Mortality by K-Means Cluster

Similarly, the same algorithm can be run against mortality data.

Summary of K-Means Mortality Clusters 1, 2 & 3

There were only 67 countries with more than 40 days of reporting deaths. Of these 67, 15 were assigned to clusters 1 - 3 and 52 were assigned to cluster 0. With mortality, the US ended with a higher number of deaths at the end of 40 days but because the curve was similar to both France and Italy, they were all assigned to the same cluster. Again, we can drop the countries in cluster 0 and focus attention on those in clusters 1 - 3.

Mortality Cluster Centers

Similarly, the mortality curve cluster centers can be extracted and plotted.

Summary of Infection and Mortality Clusters

With both sets of data, we can simply sum the two cluster numbers together to get a quick metric of how the country fared across both infections and mortality. All things being equal, you would expect close alignment between the infection and mortality clusters. Discrepancies here would be another opportunity to pull together more data and try to determine possible causes. For the sake of trimming our data set again, we will continue to ignore countries that were assigned to cluster 0 for both infections and mortality.

Further Analysis of Cluster Zero

If our interest was in studying the countries with more favourable COVID-19 trajectories over the first 40 days, we could simply focus our attention on the countries in cluster 0 and drop clusters 1 - 3. We could then continue with more k-means clustering on this sub-set to look for patterns and valid models in these countries as well. The following table shows our original cluster 0 sub-clustered into 4.

Covid-19 Data Source: https://github.com/CSSEGISandData/COVID-19

This is the data repository for the 2019 Novel Coronavirus Visual Dashboard operated by the Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE). Also, Supported by ESRI Living Atlas Team and the Johns Hopkins University Applied Physics Lab (JHU APL).

Thank you to them for making this data available.