Online newspaper subscriptions: using machine learning to reduce and understand customer churn

Figures & data

Table 1. Literature review on churn modelling in subscriptions business models.

Authors	Model	Results (R) and limitations (L)
De Caigny et al. (2018)	Support Vector Machines (SVM)	R: the SVM model outperforms a Random Forest (RF) algorithm; the ten most important predictors were extracted. The subscription’s length and the elapsed time since the last renewal are the most important predictors L: print newspaper subscription context; client-company interaction data was not considered; monetary value, frequency of renewals, and sociodemographic variables were not present in the most important variables
MittMedia (Ekholm, 2019)	Survival Analysis, Gradient Boost Machine (GBM), and Neural Network (NN)	R: Swedish media business uses models that complement each other in the churn prediction analysis. Insights into variables correlated with churn show that a high count of notification openings and female users were more correlated to churn. In contrast, a higher count of articles read or if the subscriber was a past paper subscriber were more correlated with a lower churn risk. Retention actions are focused on users with a likelihood to churn between 60% and 90%. L: scarce information about the data preparation and model performance.
Burez and Van den Poel (2007)	Logistic Regression (LR), Markov Chains, RF	R: Historical and current subscription data, socio-demographic data, and interactions with the customer churn prediction were studied on an imbalanced dataset (15% churners). The study results show that for cut-off values lower than 25%, the RF model outperforms other techniques; however, for larger cut-off values, both LR models perform as well as the RF and end up surpassing it. L: pay-TV sector.
(2006)	LR, Decision Tree (DT) with Bagging, Stochastic Gradient Boosting	R: Bagging and boosting classification techniques to predict churn have similar performance, nevertheless, bagging is considered the most competitive approach since it is conceptually less complicated than stochastic gradient boosting. Random oversampling applied to half of the training set keeping the other half of the training set with the original distribution (1.8% churn), show that a balancing schema was preferable. Adjusting the cut-off value is substantially better than assigning a weight to each observation in the training set. L: telecommunication sector; very small churn rate.
C. Tsai and Lu (2009)	NN, Self-Organizing Maps	R: By sequentially combining NN with NN and Self-Organising Maps with NN, the first technique filtered out unrepresentative training data, then, the second used the representative outputs to create the prediction model. Results indicated that the single NN was outperformed by both hybrid models in accuracy and Type I and II errors, being the combination of two NN the best performing model. L: telecommunication sector; black-box models.

Figure 1. Research framework.

Table 2. Features meaning, and periods covered.

	Variable Name	Type	Variable meaning	Period
Usage Variables	M_RFV_60D M_RFV_30D	Ordinal (e.g.: 597, 981, …)	Maximum RFV (Recency, Frequency, Volume) value	0D − 30D&30D − 60D
	N_visit_days_60D N_visit_days_30D	Numerical	Number of days with visits
	N_last_visit_60D N_last_visit_30D	Numerical	Days since the last visit
	N_articles_N30 N_articles_N60	Numerical	Total articles read
	N_exclusive_60D N_exclusive_30D	Numerical	Total exclusive-to-subscriber articles read
	N_readability_75_60D N_readability_75_30D	Numerical	Total articles in which 75% of the article was read
	engagement_Share_60D engagement_Share_30D	Numerical	Number of shares from the website to social media
	engagement_COMENT_60D engagement_COMENT_30D	Numerical	Number of comments on the website
	engagement_save_60D, engagement_save_30D	Numerical	Number of articles saved on the website
	N_30_pref_section	Categorical (e.g.: Society, Politics)	Preferred content section	0D − 30D
	NL_30_read	Numerical	Number of newsletters read
	NL_30_click		Number of newsletters clicked
Subscription Characteristics	A1_SUBSCRIPTION_ CAMPAIGN	Categorical (e.g.: Pense bem Pense Publico, Paywall, …)	Campaign related with current subscription	Subscription length
	end_date	Date	Subscription’s end date
	start_date	Date	Subscription’s start date
	SUBSCRITPION_CHANNEL	Categorical (e.g.: Online store, Call centre contact)	Channel through which the subscriber purchased the current subscription
	SUBSCRIPTION_TYPE	Binary	Binary indicating if the subscription is paid or offered
	SUBSCRIPTION_PRODUCT	Ordinal (e.g.: month, semester)	Subscription product duration
	SUBSCRIPTION_PAYMENT _MODE	Categorical (e.g.: credit card, paypal, …)	Payment method used
	SUBSCRIPTION_ ORIGINAL_DAYS	Numerical	Subscription length in days
	SUBSCRIPTION_IS _RECURRENT	Binary	Binary indicating if the subscription is automatically renewed
	SUBSCRIPTION_IS_TRIAL	Binary	Binary indicating if the price increases in the next subscription
	SUBSCRIPTION_VALUE _WITH_TAXES	Numerical	Subscription price
	SUBSCRIPTION _CLASSIFICATION	Categorical (e.g.: renovation, new client, …)	Customer’s relationship with the company at the purchase moment
Historical Variables	all_subscriptions	Numerical	Total subscriptions	Since the user registered the website
	avg_subscription_days	Numerical	Average subscription length in days
	max_subscription_days	Numerical	Maximum subscription length in days
	tenure_days	Numerical	Days since the first subscription
	fidelity_rate	Numerical	e_active_subs_days/c_tenure_days
	active_subs_days	Numerical	Days with active subscriptions
	i_product1	Numerical	Most purchased subscription length in days
	USER_create_date	Date	Subscriber registration on the website
Subscriber Socio demographics	gender	Categorical (e.g.: male, female)	Subscriber gender
	birthday	Date	Subscriber birthday
	job	Categorical (e.g.: lawyer, teacher, artist, …)	Subscriber job
Target	SUBSCRIPTION_IS_ACTIVE	Binary	Binary indicating if the subscription is active	30D after subscription ends
Stringency Index	avg_stringency_index_30, avg_stringency_index_60, avg_stringency_index_total	Numerical	Average stringency index (Global Change Data Lab, 2021)	0D − 30D,30D − 60D,0D − 60D

Table 3. Parameters considered for grid search cross-validation per model.

	XGBoost	LightGBM	Random Forest	Decision Tree
learning rate	[0.1, 0.2, 0.5]	[0.01, 0.1, 0.2]
max_depth	[2, 3, 5, 7]	[−1, 2, 3, 5, 7]	[None, 2, 3, 5, 7]	[None, 2, 3, 5, 7,8]
n_estimators	[200, 500, 1000]	[200, 500, 1000]	[200, 500, 1000]
base_score	[0.2, 0.5]
colsample_bytree	[0.5, 0.8]	[0.5, 0.8]	[“auto”, 5, 8]	[None, 5, 8]
subsample	[0.3, 0.5, 0.8]	[0.3, 0.5, 0.8]
gamma	[1, 20, 50, 100]
min_samples_leaf		[None, 100, 200]	[1, 100, 200]	[1, 100, 200]
num_leaves		[5, 10, 20, 30]
subsample_for_bin		[500, 1000, 200000]

Figure 2. Daily predictions pipeline.

Table 4. Evaluation metrics of models trained with original and balanced data for both modelling contexts.

		Accuracy		Balanced Accuracy		Precision		Recall		F1		AUC		Specificity
	Algorithm_dataset	Train	Validation (Val)	Train	Val	Train	Val	Train	Val	Train	Val	Train	Val	Train	Val
Model 1	DT_original	94.52%	94.58%	82.09%	82.65%	68.34%	64.32%	67.08%	68.52%	67.71%	66.35%	90.56%	91.86%	97.09%	96.78%
	DT_smote	82.45%	94.57%	82.45%	82.54%	95.94%	64.31%	67.77%	68.28%	79.43%	66.24%	91.90%	91.83%	97.13%	96.79%
	RF_original	92.97%	93.28%	91.42%	89.43%	55.55%	54.47%	89.54%	84.87%	68.57%	66.35%	98.00%	96.45%	93.30%	93.99%
	RF_smote	93.31%	92.24%	93.31%	88.92%	92.84%	50.18%	93.85%	84.99%	93.34%	63.10%	98.63%	95.73%	92.76%	92.85%
	LGB_original	99.48%	96.08%	99.71%	87.84%	94.24%	73.38%	100.00%	78.09%	97.03%	75.66%	100.00%	95.22%	99.43%	97.60%
	LGB_smote	94.20%	92.55%	94.20%	88.26%	93.63%	51.42%	94.85%	83.17%	94.24%	63.55%	98.67%	95.64%	93.55%	93.35%
	XGB_original	95.10%	95.15%	91.92%	88.67%	66.04%	65.27%	88.07%	80.99%	75.48%	72.29%	98.38%	96.10%	95.76%	96.35%
	XGB_smote	98.10%	92.52%	98.10%	85.47%	98.16%	51.41%	98.04%	77.12%	98.10%	61.69%	99.81%	93.13%	98.16%	93.83%
Model 2	DT_original	93.60%	91.02%	93.89%	91.03%	75.15%	76.80%	94.32%	91.05%	83.65%	83.32%	98.62%	95.41%	93.45%	91.01%
	DT_smote	88.75%	89.78%	88.75%	86.87%	92.31%	78.20%	84.54%	81.14%	88.26%	79.64%	92.80%	91.97%	92.96%	92.61%
	RF_original	95.10%	92.73%	93.63%	92.35%	82.33%	81.28%	91.37%	91.61%	86.62%	86.13%	98.82%	97.59%	95.89%	93.10%
	RF_smote	88.66%	91.97%	88.66%	89.45%	94.89%	83.16%	81.71%	84.49%	87.81%	83.82%	97.06%	96.22%	95.60%	94.41%
	LGB_original	94.66%	92.44%	93.90%	92.59%	79.75%	79.75%	92.73%	92.89%	85.75%	85.82%	98.85%	97.71%	95.06%	92.29%
	LGB_smote	94.84%	91.79%	94.84%	91.51%	94.54%	78.92%	95.19%	90.97%	94.86%	84.52%	98.94%	97.47%	94.50%	92.06%
	XGB_original	95.64%	93.70%	93.21%	92.62%	85.96%	84.92%	89.48%	90.49%	87.69%	87.62%	98.81%	97.69%	96.93%	94.75%
	XGB_smote	95.31%	92.04%	95.31%	91.71%	94.81%	79.59%	95.87%	91.05%	95.34%	84.94%	99.13%	97.65%	94.76%	92.37%

Table 5. Evaluation metrics from selected models.

Model	Dataset	Accuracy	Balanced Accuracy	Precision	Recall	F1	AUC	Specificity
1	Train	95.10%	91.92%	66.04%	88.07%	75.48%	98.38%	95.76%
	Test	95.15%	88.67%	65.27%	80.99%	72.29%	96.10%	96.35%
2	Train	95.64%	93.21%	85.96%	89.48%	87.69%	98.81%	96.93%
	Validation	93.70%	92.62%	84.92%	90.49%	87.62%	97.69%	94.75%
	Test	93.64%	92.57%	82.45%	90.64%	86.35%	98.34%	94.49%

Figure 3. Features importance ranking by SHAP values.

Figure 4. SHAP dependency plots extracted from some model 1 predictors.

Figure 5. Average stringency index per dataset.

Table 6. Retention actions results for model 2.

	Actions
Group	Newsletter	Phone Call	Active	Churn
Variation	not open NL	no phone answer	2	18
		no phone	28	36
		phone answer	20	24
	open NL	phone answer	4	8
		no phone answer	3	7
		no phone	22	21
control	No Action		86	108
non churner			804	95

De Caigny, A., Coussement, K., & De Bock, K. W. (2018). A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees. European Journal of Operational Research, 269(2), 760–772. https://doi.org/10.1016/j.ejor.2018.02.009

 Web of Science ®Google Scholar

Ekholm, P. (2019). Machine learning challenge winning solutions. https://medium.com/mittmedia/we-looked-into-why-our-subscribers-churned-with-machine-learning-d1ac1adf8009

 Google Scholar

Burez, J., & Van den Poel, D. (2007). CRM at a pay-TV company: Using analytical models to reduce customer attrition by targeted marketing for subscription services. Expert Systems with Applications, 32(2), 277–288. https://doi.org/10.1016/j.eswa.2005.11.037

 Web of Science ®Google Scholar

Tsai, C., & Lu, Y. (2009). Customer churn prediction by hybrid neural networks. Expert Systems with Applications, 36(10), 12547–12553. https://doi.org/10.1016/j.eswa.2009.05.032

 Web of Science ®Google Scholar

Global Change Data Lab. (2021). Our World Data. https://ourworldindata.org/

 Google Scholar