Revision as of 01:35, 23 May 2024

INTRODUCTION

Depression is a collective issue found globally with a rising diagnosis and frequency that negatively impacts the quality of life for those afflicted, especially in elderly populations and often requires medications (1). Vitally, diagnosis allows individuals to acquire medications, therapies, and awareness to a possibly life changing announcement from a professional. Unfortunately, diagnosis is often after extensive development of mental disease and hence detectable by professionals with a measly 50% accuracy (2).

Machine learning is an oncoming development for perceived artificial intelligence which attempts to copy the brain capabilities of a human. Notably, machine learning has proven excellence in detecting themes that are typically difficult for a human to recognize such as underlying consistency between prominent labels. Prominent evidence regarding consistent accuracy in detecting specific mental illness is uncertain, as such more research is needed. Utilizing machine learning with social media text could detect depression early to allow individuals higher risk assessment and hence more time to develop proper response to this issue.

(Figure 1) - RNN Model Visualization (Adapted from Gupta). Visual representation of the recursive nature of an RNN, both in a condensed and uncondensed form.

Natural Language Processing (NLP) is the process of turning natural language into a form which is readable to a computer typically for the purpose of machine learning. This is a broad term to encompass anything from speech recognition to machine translation. NLP has preprocessing techniques for data such as padding, tokenization, lemmatization, and truncation. Padding is used to normalize data to have all values equal length done by adding filler tokens. Tokenization is the act of turning sentences into individual tokens representing each word numerically. Lemmatization is the act of turning words into their base form regarding context to both reduce complexity and introduce normalization of words, an example would be computing turned into compute (3).

Recurrent Neural Networks (RNNs) are a form of machine learning innovative for their recursive nature and self-attention (Figure 1). These models are used in NLP for speech recognition and text generation as self-attention encompasses a contextual aspect from textual data. RNN’s are fundamental to many text-based processes which had the flaw of gradient decline resulting in loss of attention across large amounts of data (4). This was improved upon following Long Short-Term Memory (LSTM) models which introduces crucial information to all recursive nodes in the network, this line of information gathers importance and alters accordingly (5). An RNN is a fundamental aspect of the BERT model and shares similar core components (4).

(Figure 2) - Transformer Model Visualization (https://www.baeldung.com/cs/rnns-transformers-nlp) Figure representing the architecture associated with Transformers, the fundamental aspect of BERT.

Transformers added upon RNN models by utilizing the self-attention mechanism but differed by removing the recursive element (Figure 2). Furthermore, instead of individual processing of each input, the model worked utilizing parallelism of multiple tokens at once. All forms of transformers have an Encoder layer, with some having a Decoding layer. The feed-forward design introduces self-attention similarly to LSTM models, while also having a multi-head attention layer which is the core of the Transformer’s parallelism (6).

BERT stands for the Bidirectional Encoder Representations (from) Transformers. BERT was created by Google’s AI Language department to create a competitive RNN alternative specifically for word processing (7). BERT uses weighting to value all words equally with no representation to word index and will adjust these ‘weights’ to value phrases or specific keywords with higher correlation than others. This creates a model which searches for both specific words and general abstraction. BERT has been used robustly from sentiment analysis to language translation (8). This robustness is increased by the development of BERT expansions, which train the model with specific large data characteristics, such as medical or robust data (9).

Social media posts and text are an excellent database for machine learning and more specifically for detecting mental illness with high accuracy (10). While identifying information regarding supposed diagnosis legitimacy cannot be achieved, all data being self-reported sustains authenticity to a degree that falsity will be trained out given large amounts of data (11).

Social media text processing has been utilized for mental evaluation prior, especially regarding multiclass identification (12). The purpose of this project was to create a machine learning model which could accurately detect depression in social media messages with the goal of assisting psychiatrists. This was done as former studies concluded fluctuating results among attempted machine learning models on similar marks of identifying depression from social media messages (13). Furthermore, a rising need for social workers, psychiatrists, psychologists (14) infer that those available would benefit from the use of an application in their work field. Machine learning is a rapidly growing field with constant developments warranting previous research as a basis, not an expectation. The use of BERT was dictated by availability of model, learnability, and previous use. Likewise, BERT being open source allowed for flexibility of internal hidden layers. This led to the hypothesis of utilizing BERT on social media messages to detect the presence of depression.

RESULTS

Hyper Tuning trials of models created various combinations which resulted in high accuracy (above 95%) and did not contain many similarities between models (Table 1). The final model was chosen based on a combination of high dropout and low epoch count to discourage overfitting to a maximum. Utilizing a low token count allowed for less data to still give accurate results as the model was trained on minimal tokens. Difficulty comparing outputs to another dataset’s values as depressive data is often non-accessible without prior access, other forms of output do exist such as sentimental analysis but attempted output to a basis similar to sentimental (0-2) proved inconsistent results which fluctuated depending on the database. The use of a single database may introduce a selection bias as subjects are from solely Reddit, exact collection methods are not documented except binary labels which were self-reported, in aid of user anonymity. Comparing with a validation set ensures prevention of overfitting among model training and cross validation confirms accuracies with varying inputs, while using a stable randomization of data proved model consistency. Concluding, accurate results allow specific phrases or themes to be confirmed discriminants on detecting diagnosed individuals from non-diagnosed individuals, such “key words” could be used as a basis for further research.

(Table 1) - HyperTuning of Models. Accuracy Regarding Multi-Parameter HyperTuning with 128 Tokens, 33 individual pretrained models with labeled hyper parameters and accuracies.

DISCUSSION

Testing between higher tokenization values such as 256 to incorporate more textual information was tested, which resulted in similar accuracies with far longer training cycles with a model taking multiple hours compared to several minutes. Higher tokens theoretically incorporated higher contextual information from messages but finalized validation accuracy with HyperTuning proved similar results when compared with 128 tokenization (+/- 0.5%). Processing time increased dramatically, as the double of tokens across thousands of inputs increased complexity.

In addition, the implementation of multiple NLP processing techniques such as the removal of keywords, pre-tokenizing, and padding were also tested. The removal of keywords removed contextual information from messages which fundamentally altered the text, the advantage of BERT’s ability to understand context would be nullified by this. As such the removal of keywords showed a lower (85%) accuracy with no HyperTuning. Pre-tokenization conflicted with BERT’s own tokenization, while processing ran with similar efficiency to finalized models, the model lacked readability of inputs with pre-tokenized inputs. Lastly, padding is introduced in BERT’s encoder and tested individually which normalized data into a form which would prevent bias from input lengths, this was a vital part of ensuring validation accuracy. The final model’s accuracy in testing data supported the high training accuracy (Table 2).

(Table 2) – Confusion Matrix Visualized (https://abeyon.com/ai-performance-measurement-f1score/). Adapted by Student) Figure representing the highest accuracy model testing results.

Machine learning can be a highly successful application for the detection of depression in textual information. Improvement of previous studies was achieved by a vast majority, and the potential for application among the diagnosis of individuals shows to be increasingly promising. The model architecture is not defined solely for the purpose of depression, only the dataset, which allows adoption into other Binary Classifications. The availability of code for creation of models utilizing BERT is minimal, this implementation also may serve as assistance for the creation of BERT-Hybrid models.

MATERIALS AND METHODS

General Procedure

Programming followed a linear creation originating with simple experimentation of BERT encoding and single node dense layers to grid search of multi-layer dense and dropout layers. Versions visibly progressed in both complexities, testing availability, output, and model architecture. Architecture derived from the need of complexity, meaning if the model functioned with high accuracy (95%+) with minimal hidden layers, there was no need to alter this. The data’s split between Depression and Non-Depression was also confirmed (Figure 4).

(Figure 3) – Dataset Depression Non-Depression Split (Graphic via Student, TensorFlow). Figure representing the split across Depressed and Non-Depressed data samples

Final Procedure

Modules such as Tensorflow-Text, Keras-Tuner, TensorFlow, and Sklearn were imported to be utilized in the creation of the model. TensorFlow as the main module for the BERT model, Keras-Tuner for grid search, and Sklearn for test-train-split. Data was imported from kaggle/depression-reddit-cleaned as a TensorFlow dataframe, which had two columns; clean_text which contains the textual information of the message and is_depression which contains a binary value representing if the individual has depression. Each textual portion had 2-7 sentences of information which was gathered through multiple Subreddits without documentation from specific locations. Data was already cleaned and ready to be used by the dataset owner. The Sklearn; Test-Train-Validation-Split, of data with 60% used for training, 20% used for testing, and 20% used for validation. Ensuring the same random_state for comparison. Clean_text is normalized through the BERT encoder inside the model architecture, ensuring that each message has 128 tokens and identical tokenization style. The BERT tokenizer and encoder are from the smallest BERT Model; bert_en_uncased_L-2_H-128_A-2 and bert_uncased_preprocess_3. These preprocessors tokenize inputs to 128 tokens, perform lemmatization, and apply padding. The model was created with a dropout layer, hidden layer, dense layer, two activation functions of RELU and Sigmoid, and an output layer. It utilized an AdamW optimizer, which was chosen by a variety of training models, comparing results, and discovery of general machine learning models utilizing AdamW in textual applications. Keras-Tuner grid search was utilized to test random values for dropout percentage, hidden layer node amount, and HyperTuning of AdamW optimizer: such as decay_steps and decay_rate. The HyperTuning values were chosen on variable dependent specifications, such as the necessity of a Dropout, this also determined total training time across all models (Table 3). Most scaling are logarithmic based on the large difference between the minimum and maximum values. Keras-Tuner gathered best validation accuracy from various model performance, and the highest accuracy model is utilized.

Table 3 – Hyper Tuning Scale Figure. Table representing the hyperparameter values scaled and ranges.

Utilization Procedure

The appendix contains direct code relating to the entirety of the Final Procedure. Inside the code contains comments on both running and testing the model. The entire document downloaded into a python file and was intended to be run inside of Google Collab as a .ipynb file. The first section labeled

ACKNOWLEDGMENTS

Thank you to Ms. Beatty for reviewing and assisting with this project!

REFERENCES

1. Hofmann, Stefan G., et al. “Effect of treatments for depression on quality of life: A meta-analysis.” Cognitive Behaviour Therapy, vol. 46, no. 4, 25 Apr. 2017, pp. 265–286, https://doi.org/10.1080/16506073.2017.1304445.

2. Reece, Andrew G, and Christopher M Danforth. “Instagram Photos Reveal Predictive Markers of Depression - Epj Data Science.” SpringerOpen, Springer Berlin Heidelberg, 5 Sept. 2017, doi.org/10.1140/epjds/s13688-017-0110-z.

3. “What Is NLP? - Natural Language Processing Explained - AWS.” What Is Natural Language Processing (NLP)?, Amazon, 2024, aws.amazon.com/what-is/nlp/.

4. GfG. “ .” Introduction to Recurrent Neural Network, GeeksforGeeks, 4 Dec. 2023, www.geeksforgeeks.org/introduction-to-recurrent-neural-network/.

5. “Recurrent Neural Networks.” Understanding LSTM Networks -- Colah’s Blog, 27 Aug. 2015, colah.github.io/posts/2015-

08-Understanding-LSTMs/.

6. Baeldung. “From RNNs to Transformers.” Baeldung for Computer Science, 23 Jan. 2024, www.baeldung.com/cs/rnns-transformers-nlp.

7. Devlin, Jacob, et al. “Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” arXiv.Org, 24 May 2019, doi.org/10.48550/arXiv.1810.04805.

8. Turchin, Alexander, et al. “Comparison of Bert implementations for natural language processing of narrative medical documents.” Informatics in Medicine Unlocked, vol. 36, 2023, p. 101139, https://doi.org/10.1016/j.imu.2022.101139.

9. Liu, Yinhan, et al. “Roberta: A Robustly Optimized Bert Pretraining Approach.” arXiv.Org, 26 July 2019, doi.org/10.48550/arXiv.1907.11692.

10. Kim, Nam Hyeok, et al. “Analysis of depression in social media texts through the patient health questionnaire-9 and natural language processing.” DIGITAL HEALTH, vol. 8, Jan. 2022, p. 205520762211142, https://doi.org/10.1177/20552076221114204.

11. “The Importance of Having Large Sample Sizes for Your Research.” Charlesworth Author Services Authors, www.cwauthors.com/article/importance-of-having-large-sample-sizes-for-research.

12. Zanwar, Sourabh, et al. “Exploring Hybrid and Ensemble Models for Multiclass Prediction of Mental Health Status on Social Media.” arXiv.Org, 19 Dec. 2022, doi.org/10.48550/arXiv.2212.09839.

13. Seth, Pratinav, and Mihir Agarwal. “Uatta-EB: Uncertainty-Aware Test-Time Augmented Ensemble of BERTS for Classifying Common Mental Illnesses on Social Media Posts.” arXiv.Org, 10 Apr. 2023, doi.org/10.48550/arXiv.2304.04539.

14. “Projecting Health Workforce Supply and Demand.” HRSA Health Workforce, bhw.hrsa.gov/data-research/projecting-health-workforce-supply-demand. Accessed 2 Mar. 2024.

APPENDIX

1. -*- coding: utf-8 -*-

"""Depression_Classification8.ipynb

Automatically generated by Colab.

Original file is located at

https://colab.research.google.com/drive/1Fb67GkbTcbpU9aK4bTuWK6_68E0p0Z_o

"""

!pip install -U "tensorflow-text==2.13.*"

!pip install keras-tuner

import os

import numpy as np

import pandas as pd

import shutil

import tensorflow as tf

import tensorflow_hub as hub

import tensorflow_text as text

from tensorflow import keras

from sklearn.model_selection import train_test_split

import matplotlib.pyplot as plt

import keras_tuner as kt

tf.get_logger().setLevel('ERROR')

["""https://www.kaggle.com/datasets/infamouscoder/depression-reddit-cleaned """https://www.kaggle.com/datasets/infamouscoder/depression-reddit-cleaned]

>

Dataset here

"""

url = ['https://drive.google.com/file/d/1ayX-QruP29SWjSZLqT0-JSVyL5D7NrSu/view?usp=sharing'#Depression 'https://drive.google.com/file/d/1ayX-QruP29SWjSZLqT0-JSVyL5D7NrSu/view?usp=sharing'#Depression] default

[url='https://drive.google.com/uc?id=' url='https://drive.google.com/uc?id='] + url.split('/')[-2]

df = pd.read_csv(url)

df

"""# Complete Model Training"""

X_train, X_test, y_train, y_test = train_test_split(df['clean_text'],df['is_depression'], test_size = .2, random_state = 1)

X_train, X_val, y_train, y_val = train_test_split(X_train,y_train, test_size = .25, random_state = 1)

tfhub_handle_encoder = ['https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1' 'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1']

tfhub_handle_preprocess = ["'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'" "'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3'"]

print(f'BERT model selected  : {tfhub_handle_encoder}')

print(f'Preprocess model auto-selected: {tfhub_handle_preprocess}')

bert_preprocess_model = hub.KerasLayer(tfhub_handle_preprocess)

bert_model = hub.KerasLayer(tfhub_handle_encoder)

def build_hypermodel(hp):

text_input = tf.keras.layers.Input(shape=(), dtype=tf.string,name='sentences')

preprocessing_layer=hub.KerasLayer(tfhub_handle_preprocess,name="preprocessing")

encoder_inputs= preprocessing_layer(text_input,training=True)

encoder = hub.KerasLayer(tfhub_handle_encoder,trainable=True, name='BERT_encoder')

outputs = encoder(encoder_inputs)

neural_net = outputs['pooled_output']

1. Hyperparameters to tune

hp_dropout = hp.Float('dropout', min_value=0.2, max_value=0.5, step=.05)

hp_units = hp.Int('units', min_value=16, max_value=64, step=2)

neural_net = tf.keras.layers.Dropout(hp_dropout)(neural_net)

neural_net = tf.keras.layers.Dense(hp_units, activation='relu')(neural_net)

neural_net = tf.keras.layers.Dense(1, activation='sigmoid', name='classifier')(neural_net)

lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(

initial_learning_rate= hp.Float('init_lr', min_value=1e-5, max_value=5e-5, step=1e-6),

decay_steps=hp.Int('decay_steps', min_value=10000, max_value=20000, step=5000),

decay_rate=hp.Float('decay_rate', min_value=1e-5, max_value=1e-3, sampling='log')

)

1. AdamW

optimizer = tf.keras.optimizers.AdamW(learning_rate=lr_schedule,

weight_decay = hp.Float('weight_decay', min_value=0.001, max_value=0.1, sampling='log'),

beta_1=hp.Float('beta_1',min_value=0.9,max_value=0.99,sampling='log'),

beta_2=hp.Float('beta_2',min_value=0.999,max_value=0.99999,sampling='log'),

epsilon=hp.Float('epsilon',min_value=1e-8,max_value=1e-6,sampling='log'),

name='Adamw')

model = tf.keras.Model(text_input, neural_net) #Passed into model

model.compile(

optimizer=optimizer,

loss='binary_crossentropy',

metrics=['accuracy']

)

return model

tuner = kt.Hyperband(

build_hypermodel,

objective='val_accuracy',

max_epochs=3,

factor=2,

directory=os.path.join('drive','MyDrive','GridSearch'), # Directory to store logs and checkpoints, note this will break unless redefined to personal directory

project_name='Trial 4'

)

1. Get the best hyperparameters

best_hps = tuner.get_best_hyperparameters(num_trials=1)[0]

1. Build and compile the final model with the best hyperparameters

final_model = tuner.hypermodel.build(best_hps)

1. Train the final model with the entire training set

final_model.fit(X_train, y_train, epochs=5, validation_data=(X_val, y_val))

tf.keras.utils.plot_model(final_model)

final_model.save("DepressionClassificationModelV8")

"""# Load the model

> Training skipped

"""

1. Commented out IPython magic to ensure Python compatibility.

1.  %cd /content

!mkdir DepressionClassificationModelV8

1.  %cd DepressionClassificationModelV8

!mkdir variables

1.  %cd variables

!wget --no-check-certificate ['https://docs.google.com/uc?export=download&id=1-Y6J1XGkQhTXT1l1FkSpmQ2zY0H3jrrs' 'https://docs.google.com/uc?export=download&id=1-Y6J1XGkQhTXT1l1FkSpmQ2zY0H3jrrs'] -O variables.index

!wget --no-check-certificate ['https://docs.google.com/uc?export=download&id=1-PM-9FQyHjmCwv6hQG2KfkNxnOZUc_4U' 'https://docs.google.com/uc?export=download&id=1-PM-9FQyHjmCwv6hQG2KfkNxnOZUc_4U'] -O variables.data-00000-of-00001

1.  %cd /content

1.  %cd DepressionClassificationModelV8

!mkdir assets

1.  %cd assets

!wget --no-check-certificate ['https://docs.google.com/uc?export=download&id=1-Vq8Br4gtbiQaOTNaO2G2LE8cyv1DM0g' 'https://docs.google.com/uc?export=download&id=1-Vq8Br4gtbiQaOTNaO2G2LE8cyv1DM0g'] -O vocab.txt

1.  %cd /content

1.  %cd DepressionClassificationModelV8

!wget --no-check-certificate ['https://docs.google.com/uc?export=download&id=1-P4gGGS-hzxZFHIm5LCcGBJj2aHsiGuX' 'https://docs.google.com/uc?export=download&id=1-P4gGGS-hzxZFHIm5LCcGBJj2aHsiGuX'] -O saved_model.pb

!wget --no-check-certificate ['https://docs.google.com/uc?export=download&id=1-Jlz2RaFcrSR-6gAjCRvqvqpyjTTneg3' 'https://docs.google.com/uc?export=download&id=1-Jlz2RaFcrSR-6gAjCRvqvqpyjTTneg3'] -O keras_metadata.pb

!wget --no-check-certificate ['https://docs.google.com/uc?export=download&id=1-DDZEh_iH3sXEFhD592d6oaZO98fPmCb' 'https://docs.google.com/uc?export=download&id=1-DDZEh_iH3sXEFhD592d6oaZO98fPmCb'] -O fingerprint.pb

fm = tf.keras.models.load_model("/content/DepressionClassificationModelV8")

fm.summary()

"""# Test Model"""

text = np.array([input("Enter an input")])

prediction = final_model.predict(np.array(text))

print(prediction)

loss, accuracy = final_model.evaluate(x=X_test, y=y_test)

print(f'Loss: {loss}')

print(f'Accuracy: {accuracy}')