Feature Extraction

Feature extraction techniques explore the relationships and dependencies between features to create a new dataset of transformed features. Unlike feature selection methods that only select the best features from the existing ones, feature extraction models go further by generating new features that capture the essential information of the original data. This transformed dataset not only has a lower dimensionality but also retains the interesting and intrinsic characteristics of the original data.

The goal of feature extraction is to reduce the complexity of the model by transforming the original features into a new representation that captures the most relevant information. This can lead to improved efficiency, reduced generalization error, and decreased overfitting, as the model can focus on the most informative aspects of the data.

There are different perspectives on what properties or content from the original dataset should be preserved in the transformed dataset, resulting in a wide range of feature extraction techniques. Some of the most popular feature extraction algorithms include Principal Component Analysis (PCA), Linear discriminant analysis (LDA) and Non-Negative Matrix Factorization (NMF).

Principal Component Analysis

Principal Component Analysis (PCA) is a widely used feature extraction method that aims to transform the original features into a new set of uncorrelated features called principal components. The key idea behind PCA is to capture the maximum amount of variance in the data with a reduced number of features.

PCA achieves this by finding a set of orthogonal axes, called principal components, that represent the directions of maximum variance in the data. The first principal component captures the most significant variation, followed by the second principal component, and so on.

In the above image the two principal components of a dataset are shown, easily we can identify that the amount of information in the direction of blue axis is greater than the red axis (these axes are called eigenvectors or principal components). To reduce the dimensionality of the dataset, we simply have to make a projection on the first principal components (the exact number of components is chosen by the user).

By selecting a subset of the most important principal components, PCA effectively reduces the dimensionality of the data while preserving the most significant information. PCA is particularly useful when dealing with highly correlated features or when the data is characterized by a large number of dimensions. It helps in visualizing and understanding the underlying structure of the data and can be used as a preprocessing step for various machine learning tasks.

Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) is a feature extraction technique commonly used in classification tasks. Unlike PCA, which focuses on maximizing variance, LDA aims to find a linear combination of features that maximizes the separability between different classes. This is achieved by mapping the data into a lower-dimensional space while maximizing the between-class distances and minimizing the within-class variances. It finds a set of discriminant functions that maximize the ratio of between-class scatter to within-class scatter.

By projecting the data onto the derived discriminant axes, LDA creates a new feature space where the classes are well-separated, making it easier for classification algorithms to discriminate between different classes. LDA is particularly effective when there is a clear class separation in the data and can improve classification performance by reducing the dimensionality of the feature space.

Non-Negative Matrix Factorization

Non-Negative Matrix Factorization (NMF) is a feature extraction method that decomposes the original data matrix into two low-rank matrices: one representing the basis or feature vectors and the other representing the coefficients. NMF assumes that the original data can be represented as a linear combination of non-negative basis vectors.

NMF is particularly useful for data with non-negative values, such as text data or image data. It can be applied to extract meaningful components or topics from text documents or to represent images in terms of interpretable parts.

NMF iteratively updates the basis vectors and coefficients until it converges to a representation that captures the most important features of the data. The resulting low-dimensional representation can be used for various tasks, including clustering, visualization, or as input for subsequent machine learning algorithms.