DSP LOG
type: channel
description: Signal Processing
date: 2026-03-14T05:40:44Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: Loss functions for handling class imbalance
items: Word Embeddings using neural networks
items: Gradients for multi class classification with Softmax
items: Gradients for Binary Classification with Sigmoid
items: Gradients for linear regression
items: Migrated to Amazon EC2 instance (from shared hosting)
items: GATE-2012 ECE Q28 (electromagnetics)

Loss functions for handling class imbalance
type: item
creator: Krishna Sankar
date: 2026-03-05T01:05:23Z
subject: Machine Learning
subject: Assymetric loss
subject: Class Balanced Loss
subject: Focal Loss
subject: Logit Adjusted Loss
subject: Weighted Cross Entropy
description: For handling class imbalance, multiple stratetgies have emerged. Post covers Weighted Cross Entropy, Focal Loss, Assymetric Loss, Class Balanced Loss and Logit Adjusted Loss. The post Loss functions for handling class imbalance appeared first on DSP LOG.
encoded: Most of real world datasets have class imbalance, where a “majority” class dwarfs the “minority” samples. Typical examples are – identifying rare pathologies in medical diagnosis or flagging anomalous transactions to detect fraud or detecting sparse foreground objects from vast background objects in computer vision to name a few. The machine learning models we have discussed – binary classification (refer post Gradients for Binary Classification with Sigmoid) or multiclass classification (refer post Gradients for multi class classification with Softmax) needs tweaks to learn from these imbalanced datasets. Without these adjustments, the models can “cheat” by favouring the majority class and can report a pseudo high accuracy though the class specific accuracy is low. Different strategies have emerged over the years, and in this article we are covering the approaches listed below. Weighted cross entropy Foundational baseline, where a class-specific weight factor to the standard cross-entropy loss to weight the loss based on frequency of the class. Focal Loss for Dense Object Detection, Lin et al. (2017) Propose a modulating factor to the cross-entropy loss to down-weight easy/frequent examples which indirectly forces the model to focus on hard/rare examples Asymmetric Loss for Multi-Label Classification, Ridnik et al. (2021) Extended the intuition of Focal Loss by having independent hyper-parameter for positive and negative samples. This allows for more aggressive “pushing” of easy/frequent examples while preserving the gradient signal for hard/rare samples. Additionally, authors introduces a probability margin that explicitly zeros out the loss from easy/frequent samples. Class-Balanced Loss Based on Effective Number of Samples, Cui et al. (CVPR 2019) Based on the intuition that there are similarities among the samples, authors propose a framework to capture the diminishing benefit when more datasamples are added to a class. Long-tail Learning via Logit Adjustment, M…

Word Embeddings using neural networks
type: item
creator: Krishna Sankar
date: 2025-12-27T15:12:18Z
subject: Machine Learning
subject: CBOW
subject: Embeddings
subject: GLoVE
subject: Hierarchical SoftMax
subject: NCE
subject: Negative Sampling
subject: Noise Contrastive Estimation
description: The post covers various neural network based word embedding models. Starting from the Neural Probabilistic Language Model from Bengio et al 2003, then reduction of complexity using Hierarchical softmax and Noise Contrastive Estimation. Further works like CBoW, GlOVe, Skip Gram and Negative Sampling which helped to train on much higher data. The post Word Embeddings using neural networks appeared first on DSP LOG.
encoded: In machine learning, converting the input data (text, images, or time series) —into a vector format (also known as embeddings) forms a key building block for enabling downstream tasks. This article explores in detail the architecture of some of the neural network based word embedding models in the literature. Papers referred : Neural Probabilistic Language Model, Bengio et al 2003 proposed a neural network architecture to jointly learn word feature vector and probability of words in a sequence. Hierarchical Probabilistic Neural Network Language Model, Morin & Bengio (2005) given the softmax layer for finding the probability scales with vocabulary, proposed a hierarchical version of softmax to reduce the complexity from to . Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics, Gutmann et al 2012, Instead of directly estimating the data distribution, noise contrastive estimation estimates the probability of a sample being from the data versus from a known noise distribution. This approach was extended to neural language models in the paper A fast and simple algorithm for training neural probabilistic language models A Mnih et al, 2012. Efficient Estimation of Word Representations in Vector Space, Mikolov et al 2013. proposed simpler neural architectures with the intuition that simpler models enable training on much larger corpus of data. Continuous Bag of Words (CBOW) to predict the center word given the context, Skip Gram to predict surrounding words given center word was introduced. Distributed Representations of Words and Phrases and their Compositionality, Mikolov et al 2013 speedup provided by sub sampling of frequent words helps to improve the accuracy of the less-frequent words simplified variant of Noise Contrastive estimation called Negative Sampling GloVe: Global Vectors for Word Representation, Pennington et al 2014 propose that ratio of co-occurrence probabilities capture semantic information bette…

Gradients for multi class classification with Softmax
type: item
creator: Krishna Sankar
date: 2025-06-22T08:53:17Z
subject: Machine Learning
subject: AI
subject: Categorical Cross Entropy
subject: Linear
subject: Maximum Likelihood
subject: ML
subject: Softmax
description: In a multi class classification problem, the output (also called the label or class) takes a finite set of discrete values . In this post, system model for a multi class classification with a linear layer followed by softmax layer is defined. The softmax function transforms the output of a linear layer into values lying … Continue reading "Gradients for multi class classification with Softmax" The post Gradients for multi class classification with Softmax appeared first on DSP LOG.
encoded: In a multi class classification problem, the output (also called the label or class) takes a finite set of discrete values . In this post, system model for a multi class classification with a linear layer followed by softmax layer is defined. The softmax function transforms the output of a linear layer into values lying between 0 and 1, which can be interpreted as probability scores. Next, the loss function using categorical cross entropy is explained and derive the gradients for model parameters using the chain rule. The analytically computed gradients are then compared with those obtained from the deep learning framework PyTorch. Finally, we implement a training loop using gradient descent for a toy multi-class classification task with 2D Gaussian-distributed data. As always, contents from CS229 Lecture Notes and the notations used in the course Deep Learning Specialization C1W1L01 from Dr Andrew Ng forms key references. Model Let us take an example of estimating based on feature vector having features i.e. and there are examples. <img decoding="async" src="https://dsplog.com/cgi-bin/mimetex.cgi?\begin{tabular}{|c|c|c|c|c|} \hline&{example^1}&{example^2}&\ldots&{example^m}\\ \hline{feature_1}&{x_1}^{1}&{x_1}^{2}&\ldots&{x_1}^{m}\\ \hline{feature_2}&{x_2}^{1}&{x_2}^{2}&\ldots&{x_2}^{m}\\ \hline&\vdots&\vdots&\ldots&\vdots\\ \hline{feature_n}&{x_n}^{1}&{x_n}^{2}&\ldots&{x_n}^{m}&\\ \hline{output}&{y}^{1}&{y}^{2}&\ldots&{y}^{m}\end{tabular} " alt=""> Linear Layer Let us assume that the variable is defined as linear function of . For a single training example, this can be written as : where, is the vector of size i.e. , is the parameter matrix of size i.e. , is the feature vector of size i.e. and is the parameter vector of size i.e. Note : This is the definition of Linear layer in PyTorch (refer entry on Linear layer). This is alternatively called as Dense Layer in Tensorflow (refer entry on Dense) and as Fully Connected layer in deep learning literature. Softmax laye…

Gradients for Binary Classification with Sigmoid
type: item
creator: Krishna Sankar
date: 2025-05-17T13:05:07Z
subject: Machine Learning
subject: AI
subject: Binary Classification
subject: Binary Cross Entropy
subject: Maximum Likelihood
subject: ML
subject: Sigmoid
description: In a classification problem, the output (also called the label or class) takes a small number of discrete values rather than continuous values. For a simple binary classification problem, where output takes only two discrete values : 0 or 1, the sigmoid function can be used to transform the output of a linear regression model … Continue reading "Gradients for Binary Classification with Sigmoid" The post Gradients for Binary Classification with Sigmoid appeared first on DSP LOG.
encoded: In a classification problem, the output (also called the label or class) takes a small number of discrete values rather than continuous values. For a simple binary classification problem, where output takes only two discrete values : 0 or 1, the sigmoid function can be used to transform the output of a linear regression model into a value between 0 and 1, squashing the continuous prediction into a probability-like score. This score can then be interpreted as the likelihood of the output being class 1, with a threshold (commonly 0.5) used to decide between class 0 and class 1. In this post, the intuition for loss function for binary classification based on Maximum Likelihood Estimate (MLE) is explained. We then derive the gradients for model parameters using the chain rule. Gradients computed analytically are compared against gradients computed using deep learning framework PyTorch. Further, training loop using gradient descent for a binary classification problem having two dimensional Gaussian distributed data is implemented. As always, contents from CS229 Lecture Notes and the notations used in the course Deep Learning Specialization C1W1L01 from Dr Andrew Ng forms key references. Model Let us take an example of estimating based on feature vector having features i.e. . There are examples. <img decoding="async" src="https://dsplog.com/cgi-bin/mimetex.cgi?\begin{tabular}{|c|c|c|c|c|} \hline&{example^1}&{example^2}&\ldots&{example^m}\\ \hline{feature_1}&{x_1}^{1}&{x_1}^{2}&\ldots&{x_1}^{m}\\ \hline{feature_2}&{x_2}^{1}&{x_2}^{2}&\ldots&{x_2}^{m}\\ \hline&\vdots&\vdots&\ldots&\vdots\\ \hline{feature_n}&{x_n}^{1}&{x_n}^{2}&\ldots&{x_n}^{m}&\\ \hline{output}&{y}^{1}&{y}^{2}&\ldots&{y}^{m}\end{tabular} " alt=""> Let us assume that the variable is defined as linear function of . Then gets transformed into a probability score using sigmoid function. For a single training example, this can be written as : where, is the feature vector of size i.e. and is a scalar To convert t…

Gradients for linear regression
type: item
creator: Krishna Sankar
date: 2025-05-01T06:02:13Z
subject: Machine Learning
subject: AI
subject: Gradient
subject: MAE
subject: ML
subject: MSE
subject: PyTorch
description: Understanding gradients is essential in machine learning, as they indicate the direction and rate of change in the loss function with respect to model parameters. This post covers the gradients for the vanilla Linear Regression case taking two loss functions Mean Square Error (MSE) and Mean Absolute Error (MAE) as examples. The gradients computed analytically … Continue reading "Gradients for linear regression" The post Gradients for linear regression appeared first on DSP LOG.
encoded: Understanding gradients is essential in machine learning, as they indicate the direction and rate of change in the loss function with respect to model parameters. This post covers the gradients for the vanilla Linear Regression case taking two loss functions Mean Square Error (MSE) and Mean Absolute Error (MAE) as examples. The gradients computed analytically are compared against gradient computed using deep learning framework PyTorch. Further, using the gradients, training loop using gradient descent is implemented for the simplest example of fitting a straight line. As always, contents from CS229 Lecture Notes and the notations used in the course Deep Learning Specialization C1W1L01 from Dr Andrew Ng forms key references. Model Let us take an example of estimating based on feature vector having features i.e. . There are examples. <img decoding="async" src="https://dsplog.com/cgi-bin/mimetex.cgi?\begin{tabular}{|c|c|c|c|c|} \hline&{example^1}&{example^2}&\ldots&{example^m}\\ \hline{feature_1}&{x_1}^{1}&{x_1}^{2}&\ldots&{x_1}^{m}\\ \hline{feature_2}&{x_2}^{1}&{x_2}^{2}&\ldots&{x_2}^{m}\\ \hline&\vdots&\vdots&\ldots&\vdots\\ \hline{feature_n}&{x_n}^{1}&{x_n}^{2}&\ldots&{x_n}^{m}&\\ \hline{output}&{y}^{1}&{y}^{2}&\ldots&{y}^{m}\end{tabular} " alt=""> Assume that the estimate is as linear function of . For a single training example, this can be written as : where, is the feature vector of size i.e. and is a scalar Least Mean Squares To find the parameters and , based on training examples, need to formalise a metric to quantify the “closeness” of the estimate to the true value . As an arbitrary choice, let us define a metric based on the mean square error (MSE) as, Goal is to find the parameters and which minimizes the metric . This can be considered as ordinary least squares (wiki entry on ordinary least squares) model. To find the value of parameters and which minimises the metric , let us try gradient descent method where we i) start with initial random values of par…

Migrated to Amazon EC2 instance (from shared hosting)
type: item
creator: Krishna Sankar
date: 2013-03-11T01:20:38Z
subject: News
subject: amazon
subject: ec2
description: Being not too happy with the speed of the shared hosting, decided to move the blog to an Amazon Elastic Compute Cloud (Amazon EC2) instance. Given this is a baby step, picked up a micro instance running an Ubuntu server and installed Apache web server, MySQL, PHP . After doing a bit of tweaking with this new … Continue reading "Migrated to Amazon EC2 instance (from shared hosting)" The post Migrated to Amazon EC2 instance (from shared hosting) appeared first on DSP LOG.
encoded: Being not too happy with the speed of the shared hosting, decided to move the blog to an Amazon Elastic Compute Cloud (Amazon EC2) instance. Given this is a baby step, picked up a micro instance running an Ubuntu server and installed Apache web server, MySQL, PHP . After doing a bit of tweaking with this new instance, imported the SQL database and other files from the shared hosting and pointed the A name record to the new IP address. This switch happened over this weekend. One particular issue which I faced was frequent crashing of MySQL due to memory limitations. Followed few online instructions to improve the situation and the current configuration seems to be holding up (but this is a cause of worry – need to figure the right solution). Anyhow, hope you like the decreased page load time! Some helpful links from the web: a) How to install WordPress on Amazone EC2 b) Move WordPress site from shared hosting to Amazon EC2 c) DIY: Enable CGI on your Apache server d) Import MySQL Dumpfile, SQL Datafile Into My Database e) Making WordPress Stable on EC2-Micro f) how to enable mod_rewrite in apache2.2 (debian/ubuntu) The post Migrated to Amazon EC2 instance (from shared hosting) appeared first on DSP LOG.

GATE-2012 ECE Q28 (electromagnetics)
type: item
creator: Krishna Sankar
date: 2013-02-20T01:50:01Z
subject: GATE
subject: 2012
subject: ECE
subject: electromagnetics
description: Question 28 on electromagnetics from GATE (Graduate Aptitude Test in Engineering) 2012 Electronics and Communication Engineering paper. Q28. A transmission line with a characteristic impedance of 100 is used to match a 50 section to a 200 section. If the matching is to be done both at 429MHz and 1GHz, the length of the transmission line can be approximately (A) 82.5cm … Continue reading "GATE-2012 ECE Q28 (electromagnetics)" The post GATE-2012 ECE Q28 (electromagnetics) appeared first on DSP LOG.
encoded: Question 28 on electromagnetics from GATE (Graduate Aptitude Test in Engineering) 2012 Electronics and Communication Engineering paper. Q28. A transmission line with a characteristic impedance of 100 is used to match a 50 section to a 200 section. If the matching is to be done both at 429MHz and 1GHz, the length of the transmission line can be approximately (A) 82.5cm (B) 1.05m (C) 1.58m (D) 1.75m Solution To answer this question, let us first understand the propagation in a transmission line, termination and the concept of impedance matching. The section 2.1 in Microwave Engineering, David M Pozar (buy from Amazon.com , Buy from Flipkart.com) is used as reference. Consider a transmission line of very small length having the parameters as show in figure below. <img loading="lazy" decoding="async" width="378" height="150" src="https://dsplog.com/db-install/wp-content/uploads/2013/02/transmission_line_model.png" alt="transmission_line_model" class="wp-image-1943" srcset="https://dsplog.com/db-install/wp-content/uploads/2013/02/transmission_line_model.png 378w, https://dsplog.com/db-install/wp-content/uploads/2013/02/transmission_line_model-300x119.png 300w, https://dsplog.com/db-install/wp-content/uploads/2013/02/transmission_line_model-375x150.png 375w" sizes="auto, (max-width: 378px) 85vw, 378px" /> Figure : Transmission line model (Reference Figure 2.1 in Microwave Engineering, David M Pozar (buy from Amazon.com , Buy from Flipkart.com) is the resistance per unit length , is the inductance per unit length , is the conductance per unit length , is the capacitance per unit length . Applying Kirchoff’s voltage law, Applying Kirchoff’s current law, Dividing the above equations by and taking the limit , If we assume that the inputs are sinusoidal, then the above equation can be re-written as Substituting, , , where . The solution to the above equations are, . The current on the line can be alternately expressed as, , where the characteristic impedance of the line is def…