Machine Learning : One-hot encoding for inferencing

 One-hot encoding is a technique used in machine learning and data preprocessing to convert categorical data into a binary format.


This allows algorithms to handle categorical data more effectively.  In One-hot encoding, each category is represented by a binary vector where all elements are zero except for the position corresponding to the category, which is set to one. 

The ipynb file for this activity
https://github.com/niranjanmeegammana/ML-course/blob/main/projects/MLC_Onehot_Encoding_for_Inferencing.ipynb


import pandas as pd


# Data with a categorical column

data = {

    'Fruit': ['Apple', 'Orange', 'Banana', 'Apple', 'Banana', 'Orange']

}


# Convert the data into a DataFrame

df = pd.DataFrame(data)

print(df)


# Perform one-hot encoding using pandas' get_dummies() function

onehot_encoded = pd.get_dummies(df['Fruit'])


# Concatenate the one-hot encoded DataFrame with the original DataFrame

df_encoded = pd.concat([df, onehot_encoded], axis=1)


print(df_encoded)


Example with two category columns

import pandas as pd


# Sample data with two categorical columns

data = {

    'Fruit': ['Apple', 'Orange', 'Banana', 'Apple', 'Banana', 'Orange'],

    'Color': ['Red', 'Orange', 'Yellow', 'Red', 'Yellow', 'Orange']

}


# Convert the data into a DataFrame

df = pd.DataFrame(data)


print (df)


# Perform one-hot encoding on both categorical columns

onehot_encoded = pd.get_dummies(df, columns=['Fruit', 'Color'])


print(onehot_encoded)


Inferencing 

When we create a model, we use multiple records with one hot encoding. However, when Inferencing we use one record which may not have all data in category columns for one hot encoding. Therefore, the inferencing features will not match features seen by the model in training. 

To solve this problem we can create an empty record that represents all features used in training and update values from the new record we will be testing.


Create a record with fruit: Apple and    Color:  Red.

Convert the record to a data frame record. 

# Create the record as a dictionary

record = {

    'Fruit': ['Apple'],

    'Color': ['Red']

}


# Convert the record into a DataFrame

df_record = pd.DataFrame(record)

df_record 


# Perform one-hot encoding on the DataFrame

onehot_encoded_record = pd.get_dummies(df_record)

print(onehot_encoded_record)


Create an empty df record from onehot_encoded, and fill data with zero.

# Create an empty DataFrame with the same columns as onehot_encoded

empty_df_record = pd.DataFrame(columns=onehot_encoded.columns)

# Fill the empty DataFrame with zeros

empty_df_record = empty_df_record.append(pd.Series([0] * len(empty_df_record.columns), index=empty_df_record.columns), ignore_index=True)

print(empty_df_record)

Update empty_df_record with onehot_encoded_record values and save it as new_onehot_record

#Create a copy of empty_df_record 

import copy


# Create a deep copy

new_onehot_record = copy.deepcopy(empty_df_record)

# Update new_onehot_record with values from onehot_encoded_record


new_onehot_record.update(onehot_encoded_record)

new_onehot_record

This record can be used for inferencing with the model.

print(new_onehot_record)

Niranjan Meegammana 

Cyber security and  ML researcher

Sri Lanka Institute of Information Technology 

Shilpa Sayura Foundation 



Comments

Post a Comment

Popular posts from this blog

AI-Typical Phrases to Avoid in Academic Writing

Human Academic Replacements of AI Phrases 1. Filler Opening Phrases (Delete or Rewrite) Avoid (AI-like) Human Academic Alternative In today’s fast-paced world (Delete entirely) It is important to note that (Delete) It should be noted that (Delete) It is worth mentioning that (Delete) There is no doubt that Evidence shows that It can be seen that This shows that One can observe that This indicates that This paper aims to This paper examines 2. Overused AI Transition Phrases Avoid Replace With Furthermore Also Moreover In addition That being said However On the other hand In contrast At the same time Meanwhile In conclusion In summary Overall speaking Overall From a broader perspective More generally 3. AI-Style Explanatory Phrases Avoid Replace With To put it simply Simply At its core Essentially This means that As a result In other words That is This suggests that This indicates that This demonstrates that This shows that 4. Formulaic Emphasis Patterns Avoid Pattern Human Rewrite Not o...
Read more

Tips for Humanizing AI Text

Image
To reduce the AI likelihood in your document and make it more human-like, follow these strategies: 1. Personalize the Writing Style Add personal insights, opinions, or unique perspectives rather than just factual descriptions. Use more conversational or varied sentence structures to break AI-like patterns. 2. Introduce Variability in Sentence Length AI-generated text often has a consistent structure . Try using shorter and longer sentences alternately to mimic natural human writing. Use rhetorical questions or emphasize key points with italicized or bold text (where applicable). 3. Avoid Overuse of Structured Formatting AI-generated documents tend to overuse structured elements (e.g., numbered lists, bullet points, tables). Reduce the number of tables where possible and explain key points in flowing paragraphs instead. 4. Add Real-World Examples and Case Studies Instead of generic descriptions, incorporate real-world applications, anecdotes, or case studies to...
Read more

Overfilling and Underfitting in Machine Learning

Image
Overfitting occurs when a machine learning model is too complex and captures noise or random fluctuations in the training data, making it perform well on the training set but poorly on unseen data. Regularization techniques, more data, or simpler models can help mitigate overfitting. The ipynb file for this tutorial is here . https://github.com/niranjanmeegammana/ML-master/blob/main/04_TEC/MLC_overfitting_and_Underfitting_000.ipynb import numpy as np from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeRegressor import matplotlib.pyplot as plt # Generate some synthetic data np.random.seed(42) X = np.random.rand(100, 1) y = 4 * (X - 0.5) ** 2 + 0.1 * np.random.randn(100, 1) # Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Train the decision tree with different depths depths = [1, 3, 5, 10] train_scores = [] test_scores = [] for depth in depths:     mo...
Read more