Efficiently Eliminating String Repetitions In Pandas

Pandas is a powerful Python library for data analysis and manipulation. One common task when working with data is removing duplicate values from a dataset. This can be done in Pandas using the drop_duplicates() function. This function allows you to specify which columns to consider for identifying duplicates, whether to keep the first or last occurrence of a duplicate, and whether to modify the original DataFrame or create a new one. Additionally, Pandas provides the Series.str.repeat() function to duplicate each string in a Series or Index.

Removing duplicate rows in a Pandas dataframe

The `drop_duplicates() method in Pandas helps remove duplicates from a Pandas dataframe. This method returns a dataframe with duplicate rows removed, considering certain columns. Indexes, including time indexes, are ignored.

Python

DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)

• `subset`: Takes a column or list of column labels. It is optional and the default value is `None`. After passing columns, it will only consider them for duplicates.
• `keep`: Determines how to consider duplicate values. It has three distinct values, and the default is `first`. If `first`, it considers the first value as unique and the rest of the same values as duplicates. If `last`, it considers the last value as unique and the rest of the same values as duplicates. If `False`, it considers all of the same values as duplicates.
• `inplace`: A boolean value that removes rows with duplicates if `True`.

Python

Import pandas as pd

Data = {

"A": ["TeamA", "TeamB", "TeamB", "TeamC", "TeamA"],

"B": [50, 40, 40, 30, 50],

"C": [True, False, False, False, True]

}

Df = pd.DataFrame(data)

Display(df.drop_duplicates())

The output will be:

A       B       C

0  TeamA  50       True

1  TeamB  40       False

3  TeamC  30       False

You can also remove duplicates based on specific columns:

Python

Import pandas as pd

Sample data for student names and ages

Students_data = {

'Name': ["Alice", "Bob", "Charlie", "David", "Alice", "Eva", "Bob"],

'Age': [20, 22, 21, 23, 20, 19, 22],

'Grade': [85, 90, 78, 92, 85, 88, 90],

'Attendance': [95, 92, 88, 93, 95, 96, 92]

}

Create a DataFrame

Students_df = pd.DataFrame(students_data)

Remove duplicates based on specific columns (Name and Age)

Students_df_specific_columns = students_df.drop_duplicates(subset=["Name", "Age"], keep=False)

Print(students_df_specific_columns)

The output will be:

Name  Age  Grade  Attendance

2  Charlie  21  78    88

3  David  23  92    93

5  Eva  19  88    96

Removing duplicates from specific columns

When working with data in Python, it is common to encounter duplicate values, especially when dealing with large datasets. These duplicates can skew your results and lead to inaccurate analyses. Therefore, it is crucial to know how to remove duplicates effectively.

In this section, we will focus on removing duplicates from specific columns in a Pandas DataFrame. We will explore different techniques, including using the `drop_duplicates()` method and applying custom logic.

Using the `drop_duplicates()` method

The `drop_duplicates()` method in Pandas is a powerful tool for removing duplicates from a DataFrame. This method allows you to specify one or more columns to use as the basis for dropping duplicates. Here's how you can use it:

Python

Import pandas as pd

Sample data

Data = {

"Column1": ["cat", "toy", "cat"],

"Column2": ["bat", "flower", "bat"],

"Column3": ["xyz", "abc", "lmn"]

}

Df = pd.DataFrame(data)

Drop duplicates based on specific columns

Df_no_duplicates = df.drop_duplicates(subset=["Column1", "Column2"])

Print(df_no_duplicates)

In this example, we first create a DataFrame `df` with three columns: "Column1", "Column2", and "Column3". We then use the `drop_duplicates()` method to remove duplicates based on specific columns, in this case, "Column1" and "Column2". The resulting DataFrame `df_no_duplicates` will have the duplicates removed, keeping only the first occurrence of each duplicate set.

You can also specify whether you want to keep the first or last occurrence of duplicates using the `keep` parameter. For instance:

Python

Keep the first occurrence of duplicates

Df_first = df.drop_duplicates(subset=["Column1"], keep="first")

Keep the last occurrence of duplicates

Df_last = df.drop_duplicates(subset=["Column1"], keep="last")

Removing duplicates based on multiple conditions

In some cases, you may need to remove duplicates based on multiple conditions or criteria. For example, you might want to drop duplicates where the values in "Column1" are the same, but only if the corresponding values in "Column2" are also equal. Here's how you can achieve this:

Python

Drop duplicates based on multiple conditions

Df_filtered = df[df["Column1"].duplicated() & df["Column2"].duplicated()]

Print(df_filtered)

In this code snippet, we use the `duplicated()` method to create a Boolean mask that identifies duplicate values in "Column1" and "Column2". We then use this mask to filter the original DataFrame and create a new DataFrame `df_filtered` that contains only the rows with duplicate values in both columns.

Custom logic for removing duplicates

In certain situations, you may need to apply custom logic to remove duplicates. For example, you might want to keep the row with the highest value in a specific column among the duplicates. Here's an example:

Python

Sort the DataFrame by Column2 in descending order

Df = df.sort_values(by="Column2", ascending=False)

Drop duplicates based on Column1 and keep the first occurrence

Df_custom = df.drop_duplicates(subset=["Column1"], keep="first")

Print(df_custom)

In this code, we first sort the DataFrame by "Column2" in descending order. Then, we use the `drop_duplicates()` method to remove duplicates based on "Column1", keeping only the first occurrence. This ensures that we retain the row with the highest value in "Column2" among the duplicates.

Keeping the first/last occurrence of duplicates

When using the `drop_duplicates()` method in pandas, the `keep` parameter controls which occurrence of duplicates is kept. The `keep` parameter can be set to:

• 'first': Drops duplicates except for the first occurrence. This is the default value.
• 'last': Drops duplicates except for the last occurrence.
• False: Drops all duplicates.

Python

Import pandas as pd

Df = pd.DataFrame({

'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],

'style': ['cup', 'cup', 'cup', 'pack', 'pack'],

'rating': [4, 4, 3.5, 15, 5]

})

To keep the first occurrence of duplicates, you can use the `keep='first'` parameter:

Python

Df.drop_duplicates(keep='first')

This will return the following DataFrame:

Brand style rating

0 Yum Yum cup 4.0

2 Indomie cup 3.5

3 Indomie pack 15.0

4 Indomie pack 5.0

To keep the last occurrence of duplicates, you can use the `keep='last'` parameter:

Python

Df.drop_duplicates(keep='last')

This will return the following DataFrame:

Brand style rating

1 Yum Yum cup 4.0

2 Indomie cup 3.5

4 Indomie pack 5.0

You can also specify the columns to consider for identifying duplicates using the `subset` parameter. For example, to keep the last occurrence of duplicates based on the 'brand' and 'style' columns:

Python

Df.drop_duplicates(subset=['brand', 'style'], keep='last')

This will return the following DataFrame:

Brand style rating

1 Yum Yum cup 4.0

2 Indomie cup 3.5

4 Indomie pack 5.0

The `drop_duplicates()` method can also be used on pandas Series. Here is an example of a Series with duplicates:

Python

S = pd.Series(['llama', 'cow', 'llama', 'beetle', 'llama', 'hippo'], name='animal')

To keep the first occurrence of duplicates, you can use the `keep='first'` parameter:

Python

S.drop_duplicates(keep='first')

This will return the following Series:

0 llama

1 cow

3 beetle

5 hippo

Name: animal, dtype: object

To keep the last occurrence of duplicates, you can use the `keep='last'` parameter:

Python

S.drop_duplicates(keep='last')

This will return the following Series:

1 cow

3 beetle

4 llama

5 hippo

Name: animal, dtype: object

Removing duplicates using Python's inbuilt functions

There are several ways to remove duplicate values from a list in Python. Here are some common approaches:

Using a Dictionary

One approach is to use a dictionary, which does not allow duplicate keys. By creating a dictionary with the list items as keys and then converting it back to a list, you can easily remove duplicates. Here's an example:

Python

Mylist = ["a", "b", "a", "c", "c"]

Mylist = list(dict.fromkeys(mylist))

Print(mylist)

In this code, we first create a list `mylist` that contains duplicate values. Then, we use the `dict.fromkeys()` function to create a dictionary with the list items as keys. Since dictionaries cannot have duplicate keys, any duplicate values in the list will be automatically removed. Finally, we convert the dictionary back into a list, resulting in a new list without any duplicates.

Using `itertools.repeat()`

The `itertools` module in Python provides a repeat() function that can be used to repeat elements in a list. This function takes two parameters: the element to be repeated and the number of times it should be repeated. Here's an example:

Python

From itertools import repeat

Test_list = [4, 5, 6]

Res = list(itertools.chain.from_iterable(itertools.repeat(i, 3) for i in test_list))

Print(res)

In this code, we import the `repeat()` function from the `itertools` module. We then define a list `test_list` with some values. Next, we use a list comprehension to repeat each element in `test_list` three times (`3` being the magnitude of repetition) and store the result in the `res` list. Finally, we print the `res` list, which contains the original list elements repeated the specified number of times.

Using `numpy.repeat()`

The `numpy` module in Python also provides a repeat() function that can be used to repeat elements in an array. This function takes three parameters: the array, the number of repetitions, and the axis along which the elements are repeated. Here's an example:

Python

Import numpy as np

Arr = np.array([4, 5, 6])

Res = np.repeat(arr, 3)

Print(res)

In this code, we import the `numpy` module and define an array `arr` with some values. We then use the `repeat()` function to repeat each element in the array three times (`3` being the number of repetitions) and store the result in the `res` array. Finally, we print the `res` array, which contains the original array elements repeated the specified number of times.

Using `pandas.Series.str.repeat()`

The `pandas` module in Python provides a Series.str.repeat() method that can be used to repeat string values in a Series. This method takes an integer or a list of integers as a parameter to define the number of times each string should be repeated. Here's an example:

Python

Import pandas as pd

Data = pd.read_csv("nba.csv")

Data.dropna(how='all', inplace=True)

Data ["Team"] = data ["Team"].str.repeat(2)

Print(data)

In this code, we import the `pandas` module and read data from a CSV file into a DataFrame called `data`. We then use the `dropna()` method to remove any rows with null values to avoid errors. Next, we use the `str.repeat()` method on the "Team" column to repeat each string value twice. Finally, we print the updated DataFrame, which now contains the repeated string values in the "Team" column.

Removing duplicates from a Pandas dataframe using Regex

In this article, we will see how to remove continuously repeating characters from the words of the given column of the given Pandas Dataframe using Regex.

Here, we are actually looking for continuously occurring repetitively coming characters for which we have created a pattern that contains this regular expression (\w)\1+. Here, \w is for a character, and 1+ is for the characters that come more than once.

We are passing our pattern in the re.sub() function of the re library.

Syntax: re.sub(pattern, repl, string, count=0, flags=0)

The ‘sub’ in the function stands for SubString, a certain regular expression pattern is searched in the given string (3rd parameter), and upon finding the substring pattern, it is replaced by repl (2nd parameter). The count checks and maintains the number of times this occurs.

Now, let’s create a Dataframe:

Importing required libraries

Creating Dataframe with column

As name and common_comments

'name' : ['Akash', 'Ayush', 'Diksha',

'Priyanka', 'Radhika'],

'common_comments' : ['hey buddy meet me today ',

'sorry bro I can't meet',

'hey akash I love geeksforgeeks',

'Twitter is the best way to comment',

'Geeksforgeeks is good for learners']

Columns = ['name', 'common_comments']

Printing Dataframe

Now, remove continuously repetitive characters from words of the Dataframe common_comments column.

Define a function to remove

Continuously repeating characters

Def conti_rep_char(str1):

Tchr = str1.group(0)

Define a function to check

Whether a character is unique

Def check_unique_char(rep, sent_text):

# regular expression for

# repetition of characters

Convert = re.sub(r'(\w)\1+',

# returning the converted word

Df['modified_common_comments'] = df['common_comments'].apply(

Lambda x: check_unique_char(conti_rep_char,

Time Complexity: O(n) where n is the number of elements in the dataframe.

Space complexity: O(m * n) where m is the number of columns and n is the number of elements in the dataframe

Another approach

We can also use the Series.str.replace() method to replace each occurrence of a pattern/regex in the Series/Index.

Equivalent to str.replace() or re.sub(), depending on the regex value.

Pat: str or compiled regex

String can be a character sequence or regular expression.

Repl: replacement string or a callable. The callable is passed the regex match object and must return a replacement string to be used. See re.sub.

N: int, default -1 (all)

Number of replacements to make from the start.

Case: bool, default None

Determines if the replacement is case-sensitive:

If True, case-sensitive (the default if pat is a string)

Set to False for case-insensitive

Cannot be set if pat is a compiled regex.

Flags: int, default 0 (no flags)

Regex module flags, e.g. re.IGNORECASE. Cannot be set if pat is a compiled regex.

Regex: bool, default False

Determines if the passed-in pattern is a regular expression:

If True, assumes the passed-in pattern is a regular expression.

If False, treats the pattern as a literal string

Cannot be set to False if pat is a compiled regex or repl is a callable.

Series or Index of object

A copy of the object with all matching occurrences of pat replaced by repl.

Example

Import numpy as np

Import pandas as pd

Going to use this for boolean indexing

Erase = np.tile(np.array(False), 12)

Iterate over each unique word

For word in np.unique(df ['Word]):

Found = df ['Word'] == word

# check if there is more than one occurrence

If np.count_nonzero(found == True) > 1:

# get indexes

Indexs = np.where(found.values == True) [0]

FirstIndex = indexs [0]

LastIndex = indexs [len(indexs) - 1]

# update values to erase

Erase [firstIndex + 1:lastIndex + 1] = True

# update main dataframe

Df [erase] = ''

Frequently asked questions