Remove duplicate values from a list in Python

This post will discuss how to remove duplicate values from a list in Python. The solution should find and delete the values that appear more than once in the list.

1. Using a Set

A simple solution is to insert all elements from the list into a set that would eliminate duplicates. A set is an object that stores a collection of unique and unordered values. The following code demonstrates this:

nums = [1, 5, 2, 1, 4, 5]

nums[:] = list(set(nums))

print(nums) # [1, 2, 4, 5]

Download Run Code

This method is very efficient to remove any duplicate values from the original list. However, it fails to preserve the original order of elements in the list.

2. Using List Comprehension

If you need to maintain the insertion order of elements in the list, you can use list comprehension with the index() function. List comprehension is a compact way of creating a new list from an existing iterable by applying some expression or condition to each element. Here’s how the code would look like:

nums = [1, 5, 2, 1, 4, 5]

nums[:] = [x for i, x in enumerate(nums) if i == nums.index(x)]

print(nums) # [1, 5, 2, 4]

Download Run Code

This method preserves the order of the original list, but may not be very efficient. To improve performance, you can use set data structure instead of index() function:

nums = [1, 5, 2, 1, 4, 5]

visited = set()

nums[:] = [x for x in nums if x not in visited and not visited.add(x)]

print(nums) # [1, 5, 2, 4]

Download Run Code

3. Using `reduce()` function

You can also use the reduce() function from the functools module to apply a function to each pair of elements in the list. The function can check if the second element is already in the first element, which is a list, and append it if not. For example:

from functools import reduce

nums = [1, 5, 2, 1, 4, 5]

nums[:] = reduce(lambda l, x: l if x in l else l + [x], nums, [])

print(nums) # [1, 5, 2, 4]

Download Run Code

4. Using Dictionary

Another option is to use a dictionary to store the elements of the list as keys. Before Python 3.7, you can use the OrderedDict, which has the capability to remember the insertion order. The idea is to use the fromkeys() function that returns a new dictionary with value defaults to None. To get distinct keys, simply convert it to a list, as shown below:

from collections import OrderedDict

nums = [1, 5, 2, 1, 4, 5]

nums[:] = list(OrderedDict.fromkeys(nums))

print(nums) # [1, 5, 2, 4]

Download Run Code

As of Python 3.7, regular dictionary are guaranteed to be ordered. Therefore, you can use dict instead.

nums = [1, 5, 2, 1, 4, 5]

nums[:] = list(dict.fromkeys(nums))

print(nums) # [1, 5, 2, 4]

Download Run Code

5. Using `numpy` module

If you happen to be using numpy module already, you can use the unique() function. It returns the sorted unique elements and won’t preserve the original order of elements in the list. For example:

import numpy as np

nums = [1, 5, 2, 1, 4, 5]

nums[:] = np.unique(nums)

print(nums) # [1 2 4 5]

Download Code

6. Using `more_itertools` module

Finally, you can use the more_itertools module to import the unique_everseen() function, which returns an iterator over the unique elements of the list, preserving their order. You can use it as follows:

from more_itertools.recipes import unique_everseen

nums = [1, 5, 2, 1, 4, 5]

nums[:] = list(unique_everseen(nums))

print(nums) # [1, 5, 2, 4]

Download Code

That’s all about removing duplicate values from a list in Python.

Remove duplicate values from a list in Python

1. Using a Set

2. Using List Comprehension

3. Using reduce() function

4. Using Dictionary

5. Using numpy module

6. Using more_itertools module

Share this:

3. Using `reduce()` function

5. Using `numpy` module

6. Using `more_itertools` module