Remove duplicates from a list in Python
This post will discuss how to remove duplicates from a list in Python while preserving the original order of elements.
If ordering information is not required, we can simply use a set that doesn’t allow duplicates. This removes duplicates, but the original order of elements is not maintained as sets are the unordered collection.
|
1 2 3 4 |
ints = [4, 3, 5, 4, 2, 1, 3] ints[:] = list(set(ints)) print(ints) # prints [1, 2, 3, 4, 5] |
As evident from the above example, the original order of items is lost. This post provides an overview of some of the available alternatives to preserve the original order of items.
1. Using OrderedDict object
To keep the ordering information, we can use OrderedDict from Python collections module. The OrderedDict is a subclass of dict that remembers the insertion order of its entries.
|
1 2 3 4 5 6 7 |
from collections import OrderedDict ints = [4, 3, 5, 4, 2, 1, 3] ints[:] = list(OrderedDict.fromkeys(ints)) print(ints) # prints [4, 3, 5, 2, 1] |
Alternatively, we can take advantage of the fact that the original insertion position remains unchanged when a new entry overwrites an existing entry of an OrderedDict. Here is an example:
|
1 2 3 4 5 6 |
from collections import OrderedDict ints = [4, 3, 5, 4, 2, 1, 3] ints[:] = list(OrderedDict((x, None) for x in ints).keys()) print(ints) # prints [4, 3, 5, 2, 1] |
2. Using a Dictionary
Starting from Python 3.7, we can use a regular dictionary to remove duplicates from a list, as they are guaranteed to be ordered. The idea is to use the items of the original list as keys of the dictionary, and assign any value to them (such as True or None). Then we can get the keys of the dictionary as a list using the list() function and the keys() function. Here is an example of how to implement this method:
|
1 2 3 4 5 6 7 8 |
ints = [4, 3, 5, 4, 2, 1, 3] dict = {} for item in ints: dict[item] = True result = list(dict.keys()) print(result) # prints [4, 3, 5, 2, 1] |
3. Using more_itertools Library
Python’s more-itertools library provides elegant routines for working with Python iterables. It has a unique_everseen function that yields unique elements while preserving order. Following is a simple example demonstrating usage of this function:
|
1 2 3 4 5 6 7 |
from more_itertools import unique_everseen ints = [4, 3, 5, 4, 2, 1, 3] ints[:] = list(unique_everseen(ints)) print(ints) # prints [4, 3, 5, 2, 1] |
4. Using List Comprehension
Another approach is to construct a “new list” from the original list using list comprehensions. The idea is to iterate over the original list, and append each item to the new list only if it is not already in the new list. Here is an example of how to implement this method:
|
1 2 3 4 5 6 |
ints = [4, 3, 5, 4, 2, 1, 3] l = [] [l.append(x) for x in ints if x not in l] print(l) # prints [4, 3, 5, 2, 1] |
Note this approach is relatively slower than the above two approaches. We can also use a dictionary as a helper data structure.
|
1 2 3 4 5 |
ints = [4, 3, 5, 4, 2, 1, 3] result = [item for item in dict.fromkeys(ints)] print(result) # prints [4, 3, 5, 2, 1] |
That’s all about removing duplicates from a list in Python.
Thanks for reading.
To share your code in the comments, please use our online compiler that supports C, C++, Java, Python, JavaScript, C#, PHP, and many more popular programming languages.
Like us? Refer us to your friends and support our growth. Happy coding :)