Golang remove duplicates from slice: dedupe and unique values

Golang remove duplicates from slice: preserve order with a map, sort then Compact for sorted unique, counts with map, and the XOR trick for one odd-count value.

Published

Updated

Read time 3 min read

Reviewed byDeepak Prasad

Golang remove duplicates from slice: dedupe and unique values

This page targets golang remove duplicates from slice, golang dedupe slice, golang unique slice, and remove duplicates from slice golang. It is not the same topic as golang remove from slice or delete element from slice golang at a known index—use golang remove from slice when you delete by position or filter while iterating without deduping the whole set.

Examples were run with a recent Go toolchain on Linux (stdlib only, including slices).


Preserve order: golang unique slice with a map

For golang get unique values from slice while keeping the first copy of each value, track seen and append once:

go
package main

import "fmt"

func uniquePreserveOrder[T comparable](in []T) []T {
	seen := make(map[T]struct{})
	out := make([]T, 0, len(in))
	for _, v := range in {
		if _, ok := seen[v]; !ok {
			seen[v] = struct{}{}
			out = append(out, v)
		}
	}
	return out
}

func main() {
	fmt.Println(uniquePreserveOrder([]int{8, 7, 5, 0, 0, 1, 2, 2, 8}))
}
Output

Run prints [8 7 5 0 1 2] with duplicates removed in first-seen order.


Sorted unique: sort then slices.Compact

When you only need golang slice remove duplicates and sorted output is acceptable, sort a copy then call slices.Compact:

go
package main

import (
	"fmt"
	"slices"
)

func main() {
	a := []int{3, 1, 3, 2, 1, 2}
	slices.Sort(a)
	a = slices.Compact(a)
	fmt.Println(a)
}
Output

Run prints [1 2 3]; original order is not preserved.


Count occurrences: map for golang remove duplicates diagnostics

go
package main

import "fmt"

func counts[T comparable](in []T) map[T]int {
	m := make(map[T]int)
	for _, v := range in {
		m[v]++
	}
	return m
}

func main() {
	in := []int{23, 8, 7, 5, 0, 0, 1, 2, 2, 8}
	fmt.Println(counts(in))
}
Output

Run prints a map from each value to how many times it appeared; use that to decide which elements to drop or keep beyond simple dedupe.


XOR trick (narrow case only)

If every value except one appears an even number of times, XOR of all integers reveals the odd-one-out:

go
package main

import "fmt"

func xorSingleton(nums []int) int {
	x := 0
	for _, v := range nums {
		x ^= v
	}
	return x
}

func main() {
	nums := []int{8, 7, 5, 0, 0, 1, 2, 2, 8, 3, 4, 3, 4, 1, 6, 6, 7}
	fmt.Println(xorSingleton(nums))
}
Output

Run prints 5. Do not use this for general golang array unique problems; it assumes the XOR property holds.


Summary

Golang remove duplicates from slice usually means either a map-backed pass that preserves first-seen order or a sort plus slices.Compact path when sorted golang unique slice output is fine. Maps also power golang get unique values from slice workflows that need counts before filtering. The XOR section is a small specialty case, not a replacement for map-based dedupe. For deleting a single index or golang remove element from slice while iterating without deduping the collection, use golang remove from slice.


References


Frequently Asked Questions

1. Map versus sort+Compact for golang unique slice?

Use a map when you need the first occurrence order preserved. Use sort plus slices.Compact when sorted output is fine and you want contiguous duplicate removal in O(n log n) time with little extra memory beyond sorting.

2. Does Compact work without sorting first?

Compact only removes consecutive duplicates. Sorting first groups equal values together so Compact can remove all copies except one per run.

3. When does XOR find a unique value?

Only when every duplicated value appears an even number of times and exactly one value appears an odd number of times; XOR-ing all entries yields that singleton. It is not a general dedupe tool.

4. Are slice and array answers different?

In Go, arrays have fixed size; most code uses a slice over the array or converts with [:] to reuse these patterns on array-backed data.
Deepak Prasad

R&D Engineer

Founder of GoLinuxCloud with more than 15 years of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive …