Dedup: Splunk Commands Tutorials & Reference
Commands Category: Filtering
Commands: dedup
Use: Removes the events that contain an identical combination of values for the fields that you specify.
With the dedup command, you can specify the number of duplicate events to keep for each value of a single field, or for each combination of values among several fields. Events returned by dedup are based on search order. For historical searches, the most recent events are searched first. For real-time searches, the first events that are received are search, which are not necessarily the most recent events.
Avoid using the dedup command on the _raw field if you are searching over a large volume of data. If you search the _raw field, the text of every event in memory is retained which impacts your search performance.
Remove duplicates of results with the same 'host' value.
... | dedup hostRemove duplicates of results with the same 'source' value and sort the events by the '_time' field in ascending order.
... | dedup source sortby +_timeRemove duplicates of results with the same 'source' value and sort the events by the '_size' field in descending order.
... | dedup source sortby -_sizeFor events that have the same 'source' value, keep the first 3 that occur and remove all subsequent events.
... | dedup 3 sourceFor events that have the same 'source' AND 'host' values, keep the first 3 that occur and remove all subsequent events.
... | dedup 3 source hostSample Data - Download sample data for lab - ../../tutorial/splunk/labs/fundamental/Splunk_f1_Data.zip
(index=main sourcetype=access_combined_wcookie action=purchase status=200 file=success.do | dedup JSESSIONID | table JSESSIONID, action, status | rename JSESSIONID as UserSessions
index=main sourcetype=access_combined_wcookie action=purchase status=200 file=success.do | dedup JSESSIONID | table JSESSIONID | rename JSESSIONID as UserSessions
 |