fields

It's a pretty popular search command and it is used in all sorts of situations. Below are some really cool searches that use fields along with other search commands.

geo-location w/ user home base lookup

index=geod
# get some location information
| iplocation clientip
# lookup user details from a lookup table
#  including their home location
| lookup user_home_lu user as user
# calculate the distance between the login location
#  and the user's home location
#  using the haversine app (http://apps.splunk.com/app/936/)
| haversine originField=home_latlon units=mi inputFieldLat=lat inputFieldLon=lon
# limit the list to those where the distance is greater
#  than 500 miles
| where distance > 500
# clean up for reporting purposes
| strcat City ", " Region cs
# report the results
| fields user, cs, distance

purpose:

find users that are logging in from a location which is greater than 500 miles away from the registered home office

requirements:

haversine app clientip lookup table with user > home_latlon

comments:

song puzzle answer

index=music-puzzle sourcetype=test3 | rename song.parts{}.id as a__pid, song.parts{}.part as a__ppt, song.parts{}.seq as a__pseq | eval tuples = mvzip(mvzip(a__pid, a__ppt, "~~"),a__pseq, "~~") | fields - a__* | mvexpand tuples | rex field=tuples "(?<s_p_id>[^~]+)~~(?<s_p_text>[^~]+)~~(?<s_p_seq>[^~]+)" | sort song.name, s_p_seq | eval s_p_text = urldecode(s_p_text) | stats list(s_p_text) by song.name

purpose:

requirements:

comments:

json mv extraction

...
# clean up some field names for ease of typing later
| rename events{}.code AS a_c, events{}.message AS a_m, events{}.timestamp AS a_ts, events{}.priority as a_p
# combine mv fields together using mvzip (to get tuples as comma-delim'd strings)
| eval b_combined = mvzip(mvzip(mvzip(a_c, a_m), a_ts), a_p)
# get rid of the a_* fields, simply b/c we don't need them clogging up the ui
| fields - a_*
# expand out the combined fields
| mvexpand b_combined
# extract nicely named fields from the results (using the comma from mvzip as the delimiter)
| rex field=b_combined "(?<e_c>[^,]*),(?<e_m>[^,]*),(?<e_ts>[^,]*),(?<e_p>[^,]*)"
# get rid of the combined field b/c we don't need it
| fields - b_*
# urldecode the field that you care about
| eval e_m = urldecode(e_m)

purpose:

requirements:

some json data with pretty specific structure

comments:

Combine dbinspect and REST api data for buckets

| dbinspect index=*
| foreach * [eval dbinspect_<<FIELD>> = '<<FIELD>>']
| table dbinspect_*
| append [
  | rest splunk_server_group=dmc_group_cluster_master "/services/cluster/master/buckets"
  | foreach * [eval rest_api_<<FIELD>> = '<<FIELD>>']
  | table rest_api_* 
  ]
| eval bucketId=if(isNull(rest_api_title),dbinspect_bucketId,rest_api_title)
| stats values(*) as * by bucketId
| foreach rest_api_peers.*.* [eval rest_api_<<MATCHSEG2>>=""]
| foreach rest_api_peers.*.* [eval rest_api_<<MATCHSEG2>>=if("<<MATCHSEG1>>"=dbinspect_bucketId,'<<FIELD>>','<<MATCHSEG2>>')]
| fields - rest_api_peers.*

purpose:

requirements:

Needs to be executed on a search head that can query the cluster master REST API

comments:

The dbinspect API doesn't return consistent information about the size of buckets.

Indexes my user can search.

| rest /services/data/indexes
| search
    [
    | rest /services/data/indexes
    | dedup title
    | table title
    | search
        [
            | rest splunk_server=local /services/authorization/roles
            | search
                [
                        | rest splunk_server=local /services/authentication/users
                        | search
                            [
                                        | rest /services/authentication/current-context
                                        | search type=splunk
                                        | table username
                                        | rename username as title ]
| fields roles
| mvexpand roles
| rename roles as title] imported_srchIndexesAllowed=*
| table imported_srchIndexesAllowed
| rename imported_srchIndexesAllowed as title
| mvexpand title] ]
| stats values(splunk_server) as splunkserver by title
| eval splunkserver=mvjoin(splunkserver,":")
| lookup non_internal_indexes title as title OUTPUT description as description
| fields title, description, splunkserver
| rename title AS Index

purpose:

requirements:

lookup csv with columns "title" of Index and "description"

comments:

We use in a ~200 Users env to show the user in which Index he is allowed to search.

Pearson Correlation Coefficient

index=*
| fields x y
| eval n=1 | eval xx=x*x | eval xy=x*y | eval yy=y*y
| addcoltotals  | tail 1
| eval rho_xy=(xy/n-x/n*y/n)/(sqrt(xx/n-(x/n)*(x/n))*sqrt(yy/n-(y/n)*(y/n)))
| fields rho_xy

purpose:

requirements:

comments:

This SPL query calculates the Pearson coefficient of two fields named x and y.

User Search Restrictions and Last Logins

| rest /services/authentication/users splunk_server=local
| table title realname email roles 
| mvexpand roles 
| join roles type=outer 
    [| rest /services/authorization/roles splunk_server=local
    | fields imported* title srch* 
    | fields - *Quota *TimeWin *capabilities 
    | rex mode=sed field=srchFilter "s/^\*$/true/" 
    | rex mode=sed field=imported_srchFilter "s/^\*$/true/" 
    | eval search_restrictions=if(imported_srchFilter="","( ".srchFilter." )",if(srchFilter="","( ".imported_srchFilter." )","( ".srchFilter." ) OR ( ".imported_srchFilter." ) ")) 
    | fields - srchIndexesDefault 
    | eval srchIndexesAllowed_new=mvjoin(srchIndexesAllowed," OR index=") 
    | eval index_restrictions="( index=".srchIndexesAllowed_new." )" 
    | table title search_restrictions index_restrictions 
    | rename title AS roles ] 
| stats values(roles) AS roles values(search_restrictions) AS search_restrictions values(index_restrictions) AS index_restrictions by title,realname,email 
| rex mode=sed field=search_restrictions "s/\( \)//g" 
| eval search_restrictions=mvjoin(search_restrictions," OR ") 
| eval index_restrictions=mvjoin(index_restrictions," OR ") 
| eval realname=upper(realname)
| rename title AS username
| eval restrictions=if(search_restrictions="( )",index_restrictions,search_restrictions." ".index_restrictions)
| table username email username roles restrictions
| search username=* 
| rex mode=sed field=restrictions "s/^ OR //g" 
| join type=outer username 
    [ search (index=_audit "login attempt info=succeeded) OR (index=_internal "GET /splunk/en-US/account/login") 
    | stats max(_time) AS "last_login" by user 
    | rename user AS username ] 
| eval days_ago=floor((now()-last_login)/86400) 
| eval last_login=strftime(last_login,"%Y-%m-%d %H:%M:%S") 
| fillnull last_login value="Never"

purpose:

requirements:

comments:

Shows a table showing the inherited search and index restrictions for each user, represented as an SPL string. In certain cases it may not be 100% accurate, but it covers most instances that I have run into. NOTE: does not account for `searchFilterSelecting = false` defined in authorize.conf

Show links between tags, sourcetypes, apps and datamodels.

| rest splunk_server=local count=0 /servicesNS/-/-/admin/datamodel-files 
| spath input=eai:data output=base_search path=objects{}.baseSearch 
| spath input=eai:data output=constraints path=objects{}.constraints{}.search 
| eval tag_content = mvappend(base_search,constraints) 
| rex max_match=0 field=tag_content "tag=\"?(?<tag_name>\w+)\"?" 
| mvexpand tag_name 
| rename title AS datamodel 
| append 
    [| rest splunk_server=local count=0 /servicesNS/-/-/admin/eventtypes 
    | rename eai:acl.app AS app tags AS tag_name 
    | search app="*TA*" 
    | rex max_match=0 field=search "sourcetype=\"?(?<sourcetype>[^\s\"^)]+)\"?" 
    | mvexpand sourcetype 
    | mvexpand tag_name 
    | eval app_sourcetype=mvzip(app,sourcetype,"__") 
    | stats list(tag_name) as tag_name by app, sourcetype,app_sourcetype ] 
| stats list(datamodel) as datamodel, list(app) as app, list(app_sourcetype) as app_sourcetype by tag_name 
| search datamodel=* 
| stats values(datamodel) as datamodel, values(tag_name) as tags by app_sourcetype 
| eval tags=mvdedup(tags) 
| rex max_match=0 field=app_sourcetype "\"?(?<app>.+)__\"?" 
| rex max_match=0 field=app_sourcetype "__\"?(?<sourcetype>.+)\"?" 
| fields - app_sourcetype

purpose:

This search answers the questions which dashboards will my new add-on be used for in Enterprise Security.

requirements:

comments:

This is useful to see which app populates which datamodel in Enterprise Security or any other environment which datamodels.