Splunk Sourcetype Repair Workflow

This workflow repairs every event in tychon-crypto when its sourcetype is not tychon:crypto:json.

Prerequisites: copy and deletion permissions

The account performing this workflow needs run_collect to execute the copy step and delete_by_keyword to execute the final cleanup. The can_delete role grants delete_by_keyword. Being a Splunk administrator does not necessarily grant either capability.

1. Open Settings → Access controls → Roles.
2. Open the role assigned to the account that will run the deletion.
3. Enable run_collect and delete_by_keyword, then save the role.
4. Sign out and back in before running the deletion search.

Verify the account's assigned roles with:

| rest /services/authentication/users/admin
| table title roles

If the account does not have run_collect, do not run the copy search. If it does not have delete_by_keyword, stop after the copy and validation steps. Do not run the deletion SPL until the capability is confirmed.

Before starting

Confirm a recoverable backup and pause ingestion into tychon-crypto. Keep ingestion paused until the timestamp setting below has been restored. Run this copy only once; repeating it creates duplicates. If an earlier attempt ran, check its copies before proceeding. Stop on errors or partial results. Use complete, unsampled searches and select All time for validation and dashboard searches.

1. Find invalid sourcetypes

index=tychon-crypto earliest=0 latest=now
| where isnull(sourcetype) OR sourcetype!="tychon:crypto:json"
| stats count as Events dc(id) as UniqueIDs by sourcetype
| sort - Events

2. Inspect raw data and missing values

index=tychon-crypto earliest=0 latest=now
| where isnull(sourcetype) OR sourcetype!="tychon:crypto:json"
| table _time _indextime index sourcetype source host id event.dataset tychon.type _raw
| head 100

index=tychon-crypto earliest=0 latest=now
| where isnull(source) OR source=""
| stats count as Events dc(id) as UniqueIDs by sourcetype host
| sort - Events

index=tychon-crypto earliest=0 latest=now
| where isnull(sourcetype) OR sourcetype=""
| table _time index source host id event.dataset tychon.type _raw
| head 100

Before copying: temporarily disable timestamp extraction

In Settings → Source types → tychon:crypto:json → Advanced, record the existing DATETIME_CONFIG value or that it is inherited. Set DATETIME_CONFIG=NONE. Confirm it is active on the parsing tier, including any required reload or restart. In a distributed deployment, changing only the search head may not be sufficient. This prevents dates inside the JSON from replacing the original event time.

Selected events must contain valid JSON and populated host and source values. Stop if they do not.

3. Copy events with the canonical sourcetype

index=tychon-crypto earliest=0 latest=now
| where isnull(sourcetype) OR sourcetype!="tychon:crypto:json"
| fields _raw _time host source sourcetype
| eval sourcetype="tychon:crypto:json"
| collect index=tychon-crypto output_format=hec addinfo=false

collect writes new events from the search results; it does not modify or remove the originals. The copied events use the canonical sourcetype and keep the original source and host values. Both versions exist until the deletion step.

Run the copy once and wait for indexing before validating. The results returned by collect show submitted events, not confirmation that indexing finished. Do not add marker, source, sourcetype, host, or addtime options to HEC output. The older raw-output command with a marker changes the raw JSON.

4. Validate the replacement

index=tychon-crypto earliest=0 latest=now
| eval RawHash=sha256(_raw), EventTime=printf("%.9f", _time)
| eval Original=if(sourcetype="tychon:crypto:json", 0, 1),
    Copy=if(sourcetype="tychon:crypto:json", 1, 0)
| stats sum(Original) as Originals sum(Copy) as Copies
    by RawHash EventTime host source
| where Originals>0
| eval Mismatch=if(Originals=Copies, 0, 1)
| stats sum(Originals) as OriginalEvents sum(Copies) as CopiedEvents
    sum(Mismatch) as MismatchedGroups

Require OriginalEvents > 0, OriginalEvents = CopiedEvents, and MismatchedGroups = 0. This compares raw payloads, event times, hosts, sources, and duplicate counts. If any check fails, keep the originals and investigate; do not rerun the copy blindly. This comparison does not identify unrelated or malformed canonical copies from earlier attempts.

Before deleting, also run the dashboard search in step 6 and check the affected dashboards and filters. For the TQR-1659 QA sample, require 405 inventory document IDs and one RHEL 9 host.

5. Delete the originals

Run only after validation and backup confirmation. Requires delete_by_keyword and operator approval.

index=tychon-crypto earliest=0 latest=now
| where isnull(sourcetype) OR sourcetype!="tychon:crypto:json"
| delete

6. Confirm the dashboard path

`tychon_crypto_base`
| `tychon_routing_fields`
| where tychon_dataset="tychon.pqc_inventory"
| stats count as Events dc(id) as UniqueIDs by sourcetype

7. Restore timestamp extraction and resume

Restore the exact previous DATETIME_CONFIG value; remove the temporary override if it was inherited. Confirm restoration is active before resuming ingestion, even if the migration was abandoned. Check a new scanner event's sourcetype, event time, and dashboard visibility.