I have been doing a few tests on how configurations are pushed when applying a shcluster bundle. However, I would like to find some definitive answers if at all possible.
On the deployer in shcluster/apps I have a Splunk app with
- appname/default/props.conf
- appname/default/transforms.conf
- appname/default/savedsearches.conf
- appname/local/props.conf
- appname/local/transforms.conf
- appname/local/savedsearches.conf
Now it appears when I apply the cluster bundle with
sudo -u splunk /opt/splunk/bin/splunk apply shcluster-bundle -target https://10.10.1.1:8089 -auth admin:changeme
The app gets pushed to the search head cluster members.
However, on the search heads, it appears everything in appname/local has been "merged" with appname/default. This is great and I understand the reasoning behind this because it then means that users can make changes to the apps on the SH cluster and only changes are stored in the appname/local. This means that if the apps are deployed again, they won't overwrite local users changes to the app.
**First question** is. Where is this deployment behavior documented? I would assume matching stanzas in local/props.conf would override the default/props.conf, but is this documented somewhere?
What happens to local really isn't covered here
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/PropagateSHCconfigurationchanges
**Second Question** is if I want to "take a snapshot" of an app from a search head in the cluster to "update" the deployer with the most recent version is it just a matter of copying off the entire app directory?
Removing any folders like appname/default.old.20160304-103301 which appear to be backups from the last deployment. Then copy this across to the deployer as the lastest "version". I can see the documentation says you don't need to but it seems like a good idea to "track" an app as it grows.
**Bonus Knowledge**
I just discovered you have control over how the deployer handles lookups which is great. This is one of the reasons I have been hesitant to deploy at times.
splunk apply shcluster-bundle -target : -preserve-lookups true -auth :
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/HowconfrepoworksinSHC
↧
Search Head Deployer in a SH Cluster: What happens to local?
↧
When adding an indexer to a distributed environment, is there a configuration that makes indexers exchange events to auto load balance them?
Is there a configuration that makes indexers exchange events in order to auto load balance them? Let's say I add an indexer into distributed environment. I want to use it without reconfiguring syslog sources and forwarders.
Maybe it's a request - make indexers connect to each other, and move events between them to distribute in an optimal way...
Does indexer clustering with duplication of data give any advantage? Maybe then the search head is using first/second indexer to retrieve events... Not only "first copy"?
↧
↧
Where to install and configure the Splunk Add-on for Bro IDS in an indexer clustering environment?
So after spending a lot of time googling this issue, finally I get some mixed comments.
Hence thought of asking the question here to get the clarification on the issue.
Starting with the environment, I have an indexer cluster of 3 indexers, two independent search heads, and one Universal forwarder.
My question is where the BRO IDS app goes and how it works?
What I have done is - I have installed the app on both of my search heads (as per general convention while dealing with apps), and my Universal Forwarder is monitoring the Bro log directory (yes I have installed UF on my Bro sensor machine).
I am getting the monitored Bro logs in my indexers and am able to search them via search heads, but the app is just sitting there doing nothing it seems.
The documentation I have read so far says that you need to install app on the heavy forwarder that is monitoring your log dir and have to set the inputs path in the app instead of heavy forwarder's input. (So I think it's stupid for the people who just want to have a forwarder installed on their bro sensor for just forwarding bro logs and for that we need to install heavy forwarder with the app, and that too app will be doing all the forwarding and parsing and heavy forwarder will be just sitting there providing Python support to the app to do its stuff).
So my question is: is my above configuration even workable with Bro IDS add-on or do I have to just chuck the idea of using the add-on because I don't want to run a heavy forwarder on my Bro machines?
Any comments would be greatly appreciated, as I have already wasted a lot of time dealing with this issue.
Thanks,
Fatema.
↧
Why is the Splunk Add-on for Citrix NetScaler not parsing syslog data correctly in my distributed search environment?
Hi,
I have a distributed environment of Splunk running 6.3, I have a search head, cluster master, indexer & heavy forwarder. I have syslog data coming from netscalers on the heavy forwarder where I have the Splunk Add-on for Citrix Netscaler installed and all the data is being indexed correctly. The HF forwards data to my indexer and the data is coming in fine, but it has not been parsed correctly. I initially didn’t have the Splunk Add-on for Citrix Netscaler installed on the indexer so though this was the issue, so I installed it, but there is no change. Does anyone know what’s happening here? I though the HF forwarded the indexed data?
↧
Splunk Distributed Peer error on 6.2.6 a week after extending the certs
Posting a question after an year, so bear with me.
We're on Splunk 6.2.6 and recently 2 weeks back updated the default Splunk certs using the script provided by Splunk. This was done in 3 environments. Four days back, 2 out of 3 search heads were not able to connect to a few indexers. There is no common indexer b/w those two SHs, they are failing for different Indexers. We're not using any SSL.
If I try to add (after deleting the Distributed Peer entry) OR update the authentication in Settings->Distributed Search- > search Peers, it gets time out (read operation timeout error)
I can see several SSL errors logged on the indexers (to which the SH is not able to connect). Similar to this:
06-18-2016 16:55:30.036 -0500 WARN HttpListener - Socket error from X.X.X.X while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
The SH shows errors like this:
06-18-2016 16:54:30.036 -0500 ERROR HttpClientRequest - HTTP client error: Read Timeout (while accessing http://127.0.0.1:8065/dj/en-us/twitter2/setup/)
Raised a ticket with Splunk support and waiting on solution. Just wanted to check in the community if anyone else has faced this issue and/or have solution to it.
Thanks in advanced.
↧
↧
How can I find whether an environment is clustered or distributed? If it is distributed, how can I add a new index to that?
I have 4 servers in which 2 are clustered and are used as search heads, a 3rd one is Splunk Enterprise Security, and the 4th server is search head pooling. These are connected to indexers. I want to know how to find whether the environment is clustered or distributed. If it is distributed, then how should I add new index to it and pull logs into that index?
Thanks,
Nishwanth
↧
Distributed search groups not actually filtering searches
We are using distributed search groups ( http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Distributedsearchgroups ).
We have 2 sets of indexers: index group A and index group b.
We have a config similar to the following.
distsearch.conf
[distributedSearch]
servers = indexa_1:8089,indexa_2:8089,indexb_1:8089,indexb_2:8089
[distributedSearch:groupa]
default = true
servers = indexa_1:8089,indexa_2:8089
[distributedSearch:groupb]
servers = indexb_1:8089,indexb_2:8089
[distributedSearch:all]
servers = indexa_1:8089,indexa_2:8089,indexb_1:8089,indexb_2:8089
I am finding that if I check /opt/splunk/var/log/splunk/remote_searches.log on indexb_1 or indexb_2 I can see certain searches from this search head hitting them when they shouldn't.
These particular searches do not have splunk_server_group=groupb or splunk_server_group=all in the query.
They do all seem to have "presummarize" or "scheduler" in their search. I'm not seeing interactive search sessions though.
Do distributed search groups only stop searches from interactive searches?
This seems like a hole/bug.
↧
Why is one indexer faster at search than the other two - troubleshooting distributed search speed by indexer
I have three indexers. All configured the same all with the same hardware (16 cores 32 GB ram).
I have a simple search for internal data
` index=_internal host=My-License-Manager source=*license_usage.log type="RolloverSummary" earliest=-30d@d`
This search runs in just over 5 seconds on indexer #1 and times out on indexer #2 and #3
If I change the time to `earliest=-35d@d latest=-4d@d` indexer #2 returns in 5 seconds but only #3 times out.
If I change the time to `earliest=-29d@d latest=-4d@d` all three indexers return results in just over 5 seconds.
One day later or one day earlier will cause indexer #2 or #3 to time out.
how do I start to troubleshoot what is causing this. I am sure this can't be isolated to this one data set and has to be affecting other data sets as well.
I opened a Case `Number 387826 Date/Time Opened 8/23/2016 7:31 AM` with splunk support but no response yet
↧
where do you install Python for Scientific Computing (for Linux 64-bit) in a distributed search environment?
Where does the Python for Scientific Computing SA get installed in a distributed environment? Indexers? Search Heads? Both?
↧
↧
What is the difference between Cluster master and License master in a distributed Environment?
What is the difference between Cluster master and License master in a distributed Environment?
Any major differences and detailed explanation of both would be great.
↧
SSO with SAML in distributed environment : Why is data retrieved and seen in inline search, but not when in a dashboard?
Hello,
We are in a distributed configuration. We want to add SSO to Splunk Active Directory Federation Services (ADFS). We have only configured SSO with ADFS on the search head. For the authentication it works fine (with a little bit of works). But we have a strange behavior with dashboard :
- when running the inline search from a panel, no issue : datas are retrieved and displayed correctly
- when running the search from the dashboard : datas are not displayed, we get a no result found
I've tried to add a new panel on an existing dashboard, same issue. On a new dashboard (private or shared), same issue. I think this is related to a role permission, but don't know how to troubleshoot this.
Does anyone had already encountered this behavior, do we need to set up SSO on all node of the Splunk infrastructure (search head and indexer)?
Thanks for your help.
↧
Why does my search peer come up as an error on my search head?
I've added a search head as a search peer and it's come up as "sick" with the following error. Can't seem to find any reference to it here.
Error [00800000] Failed 11 out of 11 times. Servername used by peer is already in use by the search head.
It suggests a dns issue, but none exists that I can see.
↧
Can we set the ttl for knowledge bundles on indexers?
We have a version 6.3.4 search head cluster and indexers, in a distributed search environment. Noticing that the searchpeers directory has the bundle along with the deltas. Can we set a time to live for these directories? Similar to setting a ttl for dispatch artifacts from scheduled runs.
Not sure if there is a configuration in distsearch or limits that would be able to do this.
↧
↧
Search Head X running splunk version '6.4.0' does not support distributing searches to the following peers: {peer:Y version:6.4.3}
What are the reasons which can cause this error in non clustered indexes ?
As both the major and minor versions are the same between the SH and indexer (only the maintenance one is lower on the SH), according to the compatibility matrix it should work
May this be due to a bundle replication issue?
↧
Before planning to deploy a Distributed Search environment, is there a partition model recommendation before installing Splunk 6.4 on my Linux servers?
I am planning to deploy a Splunk Distributed Search Architecture in a mixed environment of 500 servers mostly Windows and some Red Hat Enterprise (RHEL) Linux 7. Splunk hosts will be RHEL 7.2 I will have two search heads: Enterprise & Security, a 3 node indexer clustered on the Splunk application level, and a separate Deployment Server.
I read that Splunk will create the necessary directories during installation. Is there partition model recommendation or LVM I should have ready before installing Splunk 6.4 in my Linux servers? Or should I just let Splunk create directories automatically during install?
See my current Linux partitions below:
/root 50G
/home/ 200 G
/boot 500mb
/swap/ 8G
/tmp
/var
/var/tmp/
/var/log/
/var/log/audit/
↧
How to get existing KV Store to initialize after replacing one of the three (3) members with a new instance?
Splunkers,
Having trouble getting the kvstore to indicate that it is ready on any of the three members of the shcluster running Splunk 6.4.0 on CentOS 6.7.
There are 5 existing KV Stores and none of them can be accessed.
The trouble began when an overzealous admin accidentally deleted directories in one of one of our running shcluster members while it was running.
Attempted to use CLI commands to remove the corrupted member from one of the other members which seemed to work.
Then killed the Splunk related zombie processes left due to the pid file and bin directory being deleted the corrupted instance CLI could not be used.
Deleted the corrupted /opt/splunk instance then un-tar-ed another instance of Splunk 6.4.0 into a new /opt/splunk to replace the corrupted instance.
Followed the Splunk docks for "init" and "add new" to the shcluster. Once started issued CLI commands to make sure the new instance was properly configured.
The shcluster status is good and searches are possible from any shcluster member.
Attempt to do a simple search such as: | inputlookup
Yields errors indicating that the KV Store was not properly initialized.
If we had backups of the activity store data from the original three member shcluster a clean restart would make sense i.e. rebuild all the stores from scratch. We have files in folders contained in two of the three members, and can not access them via Splunk to create backup CSV files. We are hoping someone can guide us through getting the activity store initialized.
The mongod.log on the two remaining original shcluster members contain events such as:
Error in heartbeat request to ------------ InvalidReplicaSetConfig Our replica set configuration is invalid or does not include us
We have tried most of the non-destructive suggestions provided in Answers and Google searches.
THX
↧
Splunk Enterprise Security: Is it possible to implement multi-tenancy in a distributed search environment?
Hello everybody.
I deployed a Splunk Enterprise Security in a distributed environment for our customer. He also has many customers and he doesn't want to see all the logs together. I've heard ES does not support multi-tenant natively, but at the moment, he wants to have separable reports for customer or see in the dashboard which data belongs to whom.
I don't know if there is a way to reach that. If you know, I will appreciate any help.
I've been looking for something similar and I got this:
https://answers.splunk.com/answers/236674/security-app-with-multi-tentant.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev
Best regards.
↧
↧
Why am I getting duplicate results after adding indexer cluster to distributed search?
I am testing our new indexer cluster using our existing search head. I added the indexer cluster servers to "dist_search" and created an indexer group so I can search just the cluster. However, all of the logs I am searching are duplicated.
I am not sure if this is because of the dist_search settings I have, a mis-configuration of the index cluster, or something else.
I did have load balancing set up on the heavy forwarders, sending to all 3 indexers. I removed that config and am only sending to 1 indexer, but the events are still duplicated.
Please help! Thanks!
↧
How to create an alert to trigger if one of the indexers is not reachable in a distributed search environment?
Hi,
I have 1 search head and 3 indexers where one of them is working as a license node.
I've had a situation where one of them lost connection (service was down).
How do I create an alert for the search head to inform if one of the indexers is not reachable?
↧
How to edit my props.conf for proper event line breaking based on my sample data?
Ok, I give. I can't seem to figure out why this is failing...
This is the log: (Suitably neutered)
2016-11-03 13:34:00,654 [10] INFO XXXXXXX_YYY.XXXXXXX - Script Name Input:
2016-11-03 13:34:00,716 [10] INFO XXXXXXX_YYY.XXXXXXX - account: zzzzzzz
2016-11-03 13:34:00,716 [10] INFO XXXXXXX_YYY.XXXXXXX - No Parameters supplied
2016-11-03 13:34:00,716 [10] INFO XXXXXXX_YYY.XXXXXXX - Total Parameters:0
2016-11-03 13:34:03,259 [10] ERROR XXXXXXX_YYY.XXXXXXX - Powershell script '' does not exist.
2016-11-03 13:34:03,758 [13] INFO XXXXXXX_YYY.XXXXXXX - Script Name Input:lync_provisioning.ps1
2016-11-03 13:34:03,758 [13] INFO XXXXXXX_YYY.XXXXXXX - account: Abcdef.Hijklm@domainname.com
2016-11-03 13:34:03,758 [13] INFO XXXXXXX_YYY.XXXXXXX - Total Parameters:4
2016-11-03 13:34:03,836 [13] INFO XXXXXXX_YYY.XXXXXXX - --------------------Powershell Execute-------------------------
2016-11-03 13:34:03,914 [13] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: DomainController|HOSTNAME
2016-11-03 13:34:03,914 [13] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: sAMAccountName|zzzzzzz
2016-11-03 13:34:03,914 [13] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: PrimarySMTP|Abcdef.Hijklm@domainname.com
2016-11-03 13:34:03,914 [13] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: Action|Disable
2016-11-03 13:34:03,914 [13] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: ScriptRoot|C:\PowerShell Scripts\
2016-11-03 13:34:16,176 [13] INFO XXXXXXX_YYY.XXXXXXX - Powershell Script Return Value: True
2016-11-03 13:34:16,176 [13] INFO XXXXXXX_YYY.XXXXXXX - ------------------Powershell END Execute-----------------------
2016-11-03 13:38:58,650 [12] INFO XXXXXXX_YYY.XXXXXXX - Script Name Input:
2016-11-03 13:38:58,650 [12] INFO XXXXXXX_YYY.XXXXXXX - account: zzzzzzz
2016-11-03 13:38:58,650 [12] INFO XXXXXXX_YYY.XXXXXXX - No Parameters supplied
2016-11-03 13:38:58,650 [12] INFO XXXXXXX_YYY.XXXXXXX - Total Parameters:0
2016-11-03 13:38:58,744 [12] ERROR XXXXXXX_YYY.XXXXXXX - Powershell script '' does not exist.
2016-11-03 13:38:59,258 [19] INFO XXXXXXX_YYY.XXXXXXX - Script Name Input:home_drive_provisioning.ps1
2016-11-03 13:38:59,258 [19] INFO XXXXXXX_YYY.XXXXXXX - account: zzzzzzz
2016-11-03 13:38:59,258 [19] INFO XXXXXXX_YYY.XXXXXXX - Total Parameters:6
2016-11-03 13:38:59,321 [19] INFO XXXXXXX_YYY.XXXXXXX - --------------------Powershell Execute-------------------------
2016-11-03 13:38:59,368 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: HomeDrivePath|\\ZZZZZ\home$
2016-11-03 13:38:59,383 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: IsilonPath|/ifs/pathing/home/
2016-11-03 13:38:59,383 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: Username|userprovisioning
2016-11-03 13:38:59,383 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: Password|**********
2016-11-03 13:38:59,383 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: Account|zzzzzzz
2016-11-03 13:38:59,383 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: TermDate|11/03/2016 00:00:00
2016-11-03 13:38:59,383 [19] INFO XXXXXXX_YYY.XXXXXXX - Parameter Added: ScriptRoot|C:\PowerShell Scripts\
2016-11-03 13:39:01,567 [19] INFO XXXXXXX_YYY.XXXXXXX - Quota Removed from /ifs/nasprod/home/zzzzzzz
2016-11-03 13:39:01,567 [19] INFO XXXXXXX_YYY.XXXXXXX - zzzzzzz moved to _Deleted User Folders_\zzzzzzz 11-03-16
2016-11-03 13:39:01,567 [19] INFO XXXXXXX_YYY.XXXXXXX - Powershell Script Return Value: True
2016-11-03 13:39:01,567 [19] INFO XXXXXXX_YYY.XXXXXXX - ------------------Powershell END Execute-----------------------
I want it broken into exactly two events, based on `------------------Powershell END Execute-----------------------` I got 7.
This is what I have in props.conf.
[SourceType]
NO_BINARY_CHECK = true
# MUST_BREAK_AFTER = Powershell END Execute
# EVENT_BREAKER = Powershell END Execute
category = Custom
description = Log
disabled = false
pulldown_type = true
BREAK_ONLY_BEFORE = Powershell END Execute
DATETIME_CONFIG =
inputs.conf has the correct SourceType for the monitor statement.
This is in a distributed environment. The props.conf is pushed to both the Universal Forwarder, and the Indexer(s). We're running 6.3.3
Seems to work fine, if I put it on a single node, and manually add data.....
So like, what am I missing?
Thanks,
David
↧