I recently added a new splunk server in a distributed environment. Now, when I do this search:
index=os earliest="09/01/2015:09:30:00" latest="09/01/2015:09:35:00" | timechart count by splunk_server
the new splunk server does not show up in the results. However, if I do this search,
index=os splunk_server=* earliest="09/01/2015:09:30:00" latest="09/01/2015:09:35:00" | timechart count by splunk_server
then, it shows up.
Can anyone tell me why? I have the search load-balanced so I have about the same number of events going into each indexer.
Thank you in advance.
↧
After adding a new Splunk server in a distributed environment, why does it not show up in results unless I include splunk_server=*?
↧
How to change the index for the Splunk App and Add-on for Unix and Linux after installation in a distributed search environment?
We are in the process of deploying the Splunk App for Unix and Linux on our Linux servers in a distributed Splunk environment. I was able to successfully change the indexer from the default (os) to the one that we want to use in a standalone instance by modifying the instance name in the untarred source files for Unix app, then installing from those modified files. However, in the distributed environment, we want to be able to install from the source files and then be able to change the index after the install. We already have the index name that we want to use defined on our indexers, but I don't really understand how we can change the indexes after the app is installed. Can anyone give me a hand with this?
↧
↧
How will the S.o.S. - Splunk on Splunk app impact my license usage in a distributed search environment?
I tried to search this, but didn't seem to find an answer. I understand that all the logs that come to a Splunk Indexer from _INTERNAL does not count under Splunk licensing.
I have a distributed architecture in my Organization with Multiple Search Heads, Dispatchers, Indexers, and Forwarders, and I want to Start System Health Check using S.O.S. App. However, will this add additional data to indexer since the performance data from other servers (Forwarders etc) also needs to be indexed?
Can somebody please throw some light on this topic?
Thanks In advance
Best Regards,
Neel Shah
↧
Why isn't my index available for search in a distributed search environment?
Hi to everyone
I have a "Distributed Environment", with two indexers, and two search heads.
In the Master Node Indexer, I have an index called ftp, with a lot of data (I want this data available for distributed search). I've deployed "indexes.conf" to "search peers", and I can see the ftp index created in the search peer, but I can't see any data.
What can i do for have this data available for distributed search?
Regards
↧
After installing Cisco Security Suite, why am I getting "KeyError: 'elements'" during setup in a distributed search environment?
I've installed Cisco Security Suite 3.1.1 on my Splunk Enterprise search head and restarted Splunk.
When prompted to run the setup, I get an error message:
KeyError: 'elements'
View more information about your request (request ID = 55f6ece9d64122780) in Search
This page was linked to from http://mysplunkserver:8000/en-US/manager/appinstall/Splunk_CiscoSecuritySuite/checkstatus?state=eJx1jrEKwkAQRH_l2MIqcCDYCEH8BrUKIWwum0TY7B17d4WI_-6ChTZ2w7zhMU8YlXAKWrcxw9F1HVyolLssGRoHfkPBhdQzVgmrBegb18E5pT8cjXiOAflkcYjCj_aqlXZxnjOVdn_4GG6JI07ONuaRytxbDUqlqgwl2pWvPBNqWH_U8HoDupo_RA%3D%3D.
We run a distributed search environment where the search head and indexer are different physical machines, if that matters.
↧
↧
Does decrypt work in distributed search environments?
I can get this app to work fine, if I'm running in locally on an indexer. But not from a distributed search head.
index=_internal | decrypt field=sourcetype hex() emit('sourcetype')
Corresponding Errors:
[xxxxx] Streamed search execute failed because: Error in 'decrypt' command: Cannot find program 'decrypt' or script 'decrypt'.
[xxxxx] Streamed search execute failed because: Error in 'decrypt' command: Cannot find program 'decrypt' or script 'decrypt'.
[xxxxx] Streamed search execute failed because: Error in 'decrypt' command: Cannot find program 'decrypt' or script 'decrypt'.
[xxxxx] Streamed search execute failed because: Error in 'decrypt' command: Cannot find program 'decrypt' or script 'decrypt'.
Works when I go to each indexer and run the command but not from the search head.
I basically looking for any app/script that will do base64 decoding from a distributed set up. Thus far I can seem to find one.
Thanks,
Lisa
↧
Distributed Search: Is it possible to configure a search head to search a remote search head that is within an indexer clustering environment?
I'm having a hard time finding anything regarding this setup, so I'm trying my luck here.
Is it possible to configure a Search Head to search a remote Search Head that is within a cluster environment?
[Search Head]
|-- [Remote Search Head]
|-- [Indexer 1]
|-- [Indexer 2]
When I configure [Remote Search Head] as a Distributed Search Peer on the [Search Head], no data is returned. Status: OK, Replication status: OK.
For testing purposes, I have connected the peer as "admin"
↧
Why is the splunkd.log reporting lots of "DistributedPeerManager - Unable to distribute to peer named...because peer has status = "Down"."?
I have a very busy search head that complains :
DistributedPeerManager - Unable to distribute to peer named slxxxxxxxxx:9089 at uri https://xxxxxxxx037:9089 because peer has status = "Down"
The messages will start in splunkd.log at 22:08:10.971 and finish at
22:09:46.994, but the message is reported about 60 times during short time period. A telnet from the SH to the indexer on 9089 shows no connectivity issues.
This has happened off and on for all indexers configured in distributed search. I am wondering if there is a setting that could be adjusted that to prevent these messages from occurring, or if there is a conf value that could be adjust to improve performance under high load. The SH is 10vpcus by 32gig, and there is a high load average on the SH and indexers (lots of searches).
There appears to be no negative impact to the messages, since searches are working. Users are not reporting any issues.
↧
Distributed Search Replication Failure after 6.3 upgrade with error "replicationStatus Failed failure info: failed_because_NONE"
I've seen a few related issues on Answers, but not this specific error.
I have a deployment with a single search head, two indexers, and a cluster master. After upgrading to 6.3, my search head can no longer replicate the knowledge bundle to both indexers. Replication status says "Failed" in distributed search and when attempting a search, I see the following error for both indexers. Identifying info redacted.
Unable to distribute to peer named at uri https://:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_NONE
Searches work just fine from my cluster master and replication says Successful there. Anyone know what's going on? I even started a completely fresh installation and rebuilt the cluster to no avail.
↧
↧
Splunk Support for Active Directory: "ERROR The default configuration stanza for ldap.conf is missing." using ldapfetch in a distributed search environment
Hi there,
I installed SA-ldapsearch as decsribed in the docs on the search head in my distributed environment. When I run a simple ldapsearch, everything works just fine.
Now I want to query stuff from my indexers and feed the results into ldapfetch (idea: find windows groups in a log and use ldapfetch to find the members of that group).
Search head and indexers run on different machines.
I get the following errors from my indexers:
[map]: [PRDS0052] External search command 'ldapfetch' returned error code 1. Script output = " ERROR The default configuration stanza for ldap.conf is missing. "
[map]: [PRDS0053] External search command 'ldapfetch' returned error code 1. Script output = " ERROR The default configuration stanza for ldap.conf is missing. "
How can I prevent the ldap command from run on the indexers? For the lookup command, there is a "local" attribute. I cannot find sth like that in ldapfetch.
Any idea?
Best,
Bernd
↧
Why am I seeing "Authentication failed" in the Distributed Management Console for search peers added using CLI commands in a script?
Hi all,
I add the search peers by using the CLI commands in a script. When I check the Distributed Management Console UI, I see that the status for all of them is "Authentication failed". The credentials provided are correct. If I delete them and re-add them manually (I just copy the exact servername:port) they are instantly "Up". What extra step am I missing in my script to have them up instantly?
Thanks,
Kimche
↧
How to migrate a Splunk 4.3 indexer to two new indexer servers that will be running Splunk 6.x for distributed search?
I will move the indexer server to new indexer servers
Environment
- before : 1 indexer server Splunk 4.3 (not distributed, will disuse)
- after : 2 indexer servers Splunk 6.x (distributed search)
I can't find what is the best way
help me plz
↧
How to configure an automatic lookup in a distributed search environment with indexer clustering?
I've been trying to find how to create automatic lookups on a distributed deployment.
I have a fairly large collection of normal search time lookups on my search head cluster, but when I try to make one work as an automatic lookup, I get errors saying the lookup table doesn't exist on the indexers involved (also clustered).
So, do I create the lookup table and transforms/props config on the cluster master and push to the indexers, or is there a way of telling the system to run that auto-lookup locally on the search heads? My google-fu doesn't seem to be good enough to filter out all the info regarding either (but not both of) automatic lookups or distributed deployments.
I'm doing the lookup as automatic as it seems the only way to do a cidr based lookup as per https://answers.splunk.com/answers/93620/lookup-with-cidr.html
Again, I could be wrong there.
:-)
↧
↧
How to install the Security Query based App for RSA Security Analytics in a distributed search environment, and how to poll my queries with separate brokers and concentrators?
Hello;
I am excited to try this newly released app, and have a few questions:
1. My setup has several brokers and concentrators, none combined; each broker and concentrator a separate server. Any recommendations for best way to deploy/poll my queries?
2. My Splunk setup is a distributed. Should this app be installed on my indexers only?
Thank you,
-mi
↧
Understanding distributed search replication blacklisting behaviour
I'm trying to understand what happens to distsearch when you black list something. For example a csv file.
I've been looking into what is the best methodology for stopping large csv files from being sent to indexers via bundle replication. We have noticed recently power users creating ever growing lookup files. These eventually result in field extraction issues as normal props/transforms don't get replicated in a timely fashion. As such we're looking to limit csv's in the bundle.
Blocking them is the easy part. ie distsearch.conf [replicationBlacklist]
My issue becomes what is the flow on effect of doing this? Indexers can no longer reference the lookup file in a search so what happened then? The indexer requires is for the search, it doesn't find it so it streams back all the results instead? Does the search just fail to return anything if it uses a inputlookup early in the search?
What is actually happening under the hood when you blacklist a lookup?
↧
Why are Threat, Traffic, and Content dashboards in the Splunk for Palo Alto Networks app showing "No results found" in my distributed search environment?
Hi,
I just recent installed the Splunk for Palo Alto Networks app. After digging around and changingthe index to match what we built in-house, I was able to see the main dashboard populating data. The other tabs, however, are all empty. Threat, Traffic, Content are all showing no results found. If someone already has a solution for this in a different post, please point me to it. Or if you know where to start digging to get data generated please let me know.
Thanks!
↧
Why is the Splunk App for Stream not extracting the correct _time field in my distributed search environment?
Hello,
I have a universal forwarder and an indexer, and deployed the Splunk App for Stream app as documented.
My problem is about extracting _time.
From the Splunk_TA_stream/default/props.conf file of the indexer, I could find the following about the _time extraction.
[source::stream:*]
KV_MODE = json
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE=false
# stream events use "timestamp" for the start of the event, and
# "endtime" for the end of the event; the latter is used for _time
TIME_PREFIX=\"endtime\":\"
It is configured to extract _time value from the "endtime" tag of json, but as you can see from the attached image, the extracted _time has different values. Because I'm in GMT+9, the difference between endtime and _time should be 9 hours exactly, but the seconds or minutes values are different. It looks to use 'current time'. I tried to change several configurations including MAX_TIMESTAMP_LOOKAHEAD, but not successful.
The _time was extracted correctly in a single node environment. This problem only seems to happen in a distributed env.
Is there anything that I need to check?
![alt text][1]
[1]: /storage/temp/69181-stream-time.png
↧
↧
Distributed Search: Should I install the search head on the same server as an indexer in my environment?
Hello,
I have 2 servers available to deploy Splunk. If I read this doc : http://docs.splunk.com/Documentation/Splunk/6.2.4/Capacity/Referencehardware I understand that I should put a Search head on one server and the Indexer on the other.
But I was thinking: Isn't it better to deploy 1 indexer on each server and install the search head role on one of the servers? This way, I can distribute the cost of a search since the search engine is on the Indexers.
Since the search head is just a web application, and I will have only 2-3 users using this web application, so I think I don't need to dedicate a whole server for that role.
Moreover, when running a search with the `stats` command, can you tell me the job of the search head? Is it just displaying data?
Or doesn't it take the data from the indexer and organize it?
What do you think?
↧
Why are my search peers showing a status of "duplicate license"?
I recently set up two dedicated search heads in my Splunk environment. After installing Splunk Enterprise, I cut & pasted the contents of my valid enterprise license XML file in the licensing section on each search head. However, after 72 hours, searching is disabled on my search head and when I look in "distributed search" at my search peers, they all show a status of "duplicate license".
Is there a reason why my search peers think that there is a duplicate license issue?
↧
Why are searches that use lookups failing in a search head clustering environment unless we use local=t?
Hi,
We have recently migrated to a search head clustering environment, but unfortunately, all the searches using lookups are failing, but they work when we use local=t.
Does this mean that the lookup replication is not happening properly in Splunk?
Kindly advise.
↧