Conversation

Notices

1. Douglas A. Whitfield (musicman@nu.federati.net)'s status on Wednesday, 13-May-2020 21:21:51 UTC Douglas A. Whitfield
   It seems unlikely to me that something like #Cassandra would be missing such a file. Do I need to build Cassie from source to get this or something? I haven't been able to find anything to suggest this is a common issue. Is it lying and the javaagent is actually the issue? #java
   
   [root@localhost tmp]# java -javaagent:./jmx_prometheus_javaagent-0.12.0.jar=8080:jmx-prom.yaml -jar /usr/share/cassandra/apache-cassandra-3.11.6.jar -Djava.util.logging.config.file=./program.log
   no main manifest attribute, in /usr/share/cassandra/apache-cassandra-3.11.6.jar
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Wednesday, 13-May-2020 21:58:04 UTC Douglas A. Whitfield

     in reply to
     I was able to unjar the file and then in the manifest I noticed it pointed to a different file for classpath.
     
     same issue with that file though
     
     [root@localhost lib]# java -javaagent:./jmx_prometheus_javaagent-0.12.0.jar=8080:jmx-prom.yaml -jar /usr/share/cassandra/apache-cassandra-thrift-3.11.6.jar -Djava.util.logging.config.file=./program.log
     no main manifest attribute, in /usr/share/cassandra/apache-cassandra-thrift-3.11.6.jar
   • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} (lnxw48a1@nu.federati.net)'s status on Wednesday, 13-May-2020 21:58:57 UTC LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}

     in reply to
     @musicman It seems strange that something required like that would be missing. Have you looked at a different installation package?
   • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} (lnxw48a1@nu.federati.net)'s status on Wednesday, 13-May-2020 22:15:56 UTC LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     @musicman
     Sounds like you're on your way. By now, I'm sure you've tried unjarring the next file.
     
     Take good notes, because I would not be surprised if some of your employer's clients will run into similar issues with it.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Thursday, 14-May-2020 01:41:04 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     it's the official repo:
     
     [root@localhost lib]# yum repolist
     Last metadata expiration check: 0:00:14 ago on Wed 13 May 2020 08:38:52 PM CDT.
     repo id repo name status
     AppStream CentOS-8 - AppStream 5,318
     BaseOS CentOS-8 - Base 1,661
     cassandra Apache Cassandra 3
     *epel Extra Packages for Enterprise Linux 8 - x86_64 5,506
     *epel-modular Extra Packages for Enterprise Linux Modular 8 - x86_64 0
     extras CentOS-8 - Extras 16
     zabbix Zabbix Official Repository - x86_64 18
     zabbix-non-supported Zabbix Official Repository non-supported - x86_64 1
     
     I guess Prometheus is pretty popular, but I suspect this is a problem with the prometheus agent and not Cassandra.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Thursday, 14-May-2020 19:33:43 UTC Douglas A. Whitfield

     in reply to
     so now I have no idea where to go:
     
     [root@localhost META-INF]# cat MANIFEST.MF
     Manifest-Version: 1.0
     Ant-Version: Apache Ant 1.9.3
     Created-By: 1.8.0_20-b26 (Oracle Corporation)
     Agent-Class: org.github.jamm.MemoryMeter
     Premain-Class: org.github.jamm.MemoryMeter
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Thursday, 14-May-2020 20:28:19 UTC Douglas A. Whitfield

     in reply to
     I don't see anything in the release notes to suggest it will help, but there is a new version of the jmx_exporter
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Thursday, 14-May-2020 21:06:52 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     you know, searching the Cassandra source is probably an easier way that unjarring things. I don't understand these premain classes. I was able to build a simple jar. I need to see if the jmx exporter works on it
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Thursday, 14-May-2020 21:13:39 UTC Douglas A. Whitfield

     in reply to
     the main thing I am learning is that I have no idea what I am doing. git cloned the source though as suggested at https://github.com/prometheus/jmx_exporter
   • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} (lnxw48a1@nu.federati.net)'s status on Thursday, 14-May-2020 21:22:53 UTC LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}

     in reply to
     @musicman That sounds like a good idea.
     
     Being that this _is_ #Java, also take a good look at the command line and environment variables needed for starting it. In the olden days, variables like $JAVA_PATH and $CLASSPATH had to be defined before the "java -jar somejarfile.jar" command ... some variables were small enough to define on the command line with "-Dsomevar=somevalue".
   • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} (lnxw48a1@nu.federati.net)'s status on Thursday, 14-May-2020 21:37:21 UTC LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}

     in reply to
     @musicman It has been a loooong time since I've messed around with #Java (I'd say 13-14 years), but if the CLASSPATH was set correctly, I never had to unjar jarfiles. I never had the situation where a required file was missing from the jar, so you may be in unknown territory.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 14:14:23 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     Java path is too long to post...I guess there is pastebin
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 14:15:13 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     https://pastebin.com/zAdiexNp
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 14:18:00 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     I can confirm this works on Cassandra 3.11.6: https://www.robustperception.io/monitoring-cassandra-with-prometheus
     
     so, for now, I can use that and not the packaged CentOS version, though figuring out what the hell is different about the packaged version is probably a good exercise for understand this. Cassandra is one of the things we are week on in the team (though not java, that's a personal weakness). While I am obviously going to be the SME for monitoring, I think I am going to end up that way for Cassandra as well just out of necessity.
     LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} likes this.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 14:39:08 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     one thing I immediately notice looking at the command line of the working and non-working is that they are running as different users.
     LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} likes this.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 15:24:10 UTC Douglas A. Whitfield

     in reply to
     I had long suspected that this pidfile might be the source of the issue at least regarding not having a jar at the top of the tree and the fact that the non-package version doesn't have it leads me to believe that is the case:
     -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid
     
     Now that I have something that works, it should be much easier to figure out what is going on.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 15:51:34 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     ok, this is weird...
     
     https://pastebin.com/S0ZT0X2u
     
     My guess is the two Cassandras are fighting. Should be easy enough to test...
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 15:54:08 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     it's working. I think all I did was take out the hostname...well, it's working for localhost, but not even that was working before.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Friday, 15-May-2020 15:55:09 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     IP works too. The comment in the cassandra-env.sh file says:
     
     # add this if you're having trouble connecting:
     #JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=192.168.40.233"
     
     I guess the reverse is also true.
   • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864} (lnxw48a1@nu.federati.net)'s status on Friday, 15-May-2020 20:07:58 UTC LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}

     in reply to
     "It is working" are good words to hear, but don't stop there. Document everything you've found and the steps it took to get there. You're likely to find that others have similar issues. Now that you're "Cassandra Doug", those notes will help keep you from starting the search de novo next time.
   • Douglas A. Whitfield (musicman@nu.federati.net)'s status on Saturday, 16-May-2020 00:15:08 UTC Douglas A. Whitfield

     in reply to

     • LinuxWalt (@lnxw48a1) {3EB165E0-5BB1-45D2-9E7D-93B31821F864}
     I have a feeling there was a port conflict between me trying to run the jar manually and the jar running as part of Cassandra...at least at some point.
     
     I doubt my process would be very useful for anyone that knows much of anything about Java.
     
     That said, once I have an understanding about what to alert on and if this jmx_exporter has everything we need, there might be a blog at https://www.openlogic.com/blog