I was attempting to re-install the Grid Control agents on two RHEL 6 servers today today. They’d had been upgraded from RHEL 5 to 6 and I wanted to see what needed to be done to re-attach them to Grid Control if we didn’t take the existing installation. I was having some issues with it (the agent wouldn’t start up – complained about “OMS decided to shutdown the agent because of the following reason sent from OMS: AGENT_TZ_MISMATCH”) After a bit of digging I started to find various levels of weirdness occurring.
I started off looking in Grid Control.
If you look at the hosts table of grid control:
select Timezone_region, count(*) from sysman.mgmt_targets group by Timezone_region order by 2;You see we have a nice interesting selection of TimeZones in use:
+00:00 2 Europe/Isle_of_Man 6 +01:00 8 Europe/Guernsey 26 Europe/Lonon 49 Europe/Jersey 59 GB 80 Europe/Belfast 85 GB-Eire 367
That’s a wee bit scary – why on earth do all the servers think they are scattered over most of England (including the Channel Islands). We are based in London. Now I have checked a bit and at the OS level we set the timezone using:
/etc/sysconfig/clock
To:
ZONE="Europe/London"And date is always sensible too:
Wed Feb 20 15:39:01 GMT 2013So there I think there is some weirdness at work when the agent tries to work out the time zone in use. You can see what the agent thinks the zone is using:
cd [agent home]/bin ./emctl config agent getTZ
Depending on the server you run it on you get different answers:
Europe/Guernsey GB GB-Eire GB
Ignore the apparent consistency here, it isn’t true…. I found a chunk of Java code in a Oracle Support note (330737.1) that seemed show the same results (I expanded it slightly to check another set of values):
import java.util.*;public class TestTZ { public static void main(String[] args) { System.out.println("Local Timezone: "+(TimeZone.getDefault()).getDisplayName()); System.out.println("Local ID: "+(TimeZone.getDefault()).getID()); TimeZone tz = Calendar.getInstance().getTimeZone(); System.out.println("Calendar TimeZone: "+tz.getDisplayName()); System.out.println("Calendar ID: "+tz.getID()); } }
The output looks the same as what the agent reports. But I couldn’t find a pattern to what was goin on. In desperation I moved to looking at the actual timezone files … Its safest note to ask why.. lets just say I was desperate and looking for any possible reason for this now. I used the md5sum of localtime:
$ md5sum /etc/localtime 410c65079e6d14f4eedf50c19bd073f8 /etc/localtime
And then found zone files that had the same md5sum (just don’t ask why I started to do this):
$ find /usr/share/zoneinfo -type f | xargs md5sum | grep 410c65079e6d14f4eedf50c19bd073f8 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Guernsey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB-Eire 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Guernsey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB-Eire
Its worth pointing out here that they are actually all just links to the same file (the file system inodes match). Now when I looked at these I (finally) started to see some consistency. The file at the top of this list matches the zone reported back. So on the first server I see:
-bash-4.1$ find /usr/share/zoneinfo -type f | xargs md5sum | grep 410c65079e6d14f4eedf50c19bd073f8 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Guernsey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB-Eire 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB-Eire -bash-4.1$ ./emctl config agent getTZ Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Europe/Guernsey
On a second server, I see:
-bash-4.1$ find /usr/share/zoneinfo -type f | xargs md5sum | grep 410c65079e6d14f4eedf50c19bd073f8 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Guernsey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Guernsey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB-Eire 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB-Eire -bash-4.1$ ./emctl config agent getTZ Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. GB
For the OS, much like a database, the order that files are returned here is indeterminant. To make it determinant, and order would have to be imposed. It hasn’t been, and it looks like the method being used by the agent (and Java) to determine the timezone is equally unordered. An OS colleague and I then tried an experiment. He removed the top file (Europe/Guernsey) from the first system:
-bash-4.1$ find /usr/share/zoneinfo -type f | xargs md5sum | grep 410c65079e6d14f4eedf50c19bd073f8 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/posix/GB-Eire 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/London 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Isle_of_Man 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Jersey 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/Europe/Belfast 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB 410c65079e6d14f4eedf50c19bd073f8 /usr/share/zoneinfo/GB-Eire
Once he did this, the zone reported shifted to the next line:
-bash-4.1$ ./emctl config agent getTZ Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Europe/London -bash-4.1$ java TestTZ Local Timezone: Greenwich Mean Time Local ID: Europe/London Calendar TimeZone: Greenwich Mean Time Calendar ID: Europe/London
So what does all of this tell me (and therefore you)? We have a bit of a problem (especially with java). If (for whatever the reason) code wants to look up what the timezone is, it just gets the top row of the unsorted list. Which is ridiculously stupid. The way to avoid this it to impose our own Timezone. If we set the TZ environment value:
export TZ=Europe/Belfast
It will then be picked up by java in preference:
-bash-4.1$ java -cp ~ TestTZ Local Timezone: Greenwich Mean Time Local ID: Europe/Belfast Calendar TimeZone: Greenwich Mean Time Calendar ID: Europe/Belfast -bash-4.1$ ./emctl config agent getTZ Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Europe/Belfast
So we can impose consistency again (although don’t use Belfast, that was to prove a point). This means two things: Going forward, we will have to add the setting of the TZ environment variable at the server level. This value will be Europe/London (so its consistent with /etc/sysconfig/clock). This means things running on new servers will know they are in Europe/London and going forward things will be fine. We also need to set the TZ value in .profile and .bash_profile for any Oracle Agent user especially where Java is being used (so WebLogic). This will then bring consistency back to the agents (and any other java installs). That said, for agents you will need to have the following run on them as they are fixed:
./emctl stop agent ./emctl resetTZ agent
This sets the values in the properties file correctly. After that has happens the agent setup also needs changing in the database:
alter session set current_schema=[oms user];
exec mgmt_target.set_agent_tzrgn('[oms server]:[port]','Europe/London');
commit;
And then the agent can be started:
./emctl start agent
All of this will make agents consistent in the system again, and should prevent weird answers about timezones from java going forward (which hopefully should prevent java date/time zone weirdness in the future). Its important to note this is a general Java issue – any java app could show similar symptoms. Now if anyone wants me I’ll be in a darkened room bashing my head against a wall and mumbling about Oracle and my hatred of Java.