Netcool internationalization revised

I simplified the rules for handling of Japanese characters in Netcool:

Everything in Netcool should be in shift-jis encoding.

This ensures that Japanese text is readable in the Netcool event viewer.

Everything in our tools is in UTF-8

This works best with webservices and browsers

The new Perl DBI dataserver encodes and decodes from and to UTF and shift-jis to ensure that. All data coming back from the Perl dataserver is already translated to UTF-8. Data going into the dataserver is expected to be UTF-8 as well. The dataserver than translates to shift-jis.

The SocketGatewayListener and Eventviewer scripts have been modified to do the same thing as well.

New Perl DBI / Freetds dataserver configuration

The new dataserver is installed in Development now, on ip.ad.dr.ess.

access

Urls:
http://ip.ad.dr.ess/query?sql=select * from alerts.status&format=html

https://ip.ad.dr.ess/update

The query URL is for selects and will return results. The update URL does not return results, but just a success indication.

The web app takes 2 parameters, both using GET and POST:

• sql – the sql statement to be executed. No ; at the end
• format – either xml or html. If nothing is specified, html will be returned

The format of the result is a

. The first row contains the column headers. Errors are returned in a table with a single cell.Error handling is very limited, since DBI is not returning any errors even if a completely invalid sql statement is passed in. Only parameter checks and login errors. configuration /apps/NetcoolData.conf contains the following: LogFile /apps/logs/NetcoolData.log ErrorLogFile /apps/logs/NetcoolData.errors DebugLogFile /apps/logs/NetcoolData.debug Server OBJSERV Database alerts User root Passwd well-yeah Debug 1 Debug = 1 enables verbose logging. Debug = 0 only logs errors. /usr/local/apache2/conf/httpd.conf contains the following: <Location /query> SetHandler perl-script PerlResponseHandler Lib::NetcoolDataQuery </Location> <Location /update> SetHandler perl-script PerlResponseHandler Lib::NetcoolDataUpdate </Location> /apps/dbi/freetds-0.62.4/freetds.conf contains the following [OBJSERV] host =ip.ad.dr.ess port = 4100 tds version = 5.0 The following environment variables need to be set in the Apache environment: FREETDSCONF=’/apps/freetds.conf’ LD_LIBRARY_PATH=’/usr/local/lib’ These are set (for now) in /etc/init.d/apache After a config change you need to restart or HUP apache. (/usr/local/apache2/bin/apachectl restart)

Japanese characters in Netcool

While testing the new Perl DBI netcool dataserver, it was found that the Netcool event viewer displays Japanese characters correctly only if they use shift-JIS encoding. It will preserve any 8-bit encoding (like UTF-8), but will scramble 16 bit encodings (like UTF=16). The SSM log monitor works correctly only for Shift-JIS log files (fortunately, most use Shift-JIS). Events from the dispatcher had their text scrambled in Netcool because the text is in UTF-8

The Perl DBI dataserver was modified to return the right encoding (both in HTTP header (charset=ShiftJIS) and in the XML (encoding=”shift-jis”).

In addition, the EventAdapter was modified to encode the incoming UTF-8 XML  events into shift-jis before it is entered into Netcool.

One final thing to do is to convert data from shift-jis to UTF-8 in the socket gateway listener.

Thus, from now on the assumption is that

1. Data in netcool is in Shift-JIS
2. Data in XML events in all other tools is in UTF-8
3. Data pulled from netcool using the dataserver is in Shift-JIS but with the correct encoding specified in the XML, so that it can be converted by the client application. (It is an option to encode in UTF-8 if this causes problems).
4. Data send out from netcool using the Socket gateway is converted to UTF-8
5. Applications modifying the netcool db directly have to take care that the right encoding (Shift-JIS) is used.

Accessing Objectserver Sybase

To replace the Java dataserver application that we are using to access the in-memory Sybase database of Objectserver, I tested Perl::DBI and freetds. I got it to work by compiling a patched freetds and DBD-Sybase using some hints from the Netcoolusers.org site. It seems to work fine without hanging up even with all unicode chars present. The existing Java dataserver app hangs up on this test

packages used:

• packages:
• freetds-0.62.4
• freetds patch (netcoolusers.org)
• DBD-Sybase-1.04
• dbd-sybase patch (netcoolusers.org)
• DBI-1.48
• Gnu patch utility

how:

unpack freetds and build

To build the FreeTDS makefile use this command line:

   ./configure --prefix=/usr/local --enable-shared

 --disable-static --disable-libiconv --disable-odbc --disable-server --disable-pool --enable-sybase-compat

--disable-debug --with-pic --with-tdsver=5.0

for testing configure freetds.conf and PWD files with right data

set load path so that dynamically loaded libs can be found

use version 5 of driver

build dbd-sybase

(not all tests will work)

env variables:

export FREETDSCONF=/apps/dbi/freetds-0.62.4/freetds.conf

export LD_LIBRARY_PATH=/usr/local/lib

in contrast to docs, freetds.conf is used, not interface file

To enable Apache mod perl:

Add the following to httpd.conf

    SetHandler perl-script

    PerlResponseHandler Smart24::NetcoolData

In addition the env vars mentioned above need to be in the apache environment. This was done by adding them to the apachectl script