My Technical Notes Blog

Indexing Unstructured Data

pauldavidgilligan — Sun, 19 Jul 2009 21:55:51 +0000

Introduction

There are a number of Document Management Systems (DMS) on the market and some are very expensive, so why not build your own ? You could use the tips here also for other applications when you want to index data with a low selectivity or data that is classed as unstructured [http://en.wikipedia.org/wiki/Unstructured_data]. I have Oracle relational databases in mind in this article but the same techniques could be used for other relational database types.

Objective

The problem with most indexes is that in general they need to match the data in the columns used in the table in order to be selective. There are plenty of articles from Oracle on this and other sources so I won’t go into it here.

Unstructured data is difficult to index and often you do not know in advance the values that will be contained in the columns.

To aid in locating or classifying documents often phrases or key words are used and or other data that makes up a bibliographical reference for the document [http://en.wikipedia.org/wiki/Bibliography].

Consider you are loading many documents with a simple key phrase, e.g. GOOSE, GOOSEBERRY, GOOSECOID PROTEIN, GOOSEFLESH, GOPHER.

This sequence of phrases or key words represents 5 phrases of index key “GO”, if this was followed by 100 phrases of “JA” and 1000 phrases of “WA” you can soon see that the selectivity is difficult to control and predict. Oracle’s Cost Based Optimiser and various other indexing techniques can help here but in production environments you really want to guarantee the time it takes to access the documents that you are searching for.

The objective here is therefore to provide a framework around the document data that can be used to guarantee document access times.

Simple Data Model

One answer or tip here is very simple, you have to turn the problem around in order to solve it. If we step back from the problem and think what is the fastest way to access data in a logical database ?

The answer is the primary key.

I am of cause ignoring the physical keys such as rowid in an Oracle database as I want to keep the solution logical and so you might come up with the data model below:

Simple DMS indexing

In the data model above we make the PHRASE unique and a primary key.

We generate a unique identifier for the row GID, I have used the global identifier that Oracle/JAVA can generate in this example, this is a technical unique identifier and has little meaning for the end user.

The detail table shows the same GID row, with a sequence and then a pointer to the actual document we are trying to locate. This document pointer could be anything of cause and depends on your own design.

So you need an application to handle to incoming data that says:

I have a new PHRASE, so generate a new GID value and add rows in the master detail tables.
I have an existing PHRASE so add a new row to the detail table with a new sequence value.

A search for a given phrase you then lookup the PHRASE in the master table and get a list, via the GID, of the documents from the detail table. The lookup time should be stable and predictable even with larger row counts for the detail table.

Further more both tables could in fact be Index Oriented Tables (IOT) in the Oracle case.

State Machine

Bibliographical records for documents can contain a number of data field types, so you could have a number of master detail tables for each field type and PL/SQL routines to maintain the data.

The application logic to control this might look like the diagram segment I have drawn below:

State Machine Section

The diagram divided into three sections as you often load initial data into a database temporary section which I have called load time. Maintain and query time are the sections that deal with the query framework. The CAR objects are the database objects that store the document data and the CAM objects store the master detail objects in the data model given in this tip. I am using a car engine parts for the object group names which is just my sense of humor. Ignore the CIT objects as they have another purpose.

The diagram really says at X2.10 that you need to generate the required data for each incoming documents bibliographical fields. Each CAM would represent each field, PUBLISHED DATE, AUTHOR, TITLE, PHRASE etc. so you might have to add rows to each CAM depending on the availability of the field data.

X2.11 deals with the removal of a document which should be a rare operation.

So there is a concept of promoting a document for query and then demoting it again if the document is in error.

Disadvantages

The tip here is more complex than using standard indexes and should be considered for large document, but most DMS systems are complicated. Disk storage, CPU and memory are though cheep these days.

Setup SSH public key authentication

pauldavidgilligan — Tue, 07 Jul 2009 17:00:31 +0000

Introduction

Just some quick notes on how to setup SSH public key authentication on unix/linux type systems. There are various notes around but it can be a pain sometimes to setup and I have some useful tips.

Setup A (normally in .ssh)

You can generated an rsa keypair with : ssh-keygen -b 2048 -t rsa
Then make a copy of the public ket file with a name that indicates the source host and user e.g. cat id_rsa.pub > myuser_myhostid_rsa.pub
Copy the public key file you have generated onto each host you want to connect to under the same system user account which is myuser in this case.
So on the source host myhostid in this example we have ~myuser/.ssh/id_rsa & ~myuser/.ssh/myuser_myhostid_rsa.pub
On the target hosts we have ~myuser/.ssh/myuser_myhostid_rsa.pub (which we copied)
On the target hosts you need open a new terminal window and find the the log file that contains the sshd entries.
You should now tail -f the sshd log file
Now if you try and connect to the target host you will find it will not work, the sshd has made error entries in the log file informing you that there is a problem with configuration – this could be host not accepted or file permission errors
The reason for this is that the sshd is very strict about the locations and permissions of the files under the .ssh directory

Setup B

On the target machine you now need to correct these errors:-

host permission with cat ‘myuser_myhostid_rsa.pub‘ >> authorized_keys
chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/identity

Now from the source host you should be able to connect with little problems and see the connection data logged.

Using Apache Ant to handle Peoplesoft Cobol Build

pauldavidgilligan — Sun, 05 Jul 2009 10:14:06 +0000

Introduction

The ant build script in this blog is a little out of date as Peoplesoft(now Oracle) rely less and less on cobol these days but never the less I thought to write up the techniques here as they can be applied to other situations.

The issues we were facing were:

Peoplesoft is developed for both Windows and Unix platforms so CRLF issues vary from file to file.
Peoplesoft file extensions are not always all upper case or lower case creating yet more issues for compilation.
File permissions had to be set correctly (as in most cases).
A bundle for distribution was required.
A call would be required for the pscbl.mak command to build the cobol source files.

What It Does

The convert task does all the file type and content corrections, ant does not have the built in functions for CRLF correction at the time of development so I made some quick and dirty plugins (fixfilecase & fixcrlf).

The compile tasks excecutes a skeleton Peoplesoft environment that has just enough of an install for to run.

The Ant Build Script

This build project allows you to build a distribution of cobol
source from NT based development, correcting for CRLF chars and
source file case sensitivity which can then be distributed to unix
application servers.

basedir=”${aabhrms.cobolsrc}/${build}/${release}”/>
src=”${aabdist}/cobolsrc.${release}.tar”/>

basedir=”${aabhrms.dmssrc}/${build}/${release}”/>
src=”${aabdist}/dmssrc.${release}.tar”/>

basedir=”${ps_home}/cblbin”/>
src=”${aabdist}/cobolbin.${release}.tar”/>

Importing pkcs12 data into jks keystore

pauldavidgilligan — Fri, 03 Jul 2009 12:19:33 +0000

Introduction

Recently I was working on a minor tool in java to organise and import various pkcs12 data files that were provided by third partly providers to allow SSL over HTTP connections to be made. The recommended and prefered local storage is the jks format so these data files needed to be converted and imported in bulk into a jks data file. Of cause you can do this by hand but I had many files to look at so I tried to program a tool to help out, the funny thing was it was a very simple process and worked well – I did not ever have to worry if the same key entry existed in the jks data file so I could re-run the same bulk import many times with no problems.

Development Links

http://en.wikipedia.org/wiki/PKCS
http://java.sun.com/javase/6/docs/technotes/tools/solaris/keytool.html
http://java.sun.com/javase/6/docs/technotes/guides/security/SunProviders.html

Example Java Code

public static void ImportPKCS12toJKS( File p12keystorefile,
String p12keystorepass,
String p12keyalias,
String jkskeystorename,
String jkskeystorepass) {

File jksfile = null;

FileOutputStream jksfos = null;
FileInputStream p12fis = null;
FileInputStream jksfis = null;

try {
jksfile = new File(jkskeystorename);
p12fis = new FileInputStream(p12keystorefile);
jksfis = new FileInputStream(jksfile);

final KeyStore kspkcs12 = KeyStore.getInstance(“pkcs12″);
final KeyStore ksjks = KeyStore.getInstance(“jks”);

kspkcs12.load(p12fis, p12keystorepass.toCharArray());
ksjks.load(jksfis, jkskeystorepass.toCharArray());

final Enumeration en = kspkcs12.aliases();
int n = 0;
while (en.hasMoreElements()) {
String strAlias = (String) en.nextElement();
logger.info(“Alias ” + n++ + “: ” + strAlias);
if (kspkcs12.isKeyEntry(strAlias)) {
logger.info(“Adding key for alias ” + strAlias);
final Key key = kspkcs12.getKey(strAlias, p12keystorepass.toCharArray());
final Certificate[] chain = kspkcs12.getCertificateChain(strAlias);

if(p12keyalias != null){
logger.warn(“keyalias option set will overide ” + strAlias + ” with ” + p12keyalias);
strAlias = p12keyalias;
}
ksjks.setKeyEntry(strAlias, key, jkskeystorepass.toCharArray(), chain);
}
if(n>1){
logger.fatal(“Only expecting one alias entry for key pair”);
}
}
logger.info(“JKS keystore count: ” + ksjks.size());
jksfos = new FileOutputStream(jksfile);
logger.info(“JKS keystore commit to: ” + jksfile.getAbsolutePath());
ksjks.store(jksfos, jkskeystorepass.toCharArray());

} catch (UnrecoverableKeyException uke) {
logger.warn(“UnrecoverableKeyException”, uke);
} catch (IOException ioe) {
logger.warn(“IOException”, ioe);
} catch (NoSuchAlgorithmException nsae) {
logger.warn(“NoSuchAlgorithmException”, nsae);
} catch (CertificateException ce) {
logger.warn(“CertificateException”, ce);
} catch (KeyStoreException kse) {
logger.warn(“KeyStoreException”, kse);
} finally {
try {
jksfos.close();
} catch (Exception e) {
}…

Test and configure oracle databases

pauldavidgilligan — Thu, 02 Jul 2009 19:21:34 +0000

Introduction

In some situations when you have many Oracle databases to maintain and you do not want to use the Oracle Enterprise Manager (OEM) for this due to complexity, costs, etc. you often find that Oracle DBA’s write their own scripts.

The testing of the Oracle databases is thus a simple matter of writing a script, using perl, python, java, etc. to connect to a database and then run some test SQL.

The results of these tests are also sometimes written to another non-production database to record history.

What is however sometimes missed by DBA’s is that you can also automate the configuration of various oracle database files such as oratab, tnsnames ans init.ora files where required.

A Configuration Table

You may have for example created a table like the one below:

CREATE TABLE “MONITOR”.”TARGETS”
(
“SID” VARCHAR2 (9) NOT NULL,
“TYPE” CHAR (1) NOT NULL,
“HOSTNAME” VARCHAR2 (64) NOT NULL,
“PORT” NUMBER NOT NULL,
“TNS” VARCHAR2 (255),
“STATUS” VARCHAR2 (12),
“BUILD” CHAR (1),
“REGION” CHAR (2)
) TABLESPACE “TOOLS”;

[don't forget primary key]

In this example the SID (or service name these days), HOSTNAME, PORT are used to connect to the database for testing.

The Process

The test and configuration process – here is the trick.

We use the table data (SID, HOSTNAME, PORT) to generate a TNSNAMES.ORA dynamically and store it external to the script in a simple file.
The environment for the script, for example $TNS_ADMIN, points to this generated file so that the oracle client used can connect to the target database for testing.
Call a PL/SQL routine to update the TNS column in the TARGETS table with the required oracle r
The the script can connect to the target database defined in the TARGETS table above. The script can thus loop through the records in the TARGETS table and connect to all the required databases.
Interesting point. If you add/change or remove records in the TARGETS table and then commit the changes the next time the script runs you have a NEW set of targets to process via the changed TNSNAMES.ORA that is generated each time the script runs.
You can of cause generate other oracle configuration files too.

PL/SQL Part Example

/**
* build, builds setup data
*/
function build return varchar2 is

v_cnt number := 0;
v_tns targets.tns%TYPE;

cursor ct is select sid, hostname, port
from targets
for update of tns, status;
begin
DBMS_APPLICATION_INFO.SET_MODULE(
module_name => ‘build’,
action_name => ‘build pounce application data’);

dbms_utility.analyze_schema(‘MONITOR’, ‘COMPUTE’);

for rec in ct loop
v_tns := ‘(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=’
|| rec.hostname || ‘)(PORT=’
|| rec.port || ‘)))(CONNECT_DATA=(SID=’
|| rec.sid || ‘)(SERVER=DEDICATED)))’;
update targets set tns = v_tns, build = BUILD_NEW
where sid = rec.sid;
end loop;
commit;

select count(*) into v_cnt from targets;
return v_cnt;
end build;

Python Part Example

#######################
# define writetnsnames
#
def writetnsnames(logging, orcl, ENV):
BEHEER_TXT=ENV['BEHEER_TXT']
tnsfilename = os.path.normcase(BEHEER_TXT+”/tnsnames.ora”)
tnsfilenamebck = os.path.normcase(BEHEER_TXT+”/tnsnames.ora.bck”)
shutil.copy(tnsfilename, tnsfilenamebck)
f = open(tnsfilename, ‘w’)
f.writelines(“# \n”)
f.writelines(“# tnsnames.ora generated by pounce script on “+time.ctime()+”\n”)
f.writelines(“# \n”)
rows = alltargets(orcl)
for row in rows:
(sid, tns) = (row[0], row[4])
f.writelines(sid+”=”+tns+”\n”)

f.close()
logging.info(“Generated new tnsnames.ora ” + tnsfilename)
return tnsfilename

Resultant File

#
# tnsnames.ora generated by pounce script on Wed May 30 11:05:43 2007
#
A1ARBAT=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.64.110.142)(PORT=1721)))(CONNECT_DATA=(SID=A1ARBAT)(SERVER=DEDICATED)))
A1ARBIE=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.64.110.142)(PORT=1721)))(CONNECT_DATA=(SID=A1ARBIE)(SERVER=DEDICATED)))

HTML Results

Publish results and configurations in XML -> ~HTML too:

Summary Results in HTML