Java Keytool Commands

Java Keytool Commands for Creating and Importing

These commands allow you to generate a new Java Keytool keystore file, create a CSR, and import certificates. Any root or intermediate certificates will need to be imported before importing the primary certificate for your domain.

• Generate a Java keystore and key pair
  keytool -genkey -alias mydomain -keyalg RSA -keystore keystore.jks -keysize 2048
• Generate a certificate signing request (CSR) for an existing Java keystore
  keytool -certreq -alias mydomain -keystore keystore.jks -file mydomain.csr
• Import a root or intermediate CA certificate to an existing Java keystore
  keytool -import -trustcacerts -alias root -file Thawte.crt -keystore keystore.jks
• Import a signed primary certificate to an existing Java keystore
  keytool -import -trustcacerts -alias mydomain -file mydomain.crt -keystore keystore.jks
• Generate a keystore and self-signed certificate (see How to Create a Self Signed Certificate using Java Keytool for more info)
  keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass password -validity 360 -keysize 2048

Java Keytool Commands for Checking

If you need to check the information within a certificate, or Java keystore, use these commands.

• Check a stand-alone certificate
  keytool -printcert -v -file mydomain.crt
• Check which certificates are in a Java keystore
  keytool -list -v -keystore keystore.jks
• Check a particular keystore entry using an alias
  keytool -list -v -keystore keystore.jks -alias mydomain

Other Java Keytool Commands

• Delete a certificate from a Java Keytool keystore
  keytool -delete -alias mydomain -keystore keystore.jks
• Change a Java keystore password
  keytool -storepasswd -new new_storepass -keystore keystore.jks
• Export a certificate from a keystore
  keytool -export -alias mydomain -file mydomain.crt -keystore keystore.jks
• List Trusted CA Certs
  keytool -list -v -keystore $JAVA_HOME/jre/lib/security/cacerts
• Import New CA into Trusted Certs
  keytool -import -trustcacerts -file /path/to/ca/ca.pem -alias CA_ALIAS -keystore $JAVA_HOME/jre/lib/security/cacerts

WebLogic Server: Intermittent Stuck Threads Caused Due to the 11.1.x Oracle JDBC Thin Driver (Doc ID 1083794.1)

The issue you're encountering seems to be related to a known 11.1.x Oracle JDBC Thin Driver issue, please review Note 1083794.1 - WebLogic Server: Intermittent Stuck Threads Caused Due to the 11.1.x Oracle JDBC Thin Driver to download and apply the latest JDBC driver for your version.


1. Login to server dev-xxxxx
2. Take backup of ojdbc6.jar files
3. Copy ojdbc6.jar with new one provided by vendor in weblogic /server/lib location.
4. Restart ms01 and ms02 one leg at a time.

There were reports of timeouts noticed on different interfacing systems and there were Stuck threads in the SOA service.

Note:
that in WLS 10.3.6 and 12.1.1 the drivers are under the $WL_HOME/server/lib directory, but in WLS 12.1.2 and 12.1.3 the drivers are under the $ORACLE_HOME/oracle_common/modules directory

2 Issue:"Invalid or corrupt jarfile ojdbc6.jar"


After downloading ojdbc6.jar per instructions in Document: 1094166.1 How to Upgrade ojdbc6.jar and orai18n.jar Bundled With Weblogic 10.3, the following error is obtained:

"Invalid or corrupt jarfile ojdbc6.jar"

Steps to reproduce the issue:

1. Download ojdbc6.jar from JDBC/UCP Download Page
2. Execute java -jar ojdbc6.jar -version
3. This command shows the error "Invalid or corrupt jarfile ojdbc6.jar"


Application Issue: [Expedited] PROD - Apply Bug fix patch 26442748 in Shared  Service

  1. Download the patch 26442748 from Oracle Support.
  2. Stop Services on Shared service PRD application server on prd-xx & prd-xx.
  3. Apply patch on prd-xx & prd-xxx.
  4. Start Services on Shared service xx  PRD application server on prd-xx & prd-xx
  5. Sanity test from Infrastructure persepctive.

We have received a request from business to provide a fix for heritage admin url issue in PROD.
After  applying latest Security patch set (26519417) in Shared Services environment,one of the heritage admin url stops working.
Vendor has identified as bug and provied a patch 26442748 as fix.
Shared Service consist of multiple applications like xxxxx

Apache 2.4.x installation in Linux

Installation of apache in linux is bit different than windows. You have to install few dependencies first and then you have to compile the source code of apache. I am sharing step by step information of installing apache 2.4.x in Linux.

Download apache 

If you are on internet enabled linux server and wget utility is installed then you can download using below command

$wget http://www-us.apache.org/dist//httpd/httpd-2.4.29.tar.gz

(if wget is not there then you can install it using $yum install wget)

If you don't have internet enabled Linux machine then you can first down load in your local machine (http://www-us.apache.org/dist//httpd/httpd-2.4.29.tar.gz) and then transfer using some ftp/ssh client like filezilla.

Download dependencies 

Similarly use wget or filezilla to transfer below utilities 

$ wget http://www-us.apache.org/dist//apr/apr-1.6.3.tar.gz

$ wget http://www-us.apache.org/dist//apr/apr-util-1.6.1.tar.gz

Installation

Now unzip & untar downloaded apache and both utilities using below commands

Suppose you have copied all above inside /opt/ directory 

Go inside directory (if you are not there)

$ cd /opt

unzip & untar apache

$gunzip httpd-2.4.29.tar.gz  (If gunzip not working then install it using $yum install gunzip)

$tar -xvf httpd-2.4.29.tar

it will create httpd-2.4.29 directory inside /opt, that means apache source files rctracted inside /opt/httpd-2.4.29 now

unzip and untar utilitities 

$gunzip apr-1.6.3.tar.gz

$tar -xvf apr-1.6.3.tar

$gunzip apr-util-1.6.1.tar.gz

$tar -xvf apr-util-1.6.1.tar

Now you will have apr-1.6.3 & apr-util-1.6.1 directory in side /opt

copy both directories inside /opt/httpd-2.4.29/srclib

$cp -r /opt/ apr-1.6.3 /opt/httpd-2.4.29/srclib

$cp -r /opt/apr-util-1.6.1 /opt/httpd-2.4.29/srclib

Inside /opt/httpd-2.4.29/srclib, rename both folders as below

$ mv apr-1.6.2 apr

$ mv apr-util-1.6.0.tar apr-util

Now inside /opt/httpd-2.4.29/srclib run below commands to install dependencies

$yum install apr-util-devel

$yum install apr-devel

$yum install pcre-devel

After successfully installation of above utilities

Go inside base directory

$cd /opt/httpd-2.4.29

Create a directory for apache 

$mkdir /opt/httpd-2.4.29/apache2.4

Now to install apache with enabling ssl run below command 

$./configure --prefix=/opt/httpd-2.4.29/apache2.4 --enable-mods-shared="all" --enable-so --enable-ssl 

To install with enabling ssl modules run below command

$./configure --prefix=/opt/httpd-2.4.29/apache2.4

It will take some some and at last make it's successful and not land up with any error

Now run below two commands

$make

$make install

bingo!! you are done with installation

To start/stop apache 

Go inside /opt/httpd-2.4.29/apache2.4/bin and run below commands

To start 

./apachectl -k start

To stop

./apachectl -k stop

To test just hit server i.p 

http://localhost, you will get a page with message "It Works!" 

Other Important Considerations - 

There are few important configuration files of apache but main configuration file which calls other configuration files and modules if httpd.conf and you will find it inside /opt/httpd-2.4.29/apache2.4/conf, by default apache use 80 port and only root have privileged to start services on port 80 (As in linux only root have privileged to start process on all ports <= 1024). so either you can start apache process from root user otherwise you have change port inside httpd.conf file (which should be any port not in use by any other process in your server and should be > 1024) to start from normal user. Generic practice is to use 8080

Oracle Enterprise Service Bus (Oracle ESB) Installation & Domain Creation

Oracle ESB is part of Oracle SOA Suite which is a framework to develop service based composite applications. Oracle SOA Suite is a Suite of different components like ESB, Oracle BPEL Process Manager, BAM (Business Activity Monitoring), Oracle B2B, Oracle Event Processing etc. Don't think to learn ESB you need to understand all components. Every component itself a big product on it's own. Good to have knowledge specially from developer perspectives but not really a prerequisite to learn ESB. It's comes as a separate installer as well.

If you know the Architecture of Fusion Middleware, then it's very easy for you to install and configure Oracle ESB. If you are not aware about Fusion Middleware Architecture, then read my this blog post "Fusion Middleware Architecture", it will hardly take 10 to 15 minutes to understand.

Oracle ESB is referred as OSB (Oracle Service Bus).

So, to install OSB, you must have to install SOA Suite, and to install SOA Suite you must have to install Database and Weblogic, and have to run rcu (repository creation utility) in database to create necessary schema's.

So, the steps to install OSB are as below - 

Process 1 - 

• Install JDK
• Install Oracle database
• Install Weblogic Server 12C (Using fmw infrastructure installer)
• Install SOA Suite 12c (It required a valid weblogic home also called Oracle Home)
• Execute RCU database scripts (It's bundled with SOA 12c installer)
• Create Weblogic domain by selecting Oracle Service Bus component 
• Once created, start your server and access OSB console

 Process 2 - 

• Install JDK
• Install Oracle database
• Install Weblogic Server 12C (Using fmw infrastructure Installer)
• Install fmw Oracle Service Bus (Download OSB installer separately and it required a valid weblogic home or Oracle Home to install)
• Execute RCU database scripts (It's bundled with installer)
• Create Weblogic domain by selecting Oracle Service Bus component
• Once created, start your server and access OSB console

I will explain according to process 1 however there is no difference apart from in process 2 you will execute only osb installer and in process 1 you will install complete SOA Suite which will install other suite components as explained above including OSB.

Now, till step SOA Suite installation, follow my this blog page - "Install SOA Suite 12C". During domain creation either you can select all components (like BAM, BPEL, B2B, OSB) or only the required one. In my this blog post I will select only OSB component during domain creation. 

To create Domain, Go to $ORACLE_HOME/oracle_common/common/bin and run configuration wizard (config.sh/cmd)

Select Create a new Domain and enter Name & Location

Now select OSB application components. Here you have to select only Oracle Service Bus, and it will select other related components automatically. Since picture is not clear so I am writing all components it will select - 

You have to select "Oracle Service Bus - 12.1.3.0 [osb]", and selecting this template automatically selects the following as dependencies:

1. WebLogic Advanced Web Services for JAX-RPC Extension - 12.1.3 [oracle_common]

2. Oracle Enterprise Manager - 12.1.3.0 [em]

3. Oracle WSM Policy Manager - 12.1.3.0 [oracle_common]

4. Oracle JRF - 12.1.3.0 [oracle_common]

5. WebLogic Coherence Cluster Extension - 12 1.3.0 [wlserver]

6. ODSI XQuery 2004 Components - 12.1.3.0 [oracle_common]

Select application location (select the default one)

 Enter admin user name and password

 Select development or production mode

Here you have to provide schema details you have configure when you ran RCU initially. 

First you have to provide database details with STB schems. click on "Get RCU Configuration" to see your configuration is correct and your STB schema able to fetch all other necessary schema details to configure domain.

It will display fetched schema details here (I have hided unnecessary schema details). You can update password if it's different for all.

 Next it will show you the results if it's able to connect with db using all schemas.

You can skip this one, just select next

I haven't configured anything here so didn't selected anything and just skipped with next. If you wanted to configure different Admin server name, wanted to configure default node manager configuration, add delete managed servers, clusters etc then you can click on respective component and you will get respective screen to configure. 

 On final screen click create

 Next status screen you will see like below and once done click on finish.

Once domain created, Go to your domain/bin directory, start adim server (by startWeblogic.sh/cmd), then osb_server1 managed server (by - startManagedWeblogic.sh/cmd) abd access osb console as below.

Start AdminServer 

./startWeblogic.sh/cmd

Start Managed Server 

./startManagedWeblogic.sh/cmd osb_server1 t3://localhost:7001

Access osb console 

http://localhost:7001/servicebus

Weblogic Server : List the admin accounts

1. create a .py file, suppose admindisplay.py and copy below contents


import os
import sys
username='weblogic'
password='weblogic123'
url='t3://localhost:40510'
domain='JJTU_Domain1'
groupName='Administrators'
connect(username, password, url)
cd('SecurityConfiguration/'+domain+'/DefaultRealm/myrealm/AuthenticationProviders/DefaultAuthenticator')
ListUsersInGroups=cmo.listAllUsersInGroup(groupName,'*',0)
print ListUsersInGroups

Note - Change username, password, host, port and domain name accordingly

2. go to your_domain/bin and run setDomainEnv script

3. execute the admindisplay.py script like below ( see command in red )

--------------------------------------------------------------------------

C:\Oracle\Middleware2\user_projects\domains\JJTU_Domain1\bin>java weblogic.WLST admin.py

( Output )

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://localhost:40510 with userid weblogic ...
Successfully connected to Admin Server 'JJTU_AdminServer' that belongs to domain 'JJTU_Domain1'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

array(java.lang.String,['weblogic', 'administrator'])

Weblogic STUCK & Hogging Threads, How to deal with STUCK & HOGGING Threads.

What is STUCK thread?

How to deal with STUCK thread?

What is HOGGING thread?

How Weblogic determine a threadto declare as Hogging?

What is a STUCK thread?

We know that a STUCK thread is a thread which is processing a request for more than maximum time defined for a thread to complete the request which is default 600 and can be configured from admin console.  Based on different technical circumstances like due to some intermittent issues with network, database, application server etc a STUCK can be release after some time, but most of the time is certain thread or threads declared as STUCK then there would be some problem either temporary or permanent which need some fix.

How to deal with STUCK thread?

As of now there is no way to deal with a STUCK thread like, sometime end users ask if there any way to kill STUCK thread. No, there is no way to deal with STUCK thread.

To deal with STUCK thread –

·                     Take multiple thread dumps immediately.

·                     Review thread dumps or from console (managed server > monitoring > threads) Where it exactly got stuck?

·                     See how many threads got stuck?

·                     If the stuck thread count is increasing or constant?

·                     If constant then if got stuck on same area (application code etc ) or at different places ?

·                     If getting increase then there would be some serious problem and you have to do a quick health check of youapplication server, database and other integrated technologies wherever your application reaching like ldap server for authentication, some other API’s or web services etc, and in parallel review thread dumps for STUCK threads and share same with your developers to analyze quickly.

·                     If you have one, two or few constant STUCK threads and it’s not increasing then you can monitor it for some more time to check if they get clear or not, if not then to clear them you have only option to restart your managed server(s), and its better to restart and clear them before they make further any impact.

  

What is Hogging thread?  

I am sure if you are going to read this post then you must aware about what is hogging thread. Ok, let me define it again in a single line, “I hogging thread is a thread which is taking more than usual time to complete the request and can be declared as STUCK”.

How Weblogic determine a thread to declare as Hogging?

As we know a thread declared as STUCK if it runs over 600 secs (default configuration which you can increase or decrease from admin console).

Now, How Weblogic determines a thread to declare as Hogging? ok, here is the logic which 

I had learn from some of the Oracle internal portal note.

1.             There is an internal WebLogic polar which runs every 2 secs  (by default 2 secs and can be alter)

2.             It checks for the number of requests completed in last two minutes

3.             Then it checkhow much times each took to complete

4.             Then it takes the average time of all completed request (completed in last 2 sec)

5.             Then multiply average time with 7, and the value came consider as “usual time to complete the request”

6.             Now weblogiccheck each current executed thread in last 2 secs and compare with above average time, if for any of the thread it’s above this value then that thread will declare as Hogged thread.

For example –

1.             At a particular moment,  total number of completed requests in last two seconds – 4

2.             Total time took by all 4 requests – 16 secs

3.             Req1 took – 5 secs, Req2 took – 3 secs, Req3 took – 7 secs, Req4 took – 1 sec

4.             Average time = 16/4 = 4 secs

5.             7*4 = 28 secs

6.             Now weblogic check all executed threads to see which taking more than 28 secs, if any then that thread(s) declared as Hogged Thread.

Only the thing you can change with respect to hogging threads configuration is Polar time (Stuck Thread Timer Interval parameter) which is 2 secs by default. You can change this polar value to some different value like 4 secs if you want polar to run in every 4 secs instead of 2 secs.

Logs Rotation

Each WebLogic Server instance writes all messages from its subsystems and applications to a server log file that is located on the local host computer. By default, the server log file is located in the logsdirectory below the server instance root directory; for example, DOMAIN_NAME\servers\SERVER_NAME\logs\SERVER_NAME.log, where DOMAIN_NAME is the name of the directory in which you located the domain and SERVER_NAME is the name of the server.


In addition to writing messages to the server log file, each server instance forwards a subset of its messages to a domain-wide log file.The domain log file provides a central location from which to view the overall status of the domain. The domain log resides in the Administration Server logs directory. The default name and location for the domain log file is DOMAIN_NAME\servers\ADMIN_SERVER_NAME\logs\DOMAIN_NAME.log, where DOMAIN_NAME is the name of the directory in which you located the domain and ADMIN_SERVER_NAME is the name of the Administration Server.




You can rotate log files - 


By size ( default)
By Time

By default, when you start a WebLogic Server instance in development mode, the server automatically renames (rotates) its local server log file as SERVER_NAME.log.n. For the remainder of the server session, log messages accumulate in SERVER_NAME.log until the file grows to a size of 500 kilobytes.

Each time the server log file reaches this size, the server renames the log file and creates a new SERVER_NAME.log to store new messages. By default, the rotated log files are numbered in order of creationfilenamennnnn, where filename is the name configured for the log file. You can configure a server instance to include a time and date stamp in the file name of rotated log files; for example, server-name-%yyyy%-%mm%-%dd%-%hh%-%mm%.log.

By default, when you start a server instance in production mode, the server rotates its server log file whenever the file grows to 5000 kilobytes in size. It does not rotate the local server log file when you start the server.

You can change these default settings for log file rotation. 


For example
you can change the file size at which the server rotates the log file or 
you can configure a server to rotate log files based on a time interval. 
You can also specify the maximum number of rotated files that can accumulate. After the number   of log files reaches this number, subsequent file rotations delete the oldest log file and create a new log file with the latest suffix.


To change log size for rotation or by time -

Login to Administration Console
click on the server
On right hand side click on logging tab
under default selected general tab

update the rotation file size to the size when you want your server rotate log file

Or select By time option under rotation type drop box and enter time in begin rotation time at the you want your server rotate log file

To update for access logs

Login to Administration Console

click on the server

On right hand side click on logging tab

Select HTTP tab and follow the same above instructions.