POST data to any endpoint via REST API

curl -v -X POST -u "user:password" --header "Content-Type: application/json" 'http://<host>:<port>/<endpoint>' -d @test.json

You can change the Content-Type based on your required type and supported by endpoint.

Type application 

================

application/javascript

application/octet-stream

application/ogg

application/pdf

application/xhtml+xml

application/x-shockwave-flash

application/json

application/ld+json

application/xml

application/zip

Type Audio

==========

audio/mpeg

audio/x-ms-wma

audio/vnd.rn-realaudio

audio/x-wav

Type image

==========

image/gif

image/jpeg

image/png

image/tiff

image/vnd.microsoft.icon

image/x-icon

image/vnd.djvu

image/svg+xml

Encode Decode error in python : UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0

Error desc:

Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
                    ordinal not in range(128)

This type of error comes when some special character is not in range of selected encoding type.

Solution :

Find the default encoding type of python.
             python -c 'import sys; print(sys.getdefaultencoding())'
             o/p : ascii

We need to change the encoding type and try with different type of encoding. e.g.

latin-1
utf-8
utf-16-be


Create a file sitecustomize.py under this location : /usr/lib/python2.7/site-packages/sitecustomize.py

echo "import sys; sys.setdefaultencoding('latin-1')" > sitecustomize.py


Compile the python script and check the encoding type is changed by below command.
python -m py_compile sitecustomize.py

PYTHONPATH=".:$PYTHONPATH" python -c 'import sys; print(sys.getdefaultencoding())'

o/p : latin-1

How to configure postgres database in SqlDeveloper

Download postgressql*.jar and configure it in SqlDeveloper using steps below.

1. Open Sqldeveloper 
2. Click on Tools tab and select Preferences. 
3. Under database tab select ThirdParty JDBC driver and add a new Entry with downloaded postgres jar.

Connect to database from Jenkins Active Choice parameter to retrieve data

We can connect to database and get the data to show in Jenkins job as a parameter.


import groovy.sql.Sql
import java.sql.*

def output = []

def sql = Sql.newInstance("jdbc:oracle:thin:@<host>:<port>:<instance>","<user>","<password>","oracle.jdbc.driver.OracleDriver");
  sql.eachRow("SELECT EMP_ID FROM SCHEMA.EMPLOYEE WHERE EMP_NAME='TEST' AND POSITION LIKE '%Manager'"){ row ->
            output.push(row[0])
  }
 
return output

Make sure you have driver jar present in Jenkins classpath.
Or add the ojdbc7.jar under /usr/java/packages/lib/ext and restart Jenkins service

Auto Refresh specific tab in browser and specific area of Page by Javascript

I had one requirement to auto refresh any page after specif interval and that should show only that part of the page which content I want to see.

Below is the Java script code which has to be added as a bookmark and click that bookmark after opening the page and scroll down to specific part you want to see after each refresh.



javascript:
refresh_interval=prompt("Set Refresh Interval in Seconds [s]");
current_page=location.href;
if(refresh_interval>0) {
  selectedFrame='<frameset cols=\'*\'>\n<frame id="refreshframe" src=\''+current_page+'\'/>';
  selectedFrame+='</frameset>';
  with(document) {
    write(selectedFrame);
    void(close())
  };
  setTimeout('reload()',1000*refresh_interval);
} else {
  location.replace(current_page);
}
function reload() {
  setTimeout('reload()',1000*refresh_interval);
  var frameToSet = document.getElementById('refreshframe');
  frameToSet.contentWindow.location.reload(false);
}

How to get port number of a process by pid in Unix

Get the pid by ps -ef | grep <processName>

Get the port in which this process is listening by below command.

netstat -lanpt | grep LISTEN | grep <pid>

Clean %temp% in windows by batch file

Create a batch file with below content and double click it to clean temp folder in windows machine.

@echo off
cd %temp%
for /d %%D in (*) do rd /s /q "%%D"
del /f /q *

How to generate random port number

Below is the code to generate the random port number based on the port range configuration in machine level . It will generate a port by calculating ranges define and check if this port is being used in that machine. If port is not in use then it will return that number. This is useful for creating docker containers where you want to generate some random port to expose.

#!/bin/ksh

read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range
while :
do
       PORT="`shuf -i $LOWERPORT-$UPPERPORT -n 1`"
        ss -lpn | grep -q ":$PORT " || break
done
echo "Your port is : $PORT"

Get certificate of any website and add it to keystore

Below is the command to export certificate of any website and import it to local.
Default password for importing is "changeit". To import certificate in Unix it may need root permission as generally java installed with root.

Export :
----------
openssl s_client -connect <host>:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > certName.crt


Import :
---------
$JAVA_HOME/bin/keytool -import -alias <alias> -keystore $JAVA_HOME/jre/lib/security/cacerts -file <pathToCert>/certName.crt

How to find all hyperlinks present in a webpage by Unix script

For e.g. If you want to get all the group id present in maven central repository.

https://repo1.maven.org/maven2/log4j/


#!/usr/bin/bash
URL="https://repo1.maven.org/maven2/log4j/"

wget -O - $URL | \
  grep -o '<a href=['"'"'"][^"'"'"']*['"'"'"]' | \
  sed -e 's/^<a href=["'"'"']//' -e 's/["'"'"']$//' >> hyperlink.lst


It will print all the hyperlink present in the webpage .

Output:

https://repo1.maven.org/maven2/log4j/apache-log4j-extras/
https://repo1.maven.org/maven2/log4j/log4j/

Download a directory by wget in UNIX

Download entire directory from Nexus by wget command:

 wget -r -nH --cut-dirs=6 --no-parent --reject="index.html*"  --user=<USER> --ask-password  http://<host>:<port>/nexus/content/repositories/Test-Releases/com/edu/test/

For e.g.  In Nexus there is a folder test and inside that there are different folders with version number.

/test/1.0
/test/1.1

Below are arguments of wget commands.

-r  : Recursive
-nH : Ignore the hostname. If this parameter is not passed then it will create the folder with name of nexus host and port

--cut-dirs : Ignore the number of directories post Nexus host and port. If 5 is passed in this case then in local directories will be like edu/test/*

--reject="index.html*" : Download the actual content but not the index.html

--ask-password  : It will ask for password after running command

Call shell script from Java code

Below is the code to call the shell script from Java code. This will call the shell script and wait for it's execution and print the output.

public static void main(String[] args) throws Exception {
        try {
                String param1=args[0];
               
                String scriptName = "/root/script.ksh";
                String commands[] = new String[]{scriptName,param1};

                Runtime rt = Runtime.getRuntime();
                Process proc = rt.exec(commands);
                proc.waitFor();
                StringBuffer output = new StringBuffer();
                BufferedReader reader = new BufferedReader(new       InputStreamReader(proc.getInputStream()));
                String line = "";                      
                while ((line = reader.readLine())!= null) {
                        output.append(line + "\n");
                }
                System.out.println("===> " + output);
        } catch (Exception e) {
                e.printStackTrace();
        }

Clone/Checkout specific file/folder from GIT

By default git checkouts the entire repository to local. As it used to compress the data during storage and checkout so it takes very less time. But if you want to checkout only one file then use below command.

git archive --remote=ssh://git@host:port/project_key/repos/repoName.git HEAD <fileName/FolderName> | tar -x

Maven command to download and deploy to Nexus from command line

This will deploy any artifacts from command line without pom.xml. Here repositoryId will be the id defined in settings.xml.

mvn deploy:deploy-file -Dfile=file_FullPath
-DgroupId=*** -DartifactId=*** -Dversion=***
-Dpackaging=jar   -Durl={repo_url} -DrepositoryId=releases


This command will download the artifacts from Nexus via command line.

mvn -C org.apache.maven.plugins:maven-dependency-plugin:2.4:get
-DremoteRepositories=http://<host:port>/nexus/content/repositories/Test/
-DgroupId=com.edu.test -DartifactId=testArtifact-Dversion=1.0 -Dpackaging=jar

rsync with compress option

This is the fastest way to copy data from one server to another.It will compress the data and transfer .

1) rsync -avxH ~/source_dir/  username@remote_host:destination_directory

This will sync data under source_dir to destination_directory

2) rsync -avxH ~/source_dir  username@remote_host:destination_directory

This will create a directory source_dir under destination_directory and copy data under source_dir


In case of local server

rsync -avxH /source /destination

Run jar file in Unix machine

java -classpath "/devuser/test/test/bismaya/TestPOC/poc.jar:/devuser/test/test/bismaya/TestPOC/ojdbc7.jar" com.edu.test.main.Main

While creating the jar make sure you have selected Main as the class containing main method. Add all the jars to classpath by separating :.

Or

java -classpath `pwd` com.edu.test.main.Main.

In this case all the jars should be present in the current dir.

Or

java -jar jarName.jar --> provided there is no dependency jar is required.

Unix commads : Find command example

find . -not -path '*/\.*' -type d \( -iname "$version_id" \) ! -newermt `date +%y%m%d -d "30 days ago"` | tee -a "$log_file"

find . -not -path '*/\.*' -type d \( -iname "$version_id" \) - Find the directory not present in hidden directories

! -newermt `date +%y%m%d -d "30 days ago"`  - Older than 30 days

find . ! -newermt 2016-02-22 ! -type d -delete - Delete the older before the specified date

find . -type l -ls - Find all links

find . -not -path '*/\.*' -type d \( -iname "$version_id" \) ! -newermt `date +%y%m%d -d "63 days ago"` | xargs  rm -rf -- To delete

Upload and Download artifacts from Nexus by cmd line

Download command
-------------------------
mvn -C org.apache.maven.plugins:maven-dependency-plugin:2.4:get
-DremoteRepositories={URL of the repository}
-DgroupId={groupID} -DartifactId={artifactId} -Dversion={version} -Dpackaging=jar -DupdateReleaseInfo=true


Upload command
----------------------
mvn deploy:deploy-file -Dfile=file_FullPath
-DgroupId=*** -DartifactId=*** -Dversion=***
-Dpackaging=jar   -Durl={repo_url} -DrepositoryId={id from setting.xml}

It will download the respective maven plugins to the M2_REPO and then it will upload the artifacts specified. There should be maven installed in the machine where you are trying to upload/download.


Setting.xml
--------------

 <localRepository>C:\M2_REPO</localRepository>

<servers>
  <server>
      <id>releases</id>
      <username>deployment</username>
      <password>deployment123</password>
    </server>
  <server>
        <id>snapshots</id>
        <username>deployment</username>
        <password>deployment123</password>
  </server>



<mirrors>
    <mirror>
      <id>nexus</id>
      <mirrorOf>external:*</mirrorOf>
      <url>http://indlin714:28081/nexus</url>
    </mirror>


<profile>
      <id>nexus</id>
      <!--Enable snapshots for the built in central repo to direct -->
      <!--all requests to nexus via the mirror -->
      <activation>
              <activeByDefault>true</activeByDefault>
         </activation>    
<repositories>
        <repository>
          <id>releases</id>
          <url>http://host:port/nexus/content/repositories/testRepository</url>
          <releases><enabled>true</enabled></releases>
        </repository>
     
  <repository>
         <id>snapshots</id>
          <url>http://host:port/nexus/content/repositories/isr-fnd-public</url>
          <snapshots><enabled>true</enabled></snapshots>
         </repository>

</repositories>
     <pluginRepositories>
        <pluginRepository>
          <id>central</id>
          <url>http://central</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>true</enabled></snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>


<activeProfiles>
    <!--make the profile active all the time -->
    <activeProfile>nexus</activeProfile>
  </activeProfiles>

Create/update Jar file in Unix without affecting the Manifest.MF

Update a jar file :  jar -uf  <jar_name>  <path_to_file/folder to be updated>.
 The <Path_To_File> should be same as the path inside the jar file .

Update jar without affecting existing Manifest : 

jar  -cmf  META\-INF/MANIFEST.MF  <Jar_File_Name>  <Files_To_Be_Packed>

Create Package by Maven

Create the assembly.xml and call it from pom.xml.

assembly.xml
-----------------

<?xml version="1.0" encoding="UTF-8"?>
<assembly xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/assembly-1.0.0.xsd">
<id>assembly</id>
  <formats>
    <format>jar</format>
  </formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>

<fileSet>
<outputDirectory>product/template-topologies</outputDirectory>
<directory>${env.HOME}/workspace/src/main/java/topologies</directory>
<includes>
<include>*.topology</include>
<include>**/*.*.topology</include>
</includes>
<fileMode>744</fileMode>
</fileSet>

 </fileSets>

 </assembly>



Main Pom.xml
------------------

    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.5.5</version>
        <dependencies>
          <dependency>
            <groupId>your.group.id</groupId>
            <artifactId>my-assembly-descriptor</artifactId>
            <version>1.0-SNAPSHOT</version>
          </dependency>
        </dependencies>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
            <configuration>
              <!-- This is where we use our shared assembly descriptor -->
              <descriptorRefs>
                <descriptorRef>assembly.xml</descriptorRef>
              </descriptorRefs>
            </configuration>
          </execution>
        </executions>
      </plugin>