Thursday, August 6, 2020

Curl examples

1. Usage: Custom headers.

$: curl -v --cookie "_nmid_secure=YOUR_COOKIE" \
-H "n-user-forwarded={\"customerId\":\"\"}" \

2. Usage: Cookies.

$: curl -v --cookie "_nmid_secure=YOUR_COOKIE" \

Tuesday, March 24, 2020

Kubernetes, helm, microk8s, Docker: Tips and tricks

1. Usage: Delete all completed pods by namespace(say namespace is "internal").

$: kubectl delete po --namespace internal $(kubectl get po --namespace internal | grep Completed | awk '{if ($1 !="") print $1}')
2. Usage: Delete all pods by namespace(say namespace is "default").

$: kubectl delete po $(kubectl get po --namespace default | awk '!/NAME/'| awk '{if ($1 !="") print $1}')
3. Usage: Delete all releases from helm except few

$:helm delete --purge $(helm list | awk '!/NAME/' | awk '!/mysql/' | awk '!/redis/' | awk '!/php-myadmin/' | awk '{if ($1 !="") print $1}')
4. Usage: Cleans all dangling images. This is useful for removing intermediate images left over from multiple builds

$:alias docker_clean_images='docker rmi $(docker images -a --filter=dangling=true -q)'
5. Usage: Removing stopped containers

$:alias docker_clean_ps='docker rm $(docker ps --filter=status=exited --filter=status=created -q)'
6. Usage: To remove ALL of your cache, make sure all containers are stopped and removed, So

$:docker kill $(docker ps -q)
$:docker rmi $(docker images -a -q)

Wednesday, January 30, 2019


SI Content Description
1. const sql = connection.format("SELECT * FROM table WHERE foo = ?", ["bar"]);
Build the query before execution
2. "debug-test": "mocha --inspect-brk -t 1500" //Add above line to package.json and run
npm run debug-test
Debug test in local
3. Merge objects with key value, values being integer, sum the values
_ = require('lodash');
//Merge the array and sum values corresponding to same carrier
//Example: x={ abc: 1, abd: 5 }, y= { abd: 2, abe: 5 } =>{abc: 1,abd: 7, abe: 5 }
merged = _.mergeWith({ ...x }, { ...y }, (a, b) => {
    a = a && _.isNumber(parseInt(a)) ? parseInt(a) : 0;
    b = b && _.isNumber(parseInt(b)) ? parseInt(b) : 0;
    return a+b;

Sunday, July 1, 2018

Performance Optimization of Relational Database

Case study based on the experience in performance enhancement of an application done with Qburst Technologies Pvt Ltdhttp://www.qburst.com/

MySql Based Applications - Improve Performance with Application Level Optimization

MySql is one of the data base solution used by most of the small and medium scale organization for their web applications. MySql delivers good performance with well designed table structures an relations. Most web applications deals with mysql performance issues when volume of data grows beyond expectations.
Usually when we design web based applications with mySql in the backend we follow normalized design pattern with look-up tables and foreign key relations. This kind of normalized design ensures good performance and less read write time when the data volume is within expected limits. (ie, practically there is an upper limit on volume of data that can be handled with relational databases and is defined by type of data and nature of data access).
Data volume within limit:
Usually large volumes of data are populated when we deal with real world scenario like daily activities, stock market data, social media data etc which are usually dependent on human interactions. We can predict user behavior only upto a limit and so is the volume of data being populated in such scenarios.

When data volume grows:
If we go for normalized design, we should keep similar kind of data in same place and searching for mutually related data may force us to go for self joins in the heavily populated tables. Self join of heavily populated tables is going to be a big issue in the performance point of view. When data volume grows the normalized structure may not deliver expected performance due to many factors.
One of them is foreign key relations and another is the normalized structure itself. We use foreign keys to keep track of relational data kept in normalized tables, which can cause issues like increased insert/update time caused by the overhead on the database to create and maintain indexes.
There is a cap on volume of data that can be effectively handled by mysql with normalized structure, which force us to think about alternatives like non-relational database like mongo db. An issue we are going to face with non-relational database is handling relation of mutually dependent real world data. There are ways to tackle the situation.
Even though non relational databases promise faster reading, there are issues like poor write speed. Obviously there are techniques to tackle the issues in non relational database, most of them are heavy in terms of cost and/or effort for small and medium scale organizations.
Economic alternatives:
The main question that gains popularity is "Is it possible to extend the practical upper limit on volume of data that can be handled effectively?" and the answer is YES
We can increase the upper limit on volume of data that can be handled with mysql by following some techniques like de-normalization, archiving etc.
MySql can respond to normal select queries better compared to select queries including joins with same or other tables.
De-normalization can help improve read time in many ways. One of them is keeping data in a purpose oriented manner. De-normalization can improve performance of data read by duplicating data to multiple tables that are meant for display purpose, thus avoiding joins.
For example to make display of data faster, we can keep data optimized for display in a separate table that is used for display regardless of duplication. Now one possible issue is to manage consistency of data. In normalized and properly designed databases, consistency of data is assured by database to a large extend. But as we keep duplicate copies of data for making data read faster, the responsibility of keeping the data kept in multiple locations(data duplication) consistent should be taken from database to your application. This approach can increase the coding effort that take care of data handling and update in multiple copies of data.
De-normalization can help in reducing reading time if we include additional tables that can keep collective data. We can do this by creating rules for data display in all possible scenarios and combined display rules can be handled together with less overhead and duplicate data to multiple tables in a way that each one is optimized for its purpose. This can improve performance of data population at front end, at the same time increasing processing overhead to the code.
Real world data usually become in consistent or less accessed as time pass by. For example if an application manages marketing statistics, data of last 6 months or 1 year may be relevant and previous data becomes less relevant or less frequently accessed. The less relevant or less accessed data can be moved from main database to an archiving system, which can be another database or even tapes or disks based on demand for old data. If the user needs previous data, which can be provided on demand.
Observe your data:
There are techniques that can make mysql perform better but depends on structure and behaviour of data. Structured monitoring can give an idea on the root cause of the problem and based on which proper solution can be formed. Log your data access, create metrics of data access and find data access peeks and performance falls. These peeks and falls can explain the real problem/root cause. Solution can be formed based on this real problem.

Based on optimization tasks I worked on a project for Qburst Technologies Pvt Ltd.




Tuesday, November 28, 2017

Postgress Tips and Tricks

1. Usage: Login to remote postgres.

$: psql -h xxx.xxx.com -U postgres -W

2. Usage: Create database.

$: CREATE DATABASE databaseName;
$: sudo adduser databaseName
$: sudo passwd databaseName
$: su - databaseName
$: psql -d template1 -c "ALTER USER databaseName WITH PASSWORD 'newpassword';"

3. Usage: Common command prompt options.

$: \dt //- list tables
$: \l //- list databases
$: \c //- Connect database
$: \q //- dis-connect database
$: DROP TABLE test; //- delete table;
$: TRUNCATE TABLE profile_events; // Drop table
$: ALTER SEQUENCE seq RESTART WITH 1; //- restart sequence after truncate table
$: UPDATE TABLE SET idcolumn=nextval('seq'); //- update sequence of table

4. Note:

PostgreSQL uses sequences to generate auto-numeric values, by default, Phalcon tries to obtain the generated value from the sequence table_field_seq, for example: robots_id_seq, if that sequence has a different name, the getSequenceName() method needs to be implemented:

Sunday, July 23, 2017

Mercurial: Tips and tricks

1. Usage: Completely remove a named branch.

$: hg strip "branch(${BRANCHNAME})"

2. Now re-iterate for all the branches you have, that's it

$:hg pull -u

3. To check if you have any unwanted changes

$:hg outgoing

4. Add, Remove tag

$:hg tag -r newrevisionhash stable
$:hg tag --remove stable

5. Incoming and Outgoing

$:hg incoming
$:hg incoming -p
$:hg incoming -b <branchname>
$:hg incoming -p -b <branchname>

6. Copy files across branches

$:hg update -r to-branch
$:hg revert -r from-branch file
$:hg ci -m 'copied single file from from-branch to to-branch

7. Dirty merge If both branches have made changes to a file that you want to keep, a dirty trick would be to create a merge of the two branches using hg merge, possibly/probably on still another branch, check that in, and then copy a single file between the merge and the to-branch(Reference: Stackoverflow)

$:hg update -r to-branch
$:branch merge-branch
$:hg merge -r from-branch
$:hg ci -m 'temp merge to be discarded"
$:hg update -r to-branch
$:hg revert -r merge-branch single-file
$:hg ci -m 'merged single-file from from-branch to to-branch"
$:hg strip merge-branch

8. To see heads in a particular branch <branch name>

$:hg heads <branch name>

9. To commit as a user <userDisplayName>

$:hg commit -u <userDisplayName>

Monday, November 28, 2016

Mysqli Bind dynamic parametere

Trying to create a function to insert data into a SQL table, using MySQLI. I want to create a generic function, to insert different types of data in different databases.
The issue is at binding the parameters. I can't find a good way to bind them, as I've got multiple variables with values and keys, but they're all in array format, and  bind_param  requires a new variable for each
$sql = 'SELECT id, lastname FROM customers WHERE ' .
  'category = ? AND ' .
  'lastname LIKE ?';

/* Prepare statement */
$stmt = $conn->prepare($sql); if($stmt === false) {
  trigger_error('Wrong SQL: ' . $sql . ' Error: ' . $conn->errno . ' ' . $conn->error, E_USER_ERROR);
$category_id = 1;
$lastname = '%Smith%';
/* Bind parameters. Types: s = string, i = integer, d = double,  b = blob */
$stmt->bind_param('is', $category_id, $lastname);
/* Execute statement */
/* Fetch result to array */
$res = $stmt->get_result();
while($row = $res->fetch_array(MYSQLI_ASSOC)) {
  array_push($a_data, $row);
The problem
$stmt->bind_param() does not accept params array. So, how to bind params, if their number is variable, depending on user input in your application?

A workaround is to use call_user_func_array to pass dynamically the params array.

The solution
In the following code:

  • $conn is the connection object
  • $a_bind_params is the array of the parameters you want to bind
  • $a_param_type is an array with the type of each parameter (Types: s = string, i = integer, d = double, b = blob). This is another disadvantage of MySQLi API. You have to maintain this array some way in your application.
  • With call_user_func_array, array params must be passed by reference. See notes in manual page.

  • The code:

    /* Bind parameters. Types: s = string, i = integer, d = double,  b = blob */
    $a_params = array();
    $param_type = '';
    $n = count($a_param_type);
    for($i = 0; $i < $n; $i++) {
      $param_type .= $a_param_type[$i];
    /* with call_user_func_array, array params must be passed by reference */
    $a_params[] = & $param_type;
    for($i = 0; $i < $n; $i++) {
      /* with call_user_func_array, array params must be passed by reference */
      $a_params[] = & $a_bind_params[$i];
    /* Prepare statement */
    $stmt = $conn->prepare($sql);
    if($stmt === false) {
      trigger_error('Wrong SQL: ' . $sql . ' Error: ' . $conn->errno . ' ' . $conn->error, E_USER_ERROR);
    /* use call_user_func_array, as $stmt->bind_param('s', $param); does not accept params array */
    call_user_func_array(array($stmt, 'bind_param'), $a_params);
    /* Execute statement */
    /* Fetch result to array */
    $res = $stmt->get_result();
    while($row = $res->fetch_array(MYSQLI_ASSOC)) {
      array_push($a_data, $row);