Tuesday, December 17, 2013

Locking in SQL Server with Practical Examples

Locking in SQL Server is not physical. Understanding such logical concepts takes a lot of time and baffles you a lot. I'm one of such victims. I'll try to make your work easy by writing what I got from my effort.

Locking is necessary to avail protection on the objects (table). It is applied when a particular operation is performed. These locks are applied at different levels in the database engine.

Have an important point in your mind. As I said, Locking is a logical concept. It can be understood only if you use Transactions. Otherwise the locking mechanism gets completed in a fraction of second such that you blink your eye slower than that. One more point is, you can observe Locking well if your table has a lot of rows (1 lakh or more).

For our learning purpose, let's create a table and insert 100000 records into it:

CREATE TABLE mySampleTable
(
          ID1 int, 
          ID2 int,
          SomeData varchar(100)
)

INSERT INTO mySampleTable (ID1,ID2,SomeData)
SELECT TOP 100000 ROW_NUMBER() OVER (ORDER BY o1.name),
ROW_NUMBER() OVER (ORDER BY o2.name),
o2.name
FROM sys.all_objects o1
CROSS JOIN sys.all_objects o2

Thus a table is created and 100000 records are inserted into it.

Create an index on one of the columns now. Let it be a Clustered or Non-Clustered. For now, I'm creating a Non-Clustered Index.

CREATE NONCLUSTERED INDEX NON_IX ON mySampleTable(ID1 ASC)

Creating an index demonstrates Object and Page Level Locking

SQL Server applies locks at 3 levels mainly

                        1. Row Level
                        2. Page Level
                        3. Object Level (Table Lock)

All the above levels of locks are applied at a time depending on the transaction. I'll try to show them with examples of different types of locks.

Shared Lock (S) / Intent Shared Lock (IS)


When a transaction tries to read data from a table, a Shared Lock is applied. When some transactions acquired shared locks on a page, no other transaction can get an Exclusive Lock (X) on the page unless all the existing shared locks are released. But Shared Lock gets released as soon as the data is read.  

We created an index above. When query predicate contains Indexed Column, Shared Lock (S) is applied at Row Level while Intent Shared Lock (IS) is applied at Object and Page levels. If the query predicate contains non-indexed column then no intent locks are applied. Only a Shared Lock (S) is applied.

Shared Lock cannot be observed though you read the data through a transaction because the lock is released as soon as the data is read without depending on the completion of the transaction. In such case, you need to use a table hint to observe the shared lock.

Let me clear your confusion by showing a practical.

Now, Run the following query in a Query Window

BEGIN TRAN
      SELECT * FROM mySampleTable WITH (HOLDLOCK)
      WHERE ID1 = 2549801   --- (Index is created on column ID1)

You will get data. Don't end the transaction with a COMMIT or ROLLBACK.
Open a new Query Window and run the following query

SELECT resource_type as type, request_status as status,
request_mode as mode, request_session_id as spid,
resource_description as resource, resource_database_id as dbid
FROM sys.dm_tran_locks
WHERE resource_database_id = 6;

(NOTE: Please make a note that the query predicate resource_database_id differs from database to database. My database's id is 6. To know the id of your database use SELECT DB_ID('<your-database-name>') and enter the result in the query predicate)

Now see the result pane. You can observe that there is a Shared Lock (S) applied at the Row Level and an Intent Shared Lock (IS) is placed at Page Level and Object Level. 


Intent Shared Lock (IS) on Object and Page means, on this particular Page and Object, other locks like Intent Exclusive (IX) or one more Intent Shared (IS) can be placed. In the above situation, IS on Page and Object, S on Row are applied because in the transaction we searched data based on the column on which Index is created. When an index is created it divides the table into pages and accommodates some number of rows per page. It applies Intent Locks at Object and Page levels, normal locks at row levels.

If you search based on non-indexed columns, then no page and row level locks are applied. Only a lock at the Object Level is applied. Here's a small example of that.

Run the following query:

BEGIN TRAN
             SELECT * FROM mySampleTable WITH (HOLDLOCK)
             WHERE ID2=1

Note that the query predicate contains non-indexed column in the above query. If you look at the lock applied by above query then


Here a normal Shared Lock (S) is applied at Object level as we searched based on the non-indexed column.

One more thing about Shared Lock is, it can allow other transactions to acquire a Shared Lock on the same rows and pages to read the data. It doesn't allow an Exclusive Lock (X) which tries to update the data. An Exclusive Lock (X) is allowed only after all the shared locks are released. Let's see an example of this situation.

Exclusive Lock (X) / Intent Exclusive Lock (IX)


This lock is like a possessive wife. It doesn't allow any other locks to be applied on the page simultaneously. Shared Lock (S) allows another shared lock but only one Exclusive Lock(X) can be applied at one time. This is applied when DML operations are performed on the table.

To apply this, there is no need to use any table hint like we used for Shared Lock (S). Let's have a small demonstration on Exclusive Lock (X):

Run the following query in a query window:

BEGIN TRAN
       UPDATE mySampleTable SET ID2=3 WHERE ID1 = 2549801 --- (Index is created on  ID1)

Don't commit or rollback this, open a new query window and run the following query:

SELECT resource_type as type, request_status as status,
request_mode as mode, request_session_id as spid,
resource_description as resource, resource_database_id as dbid
FROM sys.dm_tran_locks
WHERE resource_database_id = 6;


You can observe that Intent Exclusive Locks (IX) are applied at Object and Page levels and an Exclusive Lock (X) is applied at Row Level. As I said earlier, Intent locks allow other intent locks but normal locks do not. Now, open a new query window and run the following query:

SELECT * FROM mySampleTable WHERE ID1 = 2549801

You can see that this query keeps executing but no result comes out. Let's see what's happening now. Run the following query:

SELECT resource_type as type, request_status as status,
request_mode as mode, request_session_id as spid,
resource_description as resource, resource_database_id as dbid
FROM sys.dm_tran_locks
WHERE resource_database_id = 6;


You can observe what I said. At the Page and Object levels, there are two different locks applied. One is Intent Shared (IS) and another Intent Exclusive (IX). But at the row level, there is a Shared Lock (S) with status WAIT. This means existing Exclusive Lock (X) at Row Level is not allowing Shared Lock (S) to read the same row. The S is waiting for X to be released. To release that, end the first update transaction by COMMIT or ROLLBACK. Then you can read the data from the other transaction.

To make Shared Lock (S) not to wait, you can use table hint (NOLOCK) but this just gets the data from table. It doesn't guarantee that it is the correct data because the user who initiated update transaction hasn't ended it. He may COMMIT it or ROLLBACK it. If he rolls back then you may get wrong data. This concept is called DIRTY READS.

That's all about Exclusive Lock (X).

Update Lock (U) / Intent Update Lock (IU)


Update Lock (U) is considered as a child of above two locks. It is difficult to show practically. It is applied by a transaction before performing a DML operation.

Suppose a transaction needs to update data in a table as follows:

BEGIN TRAN
            UPDATE mySampleTable SET ID2=1 WHERE ID1=2549801

The Query Optimizer, before updating the row, it has to read the row which needs to be updates. Here in the above transaction, firstly the row with ID1=2549801 has to be read and then Update should be done. So while reading that row, an Update Lock (U) is applied. After reading, existing Update Lock is converted to Exclusive Lock (X) and updating is done. So, we can't notice when an Update Lock (U) is applied. But we can force the optimizer to hold update lock as follows:

BEGIN TRAN
       SELECT * FROM mySampleTable WITH (UPDLOCK) WHERE ID1 = 2549801

Run the following query again

SELECT resource_type as type, request_status as status,
request_mode as mode, request_session_id as spid,
resource_description as resource, resource_database_id as dbid
FROM sys.dm_tran_locks
WHERE resource_database_id = 6;


See that there are both (IX) and (IU) applied, one at Object level and the other at Page level respectively and a (U) at Row Level. If the same transaction tries to update data then an (X) is applied at Row level and update is done. Update Lock (U) needs to be escalated to Exclusive Lock (X) always.

The Update Lock (U) is used mainly when transaction isolation level is set to either REPEATABLE READ or SERIALIZABLE. At those levels, Update Locks are applied to avoid Deadlock problems.

Shared with Intent Exclusive Lock (SIX)


Shared with Exclusive Lock (SIX) is the combination of Shared and Exclusive Locks can be applied if a single transaction tries to read all the data as well as modify some data. It is applied at Object Level.

Take a transaction as:

BEGIN TRAN
         SELECT * FROM mySampleTable WITH (HOLDLOCK)
         UPDATE mySampleTable SET ID2=98146 WHERE ID1=53

Now in another Query Window, run the following query:

SELECT resource_type as type, request_status as status,
request_mode as mode, request_session_id as spid,
resource_description as resource, resource_database_id as dbid
FROM sys.dm_tran_locks
WHERE resource_database_id = 6;

Now see the result pane, 



You can see that an SIX is applied at Object Level while an IX and an X are applied at Page and Row levels respectively. The SIX doesn't allow any other transaction to read or modify the data.

Now run the following query in a new Query Window.

SELECT * FROM mySampleTable

The query keeps executing without displaying any results.

To get theoretical knowledge types of Locks, read official documentation of Microsoft


Sunday, December 15, 2013

Locking in SQL Server

Locking in SQL Server is not physical. Understanding such logical concepts takes a lot of time and baffles you a lot. I'm one of such victims. I'll try to make your work easy by writing what I got from my effort.

Locking is necessary to avail protection on the objects (table). It is applied when a particular operation is performed. These locks are applied at different levels in the database engine.

The following are some main types of locks applied by SQL Server frequently,


                                                    1. Shared Lock (S)
                                                    2. Exclusive Lock (X)
                                                    3. Update Lock (U)
                                                    4. Intent Shared Lock (IS)
                                                    5. Intent Exclusive Lock (IX)
                                                    6. Intent Update Lock (IU)
                                                    7. Shared with Intent Exclusive Lock (SIX)

These locks are applied automatically by SQL Server depending upon the type of columns that are being modified and the query that is running.

SQL Server applies locks at 3 levels mainly

                                                    1. Row Level
                                                    2. Page Level
                                                    3. Object Level (Table Lock)

To get information about the levels of locks, we need to know about the type of locks in advance. Now I'll go theoretically about the basics of Locking. I'll show practically about them in my coming posts. But just keep an important point in mind, Locking is a logical concept which cannot be observed like normal features. So, to see Locking, Transactions must be used.


Shared Lock (S)


As the name says, Shared Lock (S) is a lock which can be shared by any number of transactions. It is used for just reading the data from a table. Suppose a transaction TRAN1 reads some data from a table then it applies a Shared Lock (S) on it. While TRAN1 is reading the data, another transaction TRAN2 can also read the same data by sharing the lock applied by TRAN1. Any number of transactions can read the same data by sharing the same lock.

Shared Lock (S) is applied at different levels depending on the query predicate used in the transaction. If the query predicate contains Indexed Column (Clustered or Non-Clustered or any other) then Shared Lock (S) is applied at Row Level while Intent Shared Lock (IS) is placed at Object and Page Levels. I'll write about Intent Locks in coming paragraphs of this article. Suppose the query predicate contains non-indexed column and transaction reads the data then Shared Lock (S) is applied at Object Level.

A Shared Lock (S) allows only shared locks of other transactions to be applied on the same row and it doesn't allow any other locks like Exclusive or Update.

Shared Lock (S) is released as soon as the data is read. It doesn't depend on the ending of transaction. Though the transaction is yet to be committed or rolled back it is released. So other locks like Exclusive (X) can be applied at this time.


Exclusive Lock (X)


Exclusive Lock (X) acts as a possessive wife. Shared Lock (S) allows other shared locks to be applied but Exclusive Locks (X) do not allow even another Exclusive Lock (X) to be applied. It is applied when a transaction tries to modify data in a table.

Suppose a transaction TRAN1 tries to modify the data keeping the indexed column in the query predicate then it applies an Exclusive Lock (X) at the row level and Intent Exclusive Lock (IX) at Object and Page Levels. At this situation no other transaction can apply neither an Exclusive Lock (X) nor a Shared Lock (S). The applied Exclusive Lock (X) is released only when the transaction TRAN1 is ended with a COMMIT or a ROLLBACK. If any other transaction tries to apply an Exclusive Lock (X) then it has to wait until TRAN1 is completed.

Exclusive Lock (X) causes deadlock between the transactions which results in the performance degradation in the server.

Update Lock (U)


Update Lock (U) can be taken as a mixture of above both. It is used to avoid deadlock situation. It is applied when a transaction executes UPDATE statement.

Suppose a transaction TRAN1 is trying to update data as follows:


BEGIN TRAN
      UPDATE mySampleTable SET ID2=104 WHERE ID1=100

Now before updating ID2, the query optimizer has to read the row where ID1 = 100. While reading the row, it applies an Update Lock stating that this row needs to be updated. After reading this Update Lock (U) is converted into an Exclusive Lock (X) which states that no other transaction can do anything on the row. While reading the row, Update Lock (U) allows other transactions to apply a Shared Lock (S) for reading the data. All this operation is done in minute fraction of a second. So, you can't see specifically in result pane that an update lock is applied unless you force the optimizer to keep lock. Even this happens with SELECT statement.

Update Lock (U) cannot be held by optimizer. It needs to be escalated to an Exclusive Lock (X).

Intent Shared Lock (IS)


Intent Locks are different from above locks. These are applied only when query predicate contains Indexed Column. When an indexed column is contained in the query predicate, Intent Locks are applied at Object and Page levels.

Intent Shared Lock (IS) functionality is same as Shared Lock (S) but the difference is, it allows other Intent Locks like Intent Exclusive (IX) or Intent Update Lock (IU) to be applied on same Object and Page though it is still in hold. But an Intent Lock is compatible with another Intent Lock only. If you try to apply normal lock on the object which Intent Lock is holding, it won't allow until it is released.

Intent Exclusive Lock (IX)


Intent Exclusive Lock (IX) shares the same functionality as Exclusive Lock (X) with a difference that it is applied at Object and Page levels. It also allows other locks to be applied on the same Object and Page by other transactions. It is compatible with only other Intent Locks but not with any normal lock.

Intent Update Lock (IU)


Intent Update Lock (IU) shares the same functionality as Update Lock (U). It is applied at Object and Page levels. It is compatible with intent locks only. If another transaction tries to apply one more Intent Update Lock (IU) then it allows that transaction to apply on the same page and object.

Shared with Intent Exclusive Lock (SIX)


Shared with Intent Exclusive Lock (SIX) is placed at Object Level. For SIX to be applied, a single transaction should perform read as well as update operations on a table. 

Suppose TRAN1 reads some rows from a table, updates some rows of it then an SIX is placed at Object Level, IX is placed at Page Level and an X is placed at Row Level. SIX allows only Intent Locks at Object and Page levels but doesn't allow normal locks like other Intent Locks.

_________________________________________________________________________________

                        This is the theoretical explanation about Locking which I understood. I'll try to explain Locking concepts with examples in my coming posts. Follow the official documentation by Microsoft on Locking.

Sunday, December 1, 2013

NOT IN vs NULL in SQL Server

Recently, I was given a small challenge regarding NOT IN and NULL operators in SQL Server. I was able to crack it after doing some research online. I'd like to share the challenge and its solutions.

I was given two tables with their definitions and INSERT statements for it. It went as follows:


CREATE TABLE table1
(
      firstname VARCHAR(50),
      lastname VARCHAR(50)
);

CREATE TABLE table2
(
      address1 VARCHAR(50),
      address2 VARCHAR(50),
      zip VARCHAR(50),
      name VARCHAR(50)
);

INSERT INTO table1
VALUES ( 'f1', 'l1' ),
              ( 'f2', 'l2' ),
              ( 'f3', 'l3' ),
              ( 'f4', 'l4' ),
              ( 'f5', 'l5' ),
              ( 'f6', 'l6' ),
              ( 'f7', 'l7' );

INSERT INTO table2
VALUES ( 'add1', 'add2', '12300-12', 'f2' ),
              ( 'add3', 'add4', '12300-14', 'f4' ),
              ( 'add5', 'add6', '12300-20', 'f1' ),
              ( 'add7', 'add8', '12330', NULL );

Now I was asked to execute the following query after creating the tables and inserting values into them.

SELECT t1.firstname, t1.lastname
FROM table1 AS t1
WHERE t1.firstname NOT IN ( SELECT t2.name FROM table2 AS t2 );

It returned zero records.



I was asked to explain why zero records are returned and what can be the remedy for this. Then I went reading about NOT IN because on seeing the query I doubted there is something wrong with it. I told a solution which I got.

Remove NOT in the query and then execute, the query goes as follows:

SELECT t1.firstname, t1.lastname
FROM table1 AS t1
WHERE t1.firstname  IN ( SELECT t2.name FROM table2 AS t2 );



Now I don't get zero records. This is because the operator IN performs a kind of MATCHING operation. It is used mainly when sub-querying is used. There are two queries in the above SQL statement. The first statement "SELECT t1.firstname, t1.lastname
FROM table1 AS t1 WHERE t1.firstname" gets all the records of columns firstname and lastname from table1 and the second query ( SELECT t2.name FROM table2 AS t2 ) gets all the records of column name from table2. Now the IN operator residing between those two queries performs a matching operation between the values returned by them and returns the records that are matched. It means the values which are common in in the columns (firstname column in table1 and name column in table2) are returned.

But the person who gave me this challenge was not satisfied. He wanted the result using NOT IN the query. Then I read about the purpose of NOT IN, analyzed why zero records are returned and found one more solution.

The reason for getting zero records is, there is a NULL value in the column "name" of "table2". NOT IN is the negation of IN operator. If IN returns the values that are matched then NOT IN returns the values that are not matched. It means we should the result with records that are not matching in both the tables. Also, I found that NOT IN operator performs a kind of AND operation internally. As per basics of statements, AND returns true if both the input values are true otherwise false. It requires the input a TRUE or a FALSE. But here AND operation fails due to NULL value because NULL is neither TRUE nor FALSE and violates the condition of AND. So NOT IN fails and zero records are returned.

Now the question is "Can we get some records using this NOT IN operator and keeping the NULL value as it is?"

Yes, we can get the records even by having NOT IN in the query. It is possible by using an expression COALESCE. By using this expression, the given query goes as follows:

SELECT t1.firstname, t1.lastname
FROM table1 AS t1
WHERE t1.firstname NOT IN ( SELECT COALESCE(t2.name,' ') FROM table2 AS t2 );

Run this query and you can find some records in the Result Pane. 


Let me tell the purpose of COALESCE expression. It performs a kind of OR operation. As per basics of statements, OR returns TRUE values unless both the inputs are FALSE. Here you can see that I changed the query. After "t2.name" I specified a ' ' which is the indication of empty string in SQL. This was done to effect the COALESCE expression.

The basic function of COALESCE as per Microsoft's Official Document, 

"Evaluates the arguments in order and returns the current value of the first expression that initially does not evaluate to NULL".

It means it ignores the NULL values and returns the values that are NOT NULL. Here it ignores the NULL value of the column "name" of table2 and returns the values that are not equal to NULL. I also found that COLAESCE is equivalent to LEFT OUTER JOIN. It combines all the values of the expression including NULL into a single value and returns the first non-NULL values.

Now the NOT IN operator compares the values returned by both the queries of that SQL statement and returns the values that are not matched.

Finally, to conquer its enemy NULL, NOT IN needed a friend COLAESCE.