Friday, December 13, 2013

SQL Server - Auditing sessions over time - Code example

So today I had a requirement to provide a quick and dirty way of collecting data on the sessions connected to SQL Server.  Specifically, I had to write a job that collected basic data on the current sessions, and if these sessions had not been previously recorded, record them into a table.  If the session was identical in all ways other than the session ID, then a counter called 'session count' is incremented for that session.

The challenging part here was providing what is basically a mashup of an INSERT and UPDATE statement, sometimes called an UPSERT.  In SQL Server 2008 and later versions, this functionality is provided using the MERGE statement.  My requirement here is to update a table with values if they exist; otherwise, insert them.  

So here's an example of how to use this MERGE statement in the context of my requirement. The example code is below.  You can modify it to suit your purpose - for example, write out to a permanent table, put it in a job, query the table in a different way, monitor requests rather than sessions, etc. or re-write it entirely, or simply use it as an example of MERGE for reference.

There's also a self-join in there which provides the columns on which to match.  This isn't strictly essential but I needed to match on columns then only update the last login time and last request end time if matched.    The outer query simply aggregates the results in the target table to provide a good view of the data.

Note this won't collect data on sessions which are opened and closed between running the collection.  So if you modify this code by e.g. using a permanent table and putting this code into a job (can't use a temp table in this context), run the job frequently.  If you run it e.g. every 15 minutes and a session is created and destroyed between the job execution intervals, you won't capture the data.



 -- ---------------------------------------------------------------------------------------------  
 -- Script to audit sessions on the server, accumulate session counts per distinct group of   
 -- values from sys.dm_exec_sessions, will ignore duplicate sessions (by session_id and unique   
 -- set of values), will increment when a new session is started that has the key values   
 -- matching, will insert when a brand new session is opened. Uses MERGE.  
   
 -- Run inside a stored procedure, or on the query window, multiple times over the course of X   
 -- hours to get a full list of sessions.  
 -- Add filters if necessary on the final SELECT.   
 -- Change #Logins to a permanent table if required.  
   
 -- Works only in SQL Server 2008, R2, 2012 and 2014.  
   
 -- Author: Derek Colley, 13/12/2013  
 -- ---------------------------------------------------------------------------------------------  
   
 -- TRUNCATE TABLE #Logins -- for debug  
   
 IF NOT EXISTS ( SELECT name FROM tempdb.sys.tables WHERE name LIKE ('#Logins%') )  
     CREATE TABLE #Logins (   
         uqid INT IDENTITY(1,1),   
         session_id INT,   
         last_login_time DATETIME,   
         host_name NVARCHAR(128),   
         program_name NVARCHAR(128),   
         client_interface_name NVARCHAR(32),   
         login_name NVARCHAR(128),   
         last_request_end_time DATETIME )  
   
 DECLARE @CurrentSessions TABLE (   
     session_id INT,   
     login_time DATETIME,   
     host_name NVARCHAR(128),   
     program_name NVARCHAR(128),   
     client_interface_name NVARCHAR(32),   
     login_name NVARCHAR(128),   
     last_request_end_time DATETIME )   
   
 INSERT INTO @CurrentSessions  
     SELECT        session_id, login_time, host_name, program_name, client_interface_name,   
                 login_name, last_request_end_time  
     FROM        sys.dm_exec_sessions  
     WHERE        session_id <> @@SPID   
     AND            session_id > 50  
       
 MERGE INTO #Logins   
 USING (   
         SELECT        c.session_id, c.login_time, c.host_name, c.program_name, c.client_interface_name,   
                     c.login_name , c.last_request_end_time  
         FROM        @CurrentSessions c  
         LEFT JOIN    #Logins l   
         ON            c.session_id = l.session_id   
         AND            c.host_name = l.host_name   
         AND            c.program_name = l.program_name   
         AND            c.client_interface_name = l.client_interface_name   
         AND            c.login_name = l.login_name    ) AS CurrentSessions  
     ON        #Logins.session_id = CurrentSessions.session_id   
     AND        #Logins.host_name = CurrentSessions.host_name   
     AND        #Logins.program_name = CurrentSessions.program_name   
     AND        #Logins.client_interface_name = CurrentSessions.client_interface_name   
     AND        #Logins.login_name = CurrentSessions.login_name  
 WHEN MATCHED THEN   
     UPDATE          
     SET            last_login_time = CurrentSessions.login_time,  
                 last_request_end_time = CurrentSessions.last_request_end_time  
 WHEN NOT MATCHED THEN   
     INSERT        (    session_id, last_login_time, host_name, program_name, client_interface_name,   
                     login_name, last_request_end_time )      
     VALUES        (session_id, login_time, host_name, program_name, client_interface_name,   
                 login_name, last_request_end_time );  
   
 SELECT        COUNT(*) [session_count], MAX(last_login_time) [last_login_time], host_name, program_name,   
             client_interface_name, login_name, MAX(last_request_end_time) [last_request_end_time]  
 FROM        #Logins   
 GROUP BY    host_name, program_name, client_interface_name, login_name  
 ORDER BY    COUNT(*) DESC   
   
   
   
        
           
       
   
   




Monday, June 24, 2013

Vendors! Wake up and hire a DBA!


Time to have a rant about third-party vendor databases.  Yes, I'm looking at you Sage, VersionOne, and most especially, YOU, Hybris CRM.

Yes, you.  Mr. 'Let's not use clustered indexes.'  'Let's model our schemata using ORM.'  'Let's call our primary key column 'PK' on almost every table.  Let's not bother naming any of our constraints.  And using stored procedures or functions - that's so 1990s, man.

I mean, why NOT use BIGINT for every single integer column.  Or VARCHAR(255).  No, wait a minute, that's way too efficient.  Let's use NVARCHAR(255) instead.  For everything.

Take a look at this, part of the table definition from a vanilla Hybris CRM installation:


CREATE TABLE [dbo].[addresses](
[hjmpTS] [bigint] NULL,
[TypePkString] [bigint] NOT NULL,
[PK] [bigint] NOT NULL,
[createdTS] [datetime] NOT NULL,
[modifiedTS] [datetime] NULL,
[OwnerPkString] [bigint] NULL,
[aCLTS] [int] NULL,
[propTS] [int] NULL,
[p_dateofbirth] [datetime] NULL,
[p_middlename2] [nvarchar](255) NULL,
[p_streetname] [nvarchar](255) NULL,
[p_contactaddress] [tinyint] NULL,
[titlepk] [bigint] NULL,
[p_phone1] [nvarchar](255) NULL,
[p_remarks] [nvarchar](255) NULL,
[p_firstname] [nvarchar](255) NULL,
[p_phone2] [nvarchar](255) NULL,
[originalpk] [bigint] NULL,
[p_fax] [nvarchar](255) NULL,
[p_shippingaddress] [tinyint] NULL,
[p_streetnumber] [nvarchar](255) NULL,
[p_gender] [bigint] NULL,
[p_url] [nvarchar](255) NULL,
[p_district] [nvarchar](255) NULL,
[countrypk] [bigint] NULL,
[p_lastname] [nvarchar](255) NULL,
-- continues on for a loooooong time...

'p_gender' - BIGINT??? Really?  So there's 2^64 -1, or 18,446,744,073,709,551,615 possible genders, are there?  On what planet?  Or how about 'PK' for the primary key?  In column 3, of all places?  I sincerely hope the user isn't intending to put some long ASP.NET-generated URL in 'p_url', since it's limited to 255 characters.  And 'p_dateofbirth' clearly needs to store time information too, to the millisecond, as that's important when profiling your customers.

I'm dreading the day when we actually start using your hated system in production and your table scans and parameter-sniffed execution plans slam into my servers like a colossal tidal wave of shit.  I can absolutely guarantee that when I ring the vendor to complain I'll get through to some poor sod on a mere handful of Vietnamese Dong per hour telling me 'prease to call back in mornink'.  Believe me when I say I'm beginning to batten down the hatches now, and when 'main screen turn on' I'll be hiding in the server room, watching our monitoring software with my fist in my mouth.

Sage, you're not off the hook.  No matter how hard you insist, a database is not a 'company'.  It's a database.  I admire your overall design, placing your metadata in a separate DB and company / entity-specific information in separate DBs.  I don't appreciate you writing metadata to 'master' and bursting into tears when I remove it.  Nor, frankly, are your table structures much cop either.  It's not hilarious to dress up your SQL Server-related errors in another error wrap, spreading confusion and delay among the various support teams:

'An unexpected error has occurred.  SQL code 208.' (Sage)
=
'Msg 208, Level 16, State 1, Line 1 - Invalid object name ...' (SQL Server)

Why not just return the second message?  The problem will land in the lap of the DBAs anyway, won't it?  Save time!  'An unexpected error has occurred' simply means 'Something went wrong and I (the developer) can't be fucked to write a coherent error message, preferring to let support teams scramble to look up SQL Server error codes in sys.messages until the application dies a death and they switch to something written properly.'

Vendors!  Wake up!  It's time to start reconsidering old-fashioned concepts like 'testing' and 'good design'.  I know we're all about Agile now, but 30 years ago we had good, solid texts like 'The Theory of Relational Databases' (David Maier - free eText here) full of things like ... wait for it ... relations!  Functional dependencies!  Normalisation!  I know, yawn, yawn, why not just let nHibernate create it all for us, yeah, yeah ... But there's good, solid systems out there right now using databases built on these basic principles.  

Vendors, don't give up your day job.  Develop great apps and leave the database administration to the professionals.


The information and views expressed in this article are my own and do not represent the opinion or views of my employer or any third-party. 


Thursday, June 13, 2013

Killer Code - Nested Aggregates using CTEs 

(Or, 'what NOT to do on a production SQL Server box')


Yesterday, I was experimenting with pulling out some figures in SQL Server using windowed functions.  SQL Server 2012 comes with a couple of really neat system functions called LAG and LEAD, and these are useful for selecting values from n rows behind or in front of a particular row, denoted by column.  There's other articles out there that deal with using LAG/LEAD, so I'm not going to cover it here, save to give an example of using this and the requirement I was aiming at.

For reasons of confidentiality I sadly cannot reproduce here the exact code I was using.  However here's a simplified example.  Here's an initial table configuration:

CREATE TABLE dbo.TestData (
DateCreated DATE,
ArbitraryValue NUMERIC(16,2) )

INSERT INTO dbo.TestData
SELECT '2013-06-01', 342.14 UNION ALL
SELECT '2013-06-01', 659.45 UNION ALL
SELECT '2013-06-01', 283.49 UNION ALL
SELECT '2013-06-01', 903.34 UNION ALL
SELECT '2013-06-01', 129.06 UNION ALL
SELECT '2013-06-01', 756.65 UNION ALL
SELECT '2013-06-01', 239.05 UNION ALL
SELECT '2013-06-01', 194.52 UNION ALL
SELECT '2013-06-01', 804.44 UNION ALL
SELECT '2013-06-01', 116.69 


So, what I wanted was to have a third column, produced during a SELECT, which gave me the difference between each ArbitraryValue and the value in the row preceding it.  Using LAG, this was quite easy:

SELECT         DateCreated, ArbitraryValue,
ArbitraryValue - LAG(ArbitraryValue, 1, NULL)
                  OVER ( ORDER BY DateCreated ASC ) [Difference]
FROM dbo.TestData
ORDER BY DateCreated ASC

Next, I decided I wanted a fourth column, to work out the average of the differences - I'm sure there's a mathematical term for this not dissimilar to standard deviation but I'm not a mathematician - and I tried defining a fourth column that averaged the values in all preceding rows using the AVG aggregate, inside a subquery (with DateCreated standing in as a key).  The purpose of this column was to, over the DateCreated column, illustrate the regression to the mean of the differences in the ArbitraryValue column, with the latest value being the 'best' average of the differences available:

SELECT t1.DateCreated, t1.ArbitraryValue,
t1.ArbitraryValue - LAG(t1.ArbitraryValue, 1, NULL) 
          OVER ( ORDER BY t1.DateCreated ASC ) [Difference],
        ( SELECT AVG(t2.ArbitraryValue)
FROM dbo.TestData t2
WHERE t2.DateCreated < t1.DateCreated
        ) [Average_Difference]
FROM         dbo.TestData t1
ORDER BY t1.DateCreated

This works wonderfully - for a small result set.  When I tested this with absolute values in rows, there was no problem.  The execution plan doesn't look bad either - table scans, but then again I haven't defined an index.

So, I modified the query to use aggregates in place of absolute values, as this was the actual requirement.  Let's modify the table definition and content as follows which will illustrate what I mean:

TRUNCATE TABLE dbo.TestData
ALTER TABLE dbo.TestData DROP COLUMN ArbitraryValue
INSERT INTO dbo.TestData ( DateCreated )
SELECT '2013-06-01' UNION ALL SELECT '2013-06-01' 
        UNION ALL SELECT '2013-06-01' UNION ALL 
        SELECT '2013-06-01' UNION ALL SELECT '2013-06-02' 
        UNION ALL SELECT '2013-06-02' UNION ALL 
SELECT '2013-06-03' UNION ALL SELECT '2013-06-03' 
        UNION ALL SELECT '2013-06-03' UNION ALL 
SELECT '2013-06-04' UNION ALL   SELECT '2013-06-05' 
        UNION ALL SELECT '2013-06-05' UNION ALL 
        SELECT '2013-06-05' UNION ALL SELECT '2013-06-06' 
        UNION ALL SELECT '2013-06-06' UNION ALL 
        SELECT '2013-06-06' UNION ALL SELECT '2013-06-06' 
        UNION ALL SELECT '2013-06-07' UNION ALL 
        SELECT '2013-06-07' UNION ALL SELECT '2013-06-08' 
        UNION ALL SELECT '2013-06-08' UNION ALL 
        SELECT '2013-06-08' UNION ALL SELECT '2013-06-09' 
        UNION ALL SELECT '2013-06-09' UNION ALL 
SELECT '2013-06-10' 

So now I have a table with ten distinct values in it, a total of 25 rows.  I now want to modify my SELECT query to give me a count of the number of rows per distinct row (i.e. a row count per distinct day), and calculate the difference, day to day, of these values.  In reality, this example had a direct correlation that a row was inserted into the table for an event E.  I needed to count the events, grouped by day, the differences between these counts, and the average of these differences.  This, unfortunately, didn't work, throwing up an interesting error message:

SELECT    t1.DateCreated, COUNT(t1.DateCreated),
   COUNT(t1.DateCreated) 
           - LAG(COUNT(t1.DateCreated), 1, NULL) 
           OVER ( ORDER BY t1.DateCreated ) [Difference],
(  SELECT  AVG(COUNT(t2.DateCreated))
   FROM    dbo.TestData t2
   WHERE   t2.DateCreated < t1.DateCreated 
        )  [Average_Difference]
FROM    dbo.TestData t1
GROUP BY   t1.DateCreated 


Msg 130, Level 15, State 1, Line 3
Cannot perform an aggregate function on an expression containing an aggregate or a subquery.


This is due to the SELECT AVG(COUNT(t2.DateCreated)).  Fair enough.  SQL Server won't allow this as by itself, this isn't a column - it's a statement, and pointless since AVG(COUNT(t2.DateCreated)) == COUNT(t2.DateCreated) for a single value.  However, not only have I defined a WHERE clause, I've included it as an extra column using the subquery.  While logically this should be fine, it's still a case of 'computer says no'.

I decided to get around this restriction by writing the query as follows, using a CTE for the outer query in place of the subquery, meaning I didn't have to calculate the AVG and COUNT in the same statement:

;WITH t2 (DateCreated, DateCount, Difference) AS (
SELECT t1.DateCreated, COUNT(*),
COUNT(*) - LAG(COUNT(*), 1, NULL) 
        OVER ( ORDER BY t1.DateCreated ) [Difference]
        FROM dbo.TestData t1
GROUP BY t1.DateCreated )
SELECT t3.DateCreated, t3.DateCount, t3.Difference, 
( SELECT AVG(t2.Difference) 
FROM t2 
WHERE t2.DateCreated <= t3.DateCreated 
        )  [Average_Difference]
FROM         t2 t3
ORDER BY t3.DateCreated ASC

For a small result set, this worked beautifully, returning the average difference in the fourth column.  The problem came when I tried to scale it up.  Result sets for sub-10,000 rows take a few seconds at most.  Running this query on a set of 320,000 rows (specified using a WHERE clause which I've omitted from the example above, for clarity) took around 10sec.  Scaling it up a bit, I ran this query on 10,000,000 rows (about a year's worth of data) to get the results back in about 1m 30sec.  And finally, I attempted to run this for three years of data, estimated about 30m rows.

Oh dear. 

Immediate, total devastation.  A quick look at Spotlight showed all 16 cores going flat-out at 100% CPU.  Alarms started going off.  A quick check of IOSphere, the FusionIO software, showed I had managed to achieve what 256 threads of SQLIO at full-tilt could not - complete saturation of FusionIO read-write capability (and bear in mind this can cope with about 24Gbps, around 500k IOPS read, 500k IOPS write - see here for more, http://www.fusionio.com/products/iodrive2-duo/).  I stopped the query immediately, and went for the autopsy.

The execution plan showed massive activity with scalar computations with the nested 'call' to t2.Difference causing the first COUNT(*) (in line 3) and the second COUNT(*) (in line 3) to be executed for each and every row returned.  This, combined with the simple arithmetic in line 3, combined with the cost of working with the CTE, combined with the subquery AVG calculation, created a massive amount of set-based load that smashed into the CPU schedulers like an iron fist.  

I would love to reproduce the execution plans here but shouldn't due to NDA-related reasons.  I've got my three years of average differences by using Excel to generate the fourth column instead of SQL.  And I'm sure (and if anyone's reading this, they may post) there's fifty different ways of refactoring this query.  My next read is going to be Itzik Ben-Gan's 'Microsoft SQL Server 2012 High Performance T-SQL Using Window Functions', so next time I can do it properly.

Lessons learned - don't test on production, no matter how tempting.  And don't nest aggregates.

Wednesday, May 15, 2013

From 0 to sysadmin in 30 seconds...

This is a quick tip for anyone put on the spot like I was today...

Picture the scene.  It's five to five, I'm packing up my laptop and finding my headphones.  I'm about to leave the office.  In a hurry, one of our engineers comes over and explains how the on-site supplier is having a problem accessing an obscure, unsupported SQL Server database.  They can't get access.

So I go over to have a look.  It's a SQL Server 2005 DB on the application server.  I've never seen it before.  The engineer explains how they have a username, but the password is lost.  SQL Server Management Studio isn't installed.  No-one knows how to retrieve or reset the password for the user they know of.

All faces turn to you.  What do you do?

In summary:

Start -> Run -> cmd
sqlcmd -Slocalhost -E
(if you're lucky, you'll get in, as 2005 has local admins as sysadmins by default).
If this doesn't work - sqlcmd -Slocalhost -E -A

You're in.  If you're not, get local admin access on the machine first.  Now...

create login 'me' with password 'Pass1234';
go
exec sp_addsrvrolemember @loginame = 'me', @rolename = 'sysadmin';
go
exit
sqlcmd -Slocalhost -Ume -PPass1234

You're now sysadmin.

alter login 'someLogin' with password = 'new password';
(this is the username they've supplied.  Create a new password).
go
exit

Job done.  Coat on, go home.

Friday, November 2, 2012

Keeping an Eye on Server Side Traces in SQL Server


Just a quick snippet, thought I'd share a method of keeping an eye on server-side traces.

When you've set up your SST and it's running successfully, you will often use something like:

SELECT * FROM sys.traces WHERE id = [your trace id];

This renders a whole load of useful information - id, path, stop time, etc.  But there's a couple of noticeable features which could be handy.  Sometimes I'd like to know a) how long the trace has left to run and b) how large the file size will get.

We can work out these additional two parameters with some simple mathematics.

So, first - how long left.  Well, we need to work out the difference between the current time and the planned end time, and display the difference.  Nice and easy.

[SELECT...] DATEDIFF ( minute, GETDATE(), stop_time ) [minutes_remaining]

The second requirement is a little trickier.  We want to know how large the file size will get.  We can work this out dynamically by measuring a) how much time has passed so far since the trace was started, b) how large the file size is now.  We then divide b) by a) to get an estimation of file growth per minute.  We then multiply by the number of minutes the trace will run for.

Note this method isn't foolproof since for a start the file growth happens in intervals, not continuously, and that file growth won't be linear under a skewed load.  However, it's reasonably accurate.

First, work out how much time has passed in minutes, CAST to FLOAT for greater accuracy.

CAST(( DATEDIFF (minute, start_time, stop_time ) - DATEDIFF ( minute, GETDATE(), stop_time ) AS FLOAT )

Now include the division:

file_position / CAST(( DATEDIFF (minute, start_time, stop_time ) - DATEDIFF ( minute, GETDATE(), stop_time ) AS FLOAT )

This is the estimated growth per minute.

Now multiply by the number of minutes in the trace interval (difference between start and end times):


file_position / CAST(( DATEDIFF (minute, start_time, stop_time ) - DATEDIFF ( minute, GETDATE(), stop_time ) AS FLOAT ) *  DATEDIFF (minute, start_time, stop_time )

Now round off to 2 d.p for cosmetic neatness:

ROUND((file_position / CAST(( DATEDIFF (minute, start_time, stop_time ) - DATEDIFF ( minute, GETDATE(), stop_time ) AS FLOAT ) *  DATEDIFF (minute, start_time, stop_time )),2) 

Now alias it for a column name, combine with our earlier SELECT query for the minutes remaining, and some other useful info from the sys.traces view:

SELECT  id, path, stop_time, file_position, last_event_time, event_count, 
                DATEDIFF ( minute, GETDATE(), stop_time ) [minutes_remaining],
                ROUND((file_position / 
                    CAST(( DATEDIFF minute, start_time, stop_time ) 
                    -  DATEDIFF ( minute, GETDATE(), stop_time )) AS FLOAT )
                    *  DATEDIFF (minute, start_time, stop_time )),2)
                    [forecast_final_size]
FROM    sys.traces
WHERE   id = -- insert your id # here

Which yields a result like the following:



As you can see, this gives us extra, useful information that's easy to refresh with F5 (or put into a scheduled job and output elsewhere, or similar) that can allow us to avoid problems like runaway trace file sizes - and stop us forgetting about traces.




Thursday, October 18, 2012

Scripting CREATE INDEX Statements Automatically From Your Table Indexes


Just thought I'd share a script I wrote recently to get full CREATE INDEX statements (including filters, filegroup information and all the WITH options) from existing indexes you have.  This is useful when duplicating tables, e.g. when you use SELECT * INTO... you'll find you leave your NC indexes behind.  Or if you're dropping some indexes and want to be able to recreate them in a hurry.

When I Googled for this I couldn't find a single satisfactory solution to this problem, so here's mine.  Tested OK, read all the comments and if you want to test it for yourself, you'll find some test code to create sample objects at the end of the code segment.

*  NOTE:  COPY AND PASTE THE CODE BELOW INTO A NOTEPAD WINDOW / SSMS QUERY WINDOW TO RENDER THE LINE BREAKS PROPERLY.  APOLOGIES FOR BLOGGER'S POOR CODE RENDERING SUPPORT :-S



dbo.recreateIndexes ( @schema_name VARCHAR(100) = NULL, @table_name VARCHAR(MAX) = NULL, @sort_in_tempdb BIT = 0, @statistics_norecompute BIT = 0, @drop_existing BIT = 0, @online BIT = 0, @maxdop TINYINT = 1, @data_compression BIT = 0, @data_compression_type VARCHAR(4) = 'NONE' ) AS BEGIN /* Title: recreateIndexes Summary: Procedure to script out full CREATE INDEX statements for non-clustered indexes. Author: Derek Colley, derek@derekcolley.co.uk, blog: http://uksqldba.blogspot.com Date: 18/10/2012 --- Parameters: @schema_name VARCHAR(100) -- the schema name of the table to find indexes for, optional. @table_name VARCHAR(MAX) -- the table name of the table to find indexes for, optional. @sort_in_tempdb BIT = 0 -- defines whether or not to create the index using TEMPDB @statistics_norecompute BIT = 0 -- defines whether to UPDATE STATISTICS after index creation @drop_existing BIT = 0 -- specifies whether to drop an existing index of the same name @online BIT = 0 -- specifies whether to do the operation online (unavailable sub-Enterprise edition) @maxdop TINYINT = 1 -- max degree of parallelism, play with this at your peril @data_compression BIT = 0 -- whether you want data compression on your indexes @data_compression_type VARCHAR(4) = 'NONE' -- valid parameters are 'NONE', 'PAGE', 'ROW'. Set @data_compression = 1 too. Behaviour: Returns full CREATE INDEX statements to console (Messages tab). Limitations: 1) Although some WITH parameters are drawn from sys.indexes, the procedure parameters are set in stone for all indexes. 2) I'm using a cursor :-( 3) Limited to NONCLUSTERED indexes only at present. To be fair your CLUSTERED index should be scripted at CREATE TABLE time. Misc: 1) I have included some commented-out test code you can use to create some indexes to test this procedure on. 2) I've not included functionality to pick a database or (linked) server, simple enough so mod if you like. 3) Use this procedure at your own risk, ALWAYS test on your test/QA BEFORE deploying to production. 4) If you find any errors or can improve upon this code, tell me! E-mail address above. 5) Tested on SQL Server Std Ed 2008 SP2 successfully. */ IF EXISTS (SELECT * FROM [tempdb].[sys].[objects] WHERE [name] LIKE '#indexProperties%') DROP TABLE #indexProperties SELECT SCHEMA_NAME(t.[schema_id]) [schema_name], t.[name] [table_name], i.[name] [index_name], c.[name] [column_name], ic.key_ordinal [column_position], i.[type_desc] [index_type], i.fill_factor [fill_factor], i.is_padded [is_padded], i.is_disabled [is_disabled], i.[allow_row_locks] [allow_row_locks], i.[allow_page_locks] [allow_page_locks], i.has_filter [has_filter], i.filter_definition [filter_definition], ic.is_included_column [is_included], ic.is_descending_key [is_descending_key], i.[ignore_dup_key] [ignore_dup_key], i.data_space_id [data_space_id] INTO #indexProperties FROM sys.indexes i INNER JOIN sys.tables t ON i.[object_id] = t.[object_id] INNER JOIN sys.index_columns ic ON i.object_id = ic.object_id AND i.index_id = ic.index_id INNER JOIN sys.columns c ON t.object_id = c.object_id AND ic.column_id = c.column_id WHERE i.[name] IS NOT NULL AND i.type_desc = 'NONCLUSTERED' AND SCHEMA_NAME(t.[schema_id]) = CASE WHEN @schema_name IS NULL THEN SCHEMA_NAME(t.[schema_id]) ELSE @schema_name END AND t.[name] = CASE WHEN @table_name IS NULL THEN t.[name] ELSE @table_name END ORDER BY i.[name], ic.index_column_id ASC DECLARE @schemaName VARCHAR(MAX), @tableName VARCHAR(MAX), @indexName VARCHAR(MAX) DECLARE @colList VARCHAR(MAX), @includedColList VARCHAR(MAX), @withOptions VARCHAR(MAX) DECLARE @fillFactor INT, @is_padded BIT, @is_disabled BIT, @allow_page_locks BIT, @allow_row_locks BIT DECLARE @is_filtered BIT, @filterDefinition VARCHAR(MAX), @ignoredupkey BIT, @filegroup VARCHAR(100) DECLARE cur_ForEachIndex CURSOR LOCAL FAST_FORWARD FOR SELECT DISTINCT ip.[schema_name], ip.[table_name], ip.[index_name] FROM #indexProperties ip OPEN cur_ForEachIndex FETCH NEXT FROM cur_ForEachIndex INTO @schemaName, @tableName, @indexName WHILE @@FETCH_STATUS = 0 BEGIN -- aggregate the main columns SELECT @colList = ISNULL(@colList,'') + ip.[column_name] + ', ' FROM #indexProperties ip WHERE ip.[schema_name] = @schemaName AND ip.[table_name] = @tableName AND ip.index_name = @indexName AND ip.is_included = 0 ORDER BY ip.column_position ASC -- trim the trailing comma SET @colList = LEFT(@colList,(LEN(@colList) - 1)) -- aggregate the included columns SELECT @includedColList = ISNULL(@includedColList,'') + ip.[column_name] + ', ' FROM #indexProperties ip WHERE ip.[schema_name] = @schemaName AND ip.[table_name] = @tableName AND ip.index_name = @indexName AND ip.is_included = 1 -- trim the trailing comma SET @includedColList = LEFT(@includedColList,(LEN(@includedColList) - 1 )) -- now get special options per index SELECT @fillFactor = ip.fill_factor, @is_padded = ip.is_padded, @is_disabled = ip.is_disabled, @allow_row_locks = ip.[allow_row_locks], @allow_page_locks = ip.[allow_page_locks], @is_filtered = ip.[has_filter], @filterDefinition = ip.[filter_definition], @ignoredupkey = ip.[ignore_dup_key], @filegroup = fg.[name] FROM #indexProperties ip LEFT JOIN sys.filegroups fg ON ip.data_space_id = fg.data_space_id WHERE ip.[schema_name] = @schemaName AND ip.[table_name] = @tableName AND ip.index_name = @indexName -- deliver an output PRINT 'CREATE NONCLUSTERED INDEX [' + @indexName + '] ON [' + @schemaName + '].[' + @tableName + '] (' + @colList + ') ' + CASE WHEN @includedColList IS NOT NULL THEN ' INCLUDE (' + @includedColList + ') ' ELSE '' END + CASE WHEN @is_filtered = 1 THEN ' WHERE ( ' + @filterDefinition + ' ) ' ELSE '' END + ' WITH ( PAD_INDEX = ' + CASE WHEN @is_padded = 1 THEN 'ON,' ELSE 'OFF,' END + ' FILLFACTOR = ' + CASE WHEN @fillfactor = 0 THEN '100' ELSE CAST((ISNULL(CAST(@fillfactor AS INT),100)) AS VARCHAR(3)) END + ', ' + ' IGNORE_DUP_KEY = ' + CASE WHEN @ignoredupkey = 1 THEN 'ON,' ELSE 'OFF,' END + ' ALLOW_ROW_LOCKS = ' + CASE WHEN @allow_row_locks = 1 THEN 'ON,' ELSE 'OFF,' END + ' ALLOW_PAGE_LOCKS = ' + CASE WHEN @allow_page_locks = 1 THEN 'ON,' ELSE 'OFF,' END + ' SORT_IN_TEMPDB = ' + CASE WHEN @sort_in_tempdb = 1 THEN 'ON,' ELSE 'OFF,' END + ' STATISTICS_NORECOMPUTE = ' + CASE WHEN @statistics_norecompute = 1 THEN 'ON,' ELSE 'OFF,' END + ' DROP_EXISTING = ' + CASE WHEN @drop_existing = 1 THEN 'ON,' ELSE 'OFF,' END + ' ONLINE = ' + CASE WHEN @online = 1 THEN 'ON,' ELSE 'OFF,' END + ' MAXDOP = ' + CAST((ISNULL(CAST(@maxdop AS INT),0)) AS VARCHAR(1)) + ', ' + ' DATA_COMPRESSION = ' + CASE WHEN @data_compression = 1 THEN QUOTENAME(@data_compression_type,'''') ELSE 'NONE' END + ' );' -- reset and increment SET @colList = NULL SET @includedColList = NULL FETCH NEXT FROM cur_ForEachIndex INTO @schemaName, @tableName, @indexName END ---- BEGINNING OF TEST CODE --CREATE SCHEMA test --GO --CREATE TABLE test.TableWithIndexes ( -- productId INT PRIMARY KEY NOT NULL, -- productName VARCHAR(100) NOT NULL, -- colour VARCHAR(20) NULL, -- dateAvailable DATE NULL, -- price MONEY NULL, -- discontinued BIT DEFAULT 0 NULL -- ) --INSERT INTO test.TableWithIndexes --VALUES (1, 'Defrabuliser', 'Blue', '2009-04-30', 19.99, 0), -- (15, 'Unbobulator', 'Green', '2012-01-13', 85.00, 0), -- (22, 'Disbibulator', NULL, '2012-11-18', 12.50, 0), -- (89, 'Bishbosher', 'Orange', '2008-05-23', 109.99, 1), -- (101, 'Jambasher', 'Yellow', '2001-03-03', 3.99, 0) ---- create PK/clustered index with padding/fill factor --ALTER TABLE test.TableWithIndexes ADD CONSTRAINT ix_pk_productId PRIMARY KEY CLUSTERED (productId) --WITH (PAD_INDEX = ON, FILLFACTOR=90) ---- create non-clustered non-covering indexes with single column --CREATE INDEX ix_nc_productName ON test.TableWithIndexes (productName) --CREATE INDEX ix_nc_dateAvailable ON test.TableWithIndexes (dateAvailable) ---- create non-clustered non-covering index with multiple columns --CREATE INDEX ix_nc_productName_price ON test.TableWithIndexes (productName, price) ---- create non-clustered covering index with multiple columns and INCLUDEs --CREATE INDEX ix_nc_productId_productName_dateAvailable_iPrice_iDiscontinued ON test.TableWithIndexes (productId, productName, dateAvailable) --INCLUDE ( price, discontinued ) ---- create non-clustered filtered covering index with an INCLUDE --CREATE INDEX ix_nc_productId_iPrice_filtered ON test.TableWithIndexes ( productId ) INCLUDE ( price ) WHERE ( price < 10 ) ---- execute the procedure --EXEC dbo.recreateIndexes ---- now test the indexes work! --DROP INDEX [ix_nc_dateAvailable] ON test.TableWithIndexes --DROP INDEX [ix_nc_productId_iPrice_filtered] ON test.TableWithIndexes --DROP INDEX [ix_nc_productId_productName_dateAvailable_iPrice_iDiscontinued] ON test.TableWithIndexes --DROP INDEX [ix_nc_productName] ON test.TableWithIndexes --DROP INDEX [ix_nc_productName_price] ON test.TableWithIndexes ---- pasted from Messages tab --CREATE NONCLUSTERED INDEX [ix_nc_dateAvailable] ON [test].[TableWithIndexes] (dateAvailable) WITH ( PAD_INDEX = OFF, FILLFACTOR = 100, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, SORT_IN_TEMPDB = OFF, STATISTICS_NORECOMPUTE = OFF, DROP_EXISTING = OFF, ONLINE = OFF, MAXDOP = 1, DATA_COMPRESSION = NONE ); --CREATE NONCLUSTERED INDEX [ix_nc_productId_iPrice_filtered] ON [test].[TableWithIndexes] (productId) INCLUDE (price) WHERE ( ([price]<(10)) ) WITH ( PAD_INDEX = OFF, FILLFACTOR = 100, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, SORT_IN_TEMPDB = OFF, STATISTICS_NORECOMPUTE = OFF, DROP_EXISTING = OFF, ONLINE = OFF, MAXDOP = 1, DATA_COMPRESSION = NONE ); --CREATE NONCLUSTERED INDEX [ix_nc_productId_productName_dateAvailable_iPrice_iDiscontinued] ON [test].[TableWithIndexes] (productId, productName, dateAvailable) INCLUDE (price, discontinued) WITH ( PAD_INDEX = OFF, FILLFACTOR = 100, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, SORT_IN_TEMPDB = OFF, STATISTICS_NORECOMPUTE = OFF, DROP_EXISTING = OFF, ONLINE = OFF, MAXDOP = 1, DATA_COMPRESSION = NONE ); --CREATE NONCLUSTERED INDEX [ix_nc_productName] ON [test].[TableWithIndexes] (productName) WITH ( PAD_INDEX = OFF, FILLFACTOR = 100, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, SORT_IN_TEMPDB = OFF, STATISTICS_NORECOMPUTE = OFF, DROP_EXISTING = OFF, ONLINE = OFF, MAXDOP = 1, DATA_COMPRESSION = NONE ); --CREATE NONCLUSTERED INDEX [ix_nc_productName_price] ON [test].[TableWithIndexes] (productName, price) WITH ( PAD_INDEX = OFF, FILLFACTOR = 100, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, SORT_IN_TEMPDB = OFF, STATISTICS_NORECOMPUTE = OFF, DROP_EXISTING = OFF, ONLINE = OFF, MAXDOP = 1, DATA_COMPRESSION = NONE ); -- END OF TEST CODE END

Monday, September 10, 2012

SQL Saturday #162 - Cambridge, UK


SQL Saturday #162 - Cambridge, UK


After a perilous drive across the foggy moors of the Pennines, and a nice, straight run down the Great North Road, I arrived in Cambridge at 07:45 on Saturday morning and - wow, what a crowd!

Being my first SQL Saturday, I was a bit ignorant of the drill, so waded right in with my SpeedPASS admission and raffle tickets, and went and found some familiar faces.

The sponsors were about doing their bit - Fusion IO had the most notable stall, with a $16,000 PCI-e Flash memory card available to take a look at, and a live demonstration of the speed increase of Fusion IO kit (log flushes measured in GB/sec!).  I'm definitely sold, although at the current high price of Fusion IO kit, my manager might not yet be.  Still, I can see it replacing the SAN, with long-term durability and the capability to fit many more GB/cm3 than currently possible with rotational drives.

Red Gate were out in force too, with an army of red-clad salespeople with leaflets and live demos of their SQL Monitor and SQL Storage Compress technologies.  I'm not the biggest fan of Red Gate products, finding that some are excellent (SQL Prompt, Data Generator, Search in particular) and some are just not right for me - SQL Backup (2 licenses required to restore to a secondary server - nice to find that out at 2am) and SQL Monitor (too simple, not enough depth or breadth).

SQL Sentry provided the best demo for me - the sheer amount of complex information available from one screen that will enable me to make the correct decisions in a disaster scenario (or simply check the health of my databases in a glance) was astounding.  Their software has features that would take me weeks to code up individually.  Note to Red Gate.  I like complexity!  Give me complexity!  The K.I.S.S. principle is NOT for me!

The sessions were great, too.  I started with Neil Hambly's session on Extended Events, which I've read a little about before but had trouble getting to grips with some of the concepts.  Neil explained it well, unfortunately my core environment is 2005 Standard (temporarily) but I look forward to applying some of his tips.  Buck Woody followed Neil's session with a well-delivered and uplifting keynote speech, with a brief history of SQL Server (tip: did you know that the silhouette icon in Outlook is modelled on the mugshot of Bill Gates as a teenager?).

Next up for me was Tobiasz Koprowski, an engaging Polish DBA with a great session on disaster recovery.  He spoke from experience, detailing the steps you need to take in an emergency (be prepared, basically), and including tips on the things you would perhaps never normally think about.  How to open a server rack when you don't have the keys.  Who to get out of bed when your server falls over at 2am.  Why hand warmers are important in a server room. 

I followed this with Niko Neugeberger's talk on inheriting a database for developers.  I was particularly interested in this, since I have also inherited a large number of databases from developers with a limited amount of DBA participation before I joined.  He spoke about improving performance, about checking how tables have been built and maintained, and although geared at developers the talk was very relevant to my work as a DBA, detailing for example the importance of keeping statistics updated on tables and about index fragmentation.

Straight afterwards was Martin Cairns session on Partitioning Design for Performance and Maintainability.  Martin gave an overview of partitioning and explained at length about how to design partition functions and schemas, and how to improve performance by using techniques such as partition elimination (similar to Denali's COLUMNSTORE index using segment elimination) and the different types (horizontal, vertical and filegroup) of partitioning (and when to use them).  What Martin lacked in engagement he made up for with content, as I scribbled down 3 or 4 pages of useful notes during his talk.

Straight after, I delved into the Fusion IO sponsor session, where I found out the difference between Fusion IO cards and SSDs (there IS a difference!) and just how quickly this new technology works.  

After the sponsor session, I had to stop for lunch - information overload.  The Crowne Plaza had provided some bagged lunch, so I took it outside and got chatting to a few people.  

After lunch, another marathon session.  I went for Hugo Kornelis' session on Making Your Queries Fly with COLUMNSTORE Indexes - this was my first introduction to the 2012 index type and I was extremely impressed.  Hugo had set up working demos, with performance gains demonstrated of 75x the speed of ordinary clustered indexes.  This was a very popular session, with the room hitting maximum occupancy.

Back in to Tobiasz Koprowski's licensing session (of relevance to me, since I recently had to navigate the murky waters of re-licensing under the 2012 per-core model) then onto Phil Quinn's session on XML.  Having had a bad experience with the XQuery features of 2005/08 recently (XML queries blackboxing as 'Remote Query' in the execution plan, leading to unacceptable delays during shredding) I was keen to get his views.  Phil went through 4 or 5 methods of using XML effectively in SQL Server, and referenced the very performance problem I had noticed.  Very helpful session with lots of reference material to take away and a big thank you to him for his time afterwards, patiently answering my XML questions.

After Phil, straight into Mark Broadbent's session (he is the SQL Cambs PASS chapter leader / user group organiser) on READPAST and the true atomicity (as per ACID) of SQL Server.  He went through several demos of where SQL Server does NOT treat transactions atomically (i.e. 'all or nothing') and despite some good-natured heckling from the back (Hugo!) he stepped through an innovative method of processing bucketised data using READPAST, as an alternative to READ COMMITTED / SERIALIZABLE-based locks, to ensure clean data without waits.

After this last session was the prize draw, where some lucky soul won both the OCZ card from Fusion IO AND an Amazon Kindle (fix!) and we heard again from Buck Woody and the organisers.  I left with three new books, a ton of other swag (laptop stickers, stress balls, cups, you name it) and the firm resolve to go again - an excellent experience that I'd recommend to anyone.