Wednesday, November 26, 2014

When the Problem is Not Them. It’s You.



Hello Dear Reader!  I’ve been the Data Platform Management Lead at Pragmatic Works for almost two years now.  It’s been an interesting journey that I’m enjoying.  I’m the tier 3 support in a lot of cases, play the role of manager, mentor, and fellow fighter in the trenches when required.  Today I wanted to write about something that deals with professional development.

In IT facts are facts, data is data, things perform well or they do not.  All of this leads to trust.  I do a lot of presentations on the Internals of SQL Server.  It is a fun and topic I enjoy it. *cough* nerd alert *cough*.  I do this not just for fun, but because knowledge is powerful.  It gives me the ability to let my clients have confidence that I understand what I’m talking about.  Very rare are any instances where I have been paid when I actually needed to flip hex to decimal and look through the binary that was produced.  The ability to display that knowledge when needed is crucial to ensuring you’ve got a second shot when you need it.

Projects, jobs, and opportunities come and go.  I’ve seen cases of spectacular success.  As anyone who has experienced success well knows, the path is littered with those who have failed before you.  Some projects are easy, some are hard.  All can be successful, but you have to have the right mindset.  When projects do fail it can be for a variety of reasons, typically it comes down to poor communication. 

I could be the typical marketing magazine and now list out ways projects fail, how communication is important, but Dear Reader you deserve better.  What you really want to know is how you fix it.  

IS IT TOO LATE FOR A RETRY
Unless all trust is gone it’s not too late.  When you are walking out the door, or being asked to walk out the door it is too late.  As long as you are still on the team, you should take that as validation to be part of the process.  In business everyone wants to be successful.  Everyone wants to win.  If you can help with that, then it is never too late.

The Upgrade Part 1
I was with a billion dollar company and we were upgrading their SQL Server boxes for their main application that the entire business ran off of from SQL Server 2000 to 2008 R2.  We had a very well thought out project plan, an itemized list and timeline of what needed to occur, and over 100 people including a team of VP’s that were on hand to monitor the event.  Leading up to it we hit a pretty big snag.

We were migrating the users from SQL Server 2000 to 2008 R2 and an odd thing happened.  The logins stopped working.  We caught it in Staging.  There was another consulting company that was also working with my client.  Often you will work with consultants or contractors for many different organizations, and you have solid and good relationships.  This was not one of those cases.  

The competing company took this opportunity to state very loudly that I didn’t know what I was doing.  That we could not just go from SQL 2000 to 2008 R2.  The hash had changed for logins, and you had to stutter step to SQL Server 2005, then re-export the logins to SQL 2008 R2.  This added an increased level of complexity, and threatened to blow our timeline.  I left the office dejected and the client questioning my experience.
You don't want to do IT.
You want to go home and rethink your life.

The EDW Part 1

I was with a client in the financial industry.  The mission to help pull data from a database, not SQL Server, to a new Enterprise Data Warehouse (EDW).  The plan would have us pull from our Online Transaction Processing (OLTP) system to move the data over.  They already had an Online Analytical Processing (OLAP) DW.  The data was Extracted, Transformed, and Loaded (ETL) from the OLTP system to the OLAP system.  The columns where not the same, they didn’t match up fully.  This was using a 3rd party product so we didn’t have full visibility into the data mappings.

The business trusted the OLAP system.  Even though the OLTP was the source, they wanted us to validate our imported data against the OLAP, not the business rules and calculations provided to us using the raw OLTP data.

I didn’t agree with this approach.  A source system of this type that had pure 100% financial data, that financial decisions were made off of daily, should be the trusted source.  We should not be replicating the OLAP data but finding that the business calculations produced, validating them off of the source data and moving that data.  After many meetings on this subject I found myself in a room with the entire project team.  I realized that I was the only one passionately arguing this point of view.  In fact I found everyone was unified in one direction.  That direction was that I was wrong.

FIXING THE ISSUE (YOU)
Once you have realized that you are the issue the next step is to find a way to reestablish trust.  Learning, adapting, and putting the goals of team forward showing you can be a productive member goes a long way.

The Upgrade Part 2
When I started thinking about the issue I knew I wasn’t wrong.  Either that or I had gotten unbelievably lucky for every upgrade from 2000 to 200x that I’ve ever done.  Either way I wanted to know which it was.  If I had been lucky, well let's just say I would have a lot of phone calls I would need to make.

In order to be sure it was time to take a close look at the Microsoft provided scripts for transferring logins for SQL 2000 and guidance for SQL 2005 and up.  What I found was that the hash output was indeed different.  To test this I stood up a SQL 2000 box, made a user and a password and exported them.  I did the same on SQL 2005.  There were differences.  The length was different.  The 2000 scripts generated a password hash that was 102 characters long.  The 2005 and up were only 42 characters long.  Then it struck me those characters matched the SQL 2000 output exactally.

I used the SQL 2000 script to transfer the user to a SQL 2008 R2 instance.  Then I logged on as the user.  It worked.  I deleted the user, and ran the 2005 scripts.  It worked.  Those 42 characters were the important part.  That didn’t solve my issue with the client.  Our passwords were not working.  On a wild hare it struck me that the passwords were all lower case.  I tried camel casing them.  They worked.  I found by default that SQL 2000 passwords were case insensitive.

Poor password management had led them to record the passwords as lower case.  The hash had saved the camel case, and on export had enforced it by default on 2005 and up.  I brought this to my client the next day.  We realized that the passwords they had stored were not always correct.  In order to fix this we had to reset and update every application and it’s password before we could proceed. 

We did, and the upgrade rolled on.

The EDW Part 2

I stood at the front of the room and realized every eye was on me.  I realized that the business firmly believed in the process they had laid out.  I also realized, I was the only one not on board.  This was a definitive moment.  I could stand my ground, but very quickly I could see that would lead me off this project.  Failure is not an option.  So I went for option 3.

The process of not trusting the source is, and this is an understatement, not good.  Even with that staring me down, I could see an opportunity.  If I was onboard with the plan, I could work from the inside to validate that the source data was what we needed.

So I did just that.  I wasn’t excited, but I smiled big.  I wasn’t 100% convinced, but I threw in my hat, hard work, and rolled up my sleeves.  That first day the grin on my face may have been strained, by the end of the next 4 weeks it wouldn’t be at all.

We didn’t have all the business rules or the ETL logic for the 3rd party data warehouse that the business trusted.  The people in the room knew that.  They were committed to getting this right, and putting the puzzle pieces together.  The process would be painstaking, but as I made my change of heart and discussed the validation we would need to do, mapping documents we would need to create, business logic we would need to confirm, validate, and how we would validate the imports it started to come together.

PULLING IT ALL TOGETHER

“So Balls,” you say, “That’s great that you fixed it. How do we fix it?”

Great question Dear Reader.  These stories both had a positive outcome.  They happened because I didn’t give up, and I continued to work to help the team succeed.  I don’t want to paint this as if I’m never wrong.  Just ask Adam Jorgensen (@AJBigdata | Blog), he’ll tell you that I’m wrong quite often.  I’ve also blogged about learning from mistakes, mistakes are important.  Be humble in the way you interact and listen to people.  People who believe they are never wrong, suffer defeat far worse than those of us that recognize it as part of growing.  

 
If I was wrong about the passwords from 2000 to 200x I wanted to know.  I wanted to understand why, and what we could do to fix it.  Had I been wrong, I would have sung it from the mountain tops.  I would have blogged on it and gladly advised people on it.  Instead I found the answer I was looking for and the logic to back it up, I share that just as freely as I would have my failure.

When everyone is fully committed to an idea that you do not agree with instead of assuming they are wrong, consider that you are.  It changed my point of view on the project.  It allowed me to embrace a team that wanted someone to take them on the path towards success and be part of that team.  It also reinforced a very simple fact, I am not always right and I need to listen to others.

In each case you have to want to achieve the end goal, and that has to be more important than being right.

As always Dear Reader, thanks for stopping by!

Thanks,


Brad

Wednesday, November 5, 2014

PASS Summit 2014 Streaming Keynotes

Hello Dear Reader!   It's that time of the year again, time for the PASS Summit and the streaming Keynote.  Here we go!

11/5/2014 10:05 AM
T.K. takes the stage again.  Announces that Azure ML is available for free today, you just need a live ID.  He's got the slide from the beginning up.  They did a really good job of showing an end to end story here.



11/5/2014 9:53 AM
James Phillips takes the stage to talk about Power BI and Power Pivot, he's showing us the visualizations, and how to take the big data that Joseph showed us and consume it as a user.

We are getting a really nice overview of PowerBI on the web.  When the video posts I'll get a link in there, because this is moving to fast for good descriptions.


11/5/2014 9:38 AM
Demo of Stretch table was really awesome.  Not sure how much I can say about this at this point. 
But this could have huge potential.

Sanji comes out to do a demo in-store analytics using an x-box Kinect.  Showing heat map made in power maps.  They can see where people are hanging out, so they can understand shopping behavior in a store.  How do they consume that data quickly.  Using Azure Data factory to show orchestration, and lineage in the cloud. 

They are walking through Azure ML, showing how to use a work flow to process past data to make predictions on future buying patterns.

We now see how this will work on a mobile app. Cortana is on the mic talking, making product recommendations after Sanji check's in.  She recommends beer mugs.  Then she shows him where to find them on the in store map.  Very very cool.

11/5/2014 9:19 AM
Sneak peak of the technology with Mike Zwillig from the SQL Team takes the stage.  He starts with a demo on Pier 1 imports.  He talks about the issues of getting data reporting from a data mart.

Neat demo he asks the audience to go to a website and click on items to buy them.  He then shows real time reporting of what is being purchased using an *writeable nonclustered Columnstore on an IN-MEMORY OLTP TABLE*.  Awesome.



11/5/2014 9:10 AM
Tracey leaves the stage and we move on to Maximizing performance & availability.  T.K. talks about how running SQL DB across all of Microsoft has forced them to think not just as engineers but as DBA's. 

Not sure if I heard this right but T.K. just said they are working on In-Memory ColumnStore.  That would be HUGE.  We see a list of business that are currently using the cloud and on-prem in very clever and very performant ways ensuring millisecond response times.  He discusses that Samsung has achieved x24 performance with In-Memory OLTP.

We now play a video from Dell on how they are using In-Memory OLTP.  They say they have achieved x4, performance.

T.K. is announcing that SQL DB will have significant improvements in T-SQL Compatibility, larger indexes, Parallel queries, extended events, and In-Memory Columnstore for data marts.  Wow.  There it is.  If it is in SQL DB, does that mean we will see this in the box product?


11/5/2014 9:01 AM
Tracey continues his demo showing how they can use JSON files to change the schema of their search pattern, based on marketing promotions. Using the Shard Utilization heat map that shows how to add processing power, monitor usage, how to scale over different regions.  How you can change the SQL DB scale.  This was very cool.

One more thing.  They GA'ed a new feature last week, they are going to take their Pier 1 main database, they show the load on it.  They want to configure geo-replication.  On the map they show all of their deployments.  Tracey is replicating the main database to Asia.  There were three clicks to do that.  Very impressive.  I would like to see what the bandwidth requirements are for that, this may require a further look.



11/5/2014 8:46 AM

T.K. continues, we are adding 40% data in the world every year.  We need to have and form a Data culture.  Three key tenants:

1. Harness new data types
2. Tackle big data
3. Deliver solutions faster

His vision of the system development lifecycle is to transform & analyze, Visualize & decide, and capture & manage.  He puts up a slide showing how the different Microsoft platform stack relate to these items.  We will focus on capture & manage because that is where the data platform stack fits in.  He's stressing that this is an 'and' world not an 'or' world.  They want their platform to tie together like never before.  More talking points, capture a diverse set of data, achieve elastic scale, maximize performance & availability, and simplify things with the cloud.

Here's a couple of highlights from the past year.  Cloud first, Mobile first.  Get used to hearing that.  He is discussing advances in Azure Document DB, Azure HDInsight, Analytics Platform System, and Azure Search.
The ability to scale for 2014 is very impressive.  640 logical cores. 
A partner, Tracey, from Peir 1 imports comes out to display intelligent search using fuzzy logic.  He is demoing searching for Orange Pumpkins.  They are using Azure search and two lines of code. 




11/5/2014 8:35 AM
Tom introduces the Executive VP's of SQL Server T.K. Rengarajan, James Phillips, and Joseph Sirosh to help share the vision they have for the Data Platform stack.  A video shows of the three talking about their backgrounds and area's of expertise.  It's interesting that they are all relatively new to being the EVP's for SQL Server.  Should be interesting to see how this brings about new features, or what their view on expanding the stack is.

A video is now playing of different vendors and early adopters of the Microsoft stack, discus and how they are currently using the stacks.  T.K. Rengarajan takes the stage.  He talks about his ba


ckground and studying under Dr. Dewitt.  He speaks about his family and two daughters, he is invested in WIT because he hopes they have wonderful career in technology.

He discusses about how he loves MAPs, using maps to navigate family vacations.  He discusses using MapQuest ('remember MapQuest, he says').  Then GPS made trips even easier, and this is a natural way to see how data and technology has changed our lives.


11/5/2014 8:25 AM

PASS President Tom Larock starts off on stage.  He welcomes the first timers and points out that we are currently streaming live to over 50 countries.  Tom ask for a round of applause for Microsoft for making the best data platform on the planet.

Tom is giving us an overview of the road so far.  How PASS was started and founded.  From the first
PASS Summit with 1,200 attendees to today with 5,899 attendees.  Since it's inception PASS has produced 1.3 Million hours of training.


Tom asks us to think about where we were five years ago, and where you are now.  He asks you to look at your future in the same way.  "Growth is never by mere chance; it is the result of forces working together."- James Cash Penney.

Tom is addressing the changes PASS has made to have a larger umbrella.  He is asking people to get involved in growing PASS.  Meet new people, learn & grow, volunteer, join a chapter, or talk to the Board of directors.

Tom is discussing some of the events available at the Summit.  The SQL Cat Team, 50% off Microsoft Certifications, Birds of a feather luncheon, the Women in Technology Luncheon, and Speaker Idol.  Tom Thanks the exhibitors and vendors.  There is an Exhibitor Reception tonight form 6 pm to 8 pm.

Summary

Okay Dear Reader, this was a great way to kick off the Summit.  I’m off to finish up some prep before my first presentation today.  See you soon!

Thanks,


Brad