Data Science: What Creeped Us Out Early On

“Data scientists solve complex data problems through employing deep expertise in some scientific discipline. It is generally expected that data scientists are able to work with various elements of mathematics, statistics and computer science, although expertise in these subjects are not required.[3] However, a data scientist is most likely to be an expert in only one or two of these disciplines and proficient in another two or three. Therefore data science is practiced as a team, where the membership of the team have a variety of expertise.” – Wikipedia Reference

The “information explosion” started around 1941.  (see reference) Believe it or not, the phrase “data scientist” was coined in 1960 by Peter Naur (Wikipedia Peter Naur) as an interchangeable noun with “computer scientist”.  However, the term wasn’t related directly to statistical data management until 1997 when, in an inaugural lecture at H. C. Carver Collegiate Professorship in Statistics at the University of Michigan, Professor Naur defined statistics in science.  The name of his lecture was: “Statistics = Data Science?”

It was a very modest beginning to something which has exploded into a entirely new profession today.

Fast forward:

In 2006 I was 10 years into my career in data management, manipulation, and adoration.  I also had a liberal arts education in political science – clearly unrelated to data science.  I had no idea what data science was, but I knew I loved data.

In that year a colleague of mine and I sat down with a quad processor server and MS SQL (some antiquated version) and a terabyte of data getting moved around every night to figure out what to do with it.  To our knowledge there was no name for what we were doing.  It was really more a pain in the proverbial … well, you know what I mean… for the company we were consulting with.  They knew there was value in that data, but at that point there was no popular concept of what that value was.  At least not in our little corner of the world.

The data came from click events in a few major websites – a LOT of click events – one terabyte a day worth.  My colleague and I were making great strides in ensuring the four processors were working overtime all night every night (one thread per processor running parallel).  One day I was discussing the project with a non-technical relative of mine when she stopped me and said in a very startled tone, “You mean you track everything I click on?  Creepy!”

Yep, I realized at that moment we were headed right for the last chapter of an Orson Wells novel.

Don’t get me wrong.  I still love data.  I still ‘practice” data science as my profession and hopefully I get better every day at it (I should be getting better after almost 20 years in this profession).  However, that realization was really creepy.  We were tracking every click event our customers created on those websites.  What else was being tracked in other organizations?

Today we know that everything we touch, every comment we make, every photo we post somewhere is tracked, analyzed and reported on to some organization.  I think some of the creepiness has been washed out of the concept simply because we’ve come to accept this “Big Brother” technology as an inevitable part of progress. 

The unasked question remains, however:  will anything “creep us out” going forward in this scientific practice?

I can think of a few things if I give it enough energy, but I try not to because like most people in this profession I love the story data tells.  It is not unlike the feeling of getting into a new car every day and smelling that “new car smell”.  It’s something intangible that you cannot define for someone who hasn’t experienced it first hand.  Every day I “get in that new car and inhale”.  It is as beautiful a high today as it was in 2006 when I couldn’t sleep at night wondering what story our click event data would eventually reveal.



Software Project Management: How to eat an elephant

If you’ve ever been through the entire development cycle of a software project (and actually seen the project to fruition in production) you know the madness that can ensue at the very end of a project.  I would suggest that it doesn’t have to be painful at the end like that.  When you have the stress of all eyes on you (which happens when you have an actual single deliverable with an actual deadline and budget), the best way to eat the elephant is one bite at a time. 

The elephant is the complete and utter satisfaction of your customer.

How do you give the customer complete and utter satisfaction during the journey to fruition and once the product is delivered?

***** Information *****

I know.  Scary isn’t it?  Keeping your customer thoroughly informed along the way leads to lots of meetings and wasted time, right?  It doesn’t have to.

The big question is how to disseminate that information to the customer.  I have four high-level suggestions listed here:

1.)  Every project I’ve ever been on has some collaborative software program (these days SaaS) that lets the client see what he/she is given access to during the progress of the project.

Some project managers open the floodgates to all the information in the project (even that sacred communication between developers).  In my opinion, this is disastrous.  Not because you should have anything to hide, but because if your client wanted to manage this project he wouldn’t have hired you in the first place.

I’ve found one of the ideal ways to set up this collaborative space is much like you set up your project management software (whether it is MS Project or any other application).  Define timelines for each task and keep those timelines updated.  Only give your client access to the highest level of those tasks (he really doesn’t care that Bob needs to change a transformation in his ETL process for the database transfer).  Clients want to see those bars moving.  Yes, they might have questions, but I will touch on how to handle those questions below.

2.)  Engage your development team every single week – without exception – from the start.  Require written updates input into the project/task management system with details from each team member on: where they are at on that task, what roadblocks they have to completing a taks, and how long it will be before the task is done. Yes, it’s a pain for developers and team members to do (I’ve been a software developer for almost 20 years now – trust me I know).  However, this rolls up into what is visible for your customer so they remain engaged in seeing those bars move toward completion.

3.)  FOLLOW UP.  Your job as a project manager is to MANAGE.  Don’t slack.  Thinking that taking one week off from following each members progress is deadly.  Ask questions when you don’t understand something and make sure you really understand the answer you are given.  Post follow-up comments your customer can see so they know you are actively managing the team (and subsequently the client’s money and time)

4.)  The final point I would make is to be honest with your clients.  When you are going to miss a deadline or go outside a budget, tell the client up front.  Withholding that information is deadly to your relationship with that client.  Come prepared with an explanation and your plans to resolve the issue.  Anticipate your client will be frustrated, but likely will be comforted by your clear plans.

Most projects can be completed on time and within budget with the right planning and experience.  You will make numerous mistakes along the way, but mistakes lead to wisdom.  Wisdom makes for a great project manager and projects completed to a customer’s satisfaction.


What happens to old software engineers?

“For of All Sad Words of Tongue and Pen, the Saddest are These, It Might Have Been…..” John Greenleaf Whittier

I worship my father.  Well, I worship a Holy Father, but I adore my earthly father.  He’s incredible.  he is 73 years old and one of the smartest and most successful men I know.  Plus, he’s my daddy and I love him with all my heart.

So yesterday we got in a discussion because he’s thinking about retiring from practicing law and it really got the wheels turning in my own head.

I am 48 years old.  If I were a doctor or lawyer or priest or librarian I would be viewed as “in my prime” – wise, knowledgeable, experienced and highly prized for my years in the industry.  On the other hand, what do we think of people over 45 who are “still programming”?

The assumption, of course, is that as you age you should climb the ladder and get away from hands-on development.  I mean, how sharp can you stay after 45?  Let’s face it – it’s hard enough to stay on top of the latest technologies when you are young.

There’s one thing missing though: a clear definition of “young” in 2014.  You see, when I was growing up old people included anyone over 40.  In 1984 when I graduated high school I can remember looking at one of my favorite teachers who, at the time, would have been about 60 and thinking “Holy cow, he won’t be here next year.”  He taught high level math courses – how could he possibly keep up?

25 years later, that teacher retired.

So back to what happens to “old” software engineers?  There’s only one way to keep yourself “young” in this industry: education.  NEVER stop learning.  The way I see it, I can start climbing the ladder of management (again – I’ve done it several times) and get further and further away from the hands-on software development so it won’t matter how sharp I am.  However, I would not be happy.

I can remember talking to my husband (also a software engineer) when we were in our early 30’s about how obsolete “old developers” were and how out of touch they were with technology developments: stuck in time – old time.

Fortunately, God blessed me with a fabulous mind.  It’s no different than most folks – maybe one or two standard deviations above average, but all-in-all a good, solid working mind.  He also blessed me with a passion for programming and learning (albeit in very unorthodox learning methods – usually outside a classroom).

The logic, math, statistical reasoning – it all captivates me.  I’m a business intelligence specialist / architect / developer / designer.  Who knows what it’s called these days.  Now we have official terms for everything that once was just “mystery technical voodoo” we performed magically behind a curtain (or in my case, often tucked away in a server closet working out of sight of the rest of the human race).

So now we have two questions to answer:

  1. What’s “old” in the programming industry?
  2. What happens to “old” software engineers?

Well, to the first question I have to answer there is no such thing as “old” software engineers just because of their chronological age.  There are wise and experienced software engineers, but they aren’t old until they stop learning new technology.  Then they are “old” by choice.

To the second question, I propose we remain software engineers.  I propose we nestle in for the long haul and run like madmen to keep up with the ever-changing world of programming languages and concepts  (I’m told exercise is good and coffee is bad – ok, pots of coffee consumed throughout the night in pursuit of the solution to that last final “bug” fix is bad).

Frankly, those of us who are “wise” programmers kind of  have a huge advantage over young whipper-snappers: we know REAL programming.  We know things like “assembly language”, “DOS programming”, etc.  Ask a newly graduated CS major to build something in assembly and unless he/she attended one of the top universities you are liable to get a very blank stare.

I find that tragic.

I can bounce from Unix/Linux programming to DOS shell scripting to .NET C# to MS BIDS to Java to proprietary BI suites and more!  I believe that’s because I know what’s truly happening under the covers of today’s IDE’s.  The logic remains the same – no matter what we call it this decade.

So what makes me think I can keep this up for a long, long time?  Here are a few examples that life doesn’t even begin until after 45:

Roget Invented the Thesaurus at Age 73

Grandma Moses  Anna Mary Robertson Moses is one of the biggest names in American folk art, and she didn’t even pick up a brush until she was well into her eighth decade.

Grandma Moses was originally a big fan of embroidery, but once her arthritis grew too painful for her to hold a needle, she decided to give painting a try in the mid-1930s.

She was 76 when she cranked out her first canvas, and she lived another 25 years as a painter — long enough to see the canvases she had sold for $3 fetch prices north of $10,000.

Benjamin Franklin At age 70 in 1776 Franklin played an instrumental role in draftingand signing the Declaration of Independence.  At age 81, Franklin signed the Constitution of the United States of America. ·

Winston Churchill FIRST became Prime Minister at 65 ·

Laura Ingalls Wilder STARTED writing the “Little House on the Prairie” series at 65

Edmond Hoyle Whether or not you know it, you probably owe Hoyle a tip of the cap each time you reach for a deck of cards. The Englishman is considered to be the world’s first technical writer on the rules of card games, and he didn’t put pen to paper as a young card sharp. Hoyle was around 70 years old when he first began recording the rules of various card games in 1741; over the last 27 years of his life, his smash hit A Short Treatise on the Game of Whist went through over a dozen editions.

Jack Weil, Age 107 d. Aug 2008 Jack A. Weil passed away on August 13, 2008 at the age of 107. The fact he lived to the ripe old age of 107 is impressive. The fact that he was still the chief executive of the company he founded and working 40+ hours a week until his final days is plain old amazing.

In 1946, he formed Rockmount, a western high fashion clothing retailer that continues to manufacturer it’s shirts in the US after many competitors moved offshore. Achieving historical fame, various accounts state Mr. Weil either invented the modern bolo tie or named it.

His secret to heath, wealth and happiness? “He loved his work.”

Poppy Bridger, Age 84 After working as a PhD chemist for 45 years, Poppy Bridger, retired at the age of 69 to care for her ailing mother. But her 72nd birthday gift was an opportunity to buy and operate the lab she had worked at. With about $250K in savings, back to work she went!

On any given day, you will find Bridger testing the authenticity of a precious heirloom or analyzing the properties of metal fatigue. To help with the growing work load at the lab, she has subsequently hired her son and daughter to work with her.

She goes to work every day, and at the age of 84 is bringing into the business about $350K annually.

 Colonel Harland Sanders, Age 90 d. Dec 1980 The world famous Colonel Sanders launched his business at the age of 65, using his first Social Security check as start up funds. A master of personal branding, Sanders leveraged his honorary “Colonel” title and constantly wore the stereotypical “southern gentleman” white-suit and black tie. The rocket like growth of KFC is now legendary, and prior to his death Colonel Sanders’ restaurant chain had achieved over 6,000 locations with sales of more than $2 billion.

During his entrepreneurial tenure Sanders met with the U.S. Congressional Committee of Aging and spoke against mandatory retirement, highlighting the love for work and the value of wisdom in the work place.

Not a bad run, old chap! Not bad at all.

 Barbara Miller, Age 74
Being an entrepreneur was never really a consideration in Barbara Miller’s life. After quitting her job in the paper industry after 30 years of service, she assumed she was done. But as she packed her stuff, her former colleagues begged her to start a new business… so she did.

In January of 1995, Miller opened the doors to Miller Paper Company and started with $300K in savings and 15 employees. Today the business is generating over $7M in annual revenue and has been on D&B’s list of the nation’s fastest growing companies.

Business has not been a walk in the park, to say the least. Miller started her company and was immediately sued by her former employer. A few months later she struggled with ovarian cancer.

 Sylvia Lieberman, Age 91 Sylvia Lieberman became an entrepreneur in fall 2007 when she was 90. This is when she realized her dream of having her first children’s book published. So why not start a company to author and promote the book?

Archibald”s Swiss Cheese Mountain is an award-winning book about a little mouse with a big heart who teaches children how to reach their big dreams. Not only is she an entrepreneur, but a philanthropic one! A portion of the proceeds goes to two children’s charities.

Despite her age, Sylvia works tirelessly promoting her book at book-signings and readings, TV appearances, radio and print interviews, and even appeared on a float in a parade. And all these efforts increase the amount she donates to charities.

Do you want to be the first person he ever manages?

Ok, so everyone starts somewhere.  However, do you remember the first time you managed a staff (even if just one person?)  I don’t know about you – I was a HORRIBLE manager!  I had no clue what I was doing.  It’s like parenting. One day you are sailing along by yourself just working like crazy on what you love to do and the next day you have this human being dependent on you for his/her sustenance.  No instruction manuals.  No “do overs” when you screw up.

Well, ask yourself this:  If you are offered a job at a company where you’d be the first person ever to work for the hiring manager, would you take it?

Most people don’t think about it before they take a job.  Most people just assume the person hiring them knows how to manage, right?  Have you ever interviewed for a job and asked the hiring person if he/she has ever manage anyone before?

I bring all this up because of a recent experience I’ve had with my own manager being a virgin to that position.  Wow.  What a roller coaster.  I think the situation was exacerbated by his short temper, but it was definitely a volatile environment.  I think I should have earned hazard pay.

The one takeaway I have from this whole experience is to really, really understand how much experience a manager has before you accept a new position.  It’ll definitely make my short list of questions I ask in interviews going forward.

~ Amy


Do you really want to rule the world? Sure, why not.

If you ask any technical manager what his objective is for the day, he will usually have a reasonably normal response.  If you ask my husband he invariably responds with, “The same thing we do every day.  Take over the world, Pinky.”

Now, that’s not to say my husband has aspirations to take over the world.  Ok, maybe he does (he is a programmer, after all), but that’s not my point.  Most people get out of bed every day and think, “I’ll do what I do every day today once again.”  It’s easy.  It’s comfortable.  It’s what has become normal to them.

What if today is the day?  Today you are going to do something so radical everyone is shocked.

I’m not talking radical like Walter White Breaking Bad crystal meth radical.  

I’m talking about changing your whole view on the day the moment you crawl out of bed today.

Stop and ask yourself one thing:  what could I do today that would make one of my employees feel fantastic about themselves and in turn make them thoroughly enjoy working today?

Sound insane?  Really? Then maybe you should rethink your career choice in management.

Yes, it’s easy to find employees doing things “wrong” but why don’t you take today – just one day – and really look for someone doing something super right?

“Word up” (slang for listen) for the day – and in a really good way.  Tonight someone on your team will go home and brag to their spouse about the great day they had and you’ll come home and brag to yours about how productive the team seemed today.




He’s a great programmer so he’ll make a great manager

That statement is akin to saying, “That cat fetches like a dog.  She must be a dog!”  Programming (or software engineering) has absolutely nothing to do with management.  There aren’t even any similarities between the two.

I get it: finding someone to manage technical staff is like finding a needle in a haystack. 

1.)  They have to be “normal” enough to be able to communicate with senior management while they manage a group of highly unusual and self-proclaimed intellectuals.  (Therefore, senior management often mistakes a relatively normal techie for management potential.) 

2.)  They have to be able to communicate with these intellectuals using “tech speak”.  (The worst example of a technical manager is always when senior management assumes someone who is gifted at software engineering makes a great manager of caffeine-addicted geeks in superhero t-shirts.)

Today, we’re going to look at the latter: technical managers chosen to supervise and direct a programming team simply because they rose to the top of the group with ninja technical skills.

First let me be very clear about one thing: I cannot stake claim to being the most intellectually gifted programmer in the room – anywhere.  I’m always learning and always growing (even though my feeble old mind fights me over it these days).  For the most part those of us who are programmers/software engineers/computer geeks chose this profession because:

  • We like working alone
  • We like math and logic
  • We adore caffeine, late night debates on the possibility of black holes and which is better: Star Trek or Star Wars (I lean toward the trekkie side as I’m of an older generation)
  • We think we can do things no one else can do – sort of
  • We have some hidden insecurity about our intellectual prowess

Using that knowledge it is pretty simple to see that most programmers have some deep-seated insecurities which rarely makes for good management potential.

There are, of course, great examples of exceptions to that assumption (besides which, we all know about the trichotomy of assumption, right?)  However, I am willing to stake my reputation on the fact that those examples are few and far between.

What makes a great technical manager, you ask?  There are many features of a great technical manager, but the one most often used to judge capability is really only a fraction of a much larger picture:

  1. The ability to understand technical jargon and translate it for senior management
  2. A genuine concern for those who work for him/her
  3. Understanding what motivates each person on his/her staff – why are those people really working there?  (When it comes to programmers, the answers will always surprise you.  Sure, money is a huge motivator, but what almost always ranks second?  A really cool, cutting edge project to work on.  If you want to demotivate a programmer tell them they have to do maintenance on a project using 10 year old technology.  Nightmare job!)
  4. A willingness to admit his/her own weaknesses, strengths, mistakes and successes (setting an example of humility while maintaining an official managerial demeanor)
  5. Respect for those who work for him/her and their natural individuality (some might see this as a respect for each extreme of geekiness)
  6. Being able to identify and tap into the true strength in each resource on the team while encouraging each person to take on roles that might fall outside their normal comfort zone
  7. Being able to keep the team focused and drive a project through to the end without making everyone on the team insane.  If I had a dollar for every project I saw fail because a team was not cohesive with clear direction I’d be very wealthy.  (One of my favorite interview questions is to ask programmers about projects they finished.  You would be amazed at how many programmers cannot answer that question because they never finish anything.)
  8. Giving constructive feedback.  Yelling, manipulation, and vicissitude have no place in management.
  9. Appreciating the difference between “downtime” and “avoiding work” and being able to identify each
  10. Personal accountability

Clearly I’m not the resident expert on all that makes a great technical manager, but as someone who has been on both sides of the fence I can assure you I know a bad technical manager when I see one.  Usually (but not always) that person was once the “Kid Rock” of programming and rose to the top of the ranks with his or her technical aptitude.  As you can see from the brief list I’ve made, being a great programmer has absolutely nothing to do with great technical management skills.

In fact, let’s take a look at your typical premium code jockey (here I will use the term “he” for the sake of brevity, but clearly I mean both genders of programmer):

  1. He views himself as cerebrally superior to most of humanity
  2. His self-opinion is usually reinforced daily by those outside the technical community who come to him for guidance and view him as a god of technology
  3. He views the other members of his team as inferior because they tend to view him as a god of technology
  4. He rarely admits to making mistakes
  5. He has absolutely no desire to “teach” anyone anything (In his mind they should have been gifted with his cerebral talent in the beginning.  He can’t fix their sad plight.)
  6. He believes people who make mistakes are idiots and has no respect for them
  7. His communication with senior management is often very smooth – I mean, he’s the god of technology, right?  Senior management loves him.  He can “fix” anything quickly.  He is, indeed, the god of technology
  8. He has no clue what his peers want out of life.  He couldn’t care less what motivates them.
  9. He can finish anything, but the boobs he works with are incompetent.  If he wants something done right he does it himself.
  10. He loves to humiliate potential new programmers in an interview.  He spends days brewing up the most insane technical interview questions he can find on the internet…umm…I mean in his mind…and then proudly prances into the interview room and swooshes out the white board markers to scrawl out a complex algorithm that encircles the room in a hush of reverence for his vast intellect.

Sound like someone you would want to work for?  I can imagine your answer is a resounding “No”.  So, perhaps you’ll take time to post a few stories below about someone technical you worked for who was really, really a great manager?  I’d love to hear your stories!

Until the next time my Tardis passes through,