So if data modeling has been around for decades, I thought I'd check Google Trends to see how some of the newer terms would show.  First, I started with ITIL.    Google Trends allows you to report on trends only as far back as Jan 2004, so ITIL shows a fairly slow rise in search popularity.

Next I tried IT Service Management (ITSM), a related term.  The trend shows that ITSM was not popular enough to be tracked by Google Trends until about February 2004.  By watching those trend lines, you can pretty much tell when a term got buzz.

So in the above sample, you can see the birth of a buzzword.  My guess is that as long as the lines remain steady or increasing, the term may progress from buzzword to silver bullet to standard of practice.  Maybe.

Sometimes terms are popular in a small enough user base, or have small popularity in a larger user base that there is insufficient data for reporting.  For example, if you type in agile modelings, you will get a page that looks like the sample figure below.

This just means that the term has not had enough searches to reach the popularity requirement for reporting in Google Trends.  I'm curious, though, as to when terms "make it". 

I'll write more later about trends and looking to the past.

Google Labs always offers a set of nifty tools for sorting through the billions of pieces of data and information that pass through their servers.  One new and interesting feature is Google Trends.  Trends reports on the history of search terms, reporting the top search origins for a search term as well as the trend in how often it has been searched for.

So I typed in data model and found something that surprised me.  The top location of searchers for data model is....Bangalore, India.  In fact, the top 5 positions for people seeking more information about data modelings is India.  The figure below is a sample of the information returned by Google trends.

Singapore is next, and then it is Washington, DC breaking in to the list at position seven.    If you read into the help portion of this tool, you'll find that the result are determined based on sampling the massive amount of server log data they collect as each visitor uses their sight. 

So does the fat that 7 of top ten positions on this list are located outside North America mean that more non-NA people are learning data modeling?  Could it mean that more outside the US are working on data models?  Could it mean that there aren't enough people in the US working in IT to break out higher in the list?  We won't know just from looking at this data.  Perhaps a better test would be to look at sales of data modeling books in each country?  Or the numbers of students taking courses in data management.  Or...

What are your guesses as to the significance of this data?

I've added a new article by Peter Stiglich on Enabling High Quality Analytics Through a Data Validity Dimension.  Peter's approach is interested -- who knew that using the cartesian product was going to be so helpful?

Data Quality

While working on an Enterprise Data Warehouse for a state court system the issue of poor data quality in the source systems became apparent.  Referential integrity was not strictly enforced and there was very little in the way of attribute level constraints.  One normally expects that these types of constraints be enforced for an OLTP application, whether through the application, in the database, or both.     

 

Of course, one should never be surprised when there is poor data quality in the source systems – poor data quality is the norm rather than the exception.  According to The Data Warehouse Institute (TDWI) over $600 billion a year is lost due to poor data quality. 

If you are interested in submitting an articled, send it to website@infoadvisors.com for review.

Barry Williams, of Database Answers,  has one of the most comprehensive lists of data model tools I've seen.  Some of the entries are fairly obsolete, but if you wanted to know what's out there, this would be a good place to start.  This list includes open source, commercial, free, modeling tools (or modelling tools, if you spell it that way).

This recent CNET News.com article about owners of Toyota Prius hacking the control systems of their automobiles started me thinking about hacks, easter eggs, and other mods users make to sofware applications.

It has only been a few years that we modeling tool users have had the ability to change the fonts, backgrounds, colors, and other usability features in our tools.  Before that, we often had to resort to printed material or image capture to highly parts of our models or to get extended features.

I've often wondered if there are easter eggs in any of these products.  From eeggs.com:

What is an "Easter Egg"?

In the context of software (get that Cadbury Bunny out of your head!), an Easter Egg is a hidden feature or novelty that the programmers have put in their software. In general, it is any hidden, entertaining thing that a creator hides in their creation only for their own personal reasons. This can be anything from a hidden list of the developers, to hidden commands, to jokes, to funny animations. You'd be surprised just how many things contain Easter Eggs... just look at the list that has accumulated here!

A true Easter Egg must satisfy the following criteria:

1. Undocumented, Hidden, and Non-Obvious
   An Easter Egg can't be a legitimate feature of a product, or be an obvious part of a storyline. Easter Eggs will usually stand out either because they totally don't fit with their context (like a pinball game in a word processor), or because they have a deeper hidden personal meaning to the creators, so they threw it in for entertainment.
2. Reproducible
   Every user with the same product or combination of products must be able to produce the same result given the instructions. If others can't reproduce an Egg, then it doesn't belong in this archive.
3. Put There by the Creators for Personal Reasons
   The Egg must have been put there on purpose, and furthermore have a personal significance to the creators beyond just making a better product (movie, TV show, software program, etc).
4. Not Malicious
   Easter Eggs are there for fun, not to do damage.
5. ENTERTAINING!
   The most important element... if it's not there for entertainment, it's not an Egg.

I've never seen anyone report an easter egg in a modeling tool...have you?

John Zachman is interviewed by SearchDataManagement.com about the origins and benefits of using the Zachman Framework.

What exactly is the Zachman Framework?
Zachman: The best way to understand it is through a metaphor. The whole idea has to do with architecture. In the early 70s, we [IBM] were doing information strategy work at airplane manufacturing companies. It was obvious that airplane manufacturers knew what architecture was relative to airplanes. They were producing extremely complex objects - airplanes. And they could maintain them for an extremely long time. So, they could deal with extreme complexity and extreme change.

If you're going to create a complex object, you need to be able to describe it. For example, if you cannot describe the airplane, you cannot create the airplane. On top of that, once you get it created and then you want to change it -- how do you do that? You go back to your descriptive representations, which are the basis for change. If you want to deal with complexity and change, then you are going to deal with architecture. It doesn't make any difference if you're talking about buildings, or airplanes, or supercomputers or enterprises.

Book Study:  Database in Depth - 5 June 2006

We will be starting the book study in our Data Model Discussion Group on CJ Date's Database in Depth : Relational Theory for Practitioners

Since most data modelers have learned on the job instead of receiving formal education on the topics covered in this work, I believe it is very important that we understand the theory that forms the foundation of what we do.  It has been twenty years since I studied data management theory, so it will be good to read these topics again.

We will cover a chapter a week and will start the discussion on 5 June.  That gives you two weeks to get your hands on a copy and read the first chapter.

The way we will work this is each week I will start a new topic for each chapter in the Data Management Resources conference on the Data Modeling Group Board and we can discuss any questions, issues, or comments you may have about that chapter.

So check to see if you are set up for the mailing list for this conference by logging in to the Data Model Group and going to Options/My Mailing Lists.  You will also want to check to see your e-mail address is correct by logging in to the board and going to Options/My Profile.

Place your orders to get your copy of the book - I've included links via our affiliates below.  You are in no way required to purchase via these links.  I have seen this book on the shelf at some Borders Books, but I'm guessing you will have to special order this work.

I'm looking forward to this book study and I hope you are, too.

Amazon.com    Database in Depth : Relational Theory for Practitioners

Amazon.ca       Database in Depth: Relational Theory for Practitioners

Amazon.co.uk  Database in Depth: The Relational Model for Practitioners

Over the last 20 years I have worked for and with professional services companies.  Many of these organizations employed brilliant people who had decades of experience in their specialization.  Other companies, though, focused on the consulting model of hiring the least experienced people they could hire, then pass them off as experts -- because least experienced means lesser pay and therefore a higher margin.

Some companies specialized in what one of my clients called the IROC approach - hirer Idiots Right Out of College and send them to a crash course in IT, usually just a couple of weeks of training at best.  Often the $250 an hour consultant knew less about the technology and business at hand than the full time employees that had researched the topic in their spare time.  One of the highest margins a consulting company can make is by hiring 'professionals' from other fields and passing them off as professionals in IT. 

Take, for example, this article in SearchCIO.com:

Matt Sorge graduated from MIT in January with a degree in mechanical engineering and a respectable 4.2 (out of a possible 5.0) grade-point average (GPA). When company recruiters descended on the campus in Cambridge, Mass., last fall, Sorge bypassed the big American automobile companies, a common first stop for many MIT mechanical engineers, and spent most of his time talking to Infosys Technologies Ltd., the Indian information services and consulting firm.

"I have very limited background in computer science, but they told me they were looking to hire for potential. I liked that," Sorge said.

Come July, the 23-year-old "military brat" from Lawton, Okla., will head to Infosys' corporate training center in Mysore, India, for an intensive 16-week crash course in software engineering. He's one of 100 students tapped for what Infosys said is its "first major college recruiting effort in the United States." The company plans to recruit a total of 300 students right out of school over the next 12 months.

This does not mean that there are not talented and brilliant mechanical engineers out there working in IT and making their projects a success.  Indeed our candidate above is most likely incredibly bright, intelligent, and well-educated in the areas of thermodynamics, strength of materials and engineering ethics.   He admits, though, that he has almost no background in computer science and I'd bet he knows nearly nothing about information systems -- except as a user.  He will be sent on a crash course in IT topics, then sent off to work on some large information systems projects with little knowledge of relational databases, networking, IT security, reliability, maintainability, and all the other abilities.   He will most likely learn from his clients how these things work.  

I know some incredibly wonderful IT professionals who have no academic background at all.  They have self-studied and trained themselves to an exceptionally high level of competency  -- and most of this on their own dime.  The issue I have is not with an one individual's credentials - it is with the decreasing competency of organizations as a whole.

People who cross over into IT had to learn on the job.  That means that they apprenticed into IT, perhaps learning from others who apprenticed in, etc.  That leads to the types of solutions I've come across from experts who entered the IT profession by word-of-mouth training (none of these had any formal education in IT)

• The developer came across a business problem that needed multiple instances of data where only one was allowed by the data model.  Did he raise the issue as a missed requirement?  No, he just added comma separated values in the column in the database.  It worked in development, but failed my minimum level of professional competency test.
• A developer on a system that required encryption for sensitive data designed a solution that put the encryption key in plain text in a file on the root drive of the server.  Again, it worked in development, but failed my test for a minimum level of professional competency.
• A developer wrote up a lengthy message to the CIO that the database design was keeping him from meeting the needs of the business.  This led to an emergency meeting between the data management and development staff, with the CIO mediating.  What design strategy was keeping his application from working?  Everytime he went to put data in the database, the database would spit out an error, no matter what he typed.   The error messages kept telling him that he needed to create a customer or an item before an order and everyone knows that orders come before customers or items.  It turns out that this consultant had never programmed against a database, only "against HTML".
• A senior IT consultant claimed that there were thousands of errors in the data models that he had to correct to get the application working.  What types of errors did he correct?  The most prominent one was that the Chart of Accounts called for 14 digit accounting codes and his application only needed a couple of hundred, so he changed the Account Code to be 3 digits.  And the employee given name field only required 14 characters because he looked in the phone directory and determined that his application only needed that many characters...and there were more "corrections" just like that.
• A senior developer unfamiliar with relational database concepts wrote code that took more than 17 minutes to run, even though all it was doing was starting the application.  Very little processing was supposed to be happening.  She spent weeks trying to tune the system, but could only shave off processing time by removing all graphics.  No peer reviews were allowed because she was a senior developer.  A quick peek at her code showed the problem - to check to see whether the database was available to the application, her code was sequentially counting (using GET NEXT RECORD) all the rows in all the tables, then comparing the counts to the number zero.  If they were equal, then the database was not available.   When the code was fixed to only test the connection, her response was that she was "not a database person and could not be expected to know every nuance of every database on the planet." Her background?  She was a Ballet major who had been sent to a 3 week boot camp training session in IT.  She was classified as a senior developer because her degree was from a prestigious private college.

We've all made embarrassing mistakes while learning a new technology or approach, but what happens if your entire project is staffed with non-profesisonals?  I believe it is fine to staff individuals based on "potential" as described in the article.  However, too many organizations staff their entire resource pool with potential, leaving the client organizations responsible for the apprenticing in the experts just so that they can meet the minimum level of professional competency that they should have had before they were set loose on a real project.  Again, I know and respect many IT professionals who learned their craft in the school of hard knocks -- but you can't staff an entire project with crash test...er...professionals.

All those companies that have outsourced their most difficult projects roles to "cheaper" resources had better be careful - they might just get what they wished for.

This week I'll be writing about how to ensure that your consulting staff has the right stuff to lead your project to success.

Guess what?  Not all women have the same career goals, work habits, or family constraints.  Unbelievable, isn't it?  I know that our readers understand this, but I think it is important that employers understand it. 

My first job out of school was for a large consulting firm that focused primarily on Defense and Federal government projects.  My assignments were exciting, very challenging, and fun.  Then I got married and my projects changed.  I was given assignments that were boring and not challenging.  When I asked my project manager why, I was told that I had married and would probably leave the company to become a housewife, so he wasn't going to put me in a position that would be at risk when I left.  This is also the same guy that put in my performance appraisal under Technical Skills: "None Observed" when I had removed 7 minutes (not seconds, but minutes) from the application start up, designed all the databases,  developed code, designed user interfaces, fixed thousands of bugs in the existing app, you name it.  When I questioned him on this, he said "Joe has technical skills, ask him".  Joe was our show guy, the one sent to conferences to demo the product.  He had very few technical skills.   My appeal of the appraisal was the first of many that others on our project filed.

The sad part is that I finally left that company for better assignments in another company, thus proving to this idiot that married women leave the company.  I don't think he ever clued in that he was the cause. 

But back to women and careers.  Eileen Trauth, professor of information sciences and technology at Pennsylvania State University has presented her findings on why women are opting out of IT:

"Policy makers, educators, managers need to recognize that you can't generalize to all women," said Eileen Trauth, professor in the College of Information Sciences and Technology (IST). "There is far too much variation in the paths that women take for anyone to assume that women's career motivations are the same, their methods of balancing work and family are the same, or their responses to motherhood are the same."

Trauth conducted interviews with 167 women who were working in IT in Australia, New Zealand, Ireland and the United States. Besides their place of residence, the women also represented a range of racial and ethnic backgrounds.

Those interviews suggested women's career choices were influenced by a wide range of factors including gender stereotypes, societal messages and family dynamics, Trauth said. But she also recorded a wide range of responses to the motherhood, career and educational choices and gender stereotypes, reinforcing her belief that recognizing such diversity may yield more opportunities for women.

While I'm thrilled that research is being completed in this area, I am saddened by the fact that employers and project managers think that all women are the same or that all women will leave.  My guess is that managers who believe this have behaved in a way that drives women to other projects or companies -- and that these ill informed managers see this as proof of their beliefs.

I attended a very interesting public workshop hosted by the Anti-Spyware Coalition - Developing International Solutions for Global Spyware Problems.  This one day event featured speakers and attendees from the US Federal Trade Commission, Microsoft, CA, Google, Symantec, McAfee, CAUCE Canada, The National Network to End Domestic Violence, and more.

The day started with Eileen Harrington. Associate Director in the Bureau of Consumer Protection, who highlighted the need for global co-operation in fighting malware.  She spoke about pending legislation, the US Safe Web Act, that would enable law enforcement to share information about investigations with foreign law enforcement and the ability to request information from ISPs without their disclosure to the suspect.

The first panel involved discussions and demonstrations of spyware technologies and approaches.  The second panel, Harms, discussed the impact of spyware on society, including spyware used to track and stalk victims of domestic violence. 

Micheal Geist led a panel on legislative solutions.  The final panel focused on Increasing Consumer and Enterprise Awareness.

One of the most interesting presentations was by Joe Jarzombeck, the Director of Software Assurance for the US Department of Homeland Security.  Software Assurance is Joe's doing at DHS.  His role, which he was allowed to design, focuses on increasing integrity and reliability of software systems in government and private practice.   The DHS has developed several products that are free to use.  I will be writing more on this topic soon.