Espionage and the Covert Art of Data Warehouse Management

data warehouse testing

I don’t know how the secret agent world of intelligence works, but I do know how data warehouses work, and I know how secret agents work in the movies.  So let’s see what happens if I make the “logical connections.”

I am a fictional secret agent who works for MI6.  I have just heard a foreign agent refer to an upcoming event as “Operation Grand Slam.”  I know that the word “Operation” was in front, so we’re not talking about a “Grand Slam” in baseball, tennis, or even a Denny’s menu.  We are talking about some covert action that is going to take place in the near future, and lives may be at risk!  If you know your movies, you know that the plot will involve the fate of all of the gold in Fort Knox and a very destructive weapon (I am not trying to spoil a 52-year-old movie today – I’ll just say that the climax is shocking, to which I’ll tip my hat).

Let’s bring this to the modern day so we can add in our data warehouse knowledge to assist.  You Google this term, and immediately find the movie reference.  The End.  Or is it?  No, the covert name will be something different today, something that fails on a Google search.  Fortunately, you’ve got access to the Utah Data Center, the world’s largest repository of intelligence material.  And data warehouse testing is what you’ll need to solve this dilemma.  But you can’t search a large collection of audio files easily, so there has to be another way.  An easier way to parse the data before we ever ask to generate a report from queried data.  And let me tell you what it is.

The old way of building a data warehouse was to use ETL.  The E and L are not particularly exciting here – they just move the data from one place to another in the same form.  But the T, that’s exciting.  That’s where the magic happens.  T stands for Transform.  And that’s what makes it possible to find that phrase easily.  I was once talking to a headhunter – I mean career placement specialist – who told me that my resume would be scanned to have text pulled from it, so that the .doc or .docx would be irrelevant.  Part of the Transform here will involve a similar process, one aimed at extracting flat text from a file in a different format – in this case an audio file, the same way that Siri can pull real words from audio today.

To get the details of the actual spoken content of a phone call, you need to do one of 2 things: tap the line (if you are using POTS), or copy the assembled packets (if you are using VOIP).  POTS landlines are rapidly disappearing, limiting the need for old-fashioned line-tapping.  To get the metadata, you simply need for the carrier to be required by federal law to push call data toward your aggregation center, to help tag your voice packet collection audio files.  The aggregator then cleanses the data through this Transformation procedure we were just talking about, so that we have a flat text file to scan.  We still might want to hold onto the original audio file for playback at a later time, so we can say, “That’s the voice of the person we are looking for.”

Perhaps the federal government also requires data pushes from other methods of VOIP or text communication, like Skype or FaceTime or gotomeeting or IM or email (pulls would cause too much latency in the communications system, and we can’t shut down communication without someone getting suspicious).  I say perhaps – I have no official knowledge here of what the U.S. government has access to.  I am only saying what I would do if I had ultimate control and wanted this end-goal of communication data collection.  And if you know me, you know how much I would enjoy having ultimate control.  Or maybe my tin-foil hat is pinching my brain too much and requires adjustment.

The point is that we know what we have to do.  We have collected and stored lots of information.  We filter, if needed, by using a Transform so that it is in a flat text form, which is well-designed for querying at a later time.  We give ourselves the ability to query a phrase from our collected flat text.  We use this to generate a report of all of the text matches for things that contain the danger phrase we seek.  The report contains links back to the original audio files or audio script of the conversation, for more subtle analysis.  We sort our report by date, so we can track the genesis of the topic and walk through the later conversations.  All sewn up rather tidily, wouldn’t you say?  All that’s left for us to do now is to send out our best agents out to apprehend the scofflaws, now that we have uncovered their nefarious plot.  And we have the intelligence gathered by our ginormous data warehouse to thank.  Well done everyone, good show!  On to your next assignment …

Similar Articles


Prompt and accurate invoicing is an asset for any business. It ensures that the company is timely paid for its services. As the company grows, managing and tracking invoices belonging to different projects manually is impractical.

Business Expense Tracking Software

Nowadays, every business organization can be seen looking for high-end software solutions to increase the productivity and overall efficiency of their business operations. One such digital tool that has gained immense popularity

Mobile Apps

Today, we exist in a highly digital world. After all, we have come a long way from doing processes manually, in-person; instead, we mostly rely on digital tools and solutions, especially mobile apps, to execute most of our day-to-day tasks

Online Proctoring Software: Factors to Keep in Mind

The world had started to, slowly but steadily, engage with the concept of digital learning and online exams long before the coronavirus pandemic came along and disrupted the education ecosystem across the globe.

Online Food Ordering System: Key Features

Unless you have been living under a rock, there's a good chance that you have ordered online at least a handful of times, if not more. Do you know whom you have to thank for that? Online food ordering systems.

5 Ultimate Mobile App Development Trends for 2021

The average time spent on smart devices daily has been increasing over the years. Mobile app development technology is continuously evolving and flourishing in this digital era. Thus, you have to keep up with the latest mobile app development trends.

Penetration Testing

For security-conscious companies, being compliant with industry-standard security policies is a top priority.  Being in non-compliance with industry standards can result in fines.  More importantly, it can lead to increased cyber-attacks on your company, employee data, and customer data.

Magento vs Drupal Commerce: Which One Should You Pick?

As the world continues to evolve at a break-neck pace, the market has witnessed numerous changes. Among all these changes, perhaps none has proven to be quite as consequential as the emergence of e-commerce.

Time Tracking Software

Tracking your employee's time is an essential component of many businesses. It enables business owners to accurately charge their clients for the time spent on their projects. Luckily, time tracking in the digital age is not a challenge anymore.