Espionage and the Covert Art of Data Warehouse Management

data warehouse testing

I don’t know how the secret agent world of intelligence works, but I do know how data warehouses work, and I know how secret agents work in the movies.  So let’s see what happens if I make the “logical connections.”

I am a fictional secret agent who works for MI6.  I have just heard a foreign agent refer to an upcoming event as “Operation Grand Slam.”  I know that the word “Operation” was in front, so we’re not talking about a “Grand Slam” in baseball, tennis, or even a Denny’s menu.  We are talking about some covert action that is going to take place in the near future, and lives may be at risk!  If you know your movies, you know that the plot will involve the fate of all of the gold in Fort Knox and a very destructive weapon (I am not trying to spoil a 52-year-old movie today – I’ll just say that the climax is shocking, to which I’ll tip my hat).

Let’s bring this to the modern day so we can add in our data warehouse knowledge to assist.  You Google this term, and immediately find the movie reference.  The End.  Or is it?  No, the covert name will be something different today, something that fails on a Google search.  Fortunately, you’ve got access to the Utah Data Center, the world’s largest repository of intelligence material.  And data warehouse testing is what you’ll need to solve this dilemma.  But you can’t search a large collection of audio files easily, so there has to be another way.  An easier way to parse the data before we ever ask to generate a report from queried data.  And let me tell you what it is.

The old way of building a data warehouse was to use ETL.  The E and L are not particularly exciting here – they just move the data from one place to another in the same form.  But the T, that’s exciting.  That’s where the magic happens.  T stands for Transform.  And that’s what makes it possible to find that phrase easily.  I was once talking to a headhunter – I mean career placement specialist – who told me that my resume would be scanned to have text pulled from it, so that the .doc or .docx would be irrelevant.  Part of the Transform here will involve a similar process, one aimed at extracting flat text from a file in a different format – in this case an audio file, the same way that Siri can pull real words from audio today.

To get the details of the actual spoken content of a phone call, you need to do one of 2 things: tap the line (if you are using POTS), or copy the assembled packets (if you are using VOIP).  POTS landlines are rapidly disappearing, limiting the need for old-fashioned line-tapping.  To get the metadata, you simply need for the carrier to be required by federal law to push call data toward your aggregation center, to help tag your voice packet collection audio files.  The aggregator then cleanses the data through this Transformation procedure we were just talking about, so that we have a flat text file to scan.  We still might want to hold onto the original audio file for playback at a later time, so we can say, “That’s the voice of the person we are looking for.”

Perhaps the federal government also requires data pushes from other methods of VOIP or text communication, like Skype or FaceTime or gotomeeting or IM or email (pulls would cause too much latency in the communications system, and we can’t shut down communication without someone getting suspicious).  I say perhaps – I have no official knowledge here of what the U.S. government has access to.  I am only saying what I would do if I had ultimate control and wanted this end-goal of communication data collection.  And if you know me, you know how much I would enjoy having ultimate control.  Or maybe my tin-foil hat is pinching my brain too much and requires adjustment.

The point is that we know what we have to do.  We have collected and stored lots of information.  We filter, if needed, by using a Transform so that it is in a flat text form, which is well-designed for querying at a later time.  We give ourselves the ability to query a phrase from our collected flat text.  We use this to generate a report of all of the text matches for things that contain the danger phrase we seek.  The report contains links back to the original audio files or audio script of the conversation, for more subtle analysis.  We sort our report by date, so we can track the genesis of the topic and walk through the later conversations.  All sewn up rather tidily, wouldn’t you say?  All that’s left for us to do now is to send out our best agents out to apprehend the scofflaws, now that we have uncovered their nefarious plot.  And we have the intelligence gathered by our ginormous data warehouse to thank.  Well done everyone, good show!  On to your next assignment …

Similar Articles

UFT Test Automation Frameworks: What You Need to Know

As the importance of software continues to grow, there are several factors whose influence have grown right along with it. And out of all of them, test automation frameworks have garnered their fair share of attention from the market. Why is that?

Top Ways in Which EHR Systems Enable Better Patient Care

Electronic Health Records system offers benefits that have enabled an industry as tricky as healthcare. The software allows for medical professionals and other professionals across the ecosystem to quickly input as well as access information about a patient.

Why You Should Choose Angular to Build Web Apps

Development, while eventually rewarding, can be a very tedious process. But that’s the thing about technology; it always manages to find a solution, no matter the problem. So, in the context of development, experts came up with frameworks that would help programmers do away with tedious and arduous coding practices.

Issue Tracking System in DevOps

DevOps and Agile approach in an organization is not merely an implementation of certain tools and techniques, instead, it is cultural change. Many organizations in the software industry have either adopted these methodologies or are making accommodations to incorporate them in their system.

SharePoint for HR Management

Running a company, no matter how big or small is no easy task. It has so many different parts and components that all need to be managed correctly to ensure seamless operations for the company. Now, different people may have different opinions about how to ensure the smooth functioning of a business.

text editor

These days, collaboration has become a business need to support innovative ways of working. Companies are well beyond a realm where employees are destined to their counters. The staff is hardly in the same place at the same time

Code Review Feedback

Almost every single person in the software industry must have heard about code reviews and may be familiar with it. Even then the concept of reviewing codes is misinterpreted. People in this industry generally assume that performing tests must be sufficient to meet the requirements of the verification process, yet it is mostly ignored in the software development life cycle. 

Blind Spots in Software Testing

Have you ever accidentally added spice to a cake, because you were baking a cake and a steak at the same time? Have you dumped a spoon in a dustbin instead of a washbasin?  

Online Exams: Best Practices to Ensure Security

Online exams may seem like a modern contraption and, quite frankly, they are indeed quite advanced. But what they are not is brand new — yep, the concept of online exams has been around for nearly as long as the concept of online learning has been around.