Espionage and the Covert Art of Data Warehouse Management

data warehouse testing

I don’t know how the secret agent world of intelligence works, but I do know how data warehouses work, and I know how secret agents work in the movies.  So let’s see what happens if I make the “logical connections.”

I am a fictional secret agent who works for MI6.  I have just heard a foreign agent refer to an upcoming event as “Operation Grand Slam.”  I know that the word “Operation” was in front, so we’re not talking about a “Grand Slam” in baseball, tennis, or even a Denny’s menu.  We are talking about some covert action that is going to take place in the near future, and lives may be at risk!  If you know your movies, you know that the plot will involve the fate of all of the gold in Fort Knox and a very destructive weapon (I am not trying to spoil a 52-year-old movie today – I’ll just say that the climax is shocking, to which I’ll tip my hat).

Let’s bring this to the modern day so we can add in our data warehouse knowledge to assist.  You Google this term, and immediately find the movie reference.  The End.  Or is it?  No, the covert name will be something different today, something that fails on a Google search.  Fortunately, you’ve got access to the Utah Data Center, the world’s largest repository of intelligence material.  And data warehouse testing is what you’ll need to solve this dilemma.  But you can’t search a large collection of audio files easily, so there has to be another way.  An easier way to parse the data before we ever ask to generate a report from queried data.  And let me tell you what it is.

The old way of building a data warehouse was to use ETL.  The E and L are not particularly exciting here – they just move the data from one place to another in the same form.  But the T, that’s exciting.  That’s where the magic happens.  T stands for Transform.  And that’s what makes it possible to find that phrase easily.  I was once talking to a headhunter – I mean career placement specialist – who told me that my resume would be scanned to have text pulled from it, so that the .doc or .docx would be irrelevant.  Part of the Transform here will involve a similar process, one aimed at extracting flat text from a file in a different format – in this case an audio file, the same way that Siri can pull real words from audio today.

To get the details of the actual spoken content of a phone call, you need to do one of 2 things: tap the line (if you are using POTS), or copy the assembled packets (if you are using VOIP).  POTS landlines are rapidly disappearing, limiting the need for old-fashioned line-tapping.  To get the metadata, you simply need for the carrier to be required by federal law to push call data toward your aggregation center, to help tag your voice packet collection audio files.  The aggregator then cleanses the data through this Transformation procedure we were just talking about, so that we have a flat text file to scan.  We still might want to hold onto the original audio file for playback at a later time, so we can say, “That’s the voice of the person we are looking for.”

Perhaps the federal government also requires data pushes from other methods of VOIP or text communication, like Skype or FaceTime or gotomeeting or IM or email (pulls would cause too much latency in the communications system, and we can’t shut down communication without someone getting suspicious).  I say perhaps – I have no official knowledge here of what the U.S. government has access to.  I am only saying what I would do if I had ultimate control and wanted this end-goal of communication data collection.  And if you know me, you know how much I would enjoy having ultimate control.  Or maybe my tin-foil hat is pinching my brain too much and requires adjustment.

The point is that we know what we have to do.  We have collected and stored lots of information.  We filter, if needed, by using a Transform so that it is in a flat text form, which is well-designed for querying at a later time.  We give ourselves the ability to query a phrase from our collected flat text.  We use this to generate a report of all of the text matches for things that contain the danger phrase we seek.  The report contains links back to the original audio files or audio script of the conversation, for more subtle analysis.  We sort our report by date, so we can track the genesis of the topic and walk through the later conversations.  All sewn up rather tidily, wouldn’t you say?  All that’s left for us to do now is to send out our best agents out to apprehend the scofflaws, now that we have uncovered their nefarious plot.  And we have the intelligence gathered by our ginormous data warehouse to thank.  Well done everyone, good show!  On to your next assignment …

Similar Articles

Increase Productivity with SharePoint

It is a highly competitive world, and companies are continually striving to do better each day to improve productivity and increase overall efficiency. In any business, managing documents, exchanging data through emails, and scheduling meetings requires continuous collaboration within the team.

10 Best Practices for Choosing a Penetration Testing Company

Penetration testing has grown into one of the most common engagements for the current security-aware companies. There are numerous reasons for running a pentest, such as better security guards, diminished risk levels or meeting strict compliance requirements

Software Testing in 2020:  Biggest Trends

The massive demand for high-quality products created in the shortest period possible made testing a crucial success factor of the software development procedure.

6 Steps for Succeeding with Test Automation in Agile

To be able to keep up with this ever-shorter release cycles that come with the adoption of agile software development, many development teams are embracing test automation as a means to continuously make sure that every software release conforms to the desired degree of quality.

Mobile Application Security Testing Facts

Mobile application security testing can help ensure there aren't any loopholes in the software that might lead to information loss. The sets of tests are meant to assault the program to recognize potential threats and vulnerabilities that would allow external systems or persons to get private data stored on the mobile device.

7 Steps for Obtaining Successful QA Tests

Today’s enterprises are more focussed on faster releases and faster developments but with less focus on the Quality Assurance (QA) part of the SDLC. But, this hard-hitting as when the defects are more, the final software is affected and does not enjoy customer preference.

Why Vue.js is Worth Considering for Your Next Project

Front-end development is evolving at a rapid pace. As a developer, you need to cope up with the modern technology trends to provide the best user experience on your application. As part of this, advanced web development technologies such as Vue.js help to create interactive UI and maximize ROI.

In-house vs Outsource Software Development

Businesses often face the challenge to opt between in-house software application development or outsourcing software development companies for application development. Which one to choose is the real fight. We have the answers though.

4 Ways Microsoft Dynamics 365 Transforming Healthcare Industry

The need to revolutionize patient care has never been significant than it is today. Every healthcare professional wants to know as much as possible about the vitals of patients to improve diagnosis and deliver efficient treatment.