Financial institution testing: why you can't see it.

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Many companies are tight lipped about our efforts because of the litigation concerns.

Some financial services institutions also have not released their testing data for a very important reason: they used live data. Not mockup data. If they used mocked data, testing wouldn't be as accurate.

So, if they released their actual test results (hard copy reports etc.) it would be a breach of confidentiality. I'm sure none of you would want YOUR financial information passed around.

But they do send test summaries and statistics. Institutional clients can come in a review vendor test results, but they can't copy or take back with them (see above).

Now, which testing method would YOU prefer your bank do? Mock data and release the info, or use REAL data, but keep actual reports (not testing results) confidential?

Now, this also does not assume that ALL financial institutions have done so. Some may actually have lied. But that's a different post....

-- JAW (clueless@pollyanna.com), July 01, 1999

Answers

I'd prefer a little honesty from the banks without them having to have a gaggle of lawyers in their corner. I'd also prefer them not be releasing MY CONFIDENTIAL bank transactions to the FBI. Not like there is much to report but it's the principle(?),...principal (?)...it's the fact that it's my business not the FBI's. Know Your Customer, ha. Whatever happened to Know Your Government?

-- (AtlantaAS@aol.com), July 01, 1999.

And, unfortunately, some may still be having "some problems" and crossing their fingers and hoping "real hard".

An old joke about a man who had just been promoted to president of a company. At the reception honoring this event, the CEO came up to him and said: "Congratulations on your promotion. You've just been told the truth for the last time......"

-- Jon Williamson (jwilliamson003@sprintmail.com), July 01, 1999.


Err, perhaps not. In order to get an accurate test of data you would need to create a database with the same fields or, in the case of a relational database, the same fields, tables and views all with the same interdependencies. Now you have an acurate data model. Well, after you make sure you have the database on a system with compatible hardware, firmware, the same version of the OS with the same patch levels(That's "service pack" to you wintel types). The actual data itself does not have to match what's on production. If it works with bogus test data, it works with real values.

But in my experience companies just export/import their production database to recreate it for testing, so maybe you have a valid point.

And why would the test data itself show up on the results? we just report failures and what fixes where put in to place to correct the problem.

Anyway, hope that helps and keep your...

-- eyes_open (best@wishes.net), July 01, 1999.


JAW-- This is undoubtedly a factor, as you say. Even more so for defense-related remediation. Far more doubtful is whether financial testing of significance is going on cross-border, IMO. And, as you yourself say, some lying can't be ruled out. Obviously, in any situation where results are not made "visible" externally, ethics tend to run the gamut, at least in my experience.

-- BigDog (BigDog@duffer.com), July 01, 1999.

big dog, this may sound dumb, but if EVERY business and instit. lies & y2k is a 10, won,t they be in the same canoe, as the rest of us?

-- hope (dogs@zianet.com), July 01, 1999.


If enough banks remain non-compliant then this is irrelevant. At the moment enough banks are non-compliant.

-- Mike Lang (webflier@erols.com), July 01, 1999.

If enough banks remain non-compliant then this is irrelevant. At the moment enough banks are non-compliant.

And what, praytell, are you basing this opinion?

Yes, it is uncomfortable seeing the gaggle of lawyers and leagalspeak in documents, but again, they are afraid of what they say as being used against them.

But in my experience companies just export/import their production database to recreate it for testing, so maybe you have a valid point.

Yes, indeed. If you "falsify" or mock data, how do you know that the testing data hasn't been manipulated to exclude critical conditions?

Also, we do have test regions, but even so, error that do not appear in our test region only show up in production sometimes. Why? Because you don't have all of the jobs running together. You don't have the exact data in fields being passed around. You don't have the exact same job streams running together.

There isn't any way to mock data to get a clean test. The best you can do is use real data.

That's doing it the right way.

-- JAW (clueless@pollyanna.com), July 01, 1999.


I too would like to know the basis of such a statement that the majority of banks are not compliant. Surely that is just an opinion. If these banks are not revealing their status, where is the evidence of noncompliance?

-- DG (gald0001@unf.edu), July 01, 1999.

FOLKS!!!

Repeat after me...THERE IS NO SUCH THING AS A NON MISSION CRITICAL SYSTEM!!!

Seriously...I have a passing acquaintance with a New York bank. It has TWO noncritical systems...one to manage its ART Collection (including a famous matched set of DUELING Pistols), and one to manage an annual Swim Meet in Westchester County.

Name me ANY banking system which can go unfixed. The Trading Room, which clears $200 BILLION a day? Demand deposits (checking)? Trust?

Folks, it ALL must be fixed. There are no workarounds!!



-- K. Stevens (kstevens@It's ALL going away in January.com), July 01, 1999.


FOLKS!!!

Repeat after me...In banking THERE IS NO SUCH THING AS A NON MISSION CRITICAL SYSTEM!!!

Seriously...I have a passing acquaintance with a New York bank. It has TWO noncritical systems...one to manage its ART Collection (including a famous matched set of DUELING Pistols), and one to manage an annual Swim Meet in Westchester County.

Name me ANY banking system which can go unfixed. The Trading Room, which clears $200 BILLION a day? Demand deposits (checking)? Trust?

Folks, it ALL must be fixed. There are no workarounds!!



-- K. Stevens (kstevens@It's ALL going away in January.com), July 01, 1999.



It is, today, July 1, 1999. We are still debating whether there is or is not a mission critical system, whether the test procedures are being done correctly, etc. If it were 1998, it might not be so worrisome, but given we now have less than 6 months until the Big Day, I would say this is very worrisome!

I sure would plan on the banks not making it....

-- Jack (jsprat@eld.net), July 01, 1999.

JAW,

::So, if they released their actual test results (hard copy reports etc.) it would be a breach of confidentiality. I'm sure none of you would want YOUR financial information passed around.

Programmers have faced this problem before. It's trivial to write a routine to go in and globally change key fields that might be used to identify a customer. Besides, the test data sets ALREADY exist. In fact, banks are likely using the SAME test suites to check Y2K repairs as they did to check the systems out in the first place. After all, a Y2K COMPLIANT system should have EXACTLY the same functionality as it's pre-compliant counterpart.

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), July 01, 1999.


Seriously...I have a passing acquaintance with a New York bank. It has TWO noncritical systems...one to manage its ART Collection (including a famous matched set of DUELING Pistols), and one to manage an annual Swim Meet in Westchester County.

Well, looky here. Then we have two noncritical systems. In a bank. If those systems aren't compliant, the whole system will crash. Sorry, not gonna buy it, and you shouldn't either. You just made my point.

It's trivial to write a routine to go in and globally change key fields that might be used to identify a customer.

No sir. We have several clients, who are competitors. Not only is the data real, but it belongs to several of our clients. They would see vital information on their competitors and they would see their own information walk out the door. There wasn't any way to hide it and get accurate tests.

Most big iron shops have a test system or LPAR, but that is expensive to maintain. You can have a test region on the same machine, but no one in their right might would IPL the system that's used in production with a future date. So you need to do it separately. And might as well use real data. And to make it look like production as much as possible, you need to have the production data.

What is there was a client number of 09999? If you fudged the data fields you'd throw your results off. Or you'd get "false" flags where you'd get error conditions that wouldn't happen in production.

You also would not be able to use automated tools to look for differences. You run one report in production, do a forecast, then future date the test system and do an automated compare.

Most systems are WAY too big to do manual checking.

-- JAW (clueless@pollyanna.com), July 01, 1999.


JAW,

:It's trivial to write a routine to go in and globally change key fields that might be used to identify a customer.

No sir. We have several clients, who are competitors. Not only is the data real, but it belongs to several of our clients. They would see vital information on their competitors and they would see their own information walk out the door. There wasn't any way to hide it and get accurate tests.

Oh puhleeze! If you're not competent enough to dummy up your data and still keep the tests valid you shouldn't be in the IT biz in the first place. Period.

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), July 01, 1999.


JAW

Typical Polly response! Obviously you mised it. The Art system doesn't have to be fixed now. And the SWIM Meet system isn't even on IBM Mainframe equipment and has NO FINANCIAL interfaces.

EVERY OTHER F***ING SYSTEM HAS TO BE FIXED OR THIS BANK IS OUT OF BUSINESS THROUGH CROSS DEFAULTS!!!



-- K. Stevens (kstevens@It's ALL going away in January.com), July 01, 1999.



I didn't say we couldn't mock the data. I said it wouldn't be accurate. It seems that you didn't read my posting. Every system that a company builds is different. If you are such an expert, tech32, you would know that. You would also know that you can test in a phony environment and still get phony errors in test that you won't see in production and then get errors in production that you didn't see in test because the conditions aren't the same (see above). Which, again, is my point.

We take our jobs seriously, and if we did test with faked data, we would be derelict in our duty.

EVERY OTHER F***ING SYSTEM HAS TO BE FIXED OR THIS BANK IS OUT OF BUSINESS THROUGH CROSS DEFAULTS!!!

No shit, Sherlock. But when you say "every system" that does not include the art programming nor the swimmers, does it?

A mission critical system is just that. If it isn't mission critical, it isn't needed. I'm not arguing that mission critical systems don't need to be fixed. I never said that. Again, you just made my point so we really aren't arguing anything.

We have some programs that are nice to have. We even have some that we (dare I say it?) AREN'T GOING TO RENOVATE. There are also some utilities that we aren't going to upgrade, either. So does that mean we're noncompliant and we need to fix them in order to keep the system running? Nope.

This was a rather civil thread untill you ferret friggers decided to call me names. That's not nice. But expected behavior for porgy excrement peddling peasants.

Now, back to my original question: do you piss peddlers want your bank testing with dummy data? I don't.

What would you say if your bank told you they tested using mock data? You butt bastards would be all over them in a heart beat!

-- JAW (clueless@pollyanna.com), July 02, 1999.


JAW,

You would also know that you can test in a phony environment and still get phony errors in test that you won't see in production and then get errors in production that you didn't see in test because the conditions aren't the same (see above). Which, again, is my point.

Well DUH!! They're called BUGS and they happen in real life even if you test with 'live' data, and they happen even if you test with dummy data. Something unexpected comes down the wire and your system crashes.

Your argument was that it is not possible to scramble the data sufficently to hide the identities of the customer. That's just plan crap. Give me a sample dataset and I'll prove it (oh, wait, you can't do that because it would violate your customers privacy!)

Now, back to my original question: do you piss peddlers want your bank testing with dummy data? I don't.

What would you say if your bank told you they tested using mock data? You butt bastards would be all over them in a heart beat!

You know, I don't care as long as it works. Really.

-TECH32-

-- TECH (TECH32@NOMAIL.COM), July 02, 1999.


No, my argument is that we couldn't hide the identity of the customer, and also hide the statistics for our clients (our clients are banks) AND get clean test results.

If we covered ourselves on masking client and customer data, it would be mocked data, which is not as reliable as using live data.

We could have mocked data, but it would not have been as accurate and it would have required a more human intensive test approach.

That's my point.

-- JAW (clueless@pollyanna.com), July 02, 1999.


This came from the JAWbone:

Yes, indeed. If you "falsify" or mock data, how do you know that the testing data hasn't been manipulated to exclude critical conditions?

Its called setting boundary conditions in the test data set, slick. To check the system operation at every extreme value encountered. A MUST in thorough testing. And although it is correct that test data is usually created from live data, there is no reason why it could not be sanitized, and even if it weren't, why would the actual data set be listed in the report????

JAW you obviously don't know SQUAT about what you're yacking about and are not as close to the shop floor as you insinuate. If I'm wrong, and folks like you are involved, then banks would appear to be in a whole lot worse shape than I thought.

-- a (a@a.a), July 02, 1999.


JAW,

No, my argument is that we couldn't hide the identity of the customer, and also hide the statistics for our clients (our clients are banks) AND get clean test results.

Don't buy it.

If we covered ourselves on masking client and customer data, it would be mocked data, which is not as reliable as using live data.

Don't buy it. You can use BOTH if you want (and you should!)

We could have mocked data, but it would not have been as accurate and it would have required a more human intensive test approach.

Now we get to the heart of the matter. You're just being lazy...

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), July 02, 1999.


JAW

Actually, I can offer some insight on that. ALL the major NY banks were undergoing MAJOR upgrades to their Commercial Loan Systems. One bank used dummy data to conceal their OBLIGOR (customer) file from their programming staff. The other bank used live data, and SOME of the loans were VERY interesting!

Suffice it to say that the LEFT is right...MONEY talks and we have a 'free press' only if it takes NO LOANS from the banks!!!



-- K. Stevens (kstevens@It's ALL going away in January.com), July 02, 1999.


TECH32: "After all, a Y2K COMPLIANT system should have EXACTLY the same functionality as it's pre-compliant counterpart."

Thank you. You made my point for me again: that's why we used production data.

You still didn't answer my question. If we used mock data, what would you think about your institution's testing?

Now we get to the heart of the matter. You're just being lazy...

If you call making the decision to not do testing that is worthless, then call my company lazy. It is a waste of company resources, and being a stockholder myself, I'm not interested in seeing my money spent that way, nor do our other stockholdeers.

Not only by scrambling the data give us unreliable results, we'd need several hundred people going through paper reports looking for errors, then explaining every one - full time. Then also wading through "expected" or "explained" errors and correcting the other problems left over that need correction. You wouldn't want your programmers to wade through reports all day...they need to be fixing code.

But if you were a multi million dollar client and wanted us to do fake testing, we'd oblige and charge you out the ying yang for it, if it would make you feel better.

a,

Too bad you chimed in. This was almost cerebral discussion.

"And although it is correct that test data is usually created from live data, there is no reason why it could not be sanitized, and even if it weren't, why would the actual data set be listed in the report????"

You are referring to data sets, and I am referring to the actual data contained in them. The reports one generates to vailidate the data that the client sees is where we find errors, not necessarily examining data sets. Why would you scramble the very data fields that you are checking for accuracy? That's wasted testing.

I'm not going to stoop to your level and insinuate that you don't know what you are talking about, but I will say that every shop is different. And it sounds to me like you might technically know what you are talking about, but your perspective sounds like one of a piece of crap programmer locked in the basement without understanding why you were tasked with items to do by people who run the company. I suspect this is your perspective. Well, I guess I did stoop to your level.

We have several hundered people working on this full time and you think you know how to simply test a system that has a portfolio of $1 trillion? If it was simple we wouldn't need that many people to do it the "lazy" way.

Extremity testing is nice, a, but you've a large range of stuff in the middle. That's where the larger number of the errors are.

I suspect you never had to deal personally with a client on the phone who's error was in the middle of the extremity of testing.

-- JAW (clueless@pollyanna.com), July 02, 1999.


Gotta close the tags. Bad HTML editor, bad.

-- JAW (clueless@pollyanna.com), July 02, 1999.


TECH32: "After all, a Y2K COMPLIANT system should have EXACTLY the same functionality as it's pre-compliant counterpart."

Thank you. You made my point for me again: that's why we used production data.

You still didn't answer my question. If we used mock data, what would you think about your institution's testing?

Now we get to the heart of the matter. You're just being lazy...

If you call making the decision to not do testing that is worthless, then call my company lazy. It is a waste of company resources, and being a stockholder myself, I'm not interested in seeing my money spent that way, nor do our other stockholdeers.

Not only by scrambling the data give us unreliable results, we'd need several hundred people going through paper reports looking for errors, then explaining every one - full time. Then also wading through "expected" or "explained" errors and correcting the other problems left over that need correction. You wouldn't want your programmers to wade through reports all day...they need to be fixing code.

But if you were a multi million dollar client and wanted us to do fake testing, we'd oblige and charge you out the ying yang for it, if it would make you feel better.

a,

Too bad you chimed in. This was almost cerebral discussion.

"And although it is correct that test data is usually created from live data, there is no reason why it could not be sanitized, and even if it weren't, why would the actual data set be listed in the report????"

You are referring to data sets, and I am referring to the actual data contained in them. The reports one generates to vailidate the data that the client sees is where we find errors, not necessarily examining data sets. Why would you scramble the very data fields that you are checking for accuracy? That's wasted testing.

I'm not going to stoop to your level and insinuate that you don't know what you are talking about, but I will say that every shop is different. And it sounds to me like you might technically know what you are talking about, but your perspective sounds like one of a piece of crap programmer locked in the basement without understanding why you were tasked with items to do by people who run the company. I suspect this is your perspective. Well, I guess I did stoop to your level.

We have several hundered people working on this full time and you think you know how to simply test a system that has a portfolio of $1 trillion? If it was simple we wouldn't need that many people to do it the "lazy" way.

Extremity testing is nice, a, but you've a large range of stuff in the middle. That's where the larger number of the errors are.

I suspect you never had to deal personally with a client on the phone who's error was in the middle of the extremity of testing.

-- JAW (clueless@pollyanna.com), July 02, 1999.


Moderation questions? read the FAQ