I would appreciate feedback on what seems to be BC Hydro's exclusion of the bulk of testing from its testing strategy.

BC Hydro explaining testing (which it says it finished in Apr. 99) on its web site 1999

* program code is not always accessible by BC Hydro users, thus making it difficult, if not impossible, to make changes in programming logic; * program logic is not fully documented, thus requiring BC Hydro owners to regard that component as a "black box"; * inputs/outputs for the system or device are often digital or analog signals and cannot be visually inspected or interpreted, while inputs and outputs from business systems can be inspected; * on-line systems testing may not be possible if the testing is likely to impact the production environment in any significant way (e.g. SCADA for the entire electricity grid); * a few systems may not allow the rollback of the clock after advancing the date in tests; * operation/control systems sometimes behave uniquely in each application, thus requiring testing of each physical occurrence of a device.

http://eww.bchydro.bc.ca/html/lib_news_2000_program_t

-- Anonymous, May 12, 1999

Answers

I read this as: 1. we can't get to the code 2. even if we could we wouldn't know what to do with it 3. even if we could get to it, & knew how to fix it, we couldn't actually test it cause we might blow it up

Buy some candles today.

-- Anonymous, May 13, 1999

Rodger,

Have you tested your computer for Y2K yet? How did you do it? Did you plug into your microprocessor chip and analyze the data flow? Did you review the program code? Or did you advance the date and let it roll? As long as you can confirm that all clocks are rolled over, the black box approach is very valid.

-- Anonymous, May 13, 1999

Cl,

Our small business has five Macs. I take from your answer that you are comfortable with BC Hydros strategy of excluding areas of testing due to level of difficulty, lack of expertise and inconvenience. I dont understand your position

-- Anonymous, May 13, 1999

CL,

I can't believe you wrote the above. What you're describing in no way tests the computer, other than to confirm the RTC/BIOS can handle the rollover.

The software, the software!!!

How do you test all the logic paths of the software, with all possible inputs and outputs, and make sure it works correctly?

Even in a full embedded system, the micro-controller software may go down certain logic paths only under certain conditions, which may only come about due to a unique combination of inputs and time/date.

Jon

-- Anonymous, May 13, 1999

Jon,

Calm down. Before you test the software, you better make sure you have a compliant RTC/BIOS or your scientific method has no standard. Isolate the variable. If you don't know your PC is ok first, what failed??? - Gotta have one variable.

The point was that you can take a black box approach without analyzing the actual code and reading the internal registers of the RTC. Good luck with your new "big guy toys". Hope they work well for you. Please be careful and work safely.

-- Anonymous, May 13, 1999

CL,

Yeah, I know. Sorry for jumping :-)

Anyway, you didn't answer my question. How *do* you test the all logic paths of the embedded device?

Jon

-- Anonymous, May 13, 1999

Jon,

Our devices primarily measure voltage, current, frequency, phase angles representative of the actual quantities on the line. They measure for faults and take actions to trip to isolate the faults. We know they measure, and perform outputs A,B,C for a given fault. We simulate faults, record the outputs and timing sequences and store event reports for the simulated fault. We roll the date and apply the same fault and verify the outputs and responses are identical. Any functions that are not supposed to operate, we exercise to make sure they are still working. Repeat often with many dates and once again with power off to check battery backed up clock functions ok.

-- Anonymous, May 13, 1999

CL,

Okay, I'll buy that. Your systems are obviously pretty simple, and mostly analog, as you have stated here before.

But let's look at something a little more complex. Say we've got an embedded system that watches the temperature of something, and when it gets above a certain point, opens a valve to cool it. I don't know if there are devices like this in the real world, but this is a hypothetical situation.

In the box, you've got a micro-controller, and a temperature probe, and a relay. Say you've got twenty of these things, and they're on a bus since they share information with a master. The bus master has a RTC, and interfaces to a printer, which does event logging.

The slaves sample the temperature from the probe through an ADC, and if the temp gets too high, flip the relay to open the valve until the temp goes low again. The slave also sends info to the master, and it time/date-stamps the event, and sends it to the printer. If the event isn't fixed in a certain amount of time, the bus master triggers an alarm or something for human intervention.

How would you test a system like that? Especially if the boxes really were black boxes, with a bus wire, power and ground input, and the input/output lines.

How do you know that your tests exercise all of the code, and all the conditions?

Jon

-- Anonymous, May 13, 1999

Jon,

RTU apps aren't that far removed. I'd test each box discreetly as best as possible 1st. You didn't say if slaves have RTC, but I assume no. Connect a system (including comm) on a bench or make the field environment safe. Heat the probe, watch the valve, observe the printer. Cool the probe and repeat. Verify todays date and proper times. Rollover the clock on the Master and repeat all the above. Our devices record sampled data so we would record (print screen) the data, do the rollover and repeat for a permanent record of an identical operation before and after. 2 different folks compare the before and after operational data to confirm no major changes in operations.

Final step, post it all on EPRI and check for other utils results.

Maybe this was more than you wanted. Alternate answer: With a BIC lighter and a can of freeze mist?

-- Anonymous, May 13, 1999

CL,

Okay, that's what I thought. With a black-box device, you have to do black-box testing.

Do you feel that when you test a device in this manner, that you have tested every possible case that can come up? In other words, do you think this type of testing puts the software that's running on the embedded system through every single logic path in the code?

Jon

-- Anonymous, May 14, 1999