The current tests are rather rigid in what they consider passing. Partners are requesting more modular tests that allow for a wider range of hardware to pass.
To address this I am proposing a tiered test infrastructure. This would allow us to certify different levels of compliance while providing maximum flexibility. The idea is to have 4 levels.
- Bare Implementation - First stage of development.
- Constructors / destructors
- Assumptions
- At least 1 of every peripheral possible implimented
- Basic Compliance - things actually start working.
- Basic Read / Write operations for each API implimented
- Single instance of each peripheral works
- Timing of API measured to 10%
- mbed Enabled Compliance - required to be mbed enabled, this level must pass for it to be marked public
- At least one of each peripheral is broken out to Arduino Hardware spec header
- Full API implimented
- Every pin available is tested with single instance
- Multiple instances of each peripheral can be implimented at same time
- peripherals work in interrupt context
- Timing of API measured to 5%
- Corner Case Compliance - Goal level, not everyone will get here, sets goals for the best platforms to achieve
- Edge / Corner case tests
- Timing of API measured to 1%
- Check for corner cases (really beat on the API hard)
- Instantiate everything and test all at once
| x | Bare (L0) All: Constructor, Destructor | Basic (L1) All: | Mbed Enabled (L2) All: | Corner Case (L3): |
|---|---|---|---|---|
| - Single Instance, full API | - single instances on each possible pin, | - Instantiate and deconstruct multiple instances within same program | ||
| - Run tests 1, 10 times each, | - Run tests 100 times, compliance within 1% | - Try to break things | ||
| - compliance within 5% | - Multiple instances at same time | - Initialize in interrupt context | ||
| - Initialize in callback | ||||
| Digital IO | - | Check set timings between read / writes | Check random timings between read / writes | - |
| - Define Time bounds | ||||
| Analog In | - | - Read values that are between 0 and 1 | Read values | x |
| - Make sure values monotonically increase | ||||
| - Read value multiple times at each stage, make sure variance is within 5%,1% each time | ||||
| Analog Out | - | - Increase value monotonically, validate with analog in (verify with return that it doesn’t blow up) | x | x |
| - Verify it can be set to 0->1 in 0.1 incriments | ||||
| I2C | - | - Read / Write 1, 10, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| SPI | - | - Read / Write 1 byte, 10 bytes, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| PWM | - | - Frequency at 10ms, 30ms, 100ms | - Frequency steps of 1ms from 10 to 200 | Random duty cycles with random frequency's to within 1% |
| - Duty Cycle at 10%,50%,90% | - Duty Cycle every 5% from 5 to 95 | |||
| - Increase monotonically | - Increase monotonically | |||
| Interrupt In | - | - Single object with set timings | - Test on all available pins individually, with random timings (one test for each pin, don’t have more than one object at a time) | - Trigger with PWM's |
| Bus IO | - | Create buses of 1 to 4 pins, read / write | Create busses of 8 - 20 pins, read / write | x |
| Arduino HW | - Validate pins are defined A0-A5, D0-D15 | x | x | x |
| - Validate that expected peripherals are on Arduino Pins per R3 spec (analog In on A0-5...etc) | ||||
| - If pin is NC then don’t use it in later tests (ie analog Out...etc) (not sure how to do this) | ||||
| - Test NC is handled correctly |
Impliment this proposal and release along side the 5.4 mbed-os release in March / April 2017
- Random Number Generation test ( rand() is not actually random between reboots)
- Use python to seed random number to rand() ?
- Some way to detect peripherals and pins available.
- Fully specified mbed_app.json? (too big / complicated?)
- Generate mbed_app.json from mbed pindefs?
This is just a proposal, I would appreciate your feedback, input, and suggestions for either improvements, more robust test requirements, or really anything else you can think of. Please submit feedback in the comments below.
I agree that we should separate Arduino-compatible form factor from mbed OS compatibility, but we should test and report both. I like using the model of layers of compatibility, however, I recommend that we define specific criteria per interface that can, should, and must be followed and then auto-generate detailed reports from the tests that show the capabilities/limitations of each platform.