Hi ,
I am referring to the code coverage as below : https://ci-builds.trustedfirmware.org/static-files/htxSEy4sEOQXtuGetFtzHqAmp...
I see a lot of files from non plat code getting covered with code coverage . can somebody help me with further information :
1. Where are cputests related unit test scripts hosted for non plat files like bl31_main.c ? 2. can we use the unit test & code coverage infra for other platforms say AMD-Xilinx? 3. How is the code coverage achieved for assembly .S files like bl31_entrypoint.S ?
Regards Amit
From: Nagal, Amit Sent: Wednesday, December 11, 2024 1:23 PM To: Gyorgy Szing Gyorgy.Szing@arm.com; Mark Dykes Mark.Dykes@arm.com; Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org; Joanna Farley Joanna.Farley@arm.com Cc: Jain, Ronak ronak.jain@amd.com; Simek, Michal michal.simek@amd.com Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi Everyone,
I see very limited number of unit tests cpp files hosted at https://git.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/refs/... Do we have more unit tests implemented for TF-A and available publicly anywhere ?
Regards Amit
From: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com> Sent: Wednesday, November 20, 2024 2:44 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
“can this be utilized to perform unit testing for platforms other than fvp” Unit-tests executables are running on the “build host” which often is the X86 based PC of the developer (or a CI worker). UT is favoring low cost and sacrificing some level of test correctness. It assumes the compiled C code is fully portable and works the same on all platforms. Moreover, it focuses on C function correctness, and checks if a specific function works as designed. Verifying higher level operation is out of scope (e.g. function implements any feature correctly or is behaving correctly when called by other functions in the system). In turn the complexity of embedded development is removed (managing the target, updating the firmware, using a JTAG interface for debugging, etc…), and the resource constraints of the embedded target is not present.
The code coverage executed on the fvp platform is a system test, it exercises the product on an emulated target. It verifies the correctness of the code as it executes like it will in a real-world scenario. This is a higher value test, but it is much more expensive in terms of time and complexity than UT.
“Is c-picker tool available inside arm still as of now ?” No, it is public, although it is a bit hidden. It lives on the c-picker branch to the Trusted Services repository (see [1) Similarly, Firmware Test Builder lives on the fwtb branch (see 2). This branch is independent of TS code and contains only the two tools. We have plans to move each to dedicated repositories, but I don’t know when I will have the time to do this.
(FWTB and c-picker was developed by my team (TS) and is used and maintained by both projects. Before the TS project was started, we were working on TF-A. I can help with the tools, but I am not familiar to TF-A specifics like, how the tools are integrated to TF-A or which parts of TF-A are covered by UT cases.)
1: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref... 2: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref...
/George
On 2024-11-20, 06:05, "Nagal, Amit via TF-A-Tests" <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> wrote:
Hi Mark,
From unit test prospective , I saw this document https://www.trustedfirmware.org/docs/TF-A-UnitLevelTesting.pdf. can this be utilized to perform unit testing for platforms other than fvp ? Is c-picker tool available inside arm still as of now ?
Regards Amit
From: Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Sent: Tuesday, October 29, 2024 9:20 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Amit, It appears I was incorrect in getting coverage outside of the flows we have internally. I apologize for the confusion...
Mark
________________________________ From: Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>> Sent: Friday, October 25, 2024 12:07 AM To: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Joanna Farley <Joanna.Farley@arm.com<mailto:Joanna.Farley@arm.commailto:Joanna.Farley@arm.com%3cmailto:Joanna.Farley@arm.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi @Mark Dykesmailto:Mark.Dykes@arm.com ,
Can you please share insights about how to get coverage information using manual tests run for FVP platform .
That will help us to atleast get the coverage information for non platform code ( we are interested for bl31) .
Regards
Amit
From: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Sent: Tuesday, September 10, 2024 12:23 AM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; joanna.farley@arm.com<mailto:joanna.farley@arm.commailto:joanna.farley@arm.com%3cmailto:joanna.farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Thank you Joanna and Mark, for your insights and response.
I wasn’t aware that these tests are running on a Virtual Platform and that’s how the coverage is being measured. I’ll look into this further, decide how we want to integrate it into our project, and get back to you.
I appreciate your help and support here.
Best regards, Senthil
From: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>> Sent: Wednesday, August 28, 2024 9:03 AM To: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Senthil,
I do have a way to get coverage for manual test runs using FVP if that helps. I can assist if needed….
Mark
From: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>> Sent: Wednesday, August 28, 2024 12:53 AM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Senthil,
* Is there a way to get code coverage for manual test runs?
Sorry, I'm not aware of such capability.
As you say code coverage implies use of CI scripting that isn't possible to reproduce locally out of the box (afaik).
Regards,
Olivier.
________________________________
From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> Sent: 26 August 2024 23:58 To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: [Tf-a-tests] Re: Query regarding the test coverage thru tf-a-tests
Hi Mark and Oliver,
A gentle reminder on the below request.
Thanks, Senthil
From: Thangaraj, Senthil Nathan Sent: Tuesday, August 20, 2024 3:44 PM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Thanks a lot Mark and Oliver for your inputs and pointers.
All of these seem to be more closely integrated with the CI/CD builds. Is there a way to get code coverage for manual test runs? Specifically, I'm looking to get coverage results after loading and executing all the tests from a developer build (e.g., using a binary like tftf.bin).
Best regards, Senthil
From: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>> Sent: Tuesday, August 20, 2024 3:16 PM To: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
All, This link is more current. The overall coverage numbers are not being reported due to an issue that is being worked on now. However you can see the individual coverage metrics if you click on a particular config where is says "success". The next page will have the link "Build Artifacts" at the top and so click that to the next where you will see the directory "trace_report". Click this and you will see a list of contents one of which should be index.html. If you click this you will see the individual coverage report for that config(test).
tf-a-ci-coverage-gateway #72 [Jenkins] (trustedfirmware.org)https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/72/
Let me know if you have questions...
Mark From: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>> Sent: Tuesday, August 20, 2024 2:27 AM To: tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi,
You can refer to this job showing code coverage results aggregation from multiple test configurations:
https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/67/
https://ci-builds.trustedfirmware.org/static-files/IK1tucCNXFdhS4AAQth8N7zWg...
I don't think there is a test level granularity. @Mark Dykesmailto:Mark.Dykes@arm.com may know better.
Regards, Olivier.
________________________________ From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>> Sent: 20 August 2024 08:40 To: tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>> Subject: [Tf-a-tests] Query regarding the test coverage thru tf-a-tests
Dear TF-A team,
I have a query regarding the TF-A test. Specifically, I would like to know if there is a method to find out the code coverage for a specific test or for all the tests in current run?
Your direction in this regard will be really appreciated.
Thank you, Senthil -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org>> -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org> -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org
Hi Amit,
Code Coverage is only available on the FVP platform as the FVP model has capabilities that allow us with a FVP plug-in to gather debug information that allow us to calculate code coverage numbers. This is why there are no physical platform code coverage figures.
Basically anything run on the model we can gather code coverage figures for (booting, testing etc). The numbers referenced in the link are generated primarily from the TF-A Tests https://trustedfirmware-a-tests.readthedocs.io/en/latest/index.html which are more functional tests rather than formal Unit tests. TF-A has some unit tests but they are minimal at this time. The TF-A Tests framework in many ways covers many of the test criteria Unit tests can cover although are more positive test focused.
TF-A Tests can and has been ported to a few physical platforms but getting code coverage numbers you would need to develop a code coverage solution for your hardware platform.
The Code Coverage solution Arm have produced for FVP’s is described here https://www.trustedfirmware.org/docs/TracebasedCodeCoverageTooling.pdf its feasible to extend this to a physical platform but its not been explored by the authors of this tooling which is available here https://gitlab.arm.com/tooling/qa-tools/-/tree/master/coverage-tool
Hope this helps.
Joanna
From: Nagal, Amit amit.nagal@amd.com Date: Monday, 16 December 2024 at 09:02 To: Gyorgy Szing Gyorgy.Szing@arm.com, Mark Dykes Mark.Dykes@arm.com, Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com, Olivier Deprez Olivier.Deprez@arm.com, tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org, Joanna Farley Joanna.Farley@arm.com, tf-a@lists.trustedfirmware.org tf-a@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com, Simek, Michal michal.simek@amd.com Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi ,
I am referring to the code coverage as below : https://ci-builds.trustedfirmware.org/static-files/htxSEy4sEOQXtuGetFtzHqAmp...
I see a lot of files from non plat code getting covered with code coverage . can somebody help me with further information :
1. Where are cputests related unit test scripts hosted for non plat files like bl31_main.c ? 2. can we use the unit test & code coverage infra for other platforms say AMD-Xilinx? 3. How is the code coverage achieved for assembly .S files like bl31_entrypoint.S ?
Regards Amit
From: Nagal, Amit Sent: Wednesday, December 11, 2024 1:23 PM To: Gyorgy Szing Gyorgy.Szing@arm.com; Mark Dykes Mark.Dykes@arm.com; Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org; Joanna Farley Joanna.Farley@arm.com Cc: Jain, Ronak ronak.jain@amd.com; Simek, Michal michal.simek@amd.com Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi Everyone,
I see very limited number of unit tests cpp files hosted at https://git.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/refs/... Do we have more unit tests implemented for TF-A and available publicly anywhere ?
Regards Amit
From: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com> Sent: Wednesday, November 20, 2024 2:44 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
“can this be utilized to perform unit testing for platforms other than fvp” Unit-tests executables are running on the “build host” which often is the X86 based PC of the developer (or a CI worker). UT is favoring low cost and sacrificing some level of test correctness. It assumes the compiled C code is fully portable and works the same on all platforms. Moreover, it focuses on C function correctness, and checks if a specific function works as designed. Verifying higher level operation is out of scope (e.g. function implements any feature correctly or is behaving correctly when called by other functions in the system). In turn the complexity of embedded development is removed (managing the target, updating the firmware, using a JTAG interface for debugging, etc…), and the resource constraints of the embedded target is not present.
The code coverage executed on the fvp platform is a system test, it exercises the product on an emulated target. It verifies the correctness of the code as it executes like it will in a real-world scenario. This is a higher value test, but it is much more expensive in terms of time and complexity than UT.
“Is c-picker tool available inside arm still as of now ?” No, it is public, although it is a bit hidden. It lives on the c-picker branch to the Trusted Services repository (see [1) Similarly, Firmware Test Builder lives on the fwtb branch (see 2). This branch is independent of TS code and contains only the two tools. We have plans to move each to dedicated repositories, but I don’t know when I will have the time to do this.
(FWTB and c-picker was developed by my team (TS) and is used and maintained by both projects. Before the TS project was started, we were working on TF-A. I can help with the tools, but I am not familiar to TF-A specifics like, how the tools are integrated to TF-A or which parts of TF-A are covered by UT cases.)
1: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref... 2: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref...
/George
On 2024-11-20, 06:05, "Nagal, Amit via TF-A-Tests" <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> wrote:
Hi Mark,
From unit test prospective , I saw this document https://www.trustedfirmware.org/docs/TF-A-UnitLevelTesting.pdf. can this be utilized to perform unit testing for platforms other than fvp ? Is c-picker tool available inside arm still as of now ?
Regards Amit
From: Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Sent: Tuesday, October 29, 2024 9:20 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Amit, It appears I was incorrect in getting coverage outside of the flows we have internally. I apologize for the confusion...
Mark
________________________________ From: Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>> Sent: Friday, October 25, 2024 12:07 AM To: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Joanna Farley <Joanna.Farley@arm.com<mailto:Joanna.Farley@arm.commailto:Joanna.Farley@arm.com%3cmailto:Joanna.Farley@arm.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi @Mark Dykesmailto:Mark.Dykes@arm.com ,
Can you please share insights about how to get coverage information using manual tests run for FVP platform .
That will help us to atleast get the coverage information for non platform code ( we are interested for bl31) .
Regards
Amit
From: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Sent: Tuesday, September 10, 2024 12:23 AM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; joanna.farley@arm.com<mailto:joanna.farley@arm.commailto:joanna.farley@arm.com%3cmailto:joanna.farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Thank you Joanna and Mark, for your insights and response.
I wasn’t aware that these tests are running on a Virtual Platform and that’s how the coverage is being measured. I’ll look into this further, decide how we want to integrate it into our project, and get back to you.
I appreciate your help and support here.
Best regards, Senthil
From: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>> Sent: Wednesday, August 28, 2024 9:03 AM To: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Senthil,
I do have a way to get coverage for manual test runs using FVP if that helps. I can assist if needed….
Mark
From: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>> Sent: Wednesday, August 28, 2024 12:53 AM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Senthil,
* Is there a way to get code coverage for manual test runs?
Sorry, I'm not aware of such capability.
As you say code coverage implies use of CI scripting that isn't possible to reproduce locally out of the box (afaik).
Regards,
Olivier.
________________________________
From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> Sent: 26 August 2024 23:58 To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: [Tf-a-tests] Re: Query regarding the test coverage thru tf-a-tests
Hi Mark and Oliver,
A gentle reminder on the below request.
Thanks, Senthil
From: Thangaraj, Senthil Nathan Sent: Tuesday, August 20, 2024 3:44 PM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Thanks a lot Mark and Oliver for your inputs and pointers.
All of these seem to be more closely integrated with the CI/CD builds. Is there a way to get code coverage for manual test runs? Specifically, I'm looking to get coverage results after loading and executing all the tests from a developer build (e.g., using a binary like tftf.bin).
Best regards, Senthil
From: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>> Sent: Tuesday, August 20, 2024 3:16 PM To: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
All, This link is more current. The overall coverage numbers are not being reported due to an issue that is being worked on now. However you can see the individual coverage metrics if you click on a particular config where is says "success". The next page will have the link "Build Artifacts" at the top and so click that to the next where you will see the directory "trace_report". Click this and you will see a list of contents one of which should be index.html. If you click this you will see the individual coverage report for that config(test).
tf-a-ci-coverage-gateway #72 [Jenkins] (trustedfirmware.org)https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/72/
Let me know if you have questions...
Mark From: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>> Sent: Tuesday, August 20, 2024 2:27 AM To: tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi,
You can refer to this job showing code coverage results aggregation from multiple test configurations:
https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/67/
https://ci-builds.trustedfirmware.org/static-files/IK1tucCNXFdhS4AAQth8N7zWg...
I don't think there is a test level granularity. @Mark Dykesmailto:Mark.Dykes@arm.com may know better.
Regards, Olivier.
________________________________ From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>> Sent: 20 August 2024 08:40 To: tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>> Subject: [Tf-a-tests] Query regarding the test coverage thru tf-a-tests
Dear TF-A team,
I have a query regarding the TF-A test. Specifically, I would like to know if there is a method to find out the code coverage for a specific test or for all the tests in current run?
Your direction in this regard will be really appreciated.
Thank you, Senthil -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org>> -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org> -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org
Hi Joana,
Thanks for information . Additionally, do we have any documentation detailing all the build/run flows to generate coverage report based on whatever cpputest based unit tests are implemented today for TF-A at https://review.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/re... . and whether these build/run flows is available for public usage.
Regards Amit
From: Joanna Farley Joanna.Farley@arm.com Sent: Monday, December 16, 2024 4:57 PM To: Nagal, Amit amit.nagal@amd.com; Gyorgy Szing Gyorgy.Szing@arm.com; Mark Dykes Mark.Dykes@arm.com; Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org; tf-a@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com; Simek, Michal michal.simek@amd.com Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
Code Coverage is only available on the FVP platform as the FVP model has capabilities that allow us with a FVP plug-in to gather debug information that allow us to calculate code coverage numbers. This is why there are no physical platform code coverage figures.
Basically anything run on the model we can gather code coverage figures for (booting, testing etc). The numbers referenced in the link are generated primarily from the TF-A Tests https://trustedfirmware-a-tests.readthedocs.io/en/latest/index.html which are more functional tests rather than formal Unit tests. TF-A has some unit tests but they are minimal at this time. The TF-A Tests framework in many ways covers many of the test criteria Unit tests can cover although are more positive test focused.
TF-A Tests can and has been ported to a few physical platforms but getting code coverage numbers you would need to develop a code coverage solution for your hardware platform.
The Code Coverage solution Arm have produced for FVP’s is described here https://www.trustedfirmware.org/docs/TracebasedCodeCoverageTooling.pdf its feasible to extend this to a physical platform but its not been explored by the authors of this tooling which is available here https://gitlab.arm.com/tooling/qa-tools/-/tree/master/coverage-tool
Hope this helps.
Joanna
From: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com> Date: Monday, 16 December 2024 at 09:02 To: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com>, Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>, Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>, Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>, tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org>, Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com>, tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>, Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests Hi ,
I am referring to the code coverage as below : https://ci-builds.trustedfirmware.org/static-files/htxSEy4sEOQXtuGetFtzHqAmp...
I see a lot of files from non plat code getting covered with code coverage . can somebody help me with further information :
1. Where are cputests related unit test scripts hosted for non plat files like bl31_main.c ? 2. can we use the unit test & code coverage infra for other platforms say AMD-Xilinx? 3. How is the code coverage achieved for assembly .S files like bl31_entrypoint.S ?
Regards Amit
From: Nagal, Amit Sent: Wednesday, December 11, 2024 1:23 PM To: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi Everyone,
I see very limited number of unit tests cpp files hosted at https://git.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/refs/... Do we have more unit tests implemented for TF-A and available publicly anywhere ?
Regards Amit
From: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com> Sent: Wednesday, November 20, 2024 2:44 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
“can this be utilized to perform unit testing for platforms other than fvp” Unit-tests executables are running on the “build host” which often is the X86 based PC of the developer (or a CI worker). UT is favoring low cost and sacrificing some level of test correctness. It assumes the compiled C code is fully portable and works the same on all platforms. Moreover, it focuses on C function correctness, and checks if a specific function works as designed. Verifying higher level operation is out of scope (e.g. function implements any feature correctly or is behaving correctly when called by other functions in the system). In turn the complexity of embedded development is removed (managing the target, updating the firmware, using a JTAG interface for debugging, etc…), and the resource constraints of the embedded target is not present.
The code coverage executed on the fvp platform is a system test, it exercises the product on an emulated target. It verifies the correctness of the code as it executes like it will in a real-world scenario. This is a higher value test, but it is much more expensive in terms of time and complexity than UT.
“Is c-picker tool available inside arm still as of now ?” No, it is public, although it is a bit hidden. It lives on the c-picker branch to the Trusted Services repository (see [1) Similarly, Firmware Test Builder lives on the fwtb branch (see 2). This branch is independent of TS code and contains only the two tools. We have plans to move each to dedicated repositories, but I don’t know when I will have the time to do this.
(FWTB and c-picker was developed by my team (TS) and is used and maintained by both projects. Before the TS project was started, we were working on TF-A. I can help with the tools, but I am not familiar to TF-A specifics like, how the tools are integrated to TF-A or which parts of TF-A are covered by UT cases.)
1: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref... 2: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref...
/George
On 2024-11-20, 06:05, "Nagal, Amit via TF-A-Tests" <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> wrote:
Hi Mark,
From unit test prospective , I saw this document https://www.trustedfirmware.org/docs/TF-A-UnitLevelTesting.pdf. can this be utilized to perform unit testing for platforms other than fvp ? Is c-picker tool available inside arm still as of now ?
Regards Amit
From: Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Sent: Tuesday, October 29, 2024 9:20 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Amit, It appears I was incorrect in getting coverage outside of the flows we have internally. I apologize for the confusion...
Mark
________________________________ From: Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>> Sent: Friday, October 25, 2024 12:07 AM To: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Joanna Farley <Joanna.Farley@arm.com<mailto:Joanna.Farley@arm.commailto:Joanna.Farley@arm.com%3cmailto:Joanna.Farley@arm.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi @Mark Dykesmailto:Mark.Dykes@arm.com ,
Can you please share insights about how to get coverage information using manual tests run for FVP platform .
That will help us to atleast get the coverage information for non platform code ( we are interested for bl31) .
Regards
Amit
From: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Sent: Tuesday, September 10, 2024 12:23 AM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; joanna.farley@arm.com<mailto:joanna.farley@arm.commailto:joanna.farley@arm.com%3cmailto:joanna.farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Thank you Joanna and Mark, for your insights and response.
I wasn’t aware that these tests are running on a Virtual Platform and that’s how the coverage is being measured. I’ll look into this further, decide how we want to integrate it into our project, and get back to you.
I appreciate your help and support here.
Best regards, Senthil
From: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>> Sent: Wednesday, August 28, 2024 9:03 AM To: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
Senthil,
I do have a way to get coverage for manual test runs using FVP if that helps. I can assist if needed….
Mark
From: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>> Sent: Wednesday, August 28, 2024 12:53 AM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Senthil,
* Is there a way to get code coverage for manual test runs?
Sorry, I'm not aware of such capability.
As you say code coverage implies use of CI scripting that isn't possible to reproduce locally out of the box (afaik).
Regards,
Olivier.
________________________________
From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> Sent: 26 August 2024 23:58 To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: [Tf-a-tests] Re: Query regarding the test coverage thru tf-a-tests
Hi Mark and Oliver,
A gentle reminder on the below request.
Thanks, Senthil
From: Thangaraj, Senthil Nathan Sent: Tuesday, August 20, 2024 3:44 PM To: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Thanks a lot Mark and Oliver for your inputs and pointers.
All of these seem to be more closely integrated with the CI/CD builds. Is there a way to get code coverage for manual test runs? Specifically, I'm looking to get coverage results after loading and executing all the tests from a developer build (e.g., using a binary like tftf.bin).
Best regards, Senthil
From: Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>> Sent: Tuesday, August 20, 2024 3:16 PM To: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
All, This link is more current. The overall coverage numbers are not being reported due to an issue that is being worked on now. However you can see the individual coverage metrics if you click on a particular config where is says "success". The next page will have the link "Build Artifacts" at the top and so click that to the next where you will see the directory "trace_report". Click this and you will see a list of contents one of which should be index.html. If you click this you will see the individual coverage report for that config(test).
tf-a-ci-coverage-gateway #72 [Jenkins] (trustedfirmware.org)https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/72/
Let me know if you have questions...
Mark From: Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>> Sent: Tuesday, August 20, 2024 2:27 AM To: tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi,
You can refer to this job showing code coverage results aggregation from multiple test configurations:
https://ci.trustedfirmware.org/job/tf-a-ci-coverage-gateway/67/
https://ci-builds.trustedfirmware.org/static-files/IK1tucCNXFdhS4AAQth8N7zWg...
I don't think there is a test level granularity. @Mark Dykesmailto:Mark.Dykes@arm.com may know better.
Regards, Olivier.
________________________________ From: Thangaraj, Senthil Nathan via TF-A-Tests <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>> Sent: 20 August 2024 08:40 To: tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>> Subject: [Tf-a-tests] Query regarding the test coverage thru tf-a-tests
Dear TF-A team,
I have a query regarding the TF-A test. Specifically, I would like to know if there is a method to find out the code coverage for a specific test or for all the tests in current run?
Your direction in this regard will be really appreciated.
Thank you, Senthil -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>> To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org>> -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.org<mailto:tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org%3cmailto:tf-a-tests-leave@lists.trustedfirmware.org> -- TF-A-Tests mailing list -- tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org To unsubscribe send an email to tf-a-tests-leave@lists.trustedfirmware.orgmailto:tf-a-tests-leave@lists.trustedfirmware.org
Hi Joana,
Thanks for information . Additionally, do we have any documentation detailing all the build/run flows to generate coverage report based on whatever cpputest based unit tests are implemented today for TF-A at https://review.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/re... . and whether these build/run flows is available for public usage.
Regards Amit
From: Joanna Farley Joanna.Farley@arm.com Sent: Monday, December 16, 2024 4:57 PM To: Nagal, Amit amit.nagal@amd.com; Gyorgy Szing Gyorgy.Szing@arm.com; Mark Dykes Mark.Dykes@arm.com; Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org; tf-a@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com; Simek, Michal michal.simek@amd.com Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
Code Coverage is only available on the FVP platform as the FVP model has capabilities that allow us with a FVP plug-in to gather debug information that allow us to calculate code coverage numbers. This is why there are no physical platform code coverage figures.
Basically anything run on the model we can gather code coverage figures for (booting, testing etc). The numbers referenced in the link are generated primarily from the TF-A Tests https://trustedfirmware-a-tests.readthedocs.io/en/latest/index.html which are more functional tests rather than formal Unit tests. TF-A has some unit tests but they are minimal at this time. The TF-A Tests framework in many ways covers many of the test criteria Unit tests can cover although are more positive test focused.
TF-A Tests can and has been ported to a few physical platforms but getting code coverage numbers you would need to develop a code coverage solution for your hardware platform.
The Code Coverage solution Arm have produced for FVP’s is described here https://www.trustedfirmware.org/docs/TracebasedCodeCoverageTooling.pdf its feasible to extend this to a physical platform but its not been explored by the authors of this tooling which is available here https://gitlab.arm.com/tooling/qa-tools/-/tree/master/coverage-tool
Hope this helps.
Joanna
From: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com> Date: Monday, 16 December 2024 at 09:02 To: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com>, Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>, Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>, Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>, tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org>, Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com>, tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>, Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests Hi ,
I am referring to the code coverage as below : https://ci-builds.trustedfirmware.org/static-files/htxSEy4sEOQXtuGetFtzHqAmp...
I see a lot of files from non plat code getting covered with code coverage . can somebody help me with further information :
1. Where are cputests related unit test scripts hosted for non plat files like bl31_main.c ? 2. can we use the unit test & code coverage infra for other platforms say AMD-Xilinx? 3. How is the code coverage achieved for assembly .S files like bl31_entrypoint.S ?
Regards Amit
From: Nagal, Amit Sent: Wednesday, December 11, 2024 1:23 PM To: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi Everyone,
I see very limited number of unit tests cpp files hosted at https://git.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/refs/... Do we have more unit tests implemented for TF-A and available publicly anywhere ?
Regards Amit
From: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com> Sent: Wednesday, November 20, 2024 2:44 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
“can this be utilized to perform unit testing for platforms other than fvp” Unit-tests executables are running on the “build host” which often is the X86 based PC of the developer (or a CI worker). UT is favoring low cost and sacrificing some level of test correctness. It assumes the compiled C code is fully portable and works the same on all platforms. Moreover, it focuses on C function correctness, and checks if a specific function works as designed. Verifying higher level operation is out of scope (e.g. function implements any feature correctly or is behaving correctly when called by other functions in the system). In turn the complexity of embedded development is removed (managing the target, updating the firmware, using a JTAG interface for debugging, etc…), and the resource constraints of the embedded target is not present.
The code coverage executed on the fvp platform is a system test, it exercises the product on an emulated target. It verifies the correctness of the code as it executes like it will in a real-world scenario. This is a higher value test, but it is much more expensive in terms of time and complexity than UT.
“Is c-picker tool available inside arm still as of now ?” No, it is public, although it is a bit hidden. It lives on the c-picker branch to the Trusted Services repository (see [1) Similarly, Firmware Test Builder lives on the fwtb branch (see 2). This branch is independent of TS code and contains only the two tools. We have plans to move each to dedicated repositories, but I don’t know when I will have the time to do this.
(FWTB and c-picker was developed by my team (TS) and is used and maintained by both projects. Before the TS project was started, we were working on TF-A. I can help with the tools, but I am not familiar to TF-A specifics like, how the tools are integrated to TF-A or which parts of TF-A are covered by UT cases.)
1: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref... 2: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref...
/George
On 2024-11-20, 06:05, "Nagal, Amit via TF-A-Tests" <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> wrote:
Hi Mark,
From unit test prospective , I saw this document https://www.trustedfirmware.org/docs/TF-A-UnitLevelTesting.pdf. can this be utilized to perform unit testing for platforms other than fvp ? Is c-picker tool available inside arm still as of now ?
Regards Amit
From: Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Sent: Tuesday, October 29, 2024 9:20 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Amit, It appears I was incorrect in getting coverage outside of the flows we have internally. I apologize for the confusion...
Mark
________________________________ From: Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>> Sent: Friday, October 25, 2024 12:07 AM To: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Joanna Farley <Joanna.Farley@arm.com<mailto:Joanna.Farley@arm.commailto:Joanna.Farley@arm.com%3cmailto:Joanna.Farley@arm.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi @Mark Dykesmailto:Mark.Dykes@arm.com ,
Can you please share insights about how to get coverage information using manual tests run for FVP platform .
That will help us to atleast get the coverage information for non platform code ( we are interested for bl31) .
Regards
Amit
Hi Amit,
https://ci.trustedfirmware.org/view/TF-A/job/tf-a-ci-coverage-gateway/ is the Jenkins Coverage job run weekly and Mark Dykes can provide you with pointers to the scripts etc that are used in this job and the jenkins job source as well as he authored most of this job and hopefully can provide guidance on how to run locally.
ATM only boot and TF-A Test runs are used to generate these numbers as mentioned below. As mentioned we have very limited set of cpputest based unit tests and I don’t believe they are integrated into the Code Coverage system and even if they were don’t expect them to have a big impact on the Code Coverage numbers.
Joanna
From: Nagal, Amit amit.nagal@amd.com Date: Tuesday, 17 December 2024 at 07:20 To: Joanna Farley Joanna.Farley@arm.com, Gyorgy Szing Gyorgy.Szing@arm.com, Mark Dykes Mark.Dykes@arm.com, Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com, Olivier Deprez Olivier.Deprez@arm.com, tf-a-tests@lists.trustedfirmware.org tf-a-tests@lists.trustedfirmware.org, tf-a@lists.trustedfirmware.org tf-a@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com, Simek, Michal michal.simek@amd.com Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi Joana,
Thanks for information . Additionally, do we have any documentation detailing all the build/run flows to generate coverage report based on whatever cpputest based unit tests are implemented today for TF-A at https://review.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/re... . and whether these build/run flows is available for public usage.
Regards Amit
From: Joanna Farley Joanna.Farley@arm.com Sent: Monday, December 16, 2024 4:57 PM To: Nagal, Amit amit.nagal@amd.com; Gyorgy Szing Gyorgy.Szing@arm.com; Mark Dykes Mark.Dykes@arm.com; Thangaraj, Senthil Nathan SenthilNathan.Thangaraj@amd.com; Olivier Deprez Olivier.Deprez@arm.com; tf-a-tests@lists.trustedfirmware.org; tf-a@lists.trustedfirmware.org Cc: Jain, Ronak ronak.jain@amd.com; Simek, Michal michal.simek@amd.com Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
Code Coverage is only available on the FVP platform as the FVP model has capabilities that allow us with a FVP plug-in to gather debug information that allow us to calculate code coverage numbers. This is why there are no physical platform code coverage figures.
Basically anything run on the model we can gather code coverage figures for (booting, testing etc). The numbers referenced in the link are generated primarily from the TF-A Tests https://trustedfirmware-a-tests.readthedocs.io/en/latest/index.html which are more functional tests rather than formal Unit tests. TF-A has some unit tests but they are minimal at this time. The TF-A Tests framework in many ways covers many of the test criteria Unit tests can cover although are more positive test focused.
TF-A Tests can and has been ported to a few physical platforms but getting code coverage numbers you would need to develop a code coverage solution for your hardware platform.
The Code Coverage solution Arm have produced for FVP’s is described here https://www.trustedfirmware.org/docs/TracebasedCodeCoverageTooling.pdf its feasible to extend this to a physical platform but its not been explored by the authors of this tooling which is available here https://gitlab.arm.com/tooling/qa-tools/-/tree/master/coverage-tool
Hope this helps.
Joanna
From: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com> Date: Monday, 16 December 2024 at 09:02 To: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com>, Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>, Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>, Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>, tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org>, Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com>, tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org <tf-a@lists.trustedfirmware.orgmailto:tf-a@lists.trustedfirmware.org> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>, Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests Hi ,
I am referring to the code coverage as below : https://ci-builds.trustedfirmware.org/static-files/htxSEy4sEOQXtuGetFtzHqAmp...
I see a lot of files from non plat code getting covered with code coverage . can somebody help me with further information :
1. Where are cputests related unit test scripts hosted for non plat files like bl31_main.c ? 2. can we use the unit test & code coverage infra for other platforms say AMD-Xilinx? 3. How is the code coverage achieved for assembly .S files like bl31_entrypoint.S ?
Regards Amit
From: Nagal, Amit Sent: Wednesday, December 11, 2024 1:23 PM To: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi Everyone,
I see very limited number of unit tests cpp files hosted at https://git.trustedfirmware.org/plugins/gitiles/TF-A/tf-a-unit-tests/+/refs/... Do we have more unit tests implemented for TF-A and available publicly anywhere ?
Regards Amit
From: Gyorgy Szing <Gyorgy.Szing@arm.commailto:Gyorgy.Szing@arm.com> Sent: Wednesday, November 20, 2024 2:44 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Hi Amit,
“can this be utilized to perform unit testing for platforms other than fvp” Unit-tests executables are running on the “build host” which often is the X86 based PC of the developer (or a CI worker). UT is favoring low cost and sacrificing some level of test correctness. It assumes the compiled C code is fully portable and works the same on all platforms. Moreover, it focuses on C function correctness, and checks if a specific function works as designed. Verifying higher level operation is out of scope (e.g. function implements any feature correctly or is behaving correctly when called by other functions in the system). In turn the complexity of embedded development is removed (managing the target, updating the firmware, using a JTAG interface for debugging, etc…), and the resource constraints of the embedded target is not present.
The code coverage executed on the fvp platform is a system test, it exercises the product on an emulated target. It verifies the correctness of the code as it executes like it will in a real-world scenario. This is a higher value test, but it is much more expensive in terms of time and complexity than UT.
“Is c-picker tool available inside arm still as of now ?” No, it is public, although it is a bit hidden. It lives on the c-picker branch to the Trusted Services repository (see [1) Similarly, Firmware Test Builder lives on the fwtb branch (see 2). This branch is independent of TS code and contains only the two tools. We have plans to move each to dedicated repositories, but I don’t know when I will have the time to do this.
(FWTB and c-picker was developed by my team (TS) and is used and maintained by both projects. Before the TS project was started, we were working on TF-A. I can help with the tools, but I am not familiar to TF-A specifics like, how the tools are integrated to TF-A or which parts of TF-A are covered by UT cases.)
1: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref... 2: https://review.trustedfirmware.org/plugins/gitiles/TS/trusted-services/+/ref...
/George
On 2024-11-20, 06:05, "Nagal, Amit via TF-A-Tests" <tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org> wrote:
Hi Mark,
From unit test prospective , I saw this document https://www.trustedfirmware.org/docs/TF-A-UnitLevelTesting.pdf. can this be utilized to perform unit testing for platforms other than fvp ? Is c-picker tool available inside arm still as of now ?
Regards Amit
From: Mark Dykes <Mark.Dykes@arm.commailto:Mark.Dykes@arm.com> Sent: Tuesday, October 29, 2024 9:20 PM To: Nagal, Amit <amit.nagal@amd.commailto:amit.nagal@amd.com>; Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com>; Olivier Deprez <Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com>; tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org; Joanna Farley <Joanna.Farley@arm.commailto:Joanna.Farley@arm.com> Cc: Jain, Ronak <ronak.jain@amd.commailto:ronak.jain@amd.com>; Simek, Michal <michal.simek@amd.commailto:michal.simek@amd.com> Subject: Re: Query regarding the test coverage thru tf-a-tests
Amit, It appears I was incorrect in getting coverage outside of the flows we have internally. I apologize for the confusion...
Mark
________________________________ From: Nagal, Amit <amit.nagal@amd.com<mailto:amit.nagal@amd.commailto:amit.nagal@amd.com%3cmailto:amit.nagal@amd.com>> Sent: Friday, October 25, 2024 12:07 AM To: Thangaraj, Senthil Nathan <SenthilNathan.Thangaraj@amd.com<mailto:SenthilNathan.Thangaraj@amd.commailto:SenthilNathan.Thangaraj@amd.com%3cmailto:SenthilNathan.Thangaraj@amd.com>>; Mark Dykes <Mark.Dykes@arm.com<mailto:Mark.Dykes@arm.commailto:Mark.Dykes@arm.com%3cmailto:Mark.Dykes@arm.com>>; Olivier Deprez <Olivier.Deprez@arm.com<mailto:Olivier.Deprez@arm.commailto:Olivier.Deprez@arm.com%3cmailto:Olivier.Deprez@arm.com>>; tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org> <tf-a-tests@lists.trustedfirmware.org<mailto:tf-a-tests@lists.trustedfirmware.orgmailto:tf-a-tests@lists.trustedfirmware.org%3cmailto:tf-a-tests@lists.trustedfirmware.org>>; Joanna Farley <Joanna.Farley@arm.com<mailto:Joanna.Farley@arm.commailto:Joanna.Farley@arm.com%3cmailto:Joanna.Farley@arm.com>> Cc: Jain, Ronak <ronak.jain@amd.com<mailto:ronak.jain@amd.commailto:ronak.jain@amd.com%3cmailto:ronak.jain@amd.com>>; Simek, Michal <michal.simek@amd.com<mailto:michal.simek@amd.commailto:michal.simek@amd.com%3cmailto:michal.simek@amd.com>> Subject: RE: Query regarding the test coverage thru tf-a-tests
Hi @Mark Dykesmailto:Mark.Dykes@arm.com ,
Can you please share insights about how to get coverage information using manual tests run for FVP platform .
That will help us to atleast get the coverage information for non platform code ( we are interested for bl31) .
Regards
Amit
tf-a@lists.trustedfirmware.org
-
Joanna Farley -
Nagal, Amit