The CI comprises of three modules. The Jenkins logic, the Python Scripts and the docker build slave.
The environment is provided by the docker build slave and provisioned into the pipeline flow when the venv is created for each stage:
virtualenv -p python3 ${VENV_P3_NAME} --system-site-packages
No requirements are installed at the Jenkins Stage, but if needed as an one-off (i.e. for staging purposes), the design supports it.
The case for installing Python requirements on the fly using requirements.txt
Extending requirements dynamically on the fly, can be quite a challenge due to the way Jenkins handles the absolute resolution of workspace directory on each step. In short Python’s virtual-environment stores the configuration paths in absolute format, while Jenkins is not guaranteed to give you the same reference to a working directory in consecutive calls in the pipeline.
So if you create a venv at stage 1, which evaluates ~/ as /server/workspace/fubar-job/venv/.. and then attempt to call it in a following parallel step, the code may be located at /server/workspace/1/fubar-job/venv
At this point you can either create the VENV in each stage, and reinstall the requirements, effectively wasting bandwidth or hack it by piping everything in SED before activating to ensure the path is resolved correctly.
For that purposes the ci-scripts level requirements.txt will be deprecated in the next feature update.
How should a user access or modify the TF-M Build environment.
Environment will be established at the docker build stage.
https://review.trustedfirmware.org/admin/repos/ci/dockerfiles
And more specifically the requirements for Python3 in this file:
https://git.trustedfirmware.org/ci/dockerfiles.git/tree/xenial-amd64-tf-m-b…
After it has been updated and merged you should raise a ticket and request the image to get rebuilt, or if it is not very critical, wait for some other party change to trigger the build.
The process requires creating local docker image, meant to test your changes but also allowing you to access the TF-M build environment as deployed on the CI. You can do that following the steps below:
# Get the docker image
$ git clone https://review.trustedfirmware.org/ci/dockerfiles && cd dockerfiles/xenial-amd64-tf-m-build
# Edit the entry point to convert it not to be a jenkins-slave
$ vi Dockerfile
# change ENTRYPOINT ["/usr/local/bin/jenkins-slave"] to ENTRYPOINT [/bin/bash"], save exit
# Build the image
$ docker build ./
# Find the image hash id
$ docker image ls
# Run an interactive bash shell, mounting a local directory as /opt/openci in instance (if required to share files)
$ docker run -it --name tf-m-build-env -v /YOUR_CUSTOM_PATH:/opt/openci 10bcb173cd39
# You can relaunch that instance in the future by starting it again.
$ docker start && docker -exec -it tf-m-build-env /bin/bash
Please let me know if you need more clarity or guidance on how to handle modifications on the CI.
Regards,
Minos Galanakis
________________________________
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> on behalf of Antonio De Angelis via TF-M <tf-m(a)lists.trustedfirmware.org>
Sent: 02 August 2019 14:57
To: tf-m(a)lists.trustedfirmware.org <tf-m(a)lists.trustedfirmware.org>
Cc: nd <nd(a)arm.com>
Subject: Re: [TF-M] Changes to CI for python dependencies
+Minos now
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Antonio De Angelis via TF-M
Sent: 02 August 2019 14:47
To: tf-m(a)lists.trustedfirmware.org
Subject: Re: [TF-M] Changes to CI for python dependencies
Minos, could you have a look at this?
Thanks,
Antonio
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Kevin Townsend via TF-M
Sent: 02 August 2019 12:44
To: Thomas Törnblom via TF-M <tf-m(a)lists.trustedfirmware.org>
Subject: [TF-M] Changes to CI for python dependencies
In an effort to migrate to the more modern 'cryptography' module in imgtool.py (which mcuboot has already switched to upstream), I created a change request here:
https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
The change fails in CI, however, due to the missing cryptography module in the CI build environment:
https://ci.trustedfirmware.org/job/tf-m-build-test-review/1740/artifact/bui…
This brings up the following issues:
- How can/should changes be made to the CI build environment?
- Can the overall TF-M installation process be improved automating
Python module installation via a requirements.txt file?
Adding a requirements.txt file means that file could be run when the CI environment starts a new test build, taking into account any dependency changes that are part of the change request (version updates, etc.).
This would also have the positive side effect of users no longer having to scan through tfm_sw_requirement.rst to see what they don't have installed, or parse build failures for missing module names.
I'm happy to make a new change request adding a requirements.txt file, and update the documentation accordingly, but t's not clear to me how to propose the required changes to the CI setup?
Best regards,
Kevin Townsend
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Hi Andrej,
Alignment with the PSA 1.0.0 APIs is on the TF-M roadmap for next quarter: https://developer.trustedfirmware.org/w/tf_m/planning/
The "release" branch of the psa-arch-tests repo should be used for functional API certification. TF-M is compatible with that branch.
Best wishes,
Jamie
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Andrej Butok via TF-M
Sent: 05 August 2019 12:47
To: tf-m(a)lists.trustedfirmware.org
Subject: [TF-M] PSA Storage API
Just only FYI:
The latest PSA Test Suite has been switched to a newer version of PSA Storage API:
https://github.com/ARM-software/psa-arch-tests/issues/105
Best regards,
Andrej Butok
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
+Minos now
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Antonio De Angelis via TF-M
Sent: 02 August 2019 14:47
To: tf-m(a)lists.trustedfirmware.org
Subject: Re: [TF-M] Changes to CI for python dependencies
Minos, could you have a look at this?
Thanks,
Antonio
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Kevin Townsend via TF-M
Sent: 02 August 2019 12:44
To: Thomas Törnblom via TF-M <tf-m(a)lists.trustedfirmware.org>
Subject: [TF-M] Changes to CI for python dependencies
In an effort to migrate to the more modern 'cryptography' module in imgtool.py (which mcuboot has already switched to upstream), I created a change request here:
https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
The change fails in CI, however, due to the missing cryptography module in the CI build environment:
https://ci.trustedfirmware.org/job/tf-m-build-test-review/1740/artifact/bui…
This brings up the following issues:
- How can/should changes be made to the CI build environment?
- Can the overall TF-M installation process be improved automating
Python module installation via a requirements.txt file?
Adding a requirements.txt file means that file could be run when the CI environment starts a new test build, taking into account any dependency changes that are part of the change request (version updates, etc.).
This would also have the positive side effect of users no longer having to scan through tfm_sw_requirement.rst to see what they don't have installed, or parse build failures for missing module names.
I'm happy to make a new change request adding a requirements.txt file, and update the documentation accordingly, but t's not clear to me how to propose the required changes to the CI setup?
Best regards,
Kevin Townsend
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Minos, could you have a look at this?
Thanks,
Antonio
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Kevin Townsend via TF-M
Sent: 02 August 2019 12:44
To: Thomas Törnblom via TF-M <tf-m(a)lists.trustedfirmware.org>
Subject: [TF-M] Changes to CI for python dependencies
In an effort to migrate to the more modern 'cryptography' module in imgtool.py (which mcuboot has already switched to upstream), I created a change request here:
https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
The change fails in CI, however, due to the missing cryptography module in the CI build environment:
https://ci.trustedfirmware.org/job/tf-m-build-test-review/1740/artifact/bui…
This brings up the following issues:
- How can/should changes be made to the CI build environment?
- Can the overall TF-M installation process be improved automating
Python module installation via a requirements.txt file?
Adding a requirements.txt file means that file could be run when the CI environment starts a new test build, taking into account any dependency changes that are part of the change request (version updates, etc.).
This would also have the positive side effect of users no longer having to scan through tfm_sw_requirement.rst to see what they don't have installed, or parse build failures for missing module names.
I'm happy to make a new change request adding a requirements.txt file, and update the documentation accordingly, but t's not clear to me how to propose the required changes to the CI setup?
Best regards,
Kevin Townsend
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
In an effort to migrate to the more modern 'cryptography' module in
imgtool.py (which mcuboot has already switched to upstream), I created
a change request here:
https://review.trustedfirmware.org/c/trusted-firmware-m/+/1695
The change fails in CI, however, due to the missing cryptography module in the
CI build environment:
https://ci.trustedfirmware.org/job/tf-m-build-test-review/1740/artifact/bui…
This brings up the following issues:
- How can/should changes be made to the CI build environment?
- Can the overall TF-M installation process be improved automating
Python module installation via a requirements.txt file?
Adding a requirements.txt file means that file could be run when the CI
environment starts a new test build, taking into account any dependency
changes that are part of the change request (version updates, etc.).
This would also have the positive side effect of users no longer having
to scan through tfm_sw_requirement.rst to see what they don't have installed,
or parse build failures for missing module names.
I'm happy to make a new change request adding a requirements.txt file,
and update
the documentation accordingly, but t's not clear to me how to propose
the required
changes to the CI setup?
Best regards,
Kevin Townsend
Hi,
I made some changes to the tfm_ns_interface_ functions.
They have common implementations that call os_wrapper_ functions.
With these changes, RTOSes only need to implement the OS dependent functions defined in os_wrappers rather than the tfm_ns_interface_ functions.
There are several changes with a same topic:
https://review.trustedfirmware.org/q/topic:%22refine_ns_interface_functions…
Please help on reviewing. Thanks.
- Kevin
[from thread: RE: Adding a platform specific tfm_svc_number_t]
Hi Andrej,
Please note that non-secure SVC handling is independent of secure SVC handling - the two are implemented separately in the code base and hardware resources are banked for their execution.
The original discussion is about secure SVC handling type and functions, which are unrelated to NS RTOS dependency on (NS) SVC.
I'm starting a separate discussion thread for NS SVC occupancy to avoid blurring the lines between the two.
Please note that any example code in the TF-M repository on NS SVC handling is for demonstration purposes and not, strictly speaking, part of TF-M core implementation. It shows how a non-secure privileged entity needs to register a client ID to the SPM on task creation, if multiple client IDs are managed by the RTOS. Whether a specific implementation uses SVC or another method for running the corresponding privileged code is out of scope of the design, only one possible option is shown, but this is an RTOS-specific problem.
Meaning that in an RTOS where the adaptation layer mustn't use SVC and is relying on some other method, there's no design limitation in TF-M that is in conflict with that - the implementation can be adjusted in line with the RTOS's method of choice, but where the NS RTOS has no such restriction, the adaptation layer can rely on SVC for this feature.
Thanks
Miklos
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Andrej Butok via TF-M
Sent: 26 July 2019 08:29
To: Ken Liu (Arm Technology China) <Ken.Liu(a)arm.com>; DeMars, Alan <ademars(a)ti.com>
Cc: tf-m(a)lists.trustedfirmware.org
Subject: Re: [TF-M] Adding a platform specific tfm_svc_number_t
Just another use-case,
FreeRTOS is using the non-secure SVC. It does not expect that it may be used by somebody else (not RTOS).
Ideally, if TFM will not occupy SVC.
-----Original Message-----
From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of Ken Liu (Arm Technology China) via TF-M
Sent: Friday, July 26, 2019 3:49 AM
To: DeMars, Alan <ademars(a)ti.com>; tf-m(a)lists.trustedfirmware.org
Cc: nd <nd(a)arm.com>
Subject: Re: [TF-M] Adding a platform specific tfm_svc_number_t
Hi Alan,
Can you share us your usage details? This could help us on defining the svc number things you mentioned.
Thanks.
-Ken
> -----Original Message-----
> From: TF-M <tf-m-bounces(a)lists.trustedfirmware.org> On Behalf Of
> DeMars, Alan via TF-M
> Sent: Friday, July 26, 2019 6:59 AM
> To: tf-m(a)lists.trustedfirmware.org
> Subject: [TF-M] Adding a platform specific tfm_svc_number_t
>
> I need to define platform specific SPM APIs that will be invoked by our SPs.
>
> Is there a convention for 'cleanly' adding platform specific SVC
> enumerations to the tfm_svc_number_t typedef in tfm_svc.h as well as
> platform specific 'case's to SVCHandler_main() and/or SVC_Handler_IPC()?
>
> Alan
>
>
> --
> TF-M mailing list
> TF-M(a)lists.trustedfirmware.org
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> s.trustedfirmware.org%2Fmailman%2Flistinfo%2Ftf-m&data=02%7C01%7Ca
> ndrey.butok%40nxp.com%7C42c1df29f3b84ac62f5708d7116b749e%7C686ea1d3bc2
> b4c6fa92cd99c5c301635%7C0%7C0%7C636997025530401902&sdata=vO0tq34jt
> zFFn9D3cnrDP3a4fnrkq4h5jvzZmob2HnU%3D&reserved=0
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.tru…
--
TF-M mailing list
TF-M(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/tf-m
Hi,
Several patches for code restructure is coming. Before I post the gerrit items, I want to collect your feedback on this. These changes contain:
- Move header files into dedicated directory for easy include, and clean the included headers in sources;
- Change some files' name to let them make more sense.
- Move SPM related files into 'spm' folder instead of putting them in 'core'.
- Move some interface files into 'ns_callable' since they are interfaces.
- Remove 'ipc' folder after all files in it are well arranged.
I will try to do these patches together so they can be merged together.
But before that I want to request for comments about this, feel free to reply in this thread or comment on the task (add yourself if you are missing as subscribers):
https://developer.trustedfirmware.org/T426
BR
/Ken
