Hi,
I'm testing with Hafnium as SPMC at S-EL2 and OP-TEE as an SP at S-EL1 on
QEMU v7.0.0. I've run into a few problems and fixed most of them.
I believe the setup is similar to what Shiju is using in this mail thread
https://lists.trustedfirmware.org/archives/list/hafnium@lists.trustedfirmwa…
My setup can be duplicated with:
repo init -u https://github.com/jenswi-linaro/manifest.git -m qemu_v8.xml \
-b qemu_sel2
repo sync -j8
(cd hafnium && git submodule init && git submodule update)
cd build
make -j8 toolchains
make -j8 all
make run-only
With this xtest -x 1034 passes, xtest 1034 often causes
ERROR: Data abort: pc=0xe1198b8, esr=0x96000006, ec=0x25, far=0x9c
Panic: EL2 exception
Xtest runs dreadfully slow, I haven't investigated why yet, but at
least it works.
This is based on patches provided by Olivier at:
[1] https://review.trustedfirmware.org/c/TF-A/trusted-firmware-a/+/16412/2
[2] https://review.trustedfirmware.org/c/hafnium/hafnium/+/16323/7
I've also encountered the problem cache maintenance problem Shiju
described in the mail thread above:
NOTICE: Trapped access to system register write: op0=1, op1=0, crn=7,
crm=14, op2=2, rt=11.
It can be worked around by compiling OP-TEE with
CFG_CORE_WORKAROUND_NSITR_CACHE_PRIME=n. I'm pretty sure we do dcache
clean+inv elsewhere so I'm surprised it fails here. Is Hafnium expected
to block dcache clean+inv?
For Hafnium I've added two patches on top of [2], available at
https://github.com/jenswi-linaro/hafnium/tree/qemu_sel2:
- 79b4d2cbe06e SPMC: add missing ME initialization for secondary cores
- 659c79d5eacf feat(mm): fix FEAT_LPA workaround
For TF-A I've added a few patches on top of [1], available at
https://github.com/jenswi-linaro/arm-trusted-firmware/tree/qemu_sel2:
- a040396cae9e feat(qemu): add tos-fw-config for sel2 spmc
- 4f7d91723485 fix(qemu): change TOS_FW_CONFIG_NAME value
- fbfc9a222c7f spmd_smc_handler() add s/ns state to SMC traces
- ca65081b9cdc feat(sptool): add dependency to SP image
- b1e1b46a0680 fix(qemu): restore code to added needed psci nodes
For OP-TEE I've also added a few patches, available at
https://github.com/jenswi-linaro/optee_os/tree/qemu_sel2:
- 1057def23777 plat-vexpress: sel2 spmc: update for hafnium
- f18a54ed3524 core: ffa: use hvc instead of smc with S-EL2
- d18bbc92f7c1 core: mobj_ffa_add_pages_at() trust addresses from SPMC
There's also one patch for QEMU on top of v7.0.0, available at:
https://github.com/jenswi-linaro/qemu/tree/qemu_sel2
- 0c1e39672dcb Read PS bits from VTCR_EL2
The QEMU problem is fixed in v.7.1.0, but I can't get that version of
QEMU to work with TF-A. I guess it's because of yet another new CPU
feature since I'm running with "-cpu max".
I'll try to upstream the Hafnium and TF-A patches that make sense on
their own.
What's the plan with the interrupt controller?
How will OP-TEE be able to handle secure interrupts?
The hafnium git pulls in a few git submodules and even the source code
for a Linux kernel.
I guess this is useful in your internal CI setup, but when used
isolated as in my setup it makes no sense at all.
It would also be nice to be able to build with an external toolchain.
I hope this is a temporary situation, I don't see why Hafnium should
be pickier about toolchain than for instance TF-A.
Speaking of building, I haven't been able to figure out how to build
only for the QEMU variant I need so right now I'm building for
everything and that takes a bit longer than necessary.
I'm going to maintain the setup above as long as it's relevant to me. I may
add more patches on the branches or even rebase as needed. So if anyone is
using this, keep in mind that my branches may change without warning.
Thanks,
Jens
Hi,
1/ About:
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
I wonder if this is a realistic value suggesting SMMU_S_IDR1.S_SIDSIZE=24?
In which case the stream table size would grow up to 1GB.
Can you confirm this parameter value on your platform?
Also can you confirm the smmu base address on your platform, as it may be that hafnium probes the smmu io space from a wrong address?
2/ can you tell which commit hash you're using for hafnium?
We've observed random stack overflows corrupting data sections (in this case the smmu static data) because of stack size limitation.
Can you confirm you have this change in your tree:
https://git.trustedfirmware.org/hafnium/hafnium.git/commit/?id=64b421e19a06…
Thanks, Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 28 November 2022 10:48
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Cc: 赵哲(为哲) <weizhe.zz(a)alibaba-inc.com>
Subject: SMMU Configuration
Hi,
when I enabled SMMU by set the config file as follows：
aarch64_toolchains("secure_aem_v8a_fvp") {
cpu = "cortex-a57"
origin_address = "0x06000000"
boot_flow = "//src/boot_flow:spmc"
console = "//src/arch/aarch64/pl011"
iommu = "//src/arch/aarch64/arm_smmuv3"
gic_version = 3
gicd_base_address = "0x2e000000"
gicr_base_address = "0x2e140000"
gicr_frames = 8
heap_pages = 100000
max_cpus = 8
max_vms = 16
# branch_protection = "standard"
toolchain_args = {
plat_ffa = "//src/arch/aarch64/plat/ffa:spmc"
plat_psci = "//src/arch/aarch64/plat/psci:spmc"
plat_interrupts = "//src/arch/aarch64/plat/interrupts:gicv3"
plat_prng = "//src/arch/aarch64/plat/prng:prng"
secure_world = "1"
pl011_base_address = "0x2A400000"
smmu_base_address = "0x3FC00000"
smmu_memory_size = "0x400000"
# enable_mte = "1"
plat_log_level = "LOG_LEVEL_VERBOSE"
}
}
I encountered a problem，The log is as follows， there is not enough memory for init SMMU， I have define “heap_pages to 100000”， but it's still not enough，Do you have any suggestions?
VERBOSE: SMMUv3 mapped at 000000003fc00000
VERBOSE: SMMUv3: write to (S_)GBPA
VERBOSE: SMMUv3: write to (S_)CR0
VERBOSE: SMMUv3: Input Addr: 48-bits, Output Addr: 48-bits
VERBOSE: SMMUv3: Total CMDQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800043000 for CMDQ
VERBOSE: SMMUv3: write to (S_)CMDQ_BASE
VERBOSE: SMMUv3: write to (S_)CMDQ_CONS, (S_)CMDQ_PROD
VERBOSE: SMMUv3: Total EVTQ entries: 524288
VERBOSE: SMMUv3: Memory allocated at 0000008800844000 for EVTQ
VERBOSE: SMMUv3: write to (S_)EVTQ_BASE
VERBOSE: SMMUv3: write to (S_)EVTQ_PROD,(S_)EVTQ_CONS
VERBOSE: SMMUv3 Total StreamTable entries: 16777216
ERROR: SMMUv3: Could not allocate memory for stream table entries
ERROR: SMMUv3: Failed to initialize driver
Panic: Could not initialize IOMMUs.
Cc Hafnium list.
________________________________
From: Olivier Deprez <Olivier.Deprez(a)arm.com>
Sent: 25 November 2022 17:41
To: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Subject: Re: 回复：run hafnium as spmc at s-el2
Hi,
See answers inline [OD]
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 25 November 2022 03:18
To: Olivier Deprez <Olivier.Deprez(a)arm.com>
Subject: 回复：run hafnium as spmc at s-el2
Hello，expert
I do intend to use OP-TEE as SEL1 payload / secure partition with hafnium running on SEL2 and run linux in the normal world.
In my environment，when TF-A and Hafnium boot, only a single primary core runs. After the os starts, the secondary core is booted by bl31.
Now, I have some other questions ：
1）When entering hafnium to initialize gic, I get the following error (the first two lines are my debug prints) :
populate_redist_base_addrs:375,typer_reg:844425014019047,current_rdist_frame:2e200000
populate_redist_base_addrs:375,typer_reg:0,current_rdist_frame:2e220000
ERROR: Data abort: pc=0xff20558c, esr=0x96000007, ec=0x25, far=0x2e240008
The gic appears to be missing the corresponding page table configuration.
Hafnium runs on s-el2, and according to the arm specification, it has a set of page tables for itself and only does stage 1 translation.
So how to configure the page table for hafnium to include the address space where the gic resides?
[OD] Are you using the Base FVP platform?
The SPMC run address suggests this might be an Infra/Neoverse platform?
Did you take care of creating a new platform, or update the FVP defaults for the GICD/GICR system addresses?
https://git.trustedfirmware.org/hafnium/project/reference.git/tree/BUILD.gn…
2）Is smmu necessary for hanfium to run as SPMC, how much running memory does hafnium need？
And how to specify the address space where its running memory resides?
[OD] This depends first if your platform implements an SMMU and it supports secure S2 translation (from >=SMMUv3.2).
Second condition is if you need to protect VMs from a peripheral upstream from this SMMU.
If not, you can omit the SMMU driver by removing the iommu property:
https://git.trustedfirmware.org/hafnium/project/reference.git/tree/BUILD.gn…
3）When TF-A and Hafnium boot, only a single primary core runs. How to configure the cpu core to boot the hafnium?
[OD] When the platform resets and starts, a single core is up (aka the primary core) and TF-A boots on this core through BL1/BL2/BL31 stages.
From there BL31 launches BL32/Hafnium on the same core.
There is no configurability from a SW perspective. It is up to the platform itself to tie onto which core the system boots.
If running on a model, that *may* be controlled by a model parameter.
In addition, there are multiple cores at runtime. How does hanfium switch cores to switch the corresponding secure partions?
How does core switching relate to vCPU switching？
[OD] Hafnium is controlled by a normal world driver e.g. the linux upstream FF-A driver: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drive… and a TEE driver e.g. OP-TEE driver https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drive…
The primary scheduler is linux' and Hafnium is not performing scheduling decisions.
As an example, a direct request on CPU1 emitted from the normal world through an SMC, is processed by the SPMC and delivered to the SP(OP-TEE) on vCPU1.
Thank you very much～
------------------------------------------------------------------
发件人：Olivier Deprez <Olivier.Deprez(a)arm.com>
发送时间：2022年11月25日(星期五) 03:24
收件人：梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
抄 送：hafnium(a)lists.trustedfirmware.org <hafnium(a)lists.trustedfirmware.org>
主 题：Re: run hafnium as spmc at s-el2
Hi Mei Jianqiang,
> In my environment, hafnium is running as spmc in s-el2, and my question is as follows:
Can you tell a bit more on your setup, if possible, do you intend to use OP-TEE as SEL1 payload / secure partition?
Are you running linux in the normal world?
> 1) In the spmc_optee_sp_manifest.dts file, there is an attribute load_address = <0x0 0xFF200000>.
> What do the two parts of load_address mean respectively? Any details on other configurations in the dts file?
In the TF-A tree plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts is a sample device tree consumed by Hafnium/SPMC to describe the system properties and declare secure partitions.
Please refer to https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit…
The load_address field specifies the location at which the SPMC finds a 'partition package'. It consists of a partition's DT blob and a partition image.
https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit…
In this sample 0x0 is the high 32b part of the physical address and 0xFF200000 the lower 32 bits.
> 2) If only one core is available during the startup of hafnium, how to configure it?
> In this case, when I set CPUS_PER_CLUSTER 1 in fvp-defs.dtsi file of ATF reference code, the building error is: Duplicate node name /cpus/cpu@20000. Why?
I don't believe you should do change directly in this file, but rather provide the system topology through the TF-A command line by using FVP_MAX_CPUS_PER_CLUSTER/FVP_CLUSTER_COUNT/FVP_MAX_PE_PER_CPU
The device tree should describe cpus as they exist when the system is booted at run-time.
When TF-A and Hafnium boot, only a single primary core runs anyways so is this really what you intend to do?
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 24 November 2022 11:33
To: hafnium-owner <hafnium-owner(a)lists.trustedfirmware.org>
Subject: run hafnium as spmc at s-el2
Hello expert,
I'm a developer at Alibaba Cloud, and I'm having some problems using hafnium.
In my environment, hafnium is running as spmc in s-el2, and my question is as follows:
1) In the spmc_optee_sp_manifest.dts file, there is an attribute load_address = <0x0 0xFF200000>.
What do the two parts of load_address mean respectively? Any details on other configurations in the dts file?
2) If only one core is available during the startup of hafnium, how to configure it?
In this case, when I set CPUS_PER_CLUSTER 1 in fvp-defs.dtsi file of ATF reference code, the building error is: Duplicate node name /cpus/cpu@20000. Why?
Looking forward to your reply. Thank you very much~
Hi Mei Jianqiang,
> In my environment, hafnium is running as spmc in s-el2, and my question is as follows:
Can you tell a bit more on your setup, if possible, do you intend to use OP-TEE as SEL1 payload / secure partition?
Are you running linux in the normal world?
> 1) In the spmc_optee_sp_manifest.dts file, there is an attribute load_address = <0x0 0xFF200000>.
> What do the two parts of load_address mean respectively? Any details on other configurations in the dts file?
In the TF-A tree plat/arm/board/fvp/fdts/fvp_spmc_optee_sp_manifest.dts is a sample device tree consumed by Hafnium/SPMC to describe the system properties and declare secure partitions.
Please refer to https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit…
The load_address field specifies the location at which the SPMC finds a 'partition package'. It consists of a partition's DT blob and a partition image.
https://trustedfirmware-a.readthedocs.io/en/latest/components/secure-partit…
In this sample 0x0 is the high 32b part of the physical address and 0xFF200000 the lower 32 bits.
> 2) If only one core is available during the startup of hafnium, how to configure it?
> In this case, when I set CPUS_PER_CLUSTER 1 in fvp-defs.dtsi file of ATF reference code, the building error is: Duplicate node name /cpus/cpu@20000. Why?
I don't believe you should do change directly in this file, but rather provide the system topology through the TF-A command line by using FVP_MAX_CPUS_PER_CLUSTER/FVP_CLUSTER_COUNT/FVP_MAX_PE_PER_CPU
The device tree should describe cpus as they exist when the system is booted at run-time.
When TF-A and Hafnium boot, only a single primary core runs anyways so is this really what you intend to do?
Regards,
Olivier.
________________________________
From: 梅建强(禹夜) <meijianqiang.mjq(a)alibaba-inc.com>
Sent: 24 November 2022 11:33
To: hafnium-owner <hafnium-owner(a)lists.trustedfirmware.org>
Subject: run hafnium as spmc at s-el2
Hello expert,
I'm a developer at Alibaba Cloud, and I'm having some problems using hafnium.
In my environment, hafnium is running as spmc in s-el2, and my question is as follows:
1) In the spmc_optee_sp_manifest.dts file, there is an attribute load_address = <0x0 0xFF200000>.
What do the two parts of load_address mean respectively? Any details on other configurations in the dts file?
2) If only one core is available during the startup of hafnium, how to configure it?
In this case, when I set CPUS_PER_CLUSTER 1 in fvp-defs.dtsi file of ATF reference code, the building error is: Duplicate node name /cpus/cpu@20000. Why?
Looking forward to your reply. Thank you very much~
Hello,
We get the following error in the non-secure terminal in the QEMU, while booting the normal world with hafnium(SPMC) at S-EL2 and SP at S-EL1.
NOTICE: Trapped access to system register write: op0=1, op1=0, crn=7, crm=14, op2=2, rt=11.
NOTICE: Injecting Unknown Reason exception into VM 0x8001.
Above system register found as DC CISW.
It found that this happened with the recent hafnium change "feat(interrupts): preferred managed exit signal"
Thanks.
