I don't understand the cache specification well enough to comment, but perhaps Andrew or Wedson can help or give some background?
On Mon, 8 Jun 2020 at 11:11, Achin Gupta via Hafnium < hafnium@lists.trustedfirmware.org> wrote:
Hi All,
In Arm, we are experimenting with running OP-TEE under Hafnium as the SPMC in S-EL2. We have been debugging this Stage 2 fault that OP-TEE runs into a during a test to share memory (xtest 1003). It seems this is due to a bug in Hafnium but want to be sure before posting a fix. Some thoughts below to this end. Apologies for the verbosity but I hope you will appreciate it is required.
The fault occurs when OP-TEE tries to access a memory region that was shared with it by the OP-TEE driver in Linux i.e. the driver has called FFA_MEM_SHARE to share the memory, OP-TEE has called FFA_MEM_RETRIEVE_REQ to map it in its S2 and Hf has called FFA_MEM_RETRIVE_RESP to describe the IPA range to OP-TEE. So, the S2 tables are created correctly before OP-TEE tries to use them.
The S2 fault is a L3 Translation fault. The L3 descriptor in S2 tables is NULL when the fault occurs. So this makes sense. This also implies that the translation is not cached in the TLBs.
The key thing is that the fault only occurs when cache state modelling is turned on in the FVP_Base_RevC-2xAEMv8A model we are using for development. The fault occurs both when the S2 tables are created and accessed on the same PE as well as different PEs. It does not matter whether the PEs are in the same or different clusters. The fault occurs both with and without a Hypervisor (Hf) in the Normal world. So presence of Hf in EL2 is not a factor.
We noticed that Hf marks its internal memory as outer-shareable. See [1] and [2]. It uses inner-shareable for S2 PTWs though. See [3]. This is a mismatch of memory attributes as per Page 2563 in ARM DDI 0487F.b. The start of the text is quoted below.
"The rules about mismatched attributes given in Mismatched memory attributes…”
And indeed, the fault is not seen if we mark Hf’s internal memory as inner shareable to match the PTWs. The DSBs after creating the S2 tables in [4] are for inner-shareable access types. It seems that the inner-shareable PTW is unable to observe the outer shareable page table write. Using the inner shareable attributes for the internal memory makes the write observable.
Alternatively, if we change shareability of PTWs in VTCR_EL2 to outer shareable then the fault is no longer observed. It is not clear how the PTWs and page table writes are synchronised in this case without a DSB OSH. This is not a violation of the architecture afaiu.
It seems that it would be worth aligning these attributes.
The next bit is why Hf uses the outer shareable attribute for internal memory in the first place. The recommendation seems to be to use inner-shareable. See [5] and [6].
So we are wondering if this should be fixed too. Please let me know if we have misunderstood anything so far. Happy to post a patch if not or provide more information.
Cheers, Achin
[1] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [2] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/mm.c#1043 [3] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [4] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [5] "Shareable Normal memory” in Pg. 154 in ARM DDI 0487F.b [6] https://linux-arm-kernel.infradead.narkive.com/RZHvk1cT/question-how-can-we-...
-- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium
Yeah, someone asked about this a while ago, maybe Marc? But yeah, there's no good reason that it isn't inner shareable, it's been that way for a long time and I expect it is because the shareability domains weren't understood at the time, given their vague definitions, and it was needed to cover all the cores. However, we now understand that inner shareable is intended to cover all the cores so Hf should switch to inner shareable.
IIRC, for stage 2, the shareability is left for EL1 to decide in case that could cause any issues with mismatching?
On Mon, 8 Jun 2020 at 11:19, Andrew Walbran qwandor@google.com wrote:
I don't understand the cache specification well enough to comment, but perhaps Andrew or Wedson can help or give some background?
On Mon, 8 Jun 2020 at 11:11, Achin Gupta via Hafnium < hafnium@lists.trustedfirmware.org> wrote:
Hi All,
In Arm, we are experimenting with running OP-TEE under Hafnium as the SPMC in S-EL2. We have been debugging this Stage 2 fault that OP-TEE runs into a during a test to share memory (xtest 1003). It seems this is due to a bug in Hafnium but want to be sure before posting a fix. Some thoughts below to this end. Apologies for the verbosity but I hope you will appreciate it is required.
The fault occurs when OP-TEE tries to access a memory region that was shared with it by the OP-TEE driver in Linux i.e. the driver has called FFA_MEM_SHARE to share the memory, OP-TEE has called FFA_MEM_RETRIEVE_REQ to map it in its S2 and Hf has called FFA_MEM_RETRIVE_RESP to describe the IPA range to OP-TEE. So, the S2 tables are created correctly before OP-TEE tries to use them.
The S2 fault is a L3 Translation fault. The L3 descriptor in S2 tables is NULL when the fault occurs. So this makes sense. This also implies that the translation is not cached in the TLBs.
The key thing is that the fault only occurs when cache state modelling is turned on in the FVP_Base_RevC-2xAEMv8A model we are using for development. The fault occurs both when the S2 tables are created and accessed on the same PE as well as different PEs. It does not matter whether the PEs are in the same or different clusters. The fault occurs both with and without a Hypervisor (Hf) in the Normal world. So presence of Hf in EL2 is not a factor.
We noticed that Hf marks its internal memory as outer-shareable. See [1] and [2]. It uses inner-shareable for S2 PTWs though. See [3]. This is a mismatch of memory attributes as per Page 2563 in ARM DDI 0487F.b. The start of the text is quoted below.
"The rules about mismatched attributes given in Mismatched memory attributes…”
And indeed, the fault is not seen if we mark Hf’s internal memory as inner shareable to match the PTWs. The DSBs after creating the S2 tables in [4] are for inner-shareable access types. It seems that the inner-shareable PTW is unable to observe the outer shareable page table write. Using the inner shareable attributes for the internal memory makes the write observable.
Alternatively, if we change shareability of PTWs in VTCR_EL2 to outer shareable then the fault is no longer observed. It is not clear how the PTWs and page table writes are synchronised in this case without a DSB OSH. This is not a violation of the architecture afaiu.
It seems that it would be worth aligning these attributes.
The next bit is why Hf uses the outer shareable attribute for internal memory in the first place. The recommendation seems to be to use inner-shareable. See [5] and [6].
So we are wondering if this should be fixed too. Please let me know if we have misunderstood anything so far. Happy to post a patch if not or provide more information.
Cheers, Achin
[1] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [2] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/mm.c#1043 [3] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [4] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [5] "Shareable Normal memory” in Pg. 154 in ARM DDI 0487F.b [6] https://linux-arm-kernel.infradead.narkive.com/RZHvk1cT/question-how-can-we-...
-- Hafnium mailing list Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium
On 9 Jun 2020, at 09:40, Andrew Scull <ascull@google.commailto:ascull@google.com> wrote:
Yeah, someone asked about this a while ago, maybe Marc? But yeah, there's no good reason that it isn't inner shareable, it's been that way for a long time and I expect it is because the shareability domains weren't understood at the time, given their vague definitions, and it was needed to cover all the cores. However, we now understand that inner shareable is intended to cover all the cores so Hf should switch to inner shareable.
IIRC, for stage 2, the shareability is left for EL1 to decide in case that could cause any issues with mismatching?
1. I think you are talking about the shareability attribute of the PA being accessed. It is a Hf implementation choice to let EL1 drive this.
2. The mismatch is between the shareabilty attributes Hf uses for the memory (in its S1 regime) to create the S2 tables and for the S2 PTW (VTCR_EL2).
1 & 2 are independent.
On Mon, 8 Jun 2020 at 11:19, Andrew Walbran <qwandor@google.commailto:qwandor@google.com> wrote: I don't understand the cache specification well enough to comment, but perhaps Andrew or Wedson can help or give some background?
On Mon, 8 Jun 2020 at 11:11, Achin Gupta via Hafnium <hafnium@lists.trustedfirmware.orgmailto:hafnium@lists.trustedfirmware.org> wrote: Hi All,
In Arm, we are experimenting with running OP-TEE under Hafnium as the SPMC in S-EL2. We have been debugging this Stage 2 fault that OP-TEE runs into a during a test to share memory (xtest 1003). It seems this is due to a bug in Hafnium but want to be sure before posting a fix. Some thoughts below to this end. Apologies for the verbosity but I hope you will appreciate it is required.
The fault occurs when OP-TEE tries to access a memory region that was shared with it by the OP-TEE driver in Linux i.e. the driver has called FFA_MEM_SHARE to share the memory, OP-TEE has called FFA_MEM_RETRIEVE_REQ to map it in its S2 and Hf has called FFA_MEM_RETRIVE_RESP to describe the IPA range to OP-TEE. So, the S2 tables are created correctly before OP-TEE tries to use them.
The S2 fault is a L3 Translation fault. The L3 descriptor in S2 tables is NULL when the fault occurs. So this makes sense. This also implies that the translation is not cached in the TLBs.
The key thing is that the fault only occurs when cache state modelling is turned on in the FVP_Base_RevC-2xAEMv8A model we are using for development. The fault occurs both when the S2 tables are created and accessed on the same PE as well as different PEs. It does not matter whether the PEs are in the same or different clusters. The fault occurs both with and without a Hypervisor (Hf) in the Normal world. So presence of Hf in EL2 is not a factor.
We noticed that Hf marks its internal memory as outer-shareable. See [1] and [2]. It uses inner-shareable for S2 PTWs though. See [3]. This is a mismatch of memory attributes as per Page 2563 in ARM DDI 0487F.b. The start of the text is quoted below.
"The rules about mismatched attributes given in Mismatched memory attributes…”
And indeed, the fault is not seen if we mark Hf’s internal memory as inner shareable to match the PTWs. The DSBs after creating the S2 tables in [4] are for inner-shareable access types. It seems that the inner-shareable PTW is unable to observe the outer shareable page table write. Using the inner shareable attributes for the internal memory makes the write observable.
Alternatively, if we change shareability of PTWs in VTCR_EL2 to outer shareable then the fault is no longer observed. It is not clear how the PTWs and page table writes are synchronised in this case without a DSB OSH. This is not a violation of the architecture afaiu.
It seems that it would be worth aligning these attributes.
The next bit is why Hf uses the outer shareable attribute for internal memory in the first place. The recommendation seems to be to use inner-shareable. See [5] and [6].
So we are wondering if this should be fixed too. Please let me know if we have misunderstood anything so far. Happy to post a patch if not or provide more information.
Cheers, Achin
[1] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [2] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/mm.c#1043 [3] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [4] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch6... [5] "Shareable Normal memory” in Pg. 154 in ARM DDI 0487F.b [6] https://linux-arm-kernel.infradead.narkive.com/RZHvk1cT/question-how-can-we-...
-- Hafnium mailing list Hafnium@lists.trustedfirmware.orgmailto:Hafnium@lists.trustedfirmware.org https://lists.trustedfirmware.org/mailman/listinfo/hafnium
hafnium@lists.trustedfirmware.org
-
Achin Gupta -
Andrew Scull -
Andrew Walbran