+Andrew Scull <ascull(a)google.com> who kindly answered this offline.
On Fri, May 22, 2020 at 9:38 AM Olivier Deprez via Hafnium <
hafnium(a)lists.trustedfirmware.org> wrote:
> Hi,
>
> Hafnium sets CPTR_EL2.TTA (bit 28), which traps accesses to trace system
> registers to EL2.
>
> https://git.trustedfirmware.org/hafnium/hafnium.git/tree/src/arch/aarch64/s…
>
> However CPTR_EL2 register has a different bit field definition depending
> on HCR_EL2.E2H state.
> When HCR_EL2.E2H=0 (Hafnium case) CPTR_EL2.TTA bit position is 20.
>
> Is this a slight issue needing fix?
>
It sounds like this was copied badly from the spec and should be fixed. We
don't enable VHE and we should do as the spec says.
HTH,
Serban
>
> Regards,
> Olivier.
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
Hi,
I think the subtlety resides in PAR_EL1.NS (bit 9). When a secure hypervisor performs AT S1E1R, then the resulting translation output is the SEL1 IPA (including the NS bit state).
I did the following trials with SPM executing AT S1E1R for a known region mapped by an SP:
-the SP does not map the region =>PAR_EL1.F=1
-the SP maps the regions secure =>PAR_EL1.NS=0
-the SP maps the region non-secure =>PAR_EL1.NS=1
This permits checking the SP region security state without trusting the SP.
On a related topic I also noticed VTCR_EL2.NSA and VTCR_EL2.NSW (and VSTCR_EL2.SA/SW=0).
With default values of 0, this means Stage-1 translations always result to Secure IPA space (also in the case S-EL1 MMU is disabled). Hence I think PTWs always go through VSTTBR.
I guess the two VTCR_EL2 bits shall be set if we adopt a model where:
-any NS shared mapping resolves to VTTBR
-remaining SP mappings resolve to VSTTBR
Regards,
Olivier.
________________________________________
From: Hafnium <hafnium-bounces(a)lists.trustedfirmware.org> on behalf of Raghu Krishnamurthy via Hafnium <hafnium(a)lists.trustedfirmware.org>
Sent: 08 June 2020 17:11
To: hafnium(a)lists.trustedfirmware.org
Subject: Re: [Hafnium] Determine security state of a memory region
Never mind question 2 below. Educated myself that there is no NS bit in
the stage 2 tables which i had assumed. I think Andrew's suggestion
should work too, however it does have the issue that the SP may see
incoherent data if there is an NS attribute mismatch. With option 4 i
suggested, you would get a fault when incorrect attributes are used. No
strong preference though.
-Raghu
On 6/8/20 7:10 AM, Raghu Krishnamurthy via Hafnium wrote:
> Hi Achin,
>
> Thanks for the explanation. Agree that this is a problem.
>
> 1) Pasting Andrew's question for continuity: Is it possible that the
> SPMC could map it in both the secure and non-secure stage 2 page
> tables, so the SP can decide which security state to map it in at stage
> 1? Or am I misunderstanding how this works?
>
> 2) Perhaps i'm missing something, but I'm surprised by the assumption
> that a given SP(which runs only in secure world) will have multiple S2
> tables. How would the SPMC decide whether to point VSTTBR_EL2 to the
> secure or non-secure S2 tables when entering an SP? Unless the way this
> works architecturally is that the stage 1 walk's NS attribute is used to
> select VSTTBR_EL2 or VTTBR_EL2. That seems backward.
>
> 3) Have you considered option 4, where an FFA_MEM_SHARE between 2 SP's
> shall always share only pages mapped as secure and between SP and a VM
> share only pages mapped as non-secure? Or is this not an option? This
> will make the SPMC design simpler and puts the onus on lower privileged
> SP's to know if they are communicating with secure world or NS world
> components. This obviously incurs overhead from the SP's point of view
> but if the more important goal is to keep S-EL2 simpler, this might be a
> good option.
>
> Thanks
> Raghu
>
> On 6/8/20 2:37 AM, Andrew Walbran wrote:
>> On Sun, 7 Jun 2020 at 16:46, Achin Gupta via Hafnium
>> <hafnium(a)lists.trustedfirmware.org
>> <mailto:hafnium@lists.trustedfirmware.org>> wrote:
>>
>> Hi Raghu,
>>
>> Howdy! CIL…
>>
>> > On 4 Jun 2020, at 16:21, Raghu K via Hafnium
>> <hafnium(a)lists.trustedfirmware.org
>> <mailto:hafnium@lists.trustedfirmware.org>> wrote:
>> >
>> > Hi Achin,
>> >
>> > Would you mind elaborating more on why the SPM needs to determine
>> the security state and why it is important to do this without
>> trusting the SP? When you say SPM, it sounds like you are talking
>> about the SPMD running in EL3 for ex., that is not a part of the
>> SPMC which perhaps runs as S-EL2 and the SPMD may need to know this
>> to figure out how to map a particular physical page. Is that the use
>> case you are thinking about?
>>
>> So this is in the context of PSA FF-A Memory management ABIs. Also,
>> I have the S-EL2 SPMC case in mind. SPMD in EL3 does not participate
>> in memory management in this case when it comes to managing any
>> architectural state i.e. translation tables, control regs etc
>>
>> Say, a SP0 invokes FFA_MEM_SHARE to share a single page A with SP1.
>> The SPMC would need to map page A in SP’s stage 2 tables. To do
>> this, it would need to determine whether the IPA of page A belongs
>> to the Secure or Non-secure IPA space. This is under the assumption
>> that some memory ranges in SP0’s IPA space will be Non-secure.
>>
>> IMO, this information can be determined in one of the following ways:
>>
>> 1. Perform PTW in SW to determine whether IPA is mapped in the
>> tables referenced by VSTTBR_EL2 or VTTBR_EL2. I am assuming the SPMC
>> maintains separate S2 translations for the Secure and NS address
>> spaces.
>>
>> 2. Through an internal data structure which tracks the attributes of
>> a memory region assigned to a guest.
>>
>> 3. SP0 specifies the security state of page A in FFA_MEM_SHARE. The
>> spec does not cover this currently. However, the SPMC cannot trust
>> that the SP0 is providing the right security state and must verify
>> this independently anyways.
>>
>> 1 seems clunky. 2 is not done in upstream Hf. 3 does not really help.
>>
>> I think I had misunderstood that a AT* instruction could be used.
>> There do not seem to be any in the Arm ARM that only perform a IPA
>> to PA i.e. a S2 translation.
>>
>> So I am wondering what can be done to solve this problem assuming we
>> agree that this is a problem in the first place.
>>
>>
>> Is it possible that the SPMC could map it in both the secure and
>> non-secure stage 2 page tables, so the SP can decide which security
>> state to map it in at stage 1? Or am I misunderstanding how this works?
>>
>>
>> Hth,
>>
>> Cheers,
>> Achin
>>
>> >
>> > Thanks
>> > Raghu
>> >
>> > On 6/4/20 3:07 AM, Achin Gupta via Hafnium wrote:
>> >> Hi All,
>> >>
>> >> I am thinking of a scenario where a SP shares Non-secure memory
>> with one or more SPs or VMs. The NS memory region could have been
>> donated to the SP by a VM earlier (far fetched but possible).
>> >>
>> >> The question is how does the SPM determine the security state of
>> the memory region being shared by the SP.
>> >>
>> >> It is especially important that the SPM does this without
>> trusting the SP.
>> >>
>> >> I don't think it should rely on the AT* instructions. The SP
>> could change the security state of the region in S1. AFAIK, there
>> are no AT* instructions that only do S2 walks with a IPA as an input.
>> >>
>> >> So is the only option to perform a walk in both the Secure and
>> Non-secure S2 tables to determine where is the address mapped.
>> >>
>> >> This seems a bit clunky. So wondering if I am missing anything
>> and there is an easier way to do this.
>> >>
>> >> What do you reckon?
>> >>
>> >> cheers,
>> >> Achin
>> >
>> > --
>> > Hafnium mailing list
>> > Hafnium(a)lists.trustedfirmware.org
>> <mailto:Hafnium@lists.trustedfirmware.org>
>> > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>>
>> -- Hafnium mailing list
>> Hafnium(a)lists.trustedfirmware.org
>> <mailto:Hafnium@lists.trustedfirmware.org>
>> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>>
--
Hafnium mailing list
Hafnium(a)lists.trustedfirmware.org
https://lists.trustedfirmware.org/mailman/listinfo/hafnium
I don't understand the cache specification well enough to comment, but
perhaps Andrew or Wedson can help or give some background?
On Mon, 8 Jun 2020 at 11:11, Achin Gupta via Hafnium <
hafnium(a)lists.trustedfirmware.org> wrote:
> Hi All,
>
> In Arm, we are experimenting with running OP-TEE under Hafnium as the SPMC
> in S-EL2. We have been debugging this Stage 2 fault that OP-TEE runs into a
> during a test to share memory (xtest 1003). It seems this is due to a bug
> in Hafnium but want to be sure before posting a fix. Some thoughts below to
> this end. Apologies for the verbosity but I hope you will appreciate it is
> required.
>
> The fault occurs when OP-TEE tries to access a memory region that was
> shared with it by the OP-TEE driver in Linux i.e. the driver has called
> FFA_MEM_SHARE to share the memory, OP-TEE has called FFA_MEM_RETRIEVE_REQ
> to map it in its S2 and Hf has called FFA_MEM_RETRIVE_RESP to describe the
> IPA range to OP-TEE. So, the S2 tables are created correctly before OP-TEE
> tries to use them.
>
> The S2 fault is a L3 Translation fault. The L3 descriptor in S2 tables is
> NULL when the fault occurs. So this makes sense. This also implies that the
> translation is not cached in the TLBs.
>
> The key thing is that the fault only occurs when cache state modelling is
> turned on in the FVP_Base_RevC-2xAEMv8A model we are using for development.
> The fault occurs both when the S2 tables are created and accessed on the
> same PE as well as different PEs. It does not matter whether the PEs are in
> the same or different clusters. The fault occurs both with and without a
> Hypervisor (Hf) in the Normal world. So presence of Hf in EL2 is not a
> factor.
>
> We noticed that Hf marks its internal memory as outer-shareable. See [1]
> and [2]. It uses inner-shareable for S2 PTWs though. See [3]. This is a
> mismatch of memory attributes as per Page 2563 in ARM DDI 0487F.b. The
> start of the text is quoted below.
>
> "The rules about mismatched attributes given in Mismatched memory
> attributes…”
>
> And indeed, the fault is not seen if we mark Hf’s internal memory as inner
> shareable to match the PTWs. The DSBs after creating the S2 tables in [4]
> are for inner-shareable access types. It seems that the inner-shareable PTW
> is unable to observe the outer shareable page table write. Using the inner
> shareable attributes for the internal memory makes the write observable.
>
> Alternatively, if we change shareability of PTWs in VTCR_EL2 to outer
> shareable then the fault is no longer observed. It is not clear how the
> PTWs and page table writes are synchronised in this case without a DSB OSH.
> This is not a violation of the architecture afaiu.
>
> It seems that it would be worth aligning these attributes.
>
> The next bit is why Hf uses the outer shareable attribute for internal
> memory in the first place. The recommendation seems to be to use
> inner-shareable. See [5] and [6].
>
> So we are wondering if this should be fixed too. Please let me know if we
> have misunderstood anything so far. Happy to post a patch if not or provide
> more information.
>
> Cheers,
> Achin
>
> [1]
> https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch…
> [2]
> https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/mm.c#1043
> [3]
> https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch…
> [4]
> https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch…
> [5] "Shareable Normal memory” in Pg. 154 in ARM DDI 0487F.b
> [6]
> https://linux-arm-kernel.infradead.narkive.com/RZHvk1cT/question-how-can-we…
>
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
Never mind question 2 below. Educated myself that there is no NS bit in
the stage 2 tables which i had assumed. I think Andrew's suggestion
should work too, however it does have the issue that the SP may see
incoherent data if there is an NS attribute mismatch. With option 4 i
suggested, you would get a fault when incorrect attributes are used. No
strong preference though.
-Raghu
On 6/8/20 7:10 AM, Raghu Krishnamurthy via Hafnium wrote:
> Hi Achin,
>
> Thanks for the explanation. Agree that this is a problem.
>
> 1) Pasting Andrew's question for continuity: Is it possible that the
> SPMC could map it in both the secure and non-secure stage 2 page
> tables, so the SP can decide which security state to map it in at stage
> 1? Or am I misunderstanding how this works?
>
> 2) Perhaps i'm missing something, but I'm surprised by the assumption
> that a given SP(which runs only in secure world) will have multiple S2
> tables. How would the SPMC decide whether to point VSTTBR_EL2 to the
> secure or non-secure S2 tables when entering an SP? Unless the way this
> works architecturally is that the stage 1 walk's NS attribute is used to
> select VSTTBR_EL2 or VTTBR_EL2. That seems backward.
>
> 3) Have you considered option 4, where an FFA_MEM_SHARE between 2 SP's
> shall always share only pages mapped as secure and between SP and a VM
> share only pages mapped as non-secure? Or is this not an option? This
> will make the SPMC design simpler and puts the onus on lower privileged
> SP's to know if they are communicating with secure world or NS world
> components. This obviously incurs overhead from the SP's point of view
> but if the more important goal is to keep S-EL2 simpler, this might be a
> good option.
>
> Thanks
> Raghu
>
> On 6/8/20 2:37 AM, Andrew Walbran wrote:
>> On Sun, 7 Jun 2020 at 16:46, Achin Gupta via Hafnium
>> <hafnium(a)lists.trustedfirmware.org
>> <mailto:hafnium@lists.trustedfirmware.org>> wrote:
>>
>> Hi Raghu,
>>
>> Howdy! CIL…
>>
>> > On 4 Jun 2020, at 16:21, Raghu K via Hafnium
>> <hafnium(a)lists.trustedfirmware.org
>> <mailto:hafnium@lists.trustedfirmware.org>> wrote:
>> >
>> > Hi Achin,
>> >
>> > Would you mind elaborating more on why the SPM needs to determine
>> the security state and why it is important to do this without
>> trusting the SP? When you say SPM, it sounds like you are talking
>> about the SPMD running in EL3 for ex., that is not a part of the
>> SPMC which perhaps runs as S-EL2 and the SPMD may need to know this
>> to figure out how to map a particular physical page. Is that the use
>> case you are thinking about?
>>
>> So this is in the context of PSA FF-A Memory management ABIs. Also,
>> I have the S-EL2 SPMC case in mind. SPMD in EL3 does not participate
>> in memory management in this case when it comes to managing any
>> architectural state i.e. translation tables, control regs etc
>>
>> Say, a SP0 invokes FFA_MEM_SHARE to share a single page A with SP1.
>> The SPMC would need to map page A in SP’s stage 2 tables. To do
>> this, it would need to determine whether the IPA of page A belongs
>> to the Secure or Non-secure IPA space. This is under the assumption
>> that some memory ranges in SP0’s IPA space will be Non-secure.
>>
>> IMO, this information can be determined in one of the following ways:
>>
>> 1. Perform PTW in SW to determine whether IPA is mapped in the
>> tables referenced by VSTTBR_EL2 or VTTBR_EL2. I am assuming the SPMC
>> maintains separate S2 translations for the Secure and NS address
>> spaces.
>>
>> 2. Through an internal data structure which tracks the attributes of
>> a memory region assigned to a guest.
>>
>> 3. SP0 specifies the security state of page A in FFA_MEM_SHARE. The
>> spec does not cover this currently. However, the SPMC cannot trust
>> that the SP0 is providing the right security state and must verify
>> this independently anyways.
>>
>> 1 seems clunky. 2 is not done in upstream Hf. 3 does not really help.
>>
>> I think I had misunderstood that a AT* instruction could be used.
>> There do not seem to be any in the Arm ARM that only perform a IPA
>> to PA i.e. a S2 translation.
>>
>> So I am wondering what can be done to solve this problem assuming we
>> agree that this is a problem in the first place.
>>
>>
>> Is it possible that the SPMC could map it in both the secure and
>> non-secure stage 2 page tables, so the SP can decide which security
>> state to map it in at stage 1? Or am I misunderstanding how this works?
>>
>>
>> Hth,
>>
>> Cheers,
>> Achin
>>
>> >
>> > Thanks
>> > Raghu
>> >
>> > On 6/4/20 3:07 AM, Achin Gupta via Hafnium wrote:
>> >> Hi All,
>> >>
>> >> I am thinking of a scenario where a SP shares Non-secure memory
>> with one or more SPs or VMs. The NS memory region could have been
>> donated to the SP by a VM earlier (far fetched but possible).
>> >>
>> >> The question is how does the SPM determine the security state of
>> the memory region being shared by the SP.
>> >>
>> >> It is especially important that the SPM does this without
>> trusting the SP.
>> >>
>> >> I don't think it should rely on the AT* instructions. The SP
>> could change the security state of the region in S1. AFAIK, there
>> are no AT* instructions that only do S2 walks with a IPA as an input.
>> >>
>> >> So is the only option to perform a walk in both the Secure and
>> Non-secure S2 tables to determine where is the address mapped.
>> >>
>> >> This seems a bit clunky. So wondering if I am missing anything
>> and there is an easier way to do this.
>> >>
>> >> What do you reckon?
>> >>
>> >> cheers,
>> >> Achin
>> >
>> > --
>> > Hafnium mailing list
>> > Hafnium(a)lists.trustedfirmware.org
>> <mailto:Hafnium@lists.trustedfirmware.org>
>> > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>>
>> -- Hafnium mailing list
>> Hafnium(a)lists.trustedfirmware.org
>> <mailto:Hafnium@lists.trustedfirmware.org>
>> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>>
On Sun, 7 Jun 2020 at 16:46, Achin Gupta via Hafnium <
hafnium(a)lists.trustedfirmware.org> wrote:
> Hi Raghu,
>
> Howdy! CIL…
>
> > On 4 Jun 2020, at 16:21, Raghu K via Hafnium <
> hafnium(a)lists.trustedfirmware.org> wrote:
> >
> > Hi Achin,
> >
> > Would you mind elaborating more on why the SPM needs to determine the
> security state and why it is important to do this without trusting the SP?
> When you say SPM, it sounds like you are talking about the SPMD running in
> EL3 for ex., that is not a part of the SPMC which perhaps runs as S-EL2 and
> the SPMD may need to know this to figure out how to map a particular
> physical page. Is that the use case you are thinking about?
>
> So this is in the context of PSA FF-A Memory management ABIs. Also, I have
> the S-EL2 SPMC case in mind. SPMD in EL3 does not participate in memory
> management in this case when it comes to managing any architectural state
> i.e. translation tables, control regs etc
>
> Say, a SP0 invokes FFA_MEM_SHARE to share a single page A with SP1. The
> SPMC would need to map page A in SP’s stage 2 tables. To do this, it would
> need to determine whether the IPA of page A belongs to the Secure or
> Non-secure IPA space. This is under the assumption that some memory ranges
> in SP0’s IPA space will be Non-secure.
>
> IMO, this information can be determined in one of the following ways:
>
> 1. Perform PTW in SW to determine whether IPA is mapped in the tables
> referenced by VSTTBR_EL2 or VTTBR_EL2. I am assuming the SPMC maintains
> separate S2 translations for the Secure and NS address spaces.
>
> 2. Through an internal data structure which tracks the attributes of a
> memory region assigned to a guest.
>
> 3. SP0 specifies the security state of page A in FFA_MEM_SHARE. The spec
> does not cover this currently. However, the SPMC cannot trust that the SP0
> is providing the right security state and must verify this independently
> anyways.
>
> 1 seems clunky. 2 is not done in upstream Hf. 3 does not really help.
>
> I think I had misunderstood that a AT* instruction could be used. There do
> not seem to be any in the Arm ARM that only perform a IPA to PA i.e. a S2
> translation.
>
> So I am wondering what can be done to solve this problem assuming we agree
> that this is a problem in the first place.
>
Is it possible that the SPMC could map it in both the secure and non-secure
stage 2 page tables, so the SP can decide which security state to map it in
at stage 1? Or am I misunderstanding how this works?
>
> Hth,
>
> Cheers,
> Achin
>
> >
> > Thanks
> > Raghu
> >
> > On 6/4/20 3:07 AM, Achin Gupta via Hafnium wrote:
> >> Hi All,
> >>
> >> I am thinking of a scenario where a SP shares Non-secure memory with
> one or more SPs or VMs. The NS memory region could have been donated to the
> SP by a VM earlier (far fetched but possible).
> >>
> >> The question is how does the SPM determine the security state of the
> memory region being shared by the SP.
> >>
> >> It is especially important that the SPM does this without trusting the
> SP.
> >>
> >> I don't think it should rely on the AT* instructions. The SP could
> change the security state of the region in S1. AFAIK, there are no AT*
> instructions that only do S2 walks with a IPA as an input.
> >>
> >> So is the only option to perform a walk in both the Secure and
> Non-secure S2 tables to determine where is the address mapped.
> >>
> >> This seems a bit clunky. So wondering if I am missing anything and
> there is an easier way to do this.
> >>
> >> What do you reckon?
> >>
> >> cheers,
> >> Achin
> >
> > --
> > Hafnium mailing list
> > Hafnium(a)lists.trustedfirmware.org
> > https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
I'm not sure about the exact issue you're seeing, but one thing that could
be helpful for debugging is to increase the log verbosity level, by
changing the log_level in build/BUILDCONFIG.gn to LOG_LEVEL_VERBOSE and
rebuilding.
Can you also share the command you are using to make the ramdisk? Changing
the order of files in it (e.g. putting the manifest file first) might work
around the issue if it is caused by alignment.
On Mon, 8 Jun 2020 at 04:26, 王华 via Hafnium <
hafnium(a)lists.trustedfirmware.org> wrote:
>
>
>
> Hi, David :
> Thanks very much for your help.
> I check the firmware version as following command , It seems that it
> is the newest stable FW. But still failed.
> pi@raspberrypi:~$ vcgencmd bootloader_version
> Apr 16 2020 18:11:26
> version
> a5e1b95f320810c69441557c5f5f0a7f2460dfb8 (release)
>
> timestamp 1587057086
>
>
> I share my config.txt in attchment and hafnium.elf in
> https://drive.google.com/file/d/12M-qqTQTF7BampvLOZLhMDZDNuQpoiEB/view?usp=…
>
>
>
>
>
>
>
> 在 2020-06-07 22:05:15，"David Brazdil" <dbrazdil(a)google.com> 写道：
>
> Hi Wanghua,
>
>
> Happy to help, the RPi4 port should still be working. Though we do not
> continuously test it, so there is a possibility that the code has rotted a
> bit.
>
> I can share my notes on how I got it working, but you seem to have figured
> out pretty much all of it.
>
>
> You're seeing the hypervisor abort when accessing address 0x19536ee (FAR)
> - that's inside the radisk as reported above the error. The value of ESR
> suggests that this is an alignment error (the address is indeed unaligned).
> So Hafnium probably crashed trying to parse something in the ramdisk.
>
>
> Before we try anything else, could you try to update your RPi4 firmware? I
> remember that I had issues with the 'initramfs' option in config.txt - it
> would load random data from the SD card. The error that you're seeing could
> be a result of that.
> Instructions on how to update your firmware:
> https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md
>
>
> If that doesn't help, could you please attach your
> out/reference/rpi4_clang/hafnium.elf? It would be good to know which
> function is at the crashing PC 0x8b0dc.
>
>
> David
>
>
>
>
> On Sun, Jun 7, 2020 at 2:08 PM 王华 via Hafnium <
> hafnium(a)lists.trustedfirmware.org> wrote:
>
> Hi all :
>
> Has anyone brought up Hafnium on Raspberry Pi 4 board ? I tried but
> failed by the following steps:
>
> 1.Make the Hafnium RAM disk with aarch64 build Pi4 kernel , RAM disk for
> Linux , manifest.dtb.
>
> The manifest.dtb buildt by :
>
> /dts-v1/;
>
>
> / {
>
> │ hypervisor {
>
> │ │ compatible = "hafnium,hafnium";
>
> │ │ vm1 {
>
> │ │ │ debug_name = "Linux VM";
>
> │ │ │ kernel_filename = "vmlinuz";
>
> │ │ │ ramdisk_filename = "initrd.img";
>
> │ │ };
>
> │ };
>
> };
>
> 2.Copy bl31.bin hafnium.bin initrd.img to fat32 boot direction and
> configure config.txt . Power up my unit and got the uart error log :
>
> NOTICE: BL31: v2.3(debug):v2.3-109-g771c676b1
>
> NOTICE: BL31: Built : 15:49:37, Jun 1 2020
>
> INFO: Changed device tree to advertise PSCI.
>
> INFO: ARM GICv2 driver initialized
>
> INFO: BL31: Initializing runtime services
>
> INFO: BL31: cortex_a72: CPU workaround for 859971 was applied
>
> INFO: BL31: cortex_a72: CPU workaround for cve_2017_5715 was applied
>
> INFO: BL31: cortex_a72: CPU workaround for cve_2018_3639 was applied
>
> INFO: BL31: Preparing for EL3 exit to normal world
>
> INFO: Entry point address = 0x80000
>
> INFO: SPSR = 0x3c9
>
> NOTICE: Initialising hafnium
>
> INFO: text: 0x80000 - 0x97000
>
> INFO: rodata: 0x97000 - 0x9a000
>
> INFO: data: 0x9a000 - 0x117000
>
> INFO: Supported bits in physical address: 44
>
> INFO: Stage 2 has 4 page table levels with 1 pages at the root.
>
> INFO: Found PSCI version: 0x10001
>
> INFO: Memory range: 0x0 - 0x3b3fffff
>
> INFO: Memory range: 0x40000000 - 0xfbffffff
>
> INFO: Ramdisk range: 0x1800000 - 0x353dbff
>
> ERROR: Data abort: pc=0x8b0dc, esr=0x96000021, ec=0x25, far=0x19536ee
>
> Panic: EL2 exception
>
> Is there anyone can help me with this issue or share me the way to bring
> up Hafnium on pi4 ? Thanks very much !
>
>
> ------
>
> By Wanghua
>
>
>
>
> Best Regards！
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
>
>
>
>
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
Hi All,
In Arm, we are experimenting with running OP-TEE under Hafnium as the SPMC in S-EL2. We have been debugging this Stage 2 fault that OP-TEE runs into a during a test to share memory (xtest 1003). It seems this is due to a bug in Hafnium but want to be sure before posting a fix. Some thoughts below to this end. Apologies for the verbosity but I hope you will appreciate it is required.
The fault occurs when OP-TEE tries to access a memory region that was shared with it by the OP-TEE driver in Linux i.e. the driver has called FFA_MEM_SHARE to share the memory, OP-TEE has called FFA_MEM_RETRIEVE_REQ to map it in its S2 and Hf has called FFA_MEM_RETRIVE_RESP to describe the IPA range to OP-TEE. So, the S2 tables are created correctly before OP-TEE tries to use them.
The S2 fault is a L3 Translation fault. The L3 descriptor in S2 tables is NULL when the fault occurs. So this makes sense. This also implies that the translation is not cached in the TLBs.
The key thing is that the fault only occurs when cache state modelling is turned on in the FVP_Base_RevC-2xAEMv8A model we are using for development. The fault occurs both when the S2 tables are created and accessed on the same PE as well as different PEs. It does not matter whether the PEs are in the same or different clusters. The fault occurs both with and without a Hypervisor (Hf) in the Normal world. So presence of Hf in EL2 is not a factor.
We noticed that Hf marks its internal memory as outer-shareable. See [1] and [2]. It uses inner-shareable for S2 PTWs though. See [3]. This is a mismatch of memory attributes as per Page 2563 in ARM DDI 0487F.b. The start of the text is quoted below.
"The rules about mismatched attributes given in Mismatched memory attributes…”
And indeed, the fault is not seen if we mark Hf’s internal memory as inner shareable to match the PTWs. The DSBs after creating the S2 tables in [4] are for inner-shareable access types. It seems that the inner-shareable PTW is unable to observe the outer shareable page table write. Using the inner shareable attributes for the internal memory makes the write observable.
Alternatively, if we change shareability of PTWs in VTCR_EL2 to outer shareable then the fault is no longer observed. It is not clear how the PTWs and page table writes are synchronised in this case without a DSB OSH. This is not a violation of the architecture afaiu.
It seems that it would be worth aligning these attributes.
The next bit is why Hf uses the outer shareable attribute for internal memory in the first place. The recommendation seems to be to use inner-shareable. See [5] and [6].
So we are wondering if this should be fixed too. Please let me know if we have misunderstood anything so far. Happy to post a patch if not or provide more information.
Cheers,
Achin
[1] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch…
[2] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/mm.c#1043
[3] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch…
[4] https://hafnium.googlesource.com/hafnium/+/refs/heads/master/src/arch/aarch…
[5] "Shareable Normal memory” in Pg. 154 in ARM DDI 0487F.b
[6] https://linux-arm-kernel.infradead.narkive.com/RZHvk1cT/question-how-can-we…
Hi Wanghua,
Happy to help, the RPi4 port should still be working. Though we do not
continuously test it, so there is a possibility that the code has rotted a
bit.
I can share my notes on how I got it working, but you seem to have figured
out pretty much all of it.
You're seeing the hypervisor abort when accessing address 0x19536ee (FAR) -
that's inside the radisk as reported above the error. The value of ESR
suggests that this is an alignment error (the address is indeed unaligned).
So Hafnium probably crashed trying to parse something in the ramdisk.
Before we try anything else, could you try to update your RPi4 firmware? I
remember that I had issues with the 'initramfs' option in config.txt - it
would load random data from the SD card. The error that you're seeing could
be a result of that.
Instructions on how to update your firmware:
https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md
If that doesn't help, could you please attach
your out/reference/rpi4_clang/hafnium.elf? It would be good to know which
function is at the crashing PC 0x8b0dc.
David
On Sun, Jun 7, 2020 at 2:08 PM 王华 via Hafnium <
hafnium(a)lists.trustedfirmware.org> wrote:
> Hi all :
>
> Has anyone brought up Hafnium on Raspberry Pi 4 board ? I tried but
> failed by the following steps:
>
> 1.Make the Hafnium RAM disk with aarch64 build Pi4 kernel , RAM disk for
> Linux , manifest.dtb.
>
> The manifest.dtb buildt by :
>
> /dts-v1/;
>
>
> / {
>
> │ hypervisor {
>
> │ │ compatible = "hafnium,hafnium";
>
> │ │ vm1 {
>
> │ │ │ debug_name = "Linux VM";
>
> │ │ │ kernel_filename = "vmlinuz";
>
> │ │ │ ramdisk_filename = "initrd.img";
>
> │ │ };
>
> │ };
>
> };
>
> 2.Copy bl31.bin hafnium.bin initrd.img to fat32 boot direction and
> configure config.txt . Power up my unit and got the uart error log :
>
> NOTICE: BL31: v2.3(debug):v2.3-109-g771c676b1
>
> NOTICE: BL31: Built : 15:49:37, Jun 1 2020
>
> INFO: Changed device tree to advertise PSCI.
>
> INFO: ARM GICv2 driver initialized
>
> INFO: BL31: Initializing runtime services
>
> INFO: BL31: cortex_a72: CPU workaround for 859971 was applied
>
> INFO: BL31: cortex_a72: CPU workaround for cve_2017_5715 was applied
>
> INFO: BL31: cortex_a72: CPU workaround for cve_2018_3639 was applied
>
> INFO: BL31: Preparing for EL3 exit to normal world
>
> INFO: Entry point address = 0x80000
>
> INFO: SPSR = 0x3c9
>
> NOTICE: Initialising hafnium
>
> INFO: text: 0x80000 - 0x97000
>
> INFO: rodata: 0x97000 - 0x9a000
>
> INFO: data: 0x9a000 - 0x117000
>
> INFO: Supported bits in physical address: 44
>
> INFO: Stage 2 has 4 page table levels with 1 pages at the root.
>
> INFO: Found PSCI version: 0x10001
>
> INFO: Memory range: 0x0 - 0x3b3fffff
>
> INFO: Memory range: 0x40000000 - 0xfbffffff
>
> INFO: Ramdisk range: 0x1800000 - 0x353dbff
>
> ERROR: Data abort: pc=0x8b0dc, esr=0x96000021, ec=0x25, far=0x19536ee
>
> Panic: EL2 exception
>
> Is there anyone can help me with this issue or share me the way to bring
> up Hafnium on pi4 ? Thanks very much !
>
>
> ------
>
> By Wanghua
>
>
>
>
> Best Regards！
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
>
Hi Raghu,
Howdy! CIL…
> On 4 Jun 2020, at 16:21, Raghu K via Hafnium <hafnium(a)lists.trustedfirmware.org> wrote:
>
> Hi Achin,
>
> Would you mind elaborating more on why the SPM needs to determine the security state and why it is important to do this without trusting the SP? When you say SPM, it sounds like you are talking about the SPMD running in EL3 for ex., that is not a part of the SPMC which perhaps runs as S-EL2 and the SPMD may need to know this to figure out how to map a particular physical page. Is that the use case you are thinking about?
So this is in the context of PSA FF-A Memory management ABIs. Also, I have the S-EL2 SPMC case in mind. SPMD in EL3 does not participate in memory management in this case when it comes to managing any architectural state i.e. translation tables, control regs etc
Say, a SP0 invokes FFA_MEM_SHARE to share a single page A with SP1. The SPMC would need to map page A in SP’s stage 2 tables. To do this, it would need to determine whether the IPA of page A belongs to the Secure or Non-secure IPA space. This is under the assumption that some memory ranges in SP0’s IPA space will be Non-secure.
IMO, this information can be determined in one of the following ways:
1. Perform PTW in SW to determine whether IPA is mapped in the tables referenced by VSTTBR_EL2 or VTTBR_EL2. I am assuming the SPMC maintains separate S2 translations for the Secure and NS address spaces.
2. Through an internal data structure which tracks the attributes of a memory region assigned to a guest.
3. SP0 specifies the security state of page A in FFA_MEM_SHARE. The spec does not cover this currently. However, the SPMC cannot trust that the SP0 is providing the right security state and must verify this independently anyways.
1 seems clunky. 2 is not done in upstream Hf. 3 does not really help.
I think I had misunderstood that a AT* instruction could be used. There do not seem to be any in the Arm ARM that only perform a IPA to PA i.e. a S2 translation.
So I am wondering what can be done to solve this problem assuming we agree that this is a problem in the first place.
Hth,
Cheers,
Achin
>
> Thanks
> Raghu
>
> On 6/4/20 3:07 AM, Achin Gupta via Hafnium wrote:
>> Hi All,
>>
>> I am thinking of a scenario where a SP shares Non-secure memory with one or more SPs or VMs. The NS memory region could have been donated to the SP by a VM earlier (far fetched but possible).
>>
>> The question is how does the SPM determine the security state of the memory region being shared by the SP.
>>
>> It is especially important that the SPM does this without trusting the SP.
>>
>> I don't think it should rely on the AT* instructions. The SP could change the security state of the region in S1. AFAIK, there are no AT* instructions that only do S2 walks with a IPA as an input.
>>
>> So is the only option to perform a walk in both the Secure and Non-secure S2 tables to determine where is the address mapped.
>>
>> This seems a bit clunky. So wondering if I am missing anything and there is an easier way to do this.
>>
>> What do you reckon?
>>
>> cheers,
>> Achin
>
> --
> Hafnium mailing list
> Hafnium(a)lists.trustedfirmware.org
> https://lists.trustedfirmware.org/mailman/listinfo/hafnium
Hi all :
Has anyone brought up Hafnium on Raspberry Pi 4 board ? I tried but failed by the following steps:
1.Make the Hafnium RAM disk with aarch64 build Pi4 kernel , RAM disk for Linux , manifest.dtb.
The manifest.dtb buildt by :
/dts-v1/;
/ {
│ hypervisor {
│ │ compatible = "hafnium,hafnium";
│ │ vm1 {
│ │ │ debug_name = "Linux VM";
│ │ │ kernel_filename = "vmlinuz";
│ │ │ ramdisk_filename = "initrd.img";
│ │ };
│ };
};
2.Copy bl31.bin hafnium.bin initrd.img to fat32 boot direction and configure config.txt . Power up my unit and got the uart error log :
NOTICE: BL31: v2.3(debug):v2.3-109-g771c676b1
NOTICE: BL31: Built : 15:49:37, Jun 1 2020
INFO: Changed device tree to advertise PSCI.
INFO: ARM GICv2 driver initialized
INFO: BL31: Initializing runtime services
INFO: BL31: cortex_a72: CPU workaround for 859971 was applied
INFO: BL31: cortex_a72: CPU workaround for cve_2017_5715 was applied
INFO: BL31: cortex_a72: CPU workaround for cve_2018_3639 was applied
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x80000
INFO: SPSR = 0x3c9
NOTICE: Initialising hafnium
INFO: text: 0x80000 - 0x97000
INFO: rodata: 0x97000 - 0x9a000
INFO: data: 0x9a000 - 0x117000
INFO: Supported bits in physical address: 44
INFO: Stage 2 has 4 page table levels with 1 pages at the root.
INFO: Found PSCI version: 0x10001
INFO: Memory range: 0x0 - 0x3b3fffff
INFO: Memory range: 0x40000000 - 0xfbffffff
INFO: Ramdisk range: 0x1800000 - 0x353dbff
ERROR: Data abort: pc=0x8b0dc, esr=0x96000021, ec=0x25, far=0x19536ee
Panic: EL2 exception
Is there anyone can help me with this issue or share me the way to bring up Hafnium on pi4 ? Thanks very much !
------
By Wanghua
Best Regards！
