Hi Xinyu,
Thanks for the escalation.
I see Ben is in the loop, so that's the correct first step. I've also cc'd
the triage maillist to make sure all stakeholders are looped in.
In general, as a best practice, I would suggest having as much quantifiable
data as possible upfront in tickets like this to help better understand the
magnitude as well as how to reproduce. It looks like the tickets have
already started to capture this, but I also see Ben in the ticket
requesting more. Datapoints of interest in my mind:
- Clear details on how to reproduce: The task(s) where noticeable
degradation is seen - is this in parallel to when large builds have kicked
off? etc.
- Tasks invoked and level of degradation: for example, "Gerrit reviews -
previously took xyz seconds/minutes, now taking 20% more time (or 2x, 10x?,
failing & never completing?)," frequency, etc. The more details the
better! This will help determine the priority we place on resolving. :)
Perhaps coming up with a general template for this could be helpful.
- Is there other degradations beyond Gerrit?
Ben is most certainly much more qualified than me in knowing what support
is needed to isolate/resolve, and, as noted in the ticket, he is asking for
more details as well. Let's get the details captured in the ticket(s) in
preparation for next Tuesday's Triage meeting where we can prioritize the
resolution over other TFC tasks. :)
Ben, feel free to chime in and correct any of my assumptions/suggestions.
Regards,
Don
On Thu, 2 Sept 2021 at 01:41, Xinyu Zhang <Xinyu.Zhang(a)arm.com> wrote:
> Hi Don,
>
>
>
> We found that trustedfirmware.org is getting slow. Daily work of some
> developers would be influenced.
>
> Could you please help to take a look on this issue? Here is the TFC link:
> https://linaro.atlassian.net/browse/TFC-172
>
>
>
> BR,
>
> Xinyu
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
>
*Attendees*: Riku, Anton, Glen, Shebu, Ben C, Joanna, Janos, Matteo, Don,
Xinyu
*Minutes*:
- Glen: Begin w/ TFC Kanban Board
- Glen: H/W Status
- Working thru getting Chromebooks imported. In progress
- Cypress boards flashing and booting, starting on jobs work. In a
week ready for boards to be shipped to Cambridge.
- NXP board is in the queue.
- MBedTLS:
- Glen: Have advanced as far as can. Need AMI files from Dean A.
Blocked awaiting AMI files.
- Janos: Progressed since last week, but working on resolving
internal issues. No estimate yet from Dean.
- Glen: Plan to move to expect script efforts with team approval
- Matteo: Agree Expect Scripts are the next item to work. Would like
completed, so good direction to go.
- Code Coverage:
- A sync w/ Leonardo and Joanna to discuss next steps/Current status
on CC shall be planned. Glen to set up
- Glen: Performance/Fixes needed?
- Riku: Nightly jobs failing due to too many parallel checkouts of
same repo/same version. TFC-20. Need to decide how to implement
- For TF-M, looks currently like 1 week work to modify CI to reduce
parallel checkouts, TF-A still looking to get an estimate.
- Ben: TF-A may be a bit larger task, but still need to look.
- Ben: Other solutions?
- Glen: Could Arm TF-A / TF-M team do this to offload the work?
- Not currently
- Ben: Need a short-term and long-term solution. Short-term - added
a server, long-term, change the configuration changes.
- Can use Staging Server to test it out.
- Git checkout taking lots of the build time.
- Shebu: Failing every night?
- Not sure.
- Riku: Last week, 1 success all other failed.
- Ben: The new server increasing capacity has caused this issue to
show itself.
- Work around is to potentially limit the number of parallel jobs.
- Next Steps:
- Leonardo scope TF-A.
- Create two subtasks for TF-A and TF-M
- Riku: Put in work-around to limit number of parallel jobs
- Need a TFC ticket here?
<end>
Attendees: Janos Fallath, Matteo, Joanna, Don, Glen, Xinyu, Ben C
Minutes:
- Cancel next week instance - folks are out
Platform enablement
- Chromebook: Arthur wrapping up the work, when receive the platforms
will be ready.
- Cypress: Got the info and moving forward again.
- NXP Platform: Some back ground work to see best way to integrate
- Focus back in Cypress
Other
- expect scripts. in holding pattern with focus on mbedTLS
- MC: Two months away from end a FY. Would like expect scripts are
finalized. Would like as a background task when things get blocked
- MC: Would like to at a minimum know what needs to be done and what
help Leonardo may need. LAVA help for example? Anything that
can be done
in parallel. So what is left and who needs to do which task?
- MBedTLS
- JF: Don't have AMI's yet, working to get them out. So blocked.
Can't set up environment.
- GV: Jumped forward to M3's for now. Will need to get to having
this up on a real system.
- GV: By end of the week, M3 should be done, having the AMI files
will be key at that point to move on.
- JF: Originally, scripts / files were to be pulled in. Now thinking
of a restricted repo and sharing it - may not have that option. What can
be used beyond a public repo?
- JF: Can Linaro see the private Arm instance?
- Glen: Have to check if can see the private repo *ACTION Glen*.
- Glen: Set up a sync to review repo sharing options *ACTION Glen*
- Movng to gitlab?
- JF: Not at this point
- JF: Could move the test repo to gitlab, but same issue.
- SC Approved:
- Code Coverage tool
- Joanna: A new ticket raised TFC-160. Can we disable code coverage
for now until the gitlab.arm.com is back up. Without it, jobs are
failing.
- MC: Should we wait?
- Joanna: TF-A has been down since Friday.
- MC: Can we see what Dean does to resolve?
- Plan to see if can disable code coverage until resolved.
- MC: Can we be explicit in the ticket on what Linaro is to do. i.e
request disable code coverage on TF-A
Attendees: Janos Fallath, Don, Glen, Xinyu, Ben, Joanna
Minutes:
- Server is ready to bring online, Ben just prioritizing time to get it in
now.
- TFC-110: Licenses available in October - pushed out till then
- TFC-92 Complete - Can now create docker containers for FVP models
- TFC-5 - Add additional FVP models
Is this needed still or will developers add their own?
- MBedTLS
Need to add Janos to the Jira
Need ANI files
Forward email to Janos
- Boards
May start NXP board since Chromebook and Cypress platforms blocked
Chromebooks in shipping
- Expect scripts
Still need some modifications, but not working on - MBedTLS higher
priority
Joanna - Will these get worked on if gaps in board come up?
Glen - yes this could happen
- Code coverage: The bigger aggregated reports are still open
Attendees: Don, Anton, Joanna, Ben, Dave R, Xinyu
FVP: TFC-92 updates?
Ben: Working in staging, needs Docs, tested and moved to staging. Leonardo
back and to work this. Subtasks are TFC-107,108,115
Anton - Planned to complete TF-M release this week. Within the next two
days.
After the release Ben to bring up the new Server. Already set up and
waiting. Should it be all production, partial staging? Plan to start out
as all Production. Can adjust if still need more staging.
Anton: Is it possible to connect two CI's to one LAVA backend?
Ben: It's possible with correct priorities. LAVA has its own Scheduler.
Anton has a private build from an internal CI that he will send binaries to
be tested in LAVA.
Dave R: Notes on MBeb TLS
- There have been some changes to how MBed TLS tests are run. Will make
this public soon. No impacts, but should be easier to use the public repo
versus a private copy
- Unable to host private repos on Github - the impact is to migrate
private repos to GitLab. Expect this to change CI triggering. May require
a detailed call on impacts.
- Dave is here Wed Thursday next week, then out for 4 weeks. Dave will
provide a delegate.
Hello;
The test with migrating Jira to the Cloud version went well. We will
migrate over on Thursday, 7/29. The transition should only take an hour
and I will send out an email when its completed. The new Jira
instance will be - https://linaro.atlassian.net/jira/projects
Thanks;
-g
On 7/27/2021 12:17 PM, Don Harbin via Tf-openci-triage wrote:
> Attendees: Don, Glen, Ben C., Joanna, Xinyu, Glen, Shebu, Anton
>
> Minutes:
>
> - Glen: Migrating Jira to Cloud - Atlassian is going Cloud only.
> - Glen: New H/W in place?
> - Joanna: No conflicts for next month or two, no issues
> - TF-M release - Anton: First Friday of August
> - Ben: Open a ticket and Ben will bring it up after 1.4.0
> - Glen: Would it help benefit TF-M to do it now?
> - Discussed the split of resources between staging vs production
> - Server load overview
> - https://tinyurl.com/486325k5
> - Change the "01" to "02" to see both servers
> - Anton: ST board can now be used to run release tests?
> - Glen: Yes, it's ready.
> - Glen: Chromebooks. Getting the final platforms to Cambridge lab.
> Arther to finish in next week prior to board arrival. Will start Cypress
> next.
> - Joanna: Need to work out who develops tests for the Chromebooks.
> - Glen: Will check with Arthur on basic boot tests
> - Joanna: Expect Scripts and Code Coverage patches reviewed. They look
> fine. What about uploading our own FVP models (TFC-108).
> - Ben: Riku has done some work on this (created CI job). See ticket
> comments. No ETA yet, but doesn't look like a long task.
> Leonardo will be
> tasked and he's out this week to move Riku's job over.
> - Joanna: How do new users leverage the download tool?
> - Raise a TFC ticket.
> - MBedTLS - SOW updated, milestones charted out. JIra cards created.
> - Support for IAR compiler?
> - Shebu: Got some licenses for Arm use, but will be a future milestone.
> - Shebu out for month of August. Dave R will need to identify a POC for
> MBedTLS guidance.
>
> Don
--
Linaro <http://www.linaro.org>
Glen Valante | /Sr. Technical Program Manager/
T: +1.508.517.3461 <tel:1617-320-5000>
glen.valante(a)linaro.org <mailto:glen.valante@linaro.org> | Skype:
gvalante <callto:gvalante>
Attendees: Don, Glen, Ben C., Joanna, Xinyu, Glen, Shebu, Anton
Minutes:
- Glen: Migrating Jira to Cloud - Atlassian is going Cloud only.
- Glen: New H/W in place?
- Joanna: No conflicts for next month or two, no issues
- TF-M release - Anton: First Friday of August
- Ben: Open a ticket and Ben will bring it up after 1.4.0
- Glen: Would it help benefit TF-M to do it now?
- Discussed the split of resources between staging vs production
- Server load overview
- https://tinyurl.com/486325k5
- Change the "01" to "02" to see both servers
- Anton: ST board can now be used to run release tests?
- Glen: Yes, it's ready.
- Glen: Chromebooks. Getting the final platforms to Cambridge lab.
Arther to finish in next week prior to board arrival. Will start Cypress
next.
- Joanna: Need to work out who develops tests for the Chromebooks.
- Glen: Will check with Arthur on basic boot tests
- Joanna: Expect Scripts and Code Coverage patches reviewed. They look
fine. What about uploading our own FVP models (TFC-108).
- Ben: Riku has done some work on this (created CI job). See ticket
comments. No ETA yet, but doesn't look like a long task.
Leonardo will be
tasked and he's out this week to move Riku's job over.
- Joanna: How do new users leverage the download tool?
- Raise a TFC ticket.
- MBedTLS - SOW updated, milestones charted out. JIra cards created.
- Support for IAR compiler?
- Shebu: Got some licenses for Arm use, but will be a future milestone.
- Shebu out for month of August. Dave R will need to identify a POC for
MBedTLS guidance.
Don
*Attendees*: Shebu, Peterson, Xinyu, Paul, Black, Ben C, Matteo, Glen,
Andon, Don, Joanna
*Minutes*:
- Discussing compiler issues used in Open CI compiler latencies that
have been seen
- Paul driving a new licensing model. Also want to try new licensing.
See performance issues on Arm CI as well.
- Matteo: Use CPU and Arch models. Have problems w/ model licenses
- Ben to provide licenses in license file.
- Running these for TF-M and TF-A.
- Paul would like to determine how much improvement he can gain. Want
to know how long should take
- Ben: TFC-110 request for long term licenses.
https://projects.linaro.org/browse/TFC-110
- Glen to give Paul full access to tickets. Add Peterson Quadros as well
Review Dashboard:
- Anton: Approaching release. No issues, but LAVA had an issue.
- Xinju: See
https://tf.validation.linaro.org/scheduler/alljobs?length=25&search=295694#…
Lava jobs cancelled themselves.
- Ben: Assumed someone did this. No way to tell who did this.
- Glen: Should we send out an email?
- Ben already told his team.
- Xinyu, another release build tomorrow.
- Anton: Expect lots of TF-M usage over next 2 weeks. Expect this is
one time event, so no ticket
- Joanna: Team is struggling w/ some use cases. TFC-87. How jobs
reported in gerrit.
- Joanna: 2nd issue - jobs taking longer to run. Running CI on patches
may take 3-4 hours. May have been tied to simultaneous TF-M builds? Not
sure. Team seeing some stability issues. Forcing multiple runs. Not
sure if servers, tests, or other. How should this be approached
- MC: Does TFC-87 require Open CI dev teams?
- JF: Can do some minor things, but not sure that's the best patch
- MC: Have agreed on a set of desired features in the in-progress
column. Want to keep the focus on these. Unless it's a blocker, then would
like to see this remain backlogged
- JF: Can try to drive from Arm, but may need Leonardo support for quick
questions, etc. Usability
- MC: We agreed to focus on more coverage focus vs better usability.
- JF: Can an infrastructure boost help? Assume it won't get better.
- GV: Maintenance piece feels missing in some ways. ST board seeing
regressions, for example. Need to determine who is watching these failures.
- GV: Why failures are occurring is the missing piece
- SK: ST a new board, not sure it falls under maintenance.
- GV: High level - who is supposed to be watching the boards
- SK: A roles and responsibility needs to be developed.
- MC: This year, features, next year, transition to improvements.
- GV: Code coverage should wrap up in the next week. Expect scripts
task moving forward as well.
- GV: Chromebooks progressing - health check should be running in the
next few days, awaiting h/w to reach Cambridge lab.
- GV: Dave, Scott, Leonardo are finalizing estimates for MBed TLS. A
sync today. Estimates are within October time frame. So want to finish
Expect and Code Coverage, then transition to MBed TLS start of August.
TFC-114, and 110 still need closed out.
- GV: Boards are looking good - Arthur may be able to pick up some
jenkins related tasks if spare time.
- GV: When we move to maintenance mode, want to evaluate the best fits.
Thanks
Don
Attendees: Don, Glen, Joanna, Anton, Xinyu, Matteo, Shebu, Ben C
Minutes:
- TFC-34: Think the board has been swapped and tests pass
- TFC-46: ST boards. A time out has been added. Board in but a test
working at Arthur's local lab but no in LAVA Lab.
- TFC-45: Chromebook - boards shipping to Cambridge.
- TFC-4: mbed TLC - Arther has first six milestones to put into the
document. Dave also owes a few more pieces of information as well as
publish the test suite.
- Shebu: Please send a note out to Matteo, Shebu and Don when the
doc gets updated.
- TFC-92 - Matteo: Ben suggested needs to be a deployable solution.
Please explain last couple of comments
- Ben: Needs to use Tuxput and get it working. Uploads files and no
uploads FVP images to FVP bucket. Now write jenkins job. Can
now be done
thru Jenkins and no longer needs Linaro Engineering.
- Joanna; Now automatic.
- Ben: Yes, but Leonardo still working this (TFC-108)
- Glen: Any Docs?
- Ben: Yes but should be simple from user side
- Has been delayed due to focus on TFC-4
- TFC-36:
- Joanna: Patches under review, progressing well. Still catching up,
expect feedback later this week.
- TFC-7: Code Coverage
- Joanna: Version using is one version ahead. Also need output
sourced from console. Have data, just need to process it.
- TFC-114 - In progress. Just got posted and moved to In Progress.
Joanna noted that once TFC-92 is done, the arm team can do these on their
own.
- New ticket - TFC-113
- TFC-110 - Compiler licenses
- Glen: Needed for release?
- Anton: No, but slows things down.
- Ben: A quick add to flexnet server once get license.
- Anton: Send to support(a)arm.com
- Ben: Did once long time ago, didn't use the support channel though.
- Shebu: If send note to support, please cc Shebu. Include Bill
Fletcher as well since has history.
- Ben: How many should be requested? Currently have a 12 seat
compiler pack.
- Use in parallel too.
- Ben: Should get 120 new licenses. The twelve are expiring August
31.
- *Glen*: Check w/ Bill F on how to acquire 120 FCP licenses.
- Matteo: really want to see current "In Progresses" Closed.
<end>
FYI;
Ticket for Musca board remove and replace -
https://projects.linaro.org/browse/LSS-2298
-g
On 6/22/2021 9:00 AM, Don Harbin via Tf-openci-triage wrote:
> *Attendees*: Matteo, Glen, Don, Xinyu, Shebu, Fathi, Ben, Anton
>
> *Minutes*:
>
> - TFC-46 ST Boards in progress - very close. Devices are on line.
> - ChromeBook - in progress. Working w/ Julius has been helping to get a
> board unlocked.
> - Cypress has committed with a board. Cypress then NXP board are the
> order to enable
> - TFC-34: Only board that can run OTP tests. Suggest swap this out swap
> out.
> - Lab could swap this out with an LSS ticket. Need to flash with
> correct certificate. Already have this certificate. MUSCA_B1 is
> the one to
> replace.
> - LSS-2020
> -
> - Recieved Dave R code last week, Leonardo / Scott have been creating
> a plan - want to go thru the plan with Dave later this week.
> - Shebu: Would like the doc link
> - Glen: Owe ETC for all in progress columns
> - Matteo: Tied to the current plan in October. Want to understand
> effort required, especially w/ mBedTLS to do. Then will make a priority
> call.
> - Glen: mBedTLS - multiple platforms, compilers, etc. Planning a
> phased in functionality. May find some lower priorities (like Mac OS).
> Then review w/ Dave.
> - Matteo: Don't want to add more to In Progress until items are
> finished.
--
Linaro <http://www.linaro.org>
Glen Valante | /Sr. Technical Program Manager/
T: +1.508.517.3461 <tel:1617-320-5000>
glen.valante(a)linaro.org <mailto:glen.valante@linaro.org> | Skype:
gvalante <callto:gvalante>
