Notes from Megameeting 26th April 2010

Attendees

BruceAmbacher UM
DavidGiaretta STFC
HelenTibbo UNC
JohnGarrett GSFC
RobertDowns CIESIN, Columbia University
SimonLambert STFC
TerryLongstreth  

Summary

Today's discussion was partly spoken, so the chat transcript below is not a full record.

Section 9 of the "Requirements for Bodies ..." document was reviewed, based on a commented version from DavidGiaretta (on the wiki) and a comparison with the corresponding section of ISO 17021 by SimonLambert (circulated by email). JohnGarrett had also circulated a diagram showing the relations between the bodies discussed in the document.

The acid test for including extra text (in addition to 17021) is whether it would make a material difference to the way a TDR audit is conducted.

The following decisions were reached:

  • 9.1.2: action on TerryLongstreth as below
  • 9.2.2.1: delete
  • 9.2.2.2.1: keep the requirement for two members being physically present, but no need to explain why
  • 9.2.2.2.2: delete all bullet points
  • 9.2.2.3: delete
  • 9.2.2.3.1: keep
  • 9.2.2.3.2: not necessary, delete
  • 9.2.2.3.3: not necessary, delete
  • 9.2.3: delete

There was a discussion of the role of the initial audit committee/TDR Accreditation Board. JohnGarrett will try to refine his diagram accordingly.

Actions

  • TerryLongstreth to propose revised wording for 9.1.2 second and third paragraphs.

Transcript of chat

BruceAmbacher >> (All): John,  Your diagram did not appear to include any links 
between the Audit standard and the repository seeking certification. Am I 
misreading it?
JohnGarrett >> (All): Hi Bruce,  no I didn't have any links between Std and 
Repository there should be one that says 'operates according to" or something 
like that.
BruceAmbacher >> (All): It is a good starting point.
BruceAmbacher >> (All): I have looked at Simon's a little, the revised section 
itslef, less so
TerryLongstreth >> (All): I just received  all 3 of them this morninig
BruceAmbacher >> (All): Wasn't the purpose to define aspects of audit that might 
be most significant for TDR audit
TerryLongstreth >> (All): In last week's meeting, we discussed some substantial 
potential changes to section 9, as regards creating ways for less labor 
intensive / expensive audit stages
TerryLongstreth >> (All): 2nd para under discussion in 9.1.2 may need modifying, 
but the issue it raises is relevan
TerryLongstreth >> (All): In the Webworld, dependence on resources outside of a 
particular implementations control can create substantial risk
TerryLongstreth >> (All): Keep but make it clearer
SimonLambert >> (All): Do we define "Statement of Applicability" elsewhere?  
What is it?
BruceAmbacher >> (All): I think keeping par 1 is more useful than 2
TerryLongstreth >> (All): I'll try to gen a replacement
Helen Tibbo >> (All): I agree that the lasst paragraph is a bit risky as we are 
going beyond the standard itself. Yes, 1 is more useful than 2
David Giaretta >> (All): ACTION ON TERRY TO CLARIFY 9.1.2
BruceAmbacher >> (All): Isn't the "covered elsewhere" the actual RAC std?
TerryLongstreth >> (All): Why do they mention the certificzation body?
TerryLongstreth >> (All): but PP 1 talks to the audit team
JohnGarrett >> (All): Yes, Audit Team is more exact than Certification Body
TerryLongstreth >> (All): Bruce's test is the heart of the matter; I think the 
issue is more cogent
RobertDowns >> (All): While I also do not feel strongly either way, these 
paragraphs should be covered in the requirements.
BruceAmbacher >> (All): If we think it is in the RAC and that will be the basis 
for the sudit, then we can drop
TerryLongstreth >> (All): since the 'scope' of the audit must entrain all areas 
of preservation and presentation that might be put at risk
TerryLongstreth >> (All): Let me submit a strawman replacement for next week
BruceAmbacher >> (All): The link to RAC that we want is in the first par to be 
deleted.  Can we move that to the first par?
JohnGarrett >> (All): OK
BruceAmbacher >> (All): OK with Terry's offer.
Helen Tibbo >> (All): I was going ot say drop but we can look at a re-write next 
week.
David Giaretta >> (All): ACTION Terry to redraft 9.1.2
SimonLambert >> (All): yes -a agreed
TerryLongstreth >> (All): Should 9.2xxx explain how RAC is split between Stage 1 
and Stage 2?
JohnGarrett >> (All): I think the split is clearly explained in 17021. Stage 1 
is remote document study and Stage 2 is on-site checks.
TerryLongstreth >> (All): If there are no considerations of remote vs. on-site 
in RAC, then expunge 
BruceAmbacher >> (All): How closely does 17021 stage i doc review align with RAC 
self-assessment?
JohnGarrett >> (All): Now, self-assessment at RAC is done by repository, but 
Stage 1 document review is done by audit team.
TerryLongstreth >> (All): 2 members needed because RAC is more complex? more 
logical? less obscure?
BruceAmbacher >> (All): But the self-assessment uncovers the docs for review
JohnGarrett >> (All): 2 members is less than the 3 that is required on the audit 
team I think.
TerryLongstreth >> (All): Then say 'to ensure confidence in certification...'
BruceAmbacher >> (All): We need to establish the order of events: a =  self-
assessment and report. (This is our unique addition) b = Stage 1 doc review. c = 
stage 2 onsite review.  Simon has outlined what we should say very succinctly.
JohnGarrett >> (All): I think there should be two "to ensure confidence"
TerryLongstreth >> (All): But our 'pre' stage included public disclosure (from 
last week)
BruceAmbacher >> (All): Terry, where do we discuss this We wanted a "multi-
person" team to instill confidence and to ensure diversity in what is examined.
JohnGarrett >> (All): I don't think we put anything in the document.  I think we 
just said that outside folks would look favorably on a repository that claimed 
it was compliant without  an audit.
BruceAmbacher >> (All): Where is the "pre" Terry refers to in the current text?
TerryLongstreth >> (All): 'pre' : what  Bruce called a=
JohnGarrett >> (All): I don't think there is in current text.  'Pre' is 
something done just by a repository and no one outside will certify those 
results.
JohnGarrett >> (All): OK with me.
RobertDowns >> (All): Fine with me, too.
TerryLongstreth >> (All): I think the issue of how many people should be a 
parameter that varies with context
JohnGarrett >> (All): Yes I agree context will matter, but 2 is the minimum.
TerryLongstreth >> (All): one person by themselves might provide enough 
confidence
BruceAmbacher >> (All): and it depends on the nation and culture
JohnGarrett >> (All): How many do other people think is the minimum?
RobertDowns >> (All): One would be sufficient as long as the findings are 
documented in light of the evidence.
TerryLongstreth >> (All): It seems strange that we'd be more strict that either 
17021 or 27006
JohnGarrett >> (All): We are being less strict that 17021 and 20006.  They 
require the whole team.  We require only a minimum of 2.
TerryLongstreth >> (All): 'the whole audit team'  could be one person
JohnGarrett >> (All): No, the whole audit team is at least 3 I think and is 
often larger.
TerryLongstreth >> (All): If my TDR is on my laptop
BruceAmbacher >> (All): Terry, I would not approve  your laptop as a TDR under 
any criteria
TerryLongstreth >> (All): Are you the whole team?
TerryLongstreth >> (All): Bruce
TerryLongstreth >> (All): We should cover the generality but not exclude the 
extremes
BruceAmbacher >> (All): One can be adequate
JohnGarrett >> (All): I would prefer 2 people just to provide more confidence in 
the impartiality and coverage
TerryLongstreth >> (All): If the Audit Team's oversight procedures are approved 
by the TAB?
BruceAmbacher >> (All): The written report, compared to the pre-assessment and 
the doc review can demonstrate to the broader designated community that the 
audit is  honest and thorough
JohnGarrett >> (All): It just seems to me that having one person who has to 
provide more documentation of their work on-site to all the off-site people 
would end up being more expensive.  I takes me more time often to do minutes of 
meeting than to attend the meeting.
BruceAmbacher >> (All): I would like to see 2 as the norm but allow for 1 when 
justified
RobertDowns >> (All): One would be appropriate if the procedures specify what 
evidence should be provided to the team.
Helen Tibbo >> (All): The written documentation protects the team if the 
approved TDR fails after an audit
TerryLongstreth >> (All):  on site give direct verification of physical 
properties, but much of OAIS is intangible
BruceAmbacher >> (All): The onsite person/people would go through an audit based 
on the entire team's review of the documents and the issues that raised.
Helen Tibbo >> (All): The on-site review teams (usually 4-5 people) from ALA 
that review ILS programs interview everyone to get the back story about what is 
really going on...
TerryLongstreth >> (All): Do we have any specific activities to be performed on
-site?
Helen Tibbo >> (All): The team gets all the written documentation in advance but 
the onsite visit is for verifying the truthfulness of it.
Helen Tibbo >> (All): YES
Helen Tibbo >> (All): I don't think we need 4-5 but perhaps at least 2
BruceAmbacher >> (All): Hence the focus could vary from audit to audit.  All 
TDRs are not the same
Helen Tibbo >> (All): I think 2 if far more than 1
TerryLongstreth >> (All): How many people do you need to interview?
Helen Tibbo >> (All): The discussions the 2 will have will be very important. 
Helen Tibbo >> (All): One person can only fumble it over in their own minds
Helen Tibbo >> (All): I would hate to be the 1 person by myself
RobertDowns >> (All): Onsite audit by one does not mean that others cannot 
conduct interviews by phone
TerryLongstreth >> (All): I think I have a completely different picture of what 
an auditor does
Helen Tibbo >> (All): Also two people can take less time than 1 so you have to 
pay for 2 but they are onsite less time
TerryLongstreth >> (All): Once an OAIS is in place, it may well be maintained by 
a single individual
Helen Tibbo >> (All): If you just believe the documentation you don't need to 
visit. But everyone can make the documentation sound great.
JohnGarrett >> (All): If I were the sponsor of an archive, I think I would tend 
to want to pay the incremental cost of the travel for a second person to be 
present.  The time spent would be much the same whether the review was on-site 
or off-site
TerryLongstreth >> (All): single individual - one person t interview
BruceAmbacher >> (All): We run the risk of all of these talks going into hearsay 
and straying from the documentable facts.  Talk also can develop aspects where 
the repository does not carry out policy or procedures.
JohnGarrett >> (All): Most single person repositories won't want to pay at all 
for an audit and will likely just self certify
BruceAmbacher >> (All): Lets just keep 2 and see how the reviewers react/respond
JohnGarrett >> (All): Yes
TerryLongstreth >> (All): on 9.2.4 - talking about someone approving or passing 
on audit team recommendation.  In our model, I think that's what the National 
body or perhaps TAB, do
TerryLongstreth >> (All): the confusion is the term "entity"
TerryLongstreth >> (All): can't we be more explicit?
RobertDowns >> (All): Should "entity" be "certification granting authority"?
BruceAmbacher >> (All): The metrics focus on what a repository should do to be a 
TDR; these bullets  (9.2.2.2.2) instruct the audit team what to use for its 
assessment.
JohnGarrett >> (All): what is TAB?  Is that Primary Audit Board?
TerryLongstreth >> (All): TDR Audit Board (from last meeting)
BruceAmbacher >> (All): I thought it was TAB - TDR Audit Board
RobertDowns >> (All): Perhaps we should define TAB in the document.
TerryLongstreth >> (All): John's picture has primary
SimonLambert >> (All): Last week we decided on "TDR Accreditation Board"
BruceAmbacher >> (All): TAB sits above the approved audit companies which 
create,support. and  field the audit teams
JohnGarrett >> (All): I would suggest that we have a Primary Accreditation Board 
(PAB) that accredits companies to do audits.  And within those companies there 
are TDR Certification Boards (TCB) that approves the Audit Team Reports and 
grants the certification seal.
BruceAmbacher >> (All): Does TAB ever review the companies' certification 
decisions?
JohnGarrett >> (All): I suppose it could review the companies certification 
decesions to determine if the companies remain accredited to do audits.
Helen Tibbo >> (All): We do have to have a bootstrap process and a point at 
which this is a going concern.
BruceAmbacher >> (All): So, as written, the entire process rests on "for-profit" 
audit teams that will want additional business (profit) from other repositories. 
 Where is the incentive to be critical?
Helen Tibbo >> (All): we lost you David
BruceAmbacher >> (All): I lost the audio.
JohnGarrett >> (All): The whole ISO 9000 process and CMMI audits all rely on for 
profit audit companies, and they seem to work
RobertDowns >> (All): me too
JohnGarrett >> (All): We just lost David from the meeting
BruceAmbacher >> (All): David, I must leave.  Please post what we will focus on 
next week
Helen Tibbo >> (All): The incentive to be critical is that a failed repository 
might sue (not sure legal) but would at least harm the reputation of the audit 
process.
Helen Tibbo >> (All): I need to run off too...
Helen Tibbo >> (All): Talk with y'all next week.
David Giaretta >> (All): Bye Helen
JohnGarrett >> (All): If I find time, I'll work more on my diagram and 
definitions.  Anyone else feel free to augment my efforts also.
RobertDowns >> (All): I lost audio from David again
TerryLongstreth >> (All): I've got to go
JohnGarrett >> (All): Bye, I'm out of here also
RobertDowns >> (All): Bye

-- SimonLambert - 26 Apr 2010

Topic revision: r1 - 2010-04-26 - SimonLambert
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback