Notes from Megameeting 16th November 2009

Attendees

Summary

There was a discussion of the meaning and implementation of "surveillance" and recertification, and of the distinction between Stage 1 and Stage 2 audits.

It was agreed that terms must be used consistently - they will either be inherited from the "parent" standards or should be defined in section 3.

Actions

• All to check definition of surveillance in the ISO documents
• All to check the wiki page on issues (AuditorGuidelinesIssues)
• All all to add things to the draft on DraftPlanForOrganisation
• BruceAmbacher to add his proposed text on recertification to the issues page (AuditorGuidelinesIssues)

Transcript of chat

BruceAmbacher >> (All): My thought on recertification is five years due to 
effort, costs, trying to win over repositories to undergo audit.  Three years 
is just too short.
Mark Conrad >> (All): 7.2.1.2 Management of the decision taking process  - Is 
there text missing?
BruceAmbacher >> (All): David, expand on your thoughts on Surveillance.  What 
"right" does the audit body have to snoop around in a certified repository?  
Would this occue only when an egregious incident occurs?  Or would it be a 
constant passive oversight?
JohnGarrett >> (All): I think ISO 9000 has 3 years before recertification.  I 
think usually if your certification is active, they only do a spot check on a 
few things for recertification.   Do we do a full audit or only spot check 
things fro the recertification?
David Giaretta >> (All): Bruce  I have no views on surveillance other than that 
I assume it works the same way as all other certifications. DO you think the 
docuemnt suggests something special - if so it may be inherited from the 
security document we borrowed text from and perhaps we can remove it.
David Giaretta >> (All): Mark - looks like missing text - we need to compare 
with the Word doc
BruceAmbacher >> (All): No, I was just going from the General points at the top.
BruceAmbacher >> (All): John, I like that spot check, less than 100% 
recertification.  Has it been the same things in each recert or does it change 
institution to institution based on weaknesses in the initial audit?
David Giaretta >> (All): John - yes, need to check but my recollection of STFC 
ISO 9000 recertification was that it was not very thorough
BruceAmbacher >> (All): Presumably then the recert costs is not nearly as high.
RobertDowns >> (All): If we are going to consider recertification as early as 
three years, we need to be sure that the costs are not prohibitive for small 
repositories.
JohnGarrett >> (All): I'm not real plugged into what happens in the recerts, but 
I think they have a different area or two that they look at each time for 
recertification.  I guess rotating through things so over time all the elements 
are eventually rechecked.  I don't know if it is based perhaps on any changes 
the organization may have identified.
BruceAmbacher >> (All): ditto for the time investment
David Giaretta >> (All): Mark - checking with the Word doc it looks as if 
7.2.1.2 is missing the name of the RAC documemnt
Mark Conrad >> (All): ok.
SimonLambert >> (All): David - yes - I assume because ref was in angle brackets 
which the wiki treats specially.
TerryLongstreth >> (All): The recert process should probably be tailored to the 
certification body's specs; some recert processing might occur as an outgrowth 
of surveillance (which I interpret more as continuous monitoring, rather than 
reconnaissance)
JohnGarrett >> (All): What is the best way to generate use/acceptance/discussion 
of our document? 3 year or 5 year certifications?
David Giaretta >> (All): I suggest my "get out of jail free" card - say 3 years 
of whatever time is specified by the initial audit committee
David Giaretta >> (All):     .."3 years or whatever time is specified by the 
initial audit committee"
BruceAmbacher >> (All): I would lean toward 5 years due to cost, time, burden on 
small repositories, especially if the initial audit involves actions by the 
repository to bring itself up to standards.
BruceAmbacher >> (All): I like the idea of doing only a partial reaudit with 
different aspects checked each time and with an emphasis on any weaknesses 
revealed in the initial audit.
BruceAmbacher >> (All): Perhaps a reaudit team could be smaller, possibly only 
one person onsite and more reliance on written self assessment by the repository 
geared to identified issues/weaknesses/changes in preservation oriented 
activities.
JohnGarrett >> (All): Yes, I like idea of any weakness in one audit being an 
emphasized area in the recertificiation audit.
RobertDowns >> (All): Perhaps we could consider 5 year recert to look at 
potential weaknesses given the potential costs and time commitments for 
recertivication.
TerryLongstreth >> (All): If certification involves continuous monitoring of 
administrative artifacts, then a time period is irrelevant
Mark Conrad >> (All): Pick one. Put it in. See what kinds of comments we get.
JohnGarrett >> (All): Looking at weaknesses works to fulfill our underlying 
reason for doing the audits to ensure that we have competant, trusted 
repositories
BruceAmbacher >> (All): Terry, do we want repositories having a sense that ig 
brother auditor is always looking over their shoulder?  Or can it be viewed as a 
resource to be called upon when the repository is facing a new issue?
TerryLongstreth >> (All): That's a decision for the committee; banks seem to get 
along with daily reporting
David Giaretta >> (All): I tend to agree with Mark
BruceAmbacher >> (All): Whatever we pick we need to define what the reaudit will 
consist of.
David Giaretta >> (All): In terms of surveillance - this is something that goes 
through all the audit/certification docs. We need to check what it means in 
these terms. I don't think there is a real "big brother" intent
Mark Conrad >> (All): Bruce, Agreed.
TerryLongstreth >> (All): I agree with David on surveillance. 
BruceAmbacher >> (All): ok
David Giaretta >> (All): I meant to collect together all the definitions in the 
hierarchy of docs so we could get the full picture but I've been tied up with 
other work.
David Giaretta >> (All): I hope to get to it this weekend
TerryLongstreth >> (All): something of a tangent, that might intersect this 
issue of continuous audit presence, is whether the audited institution can 
access an expert for judgements of future actions that might impact audit status
BruceAmbacher >> (All): As a strwaman - "Recertification will occur no  more 
than five years after the initial audit and certification.  It may occur sooner 
if the repsoitory or the audit committee believe it is warranted.  The reaudit 
process will focus on any initial weaknesses and
BruceAmbacher >> (All): a revolving subset of the audit criteria."
David Giaretta >> (All): Terry - not sure where "continuous audit presence" 
comes from
TerryLongstreth >> (All): "surveillance"
David Giaretta >> (All): Terry - not sure surveillance means continuous audit 
presence
RobertDowns >> (All): Bruce - the strawman looks good.
Mark Conrad >> (All): Bruce, I like it.
TerryLongstreth >> (All): Ok, then I don't know what surveillance means after 
all.  C A P doesn't mean intrusion, simply availability
David Giaretta >> (All): Bruce - we need to check how consistent this is with 
the ISO audit guidelines. However I would prefer not to tie the hands of the 
auditors. It may be possible to suggest that the recertification period is 
negotiable.
David Giaretta >> (All): Terry - you may be right - I guess I'm just keen that 
we use the right definitions otherwise we will get lost. Is C.A.P. used in the 
ISO docs?
TerryLongstreth >> (All): Recertification may be driven by TRP events, like 
replacing technology
Mark Conrad >> (All): TRP?
David Giaretta >> (All): Terry - what is "TRP"?
TerryLongstreth >> (All): I don't know. I was trying to bridge what I thought 
might be a cultural gap between us (on the meaning of surveillance)
JohnGarrett >> (All): For me to trust a repository, I would expect that it would 
need to pass a recertification with some specified time period.  I don't think I 
would be willing to accept that they would be recertified when they felt like 
it.
David Giaretta >> (All): I guess I don't like the definition Bruce proposes 
because in 5 years a whole heap of things could change. It seems to me that 
surveillance and recertification are tied tiogether - although I do need to 
check with the ISO docs again
TerryLongstreth >> (All): Sorry, trying to type to fast. TRP- repository being 
audited.
Mark Conrad >> (All): I vote that we go with Bruce's strawman and see what 
comments we get.
TerryLongstreth >> (All): what David said; recert on a qualifying event
RobertDowns >> (All): I agree that we should go with Bruce's strawman.
David Giaretta >> (All): ...so for example I could imagine that surveillance 
might mean a call asking if all the technology and staff have changed. If they 
have then a recertification would be in order - even if only to decide whether 
to withdraw the certification.
Mark Conrad >> (All): See 7.2.1(c).
BruceAmbacher >> (All): If SurveillanceCAP infers availability we may need a new 
section to define what that presence entails, what resources it can bring to 
that surveillance, and what "authority" it possess.
TerryLongstreth >> (All): And that's where some kind of continuous audit 
presence might be important; the repository should be responsible for notifying 
the auditors when a audit impact event happens.
BruceAmbacher >> (All): My strawman allows reaudit sooner if either the 
repository or the audit commitee think it is called for.
JohnGarrett >> (All): Again, I'm not as tied in as I used to be, but I think in 
the ISO 9000 case, there is nothing really active going on in the surveillance 
period.  However, if the certifying organization does notice something perhaps 
in the news, like a bankruptcy or some major change in the organization, then 
according to their contract, the certifying organization can pull the 
certification until they are assured that the audited organaizaition could still 
pass.
BruceAmbacher >> (All): John, That's the kind of role I see.  Perhaps reinforced 
by an annual reporting from the repository.
JohnGarrett >> (All): That sounds good to me.
David Giaretta >> (All): Re-reading Bruce's text - it looks OK 
TerryLongstreth >> (All): Yes to Bruce, but substitute periodic or continuous 
reporting.
JohnGarrett >> (All): If an organization is going to provide expert advice over 
time that would either need to be billed separately or if included in the 
original audit costs would increase initial costs quite a bit.
TerryLongstreth >> (All): strike my last,  substitute "periodic or regular 
reporting"
RobertDowns >> (All): Again, we need to ensure that the annual reporting 
requirement does not become prohibitively costly or time consuming for small 
repositories.
JohnGarrett >> (All): Do we have any view on what would be included in annual 
reporting?  Would it be detailed or would it just be a statement about major 
changes since the audit?
BruceAmbacher >> (All): Perhaps focus the annual report on actions taken to 
address weaknesses and/or actions taken that modify the preservation function
directly or indirectly.
David Giaretta >> (All): In terms of costs I guess there is a difference between 
the actual audit and any "consultants" which a repository might bring in to help 
it come up to scratch. The latter are outside our scope I would think and are 
determined by the market.
BruceAmbacher >> (All): Combine John's idea and mine
David Giaretta >> (All): Could one or other of you put text on the WIki page?
David Giaretta >> (All): ...or both
BruceAmbacher >> (All): which wiki page?  In which document?
David Giaretta >> (All):  http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/AuditorGuidelinesIssues
JohnGarrett >> (All): Bruce, if you're willing, I'll let you do it. I think your 
words are close and it would be better to have a single statement to start from.
David Giaretta >> (All): ...that is the page we are trying to put everything in. 
There may be implications for other parts of the whole doc but the rest of the 
doc is pretty generic
David Giaretta >> (All): By the way, what about my additions about the "initial 
audit committee"?
TerryLongstreth >> (All): They look good to me
Mark Conrad >> (All): Are these the places where it says lead audit committee in 
the text?
David Giaretta >> (All): If that's the case then one outstanding issue is 
"surveillance" - we should check the inherited definitions and see if we need 
something in thos doc or whether we can leave it to the operating procedures 
(not  standards) of the audit body.
David Giaretta >> (All): Mark - yes 
JohnGarrett >> (All): They look good enough for me.  I say put it in and see 
what comments we get in the review.
David Giaretta >> (All): ... assuming I was consistent
RobertDowns >> (All): There also are references to initial audit committee in 
the text
David Giaretta >> (All): Oops - I was afraid that I mixed "lead" and "initial"!!
TerryLongstreth >> (All): In an earlier discussion, there was a distinction. the 
Initial committee was responsible for bootstrapping the process
David Giaretta >> (All): Terry - yes - I rolled them together thinking that we 
would have to revise this standard at some point and perhaps refine the defition
Mark Conrad >> (All): Section 9.2.3.2 TDR 9.2.3.2 discusses a  Stage 2 audit is 
a Stage 1 Audit described some place else?
David Giaretta >> (All): ...i.e. revise in 3 or 5 years
BruceAmbacher >> (All): Mark,
David Giaretta >> (All): MArk - yes 9.2.3.1 I think
Mark Conrad >> (All): David, I don't see it.
BruceAmbacher >> (All):  Mark, I think the stage 1 audit is the background work 
by the repository - self assessment, and the offsite prep work by the audit 
team.
RobertDowns >> (All): 9.2.3.1 is not on the web page
David Giaretta >> (All): Mark - looks like Simon omitted it - it's in John's 
marke dup doc
David Giaretta >> (All): http://wiki.digitalrepositoryauditandcertification.org/pub/Main/ReqtsForAuditors/AuditorGuidelines-rev2009Sep09w17021InBoxes.doc
Mark Conrad >> (All): Need to look at the copmplete document to see if we are 
still missing anything.
David Giaretta >> (All): Yes, I guess Simon omitted very small sections which 
did not need any decisions. There are many places where the only change proposed 
is to put in the reference to the metrics doc
David Giaretta >> (All): Stage 1 is "The objective of the stage 1 audit is to 
provide a focus for planning the stage 2 audit by gaining an understanding of 
the TDR in the context of the client organization's TDR policy and objectives, 
and, in particular, of the client organization's state of preparedness for the 
audit."
David Giaretta >> (All): "The stage 1 audit includes, but should not be 
restricted to, the document review. The certification body shall agree with the 
client organization when and where the document review is conducted. In every 
case, the document review shall be completed prior to the commencement of the 
stage 2 audit."
David Giaretta >> (All): "The results of the stage 1 audit shall be documented 
in a written report. The certification body shall review the stage 1 audit 
report before deciding on proceeding with the stage 2 audit and for selecting 
the stage 2 audit team members with the necessary competence."
David Giaretta >> (All): "The certification body shall make the client 
organization aware of the further types of information and records that may be 
required for detailed examination during the stage 2 audit."
David Giaretta >> (All): Thanks about the whole of that sub-section
David Giaretta >> (All): oopos - "Thats about..."
Mark Conrad >> (All): So in this case the certification body is the audit team? 
the initial audit committee? something else?
JohnGarrett >> (All): Yes doing a  stage 1 audit (reviewing documents) is a cost 
saving measure.  You could possibly determine that an organization had no chance 
to pass and should do other work before a stage 2 audit.  And no travel expenses 
would result from a stage 1 audit.
David Giaretta >> (All): Mark - I guess in the startup period the only auditors 
will be the "initial audit committee"
David Giaretta >> (All): ...so they would make up those initial audits 
themselves
David Giaretta >> (All): .....when we then accredit other auditors then we will 
have more options
David Giaretta >> (All): .....and when we have related national certification 
bodies then things get very devolved
Mark Conrad >> (All): There are many places in this document where it is not 
clear who/what is being referred to. "Client organizations" "certifying body" 
etc.
David Giaretta >> (All): John - I guess that's a possibility
RobertDowns >> (All): Perhaps the terms intial audit committee and lead audit 
committee should be resolved so that they are consistent.
JohnGarrett >> (All): Yes, I think that at the beginning the initial audit 
committee will also make up the lead auditors, but I would hope that would 
change to having more lead auditors over time.  I think we need to keep the 
concepts separate.  Hopefully we can get a number of additional lead auditors 
before the the first time we need to update the document.
David Giaretta >> (All): Robert - yes - I intended that we use just a single 
term "initial audit committee"
Mark Conrad >> (All): Robert, I believe all the terms relating to all actors in 
the document should be clearly defined.
David Giaretta >> (All): It may be that because we are looking at extracts from 
the doc that things may be unclear.
Mark Conrad >> (All): I am not sure that terms are used consistently throughout 
the document.
David Giaretta >> (All): Once we resolve the issues we should put everything 
together and check  - but we can always go back to John's markup at the moment 
to clarify
JohnGarrett >> (All): I thought most of the terms were clear if you looked at 
the whole document.
David Giaretta >> (All): John - yes I hope so
David Giaretta >> (All): Mark - there may be two roles but initially I intended 
there be just one body
JohnGarrett >> (All): The certifying bodies are the organizations doing the 
auditing and granting certifications.  They themselves are possibly certified as 
acceptable auditing organizations by the initial audit committee.
JohnGarrett >> (All): Client organizations are the hopefully TDRs that contract 
with the certifying organizations to be audited.
BruceAmbacher >> (All): Can we address this by including clauses that will focus 
on launching the audit function and indicate they will be inoperable once the 
process is launched and will be deleted in the first revision?
David Giaretta >> (All): John - perhaps it should be "possibly accredited as 
acceptable auditing organizations by the initial audit committee. "
Mark Conrad >> (All): I believe it would be helpful if we defined terms in 
section three and used those terms consistently throughout the document.
JohnGarrett >> (All): Agreed
RobertDowns >> (All): I also agree
David Giaretta >> (All): Mark - yes
JohnGarrett >> (All): I think we need to look back and see which terms are 
already defined in the inherited documents.
David Giaretta >> (All): Bruce - yes - we can think of some wording or a 
footnote or something
David Giaretta >> (All): Mark - we should try to use the inherited definitions - 
I put my new "initial audit committee" definition on the Wiki page - we can add 
that the the full doc when we put it all together
BruceAmbacher >> (All): Here is my draft: "Recertification will occur no more 
than five years after the initial audit and certification.  It may occur sooner 
if the repository or the audit committee believes it is warranted.  The 
recertification process will focus on any initial weaknesses and a revolving 
subset of the audit criteria."“In the interim between audits the repository will 
be responsible for submitting an annual report that focuses on any significant 
changes in the repository’s procedures, staff, and/or actions taken that affect 
the repository’s mission to preserve information.” 
David Giaretta >> (All): Looks good
RobertDowns >> (All): Looks good to me, too.
Mark Conrad >> (All): and a revolving subset of the OTHER audit criteria?
BruceAmbacher >> (All): Mark, good addition.
David Giaretta >> (All): Bruce - will you put that on the Issues page ?
BruceAmbacher >> (All): I suggest locating this at a new 9.2.3.3.4. The 
recertification Process
BruceAmbacher >> (All): I will try to put it on the issues page.
David Giaretta >> (All): How about ACTIONS: (1) all to check definition of 
surveillance is the ISO docs (2) all to check the wiki page on issues (3) all to 
add things to the draft on 
http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/DraftPlanForOrganisation
and (4) Bruce to add text on recertification to issues page
JohnGarrett >> (All): Bruce, I like it.
David Giaretta >> (All): Any other actions?
BruceAmbacher >> (All): Time to sign off?
Mark Conrad >> (All): David, Are you sure that is the right URL?
David Giaretta >> (All): which one?
David Giaretta >> (All): Draft plan for organisation is http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/DraftPlanForOrganisation
David Giaretta >> (All): Issues page is http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/AuditorGuidelinesIssues
David Giaretta >> (All): John's marked up doc is http://wiki.digitalrepositoryauditandcertification.org/pub/Main/ReqtsForAuditors/AuditorGuidelines-rev2009Sep09w17021InBoxes.doc
David Giaretta >> (All): ...I think
Mark Conrad >> (All): So which one are we supposed to posting what to?
BruceAmbacher >> (All): I will post the strawman to the issues page
Mark Conrad >> (All): 1) URL?
Mark Conrad >> (All): 2) URL?
Mark Conrad >> (All): 3) URL?
David Giaretta >> (All): Issues on the issues page and ideas on how to set up 
the organisation in practice on the plan for organisation page
Mark Conrad >> (All): Issues is http://wiki.digitalrepositoryauditandcertification.org/bin/view/Main/AuditorGuidelinesIssues?
David Giaretta >> (All): Yes
David Giaretta >> (All): (1) refers to JOhn's doc or the other PDF's you have 
containing the "parent" docs
David Giaretta >> (All): (2) and (4) refer to the issues page
David Giaretta >> (All): (3) refers to the "plan for organisation" page
Mark Conrad >> (All): Where should we post anything we have for 1)?
David Giaretta >> (All): Good question - we can add it to the "issues" page I 
guess under "general points and issues to check"
Mark Conrad >> (All): Ok.
David Giaretta >> (All): ... or just email any thoughts to the list
TerryLongstreth >> (All): I vote for email
David Giaretta >> (All): ........if that's easier
JohnGarrett >> (All): OK.  I'll catch you all next week
BruceAmbacher >> (All): I am signing off now
Mark Conrad >> (All): We are going to have parts of this scattered all over the 
place.
David Giaretta >> (All): I may be in a meeting in the hague next week
David Giaretta >> (All): I think everything about the draft standard is on the 
"issues" page. Ideas about how to set the thing up are on the "plan for 
organisation" page.
TerryLongstreth >> (All): For the rest of us; same time and place next week?
Mark Conrad >> (All): See you then.
RobertDowns >> (All): Bye
David Giaretta >> (All): Once we agree on the "issues" we merge that into the 
Word doc and remove Johns text boxes at the very end
David Giaretta >> (All): Bye all

-- SimonLambert - 16 Nov 2009