Notes from Megameeting 20th April 2009

Attendees

Summary

There was discussion of how best to use the time in the face-to-face CCSDS meeting at Colorado Springs.

The comments and proposed changes in the current working document were reviewed and actions set as below.

A provisional timetable was proposed: metrics doc more or less frozen by end of April, and test audits completed by mid to end of May.

There was discussion of how to conduct test audits and the need for some degree of comparability between different auditors' assessments, bearing in mind that it is impossible to specify the metrics precisely and there will always be some judgement required.

Actions (see transcript for details):

• BruceAmbacher to amend sections A1.1 and B2.
• MarieWaltz to amend discussion/examples in A1.2.
• MarieWaltz to amend A2.1, A3.2 and A4.2 as discussed.
• JohnGarrett to add extra examples in B4.1.
• ReaganMoore to clarify C1.

Transcript of chat

David Giaretta >> (All): I've put comments on Bruce's comments, put up a draft 
Intro and also Annexes on Wiki
BruceAmbacher >> (All): Putting the references in the auditor's document makes 
the most sense.  That would give them some of the background for the document.
BruceAmbacher >> (All): We still need to finalize our appproach to B2, then kick 
my butt to get it done.
David Giaretta >> (All): So I'd like some guidance on what we can do in the 
meetings here in COlorado Springs
David Giaretta >> (All): One thing we could do is work on the Annexes IF people 
think that's a good idea
David Giaretta >> (All): Another thing would be to work on the Intro
BruceAmbacher >> (All): Analyze the CNES comments against the present document 
structure.
David Giaretta >> (All): A third would be to check with the publications editor 
about wording
BruceAmbacher >> (All): Determine a realistic timetable for the rest of the 
process.
David Giaretta >> (All): Yes 4 - speak to Danielle about her report and see how 
we can "normalise things"
David Giaretta >> (All): Assuming there are no showstoppers in the metrics then 
the Test Audits are on the critical path
BruceAmbacher >> (All): I did not fully accept Danielle's greater emphasis on 
software and the IT side.
David Giaretta >> (All): I need to read her doc more carefully - only had time 
for a quick glance
David Giaretta >> (All): As we were saying Bruce - Danielle missed the face to 
face meeting and also took a snapshot of the Wiki which is old
David Giaretta >> (All): That's why I wrote "normalise"
BruceAmbacher >> (All): I also only scanned it.  Another theme that came through 
is to reorganize and put all things relating to ingest and producer relations 
together.  I think the document flows well now and would have to be convinced 
otherwise.
David Giaretta >> (All): Yes, I'd hate to do anything more than some tweaks to 
the metrics
Barbara Sierman >> (All): David, what need to be done by us before the document 
is ready?
David Giaretta >> (All): I suggest we focus on (1) Bruce's coments (2) guidance 
for us here at Col Springs about the Intro and Annexes 
David Giaretta >> (All): and (3) organise the text audits
David Giaretta >> (All): Barbara - I think we need to all review the metrics 
then finalise the documents as a whole e.g. introductions etc, and also complete 
the text audits as we discussed in Washington - anything else?
Barbara Sierman >> (All): David, before next Monday?
BruceAmbacher >> (All): When will we address the auditor's document?
David Giaretta >> (All): Barbara - we need to have agreement on how and when 
these things are completed - we were aiming for end of April/mid May for the 
test audits 
Barbara Sierman >> (All): Yes, I know, but sometimes to state a date might help
BruceAmbacher >> (All): For the test audits, how much prep time do the 
repositories want/need?
David Giaretta >> (All): Bruce - we have a draft on the wiki of the Reqs for 
Auditors - I'd hope we can focus on that in megameetings by the end of April
David Giaretta >> (All): I hoped we had been doing some preparatory work 
already(??!!)
BruceAmbacher >> (All): Against a still evolving document?
David Giaretta >> (All): I would hope we are going to finish our comments by the 
end of the month and the changes by then would just we small ones
David Giaretta >> (All): The test audits may suggest a few more tweaks
RobertDowns >> (All): We might consider making a copy of the document that we 
will use for the test audits so that we have a common frame of reference for the 
test audits.
David Giaretta >> (All): As for dates - if we say we are running about 2-4 weeks 
late then we need test audits completed by mid to end of May
David Giaretta >> (All): and the metrics doc more or less frozen by end of April 
at the very latest
David Giaretta >> (All): So we need sign off by all by the end of the month
David Giaretta >> (All): The Reqs for Auditors doc could wait a little longer 
since I don't think it affects the test audits (much??)
BruceAmbacher >> (All): It may be hard to complete B2 by then if we don't get 
the framework for it settled today.
Helen Tibbo >> (All): So, let's get on with the framework!
David Giaretta >> (All): Good idea - but I thought B2 was done and needed 
reviewing
David Giaretta >> (All): Not sure what you mean by framework
BruceAmbacher >> (All): I did B2 but it retains the original structure of many 
requirements/subrequirements without supporting text, discussion and examples.  
It now is very different from the remainder of the document but I'm not sure 
there is enough verbage to make it conform with the others.
David Giaretta >> (All): Yes there were quite a number of sub-metrics
BruceAmbacher >> (All): That was done at the face-to-face but we did not discuss 
how to fully structure and populate those sub-metrics.
David Giaretta >> (All): If we cannot think of additional text for the sub-
metrics then I think that we leave them as they are - they are pretty explicit 
as they are
Barbara Sierman >> (All): Yes and wait for the test audits and their comments 
about how this works in practice
David Giaretta >> (All): Good point
BruceAmbacher >> (All): Ok.  I will make another pass through it to try and 
bring it more in line with the other sections.  Perhaps the meeting Thursday can 
take it a step further,
RobertDowns >> (All): It looks like about 8 sub-metrics are missing supporting 
text.
David Giaretta >> (All): Looking at 27001 our B2 sub-metrics are pretty similar 
to most of the 27001 items - they don't have supporting text because they are 
pretty explicit as they are
BruceAmbacher >> (All): Ok.  Let me see what I can complete later today.
David Giaretta >> (All): OK but I don't think we need words just for the sake of 
having words
Barbara Sierman >> (All): Was this meant by "Bruce comments? "
BruceAmbacher >> (All): Agreed
BruceAmbacher >> (All): Barbara, No.  I have attached comments in the other 
sections.  Some just edits, a few to change words, a few on things still absent.
David Giaretta >> (All): Barbara - Bruce has put comments into many of the 
sections and I have put my views immediately after his - I guess the question is 
what others think. Also Bruce raised some questions we could discuss
Barbara Sierman >> (All): OK, I saw them too
Barbara Sierman >> (All): start with A1.1?
David Giaretta >> (All): Bruce's comemnts have "16April2009" in the tag
BruceAmbacher >> (All): In A1.1 I commented: Does "highest level" refer to the 
organizational hierarchy or to the largest financial commitment possible for 
preservation activities?DG 20090418 I assumed this is oragizational aspects
Barbara Sierman >> (All): I would agree with David
Barbara Sierman >> (All): so organizational level
BruceAmbacher >> (All): That was my thought but it is ambiguous.
RobertDowns >> (All): Yes, I believe that is the case.
BruceAmbacher >> (All): We can state it explicitly.
David Giaretta >> (All): Will you make the change Bruce?
BruceAmbacher >> (All): Sure.
David Giaretta >> (All): For the next one I'll be checking with the CCSDS 
docuemnt editor this week
Barbara Sierman >> (All): ok
David Giaretta >> (All): The next few seem OK to me then in A1.2.1 there is a 
question about whether text should be in Examples or Discussion
Barbara Sierman >> (All): next one "as" is OK with me
David Giaretta >> (All): Yes those seemed pretty clear 
RobertDowns >> (All): Yes, the rest in A1.1 appear to be good suggestions.
Barbara Sierman >> (All): I agree
KatiaThomaz >> (All): ok for me
David Giaretta >> (All): Just looking at the tricky ones A1.2.1, A1.3 need some 
thought
BruceAmbacher >> (All): Now the discussion in A1.2 is all examples. My comment 
in A1.3 goes to what should be in a Collection Policy and whether it is wise to 
cram everything in one or better to let various parts address relevant issues.  
This also gets to part of Danielle's document
Barbara Sierman >> (All): How did we define discussions? I agree with Bruce, 
you expect the examples to be in Examples and not in discussion.
RobertDowns >> (All): Yes. I agree that those appear to be examples and should 
be under Examples
BruceAmbacher >> (All): A1.2 discussion is just a list.  Little or no real 
discussion of intent, how to execute, what is crucial to the requirement, or 
amplification on the supporting text..
David Giaretta >> (All): Yes, so what should be in Discussion?
David Giaretta >> (All): A was Bernie and Marie - we should give them an ACTION 
to correct this
BruceAmbacher >> (All): ok
David Giaretta >> (All): In A1.3 I agreed with Bruce
Barbara Sierman >> (All): ok, they can look at this "discussion" to have an idea 
of our worries
KatiaThomaz >> (All): ok
David Giaretta >> (All): Yes
RobertDowns >> (All): yes
BruceAmbacher >> (All): David, did preservation policy get here as part of 
your "global" insertion of preservation policy?
David Giaretta >> (All): Hmmm - could be but that was a long time ago
David Giaretta >> (All): ...but I'd only have put it there is there was some 
similar wording already - again quite a long time ago
David Giaretta >> (All): I'm quite happy with taking those things out of 
Collection Policy
BruceAmbacher >> (All): The requirement is discussing collection policy.  While 
that impacts what is preserved, is it necessary to be explicit about 
preservation policy?  that implies it is something different from the 
preservation implied in what a repository collects.
BruceAmbacher >> (All): ok
David Giaretta >> (All): In A2.1.1 I looked back at earlier versions and 
suggested the words from that
David Giaretta >> (All): In A2.1.2 Bruce wanted "preservation" instead of 
"archiving" - OK with me
Barbara Sierman >> (All): ok with me too
David Giaretta >> (All): Same in A3.2
BruceAmbacher >> (All): The rewrite of A3.2.2 does not address Katia's comment 
from a year ago.  It should be easy to give a few examples of the types of 
rights a repository might be looking to acquire along with the data objects.
RobertDowns >> (All): Yes, in both cases, the word preservatioon is more 
specific.
KatiaThomaz >> (All): agreed
David Giaretta >> (All): I guess these are ACTIONS for Marie
David Giaretta >> (All): Also A4.2 - the repeat text - ACTION for Marie
BruceAmbacher >> (All): My only other comment was in A5.2 to incorporate 
Robert's comment into the text.
Barbara Sierman >> (All): yes, sounds sensible
David Giaretta >> (All): Yes agreed
David Giaretta >> (All): In B4.1 looks as if we need some extra Examples
David Giaretta >> (All): ...that's an ACTION for John
David Giaretta >> (All): C1.1.2 - Bruce had a question about "fire-drills"
David Giaretta >> (All): C1 was Reagan's - ACTION for him to clarify
David Giaretta >> (All): Sorry to rush but what about the Intro and Annexes?
BruceAmbacher >> (All): I read it once and like what it is trying to do.  I 
assume some of it was from the TRAC intro.  I think the references can go in 
both documents.
David Giaretta >> (All): Good - we can add some more References and maybe divide 
into Normative and Informative sets
David Giaretta >> (All): I also put something about the "Plan-Do-Check-Act" but 
needs more thought
David Giaretta >> (All): I thought that we might be able to do that here at 
COlorado Springs and propose something if people thought it worthwhile
BruceAmbacher >> (All): Where is the Plan-Do-Check-Act?
David Giaretta >> (All): I I realise I took that phrase out - it was in the 
"Process Approach" section
David Giaretta >> (All): The key sentence is "This International Standard should 
be used within a process approach for establishing, implementing, operating, 
monitoring, reviewing, maintaining and improving an organization's 
trustworthiness, see for example ISO 27001:2005."
David Giaretta >> (All): You might guess that I took inspiration from 27001
BruceAmbacher >> (All): Could we also see for example ISO 9004?  That links it 
back to process and recordkeeping within the 9000 suite.
David Giaretta >> (All): Sure - feel free to add that to the Wiki
BruceAmbacher >> (All): I am less certain about using ISO 15489, the records 
management standard.
David Giaretta >> (All): Yes I agree
David Giaretta >> (All): I guess I'd like to be able to put up a pretty complete 
documents (OK still needing some tweaks for the metrics) by the end of the week 
- with proposed Intro, Annexes etc
BruceAmbacher >> (All): With my edits to B2 and the edits to A we should be 
there.
David Giaretta >> (All): We still need to discuss the Req for Auditors but I 
think we can work on the boiler plate while we meet here in Col Spr.
Helen Tibbo >> (All): The intro looks good to me. 
David Giaretta >> (All): If we do it all on the Wiki it will just be a matter of 
pasting into Word and doing a little formatting
David Giaretta >> (All): Then any substantial changes will be done by the group 
as a whole but those meeting here can be doing something useful
David Giaretta >> (All): Ok then the test audits
RobertDowns >> (All): I also agree that the intro reads well.
David Giaretta >> (All): Sorry to rush on - We had the outline of a plan and 
also work by Danielle
Helen Tibbo >> (All): To jump to another topic for a minute. Is there any sort 
of data collection template for when we do the test audits? 
David Giaretta >> (All): I suggest we make a Wiki page to track the progress of 
each test audit
Barbara Sierman >> (All): Do the audited groups like that to be known to 
everyone?
David Giaretta >> (All): Helen - good point - I was going to try to make 
available a little tool we could all use - but it would have to be next week
Helen Tibbo >> (All): Well, I am thinking more of what we actually expect to 
appear in a data collection instrument.
BruceAmbacher >> (All): How "public" will that wiki be?  That might  hinder the 
audits.
Helen Tibbo >> (All): OK,two topics at once here...
David Giaretta >> (All): Bruce - we can make pages Private - password protected
Barbara Sierman >> (All): ok
David Giaretta >> (All): So - Helen
David Giaretta >> (All): ... any suggestions about what should appear
BruceAmbacher >> (All): Helen, Doesn't your collection document come out of the 
annex that will contain all of the requirements?  with space for comments or 
scores?
David Giaretta >> (All): Yes, I assumed we would be collecting Evidence for each 
metric plus some comemnts on the quality and tests done
BruceAmbacher >> (All): A second starting point would be the audit sheets used 
in the 2007 series of test audits.
David Giaretta >> (All): Are they public?
Helen Tibbo >> (All): Well, this gets back to how an auditor will conduct the 
audit. If the repository doesn't have an item at all that's easy. But what if 
they have one, or only 1/2 have one, or have a poor one?
David Giaretta >> (All): Also we need to align with the metrics we have
BruceAmbacher >> (All): David, I was referring to any blank template.  
Helen Tibbo >> (All): Also, if there's no specific data collection guidance, it 
will be hard for repositories to use this standard for their own self-
development.
BruceAmbacher >> (All): Helen, I assume no requirement is absolute.  All are "it 
depends" and are graded thusly.
David Giaretta >> (All): Helen - yes, this gets to the heart of it - I think we 
agreed that we would not have a marking scheme but it would be the judgement of 
the auditors
Helen Tibbo >> (All): Ah, so we can all write a lot of text, but what about this 
"grading" thing??? How do we normalize our practice on that without some 
explicit guidelines?
Helen Tibbo >> (All): So, this really means that repositories all hope for the 
easy grader....
BruceAmbacher >> (All): Each requirement, while mandatory, is not of equal 
value.  That is where the experience, training, and judgment of the auditors is 
so important
Helen Tibbo >> (All): Oh, my God!
David Giaretta >> (All): That is why we need the Prime Audit Group
David Giaretta >> (All): ...to set the norm
BruceAmbacher >> (All): I thought we already hired DBD - David, Bruce, Don
Helen Tibbo >> (All): And who is the Prime Audit Group?
David Giaretta >> (All): ..... not sure how else to do it. We just cannot 
specify enough
David Giaretta >> (All): Helen - I guess we are - also all other certifiers 
would (see Req for Auditors) have worked with us or someone who had worked with 
us 
David Giaretta >> (All): ...hence we are the "Prime"
BruceAmbacher >> (All): These are the test audits - to test the requirements and 
to test the audit process and to obtain feedback so we can develop more rigid 
metrics for certain crucial requirements
RobertDowns >> (All): We might consider a place in the template for responses 
from the repository to address the auditor's comments.
David Giaretta >> (All): Bruce - yes - identify where the probelms are and do 
our best to fix them
BruceAmbacher >> (All): Very good feedback and justification
David Giaretta >> (All): Yes - I mentioned "test" but should also have "response 
from archive"
David Giaretta >> (All): Helen - not sure if we can use Drambora - need to look
BruceAmbacher >> (All): We already see in Daniele's document that others can 
take a different view of what a requirement means and how to address all aspects 
of a relationship.
Helen Tibbo >> (All): OK, in the education world we know that when there is 
fuzzy grading of standardized tests, let's say writing samples to pass out of 
8th grade or for the GREs, the graders need to have common cases that they 
evaluate and then discuss those evaluations to come to a common understanding. I 
am EXTREMELY worried that this will be very willy-nilly if we don't do this.
David Giaretta >> (All): Helen - yes - we have to get our act together but I 
think (hope) we have time to do that after we put the docs into ISO for review - 
we'll have about 6 months at least
BruceAmbacher >> (All): Helen, I share your concern for the real audits, less so 
for the tests.
David Giaretta >> (All): SO the test audits test the docs for clarity and test 
ourselves a little for consistency. But we'll have to do some more work on the 
latter
Helen Tibbo >> (All): I am not so worried about the specific metrics, but the 
process and how we apply them. What do we know from the audits that CRL has done 
with TRAC? What was their process? Their tools? 
BruceAmbacher >> (All): I doubt we ever will have a large enough set of audits 
to have multiple blind panels reviewing the results and all assigning scores, 
then averaging them.
Helen Tibbo >> (All): Also, I know the Drambora folks spent a lot of time on 
audits and have created a "tool kit" for repositories to better collect data 
about themselves.
David Giaretta >> (All): Helen - what I saw of the docs from 2007 it was not 
clear what tools they used but I can ask Marie
BruceAmbacher >> (All): This is another good discussion topic for your face-to-
face on Thursday.
David Giaretta >> (All): Helen - yes I would hope we can use Drambora but we may 
need to add some risk templates to tie in with our metrics
David Giaretta >> (All): I did speak to Seamus a while ago about this but 
nothing has happened about it yet
David Giaretta >> (All): However good tools are something we have time to tailor 
I hope
BruceAmbacher >> (All): OK.  I am signing off.  David, let us know the results 
later this week.
Helen Tibbo >> (All): So, if at the end of the day a repository lives or dies 
based on its certification, we had damn well better be able to justify 
explicitly the success and failure points or we will be open to lawsuits. This 
isn't just a wine tasting where experienced tand opinionated tastebuds win the 
day.
David Giaretta >> (All): Will do - also will send details of any MegaMeeting 
Thursday - but may have to be by phone because of Daniele's communication 
restictions
David Giaretta >> (All): ...that was one reason I added something about Process 
in the intro - if someone fails then that just leads to an improvement plan - 
and in fact no-one will be perfect
Helen Tibbo >> (All): Talk next week. Good Luck in Colorado.

-- SimonLambert - 21 Apr 2009