Notes from Megameeting 29th October 2007

Attendees:

BarbaraSierman Koninklijke Bibliotheek, Netherlands
BruceAmbacher UM
CandidaFenton HATII, Univ Glasgow
SimonLambert STFC
DavidGiaretta STFC
DonaldSawyer NASA GSFC
HelenTibbo UNC
JohnGarrett NASA/GSFC
MarkConrad NARA
RobertDowns Center for International Earth Science Information Network (CIESIN), U Columbia

Almost all the discussion at this meeting was conducted by chat, so the following transcript of the meeting (with a few typos corrected) is complete.

Action: SimonLambert to set up wiki page allowing participants to rate requirements as to whether mandatory or not.

Helen Tibbo >> (All): Where do we start today?
David Giaretta >> (All): Since I have been away for several meetings can someone volunteer to chair?
David Giaretta >> (All): How about you Helen?
Helen Tibbo >> (All): OK, but I missed much of last week.
David Giaretta >> (All): SOmeone else?
Helen Tibbo >> (All): I think the biggest issue on the table was the mandatory vs voluntary notion of the standard
Helen Tibbo >> (All): Do we want to pick up with that?
David Giaretta >> (All): Presumably this ties in with the marking scheme?
Mark Conrad >> (All): I would put it more broadly as the scope of the proposed document.
David Giaretta >> (All): I assume the scope is as broad as we can make it
Helen Tibbo >> (All): Right, we were charged with thinking about "    * "Mandatory" versus "risk assessment" approach in certification" in the minutes from last week.
Mark Conrad >> (All): A large part of the discussion centered on whether or not the document would contain only mandatory elements and if not how we would identify which elements were mandatory and which were not.
Mark Conrad >> (All): Hi John. Welcome back Bruce.
David Giaretta >> (All): Mark - it is also tied to the marking scheme in that we can show which things are mandatory using that
Helen Tibbo >> (All): It strikes me that a repository , like a chain, is only as strong as its weakest link. And that many currently "successful" repositories are only successful because the weak links have not been pulled too much.
JohnGarrett >> (All): HI All
David Giaretta >> (All): Presumably we might include non-,mandatory things which could apply in certain circumstances and not others.
David Giaretta >> (All): The problem with just "mandatory" is that for most things there are no yes/no answers - Is that right?
Helen Tibbo >> (All): What does it mean for a requirement not to be mandatory? Does that mean that it is not one of the links in the chain but an extra bell and whistle?
Mark Conrad >> (All): David, we have not really discussed a markup scheme. I believe that if the document is to contain madatory, non-mandatory and mandatory in certain cases requirements, each requirement should be explicitly identified as such.
JohnGarrett >> (All): I think that is right.  If everything is mandatory, then there will mostly be yes or no answers.
Helen Tibbo >> (All): On the other hand, very few repositories can presently meet all these requirements and they are working and continuing.
RobertDowns >> (All): Repositories could have different requirements for each collection maintained
David Giaretta >> (All): John - but most of the things are actually "grey"
JohnGarrett >> (All): And if something can be a range of things, we will have to say that it must be at least this good before you can say "yes"
Helen Tibbo >> (All): Well, aren't we assessing risk rather than talking in absolutes now?
JohnGarrett >> (All): Yes, I agree most things are grey.  That will put a large burden on us to say how good is good enough to be certified, if we make the standard just mandatory items.
David Giaretta >> (All): Helen - yes - we cannot be certain - hence the need for a marking scheme.
Helen Tibbo >> (All): So, should there be levels - silver, gold, platnium?
David Giaretta >> (All): My problem has been that I don't see such a thing in ISO 27001 for example
JohnGarrett >> (All): I've been a fan of CMMI type of things which have levels.
David Giaretta >> (All): Yes - but I was meaning for the metrics themselves
Helen Tibbo >> (All): I need to sign off as we're taking another flight
David Giaretta >> (All): Bye Helen
JohnGarrett >> (All): CMM had metrics for the same type of things but how much you did had different requirements at the different levels.
JohnGarrett >> (All): Bye Helen
David Giaretta >> (All): Is it possible to say that everything is mandatory unless explicitly stated in the explanatory text?
Mark Conrad >> (All): David, I would be happy if we had a document that explicitly identified what requirements are mandatory and which are not.
David Giaretta >> (All): I think we are saying the same thing then
Mark Conrad >> (All): I think that it would be interesting to go through the TRAC document and see how many of them are mandatory for all repositories.
candida fenton >> (All): So if we agree that we must specifiy what is mandatory and what is not, then the next step is to decide what is mandatory and what is not
David Giaretta >> (All): The problem is that many things are "grey" 
David Giaretta >> (All): ..in that we say that we need say "logs" but how many, and in what detail?
RobertDowns >> (All): For the "grey" items we might need to specify how they are measured
Mark Conrad >> (All): Are there any of the requirements that are black and white? Perhaps we should start there.
Helen Tibbo >> (All): Are they grey or is it the response to them, i.e, what type of documentation or level of infrastructure that is variable in acceptance.
Helen Tibbo >> (All): I'm back for a few minutes.
David Giaretta >> (All): Helen - I was thinking of the greyness of the possible responses
Mark Conrad >> (All): Can Robert and Candida hear David and John using the audio?
David Giaretta >> (All): We could say what the mimimum acceptable level is for any "grey" metric
RobertDowns >> (All): I do not have my headset today
candida fenton >> (All): I am not getting any audio
David Giaretta >> (All): Lets continue with typing then
JohnGarrett >> (All): OK then back to typing
cclrc >> (All): How much discretion are we allowing  to the human auditor/certifier?
David Giaretta >> (All): So... assuming we have finihsed our first run through, we can begin looking at the "greyness" or otherwise of the metrics and check that we have set a minimum level 
JohnGarrett >> (All): The question was how to handle grey criteria.  David suggested that we may state the minimum acceptable level
Mark Conrad >> (All): A lot of the discussion over the past few weeks has been about requirements not being applicable to a particular type of repository. If we are to come up with a broadly acceptable document, shouldn't we first identify requirements that all repositories should meet?
JohnGarrett >> (All): Yes I think we could go through and try to assign minimum levels to grey items.  But I don't think we made it past B4 yet in our first pass.
Mark Conrad >> (All): Presumably such requirements would require more black and white responses.
David Giaretta >> (All): Simon - I assume there will have to be some discretion since even where we set a minimum level the auditor must evaluate whether it is met
David Giaretta >> (All): I'd be a bit wary of starting from scratch if that's what you mean Mark
RobertDowns >> (All): Yes, the guidance for measuring should be directed to the repositories and to the auditors to reduce ambiguity
David Giaretta >> (All): I assume that a lot of this derives from the discussion of Provenance and Authenticity
Mark Conrad >> (All): I don't mean start from scratch. I mean go through what we have in the TRAC document and supporting documents and sort out those items that should be mandatory across all repositories.
David Giaretta >> (All): ....can we address those and maybe get to the  heart of the issue - it may throw light on what we should do in other cases
Helen Tibbo >> (All): I agree with Mark. This would be a minimum standard but this would have to be linked to outcomes because what would the non-required items mean?
Mark Conrad >> (All): David, I believe the discussions of provenance and authenticity are a symptom of not having a clear scope for the RAC document.
David Giaretta >> (All): Mark - not sure about that - I think it has to do which "truth in advertising"
Helen Tibbo >> (All): It's hard to do this in the abstract. Does someone have a repository we could test our requirements against (or could we fabricate one) so that we could see the ramifications of required and not-required?
Mark Conrad >> (All): Much of the discussion of provenance and authenticity have been around what should be mandatory and what should not across all the different types of repositories.
David Giaretta >> (All): What I mean by that is that there are very many things that a repository can do - collect provenance right the way back to the origin or not
David Giaretta >> (All): .... but as long as the repository makes it clear what it has collected from the "producer" and then maintains the record of provenance then that is OK
Mark Conrad >> (All): Don has been arguing that collecting provenence to origin should not be mandatory because some repositories do not do this.
cclrc >> (All): David - So would a requirement for provenance back to origin apply to certain (defined) types of repositories?
Helen Tibbo >> (All): Bye
Mark Conrad >> (All): The point is that the discussion focuses on what is mandatory vs. non-mandatory whether we are discussing provenance, authenticity, trustworthiness or some other concept.
David Giaretta >> (All): Simon, what I mean is that the repository does not - in general - HAVE to collect Provenance back to the origin - BUT it HAS to say what provenance it has collected from the Producer
JohnGarrett >> (All): I lot of it comes down to what the Designated Community for an Archives is comfortable with in some areas.
David Giaretta >> (All): Mark - yes I agree - I was hoping that focussing on Provenance would help clarify some issues
Mark Conrad >> (All): David, Again, I think provenence, authenticity, etc are a symptom not the underlying problem.
Mark Conrad >> (All): David, your statement about provenance to Simon is not reflected in the working document as it is written.
BruceAmbacher >> (All): Mark what is your concept of the underlying problem?
JohnGarrett >> (All): I agree that provenance, authenticity, etc are the symptoms.  But if we can figure out how to handle them, it would be useful.  Handling a particular 'grey item' might help us in knowing how to approach other grey items.
David Giaretta >> (All): John - that's what I meant exactly
BruceAmbacher >> (All): Do we use annexes to illustrate different community approaches and why they are satisfactory to that community?
Mark Conrad >> (All): I believe we need to identify the core set of mandatory requirements  - no matter what type of repository. If we then want to add non-mandatory, or mandatory in certain circumstance requirements, they should be explicitly labeled as such. If we cannot identify a core set that all repositories should meet, I do not see how we can develop an international standard.
David Giaretta >> (All): Mark - so one "core requirement" is not to lose the bits
BruceAmbacher >> (All): Mark: our discussions have demonstrated provenance can not be an absolute so do we just not mention it?  I agrue we must still mention it and its major role for most repositories
BruceAmbacher >> (All): David - or the meaning and context of the bits as a repository received and stored them.
Mark Conrad >> (All): Bruce, I agree, but it should be explicitly listed as not being mandatory across all repositories if you accept Don's arguments in this area.
BruceAmbacher >> (All): agreed
David Giaretta >> (All): Bruce - I think you put your finger on it. I think Mark's point is a good one but rather than throw the baby out with the bathwater we just need to be careful in the metrics
BruceAmbacher >> (All): So much will come back to desirables and why, with few mandatories, unfortunately.
JohnGarrett >> (All): I think Don would argue that provenance from ingest on is required and possible.  Provenance before that is desirable, but may not be possible to obtain.
Mark Conrad >> (All): David, I do not believe this can be handled in the metrics. It should be explicit in the structure of the document. I do not believe we need to throw out the baby to accomplish this.
David Giaretta >> (All): The metric on Provenance should be something like "be clear what provenance you start with from the producer - and clearly state your policy on this - and then maintain the provenance 
David Giaretta >> (All): ... in your activities
David Giaretta >> (All): Mark - not sure what change in structure you are suggesting.
JohnGarrett >> (All): I agree that will be a lot of desirable things, but I think a lot of things will have a mandatory minimum level and then additional desirable additions.
Mark Conrad >> (All): It could be something as simple as putting an asterisk in front of all requirements that are mandatory across all repositories.
David Giaretta >> (All): Can we pull these ideas together and say that there is a minimum set of metrics for "Level 1", then if you pass another set of metrics also then you get to "Level 2"?
David Giaretta >> (All): So the "core" metrics would be those for Level 1
candida fenton >> (All): If we could identify and agree core requirements, I think it would be a big step
Mark Conrad >> (All): I agree.
David Giaretta >> (All): Yes
RobertDowns >> (All): yes
BruceAmbacher >> (All): Yes
JohnGarrett >> (All): Yes
BruceAmbacher >> (All): I think there is more consistency and agreement than we suspect
David Giaretta >> (All): By the way, DID we finish our first run through TRAC?
cclrc >> (All): No, I don't think we got beyond B4.4
JohnGarrett >> (All): I don't think so
BruceAmbacher >> (All): No.  I think we are at B6.10
David Giaretta >> (All): My recollection was that there were just a few points which had comments
Mark Conrad >> (All): So should we each go through the requirements come up with a list of the ones we think are mandatory and then compare notes?
David Giaretta >> (All): Mark - sounds good
David Giaretta >> (All): Mark - but it may be that rather than simply metrics, it is also the level of evidence needed
Mark Conrad >> (All): Do we want to do this before or after we go through the requirements from B.6.10 and on?
RobertDowns >> (All): In some cases, it might be not be clear if the required evidence is ambiguous
BruceAmbacher >> (All): Now how can I dissociate myself from a document I co-chaired?
David Giaretta >> (All): Bruce - why would you need to do that?
BruceAmbacher >> (All): I am hearing the same discussion we had for so many months in developing TRAC.
Mark Conrad >> (All): For the first round I think it would be useful to identify the core requirements, then worry about the metrics and evidence in subsequent rounds.
JohnGarrett >> (All): What was conclusion from discussions during TRAC?
Mark Conrad >> (All): Bruce, Are there lessons that we can learn from the TRAC discussions?
David Giaretta >> (All): Bruce - I think we are moving on to what one might term a marking scheme
David Giaretta >> (All): ...and this is consistent with the maturity levels you advocated then
David Giaretta >> (All): ......so perhaps we are moving forwards
BruceAmbacher >> (All): I'm not saying I am uncomfortable with where we are going.  We also need to allow the discussion of additional concepts if people want them.
Mark Conrad >> (All): David, I am not ready to move on to a marking scheme until we can agree what the core requirements are.
David Giaretta >> (All): Mark - I meant that that is the first step in a rudimentary marking scheme
Mark Conrad >> (All): Ok
JohnGarrett >> (All): I think that marking scheme may be part of determining what the core requirements are.
David Giaretta >> (All): So perhaps we can hand on heart say that we have pretty well gone through the TRAC metrics, tweaking them, and now we move to new ground?
BruceAmbacher >> (All): So, for example, provenance is a required element and when it begins for a type of data or a designated community is the variable that an auditor measures?
Mark Conrad >> (All): Again, should we each go through the requirements come up with a list of the ones we think are mandatory and then compare notes?
David Giaretta >> (All): Mark - sounds like an action on us all
Mark Conrad >> (All):  or are you suggesting a different approach to identifying the core requirements?
David Giaretta >> (All): Mark - no - just flagging up the core requirements - metrics plus evidence
Mark Conrad >> (All): Is there a way to post a matrix on the wiki listing each of the requirements down one axis and our names across the other and we could each mark which requirements we think should be mandatory?
David Giaretta >> (All): We should be able to do that
BruceAmbacher >> (All): Does the "crosswalk" help in this at all?
David Giaretta >> (All): ...however we probably also need some notes if there are some extra bits about the evidence
Mark Conrad >> (All): That would make it easy to see where there are area of agreement and disagreement.
David Giaretta >> (All): Bruce - did any of the other docs specify "core" requirements?
Mark Conrad >> (All): Bruce, Which document are you calling the cross-walk?
David Giaretta >> (All): Mark - maybe you're right - just a simple yes/no for core/non-core then progress from there
BruceAmbacher >> (All): Mark, I don't think so.
cclrc >> (All): Just to clarify - these are requirements at the level of Bm.n, right?
cclrc >> (All): For example, "B5.2 Repository captures or creates minimum descriptive metadata and ensures that it is associated with the archived object (i.e., AIP). "
David Giaretta >> (All): Yes - has to be
BruceAmbacher >> (All): Mark, "Mapping of Audit & Certification Criteria for CRL Meeting (15-16 January 2007)
Don Sawyer >> (All): if we just do core, yes/no, then it would still be good to have a place for comments
Mark Conrad >> (All): Thanks, Bruce. What were you saying "I don't think so," to?
BarbaraSierman >> (All): I'm quite sure that the Y/N will be differ per category, like library/archive etc.
BruceAmbacher >> (All): I don't think the crosswalk sets requirements/mandatory
Mark Conrad >> (All): Barbara, That is good. It will get us closer to what is mandatory across all types of repositories.
BruceAmbacher >> (All): Barbara - agreed.  And that is a core issue all the way back to OAIS, through TRAC and RAC
BarbaraSierman >> (All): Yes! And in all cross sectoral discussions!
David Giaretta >> (All): If we end up ticking everything then we will need to have another way of cutting things
Don Sawyer >> (All): David, I think the comments will be instructive!
David Giaretta >> (All): ...perhaps we could rank things "1" for most important to say "5" for least - plus comments
David Giaretta >> (All): Don - yes, the comments are essential
Mark Conrad >> (All): David, Please don't introduce a new level of complexity.
Don Sawyer >> (All): maybe only 3 levels - yes, maybe, no?
JohnGarrett >> (All): Actually, I think we will end up ticking almost everything.   I think some level of provenance is required, but maybe not back to origination.
BruceAmbacher >> (All): I think the differences will lie less in what is mandatory than in when, what becomes mandatory about that item
JohnGarrett >> (All): I agree with Bruce
RobertDowns >> (All): In such cases, the measure and evidence wil be the difference
Mark Conrad >> (All): Can we just start with a simple matrix and see where it leads us rather than speculating about what we might find?
BruceAmbacher >> (All): I think we all want as much information about the provenance of our data as we can get.
David Giaretta >> (All): Mark - OK let's stay simple for the first go - and see what the comments reveal
David Giaretta >> (All): Folks I must go - have to give a talk
candida fenton >> (All): I think simple is a good level to start at
David Giaretta >> (All): Bye
Don Sawyer >> (All): David - do you have a view on how to implement this matrix?  I suspect others, perhaps not participating regularly, may want to fill it out.
BruceAmbacher >> (All): Will David/Simon set up the matrix and let us know when it is posted?
cclrc >> (All): Bruce - Yes, OK.  I might do an example first so that people can see if they like it.
cclrc >> (All): Example = just a couple of requirements
BruceAmbacher >> (All): ok.  I also must go
Don Sawyer >> (All): O'k sounds good
Mark Conrad >> (All): That would be great Simon.
BarbaraSierman >> (All): Our basis is the trac item plus the explanation? So the Y/N is also about whether the text is clear enough?
Mark Conrad >> (All): No! the basis is the requirement.
Don Sawyer >> (All): Barbara - that is what I would expect
Don Sawyer >> (All): Mark - but that is what comments are for - when there are issues
BarbaraSierman >> (All): Well. lets have a try and discuss it later when we see the results and the comments
Don Sawyer >> (All): Agreed - Must go.  Bye
Mark Conrad >> (All): OK. I just think the simpler you make the initial exercise the easier it will be to make progress.
RobertDowns >> (All): Ok, bye

-- SimonLambert - 29 Oct 2007

Topic revision: r1 - 2007-10-29 - SimonLambert
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback