Thoughts on Government Records Retention in SharePoint, Part 1

The other morning I was asked for my thoughts on applying records retention concepts to SharePoint. First, I believe that given recent legislative changes and interpretations by both the State Attorney General and Secretary of State's office, every application has a role in records retention. With the born digital concept firmly established, once the information is created in a system, this new "record" must be retained according to record retention guidelines. SharePoint is no exception and provides a good exploration of the intersection of business needs and records management needs.

For those not stuck looking at records retention issues on a daily basis, lets setup our example. From the Local Government Common Records Retention Schedule (CORE) v2.1:

Item 1.3.2: CITIZEN'S COMPLAINTS/REQUESTS Communications from citizens making a complaint or request, as well as the associated agency response. Retain until: Matter closed plus 3 years. DAN: GS50-01-09 Rev. 0.

This record series covers correspondence with citizens, specifically complaints or requests. Many public emails to agency/department accounts would be covered. For example, if I were to complain about a broken link to webmaster@agency.org, that would need to be retained in digital form, complete with metadata, until 3 years after I fixed the broken link.

So what must a system do in support of records management. It must be able to do three things. First is classification. This means identifying the record series that the records belong to and the relevant date.

Sometimes this can be implicit. We simply make notice that all records in system X are of this record series. The actual classification is not tracked with the record, however, we have documented that records matching a certain criteria are in a specific series. This is possible for those systems that have strong metadata support or where the actual content of the record clearly defines the series. For example, a Contracts management system would implicitly match the contract documents with Item 1.4.3 Contracts and Agreements (DAN GS50-01-11 r1).  The termination or expiration date is the relevant date and the record is to be retained for 6 years. This is made easier when the metadata or record construct is the same throughout the system.

Classification can also be explicit. In this case metadata used specifically for the record series identifier and the relevant date are created and managed. This is often needed when the metadata or the record constructs are not consistent throughout the system, such as SharePoint. In the State of Washington the record series is identified by a Disposition Authority Number (DAN). Both the DAN and the relevant date should be noted on each record.

Lets take a moment and explain the relevant date. This is the date when the retention clock starts. Primary records must be maintained for y years from x date. In a number of cases, they do not have to be retained once they are obsolete or superseded ("destroy when obsolete or superseded"). In this case y=0 and x=the date record is obsolete or superseded. This might mean that when a new version of the form is published, the old form get a relative date of the publish date of the new form. In other cases, like the example above, the relevant date is dependent upon user action (y=3, x=date broken link fixed). Very rarely is the relevant date the same as the creation date. To further complicate things, in some cases the relevant date can change. For example, a court case could be appealed or a new report come in and the retention clock would get reset. Finally, the relevant date is often not known at the time the record is first created.

The second records management component that systems must support is retention. Retention is simply the act of keeping the record intact from the moment it is no longer needed for business purposes to the point in time where the retention period has expired. The retention period is measured from the relevant date. The retention period and business need period may occur in sequence or run concurrently. The business need may last longer then the retention need. In any case, the document must be retained intact for which ever is longest.  Additionally, holds for non business needs should be allowed, such as for legal actions or public record requests and may super cede the retention schedule.

The system should make it very difficult to delete, destroy or change the record during the retention period. The record must be kept intact, complete with metadata. However, it is important to note that because the relevant date is often not set until later in the business process, it may be possible or even desirable to have the system allow deletion prior to establishing the relevant date. Examples include draft documents which are never finalized or data entry errors.  It is also likely that during the business need period, the record could be altered or further added to. However, once the record series and relevant date are set, care must be taken to be able to discern how the record has changed since that point.

Third, the system must support disposition of the record. Disposition is the formal act of dealing with the record post retention. There are essentially two options.

One is to transfer the record to the State archives. Some records, as established in the retention schedules, have archival value and must be assessed for transfer to the State Archives. With digital information, this is much more difficult then with paper originating records. To date, only scanned documents and email are regularly handled by the digital archives. Because of these limitations, SharePoint must be VERY carefully considered for any use and storage of primary records that have an archival value.



Lastly, a very important component of disposition is documentation. A log must be kept identifying the record, record series, relevant date, disposition, disposition date, who approved disposition and who carried out the disposition. A good system will support easily finding records ready for disposition and logging of the disposition act. Remember, it may be necessary to document two disposition dates, one when it was removed from the system and one when it was removed from backups.

So, I have just spent a very long post laying the groundwork for records management in a generic application or system. What you really wanted to know is how do we get SharePoint in on the game. I believe that by tracking DAN, relevant date, using custom Content Types, workflows, SharePoint's built in records management functionality and reporting services, it will be possible to bake records management, with explicit classification, directly into SharePoint. In the next weeks (until my Office365 demo runs out) I will be experimenting and documenting my experiences implementing records management in SharePoint 2010.  Stay tuned.