TRIXi TRIRIGA Support Portal

 

1        Scheduled Maintenance Windows

The dates for planned maintenance windows for production environments in 2023 (also shown in the calendar at the top of this page) are: January 7-8, January 21-22, February 4-5, February 18-19, March 4-5, March 18-19, April 1-2, April 15-16, April 29-30, May 13-14, May 27-28, June 10-11, June 24-25, July 8-9, July 22-23, August 5-6, August 19-20, September 2-3, September 16-17, September 30-October 1, October 14-15, Oct 28-29, November 11-12, November 25-26, December 9-10.

Production Weekend Window is Sunday 8:00 AM to 12:00 PM.

1.1       Change / Freeze period

The Change / Freeze period for year-end 2023 is December 16, 2023 through January 2, 2024.

During the Change / Freeze window, environments will be available to users and all standard automated processes (such as backups, monitoring, replication etc.) continue as normal. However, coordinated changes to environments via the TRIXi support team will not be available during this time. Some examples of coordinated changes include, but are not limited to:

  • Database configuration (configdb / updatedb)
  • Installation of TRIRIGA fixpacks or upgrades
  • Database backflows from one environment to another
  • Running SQL statements or scripts
  • EAR file rebuilds and deploys

In addition, environment maintenance and/or planned outages are not scheduled during this period.

2        Backup and Restore

2.1       Database Backups

  • Database backups for both Production and Non-Production environments are performed 7 days a week. Full backups are performed every Saturday. Incremental backups are performed Sunday through Friday.

2.2       Filesystem Backups

  • Filesystem backups for both Production and Non-Production environments are performed 7 days a week.
  • Production backups are kept for fourteen (14) days.
  • All backups are AES-256 encrypted and can only be restored to servers with the appropriate encryption key.

2.3       Restores

  • TRIXi can perform a restoration of client’s content upon client request via case Restores can be performed within 1-3 business days, depending on urgency.
  • Databases can be restored to any point in time (limitation for production is 14 days prior).

2.4       Logs

  • All database transaction logs, and application server (JVM) logs are saved to each server’s local filesystem.
  • Logs are stored in standard default locations. There is no set retention period for logs. Logs are rolled based on size. Size varies depending on logging level and amount of activity. Customers can obtain read-only access to Application Server logs via SFTP. Database transaction logs are not accessible to customers.

3        Customer Data at Contract Termination

  • Before expiration or termination of the Cloud Service, client can use any of the provided reporting or export features of the Cloud Service to extract data.
  • Exports can be requested via case
  • Upon receiving a support request from client within 30 days of the Cloud Service expiration or termination date, TRIXi will return to client an electronic copy of client’s content in the native format (DB2 or Oracle export; zip file of Application Server customisations and attached documents).

4        Incident Management

The TRIXi team has monitoring in place for all sites and infrastructure under our control.  These are designed to allow the team to pro-actively respond to service impacting or service threatening events or conditions.

At the same time, for production environments, TRIXi provides 24/7 critical outage support. The goal is to ensure our customer’s applications are running when they should, and to provide effective and timely customer communication during availability incidents or Severity 1 cases during off hours.

4.1       Client Communicator

The Client Communicator (CC) is responsible for ensuring that any customer affected by a Severity 1 incident or alert is receiving prompt and frequent communication regarding the status of their incident. This resource does not necessarily have technical skills or access to investigate / act upon systems that are failing. The CC may also be expected to triage requests that do not fall within the definition of Severity 1 and communicate with the customer regarding these issues.

4.2      First Responder

The First Responder (FR) is a technical role that requires access to systems that may be in a failed or failing state, as well as the skills required to understand what can be done to recover affected environment(s). It may not be possible for the FR to correct all problems and he/she should be equipped to escalate issues to specific individuals for resolution if necessary. The FR remains focused on incident resolution at all times and is not expected to communicate directly with customers; they remain in regular contact with the Client Communicator on duty. It is important to note that the First Responder is precisely that, the first responder – he/she is not solely responsible for solving every incident.

The first responder will respond to alerts and off hours Severity 1 cases to:

  1. Determine the impact of the alert or case
  2. Determine the cause of the alert or case
  3. Initiate corrective action if appropriate
  4. Alert the Client Communicator if escalation is determined necessary.

The TRIXi first responder’s priority will be to restore service.  The TRIXi client communicator is notified if there are any challenges to restoring service.  The TRIXi client communicator will lead the recovery activities and escalate to any personnel required to resolve the issue, while also ensuring that continuous communication is maintained with the customer throughout the length of the incident.

4.3      Escalation Manager

Additional support is provided by an Escalation Manager.

5        Change Requests

5.1      Administration Tasks

All Administration tasks are subject to the TRIXi Change Management Process and, unless otherwise agreed upon, will follow the standard change deployment path through the system, (e.g. Development, Test, Production).

5.1.1       Tasks managed by the TRIXi SRE team

For changes that require system level access, these will be managed by the TRIXi team.   For example, the tasks below are all within the scope:

  • Class file updates
  • Applying FixPacks (patches) and Upgrades
  • EAR re-build & redeploy
  • Database backups
  • Database Configuration (running configdb)
  • WebSphere Updates/Upgrades
  • HTTP Updates/Upgrades
  • Database Updates/Upgrades
  • Network Updates/Upgrades
  • Executing SQL scripts or procedures (e.g. language updates, data updates)
  • OS Updates/Upgrades
  • OS Level batch Jobs

5.1.2       Tasks managed by the Customer

For changes that can be made through the front-end UI, these should be managed by the client or the implementation team.   For example, this could include:

  • Application designer changes
  • Data loading
  • Migration manager package deployments
  • User Management (adding users/security groups)
  • Workflow updates
  • Automation Script creation

Please note:  There may be additional tasks that are not included in the lists above.

5.2      Types of Change Requests

The following is an overview of TRIXi internal change types.

5.2.1       BAU/Standard Change

BAU (Business As Usual) changes are those that are relatively low-risk and well understood. BAU Changes are processed frequently and initiated based on client requests. These changes do not have wide-ranging impacts on business-critical configuration items, and they are processed so often that they only require a basic review and approval prior to implementation.  A BAU Change will go through all the standard phases of the TRIXi change process and can be managed throughout the process by the change owner with the exception of the approval phase.

BAU changes have to be scheduled in advance.

A BAU change can be for either a production or non-production environment. Changes for development environments do not require prior testing, however all other client environments require testing of some sort; either testing evidence from the client or a previous successful change in another related client environment.

BAU Changes will follow standard back-out procedures.

Once implemented, a BAU Change is verified by the TRIXi team to confirm 1) the change was applied and 2) that the environment is accessible; after which the change record is completed. Any other verification is the responsibility of the client and is not tracked in the change record.

5.2.2       Normal (Non-BAU) Changes

Normal (non-BAU) Changes require that all of the change process phases to be completed. These changes require a full range of assessments and authorizations to ensure completeness, accuracy, and the least possible disruption across the data centre. In addition, these changes must be scheduled, implemented, verified and reviewed prior to closure. The normal type is used when a change will produce impacts on business-critical applications and other critical components of the Cloud environment.

A normal change can be for either a production or non-production environment.  Normal changes can be initiated and requested by the client or TRIXi.  Client initiated changes for development environments do not require prior testing, however all other client environments require testing. Trixi initiated changes required testing evidence to be recorded as a work log entry on the change record or on a related change record. Normal Changes will follow standard back-out procedures.

Once implemented, a Normal Change is verified by the TRIXi team to confirm 1) the change was applied and 2) that the environment is accessible; after which the change is verified by the change requestor.  Verification evidence is documented on the change records as a work log.  Availability monitoring can be relied on to verify change completion in the case where it is an TRIXi initiated change to infrastructure that may impact client environments.

5.2.3       Emergency / Retrospective Changes

An Emergency change is one that must be done immediately. It is of a very high priority. An example of an emergency change might be the installation of new antivirus software during a period of severe viral infestation across the data centre. Emergency changes are typically not performed often. An emergency Change contains all of the process steps and requirements that are followed for a normal change but the phases can be expedited outside of the normal change cycle.

Retrospective Changes are change records raised after changes have been implemented. Retrospective changes can only be raised to accommodate emergency change in 2 key scenarios:

  1. With prior approval from the TRIXi Management Team by SMS text message or email
  2. In response to incidents where client system availability is impacted and a change is required to resolve the issue in a production system. Increase to system capacity (disk, CPU or memory) are included in this scenario.

Both the Emergency and Retrospective Changes will follow the same process in the change management system and will have a change type of Emergency.  Retrospective changes differ in the fact that they are created after the change is performed.  The criteria for an Emergency / Retrospective change is the same as a Normal change.

5.3      Change Request Examples

5.3.1     Service Requests

  • Database backup request
  • Environment refresh (backflow)
  • Account unlocks – database or other
  • Application or database server restarts (assumes there is no other change activity attached)
  • General inquiries
  • Running of SQL scripts
  • Application or database investigation or troubleshooting requests

5.3.2     BAU Changes

Criteria:

  • Client initiated
  • Operational type changes – frequently requested
  • Does not meet any of the Normal Change criteria

Examples:

  • Run a SQL data update/insert/delete – backup is always required and should be kept for a period of one month
  • New access requests (RDC, VPN, etc.) or changes to access rights
  • Update a configuration file (e.g. session timeout) – not a code or database change
  • Adding a new JMS queue
  • Importing a certificate to trust store
  • SFTP account request
  • Development environment local firewall change
  • Environment Refresh/Back flow of database or code (e.g. PROD > TEST)

5.3.3     Normal (Non-BAU) Changes

 Key Requirements:

  • Approval process must be followed and includes a minimum of 2 levels of approval – Change Manager and TRIXi Management
  • Evidence of testing required
  • Lead time is expected for scheduling
  • Changes require verification prior to closing

Criteria:

  • Non-standard operational change
  • Change to Application code
  • Cost implications – additional servers or capacity
  • Security implication
  • Shared infrastructure update

Examples:

  • Adding disk capacity to a server
  • Adding a VPN or NAT/Firewall configuration
  • Deploy a custom class file or other code
  • Application upgrades or patches
  • Installation of add-ons or 3rd party software – confirm license entitlements
  • Scheduling system level jobs/crons
  • Application or database upgrade (major release such as TRIRIGA platform upgrade)
  • Operating system or middleware patching
  • Security vulnerability patching

5.3.4     Emergency / Retrospective Changes

An emergency change is one that must be done immediately. It is of a very high priority. An example of an emergency change might be the installation of new antivirus software during a period of severe viral infestation across the data center. Emergency changes are ones that are typically not performed often. An emergency change contains all of the process steps and requirements that are followed for a Normal change but the phases can be expedited outside of the normal change cycle.

Retrospective changes are change records raised after changes have been implemented. Retrospective changes can only be raised to accommodate emergency change in 2 key scenarios:

  • With prior approval from the TRIXi SRE Management Team by SMS text message or email
  • In response is incidents where client system availability is impacted and change is required to resolve the issue in a production system.

Increase to system capacity (Disk, CPU or RAM) are included in this scenario

Both the Emergency and Retrospective Changes will follow the same process in the change management system and will have a change type of Emergency. Retrospective changes differ in the fact that they are created after the change is performed. The criteria for an Emergency / Retrospective change is the same as a Normal change.

5.4      Change Request Initiation & Approval

5.4.1     Operational Change Requests Initiated by the Client

All change requests begin with the creation of a case in the support portal.  The case should include sufficient information to assess the risk of the change, the scope of change and any other details required.   The change schedule will be communicated and agreed to within the case as a comment or post entry.    It is assumed that once the change details and schedule are agreed to that the client will follow any internal change approval process that they may require on their side.  Once the case has been submitted TRIXi can create a change request (CR), which will be tied to the case.  The CR tasks are then assigned to the appropriate TRIXi task implementer (e.g.  DBA, System Administrator,  Network Administrator).

5.4.2       Change Approval Process

By default, the change request (CR) is created with a status of Waiting Approval.  All CR’s will be processed through the change management process which will include various levels of review and approval.  If implementation approval is required from the change requester the change requester will provide confirmation on the case.

5.4.3     Operational Change Requests initiated by TRIXi

All Changes initiated by the TRIXi SRE team must also be approved by the Client Delivery Manager or the TRIXi Change Manager and must also be entered as a Change Request.  This can be created by any member of the Cloud Delivery Services team and will include hardware, operating system, or middleware changes (for example, a patch to the OS or a patch to the database software).  A Change Notification will be sent to the Client outlining the date and time of the change, whether it will affect the availability of the application, the estimated duration of the change window, and the reason for the change.  TRIXi initiated changes do not require client approval, however all production changes will be scheduled in the standard maintenance windows and communicated with as much lead time as possible.

5.5      Change Request Scheduling

5.5.1       Production changes

Will be scheduled with the client at an agreed upon date and time, and in cases where application availability will be impacted, they will be scheduled during the defined outage window only.

5.5.2       Non-production changes

Will be done during normal business hours (Monday to Friday 8:00 AM – 8:00 PM AEST) although the process for requesting, approving, and executing these changes are the same otherwise.

5.5.3       Change freeze

The TRIXi SRE Team will enforce 1 (one) change freeze period in a year. Holiday season between Christmas and New Year (late December to early January). Normally the change freeze will start 2 weeks before holiday and lifted after the first week of New Year. The change freeze is imposed due to most TRIXi support team members will be taking annual leave. To ensure there is minimal risk of system instability, there will be no changes performed during the period.

5.6      Migrating Changes

The best practices that are used when implementing changes are through environment promotions. For example, when a change request is submitted, the initial implementation first occurs on a Development or Test environment. The change requester is then tasked with testing and confirming that the change meets the business needs of the requester.  If this testing process fails, further action(s) are initiated which could range from backing out of the change or other additional changes.