H· Hawiyati Documentation
Restricted — Operations

Operations Guide

Hawiyati exposes structured metrics via Prometheus and OpenTelemetry through the hawiyati-audit module. Each service also exposes a Spring Boot Actuator health endpoint at GET /actuator/health.

Restricted — Operations

Hawiyati exposes structured metrics via Prometheus and OpenTelemetry through the hawiyati-audit module. Each service also exposes a Spring Boot Actuator health endpoint at GET /actuator/health.

Hawiyati IAM Platform — Component Architecture HAWIYATI IAM PLATFORM — COMPONENT ARCHITECTURE Spring Boot 3.5 · Java 25 · gRPC EXTERNAL ACTORS ENTRY LAYER & FEDERATION HTTPS · SAML 2.0 · OIDC · WebAuthn PLATFORM SERVICES HTTPS internal · gRPC DATA & INFRASTRUCTURE JDBC · Redis pub/sub · gRPC User Browser Browser · Mobile App Administrator Admin Console SAML Service Provider SAML 2.0 OIDC Relying Party OIDC · OAuth 2.0 SCIM Client HR System · Directory hawiyati-gateway Spring Cloud Gateway · :8080 Routing · TLS termination hawiyati-http-server IdP Login · MFA · FIDO2 · :8085 SAML ACS · OIDC callback · Account BFF hawiyati-federation SAML 2.0 · OIDC · :8087 Assertion & token issuance · Policy hawiyati-admin Admin Console · :8095 · WebFlux hawiyati-account Account Portal · :8096 · BFF/OIDC hawiyati-scim SCIM 2.0 API · :8098 hawiyati-session · :9004 hawiyati-cache · :9003 hawiyati-infra · :9002 PostgreSQL hawiyati · hawiyati_identity Redis / Valkey Sessions · Cache · Rate limits Audit & Logs CEF Syslog · OTel · SIEM PKI & Keystore SAML · OIDC · mkcert TLS External Actor External System Entry / Federation Platform Service gRPC Service Data Store

The sections below cover operations, monitoring, backup, disaster recovery, and incident response. For a full module map with ports, gRPC boundaries, and deployment topology, see the Implementation Guide.

1. Monitoring & Alerting

Hawiyati exposes structured metrics via Prometheus and OpenTelemetry through the hawiyati-audit module. Each service also exposes a Spring Boot Actuator health endpoint at GET /actuator/health.

1.1 Key Metrics

The following metrics should be tracked continuously in production:

MetricDescriptionSource
Login success rateRatio of successful to total login attemptshawiyati-audit Prometheus
Login failure rateFailed logins per minute, per IP, per userhawiyati-audit Prometheus
Active session countCurrent sessions in Redishawiyati-session gRPC + Redis INFO
MFA success/failurePer-factor MFA completion rateshawiyati-audit Prometheus
SAML assertion issuance rateAssertions signed per minute per SPhawiyati-audit Prometheus
OIDC token issuance rateTokens issued per minute per clienthawiyati-audit Prometheus
KYC session completion rateKYC sessions reaching APPROVED statehawiyati-kyc Prometheus
Risk engine block rateRequests blocked by risk engine per hourhawiyati-audit Prometheus
Login latency p9595th percentile end-to-end login durationOTel traces
Redis hit/miss ratioCache effectiveness (sessions, SP config)Redis INFO stats
PostgreSQL connection pool usageActive vs max pool connections per serviceHikariCP metrics
Disk usageDB data dir, log dir, audit export dirNode exporter

1.2 Alert Thresholds

MetricWarningCriticalRouting
Auth failure rate >10% >25% Ops + Security (anomaly)
Login latency p95 >500ms >2s Ops pager
Redis connection failures — Any Ops P1 pager immediately
PostgreSQL connection pool >80% >95% Ops pager
Session creation failures >1% >5% Ops pager
Disk usage >70% >85% Ops (capacity planning)
TLS certificate expiry <60 days <14 days Ops + Security
Risk engine block spike 3× baseline 10× baseline Security team

1.3 Recommended Monitoring Stack

Hawiyati metrics are compatible with standard Prometheus scraping. Recommended stack:

  • Prometheus — scrape /actuator/prometheus on each service (port varies per service)
  • Grafana — dashboards for auth rates, session counts, latency histograms
  • OpenTelemetry Collector — aggregate OTel traces from hawiyati-audit; forward to Jaeger or Grafana Tempo
  • Alertmanager — route alerts: P1/P2 → ops pager (PagerDuty/OpsGenie), anomaly spikes → security team Slack

1.4 Health Endpoints

ServiceHealth EndpointExpected Response
hawiyati-gatewayGET http://localhost:8080/actuator/health{"status":"UP"}
hawiyati-http-serverGET https://idp.hawiyati.org:8085/actuator/health{"status":"UP"}
hawiyati-adminGET https://localhost:8095/actuator/health{"status":"UP"}
hawiyati-scimGET https://localhost:8098/actuator/health{"status":"UP"}
Tip Configure your load balancer to poll each health endpoint every 10–15 seconds and remove unhealthy instances from the pool automatically. Pair with a readiness probe that also checks Redis and PostgreSQL connectivity.

2. Log Management

Hawiyati produces two distinct log streams: structured audit logs (compliance-grade, CEF/JSON format) and standard application logs (Spring Boot Logback).

2.1 Audit Log Format

Every audit event always contains these fields:

FieldTypeDescription
@timestampISO-8601Event time in UTC
servicestringOriginating service name
categoryenumAUTH, ADMIN, SESSION, KYC, SCIM, POLICY
actionstringSpecific action (e.g. LOGIN_SUCCESS, POLICY_UPDATED)
outcomeenumSUCCESS, FAILURE, DENIED
severityenumINFO, WARN, ERROR, CRITICAL
actor.idUUIDID of the acting user or admin operator
actor.typeenumUSER, ADMIN, SYSTEM, SCIM_CLIENT
messagestringHuman-readable event summary

2.2 Retention & Rotation

Log TypeFormatRetentionNotes
Audit logsJSON / CEF syslogMinimum 12 monthsCompliance requirement; append-only
Application logsLogback text / JSON90 daysDaily rotation; compress after 7 days
Access logs (gateway)Common Log Format30 daysHigh volume — aggregate before archiving

2.3 SIEM Integration

Forward audit logs to your SIEM via syslog (TCP/TLS recommended). Hawiyati-audit supports:

  • Syslog over TCP (RFC 5424)
  • File-based JSON for Filebeat/Fluentd pickup
  • OpenTelemetry log exporter (OTLP/gRPC)
Sensitive field masking Passwords, OTP codes, session tokens, claim tokens, and completion tokens are NEVER written to any log. If a log line shows a token or credential value, treat it as a configuration or injection bug and escalate to security immediately.

2.4 Admin Console Export

Search and export audit logs from the admin console (Admin → Audit Logs). Export formats: CSV and JSON. Maximum export size: 50,000 rows per export. For larger exports, use direct DB queries against hawiyati_config.audit_events or query your SIEM.

2.5 Log Levels

Adjust Spring Boot log levels at runtime without restart via POST /actuator/loggers/{name} (admin console access only). In production, default to INFO. Enable DEBUG for specific components only during active troubleshooting — DEBUG logging at scale generates significant volume.

# Example: enable debug for the risk engine component
curl -X POST https://idp.hawiyati.org:8085/actuator/loggers/hawiyati.risk \
  -H "Content-Type: application/json" \
  -d '{"configuredLevel":"DEBUG"}'
Backup assurance and restore testing lifecycle
A backup is reliable only after integrity checks and an isolated restore test succeed.

3. Backup & Restore

Hawiyati state lives in two stores: PostgreSQL (durable — users, sessions schema, KYC, SP config, audit) and Redis (transient — active sessions, OTP codes, rate limits, SP config cache). Only PostgreSQL requires persistent backup; Redis data is rebuildable.

3.1 PostgreSQL Backup

  • Daily full backup: pg_dump -Fc hawiyati > hawiyati-$(date +%Y%m%d).dump
  • Continuous WAL archiving: enables point-in-time recovery (PITR) to any minute. Configure archive_mode=on and archive_command in postgresql.conf
  • Backup storage: separate host or encrypted object storage (S3-compatible). Never on the same host as the DB.
  • Encryption: encrypt backups at rest (GPG or storage-level encryption)
  • Verification: weekly automated restore test to a staging instance

3.2 Redis Backup

  • Enable RDB snapshots every 6 hours (save 21600 1 in redis.conf)
  • Enable AOF with appendfsync everysec for durability
  • Copy RDB file to backup storage after each snapshot
  • Note: Redis loss means active sessions are invalidated — users must re-authenticate. This is acceptable in most recovery scenarios.
RTO / RPO Targets RTO (Recovery Time Objective): <4 hours from declared incident to service restored.
RPO (Recovery Point Objective): <1 hour of data loss with WAL archiving enabled. Without WAL archiving (daily pg_dump only), RPO is up to 24 hours — this is NOT acceptable for production.

3.3 Restore Procedure

  1. Stop all Hawiyati services — stop in reverse start order: scim → admin → http-server → gateway → gRPC services
  2. Restore PostgreSQL — pg_restore -d hawiyati hawiyati-YYYYMMDD.dump
  3. Apply WAL to target point-in-time (if PITR needed) — configure recovery.conf and start PostgreSQL in recovery mode
  4. Verify Flyway migration state — SELECT version, description FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 5; — confirm V62 (or later) is present
  5. Restore Redis snapshot — copy RDB file to Redis data dir, restart Redis
  6. Start services in order — hawiyati-infra/cache/session (gRPC) first, then hawiyati-http-server, then hawiyati-admin and hawiyati-scim
  7. Smoke test — perform a test login, verify an audit event was written, check admin console loads
Disaster declaration, recovery, validation, and service resumption flow
Traffic resumes only after durable services, security controls, data integrity, and critical journeys are validated.

4. Disaster Recovery

Hawiyati supports an active-passive DR model. Active-active is not recommended due to SAML state coupling (InResponseTo correlation, session continuity).

4.1 DR Architecture

ComponentDR MethodNotes
PostgreSQLStreaming replication to standbyStandby is read-only replica; promote on failover
RedisRedis Sentinel or ClusterFor session HA; acceptable to lose in-flight sessions
Application servicesStateless — restart on standby hostsJARs and config must be available on standby
TLS certificatesCopy to standby, same hostnameCertificate must match production DNS name
Active-Active Not Supported Do not run two active Hawiyati instances sharing the same user database without a sharding or coordination layer. SAML InResponseTo validation and session state are node-local — cross-node requests will fail validation.

4.2 Failover Procedure

  1. Alert fires on primary failure — confirm primary is truly down (check /actuator/health, attempt DB connection). Do NOT proceed if split-brain is possible.
  2. Verify primary is unreachable — ping, TCP check, attempt pg_isready from standby host
  3. Promote PostgreSQL standby — pg_promote() or pg_ctl promote -D /var/lib/postgresql/data
  4. Update load balancer / DNS — point application VIP or DNS record to standby host (TTL should be pre-set low: 60s)
  5. Flush Redis on standby — FLUSHALL to clear any stale state. Active sessions are lost; users must re-authenticate.
  6. Start Hawiyati services on standby — same start order as primary: gRPC services first, then HTTP services
  7. Smoke test — test login via a canary SP, verify audit log write, check admin console

4.3 Failback Procedure

After the primary host recovers:

  1. Re-sync PostgreSQL: stream WAL from current primary (standby) back to recovered host
  2. Verify data consistency on recovered host
  3. Switch DNS/load balancer back to primary host
  4. Demote standby to replica
  5. Monitor for 30 minutes before declaring primary restored

4.4 DR Test Schedule

Conduct a full DR failover test quarterly. The test should:

  • Simulate primary host failure (stop services and DB)
  • Follow the failover procedure above end-to-end
  • Measure actual RTO against target (<4h)
  • Document findings and update the DR runbook

5. Upgrade & Rollback

Hawiyati uses Flyway for schema migrations. Flyway runs automatically on service startup — new migration scripts (Vxx__description.sql) are applied in version order. Zero-downtime upgrades are not currently supported; schedule a maintenance window.

Always back up the database before upgrading Flyway migrations are forward-only. A rollback requires restoring the database from a pre-upgrade backup — data written after the upgrade is lost. Never skip this step.

5.1 Upgrade Procedure

  1. Read the release notes — identify any breaking changes, new required configuration, or manual pre-migration steps
  2. Test in staging — run new JARs against a copy of production data. Verify all Flyway migrations succeed and smoke tests pass.
  3. Backup production database — pg_dump -Fc hawiyati > pre-upgrade-$(date +%Y%m%d-%H%M).dump
  4. Schedule maintenance window — notify SP owners and affected users; set maintenance page at load balancer if available
  5. Stop services — stop in reverse order: SCIM → Admin → HTTP Server → Gateway → gRPC services
  6. Deploy new JARs — replace JAR files and update application.yml if new configuration keys are required
  7. Start services — gRPC services first, then HTTP services. Flyway runs migrations automatically on first startup. Watch logs for migration success.
  8. Smoke test — test login, MFA, SP federation, admin console, audit log write
  9. Monitor for 1 hour — watch error rate, login latency, and session count metrics

5.2 Rollback Procedure

Data loss warning Rolling back requires restoring the pre-upgrade database backup. Any data created after the upgrade (new users, new sessions, audit events) will be lost. Inform stakeholders before proceeding.
  1. Stop all services
  2. Restore previous JARs — keep previous version in a versioned deploy directory
  3. Restore pre-upgrade database backup — pg_restore -d hawiyati pre-upgrade-YYYYMMDD-HHMM.dump
  4. Start services with previous JARs — Flyway will confirm current schema version matches and skip migrations
  5. Smoke test and monitor

5.3 Flyway Migration Naming

Migration files follow the convention: V{version}__{description}.sql (double underscore). Example: V54__add_claim_token_table.sql. The current schema head is V62. Never manually edit or delete applied migration records in flyway_schema_history.

6. Routine Maintenance Schedule

Preventive maintenance keeps the platform healthy and avoids reactive incidents. Assign ownership for each task in your operations runbook.

FrequencyTaskOwner
Weekly Check disk usage on DB host, log host, and backup storage (alert at >70%) Ops
Review failed login anomalies in audit logs — look for velocity or pattern shiftsSecurity
Verify backup completion logs — confirm daily pg_dump and RDB snapshot ran successfullyOps
Check TLS certificate expiry for all hostnames — alert if <60 days remainOps
Monthly Rotate Redis AUTH password — update in application.yml and restart servicesOps
Review admin operator accounts — disable or delete accounts for staff who have leftSecurity
Review and rotate SCIM bearer tokens — reissue tokens for connected provisioning systemsOps + SP owners
Review risk engine configuration vs recent incident data — adjust thresholds if neededSecurity
Quarterly DR failover test — follow full failover procedure, measure RTOOps
Penetration test scope review — confirm coverage of new SPs or featuresSecurity
Certificate rotation (if due) — renew all TLS certs before expiryOps
Dependency vulnerability scan — check Hawiyati's dependency tree against CVE databaseSecurity
Review and purge expired KYC sessions — clean up stuck IN_PROGRESS sessions older than policy thresholdOps + Compliance
Annually Full security audit — review access controls, encryption config, audit log integritySecurity
Key rotation review — rotate SAML signing key and OIDC JWK set (rolling update)Security + Ops
Compliance assessment — review NIST SP 800-63 alignment, assurance level configurationCompliance
Disaster recovery plan review — update runbook, retest failback procedureOps

7. Capacity Planning

Plan capacity at least 6 months ahead. The primary scaling constraints are PostgreSQL (write throughput and disk) and Redis (memory for active sessions).

7.1 Sizing Formulas

ResourceEstimate FormulaExample (10,000 concurrent users)
Redis memory (sessions) ~2 KB/session × peak concurrent sessions × 2 (buffer) 2 KB × 10,000 × 2 = 40 MB
Redis memory (OTP + rate limits) ~0.5 KB/user in active MFA flow × concurrent MFA attempts Typically <100 MB additional
Audit log disk growth ~500 bytes/event × events/day 1M events/day = ~500 MB/day = ~15 GB/month
PostgreSQL data growth Audit log dominant; user/session tables much smaller 1M users ≈ 2 GB for identity tables
Application CPU ~0.1 CPU core per 100 concurrent logins (p95 <500ms) 1,000 concurrent logins → ~10 cores across http-server instances

7.2 Scaling Approach

Horizontal scaling (application tier): hawiyati-http-server and hawiyati-admin are stateless — sessions are Redis-backed. Add instances behind a load balancer (sticky sessions not required). gRPC services (infra, cache, session) are also stateless and can be replicated.

Database scaling: PostgreSQL is the primary bottleneck. Add read replicas for reporting and audit log queries. Use connection pooling (PgBouncer) to handle high connection counts. Partition the audit_events table by month for large deployments.

7.3 Scaling Signals

Act on these signals before they become incidents CPU >70% sustained on http-server → add an instance. Memory >80% on Redis → increase maxmemory or add cluster node. Connection pool exhaustion alerts firing >5× per day → increase pool size or add a DB replica.

7.4 Redis Eviction Monitoring

# Check Redis memory and eviction stats
redis-cli INFO memory | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation'
redis-cli INFO stats | grep -E 'evicted_keys|keyspace_hits|keyspace_misses'

Set Redis maxmemory-policy allkeys-lru in production. Monitor evicted_keys — if evictions are occurring on session keys, sessions are being lost before expiry, which causes unexpected logouts. Increase maxmemory or purge stale keys.

8. Incident Response

A well-rehearsed incident response process minimises mean time to recovery (MTTR). Define roles, communication channels, and escalation paths before your first production incident.

8.1 Severity Matrix

P1 — Critical
All users unable to authenticate. Auth service is down or returning 5xx. Response: 15 min.
P2 — Major
>10% auth failure rate, or a major SP is impacted. Degraded but not fully down. Response: 1 hour.
P3 — Minor
Single MFA factor unavailable, KYC stalled, admin console slow. Response: 4 hours.
P4 — Informational
Minor issue, user question, cosmetic defect. No production impact. Response: next business day.
SeverityResponse SLACommunicationEscalation
P115 minutesImmediate pager + status page updateOps lead + Security lead
P21 hourSlack ops channel + affected SP ownersOps lead
P34 hoursTicket + affected team notificationOps on-call
P4Next business dayTicketOps team

8.2 Common Runbooks

These runbooks should be documented in your ops wiki with step-by-step procedures:

RunbookWhen to UseKey Steps
Auth service restarthawiyati-http-server unresponsiveHealth check → log triage → graceful restart → smoke test
Redis failoverRedis connection failures alert firesCheck Redis health → fail to replica → flush stale state → restart services
PostgreSQL failoverDB unreachable, primary host failureSee DR failover procedure (Section 4.2)
Emergency certificate renewalCert expired or <14 days remainingIssue new cert → update keystore → rolling restart
Account lockout stormMass lockout from velocity attack or misconfigurationIdentify source → blocklist IP → bulk unlock affected accounts via admin API
Rate limit incidentLegitimate users hit rate limitsIdentify trigger → temporary threshold increase → investigate root cause

8.3 Post-Incident Review

For every P1 and P2 incident, conduct a post-incident review within 48 hours. The review template:

SectionContent
Incident timelineExact timestamps: detection, declaration, diagnosis, mitigation, resolution
Root causeTechnical root cause (5-whys analysis)
ImpactNumber of affected users, duration, SPs impacted, estimated auth failures
DetectionHow was the incident detected? Alert, user report, or internal discovery?
Response assessmentWhat went well? What was slow or missing?
Action itemsSpecific follow-up tasks with owners and due dates to prevent recurrence
Blameless culture Post-incident reviews should be blameless. The goal is to improve systems and processes, not to assign individual fault. People make good decisions with the information they had at the time; the review improves the information and processes for next time.
HawiyatiDocumentation · v2026 · Spring Boot 3.5