H· Hawiyati Documentation

Troubleshooting

This guide covers the most common error conditions in Hawiyati — SAML, OIDC, MFA, accounts, SCIM, and infrastructure — with symptoms, root causes, and fixes. Tip: press Ctrl+F and paste your error code to jump straight to it.

This guide covers the most common error conditions in Hawiyati — SAML, OIDC, MFA, accounts, SCIM, and infrastructure — with symptoms, root causes, and fixes. Tip: press Ctrl+F and paste your error code to jump straight to it.

Hawiyati IAM Platform — Component Architecture HAWIYATI IAM PLATFORM — COMPONENT ARCHITECTURE Spring Boot 3.5 · Java 25 · gRPC EXTERNAL ACTORS ENTRY LAYER & FEDERATION HTTPS · SAML 2.0 · OIDC · WebAuthn PLATFORM SERVICES HTTPS internal · gRPC DATA & INFRASTRUCTURE JDBC · Redis pub/sub · gRPC User Browser Browser · Mobile App Administrator Admin Console SAML Service Provider SAML 2.0 OIDC Relying Party OIDC · OAuth 2.0 SCIM Client HR System · Directory hawiyati-gateway Spring Cloud Gateway · :8080 Routing · TLS termination hawiyati-http-server IdP Login · MFA · FIDO2 · :8085 SAML ACS · OIDC callback · Account BFF hawiyati-federation SAML 2.0 · OIDC · :8087 Assertion & token issuance · Policy hawiyati-admin Admin Console · :8095 · WebFlux hawiyati-account Account Portal · :8096 · BFF/OIDC hawiyati-scim SCIM 2.0 API · :8098 hawiyati-session · :9004 hawiyati-cache · :9003 hawiyati-infra · :9002 PostgreSQL hawiyati · hawiyati_identity Redis / Valkey Sessions · Cache · Rate limits Audit & Logs CEF Syslog · OTel · SIEM PKI & Keystore SAML · OIDC · mkcert TLS External Actor External System Entry / Federation Platform Service gRPC Service Data Store

The sections below cover common error conditions, troubleshooting procedures, and escalation paths. For a full module map with ports, gRPC boundaries, and deployment topology, see the Implementation Guide.

Diagnostic Approach

When something goes wrong, follow this sequence before escalating. Most issues are resolved at step 1 or 2.

1
Check Audit Logs first
Admin → Audit Log. Filter by the affected user and time range. The failureReason field usually identifies the exact problem. Authentication failures, policy denials, and token errors all produce structured events.
2
Check Application Logs
Locate the Spring Boot log for the relevant service (hawiyati-http-server for login, hawiyati-admin for admin console, hawiyati-scim for SCIM). Look for ERROR or WARN lines at the same timestamp.
3
Check Redis Health
Redis stores sessions, OTP codes, rate-limit counters, and SP config cache. Run redis-cli ping — expect PONG. Check redis-cli info memory for near-limit conditions. Session failures and rate-limit anomalies often trace to Redis connectivity.
4
Check PostgreSQL Connectivity
Run psql -h host -U hawiyati -d hawiyati_db -c "SELECT 1". Check that the Flyway migration head matches the expected version (SELECT version FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 1).
5
Check Network & TLS
Verify TLS certificates are valid and not expired (openssl s_client -connect idp.hawiyati.org:8085). Check firewall rules between IdP, SP, and admin hosts. For SAML/OIDC issues, confirm both parties can reach each other's endpoints.

The audit log is your primary diagnostic tool. Navigate to Admin → Audit Log. Events are displayed newest-first.

Useful filter combinations

What you're investigatingFilter settings
Failed login attempts for a userCategory: authentication · Outcome: failure · Actor: username
All activity for a specific sessionSearch: paste the correlationId from any event in the request
SAML assertion failuresCategory: federation · Outcome: failure
Admin changes to a userCategory: iam · Search: resource ID or username
Token issuance for an OIDC clientCategory: token · Search: client_id
KYC approval/rejection eventsCategory: iam · Action: kyc.approve or kyc.reject
SCIM provisioning eventsCategory: iam · Search: externalId value
Configuration changesCategory: configuration

Key fields to check

FieldWhat it tells you
failureReasonMachine-readable reason code for failures — most specific diagnostic signal available.
detailAction-specific metadata. Note: sensitive values (token, password, secret, key) are always redacted to ***.
correlationIdTrace ID shared across all service calls in a single request — paste into Search to see the full picture.
sourceIpClient IP — useful for identifying automated attack sources or misconfigured SP IPs.
serviceWhich Hawiyati module emitted the event — narrows which log file to inspect.
ℹ️
Note

Every audit log search you perform is itself logged as an audit.query event. The audit trail cannot be reviewed without leaving a record.

Adjusting Log Levels

To enable verbose logging for a specific component without restarting the service, use the Spring Boot Actuator log level endpoint:

Enable DEBUG for SAML processing
curl -X POST https://localhost:8085/actuator/loggers/hawiyati.saml \ -H 'Content-Type: application/json' \ -d '{"configuredLevel":"DEBUG"}'
Enable DEBUG for OIDC token issuance
curl -X POST https://localhost:8085/actuator/loggers/hawiyati.openid \ -H 'Content-Type: application/json' \ -d '{"configuredLevel":"DEBUG"}'
Restore to INFO when done
curl -X POST https://localhost:8085/actuator/loggers/hawiyati.saml \ -H 'Content-Type: application/json' \ -d '{"configuredLevel":"INFO"}'
⚠️
Warning

DEBUG logging may write sensitive request parameters to the log file. Never leave DEBUG enabled in production for longer than a diagnostic session. Restore INFO immediately after capturing the needed trace.

Log file locations (default configuration):

ServiceLog file
hawiyati-http-serverlogs/hawiyati-http-server.log
hawiyati-adminlogs/hawiyati-admin.log
hawiyati-scimlogs/hawiyati-scim.log
hawiyati-gatewaylogs/hawiyati-gateway.log
hawiyati-kyclogs/hawiyati-kyc.log

SAML 2.0 Errors

SAML errors typically surface as a redirect to an error page at the SP or IdP, or as an HTTP 400/500 at the ACS endpoint. Use the SAML Tracer browser extension to inspect raw assertions and AuthnRequests during diagnosis.

InResponseTo mismatch
Cause: The SP received an assertion whose InResponseTo attribute does not match any outstanding AuthnRequest ID. Usually caused by stale SP session state, double-submission of the AuthnRequest, or the SP's in-memory AuthnRequest store being reset (restart or load-balancer failover).
Fix: Clear the SP's session/cookie for the affected user and retry. If it happens consistently, check that the SP uses sticky sessions or stores AuthnRequest IDs in a shared store. Ensure both IdP and SP clocks are NTP-synchronized.
Signature validation failed
Cause: The SP cannot verify the signature on the assertion or response. Most commonly caused by a stale IdP certificate in the SP's metadata after a certificate rollover, or by metadata fetched before the new certificate was published.
Fix: Re-import the current Hawiyati IdP metadata (https://idp.hawiyati.org/saml/metadata) into the SP. Verify the certificate thumbprint matches what Hawiyati reports in the admin console under the SP's application settings.
ACS URL not registered
Cause: The ACS URL in the AuthnRequest or the SP's metadata does not match any registered ACS URL for that Entity ID in Hawiyati's SP registry. Hawiyati rejects all unregistered ACS URLs as a security control.
Fix: In Admin → Applications, open the SP and verify the ACS URL field exactly matches what the SP sends (including trailing slashes and protocol). Re-upload the SP's metadata if the ACS URL changed.
Assertion expired
Cause: The SP received the assertion after its NotOnOrAfter timestamp elapsed. This nearly always indicates clock drift between the IdP and SP servers. SAML assertions have a short validity window (typically 5 minutes).
Fix: Synchronize both server clocks to an NTP source. Most SP implementations allow a configurable clock-skew tolerance (e.g., ±3 minutes) — enable this as a short-term mitigation while fixing the clock drift.
NameID format not supported
Cause: The SP's AuthnRequest specifies a NameID format that Hawiyati does not support, or the format Hawiyati sends does not match what the SP expects.
Fix: Hawiyati supports urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress and urn:oasis:names:tc:SAML:2.0:nameid-format:persistent. Configure the SP to request one of these, or remove the NameIDPolicy element to accept any format.
AuthnRequest unsigned when required
Cause: The SP's registration in Hawiyati is configured to require signed AuthnRequests, but the SP is sending unsigned requests.
Fix: Either configure the SP to sign its AuthnRequests using its private key (and upload the SP's certificate into Hawiyati), or in Admin → Applications disable "Require signed AuthnRequests" for this SP if signing is not needed.
SLO not completing
Cause: Single Logout fails to propagate to all SPs. Typically the IdP cannot reach one SP's SLO endpoint, or the SP returns an error response during SLO.
Fix: Check that the SP's SLO endpoint URL is registered and reachable from the IdP's network. Review the federation category in the audit log for the specific SP that failed. If the SP does not support SLO, remove the SLO URL from its registration to prevent the chain from stalling.
400 Bad Request at ACS
Cause: The SP's ACS endpoint returned HTTP 400. Usually a malformed POST body, wrong SAML binding, or the SP's own parsing error.
Fix: Use SAML Tracer to inspect the raw POST to the ACS. Confirm the SP expects HTTP-POST binding (not HTTP-Redirect) for responses. Check for missing or incorrect Content-Type header. Review SP application logs for parsing errors.
Redirect loop at login
Cause: The browser is bouncing between the SP and IdP without authenticating. Commonly caused by the IdP session cookie not being set — usually a misconfigured SameSite or Secure attribute preventing the cookie from being sent cross-origin.
Fix: Confirm Hawiyati is served over HTTPS (required for Secure cookies). Check the hawiyati.session.cookie.same-site configuration — for SP-initiated flows across different domains, None with Secure is required. Verify the browser is not in a private window with cookies blocked.

OIDC Errors

OIDC errors are returned as JSON error responses in the authorization redirect or token endpoint response. The error and error_description fields identify the issue. Check the token category in the audit log for server-side context.

invalid_client
Cause: The client_id is not registered, or the client secret is wrong (for confidential clients).
Fix: In Admin → Applications, verify the client_id exists and the secret matches what the application is using. Re-generate the client secret if it may have been rotated.
redirect_uri_mismatch
Cause: The redirect_uri in the authorization request is not in the registered allowlist for this client. Hawiyati performs exact-match validation as a security control.
Fix: In Admin → Applications, add the exact URI the application sends (including protocol, path, and any trailing slash). Even a trailing slash difference causes a mismatch. Wildcards are not supported.
invalid_grant
Cause: The authorization code has expired (codes are short-lived, typically 60 seconds), has already been used (codes are single-use), or was issued to a different client.
Fix: Exchange the authorization code immediately after the redirect — do not delay or retry with the same code. Ensure the token endpoint request uses the same redirect_uri that was used in the authorization request. Check server clock sync if codes expire prematurely.
code_challenge_failed
Cause: The code_verifier in the token request does not match the code_challenge that was sent in the authorization request. PKCE verification failed.
Fix: Verify the PKCE implementation: generate a cryptographically random 43–128 byte code_verifier, compute code_challenge = BASE64URL(SHA256(code_verifier)), use code_challenge_method=S256. Do not reuse or mutate the verifier between the two requests.
token expired
Cause: The access token or ID token has passed its exp claim. Applications must handle token expiry by refreshing or re-authenticating.
Fix: Implement token refresh using the refresh token before the access token expires. Check the exp claim at login and schedule a proactive refresh (e.g., 60 seconds before expiry). If no refresh token was issued, the client scope may not include offline_access.
invalid_token (UserInfo)
Cause: The access token presented to the UserInfo endpoint is expired, revoked, or was not issued with the openid scope.
Fix: Confirm the authorization request included the openid scope. Refresh the access token before calling UserInfo. Check the token category in the audit log for a revocation event.
Discovery document 404
Cause: The OIDC library cannot fetch the discovery document. Wrong issuer URL configured in the application.
Fix: The discovery document is at https://idp.hawiyati.org:8085/.well-known/openid-configuration. The issuer value is https://idp.hawiyati.org:8085 — configure this exactly in your OIDC library's issuer field. Do not append a path.
RS256 signature validation fails
Cause: The application is using a stale or wrong public key to verify the ID token. This happens after key rotation or if the JWKS was cached before the current signing key was published.
Fix: Re-fetch the JWKS from https://idp.hawiyati.org:8085/jwks. Most OIDC libraries cache JWKS — force a refresh. Use the kid claim in the token header to select the correct key from the JWKS.

Authentication & MFA Errors

SymptomLikely CauseFix
Account locked, cannot log in Failed login attempt threshold exceeded (default: 5 consecutive failures) Admin → Users → find user → Unlock Account. Adjust lockout threshold in Admin → Settings if threshold is too aggressive.
TOTP code rejected immediately Clock drift between user device and server. TOTP uses 30-second windows — even a 31-second drift causes failure. Sync the server clock to NTP. Hawiyati allows ±1 step tolerance (±30 seconds) by default. Ensure user device clock is accurate. Do not advise users to try "the next code" more than once without addressing the root cause.
FIDO2 registration fails in browser HTTPS not enabled (WebAuthn requires a secure context), or the RP ID does not match the page origin. Ensure the login page is served over HTTPS. Verify hawiyati.fido2.rp-id in configuration matches the hostname (without port) of the page where registration occurs.
FIDO2 authentication fails Credential ID not found, or the authenticator's counter has advanced beyond the stored value (cloned credential detection). In Admin → Users, check the user's registered FIDO2 credentials. If the credential was deleted from the authenticator, remove it from Hawiyati too and re-enroll. A counter mismatch may indicate a cloned authenticator — treat as a security incident.
SMS OTP not delivered SMS gateway connectivity issue, MSISDN not in E.164 format, or per-MSISDN rate limit reached. Verify the SMS gateway endpoint and credentials in Admin → Settings. Confirm the user's phone number is stored in E.164 format (e.g., +213XXXXXXXXX). Check gateway rate limit logs. OTP codes have a 5-minute TTL — a delayed delivery may arrive after expiry.
Email OTP not delivered SMTP configuration issue or email flagged as spam. Test SMTP connectivity from the server. Check the spam/junk folder. Verify the sender domain has SPF and DKIM records. Review email gateway logs for bounce/rejection codes.
Recovery code rejected Recovery codes are single-use. A code that was already redeemed cannot be used again. Hawiyati shows recovery codes once at enrollment — users must have saved them. Admin can generate a new set via Admin → Users → Reset MFA → Generate Recovery Codes (this invalidates all previous codes).
Step-up MFA loop — keeps prompting for MFA The session's MFA-completed flag is not being set after a successful second factor. Often a session service connectivity issue. Check hawiyati-session gRPC service health. Look for session service errors in hawiyati-http-server logs at the time of the MFA completion. Confirm Redis connectivity (session state is stored in Redis).

Account & Session Issues

SymptomLikely CauseFix
User cannot log in — "Invalid credentials" even with correct password active=false (account inactive) or pendingApproval=true without approval (ACTIVE_LIMITED state) Admin → Users → check status flags. If inactive, activate the account. If pending approval, approve or investigate why approval is pending. Also verify the password hash is in Argon2id format — a plain-text hash from a migration will never match.
MIGRATED_UNCLAIMED user cannot log in Expected. Accounts in MIGRATED_UNCLAIMED state (active=false, pendingApproval=true) cannot log in until they claim their account via the claim token flow. Send the claim link via Admin → Users → Resend Claim Link. If the claim token has expired (15-minute TTL), generate a new one. The user must follow the link to set credentials before they can log in.
Session not persisting — logged out on every page load Session cookie not being sent back due to SameSite/Secure/domain mismatch, or Redis session expiry too short. Check hawiyati.session.cookie.same-site and hawiyati.session.cookie.secure in configuration. Cookies with Secure=true require HTTPS. Verify the cookie domain matches the request host. Check hawiyati.session.ttl — if set very low, sessions expire quickly.
Concurrent session limit reached The policy for this user or SP limits concurrent active sessions. Admin → Users → view active sessions → terminate older sessions. Or adjust the concurrent session limit in Admin → Settings or the SP's application config.
Claim token expired Claim tokens have a 15-minute TTL and are single-use. The link in the email has expired. Admin → Users → Resend Claim Link to issue a fresh claim token. Educate users to follow the link promptly. If claim expiry is too aggressive for your workflow, adjust hawiyati.registration.claim-token-ttl.
Password reset email not received SMTP delivery failure or email spam filtering. Test SMTP configuration. Check audit log for password_reset.email.send events. Review email delivery logs for bounce codes. Confirm the user's email address in Admin → Users is correct and accepts external mail.

SCIM Provisioning Errors

ℹ️
SCIM endpoint

The Hawiyati SCIM 2.0 base URL is https://idp.hawiyati.org:8098/scim/v2. All SCIM requests require a Bearer token in the Authorization header.

HTTP Status / ErrorCauseFix
401 Unauthorized Missing or invalid Bearer token. Generate a SCIM bearer token in Admin → Applications → (your SCIM client) → Generate Token. Configure the SCIM client to send Authorization: Bearer <token>. Tokens do not expire by default but can be revoked and re-generated.
404 Not Found on /scim/v2/Users Wrong SCIM base URL — missing the /scim/v2 prefix, wrong port, or wrong hostname. The correct base URL is https://<idp-host>:8098/scim/v2. Note port 8098 (not 8085). Verify in Admin → Applications under the SCIM client configuration.
409 Conflict A user with the same externalId (or same username/email) already exists in Hawiyati. Check for duplicate externalId in Admin → Users. If the existing user is from a previous sync, update (PATCH) rather than create (POST). De-duplicate the source system if the same user is being provisioned twice.
User provisioned but status is MIGRATED_UNCLAIMED This is correct and expected behaviour. SCIM-provisioned users start as MIGRATED_UNCLAIMED (active=false, pendingApproval=true) and must claim their account before they can log in. Send the claim link to the provisioned user. This is the intended workflow: SCIM provisions the account shell, the user claims it and sets credentials. See the Identity Lifecycle guide for the full claim flow.
SCIM PATCH not updating the user Unsupported SCIM attribute or incorrect JSON patch path. Hawiyati's SCIM implementation supports a defined subset of attributes. Hawiyati supports via SCIM: externalId, sourceInstanceId, pendingApproval, standard SCIM core attributes (userName, name, emails, phoneNumbers, active). Credential and assurance level fields cannot be set via SCIM. Remove unsupported attributes from the PATCH payload.
Group sync not working Groups endpoint not configured, or the group's members array is empty in the PATCH. Confirm the SCIM client is targeting /scim/v2/Groups. Use PATCH with op: replace on the members attribute. Verify group names match expected values. Check the iam category in the audit log for user_group.add_member / user_group.remove_member events.
Attribute values not mapped correctly SCIM attribute names are case-sensitive. A field named externalId is not the same as ExternalId. Use exactly the attribute names documented in the SCIM Integration Guide. Inspect the SCIM request/response body with a tool like Postman. Enable DEBUG logging on hawiyati.scim temporarily to see the parsed attribute map.

Infrastructure Issues

PostgreSQL connection refused
Cause: pg_hba.conf not allowing connections from the application host, wrong credentials, or PostgreSQL not listening on the expected interface/port.
Fix:
  1. Check postgresql.conf — ensure listen_addresses includes the application host or *.
  2. Check pg_hba.conf — add a line allowing the application user from the application host.
  3. Run psql -h <host> -U hawiyati -d hawiyati_db from the application server to verify connectivity.
  4. Check firewall — port 5432 must be open between application and database hosts.
Redis connection timeout
Cause: Redis bind address set to 127.0.0.1 only (unreachable from application host), wrong AUTH password, maxclients limit reached, or firewall blocking port 6379.
Fix:
  1. Run redis-cli -h <host> ping — expect PONG.
  2. If password-protected: redis-cli -h <host> -a <password> ping.
  3. Check redis.conf — set bind 0.0.0.0 or the specific interface the application connects to.
  4. Check INFO clients for connected_clients near maxclients.
  5. Verify the password in application.yml matches requirepass in redis.conf.
Flyway migration failed on startup
Cause: A migration script failed mid-run (leaving the database in a partially-applied state), incorrect V-file naming, insufficient DB permissions, or a migration checksum mismatch from a modified migration file.
Fix:
  1. Check flyway_schema_history for rows with success=false.
  2. Fix the failed SQL, then run flyway repair to clear the failed row before restarting.
  3. Ensure the DB user has CREATE TABLE, ALTER TABLE, and CREATE INDEX privileges.
  4. Never modify an already-applied migration file — add a new versioned migration instead.
  5. Check that migration V-numbers are sequential and there are no gaps or duplicates.
Port already in use (8085 / 8095 / 8080)
Cause: Another process is listening on the same port, or a previous instance of Hawiyati did not shut down cleanly.
Fix (Linux):
sudo lsof -i :8085 sudo kill <PID>
Fix (Windows):
netstat -ano | findstr :8085 taskkill /PID <PID> /F
TLS handshake failure / PKIX path building failed
Cause: The JVM does not trust the TLS certificate presented by another service (e.g., the IdP's dev cert, a self-signed cert, or an internal CA). This affects outbound HTTPS calls from Hawiyati modules.
Fix:
  1. Export the certificate: openssl s_client -connect host:port </dev/null | openssl x509 -outform PEM > cert.pem
  2. Import into the JVM truststore: keytool -import -alias hawiyati-idp -keystore $JAVA_HOME/lib/security/cacerts -file cert.pem (default password: changeit)
  3. For dev environments using mkcert: run mkcert -install on each machine, then import the mkcert root CA into the JVM truststore.
  4. Restart the application after importing the certificate.
gRPC service unavailable
Cause: An internal gRPC service (hawiyati-session, hawiyati-infra, hawiyati-cache) is not running or is unreachable from hawiyati-http-server.
Fix: Check the service's process status and logs. Verify the gRPC port and address configured in application.yml matches what the service is listening on. gRPC services use internal ports not exposed externally — check the service network layout in the deployment guide.

Support Escalation

Before raising a support ticket, collect the following information. Escalations with complete diagnostic data resolve significantly faster.

Information to collect

ItemHow to collect
Hawiyati versionAdmin → Settings → About, or check pom.xml root version element.
Audit log export (JSON)Admin → Audit Log → filter to the affected time window → Export → JSON. Max 50,000 rows.
Application log excerptCollect 5–10 minutes of logs from the affected service around the time of the incident. Include the full stack trace if present.
Affected SP Entity ID or OIDC client_idFrom Admin → Applications, or from the SAML AuthnRequest / OIDC authorization request.
Affected user (anonymised if needed)Internal username or user ID from Admin → Users. Do not include passwords or tokens.
Steps to reproduceExact sequence of actions that triggers the issue, including URLs visited, buttons clicked, and any error messages displayed.
Error message verbatimCopy the exact error text, error code, and HTTP status code. Screenshot the browser error page if applicable.
Correlation IDFrom the audit log event detail — allows cross-service trace reconstruction.

Severity classification

SeverityDefinitionExamples
P1 — Critical Complete authentication service unavailable. No users can log in. IdP HTTP service down; PostgreSQL unreachable; all login attempts fail with 500
P2 — High Significant degradation. A major feature or subset of users is affected. SAML broken for all SPs; MFA factor unavailable; SCIM provisioning stopped; admin console unreachable
P3 — Medium Non-critical feature issue or intermittent failure affecting some users. Intermittent TOTP rejections; one SP having SAML errors; audit log export slow; single user locked
P4 — Low Question, documentation request, or cosmetic issue with no service impact. How-to questions; configuration guidance; minor UI display issues; documentation clarification
✓
Best Practice

For P1 incidents, call the emergency support line in addition to opening a ticket. Do not rely on email alone for critical outages. Keep the on-call runbook up to date with current contact details.

⚠️
Security Incidents

If the issue involves suspected unauthorized access, credential compromise, or a security control bypass — treat it as a security incident, not a support ticket. Follow your organization's incident response procedure. Do not share audit logs or credential data over unencrypted channels.

HawiyatiDocumentation · v2026 · Spring Boot 3.5