Troubleshooting
This guide covers the most common error conditions in Hawiyati — SAML, OIDC, MFA, accounts, SCIM, and infrastructure — with symptoms, root causes, and fixes. Tip: press Ctrl+F and paste your error code to jump straight to it.
This guide covers the most common error conditions in Hawiyati — SAML, OIDC, MFA, accounts, SCIM, and infrastructure — with symptoms, root causes, and fixes. Tip: press Ctrl+F and paste your error code to jump straight to it.
The sections below cover common error conditions, troubleshooting procedures, and escalation paths. For a full module map with ports, gRPC boundaries, and deployment topology, see the Implementation Guide.
Diagnostic Approach
When something goes wrong, follow this sequence before escalating. Most issues are resolved at step 1 or 2.
failureReason field usually identifies the exact problem. Authentication failures, policy denials, and token errors all produce structured events.hawiyati-http-server for login, hawiyati-admin for admin console, hawiyati-scim for SCIM). Look for ERROR or WARN lines at the same timestamp.redis-cli ping — expect PONG. Check redis-cli info memory for near-limit conditions. Session failures and rate-limit anomalies often trace to Redis connectivity.psql -h host -U hawiyati -d hawiyati_db -c "SELECT 1". Check that the Flyway migration head matches the expected version (SELECT version FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 1).openssl s_client -connect idp.hawiyati.org:8085). Check firewall rules between IdP, SP, and admin hosts. For SAML/OIDC issues, confirm both parties can reach each other's endpoints.Reading Audit Logs
The audit log is your primary diagnostic tool. Navigate to Admin → Audit Log. Events are displayed newest-first.
Useful filter combinations
| What you're investigating | Filter settings |
|---|---|
| Failed login attempts for a user | Category: authentication · Outcome: failure · Actor: username |
| All activity for a specific session | Search: paste the correlationId from any event in the request |
| SAML assertion failures | Category: federation · Outcome: failure |
| Admin changes to a user | Category: iam · Search: resource ID or username |
| Token issuance for an OIDC client | Category: token · Search: client_id |
| KYC approval/rejection events | Category: iam · Action: kyc.approve or kyc.reject |
| SCIM provisioning events | Category: iam · Search: externalId value |
| Configuration changes | Category: configuration |
Key fields to check
| Field | What it tells you |
|---|---|
failureReason | Machine-readable reason code for failures — most specific diagnostic signal available. |
detail | Action-specific metadata. Note: sensitive values (token, password, secret, key) are always redacted to ***. |
correlationId | Trace ID shared across all service calls in a single request — paste into Search to see the full picture. |
sourceIp | Client IP — useful for identifying automated attack sources or misconfigured SP IPs. |
service | Which Hawiyati module emitted the event — narrows which log file to inspect. |
Every audit log search you perform is itself logged as an audit.query event. The audit trail cannot be reviewed without leaving a record.
Adjusting Log Levels
To enable verbose logging for a specific component without restarting the service, use the Spring Boot Actuator log level endpoint:
DEBUG logging may write sensitive request parameters to the log file. Never leave DEBUG enabled in production for longer than a diagnostic session. Restore INFO immediately after capturing the needed trace.
Log file locations (default configuration):
| Service | Log file |
|---|---|
hawiyati-http-server | logs/hawiyati-http-server.log |
hawiyati-admin | logs/hawiyati-admin.log |
hawiyati-scim | logs/hawiyati-scim.log |
hawiyati-gateway | logs/hawiyati-gateway.log |
hawiyati-kyc | logs/hawiyati-kyc.log |
SAML 2.0 Errors
SAML errors typically surface as a redirect to an error page at the SP or IdP, or as an HTTP 400/500 at the ACS endpoint. Use the SAML Tracer browser extension to inspect raw assertions and AuthnRequests during diagnosis.
InResponseTo attribute does not match any outstanding AuthnRequest ID. Usually caused by stale SP session state, double-submission of the AuthnRequest, or the SP's in-memory AuthnRequest store being reset (restart or load-balancer failover).https://idp.hawiyati.org/saml/metadata) into the SP. Verify the certificate thumbprint matches what Hawiyati reports in the admin console under the SP's application settings.NotOnOrAfter timestamp elapsed. This nearly always indicates clock drift between the IdP and SP servers. SAML assertions have a short validity window (typically 5 minutes).urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress and urn:oasis:names:tc:SAML:2.0:nameid-format:persistent. Configure the SP to request one of these, or remove the NameIDPolicy element to accept any format.SameSite or Secure attribute preventing the cookie from being sent cross-origin.Secure cookies). Check the hawiyati.session.cookie.same-site configuration — for SP-initiated flows across different domains, None with Secure is required. Verify the browser is not in a private window with cookies blocked.OIDC Errors
OIDC errors are returned as JSON error responses in the authorization redirect or token endpoint response. The error and error_description fields identify the issue. Check the token category in the audit log for server-side context.
redirect_uri in the authorization request is not in the registered allowlist for this client. Hawiyati performs exact-match validation as a security control.redirect_uri that was used in the authorization request. Check server clock sync if codes expire prematurely.code_verifier in the token request does not match the code_challenge that was sent in the authorization request. PKCE verification failed.code_verifier, compute code_challenge = BASE64URL(SHA256(code_verifier)), use code_challenge_method=S256. Do not reuse or mutate the verifier between the two requests.exp claim. Applications must handle token expiry by refreshing or re-authenticating.exp claim at login and schedule a proactive refresh (e.g., 60 seconds before expiry). If no refresh token was issued, the client scope may not include offline_access.openid scope.openid scope. Refresh the access token before calling UserInfo. Check the token category in the audit log for a revocation event.https://idp.hawiyati.org:8085/.well-known/openid-configuration. The issuer value is https://idp.hawiyati.org:8085 — configure this exactly in your OIDC library's issuer field. Do not append a path.https://idp.hawiyati.org:8085/jwks. Most OIDC libraries cache JWKS — force a refresh. Use the kid claim in the token header to select the correct key from the JWKS.Authentication & MFA Errors
| Symptom | Likely Cause | Fix |
|---|---|---|
| Account locked, cannot log in | Failed login attempt threshold exceeded (default: 5 consecutive failures) | Admin → Users → find user → Unlock Account. Adjust lockout threshold in Admin → Settings if threshold is too aggressive. |
| TOTP code rejected immediately | Clock drift between user device and server. TOTP uses 30-second windows — even a 31-second drift causes failure. | Sync the server clock to NTP. Hawiyati allows ±1 step tolerance (±30 seconds) by default. Ensure user device clock is accurate. Do not advise users to try "the next code" more than once without addressing the root cause. |
| FIDO2 registration fails in browser | HTTPS not enabled (WebAuthn requires a secure context), or the RP ID does not match the page origin. | Ensure the login page is served over HTTPS. Verify hawiyati.fido2.rp-id in configuration matches the hostname (without port) of the page where registration occurs. |
| FIDO2 authentication fails | Credential ID not found, or the authenticator's counter has advanced beyond the stored value (cloned credential detection). | In Admin → Users, check the user's registered FIDO2 credentials. If the credential was deleted from the authenticator, remove it from Hawiyati too and re-enroll. A counter mismatch may indicate a cloned authenticator — treat as a security incident. |
| SMS OTP not delivered | SMS gateway connectivity issue, MSISDN not in E.164 format, or per-MSISDN rate limit reached. | Verify the SMS gateway endpoint and credentials in Admin → Settings. Confirm the user's phone number is stored in E.164 format (e.g., +213XXXXXXXXX). Check gateway rate limit logs. OTP codes have a 5-minute TTL — a delayed delivery may arrive after expiry. |
| Email OTP not delivered | SMTP configuration issue or email flagged as spam. | Test SMTP connectivity from the server. Check the spam/junk folder. Verify the sender domain has SPF and DKIM records. Review email gateway logs for bounce/rejection codes. |
| Recovery code rejected | Recovery codes are single-use. A code that was already redeemed cannot be used again. | Hawiyati shows recovery codes once at enrollment — users must have saved them. Admin can generate a new set via Admin → Users → Reset MFA → Generate Recovery Codes (this invalidates all previous codes). |
| Step-up MFA loop — keeps prompting for MFA | The session's MFA-completed flag is not being set after a successful second factor. Often a session service connectivity issue. | Check hawiyati-session gRPC service health. Look for session service errors in hawiyati-http-server logs at the time of the MFA completion. Confirm Redis connectivity (session state is stored in Redis). |
Account & Session Issues
| Symptom | Likely Cause | Fix |
|---|---|---|
| User cannot log in — "Invalid credentials" even with correct password | active=false (account inactive) or pendingApproval=true without approval (ACTIVE_LIMITED state) |
Admin → Users → check status flags. If inactive, activate the account. If pending approval, approve or investigate why approval is pending. Also verify the password hash is in Argon2id format — a plain-text hash from a migration will never match. |
| MIGRATED_UNCLAIMED user cannot log in | Expected. Accounts in MIGRATED_UNCLAIMED state (active=false, pendingApproval=true) cannot log in until they claim their account via the claim token flow. |
Send the claim link via Admin → Users → Resend Claim Link. If the claim token has expired (15-minute TTL), generate a new one. The user must follow the link to set credentials before they can log in. |
| Session not persisting — logged out on every page load | Session cookie not being sent back due to SameSite/Secure/domain mismatch, or Redis session expiry too short. | Check hawiyati.session.cookie.same-site and hawiyati.session.cookie.secure in configuration. Cookies with Secure=true require HTTPS. Verify the cookie domain matches the request host. Check hawiyati.session.ttl — if set very low, sessions expire quickly. |
| Concurrent session limit reached | The policy for this user or SP limits concurrent active sessions. | Admin → Users → view active sessions → terminate older sessions. Or adjust the concurrent session limit in Admin → Settings or the SP's application config. |
| Claim token expired | Claim tokens have a 15-minute TTL and are single-use. The link in the email has expired. | Admin → Users → Resend Claim Link to issue a fresh claim token. Educate users to follow the link promptly. If claim expiry is too aggressive for your workflow, adjust hawiyati.registration.claim-token-ttl. |
| Password reset email not received | SMTP delivery failure or email spam filtering. | Test SMTP configuration. Check audit log for password_reset.email.send events. Review email delivery logs for bounce codes. Confirm the user's email address in Admin → Users is correct and accepts external mail. |
SCIM Provisioning Errors
The Hawiyati SCIM 2.0 base URL is https://idp.hawiyati.org:8098/scim/v2. All SCIM requests require a Bearer token in the Authorization header.
| HTTP Status / Error | Cause | Fix |
|---|---|---|
401 Unauthorized |
Missing or invalid Bearer token. | Generate a SCIM bearer token in Admin → Applications → (your SCIM client) → Generate Token. Configure the SCIM client to send Authorization: Bearer <token>. Tokens do not expire by default but can be revoked and re-generated. |
404 Not Found on /scim/v2/Users |
Wrong SCIM base URL — missing the /scim/v2 prefix, wrong port, or wrong hostname. |
The correct base URL is https://<idp-host>:8098/scim/v2. Note port 8098 (not 8085). Verify in Admin → Applications under the SCIM client configuration. |
409 Conflict |
A user with the same externalId (or same username/email) already exists in Hawiyati. |
Check for duplicate externalId in Admin → Users. If the existing user is from a previous sync, update (PATCH) rather than create (POST). De-duplicate the source system if the same user is being provisioned twice. |
| User provisioned but status is MIGRATED_UNCLAIMED | This is correct and expected behaviour. SCIM-provisioned users start as MIGRATED_UNCLAIMED (active=false, pendingApproval=true) and must claim their account before they can log in. | Send the claim link to the provisioned user. This is the intended workflow: SCIM provisions the account shell, the user claims it and sets credentials. See the Identity Lifecycle guide for the full claim flow. |
| SCIM PATCH not updating the user | Unsupported SCIM attribute or incorrect JSON patch path. Hawiyati's SCIM implementation supports a defined subset of attributes. | Hawiyati supports via SCIM: externalId, sourceInstanceId, pendingApproval, standard SCIM core attributes (userName, name, emails, phoneNumbers, active). Credential and assurance level fields cannot be set via SCIM. Remove unsupported attributes from the PATCH payload. |
| Group sync not working | Groups endpoint not configured, or the group's members array is empty in the PATCH. | Confirm the SCIM client is targeting /scim/v2/Groups. Use PATCH with op: replace on the members attribute. Verify group names match expected values. Check the iam category in the audit log for user_group.add_member / user_group.remove_member events. |
| Attribute values not mapped correctly | SCIM attribute names are case-sensitive. A field named externalId is not the same as ExternalId. |
Use exactly the attribute names documented in the SCIM Integration Guide. Inspect the SCIM request/response body with a tool like Postman. Enable DEBUG logging on hawiyati.scim temporarily to see the parsed attribute map. |
Infrastructure Issues
pg_hba.conf not allowing connections from the application host, wrong credentials, or PostgreSQL not listening on the expected interface/port.- Check
postgresql.conf— ensurelisten_addressesincludes the application host or*. - Check
pg_hba.conf— add a line allowing the application user from the application host. - Run
psql -h <host> -U hawiyati -d hawiyati_dbfrom the application server to verify connectivity. - Check firewall — port 5432 must be open between application and database hosts.
bind address set to 127.0.0.1 only (unreachable from application host), wrong AUTH password, maxclients limit reached, or firewall blocking port 6379.- Run
redis-cli -h <host> ping— expectPONG. - If password-protected:
redis-cli -h <host> -a <password> ping. - Check
redis.conf— setbind 0.0.0.0or the specific interface the application connects to. - Check
INFO clientsfor connected_clients nearmaxclients. - Verify the password in
application.ymlmatchesrequirepassinredis.conf.
- Check
flyway_schema_historyfor rows withsuccess=false. - Fix the failed SQL, then run
flyway repairto clear the failed row before restarting. - Ensure the DB user has
CREATE TABLE,ALTER TABLE, andCREATE INDEXprivileges. - Never modify an already-applied migration file — add a new versioned migration instead.
- Check that migration V-numbers are sequential and there are no gaps or duplicates.
- Export the certificate:
openssl s_client -connect host:port </dev/null | openssl x509 -outform PEM > cert.pem - Import into the JVM truststore:
keytool -import -alias hawiyati-idp -keystore $JAVA_HOME/lib/security/cacerts -file cert.pem(default password:changeit) - For dev environments using mkcert: run
mkcert -installon each machine, then import the mkcert root CA into the JVM truststore. - Restart the application after importing the certificate.
application.yml matches what the service is listening on. gRPC services use internal ports not exposed externally — check the service network layout in the deployment guide.Support Escalation
Before raising a support ticket, collect the following information. Escalations with complete diagnostic data resolve significantly faster.
Information to collect
| Item | How to collect |
|---|---|
| Hawiyati version | Admin → Settings → About, or check pom.xml root version element. |
| Audit log export (JSON) | Admin → Audit Log → filter to the affected time window → Export → JSON. Max 50,000 rows. |
| Application log excerpt | Collect 5–10 minutes of logs from the affected service around the time of the incident. Include the full stack trace if present. |
| Affected SP Entity ID or OIDC client_id | From Admin → Applications, or from the SAML AuthnRequest / OIDC authorization request. |
| Affected user (anonymised if needed) | Internal username or user ID from Admin → Users. Do not include passwords or tokens. |
| Steps to reproduce | Exact sequence of actions that triggers the issue, including URLs visited, buttons clicked, and any error messages displayed. |
| Error message verbatim | Copy the exact error text, error code, and HTTP status code. Screenshot the browser error page if applicable. |
| Correlation ID | From the audit log event detail — allows cross-service trace reconstruction. |
Severity classification
| Severity | Definition | Examples |
|---|---|---|
| P1 — Critical | Complete authentication service unavailable. No users can log in. | IdP HTTP service down; PostgreSQL unreachable; all login attempts fail with 500 |
| P2 — High | Significant degradation. A major feature or subset of users is affected. | SAML broken for all SPs; MFA factor unavailable; SCIM provisioning stopped; admin console unreachable |
| P3 — Medium | Non-critical feature issue or intermittent failure affecting some users. | Intermittent TOTP rejections; one SP having SAML errors; audit log export slow; single user locked |
| P4 — Low | Question, documentation request, or cosmetic issue with no service impact. | How-to questions; configuration guidance; minor UI display issues; documentation clarification |
For P1 incidents, call the emergency support line in addition to opening a ticket. Do not rely on email alone for critical outages. Keep the on-call runbook up to date with current contact details.
If the issue involves suspected unauthorized access, credential compromise, or a security control bypass — treat it as a security incident, not a support ticket. Follow your organization's incident response procedure. Do not share audit logs or credential data over unencrypted channels.