There are no circumstances where manually changing the passwords for the SYSMAN-named database accounts will end happily.  Never attempt to change the passwords for SYSMAN, SYSMAN_APM, SYSMAN_BIPLATFORM, SYSMAN_MDS, SYSMAN_OPSS, or SYSMAN_RO from within the database.

OEM security is managed by the WebLogic admin server.  The database passwords are only part of the puzzle.  Your WLS relies on wallets and other encrypted files to keep it all straight.  When you do the password change in the repository OEM, security validations against those files will fail and OEM will be hopelessly broken.

When you invoke this simple emctl command a slew of activities occur

emctl config oms -change_repos_pwd

That triggers this chain of actions (from .. sysman/log.secure.log):

oms.AdminCredsWalletUtil setInstanceHome.177 – Getting credentials from wallet
oms.AdminCredsWalletUtil setInstanceHome.192 – Read the credentials from wallet
util.EmctlUtil logp.251 – Connecting over t3s to: oms01.demo.com/7103 using id: weblogic
oms.ChangeReposPwd logp.251 – Getting repos conn as user sys as sysdba
oms.ChangeReposPwd logp.251 – Getting repos conn as user sys as sysdba
oms.ChangeReposPwd logp.251 – SYSMAN password changed in the backend successfully.
oms.ChangeReposPwd logp.251 – Getting lock on table EM_UPDATE_DATASOURCES_LOCK
oms.ChangeReposPwd logp.251 – Getting repos conn as user SYSMAN
oms.ChangeReposPwd logp.251 – Successfully obtained lock on table EM_UPDATE_DATASOURCES_LOCK
oms.ChangeReposPwd logp.251 – Start change SYSMAN_MDS password
oms.ChangeReposPwd logp.251 – Changed SYSMAN_MDS password
oms.ChangeReposPwd logp.251 – Changing OPSS admin user’s pwd
oms.ChangeReposPwd logp.251 – Changed OPSS admin pwd
oms.ChangeReposPwd logp.251 – Changing APM admin user’s pwd
oms.ChangeReposPwd logp.251 – Changed APM admin pwd
oms.ChangeReposPwd logp.251 – Getting repos conn as user SYSMAN
oms.AdminCredsWalletUtil setInstanceHome.177 – Getting credentials from wallet
oms.AdminCredsWalletUtil setInstanceHome.192 – Read the credentials from wallet
util.EmctlUtil logp.251 – Connecting over t3s to: oms01.demo.com/7103 using id: weblogic
util.EmctlUtil logp.251 – Updating datasource : emgc-sysman-pool
util.EmctlUtil logp.251 – Updating WLS datasource :emgc-sysman-pool: Done startEditSession()
util.EmctlUtil logp.251 – Updating WLS datasource : DB datasource :emgc-sysman-pool: IS found
util.EmctlUtil logp.251 – Updating password
util.EmctlUtil logp.251 – Updating WLS datasource :emgc-sysman-pool: Invoked save
util.EmctlUtil logp.251 – Updating WLS datasource :emgc-sysman-pool: Done activate()
oms.ChangeReposPwd logp.251 – Updating OWSM DataSource
util.EmctlUtil logp.251 – Updating datasource : mds-owsm
util.EmctlUtil logp.251 – DataSource URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST= …
util.EmctlUtil logp.251 – Updating WLS datasource :mds-owsm: Done startEditSession()
util.EmctlUtil logp.251 – Updating WLS datasource : DB datasource :mds-owsm: IS found
util.EmctlUtil logp.251 – Updating URL
util.EmctlUtil logp.251 – Updating password
util.EmctlUtil logp.251 – Updating userName
util.EmctlUtil logp.251 – Updating WLS datasource :mds-owsm: Invoked save
util.EmctlUtil logp.251 – Updating WLS datasource :mds-owsm: Done activate()
oms.ChangeReposPwd logp.251 – Updating APM DataSource
util.EmctlUtil logp.251 – Updating datasource : apm-DBDS
util.EmctlUtil logp.251 – DataSource URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST= …
util.EmctlUtil logp.251 – Updating WLS datasource :apm-DBDS: Done startEditSession()
util.EmctlUtil logp.251 – Updating WLS datasource : DB datasource :apm-DBDS: IS found
util.EmctlUtil logp.251 – Updating URL
util.EmctlUtil logp.251 – Updating password
util.EmctlUtil logp.251 – Updating userName
util.EmctlUtil logp.251 – Updating WLS datasource :apm-DBDS: Invoked save
util.EmctlUtil logp.251 – Updating WLS datasource :apm-DBDS: Done activate()
oms.ChangeReposPwd logp.251 – Updating APM-MDS DataSource
util.EmctlUtil logp.251 – Updating datasource : mds-ApplicationMDSDB
util.EmctlUtil logp.251 – DataSource URL: jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST= …
util.EmctlUtil logp.251 – Updating WLS datasource :mds-ApplicationMDSDB: Done startEditSession()
util.EmctlUtil logp.251 – Updating WLS datasource : DB datasource :mds-ApplicationMDSDB: IS found
util.EmctlUtil logp.251 – Updating URL
util.EmctlUtil logp.251 – Updating password
util.EmctlUtil logp.251 – Updating userName
util.EmctlUtil logp.251 – Updating WLS datasource :mds-ApplicationMDSDB: Invoked save
util.EmctlUtil logp.251 – Updating WLS datasource :mds-ApplicationMDSDB: Done activate()
oms.ChangeReposPwd logp.251 – Updating OPSS DataSource
util.EmctlUtil logp.251 – Destroying datasource sysman-opss-ds
util.EmctlUtil logp.251 – destroyDataSource completed with status = true
util.EmctlUtil logp.251 – Creating datasource sysman-opss-ds
util.EmctlUtil logp.251 – Datasource created. Targetting to 5 servers.
util.EmctlUtil logp.251 – Targeting to EMGC_ADMINSERVER
util.EmctlUtil logp.251 – Targeted to EMGC_ADMINSERVER
util.EmctlUtil logp.251 – Targeting to EMGC_OMS1
util.EmctlUtil logp.251 – Targeted to EMGC_OMS1
util.EmctlUtil logp.251 – Targeting to EMGC_OMS2
util.EmctlUtil logp.251 – Targeted to EMGC_OMS3
util.EmctlUtil logp.251 – Targeting to EMGC_OMS3
util.EmctlUtil logp.251 – Targeted to EMGC_OMS3
util.EmctlUtil logp.251 – Targeting to EMGC_OMS4
util.EmctlUtil logp.251 – Targeted to EMGC_OMS4
util.EmctlUtil logp.251 – Targeting to cluster BIP_cluster
util.EmctlUtil logp.251 – Targeted to BIP_cluster
util.EmctlUtil logp.251 – createDataSource completed with status = true
oms.ChangeReposPwd logp.251 – Updating mds password in domain…
util.EmctlUtil logp.251 – Updating datasource : mds-sysman_mds
util.EmctlUtil logp.251 – Updating WLS datasource :mds-sysman_mds: Done startEditSession()
util.EmctlUtil logp.251 – Updating WLS datasource : DB datasource :mds-sysman_mds: IS found
util.EmctlUtil logp.251 – Updating password
util.EmctlUtil logp.251 – Updating WLS datasource :mds-sysman_mds: Invoked save
util.EmctlUtil logp.251 – Updating WLS datasource :mds-sysman_mds: Done activate()
oms.ChangeReposPwd logp.251 – MDS password in domain updated successfully.
oms.ChangeReposPwd logp.251 – Waiting for jps/opss re-initialization to complete
oms.ChangeReposPwd logp.251 – Updating repository password in Credential Store. Try#1
mas.CredStoreUtil logp.251 – deleteCredential : Exit mapName -EM keyName – REPOS_DETAILS status = true
mas.CredStoreUtil logp.251 – setGenericCredential : Exit mapName -EM keyName – REPOS_DETAILS
mas.CredStoreUtil logp.251 – getCredential : Got creds for mapName -EM_BIP keyName – EM_BIP_DETAILS
oms.ChangeReposPwd logp.251 – Getting repos conn as user sys as sysdba
oms.ChangeReposPwd logp.251 – Changing :SYSMAN_BIPLATFORM: in back-end
oms.ChangeReposPwd logp.251 – Changed :SYSMAN_BIPLATFORM: user in back-end
util.EmctlUtil logp.251 – Updating datasource : bip_datasource
util.EmctlUtil logp.251 – Updating WLS datasource :bip_datasource: Done startEditSession()
util.EmctlUtil logp.251 – Updating WLS datasource : DB datasource :bip_datasource: IS found
util.EmctlUtil logp.251 – Updating password
util.EmctlUtil logp.251 – Updating WLS datasource :bip_datasource: Invoked save
util.EmctlUtil logp.251 – Updating WLS datasource :bip_datasource: Done activate()
oms.ChangeReposPwd logp.251 – Commiting the getLockConn to release lock on EM_UPDATE_DATASOURCES_LOCK
oms.ChangeReposPwd logp.251 – Done commiting the getLockConn

Don’t Despair

The  emctl change_repos_password command will clean up any mess you’ve made.

It starts by making a clean password change on the backend (the repository), updates the local reference files, and then propogates the change to all OMS’s in your environment.

I deliberately messed up one of our lab servers and within minutes of invoking emctl change_repos_password all my handiwork was fixed and I was back in business.